EKS Advanced Debugging and Incident Response

Supported Versions: EKS 1.28+, kubectl 1.28+ Last Updated: February 23, 2026

For stable operation of Amazon EKS clusters, a systematic incident response framework and advanced debugging skills are essential. This document provides a practical guide for quickly diagnosing and resolving complex issues that occur in production environments.

Incident Response Framework
Control Plane Debugging
Node-Level Troubleshooting
Workload Debugging
Networking Diagnostics
Storage Troubleshooting
Observability Architecture
Failure Detection Architecture
Quick Reference
Next Steps

1. Incident Response Framework

First 5-Minute Checklist (Initial Triage)

The first 5 minutes are most critical when an incident occurs. Follow this checklist in order.

bash

# Step 1: Check cluster status (30 seconds)
kubectl cluster-info
kubectl get nodes -o wide
kubectl get pods -A --field-selector=status.phase!=Running

# Step 2: Check recent events (30 seconds)
kubectl get events -A --sort-by='.lastTimestamp' | tail -50

# Step 3: Core system pod status (30 seconds)
kubectl get pods -n kube-system
kubectl get pods -n amazon-vpc-cni-system

# Step 4: Check resource usage (30 seconds)
kubectl top nodes
kubectl top pods -A --sort-by=memory | head -20

# Step 5: Determine impact scope (2 minutes)
kubectl get deployments -A | grep -v "1/1\|2/2\|3/3"
kubectl get svc -A --field-selector=spec.type=LoadBalancer

Initial Diagnostic Script

bash

#!/bin/bash
# eks-triage.sh - EKS Emergency Diagnostic Script

echo "=== EKS Emergency Diagnosis Starting ==="
TIMESTAMP=$(date +%Y%m%d_%H%M%S)
OUTPUT_DIR="/tmp/eks-triage-$TIMESTAMP"
mkdir -p $OUTPUT_DIR

# Cluster information
echo "[1/6] Collecting cluster information..."
kubectl cluster-info dump --output-directory=$OUTPUT_DIR/cluster-info 2>/dev/null

# Node status
echo "[2/6] Checking node status..."
kubectl get nodes -o wide > $OUTPUT_DIR/nodes.txt
kubectl describe nodes > $OUTPUT_DIR/nodes-describe.txt

# Unhealthy pods
echo "[3/6] Listing unhealthy pods..."
kubectl get pods -A --field-selector=status.phase!=Running > $OUTPUT_DIR/unhealthy-pods.txt

# Recent events
echo "[4/6] Collecting recent events..."
kubectl get events -A --sort-by='.lastTimestamp' > $OUTPUT_DIR/events.txt

# Resource usage
echo "[5/6] Resource usage..."
kubectl top nodes > $OUTPUT_DIR/node-resources.txt 2>/dev/null
kubectl top pods -A > $OUTPUT_DIR/pod-resources.txt 2>/dev/null

# System components
echo "[6/6] System component status..."
kubectl get pods -n kube-system -o wide > $OUTPUT_DIR/kube-system.txt

echo "=== Diagnosis Complete: $OUTPUT_DIR ==="
tar -czf $OUTPUT_DIR.tar.gz -C /tmp eks-triage-$TIMESTAMP
echo "Archive: $OUTPUT_DIR.tar.gz"

Severity Matrix

Severity	Classification	Impact Scope	Response Time	Examples
P1	Critical	Complete service outage	Within 15 minutes	Control plane failure, all nodes NotReady
P2	High	Major functionality failure	Within 1 hour	Specific workload complete failure, network connectivity issues
P3	Medium	Partial impact	Within 4 hours	Some pod restarts, performance degradation
P4	Low	Minor issues	Within 24 hours	Log collection delay, non-critical monitoring alerts

Decision Tree for Rapid Problem Identification

2. Control Plane Debugging

EKS Control Plane Log Types

EKS sends 5 types of control plane logs to CloudWatch Logs.

Log Type	Description	Primary Use Cases
api	API server logs	API call tracing, error analysis
audit	Audit logs	Security audits, change tracking
authenticator	IAM authentication logs	Authentication failure debugging
controllerManager	Controller manager logs	Resource reconciliation issues
scheduler	Scheduler logs	Pod placement issues

Enabling Control Plane Logging

bash

# Enable all log types
aws eks update-cluster-config \
  --name my-cluster \
  --logging '{"clusterLogging":[{"types":["api","audit","authenticator","controllerManager","scheduler"],"enabled":true}]}'

# Check current settings
aws eks describe-cluster --name my-cluster \
  --query 'cluster.logging.clusterLogging'

CloudWatch Logs Insights Queries

Error Analysis Query

sql

-- API server error analysis
fields @timestamp, @message
| filter @logStream like /kube-apiserver/
| filter @message like /error|Error|ERROR/
| sort @timestamp desc
| limit 100

-- Error statistics for the last hour
fields @timestamp, @message
| filter @logStream like /kube-apiserver/
| filter @message like /error|Error|ERROR/
| stats count(*) as error_count by bin(5m)
| sort @timestamp desc

Authentication Failure Analysis

sql

-- IAM authentication failure tracking
fields @timestamp, @message
| filter @logStream like /authenticator/
| filter @message like /access denied|Unauthorized|forbidden/
| parse @message /user=(?<user>[^ ]+)/
| stats count(*) by user
| sort count(*) desc
| limit 20

-- Authentication history for specific user
fields @timestamp, @message
| filter @logStream like /authenticator/
| filter @message like /arn:aws:iam::123456789012:user\/specific-user/
| sort @timestamp desc
| limit 50

API Throttling Detection

sql

-- API throttling event detection
fields @timestamp, @message
| filter @logStream like /kube-apiserver/
| filter @message like /throttl|rate limit|429/
| stats count(*) as throttle_count by bin(1m)
| sort @timestamp desc

-- Identify sources of excessive API calls
fields @timestamp, @message
| filter @logStream like /audit/
| parse @message /"user":{"username":"(?<username>[^"]+)"/
| parse @message /"verb":"(?<verb>[^"]+)"/
| parse @message /"resource":"(?<resource>[^"]+)"/
| stats count(*) as call_count by username, verb, resource
| sort call_count desc
| limit 50

IAM Authentication Troubleshooting

Check aws-auth ConfigMap

bash

# View aws-auth ConfigMap
kubectl get configmap aws-auth -n kube-system -o yaml

# IAM role mapping example
apiVersion: v1
kind: ConfigMap
metadata:
  name: aws-auth
  namespace: kube-system
data:
  mapRoles: |
    - rolearn: arn:aws:iam::123456789012:role/eks-node-role
      username: system:node:{{EC2PrivateDNSName}}
      groups:
        - system:bootstrappers
        - system:nodes
    - rolearn: arn:aws:iam::123456789012:role/admin-role
      username: admin
      groups:
        - system:masters
  mapUsers: |
    - userarn: arn:aws:iam::123456789012:user/developer
      username: developer
      groups:
        - system:developers

Authentication Testing

bash

# Check current credentials
aws sts get-caller-identity

# Test EKS cluster authentication
aws eks get-token --cluster-name my-cluster | jq -r '.status.token' | cut -d'.' -f2 | base64 -d | jq

# Check kubectl authentication status
kubectl auth can-i get pods --all-namespaces
kubectl auth whoami

IRSA (IAM Roles for Service Accounts) Troubleshooting

bash

# Check service account annotation
kubectl get sa my-service-account -n my-namespace -o yaml

# Check OIDC provider
aws eks describe-cluster --name my-cluster \
  --query 'cluster.identity.oidc.issuer'

# Check IAM role trust policy
aws iam get-role --role-name my-irsa-role \
  --query 'Role.AssumeRolePolicyDocument'

IRSA Configuration Example

yaml

# Service account (IRSA enabled)
apiVersion: v1
kind: ServiceAccount
metadata:
  name: s3-access-sa
  namespace: default
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::123456789012:role/s3-access-role
---
# Using service account in Pod
apiVersion: v1
kind: Pod
metadata:
  name: s3-access-pod
spec:
  serviceAccountName: s3-access-sa
  containers:
  - name: app
    image: amazon/aws-cli
    command: ["aws", "s3", "ls"]

IRSA Debugging

bash

# Check credentials inside Pod
kubectl exec -it s3-access-pod -- env | grep AWS

# Check token mount
kubectl exec -it s3-access-pod -- cat /var/run/secrets/eks.amazonaws.com/serviceaccount/token

# Test STS call
kubectl exec -it s3-access-pod -- aws sts get-caller-identity

Pod Identity Troubleshooting

bash

# Check Pod Identity agent status
kubectl get pods -n kube-system -l app.kubernetes.io/name=eks-pod-identity-agent

# Check Pod Identity associations
aws eks list-pod-identity-associations --cluster-name my-cluster

# Create Pod Identity association
aws eks create-pod-identity-association \
  --cluster-name my-cluster \
  --namespace default \
  --service-account my-sa \
  --role-arn arn:aws:iam::123456789012:role/my-role

Service Account Token Expiration (1-hour default TTL)

yaml

# Extend token expiration time (max 24 hours)
apiVersion: v1
kind: Pod
metadata:
  name: extended-token-pod
spec:
  serviceAccountName: my-sa
  containers:
  - name: app
    image: my-app
    volumeMounts:
    - name: token
      mountPath: /var/run/secrets/tokens
  volumes:
  - name: token
    projected:
      sources:
      - serviceAccountToken:
          path: token
          expirationSeconds: 86400  # 24 hours
          audience: sts.amazonaws.com

EKS Add-on Error Patterns

bash

# Check Add-on status
aws eks describe-addon --cluster-name my-cluster --addon-name vpc-cni

# Add-on health status codes
# ACTIVE: Normal operation
# CREATE_FAILED: Creation failed
# DEGRADED: Performance degraded
# DELETE_FAILED: Deletion failed
# UPDATING: Updating
# DELETING: Deleting

# Detailed Add-on status query
aws eks describe-addon --cluster-name my-cluster --addon-name vpc-cni \
  --query 'addon.{Status:status,Health:health,Issues:health.issues}'

# Update problematic Add-on
aws eks update-addon \
  --cluster-name my-cluster \
  --addon-name vpc-cni \
  --resolve-conflicts OVERWRITE

3. Node-Level Troubleshooting

Node Join Failure Diagnosis (8 Common Causes)

#	Cause	Symptom	Resolution
1	Bootstrap script mismatch	Node doesn't appear in cluster	Verify AMI version matches cluster version
2	Security group misconfiguration	Node-control plane communication fails	Check inbound rules for ports 443, 10250
3	VPC DNS configuration issue	DNS resolution fails	Enable enableDnsHostnames, enableDnsSupport
4	Insufficient IAM role permissions	Authentication fails	Verify required policies attached to node role
5	Missing subnet tags	Node provisioning fails	Check kubernetes.io/cluster/<name> tag
6	Private subnet missing NAT	Image pull fails	Configure NAT Gateway or VPC endpoints
7	Instance profile not attached	EC2 launch fails	Check Launch Template configuration
8	User data script error	Bootstrap interrupted	Check /var/log/cloud-init-output.log

NotReady Node Decision Tree

kubelet/containerd Debugging via SSM

bash

# Start SSM session
aws ssm start-session --target i-1234567890abcdef0

# Check kubelet status
sudo systemctl status kubelet
sudo journalctl -u kubelet -f --no-pager | tail -100

# Check kubelet configuration
sudo cat /etc/kubernetes/kubelet/kubelet-config.json
sudo cat /var/lib/kubelet/kubeconfig

# Check containerd status
sudo systemctl status containerd
sudo journalctl -u containerd -f --no-pager | tail -50

# List containers
sudo crictl ps
sudo crictl ps -a  # Include terminated containers

# Check container logs
sudo crictl logs <container-id>

# List and clean images
sudo crictl images
sudo crictl rmi --prune  # Remove unused images

# Check disk usage
df -h
sudo du -sh /var/lib/containerd/*
sudo du -sh /var/log/*

Resource Pressure Conditions

bash

# Check node conditions
kubectl describe node <node-name> | grep -A 20 "Conditions:"

# Check specific pressure conditions
kubectl get nodes -o custom-columns=\
NAME:.metadata.name,\
DISK_PRESSURE:.status.conditions[?(@.type==\"DiskPressure\")].status,\
MEMORY_PRESSURE:.status.conditions[?(@.type==\"MemoryPressure\")].status,\
PID_PRESSURE:.status.conditions[?(@.type==\"PIDPressure\")].status

Resolving DiskPressure

bash

# After connecting to node via SSM
# Clean log files
sudo journalctl --vacuum-size=500M
sudo rm -rf /var/log/*.gz
sudo rm -rf /var/log/*.[0-9]

# Clean container images
sudo crictl rmi --prune

# Clean terminated containers
sudo crictl rm $(sudo crictl ps -a -q --state exited)

Resolving MemoryPressure

bash

# Identify pods with high memory usage
kubectl top pods -A --sort-by=memory | head -20

# Memory usage by node
kubectl top nodes

# Detailed memory usage (inside node)
free -h
cat /proc/meminfo | grep -E "MemTotal|MemFree|MemAvailable|Buffers|Cached"

Resolving PIDPressure

bash

# Check current PID usage (inside node)
cat /proc/sys/kernel/pid_max
ls /proc | grep -E "^[0-9]+$" | wc -l

# Processes using most PIDs
ps aux --sort=-nlwp | head -20

Karpenter Provisioning Issues

bash

# Check Karpenter controller logs
kubectl logs -n karpenter -l app.kubernetes.io/name=karpenter -c controller --tail=100

# Check NodePool status
kubectl get nodepools
kubectl describe nodepool default

# Check NodeClaim status
kubectl get nodeclaims
kubectl describe nodeclaim <name>

# Check when Karpenter isn't creating nodes
kubectl get events -n karpenter --sort-by='.lastTimestamp'

Karpenter Configuration Example

yaml

apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: default
spec:
  template:
    spec:
      requirements:
        - key: kubernetes.io/arch
          operator: In
          values: ["amd64"]
        - key: karpenter.sh/capacity-type
          operator: In
          values: ["spot", "on-demand"]
        - key: karpenter.k8s.aws/instance-category
          operator: In
          values: ["c", "m", "r"]
      nodeClassRef:
        group: karpenter.k8s.aws
        kind: EC2NodeClass
        name: default
  limits:
    cpu: 1000
    memory: 1000Gi
  disruption:
    consolidationPolicy: WhenUnderutilized
    consolidateAfter: 30s

Managed Node Group Error Codes

Error Code	Description	Resolution
AccessDenied	Insufficient IAM permissions	Check node role policies
AsgInstanceLaunchFailures	ASG instance launch failed	Check Launch Template, subnet capacity
ClusterUnreachable	Cannot connect to cluster	Check VPC endpoints, security groups
InsufficientFreeAddresses	IP address shortage	Expand subnet CIDR or add new subnets
NodeCreationFailure	Node creation failed	Check EC2 service limits, AMI availability

bash

# Check node group status
aws eks describe-nodegroup \
  --cluster-name my-cluster \
  --nodegroup-name my-nodegroup \
  --query 'nodegroup.{Status:status,Health:health}'

# Node group issue details
aws eks describe-nodegroup \
  --cluster-name my-cluster \
  --nodegroup-name my-nodegroup \
  --query 'nodegroup.health.issues'

Node Readiness Controller (Staged Boot Verification)

yaml

# Custom node readiness validation
apiVersion: v1
kind: ConfigMap
metadata:
  name: node-readiness-config
  namespace: kube-system
data:
  config.yaml: |
    checks:
      - name: cni-ready
        probe:
          exec:
            command: ["test", "-f", "/etc/cni/net.d/10-aws.conflist"]
        initialDelaySeconds: 5
        periodSeconds: 2
        failureThreshold: 30
      - name: containerd-ready
        probe:
          exec:
            command: ["crictl", "info"]
        initialDelaySeconds: 10
        periodSeconds: 5
        failureThreshold: 12

4. Workload Debugging

Pod State Flow Diagram

Basic Diagnostic Commands

bash

# Check pod status and events
kubectl describe pod <pod-name> -n <namespace>

# Check pod logs
kubectl logs <pod-name> -n <namespace>
kubectl logs <pod-name> -n <namespace> --previous  # Previous container logs
kubectl logs <pod-name> -n <namespace> -c <container-name>  # Specific container
kubectl logs <pod-name> -n <namespace> --tail=100 -f  # Real-time follow

# Check namespace events
kubectl get events -n <namespace> --sort-by='.lastTimestamp'

# Detailed pod status
kubectl get pod <pod-name> -n <namespace> -o yaml

kubectl debug Techniques

Ephemeral Containers

bash

# Add basic debug container
kubectl debug -it <pod-name> --image=busybox --target=<container-name>

# Network debugging container
kubectl debug -it <pod-name> --image=nicolaka/netshoot --target=<container-name>

# Share process namespace
kubectl debug -it <pod-name> --image=busybox --target=<container-name> -- sh

Pod Copying

bash

# Create identical pod copy
kubectl debug <pod-name> --copy-to=debug-pod --container=debugger --image=busybox

# Copy with changed container image
kubectl debug <pod-name> --copy-to=debug-pod --set-image=*=busybox

# Copy with shared process namespace
kubectl debug <pod-name> --copy-to=debug-pod --share-processes

Node Debugging

bash

# Run debug pod on node
kubectl debug node/<node-name> -it --image=ubuntu

# Access host filesystem
kubectl debug node/<node-name> -it --image=ubuntu -- chroot /host

# Run in node's network namespace
kubectl debug node/<node-name> -it --image=nicolaka/netshoot

Deployment Rollout Management

bash

# Check rollout status
kubectl rollout status deployment/<deployment-name> -n <namespace>

# Rollout history
kubectl rollout history deployment/<deployment-name> -n <namespace>

# Specific revision details
kubectl rollout history deployment/<deployment-name> --revision=2

# Rollback
kubectl rollout undo deployment/<deployment-name> -n <namespace>
kubectl rollout undo deployment/<deployment-name> --to-revision=2

# Pause/Resume rollout
kubectl rollout pause deployment/<deployment-name>
kubectl rollout resume deployment/<deployment-name>

# Force rollout (when image is same)
kubectl rollout restart deployment/<deployment-name>

HPA/VPA Scaling Issues

HPA Debugging

bash

# Check HPA status
kubectl get hpa -n <namespace>
kubectl describe hpa <hpa-name> -n <namespace>

# Check metrics collection status
kubectl get --raw "/apis/metrics.k8s.io/v1beta1/pods" | jq

# Check HPA events
kubectl get events -n <namespace> --field-selector involvedObject.name=<hpa-name>

HPA Configuration Example

yaml

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: app-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: app
  minReplicas: 2
  maxReplicas: 10
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 70
  - type: Resource
    resource:
      name: memory
      target:
        type: Utilization
        averageUtilization: 80
  behavior:
    scaleDown:
      stabilizationWindowSeconds: 300
      policies:
      - type: Percent
        value: 10
        periodSeconds: 60
    scaleUp:
      stabilizationWindowSeconds: 0
      policies:
      - type: Percent
        value: 100
        periodSeconds: 15

VPA Debugging

bash

# Check VPA status
kubectl get vpa -n <namespace>
kubectl describe vpa <vpa-name> -n <namespace>

# Check VPA recommendations
kubectl get vpa <vpa-name> -o jsonpath='{.status.recommendation}'

Probe Configuration Best Practices

yaml

apiVersion: v1
kind: Pod
metadata:
  name: app-with-probes
spec:
  containers:
  - name: app
    image: my-app:v1
    ports:
    - containerPort: 8080

    # Startup probe: Verify app initialization complete
    startupProbe:
      httpGet:
        path: /healthz
        port: 8080
      initialDelaySeconds: 10
      periodSeconds: 5
      failureThreshold: 30  # Wait up to 150 seconds

    # Liveness probe: Verify app is alive
    livenessProbe:
      httpGet:
        path: /healthz
        port: 8080
      initialDelaySeconds: 0  # Start immediately after startupProbe succeeds
      periodSeconds: 10
      timeoutSeconds: 5
      failureThreshold: 3

    # Readiness probe: Verify ready to receive traffic
    readinessProbe:
      httpGet:
        path: /ready
        port: 8080
      initialDelaySeconds: 0
      periodSeconds: 5
      timeoutSeconds: 3
      successThreshold: 1
      failureThreshold: 3

    resources:
      requests:
        memory: "256Mi"
        cpu: "250m"
      limits:
        memory: "512Mi"
        cpu: "500m"

5. Networking Diagnostics

VPC CNI Troubleshooting

bash

# Check VPC CNI version
kubectl describe daemonset aws-node -n kube-system | grep Image

# Check CNI pod status
kubectl get pods -n kube-system -l k8s-app=aws-node

# CNI pod logs
kubectl logs -n kube-system -l k8s-app=aws-node --tail=100

# Check ENI and IP allocation status
kubectl get pods -o wide
aws ec2 describe-network-interfaces --filters Name=description,Values="*eks*"

IP Exhaustion Troubleshooting

Enable Prefix Delegation Mode

bash

# Set environment variables
kubectl set env daemonset aws-node -n kube-system \
  ENABLE_PREFIX_DELEGATION=true \
  WARM_PREFIX_TARGET=1

# Verify
kubectl get daemonset aws-node -n kube-system -o yaml | grep -A 5 ENABLE_PREFIX

Add Secondary CIDR

bash

# Add Secondary CIDR to VPC
aws ec2 associate-vpc-cidr-block \
  --vpc-id vpc-1234567890abcdef0 \
  --cidr-block 100.64.0.0/16

# Create new subnet
aws ec2 create-subnet \
  --vpc-id vpc-1234567890abcdef0 \
  --cidr-block 100.64.0.0/24 \
  --availability-zone ap-northeast-2a

# Enable CNI custom networking
kubectl set env daemonset aws-node -n kube-system \
  AWS_VPC_K8S_CNI_CUSTOM_NETWORK_CFG=true

ENIConfig Configuration

yaml

apiVersion: crd.k8s.amazonaws.com/v1alpha1
kind: ENIConfig
metadata:
  name: ap-northeast-2a
spec:
  securityGroups:
    - sg-0123456789abcdef0
  subnet: subnet-0123456789abcdef0
---
apiVersion: crd.k8s.amazonaws.com/v1alpha1
kind: ENIConfig
metadata:
  name: ap-northeast-2c
spec:
  securityGroups:
    - sg-0123456789abcdef0
  subnet: subnet-0fedcba9876543210

CoreDNS Configuration Issues

bash

# CoreDNS pod status
kubectl get pods -n kube-system -l k8s-app=kube-dns

# CoreDNS logs
kubectl logs -n kube-system -l k8s-app=kube-dns --tail=100

# Check CoreDNS ConfigMap
kubectl get configmap coredns -n kube-system -o yaml

# DNS resolution test
kubectl run dns-test --image=busybox:1.28 --rm -it --restart=Never -- nslookup kubernetes.default

ndots Issue and Solution

yaml

# Issue: DNS query delay due to default ndots=5
# Solution: Adjust ndots value in Pod

apiVersion: v1
kind: Pod
metadata:
  name: optimized-dns-pod
spec:
  dnsConfig:
    options:
      - name: ndots
        value: "2"
      - name: single-request-reopen
      - name: timeout
        value: "2"
      - name: attempts
        value: "3"
  containers:
  - name: app
    image: my-app

CoreDNS Performance Optimization

yaml

apiVersion: v1
kind: ConfigMap
metadata:
  name: coredns
  namespace: kube-system
data:
  Corefile: |
    .:53 {
        errors
        health {
            lameduck 5s
        }
        ready
        kubernetes cluster.local in-addr.arpa ip6.arpa {
            pods insecure
            fallthrough in-addr.arpa ip6.arpa
            ttl 30
        }
        prometheus :9153
        forward . /etc/resolv.conf {
            max_concurrent 1000
        }
        cache 30
        loop
        reload
        loadbalance
    }

Service Endpoint Verification

bash

# Check service endpoints
kubectl get endpoints <service-name> -n <namespace>
kubectl describe endpoints <service-name> -n <namespace>

# Service connectivity test
kubectl run curl-test --image=curlimages/curl --rm -it --restart=Never -- \
  curl -v http://<service-name>.<namespace>.svc.cluster.local:<port>

# Service DNS resolution
kubectl run dns-test --image=busybox --rm -it --restart=Never -- \
  nslookup <service-name>.<namespace>.svc.cluster.local

NetworkPolicy AND/OR Logic Debugging

yaml

# Example: Complex NetworkPolicy
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: complex-policy
  namespace: production
spec:
  podSelector:
    matchLabels:
      app: api-server
  policyTypes:
  - Ingress
  - Egress
  ingress:
  # Rule 1: Allow traffic from frontend pods
  - from:
    - podSelector:
        matchLabels:
          app: frontend
    ports:
    - port: 8080
  # Rule 2: Allow traffic from monitoring namespace (OR)
  - from:
    - namespaceSelector:
        matchLabels:
          purpose: monitoring
    ports:
    - port: 9090
  egress:
  # Allow only outbound to database
  - to:
    - podSelector:
        matchLabels:
          app: database
    ports:
    - port: 5432

bash

# NetworkPolicy debugging
kubectl get networkpolicy -n <namespace> -o yaml
kubectl describe networkpolicy <policy-name> -n <namespace>

# Connectivity test
kubectl exec -it <source-pod> -- nc -zv <target-service> <port>
kubectl exec -it <source-pod> -- curl -v --connect-timeout 5 http://<target>:<port>

Live Debugging with netshoot Container

bash

# Run netshoot debug pod
kubectl run netshoot --image=nicolaka/netshoot -it --rm -- /bin/bash

# Tools available inside:
# - curl, wget: HTTP testing
# - dig, nslookup: DNS debugging
# - tcpdump: Packet capture
# - iperf3: Network performance testing
# - mtr, traceroute: Path tracing
# - ss, netstat: Socket status

# DNS debugging
dig +short kubernetes.default.svc.cluster.local
dig +trace google.com

# TCP connection test
nc -zv <service-ip> <port>

# Detailed HTTP test
curl -v --connect-timeout 5 http://<service>:<port>/health

# Packet capture (requires root privileges)
tcpdump -i any port 80 -nn

# Network performance test
iperf3 -c <target-ip> -p 5201

6. Storage Troubleshooting

EBS CSI Driver Error Patterns

bash

# Check EBS CSI Driver pod status
kubectl get pods -n kube-system -l app.kubernetes.io/name=aws-ebs-csi-driver

# Controller logs
kubectl logs -n kube-system -l app=ebs-csi-controller -c ebs-plugin --tail=100

# Node driver logs
kubectl logs -n kube-system -l app=ebs-csi-node -c ebs-plugin --tail=100

# CSI driver status
kubectl get csidrivers
kubectl describe csidriver ebs.csi.aws.com

IRSA Permission Configuration

json

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "ec2:CreateSnapshot",
        "ec2:AttachVolume",
        "ec2:DetachVolume",
        "ec2:ModifyVolume",
        "ec2:DescribeAvailabilityZones",
        "ec2:DescribeInstances",
        "ec2:DescribeSnapshots",
        "ec2:DescribeTags",
        "ec2:DescribeVolumes",
        "ec2:DescribeVolumesModifications",
        "ec2:CreateTags",
        "ec2:DeleteTags",
        "ec2:CreateVolume",
        "ec2:DeleteVolume",
        "ec2:DeleteSnapshot"
      ],
      "Resource": "*"
    }
  ]
}

EFS Mount Target Configuration Issues

bash

# EFS CSI Driver status
kubectl get pods -n kube-system -l app.kubernetes.io/name=aws-efs-csi-driver

# Check EFS mount targets
aws efs describe-mount-targets --file-system-id fs-1234567890abcdef0

# Check security groups (NFS port 2049)
aws ec2 describe-security-groups --group-ids sg-0123456789abcdef0

EFS StorageClass and PVC

yaml

kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: efs-sc
provisioner: efs.csi.aws.com
parameters:
  provisioningMode: efs-ap
  fileSystemId: fs-1234567890abcdef0
  directoryPerms: "700"
  gidRangeStart: "1000"
  gidRangeEnd: "2000"
  basePath: "/dynamic_provisioning"
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: efs-claim
spec:
  accessModes:
    - ReadWriteMany
  storageClassName: efs-sc
  resources:
    requests:
      storage: 5Gi

PVC/PV State Management

bash

# Check PVC status
kubectl get pvc -A
kubectl describe pvc <pvc-name> -n <namespace>

# Check PV status
kubectl get pv
kubectl describe pv <pv-name>

# Check binding issues
kubectl get pvc -A -o custom-columns=\
NAME:.metadata.name,\
STATUS:.status.phase,\
VOLUME:.spec.volumeName,\
STORAGECLASS:.spec.storageClassName

Finalizer Handling

bash

# When PVC won't delete (Terminating state)
# Check pods using it
kubectl get pods -A -o json | jq -r '.items[] | select(.spec.volumes[]?.persistentVolumeClaim.claimName == "<pvc-name>") | .metadata.name'

# Remove Finalizer (Caution: Possible data loss)
kubectl patch pvc <pvc-name> -n <namespace> -p '{"metadata":{"finalizers":null}}'

# Remove PV Finalizer
kubectl patch pv <pv-name> -p '{"metadata":{"finalizers":null}}'

AZ Matching with WaitForFirstConsumer

yaml

# StorageClass with WaitForFirstConsumer
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: ebs-sc-waitforfirstconsumer
provisioner: ebs.csi.aws.com
parameters:
  type: gp3
  encrypted: "true"
volumeBindingMode: WaitForFirstConsumer  # Create volume after Pod is scheduled
allowedTopologies:
- matchLabelExpressions:
  - key: topology.kubernetes.io/zone
    values:
    - ap-northeast-2a
    - ap-northeast-2c

bash

# Check volume-pod AZ mismatch
kubectl get pv -o custom-columns=\
NAME:.metadata.name,\
ZONE:.spec.nodeAffinity.required.nodeSelectorTerms[0].matchExpressions[0].values[0]

kubectl get pods -o custom-columns=\
NAME:.metadata.name,\
NODE:.spec.nodeName,\
ZONE:'{.spec.nodeAffinity}'

7. Observability Architecture

Container Insights Setup

bash

# Install CloudWatch Agent and Fluent Bit
aws eks create-addon \
  --cluster-name my-cluster \
  --addon-name amazon-cloudwatch-observability \
  --addon-version v1.0.0-eksbuild.1

# Or install with Helm
helm repo add aws-observability https://aws-observability.github.io/helm-charts
helm install amazon-cloudwatch-observability \
  aws-observability/amazon-cloudwatch-observability \
  --namespace amazon-cloudwatch --create-namespace \
  --set clusterName=my-cluster \
  --set region=ap-northeast-2

PromQL Query Examples

CPU Throttling Detection

promql

# CPU throttling ratio
sum(rate(container_cpu_cfs_throttled_periods_total{container!=""}[5m])) by (pod, namespace)
/
sum(rate(container_cpu_cfs_periods_total{container!=""}[5m])) by (pod, namespace)
> 0.5

# Top 10 pods with high CPU throttling
topk(10,
  sum(rate(container_cpu_cfs_throttled_periods_total{container!=""}[5m])) by (pod, namespace)
  /
  sum(rate(container_cpu_cfs_periods_total{container!=""}[5m])) by (pod, namespace)
)

OOMKilled Event Detection

promql

# Pods with OOMKilled
kube_pod_container_status_last_terminated_reason{reason="OOMKilled"} == 1

# OOMKilled count in last hour
sum(changes(kube_pod_container_status_restarts_total[1h])) by (pod, namespace)
* on (pod, namespace) group_left
kube_pod_container_status_last_terminated_reason{reason="OOMKilled"}

# Pods with high memory usage (OOM risk)
(
  sum(container_memory_working_set_bytes{container!=""}) by (pod, namespace)
  /
  sum(kube_pod_container_resource_limits{resource="memory"}) by (pod, namespace)
) > 0.9

Pod Restart Rate

promql

# Restart count in last hour
sum(increase(kube_pod_container_status_restarts_total[1h])) by (pod, namespace) > 3

# Top 10 pods with most restarts
topk(10, sum(increase(kube_pod_container_status_restarts_total[1h])) by (pod, namespace))

# Pods in CrashLoopBackOff state
kube_pod_container_status_waiting_reason{reason="CrashLoopBackOff"} == 1

CloudWatch Logs Insights Search Patterns

sql

-- Error log search
fields @timestamp, @message, kubernetes.pod_name, kubernetes.namespace_name
| filter @message like /error|Error|ERROR|exception|Exception|EXCEPTION/
| sort @timestamp desc
| limit 100

-- Specific pod logs
fields @timestamp, @message
| filter kubernetes.pod_name = "my-pod-name"
| sort @timestamp desc
| limit 500

-- Response time analysis (when app logs include response time)
fields @timestamp, @message
| parse @message /response_time=(?<response_time>\d+)ms/
| stats avg(response_time) as avg_response, max(response_time) as max_response by bin(5m)

-- OOMKilled event tracking
fields @timestamp, @message
| filter @message like /OOMKilled|Out of memory|oom-kill/
| sort @timestamp desc
| limit 50

PrometheusRule Example

yaml

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: eks-alerts
  namespace: monitoring
spec:
  groups:
  - name: eks-node-alerts
    rules:
    - alert: NodeNotReady
      expr: kube_node_status_condition{condition="Ready",status="true"} == 0
      for: 5m
      labels:
        severity: critical
      annotations:
        summary: "Node {{ $labels.node }} is in NotReady state"
        description: "Node has been in NotReady state for more than 5 minutes. Immediate attention required."

    - alert: NodeMemoryPressure
      expr: kube_node_status_condition{condition="MemoryPressure",status="true"} == 1
      for: 5m
      labels:
        severity: warning
      annotations:
        summary: "Memory pressure on node {{ $labels.node }}"

    - alert: NodeDiskPressure
      expr: kube_node_status_condition{condition="DiskPressure",status="true"} == 1
      for: 5m
      labels:
        severity: warning
      annotations:
        summary: "Disk pressure on node {{ $labels.node }}"

  - name: eks-pod-alerts
    rules:
    - alert: PodCrashLooping
      expr: rate(kube_pod_container_status_restarts_total[15m]) * 60 * 15 > 3
      for: 5m
      labels:
        severity: warning
      annotations:
        summary: "Pod {{ $labels.namespace }}/{{ $labels.pod }} is crash looping"

    - alert: PodNotReady
      expr: |
        sum by (namespace, pod) (
          max by(namespace, pod) (kube_pod_status_phase{phase=~"Pending|Unknown"}) *
          on(namespace, pod) group_left(owner_kind)
          topk by(namespace, pod) (1, max by(namespace, pod, owner_kind) (kube_pod_owner{owner_kind!="Job"}))
        ) > 0
      for: 15m
      labels:
        severity: warning
      annotations:
        summary: "Pod {{ $labels.namespace }}/{{ $labels.pod }} has not been Ready for more than 15 minutes"

    - alert: ContainerOOMKilled
      expr: kube_pod_container_status_last_terminated_reason{reason="OOMKilled"} == 1
      for: 0m
      labels:
        severity: warning
      annotations:
        summary: "Container {{ $labels.namespace }}/{{ $labels.pod }}/{{ $labels.container }} was OOMKilled"

  - name: eks-resource-alerts
    rules:
    - alert: HighCPUThrottling
      expr: |
        sum(rate(container_cpu_cfs_throttled_periods_total{container!=""}[5m])) by (pod, namespace)
        /
        sum(rate(container_cpu_cfs_periods_total{container!=""}[5m])) by (pod, namespace)
        > 0.5
      for: 10m
      labels:
        severity: warning
      annotations:
        summary: "CPU throttling for pod {{ $labels.namespace }}/{{ $labels.pod }} exceeds 50%"

ADOT (AWS Distro for OpenTelemetry) Configuration

yaml

# ADOT Collector configuration
apiVersion: opentelemetry.io/v1alpha1
kind: OpenTelemetryCollector
metadata:
  name: adot-collector
  namespace: opentelemetry
spec:
  mode: deployment
  serviceAccount: adot-collector
  config: |
    receivers:
      otlp:
        protocols:
          grpc:
            endpoint: 0.0.0.0:4317
          http:
            endpoint: 0.0.0.0:4318
      prometheus:
        config:
          scrape_configs:
            - job_name: 'kubernetes-pods'
              kubernetes_sd_configs:
                - role: pod
              relabel_configs:
                - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
                  action: keep
                  regex: true

    processors:
      batch:
        timeout: 30s
        send_batch_size: 8192
      memory_limiter:
        limit_mib: 500
        spike_limit_mib: 100
        check_interval: 5s

    exporters:
      awsxray:
        region: ap-northeast-2
      awsemf:
        region: ap-northeast-2
        namespace: ContainerInsights
        log_group_name: '/aws/containerinsights/{ClusterName}/performance'
      prometheusremotewrite:
        endpoint: "https://aps-workspaces.ap-northeast-2.amazonaws.com/workspaces/ws-xxxxx/api/v1/remote_write"
        auth:
          authenticator: sigv4auth
        resource_to_telemetry_conversion:
          enabled: true

    extensions:
      sigv4auth:
        region: ap-northeast-2
        service: "aps"

    service:
      extensions: [sigv4auth]
      pipelines:
        traces:
          receivers: [otlp]
          processors: [batch, memory_limiter]
          exporters: [awsxray]
        metrics:
          receivers: [otlp, prometheus]
          processors: [batch, memory_limiter]
          exporters: [awsemf, prometheusremotewrite]

8. Failure Detection Architecture

4-Layer Detection Pipeline

Reference Architecture 1: AWS Native

yaml

# Fluent Bit ConfigMap for CloudWatch
apiVersion: v1
kind: ConfigMap
metadata:
  name: fluent-bit-config
  namespace: amazon-cloudwatch
data:
  fluent-bit.conf: |
    [SERVICE]
        Flush         5
        Grace         30
        Log_Level     info
        Daemon        off
        Parsers_File  parsers.conf

    [INPUT]
        Name              tail
        Tag               kube.*
        Path              /var/log/containers/*.log
        Parser            docker
        DB                /var/fluent-bit/state/flb_kube.db
        Mem_Buf_Limit     50MB
        Skip_Long_Lines   On
        Refresh_Interval  10

    [FILTER]
        Name                kubernetes
        Match               kube.*
        Kube_URL            https://kubernetes.default.svc:443
        Kube_CA_File        /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
        Kube_Token_File     /var/run/secrets/kubernetes.io/serviceaccount/token
        Kube_Tag_Prefix     kube.var.log.containers.
        Merge_Log           On
        Merge_Log_Key       log_processed
        K8S-Logging.Parser  On
        K8S-Logging.Exclude Off

    [OUTPUT]
        Name                cloudwatch_logs
        Match               kube.*
        region              ap-northeast-2
        log_group_name      /aws/eks/my-cluster/containers
        log_stream_prefix   fluentbit-
        auto_create_group   true

Reference Architecture 2: Open Source Stack

yaml

# Prometheus + Alertmanager + Grafana
---
# Alertmanager configuration
apiVersion: v1
kind: ConfigMap
metadata:
  name: alertmanager-config
  namespace: monitoring
data:
  alertmanager.yml: |
    global:
      resolve_timeout: 5m
      slack_api_url: 'https://hooks.slack.com/services/xxx/yyy/zzz'

    route:
      group_by: ['alertname', 'namespace', 'severity']
      group_wait: 30s
      group_interval: 5m
      repeat_interval: 4h
      receiver: 'default-receiver'
      routes:
        - match:
            severity: critical
          receiver: 'pagerduty-critical'
          continue: true
        - match:
            severity: warning
          receiver: 'slack-warnings'

    receivers:
      - name: 'default-receiver'
        slack_configs:
          - channel: '#alerts-default'
            send_resolved: true

      - name: 'pagerduty-critical'
        pagerduty_configs:
          - service_key: '<pagerduty-service-key>'
            severity: critical

      - name: 'slack-warnings'
        slack_configs:
          - channel: '#alerts-warnings'
            send_resolved: true
            title: '{{ .Status | toUpper }}: {{ .CommonAnnotations.summary }}'
            text: '{{ .CommonAnnotations.description }}'

    inhibit_rules:
      - source_match:
          severity: 'critical'
        target_match:
          severity: 'warning'
        equal: ['alertname', 'namespace']

Detection Patterns

Threshold-based Detection

yaml

# CloudWatch Alarm
aws cloudwatch put-metric-alarm \
  --alarm-name "EKS-High-CPU-Usage" \
  --alarm-description "EKS node CPU usage exceeds 80%" \
  --metric-name node_cpu_utilization \
  --namespace ContainerInsights \
  --statistic Average \
  --period 300 \
  --threshold 80 \
  --comparison-operator GreaterThanThreshold \
  --dimensions Name=ClusterName,Value=my-cluster \
  --evaluation-periods 3 \
  --alarm-actions arn:aws:sns:ap-northeast-2:123456789012:eks-alerts

Anomaly Detection

yaml

# CloudWatch Anomaly Detection Alarm
aws cloudwatch put-anomaly-detector \
  --namespace ContainerInsights \
  --metric-name pod_cpu_utilization \
  --stat Average \
  --dimensions Name=ClusterName,Value=my-cluster

aws cloudwatch put-metric-alarm \
  --alarm-name "EKS-Anomaly-CPU" \
  --alarm-description "Abnormal CPU usage pattern detected" \
  --metrics '[
    {
      "Id": "m1",
      "MetricStat": {
        "Metric": {
          "Namespace": "ContainerInsights",
          "MetricName": "pod_cpu_utilization",
          "Dimensions": [{"Name": "ClusterName", "Value": "my-cluster"}]
        },
        "Period": 300,
        "Stat": "Average"
      }
    },
    {
      "Id": "ad1",
      "Expression": "ANOMALY_DETECTION_BAND(m1, 2)"
    }
  ]' \
  --threshold-metric-id ad1 \
  --comparison-operator LessThanLowerOrGreaterThanUpperThreshold \
  --evaluation-periods 3 \
  --alarm-actions arn:aws:sns:ap-northeast-2:123456789012:eks-anomaly-alerts

Composite Alarm

bash

# Create composite alarm
aws cloudwatch put-composite-alarm \
  --alarm-name "EKS-Critical-State" \
  --alarm-description "Cluster critical state" \
  --alarm-rule "ALARM(EKS-High-CPU-Usage) AND ALARM(EKS-High-Memory-Usage)" \
  --alarm-actions arn:aws:sns:ap-northeast-2:123456789012:eks-critical-alerts \
  --ok-actions arn:aws:sns:ap-northeast-2:123456789012:eks-resolved

Log-based Metrics

bash

# Extract metrics from logs
aws logs put-metric-filter \
  --log-group-name "/aws/eks/my-cluster/containers" \
  --filter-name "ErrorCount" \
  --filter-pattern "[..., level=\"ERROR\", ...]" \
  --metric-transformations \
    metricName=ApplicationErrors,metricNamespace=EKS/Application,metricValue=1

Maturity Model

Level	Description	MTTD Target	Key Capabilities
Level 1	Basic	30 minutes	Basic metric alerts, manual log search
Level 2	Reactive	15 minutes	Threshold alerts, log-based alerts, basic dashboards
Level 3	Proactive	5 minutes	Anomaly detection, composite alarms, automated runbooks
Level 4	Predictive	2 minutes	ML-based prediction, auto-remediation, chaos engineering

EventBridge + Lambda Auto-Remediation

yaml

# EventBridge Rule
{
  "source": ["aws.cloudwatch"],
  "detail-type": ["CloudWatch Alarm State Change"],
  "detail": {
    "alarmName": ["EKS-Pod-CrashLooping"],
    "state": {
      "value": ["ALARM"]
    }
  }
}

python

# Lambda auto-remediation function
import boto3
import json
from kubernetes import client, config

def lambda_handler(event, context):
    alarm_name = event['detail']['alarmName']

    # Get EKS cluster credentials
    eks = boto3.client('eks')
    cluster_info = eks.describe_cluster(name='my-cluster')

    # Configure Kubernetes client
    # ... (kubeconfig setup)

    # Restart CrashLooping pod
    if 'CrashLooping' in alarm_name:
        v1 = client.CoreV1Api()
        # Delete problem pod (Deployment will recreate)
        v1.delete_namespaced_pod(
            name=extract_pod_name(event),
            namespace=extract_namespace(event),
            body=client.V1DeleteOptions()
        )

    return {
        'statusCode': 200,
        'body': json.dumps('Auto-remediation executed')
    }

Alert Channel Matrix by Severity

Severity	Slack	PagerDuty	Email	SMS	Auto-Remediation
P1 Critical	#incidents	Immediate	Team Lead	On-call	Yes
P2 High	#alerts-high	15min delay	Team	-	Conditional
P3 Medium	#alerts	-	Team	-	No
P4 Low	#alerts-low	-	Daily digest	-	No

9. Quick Reference

Error Pattern Lookup Table

Symptom	Cause	Resolution
CrashLoopBackOff	Application crash, invalid command, missing dependencies	`kubectl logs --previous`, review application code/config
ImagePullBackOff	Image not found, wrong tag, authentication failure	Verify image name, review `imagePullSecrets`
OOMKilled	Memory limit exceeded	Increase memory limit, fix memory leak
CreateContainerConfigError	Missing ConfigMap/Secret, invalid reference	`kubectl describe pod`, verify referenced resources exist
Pending (resources)	No node with sufficient CPU/memory	Scale up nodes, adjust resource requests
Pending (scheduling)	nodeSelector, affinity, taint mismatch	Check Events section in `kubectl describe pod`
ContainerCreating (delayed)	Volume mount failure, network plugin issue	Check PVC status, CNI pod status
ErrImagePull	Cannot connect to image registry	Check network connectivity, ECR endpoints
RunContainerError	Invalid container config, securityContext issue	`kubectl describe pod`, review securityContext
PostStartHookError	postStart hook failed	Review hook command, adjust timeout
PreStopHookError	preStop hook failed	Review hook command, adjust terminationGracePeriodSeconds
FailedScheduling	Resource shortage, PVC binding pending	Check node resources, PVC status
FailedMount	Volume mount failed, CSI driver issue	Check CSI driver logs, PV/PVC status
NetworkNotReady	CNI plugin not ready	Check aws-node pod status, CNI logs
NodeNotReady	kubelet issue, network disconnection	Check kubelet logs, node status
Evicted	Node resource pressure (disk, memory)	Clean node resources, adjust resource limits
BackOff	Retry backoff state	Check previous error logs, resolve root cause
InvalidImageName	Invalid image name format	Verify image name syntax

Essential kubectl Commands Cheatsheet

bash

# Cluster status
kubectl cluster-info
kubectl get nodes -o wide
kubectl top nodes

# Pod debugging
kubectl get pods -A -o wide
kubectl describe pod <pod> -n <ns>
kubectl logs <pod> -n <ns> --tail=100 -f
kubectl logs <pod> -n <ns> --previous
kubectl exec -it <pod> -n <ns> -- /bin/sh

# Events
kubectl get events -A --sort-by='.lastTimestamp'
kubectl get events -n <ns> --field-selector type=Warning

# Resource usage
kubectl top pods -A --sort-by=memory
kubectl top pods -A --sort-by=cpu

# Debug containers
kubectl debug -it <pod> --image=busybox --target=<container>
kubectl debug node/<node> -it --image=ubuntu

# Network test
kubectl run test --image=nicolaka/netshoot -it --rm -- /bin/bash

# Force delete
kubectl delete pod <pod> -n <ns> --grace-period=0 --force

# Rollout
kubectl rollout status deployment/<deploy> -n <ns>
kubectl rollout undo deployment/<deploy> -n <ns>
kubectl rollout restart deployment/<deploy> -n <ns>

# Scaling
kubectl scale deployment <deploy> -n <ns> --replicas=3

# ConfigMap/Secret
kubectl get configmap -n <ns> -o yaml
kubectl get secret -n <ns> -o yaml

# Service endpoints
kubectl get endpoints -n <ns>
kubectl describe svc <service> -n <ns>

Tool Recommendations

Tool	Purpose	Installation/Usage
netshoot	Network debugging	`kubectl run net --image=nicolaka/netshoot -it --rm`
eks-node-viewer	Node resource visualization	`go install github.com/awslabs/eks-node-viewer/cmd/eks-node-viewer@latest`
crictl	Container runtime debugging	On node: `sudo crictl ps`, `sudo crictl logs`
kubeval	YAML validation	`kubeval deployment.yaml`
stern	Multi-pod logging	`stern <pod-pattern> -n <namespace>`
k9s	TUI cluster management	`k9s -n <namespace>`
kubectx/kubens	Context/namespace switching	`kubectx <context>`, `kubens <namespace>`

EKS Log Collector (For AWS Support)

bash

# Download and run EKS Log Collector
curl -O https://raw.githubusercontent.com/awslabs/amazon-eks-ami/master/log-collector-script/linux/eks-log-collector.sh
chmod +x eks-log-collector.sh

# Run log collection
sudo ./eks-log-collector.sh

# Collected logs are saved to /var/log/eks_i-xxxx_$(date +%Y-%m-%d_%H-%M-%S).tar.gz
# Attach to AWS Support case for submission

Information collected:

System information (OS, kernel, memory, CPU)
kubelet logs and configuration
containerd logs and configuration
CNI plugin logs
Network configuration (iptables, routing)
Disk usage

10. Next Steps

Quiz

To test your understanding of the content covered in this document, try the EKS Advanced Debugging Quiz.

Next Document

To learn how to integrate EKS clusters with on-premises environments, see EKS Hybrid Nodes.

Traffic Management

Security

Observability

Resilience

Advanced

Comparison Guide

ArgoCD

Metrics

Logging

Tracing

Alerting

Observability End-to-End Labs

Cilium Deep Dive

Calico Deep Dive

Istio Detailed Quiz

Linkerd Detailed Quiz

Cilium Service Mesh Detailed Quiz

ArgoCD Detailed Quiz

Metrics

Logging

Tracing

Alerting

Observability Labs

EKS Advanced Debugging and Incident Response ​

Table of Contents ​

1. Incident Response Framework ​

First 5-Minute Checklist (Initial Triage) ​

Initial Diagnostic Script ​

Severity Matrix ​

Decision Tree for Rapid Problem Identification ​

2. Control Plane Debugging ​

EKS Control Plane Log Types ​

Enabling Control Plane Logging ​

CloudWatch Logs Insights Queries ​

Error Analysis Query ​

Authentication Failure Analysis ​

API Throttling Detection ​

IAM Authentication Troubleshooting ​

Check aws-auth ConfigMap ​

Authentication Testing ​

IRSA (IAM Roles for Service Accounts) Troubleshooting ​

IRSA Configuration Example ​

IRSA Debugging ​

Pod Identity Troubleshooting ​

Service Account Token Expiration (1-hour default TTL) ​

EKS Add-on Error Patterns ​

3. Node-Level Troubleshooting ​

Node Join Failure Diagnosis (8 Common Causes) ​

NotReady Node Decision Tree ​

kubelet/containerd Debugging via SSM ​

Resource Pressure Conditions ​

Resolving DiskPressure ​

Resolving MemoryPressure ​

Resolving PIDPressure ​

Karpenter Provisioning Issues ​

Karpenter Configuration Example ​

Managed Node Group Error Codes ​

Node Readiness Controller (Staged Boot Verification) ​

4. Workload Debugging ​

Pod State Flow Diagram ​

Basic Diagnostic Commands ​

kubectl debug Techniques ​

Ephemeral Containers ​

Pod Copying ​

Node Debugging ​

Deployment Rollout Management ​

HPA/VPA Scaling Issues ​

HPA Debugging ​

HPA Configuration Example ​

VPA Debugging ​

Probe Configuration Best Practices ​

5. Networking Diagnostics ​

VPC CNI Troubleshooting ​

IP Exhaustion Troubleshooting ​

Enable Prefix Delegation Mode ​

Add Secondary CIDR ​

ENIConfig Configuration ​

CoreDNS Configuration Issues ​

ndots Issue and Solution ​

CoreDNS Performance Optimization ​

EKS Advanced Debugging and Incident Response

Table of Contents

1. Incident Response Framework

First 5-Minute Checklist (Initial Triage)

Initial Diagnostic Script

Severity Matrix

Decision Tree for Rapid Problem Identification

2. Control Plane Debugging

EKS Control Plane Log Types

Enabling Control Plane Logging

CloudWatch Logs Insights Queries

Error Analysis Query

Authentication Failure Analysis

API Throttling Detection

IAM Authentication Troubleshooting

Check aws-auth ConfigMap

Authentication Testing

IRSA (IAM Roles for Service Accounts) Troubleshooting

IRSA Configuration Example

IRSA Debugging

Pod Identity Troubleshooting

Service Account Token Expiration (1-hour default TTL)

EKS Add-on Error Patterns

3. Node-Level Troubleshooting

Node Join Failure Diagnosis (8 Common Causes)

NotReady Node Decision Tree

kubelet/containerd Debugging via SSM

Resource Pressure Conditions

Resolving DiskPressure

Resolving MemoryPressure

Resolving PIDPressure

Karpenter Provisioning Issues

Karpenter Configuration Example

Managed Node Group Error Codes

Node Readiness Controller (Staged Boot Verification)

4. Workload Debugging

Pod State Flow Diagram

Basic Diagnostic Commands

kubectl debug Techniques

Ephemeral Containers

Pod Copying

Node Debugging

Deployment Rollout Management

HPA/VPA Scaling Issues

HPA Debugging

HPA Configuration Example

VPA Debugging

Probe Configuration Best Practices

5. Networking Diagnostics

VPC CNI Troubleshooting

IP Exhaustion Troubleshooting

Enable Prefix Delegation Mode

Add Secondary CIDR

ENIConfig Configuration

CoreDNS Configuration Issues

ndots Issue and Solution

CoreDNS Performance Optimization

Service Endpoint Verification

NetworkPolicy AND/OR Logic Debugging

Live Debugging with netshoot Container

6. Storage Troubleshooting

EBS CSI Driver Error Patterns

IRSA Permission Configuration

EFS Mount Target Configuration Issues

EFS StorageClass and PVC

PVC/PV State Management

Finalizer Handling

AZ Matching with WaitForFirstConsumer

7. Observability Architecture

Container Insights Setup

PromQL Query Examples