EKS Upgrades: Auto Mode Zero-Downtime Upgrade

Supported Versions: EKS 1.28+, Terraform 1.5+, Karpenter 1.0+ Last Updated: July 10, 2026

< Previous: Resource Optimization | Table of Contents | Next: Event Capacity Planning >

Overview

This document provides a comprehensive guide for upgrading EKS Auto Mode clusters with zero downtime. It covers version planning, pre-upgrade validation, the upgrade process itself, and post-upgrade verification. Both in-place upgrades and blue/green strategies are detailed with practical code examples.

1. Upgrade Planning

Kubernetes Version Support Policy

Kubernetes follows a N-3 support policy, meaning the project maintains release branches for the most recent three minor releases plus the current release.

Version	Release Date	End of Support	Status
1.32	Dec 2024	Dec 2025	Current
1.31	Aug 2024	Aug 2025	Supported
1.30	Apr 2024	Apr 2025	Supported
1.29	Dec 2023	Feb 2025	Extended Support
1.28	Aug 2023	Nov 2024	Extended Support

EKS Lifecycle: Standard vs Extended Support

AWS EKS provides two support tiers:

Standard Support (14 months)

Included in base EKS pricing
Security patches and bug fixes
Full AWS support

Extended Support (additional 12 months)

Additional cost: $0.60 per cluster per hour
Critical security patches only
Allows more time for upgrade planning

hcl

# Terraform: Enable extended support
resource "aws_eks_cluster" "main" {
  name    = "prod-cluster"
  version = "1.29"

  upgrade_policy {
    support_type = "EXTENDED"  # or "STANDARD"
  }
}

Version Compatibility Matrix

Before upgrading, verify compatibility across all components:

Component	EKS 1.29	EKS 1.30	EKS 1.31	EKS 1.32
VPC CNI	1.15+	1.16+	1.18+	1.19+
CoreDNS	1.10.1+	1.11.1+	1.11.1+	1.11.3+
kube-proxy	1.29.x	1.30.x	1.31.x	1.32.x
EBS CSI	1.25+	1.28+	1.31+	1.33+
Karpenter	0.33+	0.35+	0.37+	0.39+
AWS LB Controller	2.6+	2.7+	2.8+	2.9+
Cert Manager	1.13+	1.14+	1.15+	1.16+
ArgoCD	2.9+	2.10+	2.11+	2.12+

Add-on Version Requirements

Check current add-on versions against target Kubernetes version:

bash

#!/bin/bash
# check-addon-compatibility.sh

CLUSTER_NAME="prod-cluster"
TARGET_VERSION="1.31"

echo "=== Current Add-on Versions ==="
aws eks list-addons --cluster-name $CLUSTER_NAME --query 'addons[]' --output text | while read addon; do
  version=$(aws eks describe-addon --cluster-name $CLUSTER_NAME --addon-name $addon \
    --query 'addon.addonVersion' --output text)
  echo "$addon: $version"
done

echo ""
echo "=== Compatible Versions for EKS $TARGET_VERSION ==="
for addon in vpc-cni coredns kube-proxy aws-ebs-csi-driver; do
  echo "--- $addon ---"
  aws eks describe-addon-versions \
    --addon-name $addon \
    --kubernetes-version $TARGET_VERSION \
    --query 'addons[0].addonVersions[*].addonVersion' \
    --output text | head -5
done

Kubernetes Deprecation Policy

Kubernetes provides a structured deprecation timeline:

Deprecation announcement: API marked deprecated in release notes
Warning period: kubectl warns when using deprecated APIs (minimum 2 releases)
Removal: API removed from the codebase

Key deprecations by version:

Version	Deprecated/Removed APIs
1.29	flowcontrol.apiserver.k8s.io/v1beta2 removed
1.30	CSIStorageCapacity v1beta1 removed
1.31	PodSecurityPolicy completely removed
1.32	Several beta APIs promoted to stable

Timeline Recommendations

Production upgrade timeline:

Phase	Duration	Activities
Planning	2 weeks	Compatibility review, deprecation audit
Dev/Test	2 weeks	Upgrade non-prod, validate workloads
Staging	1 week	Full production simulation
Production	1 week	Rolling upgrade with monitoring
Stabilization	2 weeks	Monitor, document, update runbooks

Recommended upgrade cadence:

Stay within N-1 of current stable release
Upgrade every 4-6 months
Never skip more than one minor version

2. Pre-Upgrade Checklist

Deprecated API Detection with Pluto

Pluto scans for deprecated and removed APIs:

bash

#!/bin/bash
# detect-deprecated-apis.sh

# Install pluto
curl -L -o pluto.tar.gz https://github.com/FairwindsOps/pluto/releases/download/v5.19.0/pluto_5.19.0_linux_amd64.tar.gz
tar -xzf pluto.tar.gz
sudo mv pluto /usr/local/bin/

# Scan live cluster
echo "=== Scanning Live Cluster ==="
pluto detect-all-in-cluster --target-versions k8s=v1.31.0 -o wide

# Scan Helm releases
echo ""
echo "=== Scanning Helm Releases ==="
pluto detect-helm --target-versions k8s=v1.31.0 -o wide

# Scan local manifests
echo ""
echo "=== Scanning Local Manifests ==="
pluto detect-files -d ./k8s-manifests/ --target-versions k8s=v1.31.0

# Generate report
pluto detect-all-in-cluster --target-versions k8s=v1.31.0 -o json > deprecated-apis-report.json

Sample output:

NAME                           KIND                VERSION              REPLACEMENT                    REMOVED   DEPRECATED   REPL AVAIL
my-ingress                     Ingress             extensions/v1beta1   networking.k8s.io/v1           true      true         true
my-pdb                         PodDisruptionBudget policy/v1beta1       policy/v1                      false     true         true

PodDisruptionBudget Audit

PDBs can block node draining during upgrades:

bash

#!/bin/bash
# audit-pdbs.sh

echo "=== PodDisruptionBudget Audit ==="

# List all PDBs with their configuration
kubectl get pdb -A -o custom-columns=\
'NAMESPACE:.metadata.namespace,'\
'NAME:.metadata.name,'\
'MIN-AVAILABLE:.spec.minAvailable,'\
'MAX-UNAVAILABLE:.spec.maxUnavailable,'\
'CURRENT:.status.currentHealthy,'\
'DESIRED:.status.desiredHealthy,'\
'DISRUPTIONS-ALLOWED:.status.disruptionsAllowed'

echo ""
echo "=== Blocking PDBs (disruptionsAllowed=0) ==="
kubectl get pdb -A -o json | jq -r '
  .items[] |
  select(.status.disruptionsAllowed == 0) |
  "\(.metadata.namespace)/\(.metadata.name): currentHealthy=\(.status.currentHealthy), desiredHealthy=\(.status.desiredHealthy)"
'

echo ""
echo "=== PDBs with minAvailable=100% (potentially blocking) ==="
kubectl get pdb -A -o json | jq -r '
  .items[] |
  select(.spec.minAvailable == "100%" or .spec.maxUnavailable == 0 or .spec.maxUnavailable == "0%") |
  "\(.metadata.namespace)/\(.metadata.name)"
'

Best practices for PDBs during upgrades:

Set maxUnavailable: 1 instead of minAvailable: 100%
Ensure replicas > PDB minimum
Temporarily relax PDBs for maintenance windows

ETCD Backup Strategy

EKS manages the control plane, but application state backup is critical:

yaml

# velero-schedule.yaml
apiVersion: velero.io/v1
kind: Schedule
metadata:
  name: pre-upgrade-backup
  namespace: velero
spec:
  schedule: "0 */6 * * *"  # Every 6 hours
  template:
    includedNamespaces:
      - "*"
    excludedNamespaces:
      - kube-system
      - velero
    includedResources:
      - "*"
    excludedResources:
      - events
      - events.events.k8s.io
    storageLocation: aws-s3
    volumeSnapshotLocations:
      - aws-ebs
    ttl: 168h  # 7 days retention
    snapshotVolumes: true
    defaultVolumesToFsBackup: false

Velero Snapshot and Restore Verification

bash

#!/bin/bash
# velero-backup-verify.sh

BACKUP_NAME="pre-upgrade-$(date +%Y%m%d-%H%M%S)"

echo "=== Creating Pre-Upgrade Backup ==="
velero backup create $BACKUP_NAME \
  --include-namespaces=app-prod,app-staging \
  --snapshot-volumes \
  --wait

echo ""
echo "=== Verifying Backup ==="
velero backup describe $BACKUP_NAME --details

echo ""
echo "=== Backup Logs ==="
velero backup logs $BACKUP_NAME | tail -50

echo ""
echo "=== Test Restore (dry-run equivalent) ==="
# Create restore to different namespace for verification
velero restore create test-restore-$BACKUP_NAME \
  --from-backup $BACKUP_NAME \
  --namespace-mappings app-prod:restore-test \
  --include-namespaces app-prod \
  --wait

echo ""
echo "=== Verify Restored Resources ==="
kubectl get all -n restore-test

echo ""
echo "=== Cleanup Test Restore ==="
kubectl delete namespace restore-test
velero restore delete test-restore-$BACKUP_NAME --confirm

Add-on Compatibility Check Script

bash

#!/bin/bash
# addon-compatibility-check.sh

CLUSTER_NAME="${1:-prod-cluster}"
TARGET_VERSION="${2:-1.31}"

echo "=============================================="
echo "Add-on Compatibility Check"
echo "Cluster: $CLUSTER_NAME"
echo "Target Version: $TARGET_VERSION"
echo "=============================================="

# Get current cluster version
CURRENT_VERSION=$(aws eks describe-cluster --name $CLUSTER_NAME \
  --query 'cluster.version' --output text)
echo "Current Version: $CURRENT_VERSION"
echo ""

# Check each add-on
check_addon() {
  local addon_name=$1

  # Get current version
  current=$(aws eks describe-addon --cluster-name $CLUSTER_NAME \
    --addon-name $addon_name --query 'addon.addonVersion' --output text 2>/dev/null)

  if [ "$current" == "None" ] || [ -z "$current" ]; then
    echo "[$addon_name] Not installed"
    return
  fi

  # Get recommended version for target
  recommended=$(aws eks describe-addon-versions \
    --addon-name $addon_name \
    --kubernetes-version $TARGET_VERSION \
    --query 'addons[0].addonVersions[?compatibilities[0].defaultVersion==`true`].addonVersion' \
    --output text 2>/dev/null)

  # Get all compatible versions
  compatible=$(aws eks describe-addon-versions \
    --addon-name $addon_name \
    --kubernetes-version $TARGET_VERSION \
    --query 'addons[0].addonVersions[*].addonVersion' \
    --output text 2>/dev/null | head -1)

  if echo "$compatible" | grep -q "$current"; then
    echo "[$addon_name] OK - Current: $current, Recommended: $recommended"
  else
    echo "[$addon_name] UPGRADE REQUIRED - Current: $current, Recommended: $recommended"
  fi
}

ADDONS=("vpc-cni" "coredns" "kube-proxy" "aws-ebs-csi-driver" "aws-efs-csi-driver" "snapshot-controller")

for addon in "${ADDONS[@]}"; do
  check_addon "$addon"
done

echo ""
echo "=== Helm Release Versions ==="
helm list -A -o json | jq -r '.[] | "\(.name) (\(.namespace)): \(.chart)"' | sort

Node and Application Health Verification

bash

#!/bin/bash
# health-verification.sh

echo "=== Node Health Check ==="
kubectl get nodes -o wide
echo ""

# Check for NotReady nodes
NOT_READY=$(kubectl get nodes --no-headers | grep -v " Ready " | wc -l)
if [ "$NOT_READY" -gt 0 ]; then
  echo "WARNING: $NOT_READY nodes are not Ready"
  kubectl get nodes --no-headers | grep -v " Ready "
fi

echo ""
echo "=== Pod Health Check ==="
# Pods not in Running/Completed state
kubectl get pods -A --field-selector=status.phase!=Running,status.phase!=Succeeded \
  --no-headers 2>/dev/null | head -20

echo ""
echo "=== Deployment Health ==="
kubectl get deployments -A -o custom-columns=\
'NAMESPACE:.metadata.namespace,'\
'NAME:.metadata.name,'\
'READY:.status.readyReplicas,'\
'DESIRED:.spec.replicas,'\
'AVAILABLE:.status.availableReplicas' | \
awk 'NR==1 || $3!=$4 {print}'

echo ""
echo "=== StatefulSet Health ==="
kubectl get statefulsets -A -o custom-columns=\
'NAMESPACE:.metadata.namespace,'\
'NAME:.metadata.name,'\
'READY:.status.readyReplicas,'\
'DESIRED:.spec.replicas' | \
awk 'NR==1 || $3!=$4 {print}'

echo ""
echo "=== Recent Events (Warnings) ==="
kubectl get events -A --field-selector type=Warning \
  --sort-by='.lastTimestamp' | tail -20

echo ""
echo "=== Resource Pressure ==="
kubectl top nodes 2>/dev/null || echo "Metrics server not available"

Comprehensive Pre-Upgrade Script

bash

#!/bin/bash
# pre-upgrade-checklist.sh

set -e

CLUSTER_NAME="${1:-prod-cluster}"
TARGET_VERSION="${2:-1.31}"
REPORT_DIR="./upgrade-reports/$(date +%Y%m%d-%H%M%S)"

mkdir -p $REPORT_DIR

echo "=============================================="
echo "EKS Pre-Upgrade Checklist"
echo "Cluster: $CLUSTER_NAME"
echo "Target: $TARGET_VERSION"
echo "Report: $REPORT_DIR"
echo "=============================================="

# 1. Current state snapshot
echo ""
echo "[1/8] Capturing current state..."
kubectl cluster-info > $REPORT_DIR/cluster-info.txt
kubectl get nodes -o wide > $REPORT_DIR/nodes.txt
kubectl get pods -A -o wide > $REPORT_DIR/pods.txt
kubectl get pv,pvc -A > $REPORT_DIR/storage.txt
aws eks describe-cluster --name $CLUSTER_NAME > $REPORT_DIR/eks-cluster.json

# 2. Deprecated API check
echo "[2/8] Checking deprecated APIs..."
pluto detect-all-in-cluster --target-versions k8s=v${TARGET_VERSION}.0 -o json \
  > $REPORT_DIR/deprecated-apis.json 2>/dev/null || echo "Pluto not installed"

DEPRECATED_COUNT=$(cat $REPORT_DIR/deprecated-apis.json | jq 'length' 2>/dev/null || echo "0")
echo "  Found $DEPRECATED_COUNT deprecated API usages"

# 3. PDB audit
echo "[3/8] Auditing PodDisruptionBudgets..."
kubectl get pdb -A -o json > $REPORT_DIR/pdbs.json
BLOCKING_PDBS=$(cat $REPORT_DIR/pdbs.json | jq '[.items[] | select(.status.disruptionsAllowed == 0)] | length')
echo "  Found $BLOCKING_PDBS blocking PDBs"

# 4. Add-on compatibility
echo "[4/8] Checking add-on compatibility..."
aws eks list-addons --cluster-name $CLUSTER_NAME --output json > $REPORT_DIR/addons.json

# 5. Helm releases
echo "[5/8] Documenting Helm releases..."
helm list -A -o json > $REPORT_DIR/helm-releases.json

# 6. Custom resources
echo "[6/8] Documenting custom resources..."
kubectl api-resources --verbs=list -o name | while read resource; do
  count=$(kubectl get $resource -A --no-headers 2>/dev/null | wc -l)
  if [ "$count" -gt 0 ]; then
    echo "$resource: $count"
  fi
done > $REPORT_DIR/resource-counts.txt

# 7. Backup verification
echo "[7/8] Verifying backups..."
if command -v velero &> /dev/null; then
  velero backup get -o json > $REPORT_DIR/velero-backups.json
  RECENT_BACKUP=$(velero backup get --selector='velero.io/schedule-name' -o json | \
    jq -r '.items | sort_by(.status.completionTimestamp) | last | .metadata.name')
  echo "  Most recent backup: $RECENT_BACKUP"
else
  echo "  Velero not installed"
fi

# 8. Health summary
echo "[8/8] Generating health summary..."

cat > $REPORT_DIR/summary.md << EOF
# Pre-Upgrade Summary

**Cluster**: $CLUSTER_NAME
**Current Version**: $(aws eks describe-cluster --name $CLUSTER_NAME --query 'cluster.version' --output text)
**Target Version**: $TARGET_VERSION
**Generated**: $(date -Iseconds)

## Checklist

| Item | Status | Details |
|------|--------|---------|
| Deprecated APIs | $([ "$DEPRECATED_COUNT" == "0" ] && echo "PASS" || echo "REVIEW") | $DEPRECATED_COUNT found |
| Blocking PDBs | $([ "$BLOCKING_PDBS" == "0" ] && echo "PASS" || echo "REVIEW") | $BLOCKING_PDBS found |
| Node Health | $(kubectl get nodes --no-headers | grep -v " Ready " | wc -l | xargs -I{} sh -c '[ {} -eq 0 ] && echo "PASS" || echo "FAIL"') | |
| Backup Status | $([ -n "$RECENT_BACKUP" ] && echo "PASS" || echo "REVIEW") | $RECENT_BACKUP |

## Action Items

$([ "$DEPRECATED_COUNT" != "0" ] && echo "- [ ] Update deprecated APIs (see deprecated-apis.json)")
$([ "$BLOCKING_PDBS" != "0" ] && echo "- [ ] Review blocking PDBs before maintenance window")
- [ ] Notify stakeholders of upgrade window
- [ ] Prepare rollback procedure
EOF

echo ""
echo "=============================================="
echo "Pre-upgrade checklist complete"
echo "Review report at: $REPORT_DIR/summary.md"
echo "=============================================="

cat $REPORT_DIR/summary.md

3. Auto Mode Upgrade

Terraform 3-Layer Upgrade Order

EKS Auto Mode upgrades follow a strict order to maintain stability:

Layer 02 (Cluster): Upgrade control plane version
Wait: Allow control plane upgrade to complete
Layer 03 (Platform): Upgrade add-ons to compatible versions
Automatic: Node pools rotate via Karpenter drift detection

┌─────────────────────────────────────────────────────────────┐
│                    Upgrade Sequence                         │
├─────────────────────────────────────────────────────────────┤
│  1. Control Plane (02-cluster)                              │
│     └─> aws_eks_cluster.version = "1.31"                    │
│                                                             │
│  2. Wait for Control Plane (~15-20 minutes)                 │
│     └─> Verify: kubectl get nodes, cluster status           │
│                                                             │
│  3. Platform Add-ons (03-platform)                          │
│     └─> Update add-on versions in Terraform                 │
│     └─> Apply: CoreDNS, VPC CNI, kube-proxy, etc.           │
│                                                             │
│  4. Node Rotation (Automatic)                               │
│     └─> Karpenter detects AMI drift                         │
│     └─> Nodes cordoned, drained, replaced                   │
│     └─> PDBs respected during drain                         │
└─────────────────────────────────────────────────────────────┘

Layer 02 Cluster Upgrade

hcl

# 02-cluster/main.tf

variable "kubernetes_version" {
  description = "Target Kubernetes version"
  type        = string
  default     = "1.31"  # Upgrade: 1.30 -> 1.31
}

resource "aws_eks_cluster" "main" {
  name     = var.cluster_name
  version  = var.kubernetes_version
  role_arn = aws_iam_role.cluster.arn

  vpc_config {
    subnet_ids              = var.subnet_ids
    endpoint_private_access = true
    endpoint_public_access  = true
    security_group_ids      = [aws_security_group.cluster.id]
  }

  # Auto Mode configuration
  compute_config {
    enabled       = true
    node_pools    = ["general-purpose", "system"]
    node_role_arn = aws_iam_role.node.arn
  }

  kubernetes_network_config {
    ip_family         = "ipv4"
    service_ipv4_cidr = var.service_cidr
    elastic_load_balancing {
      enabled = true
    }
  }

  storage_config {
    block_storage {
      enabled = true
    }
  }

  upgrade_policy {
    support_type = "STANDARD"
  }

  access_config {
    authentication_mode                         = "API_AND_CONFIG_MAP"
    bootstrap_cluster_creator_admin_permissions = false
  }

  # Prevent accidental destruction
  lifecycle {
    prevent_destroy = true
    ignore_changes  = [
      access_config[0].bootstrap_cluster_creator_admin_permissions
    ]
  }

  tags = var.tags
}

# Output for dependency management
output "cluster_version" {
  value = aws_eks_cluster.main.version
}

output "cluster_status" {
  value = aws_eks_cluster.main.status
}

Apply the upgrade:

bash

#!/bin/bash
# upgrade-control-plane.sh

cd terraform/02-cluster

echo "=== Current Version ==="
terraform output cluster_version

echo ""
echo "=== Planning Upgrade ==="
terraform plan -var="kubernetes_version=1.31" -out=upgrade.plan

echo ""
read -p "Proceed with control plane upgrade? (yes/no): " confirm
if [ "$confirm" != "yes" ]; then
  echo "Upgrade cancelled"
  exit 1
fi

echo ""
echo "=== Applying Upgrade ==="
terraform apply upgrade.plan

echo ""
echo "=== Waiting for Cluster Status ==="
aws eks wait cluster-active --name prod-cluster
echo "Control plane upgrade complete"

echo ""
echo "=== Verify ==="
kubectl version --short
kubectl get nodes

Layer 03 Platform Add-ons Upgrade

hcl

# 03-platform/addons.tf

variable "addon_versions" {
  description = "Add-on versions compatible with target K8s version"
  type = object({
    vpc_cni       = string
    coredns       = string
    kube_proxy    = string
    ebs_csi       = string
    efs_csi       = string
  })
  default = {
    # Versions for EKS 1.31
    vpc_cni    = "v1.18.3-eksbuild.1"
    coredns    = "v1.11.1-eksbuild.9"
    kube_proxy = "v1.31.0-eksbuild.5"
    ebs_csi    = "v1.35.0-eksbuild.1"
    efs_csi    = "v2.0.7-eksbuild.1"
  }
}

resource "aws_eks_addon" "vpc_cni" {
  cluster_name                = var.cluster_name
  addon_name                  = "vpc-cni"
  addon_version               = var.addon_versions.vpc_cni
  resolve_conflicts_on_update = "OVERWRITE"

  configuration_values = jsonencode({
    enableNetworkPolicy = "true"
    env = {
      ENABLE_PREFIX_DELEGATION = "true"
      WARM_PREFIX_TARGET       = "1"
    }
  })

  tags = var.tags
}

resource "aws_eks_addon" "coredns" {
  cluster_name                = var.cluster_name
  addon_name                  = "coredns"
  addon_version               = var.addon_versions.coredns
  resolve_conflicts_on_update = "OVERWRITE"

  configuration_values = jsonencode({
    replicaCount = 3
    resources = {
      limits = {
        cpu    = "200m"
        memory = "256Mi"
      }
      requests = {
        cpu    = "100m"
        memory = "128Mi"
      }
    }
  })

  tags = var.tags
}

resource "aws_eks_addon" "kube_proxy" {
  cluster_name                = var.cluster_name
  addon_name                  = "kube-proxy"
  addon_version               = var.addon_versions.kube_proxy
  resolve_conflicts_on_update = "OVERWRITE"

  tags = var.tags
}

resource "aws_eks_addon" "ebs_csi" {
  cluster_name                = var.cluster_name
  addon_name                  = "aws-ebs-csi-driver"
  addon_version               = var.addon_versions.ebs_csi
  service_account_role_arn    = var.ebs_csi_role_arn
  resolve_conflicts_on_update = "OVERWRITE"

  tags = var.tags
}

resource "aws_eks_addon" "efs_csi" {
  cluster_name                = var.cluster_name
  addon_name                  = "aws-efs-csi-driver"
  addon_version               = var.addon_versions.efs_csi
  service_account_role_arn    = var.efs_csi_role_arn
  resolve_conflicts_on_update = "OVERWRITE"

  tags = var.tags
}

NodePool Auto-Rotation

EKS Auto Mode with Karpenter automatically handles node rotation through drift detection:

yaml

# NodePool configuration for Auto Mode
apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: general-purpose
spec:
  template:
    spec:
      requirements:
        - key: kubernetes.io/arch
          operator: In
          values: ["amd64", "arm64"]
        - key: karpenter.sh/capacity-type
          operator: In
          values: ["on-demand", "spot"]
        - key: karpenter.k8s.aws/instance-category
          operator: In
          values: ["c", "m", "r"]
      nodeClassRef:
        group: karpenter.k8s.aws
        kind: EC2NodeClass
        name: default
  disruption:
    consolidationPolicy: WhenEmptyOrUnderutilized
    consolidateAfter: 1m
    # Drift detection triggers node replacement when AMI changes
    budgets:
      - nodes: "10%"      # Max 10% of nodes disrupted at once
      - nodes: "1"        # Or at least 1 node
        schedule: "0 9 * * *"   # Maintenance window
        duration: 8h

AMI Update Strategy:

yaml

# EC2NodeClass with AMI selection
apiVersion: karpenter.k8s.aws/v1
kind: EC2NodeClass
metadata:
  name: default
spec:
  amiSelectorTerms:
    # Auto Mode uses EKS-optimized AMIs
    - alias: al2023@latest

  role: "KarpenterNodeRole-prod-cluster"

  subnetSelectorTerms:
    - tags:
        karpenter.sh/discovery: "prod-cluster"

  securityGroupSelectorTerms:
    - tags:
        karpenter.sh/discovery: "prod-cluster"

  # Block device configuration
  blockDeviceMappings:
    - deviceName: /dev/xvda
      ebs:
        volumeSize: 100Gi
        volumeType: gp3
        iops: 3000
        throughput: 125
        encrypted: true

Monitoring Upgrade Progress

kubectl commands:

bash

#!/bin/bash
# monitor-upgrade.sh

echo "=== Cluster Version ==="
kubectl version --short

echo ""
echo "=== Node Versions ==="
kubectl get nodes -o custom-columns=\
'NAME:.metadata.name,'\
'VERSION:.status.nodeInfo.kubeletVersion,'\
'OS-IMAGE:.status.nodeInfo.osImage,'\
'AGE:.metadata.creationTimestamp'

echo ""
echo "=== Node Conditions ==="
kubectl get nodes -o json | jq -r '
  .items[] |
  "\(.metadata.name): \([.status.conditions[] | select(.status=="True") | .type] | join(", "))"
'

echo ""
echo "=== Karpenter Drift Status ==="
kubectl get nodeclaims -o custom-columns=\
'NAME:.metadata.name,'\
'NODE:.status.nodeName,'\
'READY:.status.conditions[?(@.type=="Ready")].status,'\
'AGE:.metadata.creationTimestamp'

echo ""
echo "=== Add-on Status ==="
aws eks list-addons --cluster-name prod-cluster --query 'addons[]' --output text | while read addon; do
  status=$(aws eks describe-addon --cluster-name prod-cluster --addon-name $addon \
    --query 'addon.status' --output text)
  version=$(aws eks describe-addon --cluster-name prod-cluster --addon-name $addon \
    --query 'addon.addonVersion' --output text)
  echo "$addon: $status ($version)"
done

Prometheus queries for monitoring:

promql

# Node rotation progress
count(kube_node_info) by (kubelet_version)

# Pods being rescheduled during upgrade
sum(rate(kube_pod_container_status_restarts_total[5m])) by (namespace)

# Node drain rate
rate(karpenter_nodes_terminated_total[10m])

# PDB blocking status
kube_poddisruptionbudget_status_pod_disruptions_allowed == 0

# Add-on health
kube_daemonset_status_number_ready{daemonset=~"aws-node|kube-proxy|ebs-csi-node"}

Grafana dashboard panel:

json

{
  "title": "Upgrade Progress",
  "panels": [
    {
      "title": "Node Versions",
      "type": "piechart",
      "targets": [{
        "expr": "count(kube_node_info) by (kubelet_version)"
      }]
    },
    {
      "title": "Node Rotation Timeline",
      "type": "timeseries",
      "targets": [{
        "expr": "sum(karpenter_nodes_created_total)",
        "legendFormat": "Created"
      }, {
        "expr": "sum(karpenter_nodes_terminated_total)",
        "legendFormat": "Terminated"
      }]
    }
  ]
}

Handling Stuck Upgrades

bash

#!/bin/bash
# unstick-upgrade.sh

echo "=== Identifying Stuck Nodes ==="
kubectl get nodes -l "karpenter.sh/lifecycle!=spot" \
  -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.status.conditions[?(@.type=="Ready")].status}{"\n"}{end}'

echo ""
echo "=== Checking for PDB Blocks ==="
kubectl get pdb -A -o json | jq -r '
  .items[] |
  select(.status.disruptionsAllowed == 0) |
  "BLOCKED: \(.metadata.namespace)/\(.metadata.name) - expectedPods:\(.status.expectedPods), currentHealthy:\(.status.currentHealthy)"
'

echo ""
echo "=== Pods Preventing Drain ==="
# Find pods on nodes being drained
for node in $(kubectl get nodes -o jsonpath='{.items[?(@.spec.unschedulable==true)].metadata.name}'); do
  echo "Node: $node"
  kubectl get pods --all-namespaces --field-selector spec.nodeName=$node \
    -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name,OWNER:.metadata.ownerReferences[0].kind'
done

echo ""
echo "=== Force Drain Options ==="
echo "1. Temporarily relax PDB:"
echo "   kubectl patch pdb <name> -n <ns> -p '{\"spec\":{\"maxUnavailable\":1}}'"
echo ""
echo "2. Delete stuck pods (use with caution):"
echo "   kubectl delete pod <name> -n <ns> --grace-period=0 --force"
echo ""
echo "3. Skip PDB check (emergency only):"
echo "   kubectl drain <node> --ignore-daemonsets --delete-emptydir-data --disable-eviction"

Rollback Limitations

EKS Auto Mode has specific rollback constraints:

Component	Rollback Possible	Method
Control Plane	No	Create new cluster
Add-ons	Yes	Terraform apply previous version
Node AMI	Yes	Update EC2NodeClass amiSelectorTerms
Applications	Yes	ArgoCD sync to previous revision

Rollback procedure (applications only):

bash

#!/bin/bash
# rollback-applications.sh

APP_NAME="$1"
TARGET_REVISION="$2"

if [ -z "$APP_NAME" ] || [ -z "$TARGET_REVISION" ]; then
  echo "Usage: $0 <app-name> <target-revision>"
  exit 1
fi

echo "=== Current Application State ==="
argocd app get $APP_NAME

echo ""
echo "=== Available Revisions ==="
argocd app history $APP_NAME

echo ""
echo "=== Rolling Back to $TARGET_REVISION ==="
argocd app rollback $APP_NAME $TARGET_REVISION

echo ""
echo "=== Verify Rollback ==="
argocd app wait $APP_NAME --health --timeout 300

4. Blue/Green Upgrade Strategy

Strategy Overview

Blue/Green upgrades provide the safest upgrade path by running two complete clusters in parallel:

┌─────────────────────────────────────────────────────────────┐
│                Blue/Green Upgrade Flow                      │
├─────────────────────────────────────────────────────────────┤
│                                                             │
│   ┌─────────────┐     ┌─────────────┐     ┌─────────────┐   │
│   │   BLUE      │     │    NLB      │     │   GREEN     │   │
│   │  (Current)  │◄────│  Weighted   │────►│   (New)     │   │
│   │  EKS 1.30   │     │  Routing    │     │  EKS 1.31   │   │
│   └─────────────┘     └─────────────┘     └─────────────┘   │
│         │                   │                   │           │
│         ▼                   ▼                   ▼           │
│   100% → 90% → 50% → 10% → 0%            0% → 10% → ...     │
│                                                             │
│   Phase 1: Green cluster created                            │
│   Phase 2: Platform deployed                                │
│   Phase 3: ArgoCD deploys apps                              │
│   Phase 4: Traffic shifted gradually                        │
│   Phase 5: Blue decommissioned                              │
│                                                             │
└─────────────────────────────────────────────────────────────┘

Step-by-Step Blue/Green Upgrade

Step 1: Create Green Cluster

hcl

# terraform/02-cluster-green/main.tf

variable "cluster_version" {
  default = "1.31"
}

variable "cluster_suffix" {
  default = "green"
}

resource "aws_eks_cluster" "green" {
  name     = "prod-cluster-${var.cluster_suffix}"
  version  = var.cluster_version
  role_arn = data.aws_iam_role.cluster.arn

  vpc_config {
    subnet_ids              = data.aws_subnets.private.ids
    endpoint_private_access = true
    endpoint_public_access  = true
    security_group_ids      = [aws_security_group.cluster.id]
  }

  compute_config {
    enabled       = true
    node_pools    = ["general-purpose", "system"]
    node_role_arn = data.aws_iam_role.node.arn
  }

  kubernetes_network_config {
    ip_family         = "ipv4"
    service_ipv4_cidr = "10.101.0.0/16"  # Different from blue
    elastic_load_balancing {
      enabled = true
    }
  }

  storage_config {
    block_storage {
      enabled = true
    }
  }

  tags = merge(var.tags, {
    Environment = "production"
    Cluster     = "green"
    Version     = var.cluster_version
  })
}

output "cluster_name" {
  value = aws_eks_cluster.green.name
}

output "cluster_endpoint" {
  value = aws_eks_cluster.green.endpoint
}

output "cluster_ca_data" {
  value     = aws_eks_cluster.green.certificate_authority[0].data
  sensitive = true
}

Step 2: Deploy Platform Components

bash

#!/bin/bash
# deploy-platform-green.sh

GREEN_CLUSTER="prod-cluster-green"

echo "=== Updating kubeconfig ==="
aws eks update-kubeconfig --name $GREEN_CLUSTER --alias green

echo ""
echo "=== Deploying Platform Add-ons ==="
cd terraform/03-platform
terraform workspace select green || terraform workspace new green
terraform apply -var="cluster_name=$GREEN_CLUSTER" -auto-approve

echo ""
echo "=== Installing Cert Manager ==="
helm upgrade --install cert-manager jetstack/cert-manager \
  --namespace cert-manager --create-namespace \
  --version v1.15.0 \
  --set installCRDs=true

echo ""
echo "=== Installing AWS Load Balancer Controller ==="
helm upgrade --install aws-load-balancer-controller eks/aws-load-balancer-controller \
  --namespace kube-system \
  --set clusterName=$GREEN_CLUSTER \
  --set serviceAccount.create=false \
  --set serviceAccount.name=aws-load-balancer-controller

echo ""
echo "=== Installing External DNS ==="
helm upgrade --install external-dns bitnami/external-dns \
  --namespace external-dns --create-namespace \
  --set provider=aws \
  --set aws.zoneType=public \
  --set txtOwnerId=$GREEN_CLUSTER

echo ""
echo "=== Platform deployment complete ==="
kubectl get pods -A

Step 3: Register with ArgoCD Hub

yaml

# argocd/cluster-secret-green.yaml
apiVersion: v1
kind: Secret
metadata:
  name: prod-cluster-green
  namespace: argocd
  labels:
    argocd.argoproj.io/secret-type: cluster
    environment: production
    cluster-color: green
    kubernetes-version: "1.31"
type: Opaque
stringData:
  name: prod-cluster-green
  server: https://XXXXXXXXXXXXXXXX.gr7.us-west-2.eks.amazonaws.com
  config: |
    {
      "awsAuthConfig": {
        "clusterName": "prod-cluster-green",
        "roleARN": "arn:aws:iam::ACCOUNT_ID:role/argocd-hub-role"
      },
      "tlsClientConfig": {
        "insecure": false,
        "caData": "BASE64_ENCODED_CA_DATA"
      }
    }

bash

#!/bin/bash
# register-argocd-cluster.sh

GREEN_CLUSTER="prod-cluster-green"
GREEN_ENDPOINT=$(aws eks describe-cluster --name $GREEN_CLUSTER \
  --query 'cluster.endpoint' --output text)
GREEN_CA=$(aws eks describe-cluster --name $GREEN_CLUSTER \
  --query 'cluster.certificateAuthority.data' --output text)

# Create cluster secret
cat <<EOF | kubectl apply -f - --context argocd-hub
apiVersion: v1
kind: Secret
metadata:
  name: $GREEN_CLUSTER
  namespace: argocd
  labels:
    argocd.argoproj.io/secret-type: cluster
    environment: production
    cluster-color: green
    kubernetes-version: "1.31"
type: Opaque
stringData:
  name: $GREEN_CLUSTER
  server: $GREEN_ENDPOINT
  config: |
    {
      "awsAuthConfig": {
        "clusterName": "$GREEN_CLUSTER",
        "roleARN": "arn:aws:iam::${AWS_ACCOUNT_ID}:role/argocd-hub-role"
      },
      "tlsClientConfig": {
        "insecure": false,
        "caData": "$GREEN_CA"
      }
    }
EOF

echo "Cluster registered with ArgoCD"
argocd cluster list

Step 4: ApplicationSet Deploys Applications

yaml

# argocd/applicationset-production.yaml
apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: production-apps
  namespace: argocd
spec:
  goTemplate: true
  goTemplateOptions: ["missingkey=error"]
  generators:
    - matrix:
        generators:
          # Generator 1: Clusters
          - clusters:
              selector:
                matchLabels:
                  environment: production
          # Generator 2: Applications
          - git:
              repoURL: https://github.com/company/k8s-apps.git
              revision: HEAD
              directories:
                - path: apps/*
  template:
    metadata:
      name: '{{.path.basename}}-{{.name}}'
      labels:
        app: '{{.path.basename}}'
        cluster: '{{.name}}'
        cluster-color: '{{index .metadata.labels "cluster-color"}}'
    spec:
      project: production
      source:
        repoURL: https://github.com/company/k8s-apps.git
        targetRevision: HEAD
        path: '{{.path.path}}'
        helm:
          valueFiles:
            - values.yaml
            - 'values-{{index .metadata.labels "cluster-color"}}.yaml'
      destination:
        server: '{{.server}}'
        namespace: '{{.path.basename}}'
      syncPolicy:
        automated:
          prune: true
          selfHeal: true
        syncOptions:
          - CreateNamespace=true
          - PrunePropagationPolicy=foreground

Step 5: Smoke Tests

bash

#!/bin/bash
# smoke-test-green.sh

GREEN_CLUSTER="prod-cluster-green"
aws eks update-kubeconfig --name $GREEN_CLUSTER --alias green

echo "=== Application Health Check ==="
APPS=("api-gateway" "user-service" "order-service" "payment-service")

for app in "${APPS[@]}"; do
  echo "Checking $app..."

  # Deployment status
  ready=$(kubectl get deployment $app -n $app -o jsonpath='{.status.readyReplicas}')
  desired=$(kubectl get deployment $app -n $app -o jsonpath='{.spec.replicas}')

  if [ "$ready" == "$desired" ]; then
    echo "  Deployment: OK ($ready/$desired)"
  else
    echo "  Deployment: FAIL ($ready/$desired)"
    exit 1
  fi

  # Pod health
  unhealthy=$(kubectl get pods -n $app -o jsonpath='{.items[?(@.status.phase!="Running")].metadata.name}')
  if [ -z "$unhealthy" ]; then
    echo "  Pods: OK"
  else
    echo "  Pods: FAIL - $unhealthy"
    exit 1
  fi

  # Service endpoint
  endpoint=$(kubectl get svc $app -n $app -o jsonpath='{.status.loadBalancer.ingress[0].hostname}')
  if [ -n "$endpoint" ]; then
    echo "  Service: OK ($endpoint)"
  else
    echo "  Service: Internal only"
  fi
done

echo ""
echo "=== HTTP Health Checks ==="
# Internal health check via port-forward
for app in "${APPS[@]}"; do
  kubectl port-forward svc/$app 8080:80 -n $app &
  PID=$!
  sleep 2

  status=$(curl -s -o /dev/null -w "%{http_code}" http://localhost:8080/health)
  kill $PID 2>/dev/null

  if [ "$status" == "200" ]; then
    echo "$app health check: PASS"
  else
    echo "$app health check: FAIL ($status)"
    exit 1
  fi
done

echo ""
echo "=== All smoke tests passed ==="

Step 6: NLB Weight Transition

hcl

# terraform/04-routing/nlb-weights.tf

variable "blue_weight" {
  description = "Traffic weight for blue cluster (0-100)"
  type        = number
  default     = 100
}

variable "green_weight" {
  description = "Traffic weight for green cluster (0-100)"
  type        = number
  default     = 0
}

resource "aws_lb_target_group" "blue" {
  name        = "prod-api-blue"
  port        = 443
  protocol    = "TCP"
  vpc_id      = var.vpc_id
  target_type = "ip"

  health_check {
    enabled             = true
    healthy_threshold   = 2
    unhealthy_threshold = 2
    interval            = 10
    protocol            = "TCP"
  }

  tags = {
    Cluster = "blue"
  }
}

resource "aws_lb_target_group" "green" {
  name        = "prod-api-green"
  port        = 443
  protocol    = "TCP"
  vpc_id      = var.vpc_id
  target_type = "ip"

  health_check {
    enabled             = true
    healthy_threshold   = 2
    unhealthy_threshold = 2
    interval            = 10
    protocol            = "TCP"
  }

  tags = {
    Cluster = "green"
  }
}

resource "aws_lb_listener" "api" {
  load_balancer_arn = aws_lb.api.arn
  port              = 443
  protocol          = "TCP"

  default_action {
    type = "forward"

    forward {
      target_group {
        arn    = aws_lb_target_group.blue.arn
        weight = var.blue_weight
      }

      target_group {
        arn    = aws_lb_target_group.green.arn
        weight = var.green_weight
      }

      stickiness {
        enabled  = true
        duration = 3600
      }
    }
  }
}

output "current_weights" {
  value = {
    blue  = var.blue_weight
    green = var.green_weight
  }
}

Weight transition script:

bash

#!/bin/bash
# shift-traffic.sh

set -e

BLUE_WEIGHT=${1:-100}
GREEN_WEIGHT=${2:-0}

if [ $((BLUE_WEIGHT + GREEN_WEIGHT)) -ne 100 ]; then
  echo "Error: Weights must sum to 100"
  exit 1
fi

echo "=== Shifting Traffic ==="
echo "Blue: $BLUE_WEIGHT%"
echo "Green: $GREEN_WEIGHT%"

cd terraform/04-routing
terraform apply \
  -var="blue_weight=$BLUE_WEIGHT" \
  -var="green_weight=$GREEN_WEIGHT" \
  -auto-approve

echo ""
echo "=== Verifying Shift ==="
aws elbv2 describe-listeners \
  --load-balancer-arn $(terraform output -raw nlb_arn) \
  --query 'Listeners[0].DefaultActions[0].ForwardConfig.TargetGroups[*].{ARN:TargetGroupArn,Weight:Weight}' \
  --output table

echo ""
echo "=== Monitoring ==="
echo "Watch error rates: https://grafana.company.com/d/traffic-shift"

Recommended weight transition schedule:

Stage	Blue	Green	Duration	Validation
1	100%	0%	Initial	Green smoke tests pass
2	90%	10%	30 min	Error rate stable
3	50%	50%	1 hour	Latency comparison
4	10%	90%	30 min	Final validation
5	0%	100%	-	Complete migration

Rollback Procedure

bash

#!/bin/bash
# rollback-to-blue.sh

echo "=== Emergency Rollback to Blue Cluster ==="

# Immediate traffic shift
cd terraform/04-routing
terraform apply \
  -var="blue_weight=100" \
  -var="green_weight=0" \
  -auto-approve

echo ""
echo "=== Traffic restored to blue cluster ==="

# Document the rollback
cat >> upgrade-log.md << EOF

## Rollback Event - $(date -Iseconds)

**Reason**: [Document reason]
**Action**: Traffic shifted 100% to blue cluster
**Next Steps**:
- [ ] Investigate green cluster issues
- [ ] Review application logs
- [ ] Plan retry attempt
EOF

echo ""
echo "=== Post-Rollback Verification ==="
aws eks update-kubeconfig --name prod-cluster-blue --alias blue
kubectl --context blue get pods -A | grep -v Running

Data Migration Considerations

When stateful workloads exist, plan data migration carefully:

bash

#!/bin/bash
# data-migration-checklist.sh

echo "=== Stateful Workload Inventory ==="
kubectl get pvc -A -o custom-columns=\
'NAMESPACE:.metadata.namespace,'\
'NAME:.metadata.name,'\
'STORAGE-CLASS:.spec.storageClassName,'\
'SIZE:.spec.resources.requests.storage,'\
'STATUS:.status.phase'

echo ""
echo "=== Database Instances ==="
kubectl get pods -A -l 'app.kubernetes.io/component in (database,db,postgresql,mysql,mongodb)'

echo ""
echo "=== Migration Strategies ==="
cat << 'EOF'
1. AWS Native (RDS, ElastiCache, etc.)
   - No migration needed - external to cluster
   - Verify security group access from green cluster

2. In-Cluster StatefulSets
   - Option A: Velero backup/restore
   - Option B: Application-level replication
   - Option C: Shared EFS storage

3. PersistentVolumes
   - EBS: Snapshot and restore to new AZ
   - EFS: Mount same filesystem from both clusters

EOF

Alternative: Zonal In-Place Upgrade with Native Rollback

Since Amazon EKS added native Kubernetes version rollback (July 2026), teams that already run one cluster per zone behind the weighted NLB from NLB Weighted Target Groups have a lighter-weight option than maintaining a second full cluster fleet indefinitely: upgrade each zonal cluster in place, one zone at a time, using the existing weighted routing only to drain traffic during the upgrade window, and relying on EKS's native rollback — not a second cluster — as the safety net.

┌────────────────────────────────────────────────────────────┐
│      Zonal In-Place Upgrade (rollback as safety net)       │
├────────────────────────────────────────────────────────────┤
│                                                              │
│   ┌─────────────┐     ┌─────────────┐    ┌─────────────┐   │
│   │   AZ-a      │◄────│    NLB      │───►│   AZ-c      │   │
│   │  1.30→1.31  │     │  Weighted   │    │   1.30      │   │
│   └─────────────┘     └─────────────┘    └─────────────┘   │
│                                                              │
│   1. Shift NLB weight: AZ-a -> 0%, AZ-c -> 100%             │
│   2. Upgrade AZ-a's control plane + nodes in place          │
│   3. Validate AZ-a, shift weight back to 50/50              │
│   4. Repeat for AZ-c                                        │
│   5. If AZ-a misbehaves post-upgrade: use EKS's native      │
│      rollback (control plane only, N -> N-1, within 7 days) │
│      instead of standing up a third cluster                 │
└────────────────────────────────────────────────────────────┘

When this fits better than a permanent Blue/Green fleet:

You already run zonal clusters for availability, not specifically for upgrades
You want to avoid the steady-state cost of running two full cluster fleets
Your upgrade window can tolerate the ~7-day rollback eligibility instead of an instant cluster-level failback

When to keep the full Blue/Green fleet from this section instead:

You need to validate the new version against real production traffic on a fully separate cluster before cutting over — native rollback only reverts the control plane, not any node/AMI/add-on changes made in place
Rollback eligibility conditions aren't met (cluster created at the target version, more than 7 days elapsed, another upgrade already applied, or a backward-incompatible feature was enabled) — see EKS Upgrade Strategies — Rollback Procedure

(Source: Amazon EKS announces Kubernetes version rollback, July 2026)

5. Post-Upgrade Validation

Comprehensive Health Check Script

bash

#!/bin/bash
# post-upgrade-validation.sh

set -e

CLUSTER_NAME="${1:-prod-cluster}"
REPORT_DIR="./post-upgrade-reports/$(date +%Y%m%d-%H%M%S)"

mkdir -p $REPORT_DIR

echo "=============================================="
echo "Post-Upgrade Validation"
echo "Cluster: $CLUSTER_NAME"
echo "=============================================="

# 1. Cluster version verification
echo ""
echo "[1/7] Verifying cluster version..."
CLUSTER_VERSION=$(aws eks describe-cluster --name $CLUSTER_NAME \
  --query 'cluster.version' --output text)
echo "Cluster version: $CLUSTER_VERSION"

# 2. Node status
echo ""
echo "[2/7] Checking node status..."
kubectl get nodes -o wide | tee $REPORT_DIR/nodes.txt

NOT_READY=$(kubectl get nodes --no-headers | grep -v " Ready " | wc -l)
if [ "$NOT_READY" -gt 0 ]; then
  echo "WARNING: $NOT_READY nodes not ready"
  kubectl get nodes --no-headers | grep -v " Ready "
fi

# Verify all nodes on target version
NODE_VERSIONS=$(kubectl get nodes -o jsonpath='{.items[*].status.nodeInfo.kubeletVersion}' | tr ' ' '\n' | sort -u)
echo "Node versions: $NODE_VERSIONS"

# 3. Pod status
echo ""
echo "[3/7] Checking pod status..."
PROBLEM_PODS=$(kubectl get pods -A --field-selector=status.phase!=Running,status.phase!=Succeeded \
  --no-headers 2>/dev/null | wc -l)

if [ "$PROBLEM_PODS" -gt 0 ]; then
  echo "WARNING: $PROBLEM_PODS pods not in Running/Succeeded state"
  kubectl get pods -A --field-selector=status.phase!=Running,status.phase!=Succeeded
fi

# 4. Service connectivity
echo ""
echo "[4/7] Testing service connectivity..."
SERVICES=("api-gateway:api-gateway" "user-service:user-service")

for svc in "${SERVICES[@]}"; do
  ns="${svc%%:*}"
  name="${svc##*:}"

  endpoint=$(kubectl get endpoints $name -n $ns -o jsonpath='{.subsets[0].addresses[0].ip}' 2>/dev/null)
  if [ -n "$endpoint" ]; then
    echo "  $ns/$name: OK (endpoint: $endpoint)"
  else
    echo "  $ns/$name: WARNING - no endpoints"
  fi
done

# 5. Ingress reachability
echo ""
echo "[5/7] Testing ingress reachability..."
INGRESS_HOST=$(kubectl get ingress -n api-gateway -o jsonpath='{.items[0].status.loadBalancer.ingress[0].hostname}' 2>/dev/null)

if [ -n "$INGRESS_HOST" ]; then
  HTTP_STATUS=$(curl -s -o /dev/null -w "%{http_code}" "https://$INGRESS_HOST/health" --connect-timeout 5 || echo "timeout")
  echo "  Ingress ($INGRESS_HOST): $HTTP_STATUS"
else
  echo "  No ingress found or not yet provisioned"
fi

# 6. Add-on status
echo ""
echo "[6/7] Verifying add-ons..."
for addon in vpc-cni coredns kube-proxy aws-ebs-csi-driver; do
  status=$(aws eks describe-addon --cluster-name $CLUSTER_NAME --addon-name $addon \
    --query 'addon.status' --output text 2>/dev/null || echo "NOT_INSTALLED")
  version=$(aws eks describe-addon --cluster-name $CLUSTER_NAME --addon-name $addon \
    --query 'addon.addonVersion' --output text 2>/dev/null || echo "-")
  echo "  $addon: $status ($version)"
done

# 7. Critical workload health
echo ""
echo "[7/7] Checking critical workloads..."
CRITICAL_DEPLOYMENTS=("kube-system:coredns" "kube-system:aws-load-balancer-controller" "argocd:argocd-server")

for deploy in "${CRITICAL_DEPLOYMENTS[@]}"; do
  ns="${deploy%%:*}"
  name="${deploy##*:}"

  ready=$(kubectl get deployment $name -n $ns -o jsonpath='{.status.readyReplicas}' 2>/dev/null || echo "0")
  desired=$(kubectl get deployment $name -n $ns -o jsonpath='{.spec.replicas}' 2>/dev/null || echo "0")

  if [ "$ready" == "$desired" ] && [ "$ready" != "0" ]; then
    echo "  $ns/$name: OK ($ready/$desired)"
  else
    echo "  $ns/$name: WARNING ($ready/$desired)"
  fi
done

echo ""
echo "=============================================="
echo "Validation complete. Review any warnings above."
echo "=============================================="

Smoke Tests

bash

#!/bin/bash
# smoke-tests.sh

echo "=== HTTP Smoke Tests ==="

# Define test endpoints
declare -A ENDPOINTS=(
  ["api-gateway"]="https://api.company.com/health"
  ["web-app"]="https://www.company.com/"
  ["admin-portal"]="https://admin.company.com/health"
)

for name in "${!ENDPOINTS[@]}"; do
  url="${ENDPOINTS[$name]}"

  start_time=$(date +%s%N)
  status=$(curl -s -o /dev/null -w "%{http_code}" "$url" --connect-timeout 10 || echo "timeout")
  end_time=$(date +%s%N)

  latency=$(( (end_time - start_time) / 1000000 ))

  if [ "$status" == "200" ]; then
    echo "  $name: PASS (${latency}ms)"
  else
    echo "  $name: FAIL (status: $status)"
  fi
done

echo ""
echo "=== Database Connectivity ==="
# Test via application pods
kubectl exec -n api-gateway deploy/api-gateway -- \
  /bin/sh -c 'nc -zv $DB_HOST 5432 2>&1' || echo "DB connectivity test failed"

echo ""
echo "=== Message Queue Connectivity ==="
kubectl exec -n api-gateway deploy/api-gateway -- \
  /bin/sh -c 'nc -zv $MQ_HOST 5672 2>&1' || echo "MQ connectivity test failed"

echo ""
echo "=== Cache Connectivity ==="
kubectl exec -n api-gateway deploy/api-gateway -- \
  /bin/sh -c 'nc -zv $REDIS_HOST 6379 2>&1' || echo "Cache connectivity test failed"

Metrics Comparison with PromQL

Compare key metrics before and after upgrade:

promql

# Error rate comparison (5xx errors)
# Run before upgrade, save value, compare after

# Current error rate
sum(rate(http_requests_total{status=~"5.."}[5m]))
/
sum(rate(http_requests_total[5m])) * 100

# P99 latency comparison
histogram_quantile(0.99,
  sum(rate(http_request_duration_seconds_bucket[5m])) by (le, service)
)

# Resource utilization - CPU
avg(rate(container_cpu_usage_seconds_total{namespace!~"kube-system|monitoring"}[5m])) by (namespace)

# Resource utilization - Memory
avg(container_memory_working_set_bytes{namespace!~"kube-system|monitoring"}) by (namespace) / 1024 / 1024

# Pod restart rate (should be near zero post-upgrade stabilization)
sum(increase(kube_pod_container_status_restarts_total[1h])) by (namespace)

Grafana comparison dashboard:

json

{
  "title": "Upgrade Comparison",
  "templating": {
    "list": [{
      "name": "comparison_time",
      "type": "custom",
      "options": [
        {"value": "now-1d", "text": "Yesterday"},
        {"value": "now-7d", "text": "Last Week"}
      ]
    }]
  },
  "panels": [
    {
      "title": "Error Rate: Before vs After",
      "type": "timeseries",
      "targets": [
        {
          "expr": "sum(rate(http_requests_total{status=~\"5..\"}[5m])) / sum(rate(http_requests_total[5m])) * 100",
          "legendFormat": "Current"
        },
        {
          "expr": "sum(rate(http_requests_total{status=~\"5..\"}[5m] offset 1d)) / sum(rate(http_requests_total[5m] offset 1d)) * 100",
          "legendFormat": "Before Upgrade"
        }
      ]
    },
    {
      "title": "P99 Latency: Before vs After",
      "type": "timeseries",
      "targets": [
        {
          "expr": "histogram_quantile(0.99, sum(rate(http_request_duration_seconds_bucket[5m])) by (le))",
          "legendFormat": "Current"
        },
        {
          "expr": "histogram_quantile(0.99, sum(rate(http_request_duration_seconds_bucket[5m] offset 1d)) by (le))",
          "legendFormat": "Before Upgrade"
        }
      ]
    }
  ]
}

Monitoring Period

Post-upgrade monitoring checklist:

Timeframe	Focus Areas	Actions
0-1 hour	Critical failures	Watch error rates, pod restarts
1-4 hours	Performance regression	Compare latency, throughput
4-24 hours	Stability	Memory leaks, gradual degradation
1-7 days	Edge cases	Batch jobs, scheduled tasks

bash

#!/bin/bash
# monitoring-period-alerts.sh

# Create temporary high-sensitivity alerts for post-upgrade period
cat <<EOF | kubectl apply -f -
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: post-upgrade-alerts
  namespace: monitoring
  labels:
    release: prometheus
spec:
  groups:
    - name: post-upgrade
      rules:
        - alert: PostUpgradeHighErrorRate
          expr: |
            sum(rate(http_requests_total{status=~"5.."}[5m]))
            / sum(rate(http_requests_total[5m])) > 0.01
          for: 2m
          labels:
            severity: warning
            context: post-upgrade
          annotations:
            summary: "Error rate elevated post-upgrade"

        - alert: PostUpgradeLatencyIncrease
          expr: |
            histogram_quantile(0.99, sum(rate(http_request_duration_seconds_bucket[5m])) by (le))
            >
            histogram_quantile(0.99, sum(rate(http_request_duration_seconds_bucket[5m] offset 1d)) by (le)) * 1.5
          for: 5m
          labels:
            severity: warning
            context: post-upgrade
          annotations:
            summary: "P99 latency increased >50% vs yesterday"

        - alert: PostUpgradePodRestarts
          expr: |
            sum(increase(kube_pod_container_status_restarts_total[30m])) by (namespace) > 5
          labels:
            severity: warning
            context: post-upgrade
          annotations:
            summary: "Elevated pod restarts in {{ \$labels.namespace }}"
EOF

echo "Post-upgrade alerts created. Remove after stabilization:"
echo "  kubectl delete prometheusrule post-upgrade-alerts -n monitoring"

Runbook Update

After successful upgrade, update operational documentation:

markdown

# Post-Upgrade Runbook Update Checklist

## Version Information
- [ ] Update cluster version in documentation
- [ ] Update add-on version matrix
- [ ] Document any new features enabled

## Configuration Changes
- [ ] Document any API changes made
- [ ] Update Terraform module versions
- [ ] Update Helm chart versions

## Lessons Learned
- [ ] Document any issues encountered
- [ ] Note workarounds or fixes applied
- [ ] Update pre-upgrade checklist based on experience

## Timeline
- [ ] Record actual upgrade duration
- [ ] Note any deviations from plan
- [ ] Update time estimates for future upgrades

EKS Version Upgrades - Core upgrade concepts
Node Lifecycle Management - Auto Mode node rotation
ArgoCD Multi-Cluster - ApplicationSet configuration
Observability Stack - Monitoring setup

< Previous: Resource Optimization | Table of Contents | Next: Event Capacity Planning >

Traffic Management

Security

Observability

Resilience

Advanced

Comparison Guide

ArgoCD

Metrics

Logging

Tracing

Alerting

Observability End-to-End Labs

Cilium Deep Dive

Calico Deep Dive

Istio Detailed Quiz

Linkerd Detailed Quiz

Cilium Service Mesh Detailed Quiz

ArgoCD Detailed Quiz

Metrics

Logging

Tracing

Alerting

Observability Labs

EKS Upgrades: Auto Mode Zero-Downtime Upgrade ​

Overview ​

1. Upgrade Planning ​

Kubernetes Version Support Policy ​

EKS Lifecycle: Standard vs Extended Support ​

Version Compatibility Matrix ​

Add-on Version Requirements ​

Kubernetes Deprecation Policy ​

Timeline Recommendations ​

2. Pre-Upgrade Checklist ​

Deprecated API Detection with Pluto ​

PodDisruptionBudget Audit ​

ETCD Backup Strategy ​

Velero Snapshot and Restore Verification ​

Add-on Compatibility Check Script ​

Node and Application Health Verification ​

Comprehensive Pre-Upgrade Script ​

3. Auto Mode Upgrade ​

Terraform 3-Layer Upgrade Order ​

Layer 02 Cluster Upgrade ​

Layer 03 Platform Add-ons Upgrade ​

NodePool Auto-Rotation ​

Monitoring Upgrade Progress ​

Handling Stuck Upgrades ​

Rollback Limitations ​

4. Blue/Green Upgrade Strategy ​

Strategy Overview ​

Step-by-Step Blue/Green Upgrade ​

Step 1: Create Green Cluster ​

Step 2: Deploy Platform Components ​

Step 3: Register with ArgoCD Hub ​

Step 4: ApplicationSet Deploys Applications ​

Step 5: Smoke Tests ​

Step 6: NLB Weight Transition ​

Rollback Procedure ​

Data Migration Considerations ​

Alternative: Zonal In-Place Upgrade with Native Rollback ​

5. Post-Upgrade Validation ​

Comprehensive Health Check Script ​

Smoke Tests ​

Metrics Comparison with PromQL ​

Monitoring Period ​

Runbook Update ​

Related Documentation ​

EKS Upgrades: Auto Mode Zero-Downtime Upgrade

Overview

1. Upgrade Planning

Kubernetes Version Support Policy

EKS Lifecycle: Standard vs Extended Support

Version Compatibility Matrix

Add-on Version Requirements

Kubernetes Deprecation Policy

Timeline Recommendations

2. Pre-Upgrade Checklist

Deprecated API Detection with Pluto

PodDisruptionBudget Audit

ETCD Backup Strategy

Velero Snapshot and Restore Verification

Add-on Compatibility Check Script

Node and Application Health Verification

Comprehensive Pre-Upgrade Script

3. Auto Mode Upgrade

Terraform 3-Layer Upgrade Order

Layer 02 Cluster Upgrade

Layer 03 Platform Add-ons Upgrade

NodePool Auto-Rotation

Monitoring Upgrade Progress

Handling Stuck Upgrades

Rollback Limitations

4. Blue/Green Upgrade Strategy

Strategy Overview

Step-by-Step Blue/Green Upgrade

Step 1: Create Green Cluster

Step 2: Deploy Platform Components

Step 3: Register with ArgoCD Hub

Step 4: ApplicationSet Deploys Applications

Step 5: Smoke Tests

Step 6: NLB Weight Transition

Rollback Procedure

Data Migration Considerations

Alternative: Zonal In-Place Upgrade with Native Rollback

5. Post-Upgrade Validation

Comprehensive Health Check Script

Smoke Tests

Metrics Comparison with PromQL

Monitoring Period

Runbook Update

Related Documentation