EKS Cluster Creation Quiz - Part 4

This quiz tests your understanding of advanced configuration, scalability, and operational topics related to Amazon EKS cluster creation. It covers topics such as cluster scaling, automation, cost optimization, and operational best practices.

Basic Concept Questions

1. What is the main difference between Cluster Autoscaler and Karpenter in an Amazon EKS cluster?
   • A) Cluster Autoscaler is an AWS service, while Karpenter is an open-source tool
   • B) Cluster Autoscaler scales at the node group level, while Karpenter provisions individual nodes matching workload requirements
   • C) Cluster Autoscaler scales based on CPU/memory usage, while Karpenter scales based on pod count
   • D) Cluster Autoscaler only supports horizontal scaling, while Karpenter also supports vertical scaling

Show Answer

Answer: B) Cluster Autoscaler scales at the node group level, while Karpenter provisions individual nodes matching workload requirements

Explanation: The main difference between Cluster Autoscaler and Karpenter in an Amazon EKS cluster lies in their scaling approach. Cluster Autoscaler scales at the node group level based on existing Auto Scaling Groups (ASGs), while Karpenter directly provisions individual nodes that match workload requirements.

Cluster Autoscaler Characteristics:

1. Node Group-Based Scaling:

   • Scales using pre-defined Auto Scaling Groups

   • Uses the same instance type or mixed instance types within a node group

   • Example:

     # Cluster Autoscaler Deployment
     apiVersion: apps/v1
     kind: Deployment
     metadata:
       name: cluster-autoscaler
       namespace: kube-system
     spec:
       replicas: 1
       selector:
         matchLabels:
           app: cluster-autoscaler
       template:
         metadata:
           labels:
             app: cluster-autoscaler
         spec:
           containers:
           - image: k8s.gcr.io/autoscaling/cluster-autoscaler:v1.23.0
             name: cluster-autoscaler
             command:
             - ./cluster-autoscaler
             - --v=4
             - --stderrthreshold=info
             - --cloud-provider=aws
             - --skip-nodes-with-local-storage=false
             - --expander=least-waste
             - --node-group-auto-discovery=asg:tag=k8s.io/cluster-autoscaler/enabled,k8s.io/cluster-autoscaler/my-cluster

2. How It Works:
   • Scales up node groups when there are unschedulable pods
   • Scales down node groups when node utilization is low
   • Operates within the ASG's min/max size limits
3. Limitations:
   • Relatively slow scaling speed (2-10 minutes)
   • Limited to pre-defined instance types
   • Can only scale at the node group level

Karpenter Characteristics:

1. Workload-Based Provisioning:

   • Selects optimal instance types matching workload requirements

   • Directly provisions EC2 instances without ASGs

   • Example:

     # Karpenter Provisioner
     apiVersion: karpenter.sh/v1alpha5
     kind: NodePool
     metadata:
       name: default
     spec:
       template:
         spec:
           requirements:
             - key: karpenter.sh/capacity-type
               operator: In
               values: ["spot", "on-demand"]
             - key: kubernetes.io/arch
               operator: In
               values: ["amd64", "arm64"]
         - key: node.kubernetes.io/instance-type
           operator: In
           values: ["m5.large", "m5a.large", "m5d.large", "m5ad.large", "m6g.large"]
       limits:
         resources:
           cpu: 1000
           memory: 1000Gi
       provider:
         subnetSelector:
           karpenter.sh/discovery: "true"
         securityGroupSelector:
           karpenter.sh/discovery: "true"
       ttlSecondsAfterEmpty: 30

2. How It Works:
   • Analyzes requirements of unschedulable pods
   • Selects optimal instance types matching requirements
   • Directly provisions instances and schedules pods
   • Automatically terminates nodes when empty
3. Advantages:
   • Fast scaling speed (under 1 minute)
   • Selects instance types optimized for workloads
   • Cost optimization (Spot instance utilization, right-sizing)
   • Simplified configuration (no ASG management needed)

Comparison of Both Tools:

Characteristic	Cluster Autoscaler	Karpenter
Scaling Unit	Node Group (ASG)	Individual Node
Instance Selection	Pre-defined instance types	Optimal instances for workload requirements
Scaling Speed	Slow (2-10 minutes)	Fast (under 1 minute)
Configuration Complexity	Medium (ASG configuration required)	Low (only Provisioner definition needed)
Cost Optimization	Limited	High (workload-optimized instance selection)
Maturity	High (older project)	Medium (relatively new project)

Issues with Other Options:

• Cluster Autoscaler is an AWS service, while Karpenter is an open-source tool: Both are open-source tools. Cluster Autoscaler is managed by Kubernetes SIG Autoscaling, and while Karpenter was started by AWS, it is an open-source project.
• Cluster Autoscaler scales based on CPU/memory usage, while Karpenter scales based on pod count: Both fundamentally scale based on unschedulable pods (Pending state). CPU/memory usage-based scaling is the role of Horizontal Pod Autoscaler (HPA).
• Cluster Autoscaler only supports horizontal scaling, while Karpenter also supports vertical scaling: Both only support horizontal scaling (increasing node count). Vertical scaling (increasing node resources) is not supported. Pod-level vertical scaling is the role of Vertical Pod Autoscaler (VPA).

Both Cluster Autoscaler and Karpenter are tools for auto-scaling EKS clusters, but Karpenter provides faster and more flexible scaling and can select optimal instances matching workload requirements.

2. What are the valid capacity types that can be used when creating a node group in an Amazon EKS cluster?
   • A) Reserved, On-Demand, Spot
   • B) On-Demand, Spot, Dedicated
   • C) On-Demand, Spot
   • D) Standard, Burstable, Compute-Optimized

Show Answer

Answer: C) On-Demand, Spot

Explanation: When creating an Amazon EKS node group, the available capacity types are On-Demand and Spot.

On-Demand Capacity Type:

• Characteristics: Instances available reliably without interruption
• Pricing: Pay fixed hourly rate
• Suitable Workloads:
  • Production applications sensitive to interruption
  • Stateful workloads
  • Databases
  • Critical business applications

Spot Capacity Type:

• Characteristics: Utilizes AWS spare capacity, can be interrupted when AWS reclaims capacity
• Pricing: Up to 90% discount compared to On-Demand
• Suitable Workloads:
  • Fault-tolerant applications
  • Stateless workloads
  • Batch processing jobs
  • Development/test environments

Example of Specifying Capacity Type When Creating EKS Node Group:

AWS CLI example:

aws eks create-nodegroup \
  --cluster-name my-cluster \
  --nodegroup-name my-spot-nodegroup \
  --scaling-config minSize=3,maxSize=10,desiredSize=5 \
  --subnets subnet-0a1b2c3d4e5f6g7h8 subnet-0a1b2c3d4e5f6g7h9 \
  --instance-types t3.medium t3a.medium \
  --capacity-type SPOT \
  --node-role arn:aws:iam::123456789012:role/EKS-NodeInstanceRole

eksctl example:

apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig
metadata:
  name: my-cluster
  region: us-west-2
nodeGroups:
  - name: ng-on-demand
    instanceType: m5.large
    desiredCapacity: 3
    capacityType: ON_DEMAND
  - name: ng-spot
    instanceType: m5.large
    desiredCapacity: 2
    capacityType: SPOT
    spotInstancePools: 3

Issues with Other Options:

• Reserved, On-Demand, Spot: "Reserved" refers to EC2 Reserved Instances, but cannot be directly specified as a capacity type when creating EKS node groups. Reserved Instances are a billing discount model and cannot be directly selected as a node group capacity type.
• On-Demand, Spot, Dedicated: "Dedicated" refers to EC2 Dedicated Instances, but cannot be directly specified as an EKS node group capacity type. Dedicated Instances can be configured through separate tenancy settings.
• Standard, Burstable, Compute-Optimized: These represent EC2 instance family types, not capacity types. They are characteristics considered when selecting instance types (e.g., t3.medium, m5.large, c5.xlarge).

Best Practices:

1. Use Mixed Capacity Strategy:
   • Place critical workloads on On-Demand node groups
   • Place fault-tolerant workloads on Spot node groups
   • Control workload placement using node affinity and tolerations
2. Considerations When Using Spot Instances:
   • Specify various instance types (distribute interruption risk)
   • Implement proper interruption handling mechanisms (Pod Disruption Budgets, graceful shutdown hooks)
   • Deploy AWS Node Termination Handler
3. Cost Optimization:
   • Apply Savings Plans or Reserved Instances to On-Demand nodes
   • Consider Graviton (ARM) instances
   • Select appropriate instance sizes

3. What must be mandatorily specified when configuring a Fargate profile in an Amazon EKS cluster?
   • A) Instance type and capacity type
   • B) Namespace and label selector
   • C) Subnet ID and security group
   • D) Autoscaling settings and maximum pod count

Show Answer

Answer: B) Namespace and label selector

Explanation: When configuring an Amazon EKS Fargate profile, the mandatory items to specify are the namespace and optionally the label selector. EKS uses this information to determine which pods should run on Fargate.

Fargate Profile Components:

1. Required Components:
   • Profile Name: Unique identifier for the Fargate profile
   • Pod Execution Role: IAM role required to run pods on Fargate infrastructure
   • Subnets: Private subnets where Fargate pods will run (uses cluster subnets by default)
   • Selectors: Array consisting of namespaces and optional labels
2. Selector Configuration:
   • Namespace: Kubernetes namespace the pod belongs to (required)
   • Labels: Key-value pair Kubernetes labels (optional)

Fargate Profile Creation Examples:

AWS CLI example:

aws eks create-fargate-profile \
  --cluster-name my-cluster \
  --fargate-profile-name my-fargate-profile \
  --pod-execution-role-arn arn:aws:iam::123456789012:role/AmazonEKSFargatePodExecutionRole \
  --subnets subnet-0a1b2c3d4e5f6g7h8 subnet-0a1b2c3d4e5f6g7h9 \
  --selectors namespace=default,labels={app=nginx} namespace=kube-system,labels={k8s-app=kube-dns}

eksctl example:

apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig
metadata:
  name: my-cluster
  region: us-west-2
fargateProfiles:
  - name: fp-default
    selectors:
      - namespace: default
        labels:
          app: nginx
      - namespace: kube-system
        labels:
          k8s-app: kube-dns

How Fargate Profiles Work:

1. When a pod is created, EKS checks the pod's namespace and labels.
2. If the pod's namespace and labels match a Fargate profile's selectors, that pod runs on Fargate infrastructure.
3. If no matching Fargate profile exists, the pod is scheduled on EC2 nodes (if available) or remains in Pending state.

Issues with Other Options:

• Instance type and capacity type: Fargate is a serverless compute service, so there's no need to specify instance types or capacity types. AWS automatically provisions the necessary compute resources.
• Subnet ID and security group: While subnet ID is needed, cluster subnets can be used as defaults. Security groups are optional; if not specified, the cluster's security groups are used.
• Autoscaling settings and maximum pod count: Fargate automatically scales per pod, so separate autoscaling settings are not needed. Pod count is managed through Kubernetes Deployments or HPA (Horizontal Pod Autoscaler).

Considerations When Using Fargate:

1. Cost: With Fargate, you only pay for the vCPU and memory resources actually used. There's no cost for idle nodes.
2. Limitations:
   • DaemonSet pods are not supported on Fargate.
   • Privileged containers are not supported.
   • Host network modes like HostNetwork, HostPort are not supported.
   • For persistent volumes, only Amazon EFS is supported.
3. Use Cases:
   • Batch processing jobs
   • Web applications
   • API servers
   • Microservices
   • Development/test environments

4. Which of the following is NOT a configurable item in the "update configuration" when updating a node group in an Amazon EKS cluster?
   • A) Maximum unavailable node count
   • B) Maximum unavailable node percentage
   • C) Node replacement strategy
   • D) Node group update timeout

Show Answer

Answer: C) Node replacement strategy

Explanation: "Node replacement strategy" is not an official configuration item in Amazon EKS node group update configuration. EKS managed node groups use a rolling update approach by default, and there's no option to directly change this strategy.

Items Configurable in EKS Node Group Update Configuration:

1. maxUnavailable:
   • Maximum number of nodes that can be unavailable simultaneously during update
   • Can be specified as an absolute number (e.g., 1, 2, 3) or percentage (e.g., 20%)
   • Default: 1
2. maxUnavailablePercentage:
   • Maximum percentage of nodes that can be unavailable simultaneously during update
   • Specified as a value between 1 and 100
   • Cannot be used together with maxUnavailable
3. force:
   • Whether to force the node group update
   • Default: false
4. Timeout:
   • Maximum wait time for node group update operation
   • Default: 60 minutes

Node Group Update Configuration Examples:

AWS CLI example:

aws eks update-nodegroup-config \
  --cluster-name my-cluster \
  --nodegroup-name my-nodegroup \
  --update-config maxUnavailable=2

Or specifying percentage:

aws eks update-nodegroup-config \
  --cluster-name my-cluster \
  --nodegroup-name my-nodegroup \
  --update-config maxUnavailablePercentage=20

eksctl example:

apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig
metadata:
  name: my-cluster
  region: us-west-2
managedNodeGroups:
  - name: my-nodegroup
    updateConfig:
      maxUnavailable: 2

Node Group Update Process:

1. Update Start: AWS initiates the node group update.
2. Node Draining: According to maxUnavailable setting, cordons and drains the specified number of nodes.
3. Node Termination: Terminates drained nodes.
4. New Node Creation: Creates nodes with new configuration.
5. Repeat: Repeats steps 2-4 until all nodes are updated.

Explanation of Other Options:

• Maximum unavailable (maxUnavailable) node count: Valid setting for specifying maximum number of nodes that can be unavailable simultaneously during update.
• Maximum unavailable (maxUnavailable) node percentage: Valid setting for specifying maximum percentage of nodes that can be unavailable simultaneously during update.
• Node group update timeout: Valid setting for specifying maximum wait time for node group update operation.

Node Group Update Best Practices:

1. Set Appropriate maxUnavailable Value:
   • Too small extends update time.
   • Too large can affect application availability.
   • Set considering workload characteristics and node group size.
2. Configure Pod Disruption Budgets (PDB):

   • Set PDBs for critical workloads to ensure minimum availability.

   • Example:

     apiVersion: policy/v1
     kind: PodDisruptionBudget
     metadata:
       name: app-pdb
     spec:
       minAvailable: 2  # or maxUnavailable: 1
       selector:
         matchLabels:
           app: my-app

3. Test Before Updating:
   • Validate in test environment before critical updates.
   • Prepare rollback plan.
4. Monitoring:
   • Monitor application status during update.
   • Pause or rollback update if issues occur.

5. Which statement is correct about "skipping" versions when upgrading Kubernetes versions in an EKS cluster?
   • A) Minor versions can be skipped, but major versions cannot
   • B) Major versions can be skipped, but minor versions cannot
   • C) Neither major nor minor versions can be skipped
   • D) Both major and minor versions can be skipped

Show Answer

Answer: C) Neither major nor minor versions can be skipped

Explanation: When upgrading an Amazon EKS cluster, both major and minor versions must be upgraded sequentially. Skipping versions is not supported.

EKS Version Upgrade Rules:

1. Sequential Upgrade Required:
   • Each Kubernetes version must be upgraded sequentially from the previous version.
   • Example: 1.22 -> 1.23 -> 1.24 (cannot upgrade directly from 1.22 to 1.24)
2. Supported Upgrade Paths:
   • Can only upgrade from current version to next minor version
   • Major versions must also be upgraded sequentially
3. Control Plane and Node Version Difference:
   • Control plane can be up to 2 minor versions ahead of nodes
   • Nodes can be up to 1 minor version ahead of control plane

EKS Version Upgrade Process:

1. Establish Upgrade Plan:

   • Review changes between current and target versions
   • Verify application compatibility
   • Establish upgrade schedule and rollback plan

2. Recommended Upgrade Order:

   • Control plane upgrade
   • Add-on upgrades (CoreDNS, kube-proxy, VPC CNI, etc.)
   • Node group upgrades

3. Upgrade Command Examples:

   Control plane upgrade using AWS CLI:

   aws eks update-cluster-version \
     --name my-cluster \
     --kubernetes-version 1.24

   Control plane upgrade using eksctl:

   eksctl upgrade cluster \
     --name=my-cluster \
     --version=1.24 \
     --approve

   Node group upgrade:

   aws eks update-nodegroup-version \
     --cluster-name my-cluster \
     --nodegroup-name my-nodegroup

EKS Version Support Policy:

1. Support Period:
   • Each Kubernetes version is supported for approximately 14 months after release on EKS.
   • AWS always maintains support for at least 4 production Kubernetes versions.
2. End of Support Notice:
   • AWS announces approximately 60 days before version support ends.
   • After support ends, clusters continue to operate but won't receive security patches or bug fixes.
3. Version Release Cycle:
   • AWS typically supports new versions in EKS within 2-3 months after upstream Kubernetes release.

Upgrade Best Practices:

1. Upgrade in Test Environment First:
   • Validate in test cluster before production environment upgrade
2. Backup Before Upgrade:
   • etcd backup and YAML backup of important resources
3. Gradual Upgrade:
   • Progress gradually instead of upgrading all node groups at once
4. Verify After Upgrade:
   • Confirm workloads are operating normally
   • Check monitoring systems
   • Review logs
5. Maintain Latest Version:
   • Establish regular upgrade schedule
   • Track support end dates

Issues with Other Options:

• Minor versions can be skipped, but major versions cannot: Minor versions cannot be skipped either. Must upgrade sequentially.
• Major versions can be skipped, but minor versions cannot: Major versions cannot be skipped either. Must upgrade sequentially.
• Both major and minor versions can be skipped: Skipping versions is not supported in EKS.

Short Answer Questions

6. What is required to change the instance type of a node group in an EKS cluster?

Answer and Explanation

To change the instance type of an EKS managed node group, you need to create a new node group and migrate workloads, then delete the existing node group. Instance types of managed node groups cannot be directly changed after creation.

The general procedure is as follows:

1. Create a new node group with the desired instance type
2. Set Pod Disruption Budgets (PDB) if needed
3. Cordon and drain existing nodes to migrate workloads to new nodes
4. Verify all workloads have moved to the new node group
5. Delete the existing node group

If using self-managed node groups, you can update the Auto Scaling Group's launch template and perform an instance refresh.

7. What settings need to be changed to disable public access to the Kubernetes API server and allow only private access in an EKS cluster?

Answer and Explanation

To disable public access to an EKS cluster's API server and allow only private access, you need to change the cluster's endpoint access settings:

1. Via AWS Management Console:
   • Navigate to EKS console
   • Select the cluster
   • Select "Networking" tab
   • Click "Edit" in "Cluster endpoint access" section
   • Set to "Private" (disable public access, enable private access)
2. Using AWS CLI:

aws eks update-cluster-config \
    --region region-code \
    --name cluster-name \
    --resources-vpc-config endpointPublicAccess=false,endpointPrivateAccess=true

After changing this setting, the Kubernetes API server can only be accessed from within the VPC. Therefore, only systems within the VPC or networks connected to the VPC can manage the cluster.

8. What is the typical support period for Kubernetes versions available in an EKS cluster?

Answer and Explanation

Each Kubernetes version in Amazon EKS is typically supported for approximately 14 months after release. AWS strives to always support at least 4 production Kubernetes versions.

The EKS version support cycle is as follows:

1. Initial release: AWS makes new Kubernetes version available on EKS
2. Standard support: Security patches and bug fixes provided for approximately 14 months
3. End of support announcement: AWS announces approximately 60 days before version support ends
4. End of support: Clusters running on deprecated versions are not automatically upgraded but must be manually upgraded

AWS starts supporting new versions slightly later than Kubernetes community releases, but the support period is generally longer. The upstream Kubernetes project supports each version for approximately 9 months, while EKS supports for approximately 14 months.

Hands-on Questions

9. Explain how to configure Auto Scaling settings for a node group in an EKS cluster, and write a configuration that meets the following requirements:

• Minimum node count: 2
• Maximum node count: 10
• Desired node count: 3
• Scale out based on CPU utilization (when CPU utilization exceeds 75%)
• Scale out based on memory utilization (when memory utilization exceeds 80%)

Answer and Explanation

To configure Auto Scaling settings for an EKS node group, follow these steps:

1. First, configure the basic size either when creating the node group or in the existing node group's Auto Scaling Group settings:

aws eks create-nodegroup \
    --cluster-name my-cluster \
    --nodegroup-name my-nodegroup \
    --scaling-config minSize=2,maxSize=10,desiredSize=3 \
    # Other required parameters...

2. For CPU and memory utilization-based scaling, you can either set up Cluster Autoscaler or Karpenter, or add CloudWatch alarm-based policies directly to the Auto Scaling Group.

Cluster Autoscaler Approach:

Cluster Autoscaler Deployment YAML:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: cluster-autoscaler
  namespace: kube-system
  labels:
    app: cluster-autoscaler
spec:
  replicas: 1
  selector:
    matchLabels:
      app: cluster-autoscaler
  template:
    metadata:
      labels:
        app: cluster-autoscaler
    spec:
      serviceAccountName: cluster-autoscaler
      containers:
      - image: k8s.gcr.io/autoscaling/cluster-autoscaler:v1.23.0
        name: cluster-autoscaler
        resources:
          limits:
            cpu: 100m
            memory: 300Mi
          requests:
            cpu: 100m
            memory: 300Mi
        command:
        - ./cluster-autoscaler
        - --v=4
        - --stderrthreshold=info
        - --cloud-provider=aws
        - --skip-nodes-with-local-storage=false
        - --expander=least-waste
        - --node-group-auto-discovery=asg:tag=k8s.io/cluster-autoscaler/enabled,k8s.io/cluster-autoscaler/my-cluster
        - --balance-similar-node-groups
        - --skip-nodes-with-system-pods=false

CloudWatch Alarm-Based Auto Scaling Policies:

CPU utilization-based scale out policy:

aws autoscaling put-scaling-policy \
    --auto-scaling-group-name my-nodegroup-xxx \
    --policy-name cpu-scale-out \
    --policy-type TargetTrackingScaling \
    --target-tracking-configuration file://cpu-policy.json

cpu-policy.json:

{
  "TargetValue": 75.0,
  "PredefinedMetricSpecification": {
    "PredefinedMetricType": "ASGAverageCPUUtilization"
  }
}

Memory utilization-based scale out policy (requires CloudWatch custom metric):

# First need to set up agent to publish memory utilization as CloudWatch custom metric
aws autoscaling put-scaling-policy \
    --auto-scaling-group-name my-nodegroup-xxx \
    --policy-name memory-scale-out \
    --policy-type TargetTrackingScaling \
    --target-tracking-configuration file://memory-policy.json

memory-policy.json:

{
  "TargetValue": 80.0,
  "CustomizedMetricSpecification": {
    "MetricName": "MemoryUtilization",
    "Namespace": "AWS/EC2",
    "Dimensions": [
      {
        "Name": "AutoScalingGroupName",
        "Value": "my-nodegroup-xxx"
      }
    ],
    "Statistic": "Average",
    "Unit": "Percent"
  }
}

In real environments, using Cluster Autoscaler is more appropriate for Kubernetes workloads. Cluster Autoscaler scales nodes based on pod resource requests, enabling more efficient scaling.

Advanced Questions

10. Explain the strategy for upgrading node groups using a blue/green approach in an EKS cluster, and present potential issues and solutions that may occur during this process.

Answer and Explanation

EKS Node Group Blue/Green Upgrade Strategy

Blue/green deployment is an approach where you build a new environment (green) alongside the existing environment (blue), then switch traffic to the new environment. When applied to EKS node groups, the process proceeds as follows:

Blue/Green Upgrade Steps:

1. Preparation Phase:
   • Document current node group configuration (labels, taints, tags, etc.)
   • Identify current workload status and resource requirements
2. Create Green Environment:
   • Create new node group (upgraded AMI, Kubernetes version, instance type, etc.)
   • Apply same labels and taints as existing node group
   • Apply necessary additional configuration (tags, IAM roles, etc.)
3. Testing:
   • Deploy test workloads to new node group
   • Validate functionality and performance
4. Traffic Transition:
   • Verify Pod Disruption Budget (PDB) settings
   • Cordon existing nodes (prevent new pod scheduling)
   • Gradually drain existing nodes (migrate workloads to new nodes)
   • Monitor workload status
5. Completion and Cleanup:
   • Verify all workloads have moved to new nodes
   • Delete existing node group
   • Update monitoring and alerts if needed

Potential Issues and Solutions:

1. Resource Shortage Issue:
   • Issue: Double resources needed while new node group is being created, potentially exceeding service quotas
   • Solution: Check service quotas in advance and request increase if needed, or upgrade gradually in small batches
2. Stateful Workload Migration:
   • Issue: Stateful workloads using persistent volumes may have issues when moving between nodes
   • Solution: Verify PVC/PV settings, use StatefulSets, use appropriate storage classes, perform backups
3. Node Affinity and Pod Disruption:
   • Issue: Some workloads may not move to new nodes due to node affinity or pod anti-affinity
   • Solution: Review node affinity and anti-affinity rules in pod specs and adjust if needed
4. Network Policies and Security Groups:
   • Issue: Required network policies or security groups may not be applied to new node group
   • Solution: Review and replicate network configuration including security groups, network policies, CIDR ranges
5. DNS and Service Discovery Delays:
   • Issue: Temporary DNS resolution delays or service discovery issues during node transition
   • Solution: Optimize CoreDNS settings, adjust TTL values, consider service mesh
6. Monitoring and Alert Gaps:
   • Issue: Monitoring agents or log collectors may not be automatically installed on new node group
   • Solution: Use DaemonSet-based monitoring tools, include agent installation scripts in node group launch template
7. Lack of Rollback Plan:
   • Issue: No rollback method if upgrade fails
   • Solution: Don't delete existing node group immediately, keep it for a period, create state snapshots, document rollback procedures

Implementation Example (AWS CLI):

# 1. Check current node group information
aws eks describe-nodegroup --cluster-name my-cluster --nodegroup-name blue-nodegroup

# 2. Create new node group (green)
aws eks create-nodegroup \
    --cluster-name my-cluster \
    --nodegroup-name green-nodegroup \
    --scaling-config minSize=3,maxSize=10,desiredSize=5 \
    --subnets subnet-xxxx subnet-yyyy \
    --instance-types t3.large \
    --ami-type AL2_x86_64 \
    --node-role arn:aws:iam::123456789012:role/EKS-NodeInstanceRole \
    --labels environment=prod,app=myapp \
    --tags "k8s.io/cluster-autoscaler/enabled=true,k8s.io/cluster-autoscaler/my-cluster=owned"

# 3. Check node group status
aws eks describe-nodegroup --cluster-name my-cluster --nodegroup-name green-nodegroup

# 4. Cordon existing nodes (using kubectl)
# Get node list
kubectl get nodes -l eks.amazonaws.com/nodegroup=blue-nodegroup

# Cordon each node
kubectl cordon <node-name>

# 5. Drain existing nodes
kubectl drain <node-name> --ignore-daemonsets --delete-emptydir-data

# 6. Verify all pods have moved to new nodes
kubectl get pods -o wide

# 7. Delete existing node group
aws eks delete-nodegroup --cluster-name my-cluster --nodegroup-name blue-nodegroup

Blue/green upgrades minimize downtime and provide rollback capability, but require additional resources and increase complexity. Thorough planning and testing are needed, and a gradual approach is recommended especially for large-scale production environments.