EKS Cluster Creation Quiz - Part 3

This quiz tests your understanding of advanced networking, storage configuration, and multi-tenancy related to Amazon EKS cluster creation. It covers topics such as cluster networking, storage options, and multi-tenant environment configuration.

Basic Concept Questions

1. What is the main factor that limits the number of IP addresses per pod in an Amazon EKS cluster?
   • A) VPC CIDR block size
   • B) Node instance type
   • C) Cluster's Kubernetes version
   • D) Number of available IP addresses in the subnet

Show Answer

Answer: B) Node instance type

Explanation: The main factor that limits the number of IP addresses per pod in an Amazon EKS cluster is the node's instance type. The Amazon VPC CNI plugin uses each node's Elastic Network Interfaces (ENIs) and secondary IP addresses to assign IP addresses to pods. Each EC2 instance type supports a different maximum number of ENIs and maximum IP addresses per ENI, which determines the maximum number of pods that can run on a node.

Maximum Pod Count Calculation by Instance Type:

The maximum number of pods is calculated using the following formula:

(Number of ENIs × IP addresses per ENI - 1) + 2

Where:

• 1 is subtracted because the primary IP address of the primary ENI is used by the node itself.
• 2 is added because kube-proxy and aws-node pods use host networking.

Maximum Pod Count for Common Instance Types:

Instance Type	Max ENIs	IPs per ENI	Max Pods
t3.small	3	4	11
t3.medium	3	6	17
m5.large	3	10	29
m5.xlarge	4	15	58
m5.2xlarge	4	15	58
m5.4xlarge	8	30	234
c5.large	3	10	29
c5.xlarge	4	15	58
r5.large	3	10	29
r5.xlarge	4	15	58

How to Check Maximum Pod Count:

# Check maximum pod count for a node
kubectl get nodes -o jsonpath='{.items[*].status.capacity.pods}'

# Or use the max-pods script
curl -s https://raw.githubusercontent.com/awslabs/amazon-eks-ami/master/scripts/max-pods-calculator.sh | bash -s -- --instance-type m5.large

Factors Limiting Maximum Pod Count:

1. Instance Type:

   • Each instance type supports a different maximum number of ENIs and IP addresses per ENI.
   • Larger instance types generally support more ENIs and IP addresses.

2. CNI Configuration:

   • The default VPC CNI configuration uses secondary IP addresses rather than allocating a full ENI to each pod.
   • This behavior can be changed using custom networking.

3. Prefix Delegation:

   • VPC CNI 1.9.0 and later supports prefix delegation, which allocates a /28 CIDR block (16 IPs) to each ENI.
   • This can significantly increase the maximum number of pods per node.

   kubectl set env daemonset aws-node -n kube-system ENABLE_PREFIX_DELEGATION=true

4. Custom max-pods Value:

   • You can use the kubelet's --max-pods flag to limit the maximum number of pods per node.
   • This can be set to a value lower than what the instance type supports.

   eksctl create nodegroup \
     --cluster my-cluster \
     --name my-nodegroup \
     --node-type m5.large \
     --nodes 3 \
     --kubelet-extra-args "--max-pods=110"

Issues with Other Options:

• VPC CIDR block size: The VPC CIDR block size limits the total number of IP addresses available in the VPC, but it does not directly limit the maximum number of pods per individual node.
• Cluster's Kubernetes version: The Kubernetes version affects supported features, but it does not directly limit the maximum number of pods per node.
• Number of available IP addresses in the subnet: The number of available IP addresses in a subnet affects the total number of pods that can be deployed in that subnet, but it does not directly limit the maximum number of pods per individual node.

The node's instance type is the main factor that determines the maximum number of pods that can run on a node through the number of ENIs and IP addresses per ENI it supports. Therefore, it is important to select an appropriate instance type that meets your workload requirements.

2. What is the default network policy for pod-to-pod communication in an Amazon EKS cluster? - A) Allow all pod-to-pod communication - B) Allow communication only between pods in the same namespace - C) Allow only explicitly permitted pod-to-pod communication - D) Block all pod-to-pod communication

Show Answer

Answer: A) Allow all pod-to-pod communication

Explanation: The default network policy for pod-to-pod communication in an Amazon EKS cluster is to allow all pod-to-pod communication. By default, EKS does not implement network policies, and all pods can freely communicate with all other pods in the cluster. This is the default behavior of Kubernetes, and you must explicitly configure network policies to restrict pod-to-pod communication.

Default Networking Behavior in EKS:

1. Default Allow Policy:
   • By default, all pods can communicate with all other pods in the cluster.
   • Communication between namespaces is also allowed without restrictions.
   • This follows Kubernetes' "flat network" model.
2. Amazon VPC CNI:
   • The default CNI plugin for EKS is Amazon VPC CNI.
   • This plugin assigns VPC IP addresses to pods, making them directly routable within the VPC.
   • It does not implement network policies by default.

How to Implement Network Policies:

To restrict pod-to-pod communication in EKS, you need to implement one of the following network policy solutions:

1. Calico:

   # Install Calico
   kubectl create namespace tigera-operator
   helm repo add projectcalico https://docs.projectcalico.org/charts
   helm install calico projectcalico/tigera-operator --namespace tigera-operator

2. Cilium:

   # Install Cilium
   helm repo add cilium https://helm.cilium.io/
   helm install cilium cilium/cilium --namespace kube-system

3. AWS Network Firewall (VPC level):

   • You can use AWS Network Firewall to filter traffic at the VPC level.
   • This provides subnet-level control rather than fine-grained pod-level control.

Network Policy Examples:

1. Default Deny Policy:

   # Block all ingress traffic
   apiVersion: networking.k8s.io/v1
   kind: NetworkPolicy
   metadata:
     name: default-deny-ingress
     namespace: default
   spec:
     podSelector: {}
     policyTypes:
     - Ingress

2. Allow Communication Between Specific Pods:

   # Allow communication only from frontend to backend
   apiVersion: networking.k8s.io/v1
   kind: NetworkPolicy
   metadata:
     name: allow-frontend-to-backend
     namespace: default
   spec:
     podSelector:
       matchLabels:
         app: backend
     ingress:
     - from:
       - podSelector:
           matchLabels:
             app: frontend
       ports:
       - protocol: TCP
         port: 8080

3. Restrict Cross-Namespace Communication:

   # Allow access only from prod namespace
   apiVersion: networking.k8s.io/v1
   kind: NetworkPolicy
   metadata:
     name: allow-from-prod-namespace
     namespace: default
   spec:
     podSelector:
       matchLabels:
         app: database
     ingress:
     - from:
       - namespaceSelector:
           matchLabels:
             name: prod
       ports:
       - protocol: TCP
         port: 5432

Network Policy Best Practices:

1. Apply Default Deny Policy:
   • Apply a default deny policy to all namespaces to block all traffic that is not explicitly allowed.
   • Explicitly allow only necessary communication.
2. Apply Least Privilege Principle:
   • Allow only the minimum network access that pods require.
   • Allow only specific ports and protocols.
3. Namespace Isolation:
   • Use namespaces to logically separate workloads.
   • Explicitly control communication between namespaces.
4. Label-Based Policies:
   • Use pod labels to define fine-grained network policies.
   • Maintain a consistent labeling scheme.

Issues with Other Options:

• Allow communication only between pods in the same namespace: By default, EKS allows all pod-to-pod communication, including cross-namespace communication.
• Allow only explicitly permitted pod-to-pod communication: This is the behavior after implementing network policies, but it is not the default behavior.
• Block all pod-to-pod communication: By default, EKS does not block pod-to-pod communication.

The default network policy in EKS is to allow all pod-to-pod communication. While this can be convenient in development and test environments, it is important to implement appropriate network policies to enhance security in production environments.

3. What is required for pods in an Amazon EKS cluster to access the internet outside the VPC?
   • A) Attach an internet gateway to the subnet where the pod is located
   • B) Attach a NAT gateway or NAT instance to the subnet where the pod is located
   • C) Assign a public IP address to the pod
   • D) Associate an Elastic IP address with the pod

Show Answer

Answer: B) Attach a NAT gateway or NAT instance to the subnet where the pod is located

Explanation: For pods in an Amazon EKS cluster to access the internet outside the VPC, the subnet where the pod is located must have a NAT gateway or NAT instance attached. This is the standard method for allowing pods in private subnets to access the internet.

EKS Networking Architecture:

1. Pods in Private Subnets:
   • EKS worker nodes are typically placed in private subnets for security.
   • Pods in private subnets cannot directly access the internet.
   • They must access the internet through a NAT gateway or NAT instance.
2. Pods in Public Subnets:
   • Even if worker nodes are in public subnets, pods do not receive public IP addresses by default.
   • Pods access the internet through NAT via the node's network interface.

NAT Gateway Configuration:

# Create NAT gateway
aws ec2 create-nat-gateway \
  --subnet-id subnet-public1 \
  --allocation-id eipalloc-12345

# Update private subnet routing table
aws ec2 create-route \
  --route-table-id rtb-private \
  --destination-cidr-block 0.0.0.0/0 \
  --nat-gateway-id nat-12345

VPC Configuration Example Using CloudFormation:

Resources:
  VPC:
    Type: AWS::EC2::VPC
    Properties:
      CidrBlock: 10.0.0.0/16
      EnableDnsSupport: true
      EnableDnsHostnames: true
      Tags:
        - Key: Name
          Value: EKS-VPC

  PublicSubnet1:
    Type: AWS::EC2::Subnet
    Properties:
      VpcId: !Ref VPC
      AvailabilityZone: !Select [0, !GetAZs ""]
      CidrBlock: 10.0.0.0/24
      MapPublicIpOnLaunch: true
      Tags:
        - Key: Name
          Value: Public-Subnet-1

  PrivateSubnet1:
    Type: AWS::EC2::Subnet
    Properties:
      VpcId: !Ref VPC
      AvailabilityZone: !Select [0, !GetAZs ""]
      CidrBlock: 10.0.2.0/24
      Tags:
        - Key: Name
          Value: Private-Subnet-1

  InternetGateway:
    Type: AWS::EC2::InternetGateway
    Properties:
      Tags:
        - Key: Name
          Value: EKS-IGW

  VPCGatewayAttachment:
    Type: AWS::EC2::VPCGatewayAttachment
    Properties:
      VpcId: !Ref VPC
      InternetGatewayId: !Ref InternetGateway

  NatGatewayEIP:
    Type: AWS::EC2::EIP
    DependsOn: VPCGatewayAttachment
    Properties:
      Domain: vpc

  NatGateway:
    Type: AWS::EC2::NatGateway
    Properties:
      AllocationId: !GetAtt NatGatewayEIP.AllocationId
      SubnetId: !Ref PublicSubnet1
      Tags:
        - Key: Name
          Value: EKS-NAT-GW

  PublicRouteTable:
    Type: AWS::EC2::RouteTable
    Properties:
      VpcId: !Ref VPC
      Tags:
        - Key: Name
          Value: Public-RT

  PublicRoute:
    Type: AWS::EC2::Route
    DependsOn: VPCGatewayAttachment
    Properties:
      RouteTableId: !Ref PublicRouteTable
      DestinationCidrBlock: 0.0.0.0/0
      GatewayId: !Ref InternetGateway

  PrivateRouteTable:
    Type: AWS::EC2::RouteTable
    Properties:
      VpcId: !Ref VPC
      Tags:
        - Key: Name
          Value: Private-RT

  PrivateRoute:
    Type: AWS::EC2::Route
    Properties:
      RouteTableId: !Ref PrivateRouteTable
      DestinationCidrBlock: 0.0.0.0/0
      NatGatewayId: !Ref NatGateway

VPC Configuration Using eksctl:

apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig
metadata:
  name: my-cluster
  region: us-west-2
vpc:
  cidr: 192.168.0.0/16
  nat:
    gateway: Single  # NAT gateway configuration
  clusterEndpoints:
    publicAccess: true
    privateAccess: true

Issues with Other Options:

• Attach an internet gateway to the subnet where the pod is located: An internet gateway is attached to public subnets, and resources in private subnets need a NAT gateway to access the internet. Additionally, an internet gateway alone does not allow pods to access the internet.
• Assign a public IP address to the pod: The Amazon VPC CNI plugin does not support assigning public IP addresses to pods. Pods always receive private IP addresses.
• Associate an Elastic IP address with the pod: You cannot directly associate an Elastic IP address with a pod. Elastic IP addresses can only be associated with EC2 instances or network interfaces.

A NAT gateway or NAT instance is the standard method for allowing pods in private subnets to access the internet. It translates the pod's private IP address to the NAT gateway's public IP address to enable internet communication. For production environments, it is recommended to deploy a NAT gateway in each availability zone for high availability.

4. Which of the following is NOT a requirement for applying security groups to pods using SecurityGroupPolicy in an Amazon EKS cluster?
   • A) Amazon VPC CNI plugin version 1.7.7 or later
   • B) ENIConfig resource configuration
   • C) Setting hostNetwork: true on the pod
   • D) Specifying a service account for pods to apply security groups

Show Answer

Answer: C) Setting hostNetwork: true on the pod

Explanation: "Setting hostNetwork: true on the pod" is NOT a requirement for applying security groups to pods using SecurityGroupPolicy in an Amazon EKS cluster. In fact, security groups cannot be applied to pods with hostNetwork: true. To apply security groups to pods, the pods must use their own network namespace.

Pod Security Group Feature Requirements:

1. Amazon VPC CNI Plugin Version 1.7.7 or Later:

   • The pod security group feature is supported in Amazon VPC CNI plugin version 1.7.7 and later.
   • It is recommended to use the latest version.

   # Check CNI version
   kubectl describe daemonset aws-node -n kube-system | grep Image
   
   # Update CNI
   kubectl apply -f https://raw.githubusercontent.com/aws/amazon-vpc-cni-k8s/v1.10.0/config/master/aws-k8s-cni.yaml

2. Enable Pod Security Group Feature:

   # Enable pod security group feature
   kubectl set env daemonset aws-node -n kube-system ENABLE_POD_ENI=true
   
   # Or use eksctl
   eksctl utils update-cluster-config \
     --name my-cluster \
     --region us-west-2 \
     --enable-pod-security-groups

3. Specifying a Service Account for Pods to Apply Security Groups:

   • In SecurityGroupPolicy, you must specify pods to apply security groups using a pod selector or service account selector.

   apiVersion: vpcresources.k8s.aws/v1beta1
   kind: SecurityGroupPolicy
   metadata:
     name: my-security-group-policy
     namespace: default
   spec:
     podSelector:
       matchLabels:
         app: my-app
     securityGroups:
       groupIds:
         - sg-12345

Pod Security Group Configuration Examples:

1. Create SecurityGroupPolicy:

   apiVersion: vpcresources.k8s.aws/v1beta1
   kind: SecurityGroupPolicy
   metadata:
     name: db-client-policy
     namespace: default
   spec:
     serviceAccountSelector:
       matchLabels:
         role: db-client
     securityGroups:
       groupIds:
         - sg-db-client

2. Create Service Account:

   apiVersion: v1
   kind: ServiceAccount
   metadata:
     name: db-client
     namespace: default
     labels:
       role: db-client

3. Deploy Pod:

   apiVersion: v1
   kind: Pod
   metadata:
     name: db-client-pod
   spec:
     serviceAccountName: db-client
     containers:
     - name: db-client
       image: mysql:5.7
       command: ['sleep', '3600']

Issues with hostNetwork: true:

Pods with hostNetwork: true use the node's network namespace, so separate security groups cannot be applied to the pod. Such pods inherit the node's security groups.

# Security groups cannot be applied to this pod
apiVersion: v1
kind: Pod
metadata:
  name: host-network-pod
spec:
  hostNetwork: true  # Uses the node's network namespace
  containers:
  - name: nginx
    image: nginx

ENIConfig Resource Configuration:

ENIConfig resources are required when configuring custom networking, but they are not a mandatory requirement when using only the pod security groups feature. However, if you are using pod security groups along with custom networking, you must configure ENIConfig resources.

apiVersion: crd.k8s.amazonaws.com/v1alpha1
kind: ENIConfig
metadata:
  name: us-west-2a
spec:
  subnet: subnet-12345
  securityGroups:
  - sg-12345

Pod Security Group Limitations:

1. Resource Limitations:
   • Each node requires additional ENIs for pod security groups.
   • The maximum number of supported ENIs is limited by instance type.
2. Compatibility Limitations:
   • Cannot be applied to pods with hostNetwork: true.
   • Cannot be applied to pods using hostPort.
   • May not be compatible with some CNI plugins.
3. Performance Impact:
   • Since additional ENIs are required per pod, pod startup time may be longer.
   • The maximum number of pods per node may be limited.

The pod security groups feature is a powerful capability that provides granular network security at the pod level. However, since this feature cannot be applied to pods with hostNetwork: true, pods must use their own network namespace to use pod security groups.

4. What is NOT a requirement for applying security groups to pods using SecurityGroupPolicy in an Amazon EKS cluster? - A) Amazon VPC CNI plugin version 1.7.7 or higher - B) ENIConfig resource configuration - C) Setting hostNetwork: true on the pod - D) Specifying a service account for pods to which security groups will be applied

Show Answer

Answer: C) Setting hostNetwork: true on the pod

Explanation: The option that is NOT a requirement for applying security groups to pods using SecurityGroupPolicy in an Amazon EKS cluster is "Setting hostNetwork: true on the pod". In fact, security groups cannot be applied to pods with hostNetwork: true. To apply security groups to pods, the pods must use their own network namespace.

Pod Security Groups Feature Requirements:

1. Amazon VPC CNI plugin version 1.7.7 or higher:

   • The pod security groups feature is supported in Amazon VPC CNI plugin version 1.7.7 and later.
   • Using the latest version is recommended.

   # Check CNI version
   kubectl describe daemonset aws-node -n kube-system | grep Image
   
   # Update CNI
   kubectl apply -f https://raw.githubusercontent.com/aws/amazon-vpc-cni-k8s/v1.10.0/config/master/aws-k8s-cni.yaml

2. Enable pod security groups feature:

   # Enable pod security groups feature
   kubectl set env daemonset aws-node -n kube-system ENABLE_POD_ENI=true
   
   # Or use eksctl
   eksctl utils update-cluster-config \
     --name my-cluster \
     --region us-west-2 \
     --enable-pod-security-groups

3. Specify service account for pods to which security groups will be applied:

   • In SecurityGroupPolicy, you must specify pods to which security groups will be applied using pod selector or service account selector.

   apiVersion: vpcresources.k8s.aws/v1beta1
   kind: SecurityGroupPolicy
   metadata:
     name: my-security-group-policy
     namespace: default
   spec:
     podSelector:
       matchLabels:
         app: my-app
     securityGroups:
       groupIds:
         - sg-12345

Pod Security Group Configuration Examples:

1. Create SecurityGroupPolicy:

   apiVersion: vpcresources.k8s.aws/v1beta1
   kind: SecurityGroupPolicy
   metadata:
     name: db-client-policy
     namespace: default
   spec:
     serviceAccountSelector:
       matchLabels:
         role: db-client
     securityGroups:
       groupIds:
         - sg-db-client

2. Create Service Account:

   apiVersion: v1
   kind: ServiceAccount
   metadata:
     name: db-client
     namespace: default
     labels:
       role: db-client

3. Deploy Pod:

   apiVersion: v1
   kind: Pod
   metadata:
     name: db-client-pod
   spec:
     serviceAccountName: db-client
     containers:
     - name: db-client
       image: mysql:5.7
       command: ['sleep', '3600']

Issues with hostNetwork: true:

Pods with hostNetwork: true use the node's network namespace, so separate security groups cannot be applied to the pod. Such pods inherit the node's security groups.

# Security groups cannot be applied to this pod
apiVersion: v1
kind: Pod
metadata:
  name: host-network-pod
spec:
  hostNetwork: true  # Uses the node's network namespace
  containers:
  - name: nginx
    image: nginx

ENIConfig Resource Configuration:

ENIConfig resources are required when configuring custom networking, but they are not a mandatory requirement when using only the pod security groups feature. However, if you are using pod security groups along with custom networking, you must configure ENIConfig resources.

apiVersion: crd.k8s.amazonaws.com/v1alpha1
kind: ENIConfig
metadata:
  name: us-west-2a
spec:
  subnet: subnet-12345
  securityGroups:
  - sg-12345

Pod Security Group Limitations:

1. Resource Limitations:
   • Each node requires additional ENIs for pod security groups.
   • The maximum number of supported ENIs is limited by instance type.
2. Compatibility Limitations:
   • Cannot be applied to pods with hostNetwork: true.
   • Cannot be applied to pods using hostPort.
   • May not be compatible with some CNI plugins.
3. Performance Impact:
   • Since additional ENIs are required per pod, pod startup time may be longer.
   • The maximum number of pods per node may be limited.

The pod security groups feature is a powerful capability that provides granular network security at the pod level. However, since this feature cannot be applied to pods with hostNetwork: true, pods must use their own network namespace to use pod security groups.

5. What is the main benefit of the Prefix Delegation feature in an Amazon EKS cluster?
   • A) Improved communication speed between pods
   • B) Increased maximum number of pods per node
   • C) Ability to assign public IP addresses to pods
   • D) Enhanced pod network isolation

Show Answer

Answer: B) Increased maximum number of pods per node

Explanation: The main benefit of the Prefix Delegation feature in an Amazon EKS cluster is increasing the maximum number of pods per node. This feature allocates /28 CIDR blocks (16 IP addresses) instead of individual IP addresses to each Elastic Network Interface (ENI), significantly increasing the maximum number of pods that a node can support.

How Prefix Delegation Works:

1. Default VPC CNI Behavior:
   • By default, VPC CNI allocates secondary IP addresses from the ENI for each pod.
   • Each EC2 instance type has limits on the maximum number of ENIs and IP addresses per ENI.
   • This limits the maximum number of pods per node.
2. Prefix Delegation Behavior:
   • When prefix delegation is enabled, /28 CIDR blocks (16 IPs) are allocated to each ENI instead of individual IP addresses.
   • This significantly increases the number of IP addresses each ENI can support.
   • As a result, the maximum number of pods per node increases.

Enabling Prefix Delegation:

# Enable prefix delegation
kubectl set env daemonset aws-node -n kube-system ENABLE_PREFIX_DELEGATION=true

# Check prefix delegation status
kubectl describe daemonset aws-node -n kube-system | grep ENABLE_PREFIX_DELEGATION

Benefits of Prefix Delegation:

1. Increased Maximum Pods per Node:
   • Using prefix delegation significantly increases the maximum number of pods per node.
   • For example, for an m5.large instance:
     • Default configuration: maximum 29 pods
     • With prefix delegation enabled: over 110 pods maximum
2. IP Address Efficiency:
   • Optimizes IP address usage in large clusters.
   • Useful in environments with limited VPC CIDR ranges.
3. Improved Node Resource Utilization:
   • Running more pods improves node resource utilization.
   • Helps with cluster cost optimization.

Prefix Delegation Limitations:

1. EC2 Instance Support:
   • Only Nitro-based instances support prefix delegation.
   • Cannot be used with older generation instances.
2. VPC CNI Version Requirements:
   • VPC CNI version 1.9.0 or higher is required.
   • This feature cannot be used with earlier versions.
3. Subnet Size Requirements:
   • Subnets with sufficient IP address space are required.
   • IP addresses may be exhausted quickly in small subnets.
4. Migration Considerations:
   • When enabled on existing clusters, only new pods use prefix delegation.
   • Existing pods must be restarted to apply to all pods.

Prefix Delegation Configuration Example:

# eksctl configuration file
apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig
metadata:
  name: my-cluster
  region: us-west-2
vpc:
  clusterEndpoints:
    publicAccess: true
    privateAccess: true
managedNodeGroups:
  - name: ng-1
    instanceType: m5.large
    minSize: 2
    maxSize: 5
    disableIMDSv1: true
iam:
  withOIDC: true
addons:
  - name: vpc-cni
    version: latest
    configurationValues: |
      {
        "env": {
          "ENABLE_PREFIX_DELEGATION": "true"
        }
      }

Issues with Other Options:

• Improved communication speed between pods: Prefix delegation does not directly affect pod-to-pod communication speed. Pod communication performance is primarily determined by network infrastructure and CNI implementation.
• Ability to assign public IP addresses to pods: Prefix delegation does not provide the ability to assign public IP addresses to pods. VPC CNI always assigns private IP addresses to pods.
• Enhanced pod network isolation: Prefix delegation is not related to pod network isolation. Network isolation is implemented through network policies or security groups.

Prefix delegation is a powerful feature that increases the maximum number of pods per node, improving cluster density and efficiency. It is particularly useful in large clusters or environments running high-density workloads.

6. What is the correct way to customize CoreDNS in an Amazon EKS cluster?
   • A) Modify CoreDNS settings in the AWS Management Console
   • B) Modify the CoreDNS ConfigMap
   • C) Specify CoreDNS configuration during EKS cluster creation
   • D) Update CoreDNS add-on parameters using the AWS CLI

Show Answer

Answer: B) Modify the CoreDNS ConfigMap

Explanation: The correct way to customize CoreDNS in an Amazon EKS cluster is to modify the CoreDNS ConfigMap. CoreDNS is the cluster DNS server for Kubernetes and is configured through a ConfigMap. In EKS, you can customize CoreDNS behavior by modifying the coredns ConfigMap.

How to Modify the CoreDNS ConfigMap:

1. Check Current ConfigMap:

   kubectl get configmap coredns -n kube-system -o yaml

2. Edit ConfigMap:

   kubectl edit configmap coredns -n kube-system

3. Or Apply a Patch:

   kubectl patch configmap coredns -n kube-system --type=merge -p '{"data":{"Corefile":".:53 {\n    errors\n    health {\n        lameduck 5s\n    }\n    ready\n    kubernetes cluster.local in-addr.arpa ip6.arpa {\n        pods insecure\n        fallthrough in-addr.arpa ip6.arpa\n        ttl 30\n    }\n    prometheus :9153\n    forward . /etc/resolv.conf\n    cache 30\n    loop\n    reload\n    loadbalance\n    # Add custom settings\n    hosts {\n        10.0.0.1 example.com\n        fallthrough\n    }\n}"}}'

Common CoreDNS Customization Cases:

1. Adding Custom DNS Records:

   hosts {
       10.0.0.1 example.com
       10.0.0.2 api.example.com
       fallthrough
   }

2. Configuring Forwarding for Specific Domains:

   forward example.org 10.0.0.1:53

3. Adjusting DNS Caching:

   cache {
       success 10000
       denial 5000
       prefetch 10 10 10%
   }

4. Configuring Logging:

   log {
       class error
   }

5. Disabling Autopath:

   kubernetes cluster.local in-addr.arpa ip6.arpa {
       pods insecure
       fallthrough in-addr.arpa ip6.arpa
       ttl 30
       autopath off
   }

Applying Changes After CoreDNS Modification:

After modifying the ConfigMap, you need to restart the CoreDNS pods to apply the changes:

# Check CoreDNS pods
kubectl get pods -n kube-system -l k8s-app=kube-dns

# Restart CoreDNS pods
kubectl rollout restart deployment coredns -n kube-system

# Verify changes are applied
kubectl logs -n kube-system -l k8s-app=kube-dns

CoreDNS Performance Optimization:

1. Configure Auto-scaling:

   apiVersion: autoscaling/v2
   kind: HorizontalPodAutoscaler
   metadata:
     name: coredns-autoscaler
     namespace: kube-system
   spec:
     scaleTargetRef:
       apiVersion: apps/v1
       kind: Deployment
       name: coredns
     minReplicas: 2
     maxReplicas: 10
     metrics:
     - type: Resource
       resource:
         name: cpu
         target:
           type: Utilization
           averageUtilization: 60

2. Adjust Resource Requests and Limits:

   kubectl patch deployment coredns -n kube-system --type=json -p='[{"op": "replace", "path": "/spec/template/spec/containers/0/resources", "value": {"requests": {"cpu": "100m", "memory": "70Mi"}, "limits": {"cpu": "200m", "memory": "170Mi"}}}]'

Issues with Other Options:

• Modify CoreDNS settings in the AWS Management Console: The AWS Management Console does not provide an interface to directly modify CoreDNS settings.
• Specify CoreDNS configuration during EKS cluster creation: Detailed CoreDNS configuration cannot be specified during EKS cluster creation. You must modify the ConfigMap after cluster creation.
• Update CoreDNS add-on parameters using the AWS CLI: While you can update the CoreDNS add-on version using the AWS CLI, you cannot modify detailed configuration. Configuration changes must be made through the ConfigMap.

# Update CoreDNS add-on version (not configuration change)
aws eks update-addon \
  --cluster-name my-cluster \
  --addon-name coredns \
  --addon-version v1.8.7-eksbuild.2 \
  --resolve-conflicts PRESERVE

Modifying the CoreDNS ConfigMap is the standard method for customizing DNS settings in an EKS cluster. This enables various customizations such as adding custom DNS records, configuring forwarding for specific domains, and adjusting caching behavior.

7. What is the most effective way to implement multi-tenancy in an Amazon EKS cluster? - A) Create separate EKS clusters for each tenant - B) Use separate namespaces for each tenant and apply RBAC, network policies, and resource quotas - C) Create separate node groups for each tenant and use node selectors - D) Use separate VPCs for each tenant

Show Answer

Answer: B) Use separate namespaces for each tenant and apply RBAC, network policies, and resource quotas

Explanation: The most effective way to implement multi-tenancy in an Amazon EKS cluster is to use separate namespaces for each tenant and apply RBAC (Role-Based Access Control), network policies, and resource quotas. This approach efficiently isolates multiple tenants within a single cluster while allowing resource sharing.

Implementing Namespace-Based Multi-Tenancy:

1. Create Namespaces for Each Tenant:

   # Create namespaces for each tenant
   kubectl create namespace tenant-a
   kubectl create namespace tenant-b

2. Configure RBAC:

   # Create tenant role
   apiVersion: rbac.authorization.k8s.io/v1
   kind: Role
   metadata:
     name: tenant-full-access
     namespace: tenant-a
   rules:
   - apiGroups: ["", "apps", "batch"]
     resources: ["*"]
     verbs: ["*"]
   ---
   # Bind role to tenant users
   apiVersion: rbac.authorization.k8s.io/v1
   kind: RoleBinding
   metadata:
     name: tenant-a-access
     namespace: tenant-a
   subjects:
   - kind: Group
     name: tenant-a-users
     apiGroup: rbac.authorization.k8s.io
   roleRef:
     kind: Role
     name: tenant-full-access
     apiGroup: rbac.authorization.k8s.io

3. Apply Network Policies:

   # Restrict communication between tenants
   apiVersion: networking.k8s.io/v1
   kind: NetworkPolicy

metadata: name: deny-cross-tenant-traffic namespace: tenant-a spec: podSelector: {} policyTypes: - Ingress - Egress ingress: - from: - namespaceSelector: matchLabels: name: tenant-a egress: - to: - namespaceSelector: matchLabels: name: tenant-a - to: - namespaceSelector: matchLabels: name: kube-system

4. **Resource Quota Configuration**:
 ```yaml
 # Tenant Resource Quota
 apiVersion: v1
 kind: ResourceQuota
 metadata:
   name: tenant-quota
   namespace: tenant-a
 spec:
   hard:
     requests.cpu: "10"
     requests.memory: 20Gi
     limits.cpu: "20"
     limits.memory: 40Gi
     pods: "50"
     services: "20"
     persistentvolumeclaims: "30"
     secrets: "100"
     configmaps: "100"

5. LimitRange Configuration:

   # Default resource limit settings
   apiVersion: v1
   kind: LimitRange
   metadata:
     name: tenant-limits
     namespace: tenant-a
   spec:
     limits:
     - default:
         cpu: 500m
         memory: 512Mi
       defaultRequest:
         cpu: 100m
         memory: 256Mi
       type: Container

Benefits of Namespace-Based Multi-Tenancy:

1. Resource Efficiency:
   • Resource utilization improves as multiple tenants share a single cluster.
   • Control plane costs are reduced.
2. Ease of Management:
   • Operational overhead decreases by managing a single cluster.
   • Centralized monitoring and logging is possible.
3. Flexibility:
   • Adding and removing tenants is straightforward.
   • Tenant-specific policies can be easily applied.
4. Cost Efficiency:
   • Cluster overhead is shared among multiple tenants.
   • Improved resource utilization leads to cost savings.

Drawbacks of Namespace-Based Multi-Tenancy:

1. Limited Isolation Level:
   • Namespaces provide only logical isolation, not complete physical isolation.
   • May be exposed to kernel-level vulnerabilities.
2. Resource Contention:
   • Resource contention between tenants can occur.
   • Noisy Neighbor problems may arise.
3. Security Risks:
   • Risk of cluster-level privilege escalation exists.
   • May be exposed to container escape vulnerabilities.

Other Multi-Tenancy Approaches:

1. Cluster-Based Multi-Tenancy (Separate EKS Cluster per Tenant):
   • Provides the strongest isolation
   • Increased management overhead
   • Increased costs
   • Suitable for large enterprise environments or heavily regulated industries
2. Node-Based Multi-Tenancy (Separate Node Group per Tenant):
   • Provides intermediate level of isolation
   • Node-level customization possible per tenant
   • Reduced resource utilization
   • Suitable when security requirements are high but cost is also a consideration
3. Hybrid Approach:
   • Provide dedicated clusters for critical tenants
   • Separate less critical tenants by namespace in a shared cluster
   • Balance flexibility and cost efficiency

Issues with Other Options:

• Creating a Separate EKS Cluster per Tenant: Provides the strongest isolation, but significantly increases management overhead and costs. Scalability issues may arise when there are many tenants.
• Creating Separate Node Groups per Tenant and Using Node Selectors: Provides node-level isolation, but resource utilization decreases and management can become complex. Also, node groups alone do not provide complete isolation.
• Using Separate VPCs per Tenant: Since EKS clusters are created within a single VPC, using separate VPCs per tenant requires creating separate clusters per tenant. This significantly increases management overhead and costs.

Namespace-based multi-tenancy provides the optimal balance between isolation, ease of management, and cost efficiency for most use cases. However, cluster-based multi-tenancy should be considered when security requirements are very high.

8. Which of the following is NOT a factor to consider when selecting an instance type for a node group in an Amazon EKS cluster?
   • A) CPU and memory requirements of the workload
   • B) Cost optimization
   • C) Kubernetes version of the cluster
   • D) Required pod density

Show Answer

Answer: C) Kubernetes version of the cluster

Explanation: The factor that is NOT to be considered when selecting an instance type for a node group in an Amazon EKS cluster is "Kubernetes version of the cluster." While the Kubernetes version affects supported features, it does not directly impact the node group's instance type selection. Instance type is primarily determined by workload requirements, cost, pod density, and other factors.

Actual Factors to Consider When Selecting Node Group Instance Types:

1. CPU and Memory Requirements of the Workload:
   • Select instance types that match the resource requirements of your workload
   • CPU-intensive workloads: Compute-optimized instances such as c5, c6g
   • Memory-intensive workloads: Memory-optimized instances such as r5, r6g
   • Balanced workloads: General-purpose instances such as m5, m6g
   • GPU workloads: Accelerated computing instances such as p3, g4dn
2. Cost Optimization:
   • On-Demand vs Spot instances
   • Reserved Instances or Savings Plans
   • Cost savings through ARM-based Graviton instances (e.g., m6g, c6g)
   • Selecting appropriately sized instances (avoiding overprovisioning)
3. Required Pod Density:
   • Maximum supported pods varies by instance type
   • Consider the number of ENIs per instance type and IP addresses per ENI
   • For high-density workloads, select instance types that support more ENIs and IP addresses
4. Networking Requirements:
   • Network bandwidth requirements
   • Enhanced networking support (ENA, EFA, etc.)
   • Network performance varies by instance type
5. Storage Requirements:
   • Whether local instance storage is needed (e.g., i3, d3 instances)
   • EBS optimization support
   • Storage throughput and IOPS requirements
6. Availability Requirements:
   • Regional availability of instance types
   • Interruption possibility when using Spot instances
   • Instance type availability by Availability Zone

Instance Type Selection Examples:

1. Web Application Servers:

   apiVersion: eksctl.io/v1alpha5
   kind: ClusterConfig
   metadata:
     name: my-cluster
     region: us-west-2
   managedNodeGroups:
     - name: web-servers
       instanceType: m5.large
       minSize: 2
       maxSize: 10
       labels:
         role: web

2. Database Workloads:

   apiVersion: eksctl.io/v1alpha5
   kind: ClusterConfig
   metadata:
     name: my-cluster
     region: us-west-2
   managedNodeGroups:
     - name: database-nodes
       instanceType: r5.xlarge
       minSize: 3
       maxSize: 5
       labels:
         role: database

3. Batch Processing Workloads:

   apiVersion: eksctl.io/v1alpha5
   kind: ClusterConfig
   metadata:
     name: my-cluster
     region: us-west-2
   managedNodeGroups:
     - name: batch-processors
       instanceType: c5.2xlarge
       minSize: 0
       maxSize: 20
       labels:
         role: batch

4. Cost-Optimized Workloads:

   apiVersion: eksctl.io/v1alpha5
   kind: ClusterConfig
   metadata:
     name: my-cluster
     region: us-west-2
   managedNodeGroups:
     - name: spot-workers
       instanceTypes: ["m5.large", "m5a.large", "m5d.large", "m5ad.large"]
       minSize: 2
       maxSize: 10
       spot: true
       labels:
         lifecycle: spot

Relationship Between Kubernetes Version and Instance Types:

The Kubernetes version affects the cluster in the following ways, but does not directly impact instance type selection:

1. Supported Features:
   • New Kubernetes versions provide new features.
   • Some features are only available in specific versions.
2. API Compatibility:
   • Some APIs may change or be removed in new versions.
   • Version selection is important if your application depends on specific APIs.
3. Security Patches:
   • Latest versions include the latest security patches.
   • Older versions may be exposed to security vulnerabilities.
4. Support Period:
   • Each Kubernetes version is supported for a limited period.
   • EKS supports each version for approximately 14 months.

Instance type selection is primarily determined by workload resource requirements, cost optimization, pod density, and other factors, and is not directly related to the Kubernetes version. Therefore, "Kubernetes version of the cluster" is not a primary factor to consider when selecting a node group's instance type.

9. What is the most effective method to minimize pod disruption during node group updates in an Amazon EKS cluster? - A) Using a rolling update strategy - B) Configuring PodDisruptionBudget - C) Manually migrating all pods before node group update - D) Using a blue/green deployment strategy

Show Answer

Answer: B) Configuring PodDisruptionBudget

Explanation: The most effective method to minimize pod disruption during node group updates in an Amazon EKS cluster is to configure PodDisruptionBudget (PDB). PDB limits the number of pods that can be disrupted simultaneously during voluntary disruptions, ensuring application availability. Since node group updates are considered voluntary disruptions, you can maintain application availability during updates through PDB.

How PodDisruptionBudget Works:

1. PDB Definition:
   • minAvailable: Specifies the minimum number or percentage of pods that must always be available
   • maxUnavailable: Specifies the maximum number or percentage of pods that can be unavailable simultaneously
   • Only one of these two options should be specified
2. PDB Application:
   • Kubernetes respects the PDB during node draining
   • The draining process pauses if PDB is violated
   • Draining continues when new pods start running on other nodes

PodDisruptionBudget Examples:

# Ensure at least 2 pods are always available
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: app-pdb
  namespace: default
spec:
  minAvailable: 2
  selector:
    matchLabels:
      app: my-app

# Limit only up to 50% of pods to be unavailable simultaneously
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: app-pdb
  namespace: default
spec:
  maxUnavailable: 50%
  selector:
    matchLabels:
      app: my-app

EKS Node Group Update Process:

1. Update Start:
   • Create new nodes
   • New nodes join the cluster
2. Node Draining:
   • Apply cordoning to existing nodes (prevent new pod scheduling)
   • Drain pods from existing nodes (migrate pods)
   • Migrate pods while respecting PDB
3. Node Termination:
   • Terminate nodes after all pods are migrated
   • Repeat process for the next node

PDB Configuration Best Practices:

1. Set Appropriate Replica Count:
   • Sufficient replicas are needed for PDB to work effectively
   • At least 3 replicas recommended
2. Choose Appropriate PDB Values:
   • Select values appropriate for application characteristics
   • Too restrictive values can delay updates
   • Too loose values can impact availability
3. Apply PDB to All Critical Workloads:
   • Stateful applications
   • User-facing services
   • System components
4. Test PDB:
   • Test PDB behavior before updates
   • Verify availability with draining simulation

Node Group Update Configuration:

# Modify managed node group update configuration
aws eks update-nodegroup-config \
  --cluster-name my-cluster \
  --nodegroup-name my-nodegroup \
  --update-config '{"maxUnavailable": 1}'

# Or using eksctl
eksctl update nodegroup \
  --cluster my-cluster \
  --name my-nodegroup \
  --max-unavailable 1

Issues with Other Options:

• Using a Rolling Update Strategy: EKS managed node groups already use a rolling update strategy by default. However, rolling updates alone cannot control pod disruption, and they must be used together with PDB to be effective.
• Manually Migrating All Pods Before Node Group Update: This is a time-consuming and error-prone manual process. It is also not practical for large clusters.
• Using a Blue/Green Deployment Strategy: Blue/green deployment involves creating a new node group, migrating workloads, and then deleting the existing node group. This is an effective strategy, but has drawbacks of increased costs due to resource duplication and complex implementation. It is also best used together with PDB.

PodDisruptionBudget is a Kubernetes-native way to control pod disruption during node group updates, allowing you to safely update node groups while ensuring application availability. Therefore, the most effective method to minimize pod disruption during node group updates is to configure PodDisruptionBudget.

10. Which of the following is NOT used to control Auto Scaling behavior of node groups in an Amazon EKS cluster?
    • A) Cluster Autoscaler
    • B) Karpenter
    • C) Horizontal Pod Autoscaler
    • D) Vertical Pod Autoscaler

Show Answer

Answer: D) Vertical Pod Autoscaler

Explanation: The one that is NOT used to control Auto Scaling behavior of node groups in an Amazon EKS cluster is Vertical Pod Autoscaler (VPA). VPA is used to automatically adjust the CPU and memory requests of pods, but it is not used to adjust the size of node groups. Node group Auto Scaling is primarily controlled by Cluster Autoscaler, Karpenter, and indirectly by Horizontal Pod Autoscaler (HPA).

Node Group Auto Scaling Tools:

1. Cluster Autoscaler:

   • A Kubernetes component that automatically adjusts the size of node groups
   • Adds nodes when pods cannot be scheduled
   • Removes nodes when they are not sufficiently utilized
   • Integrates with AWS Auto Scaling Groups

   # Cluster Autoscaler Deployment
   apiVersion: apps/v1
   kind: Deployment
   metadata:
     name: cluster-autoscaler
     namespace: kube-system
   spec:
     replicas: 1
     selector:
       matchLabels:
         app: cluster-autoscaler
     template:
       metadata:
         labels:
           app: cluster-autoscaler
       spec:
         serviceAccountName: cluster-autoscaler
         containers:
         - image: k8s.gcr.io/autoscaling/cluster-autoscaler:v1.23.0
           name: cluster-autoscaler
           command:
           - ./cluster-autoscaler
           - --v=4
           - --stderrthreshold=info
           - --cloud-provider=aws
           - --skip-nodes-with-local-storage=false
           - --expander=least-waste
           - --node-group-auto-discovery=asg:tag=k8s.io/cluster-autoscaler/enabled,k8s.io/cluster-autoscaler/my-cluster

2. Karpenter:

   • AWS's open-source node provisioning project
   • Selects optimal instance types for workload requirements
   • Fast node provisioning (in seconds)
   • Cost optimization and unified lifecycle management

   # Karpenter Provisioner
   apiVersion: karpenter.sh/v1alpha5
   kind: NodePool
   metadata:
     name: default
   spec:
     template:
       spec:
         requirements:
           - key: karpenter.sh/capacity-type
             operator: In
             values: ["spot", "on-demand"]
         nodeClassRef:
           name: default-class
     limits:
       cpu: 1000
         memory: 1000Gi
     provider:
       subnetSelector:
         karpenter.sh/discovery: "true"
       securityGroupSelector:
         karpenter.sh/discovery: "true"
     ttlSecondsAfterEmpty: 30

3. Horizontal Pod Autoscaler (HPA):

   • Automatically adjusts the number of pod replicas
   • Based on CPU, memory, or custom metrics
   • Can indirectly trigger node group Auto Scaling
   • Works together with Cluster Autoscaler or Karpenter

   # Horizontal Pod Autoscaler
   apiVersion: autoscaling/v2
   kind: HorizontalPodAutoscaler
   metadata:
     name: my-app-hpa
   spec:
     scaleTargetRef:
       apiVersion: apps/v1
       kind: Deployment
       name: my-app
     minReplicas: 2
     maxReplicas: 10
     metrics:
     - type: Resource
       resource:
         name: cpu
         target:
           type: Utilization
           averageUtilization: 70

Vertical Pod Autoscaler (VPA):

VPA is used to automatically adjust the CPU and memory requests of pods, but does not directly adjust the size of node groups:

# Vertical Pod Autoscaler
apiVersion: autoscaling.k8s.io/v1
kind: VerticalPodAutoscaler
metadata:
  name: my-app-vpa
spec:
  targetRef:
    apiVersion: "apps/v1"
    kind: Deployment
    name: my-app
  updatePolicy:
    updateMode: "Auto"
  resourcePolicy:
    containerPolicies:
    - containerName: '*'
      minAllowed:
        cpu: 100m
        memory: 50Mi
      maxAllowed:
        cpu: 1
        memory: 500Mi
      controlledResources: ["cpu", "memory"]

VPA provides the following features:

• Automatic adjustment of pod resource requests
• Recommendations based on resource usage
• Resource request updates through pod restarts

However, VPA does not directly adjust the size of node groups and only affects resource allocation at the pod level.

Node Group Auto Scaling Strategies:

1. Reactive Scaling:
   • Uses Cluster Autoscaler
   • Adds nodes when pods cannot be scheduled
   • Removes nodes when resource utilization is low
   • Suitable for predictable workloads
2. Predictive Scaling:
   • Uses AWS Auto Scaling predictive scaling
   • Predicts future demand based on historical patterns
   • Secures capacity before demand increases
   • Suitable for workloads with periodic patterns
3. Event-driven Scaling:
   • Uses KEDA (Kubernetes Event-driven Autoscaling)
   • Scaling based on external events or metrics
   • Scaling based on queue length, event count, etc.
   • Suitable for batch processing, event processing workloads

Explanation of Other Options:

• Cluster Autoscaler: A Kubernetes component that directly controls node group Auto Scaling, adding or removing nodes based on pod scheduling requirements.
• Karpenter: An AWS open-source node provisioning project that quickly provisions optimal instances matching workload requirements. It can be used as an alternative to Cluster Autoscaler.
• Horizontal Pod Autoscaler: Automatically adjusts the number of pod replicas, which can indirectly trigger Cluster Autoscaler or Karpenter to adjust node group size when more pods are created.

Vertical Pod Autoscaler is used to adjust pod resource requests but does not directly adjust node group size. Therefore, Vertical Pod Autoscaler is the one that is NOT used to control node group Auto Scaling behavior.

Hands-on Exercises

Exercise 1: Implementing Network Policies in an EKS Cluster

Scenario: You are a security engineer at your company and need to restrict network traffic between microservices in an EKS cluster. Specifically, only the frontend service should be able to access the backend API, and the database should only be accessible from the backend API.

Requirements:

1. Install Calico network policy engine
2. Implement default deny policy
3. Allow traffic from frontend to backend
4. Allow traffic from backend to database
5. Test policies

Solution:

Show Solution

1. Install Calico Network Policy Engine

# Install Tigera Operator
kubectl create namespace tigera-operator
helm repo add projectcalico https://docs.projectcalico.org/charts
helm install calico projectcalico/tigera-operator --namespace tigera-operator

# Verify installation
kubectl get pods -n calico-system

2. Create Namespace and Sample Applications

# Create namespace
kubectl create namespace microservices

# Deploy frontend
cat > frontend.yaml << EOF
apiVersion: apps/v1
kind: Deployment
metadata:
  name: frontend
  namespace: microservices
  labels:
    app: frontend
spec:
  replicas: 2
  selector:
    matchLabels:
      app: frontend
  template:
    metadata:
      labels:
        app: frontend
    spec:
      containers:
      - name: nginx
        image: nginx:alpine
        ports:
        - containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  name: frontend
  namespace: microservices
spec:
  selector:
    app: frontend
  ports:
  - port: 80
    targetPort: 80
EOF

kubectl apply -f frontend.yaml

# Deploy backend
cat > backend.yaml << EOF
apiVersion: apps/v1
kind: Deployment
metadata:
  name: backend
  namespace: microservices
  labels:
    app: backend
spec:
  replicas: 2
  selector:
    matchLabels:
      app: backend
  template:
    metadata:
      labels:
        app: backend
    spec:
      containers:
      - name: httpd
        image: httpd:alpine
        ports:
        - containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  name: backend
  namespace: microservices
spec:
  selector:
    app: backend
  ports:
  - port: 80
    targetPort: 80
EOF

kubectl apply -f backend.yaml

# Deploy database
cat > database.yaml << EOF
apiVersion: apps/v1
kind: Deployment
metadata:
  name: database
  namespace: microservices
  labels:
    app: database
spec:
  replicas: 1
  selector:
    matchLabels:
      app: database
  template:
    metadata:
      labels:
        app: database
    spec:
      containers:
      - name: postgres
        image: postgres:13-alpine
        env:
        - name: POSTGRES_PASSWORD
          value: "password"
        ports:
        - containerPort: 5432
---
apiVersion: v1
kind: Service
metadata:
  name: database
  namespace: microservices
spec:
  selector:
    app: database
  ports:
  - port: 5432
    targetPort: 5432
EOF

kubectl apply -f database.yaml

# Verify deployment
kubectl get pods -n microservices
kubectl get services -n microservices

3. Implement Default Deny Policy

# Create default deny policy
cat > default-deny.yaml << EOF
apiVersion: projectcalico.org/v3
kind: NetworkPolicy
metadata:
  name: default-deny
  namespace: microservices
spec:
  selector: all()
  types:
  - Ingress
  - Egress
EOF

kubectl apply -f default-deny.yaml

4. Allow Traffic from Frontend to Backend

# Policy to allow traffic from frontend to backend
cat > frontend-to-backend.yaml << EOF
apiVersion: projectcalico.org/v3
kind: NetworkPolicy
metadata:
  name: frontend-to-backend
  namespace: microservices
spec:
  selector: app == 'backend'
  types:
  - Ingress
  ingress:
  - action: Allow
    source:
      selector: app == 'frontend'
    destination:
      ports:
      - 80
EOF

kubectl apply -f frontend-to-backend.yaml

# Allow frontend egress to external DNS and API
cat > frontend-egress.yaml << EOF
apiVersion: projectcalico.org/v3
kind: NetworkPolicy
metadata:
  name: frontend-egress
  namespace: microservices
spec:
  selector: app == 'frontend'
  types:
  - Egress
  egress:
  - action: Allow
    destination:
      selector: app == 'backend'
      ports:
      - 80
  # Allow DNS access
  - action: Allow
    destination:
      selector: k8s-app == 'kube-dns'
      ports:
      - 53
EOF

kubectl apply -f frontend-egress.yaml

5. Allow Traffic from Backend to Database

# Policy to allow traffic from backend to database
cat > backend-to-database.yaml << EOF
apiVersion: projectcalico.org/v3
kind: NetworkPolicy
metadata:
  name: backend-to-database
  namespace: microservices
spec:
  selector: app == 'database'
  types:
  - Ingress
  ingress:
  - action: Allow
    source:
      selector: app == 'backend'
    destination:
      ports:
      - 5432
EOF

kubectl apply -f backend-to-database.yaml

# Allow backend egress to external DNS and database
cat > backend-egress.yaml << EOF
apiVersion: projectcalico.org/v3
kind: NetworkPolicy
metadata:
  name: backend-egress
  namespace: microservices
spec:
  selector: app == 'backend'
  types:
  - Egress
  egress:
  - action: Allow
    destination:
      selector: app == 'database'
      ports:
      - 5432
  # Allow DNS access
  - action: Allow
    destination:
      selector: k8s-app == 'kube-dns'
      ports:
      - 53
EOF

kubectl apply -f backend-egress.yaml

6. Test Policies

# Get frontend pod name
FRONTEND_POD=$(kubectl get pods -n microservices -l app=frontend -o jsonpath='{.items[0].metadata.name}')

# Get backend pod name
BACKEND_POD=$(kubectl get pods -n microservices -l app=backend -o jsonpath='{.items[0].metadata.name}')

# Get database pod name
DATABASE_POD=$(kubectl get pods -n microservices -l app=database -o jsonpath='{.items[0].metadata.name}')

# Test connection from frontend to backend (should succeed)
kubectl exec -it $FRONTEND_POD -n microservices -- wget -O- --timeout=2 http://backend

# Test connection from frontend to database (should fail)
kubectl exec -it $FRONTEND_POD -n microservices -- nc -zv database 5432

# Test connection from backend to database (should succeed)
kubectl exec -it $BACKEND_POD -n microservices -- nc -zv database 5432

# Test connection from backend to external site (should fail)
kubectl exec -it $BACKEND_POD -n microservices -- wget -O- --timeout=2 https://www.example.com

7. Network Policy Visualization (Optional)

# Install Calico network policy visualization tool
kubectl apply -f https://raw.githubusercontent.com/tigera/ccol/master/manifests/tigera-policies-viewer/tigera-policies-viewer.yaml

# Set up port forwarding
kubectl port-forward -n tigera-policies-viewer svc/tigera-policies-viewer 8080:8080

# Access http://localhost:8080 in browser to visualize policies

Through this exercise, you learned how to use Calico to restrict network traffic between microservices in an EKS cluster. By implementing default deny policies and explicitly allowing only necessary traffic, you applied the principle of least privilege. These network policies help enhance security by restricting communication between services within the cluster and reducing the potential attack surface.

### Exercise 2: Configuring IRSA and S3 Access in an EKS Cluster

Scenario: You are a DevOps engineer at your company, and you have applications running in an EKS cluster that need to securely access an S3 bucket. Following security best practices, instead of sharing the node IAM role, you want to use IRSA (IAM Roles for Service Accounts) to grant only the necessary permissions to specific pods.

Requirements:

1. Associate OIDC provider with the EKS cluster
2. Create IAM role with S3 access permissions
3. Create Kubernetes service account and associate IAM role
4. Deploy pod using the service account
5. Test S3 access

Solution:

Show Solution

1. Associate OIDC Provider with the EKS Cluster

# Set cluster name
CLUSTER_NAME=my-cluster
REGION=us-west-2

# Get OIDC provider URL
OIDC_PROVIDER=$(aws eks describe-cluster --name $CLUSTER_NAME --region $REGION --query "cluster.identity.oidc.issuer" --output text | sed -e "s/^https:\/\///")

# Check if OIDC provider already exists
aws iam list-open-id-connect-providers | grep $OIDC_PROVIDER

# Create OIDC provider if it doesn't exist
if [ $? -ne 0 ]; then
  echo "Creating OIDC provider..."
  eksctl utils associate-iam-oidc-provider --cluster $CLUSTER_NAME --region $REGION --approve
else
  echo "OIDC provider already exists."
fi

2. Create IAM Role with S3 Access Permissions

# Get account ID
ACCOUNT_ID=$(aws sts get-caller-identity --query Account --output text)

# Set namespace and service account name
NAMESPACE=default
SERVICE_ACCOUNT_NAME=s3-access-sa

# Create trust policy
cat > trust-policy.json << EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Federated": "arn:aws:iam::${ACCOUNT_ID}:oidc-provider/${OIDC_PROVIDER}"
      },
      "Action": "sts:AssumeRoleWithWebIdentity",
      "Condition": {
        "StringEquals": {
          "${OIDC_PROVIDER}:sub": "system:serviceaccount:${NAMESPACE}:${SERVICE_ACCOUNT_NAME}"
        }
      }
    }
  ]
}
EOF

# Create IAM role
ROLE_NAME=eks-s3-access-role
aws iam create-role --role-name $ROLE_NAME --assume-role-policy-document file://trust-policy.json

# Attach S3 read-only policy
aws iam attach-role-policy --role-name $ROLE_NAME --policy-arn arn:aws:iam::aws:policy/AmazonS3ReadOnlyAccess

# Get role ARN
ROLE_ARN=$(aws iam get-role --role-name $ROLE_NAME --query Role.Arn --output text)
echo "Role ARN: $ROLE_ARN"

3. Create Kubernetes Service Account and Associate IAM Role

# Create service account
cat > service-account.yaml << EOF
apiVersion: v1
kind: ServiceAccount
metadata:
  name: ${SERVICE_ACCOUNT_NAME}
  namespace: ${NAMESPACE}
  annotations:
    eks.amazonaws.com/role-arn: ${ROLE_ARN}
EOF

kubectl apply -f service-account.yaml

# Verify service account
kubectl get serviceaccount $SERVICE_ACCOUNT_NAME -o yaml

4. Deploy Pod Using the Service Account

# Deploy test pod
cat > s3-test-pod.yaml << EOF
apiVersion: v1
kind: Pod
metadata:
  name: s3-test-pod
  namespace: ${NAMESPACE}
spec:
  serviceAccountName: ${SERVICE_ACCOUNT_NAME}
  containers:
  - name: aws-cli
    image: amazon/aws-cli:latest
    command:
    - sleep
    - "3600"
  restartPolicy: Never
EOF

kubectl apply -f s3-test-pod.yaml

# Check pod status
kubectl get pod s3-test-pod
kubectl describe pod s3-test-pod

5. Test S3 Access

# Test listing S3 buckets
kubectl exec -it s3-test-pod -- aws s3 ls

# Test listing objects in a specific S3 bucket (change bucket name as needed)
kubectl exec -it s3-test-pod -- aws s3 ls s3://my-bucket/

# Verify AWS credentials
kubectl exec -it s3-test-pod -- aws sts get-caller-identity

# Check environment variables
kubectl exec -it s3-test-pod -- env | grep AWS

6. Deploy a Regular Pod for Comparison

# Deploy a pod using a regular service account
cat > regular-pod.yaml << EOF
apiVersion: v1
kind: Pod
metadata:
  name: regular-pod
  namespace: ${NAMESPACE}
spec:
  containers:
  - name: aws-cli
    image: amazon/aws-cli:latest
    command:
    - sleep
    - "3600"
  restartPolicy: Never
EOF

kubectl apply -f regular-pod.yaml

# Test S3 access from regular pod (should fail if node IAM role doesn't have S3 access permissions)
kubectl exec -it regular-pod -- aws s3 ls

7. Cleanup

# Delete pods
kubectl delete pod s3-test-pod regular-pod

# Delete service account
kubectl delete serviceaccount $SERVICE_ACCOUNT_NAME

# Clean up IAM role (optional)
aws iam detach-role-policy --role-name $ROLE_NAME --policy-arn arn:aws:iam::aws:policy/AmazonS3ReadOnlyAccess
aws iam delete-role --role-name $ROLE_NAME

How IRSA Works:

1. OIDC Provider Connection:
   • The EKS cluster is configured as an OIDC provider.
   • This allows Kubernetes service account tokens to become a trusted authentication mechanism in AWS IAM.
2. IAM Role Trust Policy:
   • The IAM role's trust policy restricts which Kubernetes service accounts can assume the role.
   • Conditions are used to limit it to specific service accounts in specific namespaces.
3. Service Account Annotation:
   • The eks.amazonaws.com/role-arn annotation specifies which IAM role the service account should assume.
   • This annotation is processed by the EKS Pod Identity Webhook.
4. Environment Variable Injection:
   • The EKS Pod Identity Webhook automatically injects the following environment variables into pods:
     • AWS_ROLE_ARN
     • AWS_WEB_IDENTITY_TOKEN_FILE
     • AWS_REGION
   • AWS SDKs use these environment variables to obtain credentials.
5. Principle of Least Privilege:
   • Grant only the minimum permissions required for the application.
   • In this example, we only granted S3 read-only access permissions.

Through this lab, you learned how to configure IRSA to grant fine-grained permissions to AWS services only to specific pods running in an EKS cluster. This approach is more secure than sharing node IAM roles and follows the principle of least privilege.

Advanced Topics

The following questions are about advanced topics related to Amazon EKS cluster creation. This section tests your understanding of advanced concepts and best practices for EKS cluster creation.

1. Which of the following is NOT a change that occurs when enabling Prefix Delegation in an Amazon EKS cluster?
   • A) Each ENI is assigned a /28 CIDR block (16 IPs)
   • B) Increased maximum pods per node
   • C) Reduced pod startup time
   • D) Improved IP address usage efficiency

Show Answer

Answer: C) Reduced pod startup time

Explanation: The change that does NOT occur when enabling Prefix Delegation in an Amazon EKS cluster is "Reduced pod startup time". In reality, enabling prefix delegation does not reduce pod startup time; it may actually increase it slightly. The main benefits of prefix delegation are increased maximum pods per node and improved IP address usage efficiency.

Actual Changes When Enabling Prefix Delegation:

1. Each ENI is assigned a /28 CIDR block (16 IPs):
   • By default, VPC CNI assigns secondary IP addresses from the ENI for each pod.
   • When prefix delegation is enabled, each ENI is assigned /28 CIDR blocks (16 IPs) instead of individual IP addresses.
   • This significantly increases the number of IP addresses each ENI can support.
2. Increased maximum pods per node:
   • Using prefix delegation significantly increases the maximum pods per node.
   • For example, for an m5.large instance:
     • Default configuration: maximum 29 pods
     • With prefix delegation enabled: maximum 110+ pods
3. Improved IP address usage efficiency:
   • Optimizes IP address usage in large clusters.
   • Useful in environments with limited VPC CIDR ranges.
   • Allows running more pods within the same IP address space.

Impact of Prefix Delegation on Pod Startup Time:

Prefix delegation does not reduce pod startup time; it may actually increase it slightly for the following reasons:

1. Additional setup overhead:
   • Additional overhead may occur for CIDR block allocation and management.
   • Time may be needed for routing table updates.
2. IP address allocation complexity:
   • CIDR block allocation and management can be more complex than individual IP address allocation.
   • This may cause slight delays during pod startup.
3. Initial setup time:
   • Initial setup time for allocating CIDR blocks to new ENIs may be longer.
   • However, once set up, multiple pods can be started quickly on that ENI.

How to Enable Prefix Delegation:

# Enable prefix delegation
kubectl set env daemonset aws-node -n kube-system ENABLE_PREFIX_DELEGATION=true

# Check prefix delegation status
kubectl describe daemonset aws-node -n kube-system | grep ENABLE_PREFIX_DELEGATION

Prefix Delegation Limitations:

1. EC2 Instance Support:
   • Only Nitro-based instances support prefix delegation.
   • Cannot be used on previous generation instances.
2. VPC CNI Version Requirements:
   • Requires VPC CNI version 1.9.0 or higher.
   • This feature is not available in earlier versions.
3. Subnet Size Requirements:
   • Requires subnets with sufficient IP address space.
   • IP addresses can be exhausted quickly in small subnets.
4. Transition Considerations:
   • When enabled on existing clusters, only new pods use prefix delegation.
   • Existing pods must be restarted to apply to all pods.

Prefix delegation is a powerful feature that increases the maximum pods per node and improves IP address usage efficiency, but it does not reduce pod startup time. Therefore, "Reduced pod startup time" is NOT a change that occurs when enabling prefix delegation.

2. Which of the following is NOT a main benefit of mixing instance types in an Amazon EKS cluster node group? - A) Cost optimization - B) Improved availability - C) Selecting instance types suited to workload characteristics - D) Simplified cluster management

Show Answer

Answer: D) Simplified cluster management

Explanation: The option that is NOT a main benefit of mixing instance types in an Amazon EKS cluster node group is "Simplified cluster management". In reality, mixing various instance types can make cluster management more complex. The main benefits of mixing instance types are cost optimization, improved availability, and selecting instance types suited to workload characteristics.

Actual Benefits of Mixing Instance Types:

1. Cost optimization:

   • Mix spot instances and on-demand instances

   • Leverage price differences across various instance families

   • Select optimal price-to-performance ratio for workload requirements

   • Example:

     apiVersion: eksctl.io/v1alpha5
     kind: ClusterConfig
     metadata:
       name: my-cluster
       region: us-west-2
     managedNodeGroups:
       - name: mixed-spot-instances
         instanceTypes: ["m5.large", "m5a.large", "m5d.large", "m5ad.large", "m5n.large"]
         spot: true
         minSize: 2
         maxSize: 10

2. Improved availability:

   • Use alternative instance types when specific instance types have insufficient capacity

   • Can substitute with other types when spot instances are interrupted

   • Risk distribution through diversification across multiple instance families

   • Example:

     apiVersion: eksctl.io/v1alpha5
     kind: ClusterConfig
     metadata:
       name: my-cluster
       region: us-west-2
     managedNodeGroups:
       - name: mixed-instance-types
         instanceTypes: ["c5.large", "c5a.large", "c5d.large", "c5n.large"]
         minSize: 3
         maxSize: 10
         spotAllocationStrategy: capacity-optimized

3. Selecting instance types suited to workload characteristics:

   • Provide instance types for various workload requirements

   • C series for compute-intensive workloads

   • R series for memory-intensive workloads

   • M series for balanced workloads

   • G or P series for GPU workloads

   • Example:

     # Compute optimized node group
     apiVersion: eksctl.io/v1alpha5
     kind: ClusterConfig
     metadata:
       name: my-cluster
       region: us-west-2
     managedNodeGroups:
       - name: compute-optimized
         instanceTypes: ["c5.2xlarge"]
         minSize: 2
         maxSize: 10
         labels:
           workload-type: compute
         taints:
           - key: workload-type
             value: compute
             effect: NoSchedule
     
       # Memory optimized node group
       - name: memory-optimized
         instanceTypes: ["r5.2xlarge"]
         minSize: 2
         maxSize: 10
         labels:
           workload-type: memory
         taints:
           - key: workload-type
             value: memory
             effect: NoSchedule

Disadvantages of Mixing Instance Types:

1. Increased cluster management complexity:
   • Requires monitoring and managing various instance types
   • Troubleshooting complexity due to different performance characteristics
   • Need to adjust resource requests and limits for various instance types
2. Reduced workload predictability:
   • Performance characteristics may vary by instance type
   • Potential workload performance variations, especially when using spot instances
3. Resource allocation complexity:
   • Difficulty setting pod resource requests and limits for various instance types
   • Complex scheduling rules needed using node selectors and taints
4. Testing and validation burden:
   • Need to test applications on various instance types
   • Increased likelihood of discovering performance and compatibility issues

Strategies for Mixing Instance Types:

1. Separate node groups by workload:

   # Node group for web servers
   - name: web-servers
     instanceTypes: ["c5.large", "c5a.large"]
     labels:
       role: web
   
   # Node group for databases
   - name: databases
     instanceTypes: ["r5.xlarge", "r5a.xlarge"]
     labels:
       role: database

2. Cost optimization strategy:

   # Base on-demand node group
   - name: on-demand-base
     instanceTypes: ["m5.large"]
     minSize: 2
     maxSize: 5
     spot: false
   
   # Spot node group for scaling
   - name: spot-scaling
     instanceTypes: ["m5.large", "m5a.large", "m5d.large", "m5n.large"]
     minSize: 0
     maxSize: 20
     spot: true

3. Availability optimization strategy:

   # Diversification across multiple instance families
   - name: high-availability
     instanceTypes: ["m5.large", "m5a.large", "c5.large", "c5a.large", "r5.large", "r5a.large"]
     minSize: 3
     maxSize: 10
     spotAllocationStrategy: capacity-optimized

Mixing instance types provides benefits such as cost optimization, improved availability, and selecting instance types suited to workload characteristics, but cluster management is not simplified and can actually become more complex. Therefore, "Simplified cluster management" is NOT a main benefit of mixing instance types.

3. What is the safest node group update strategy in an Amazon EKS cluster?
   • A) In-place update
   • B) Blue/Green deployment
   • C) Canary deployment
   • D) Rolling update

Show Answer

Answer: B) Blue/Green deployment

Explanation: The safest node group update strategy in an Amazon EKS cluster is Blue/Green deployment. Blue/Green deployment creates a new node group (green), migrates workloads, and then removes the existing node group (blue). This approach is the safest because you can immediately roll back to the previous environment if issues occur during the update.

Comparison of Node Group Update Strategies:

1. Blue/Green Deployment:

   • How it works: Create new node group → Migrate workloads → Remove existing node group

   • Advantages:

     • Immediate rollback possible
     • Minimal workload disruption during updates
     • Can compare environments before and after update
     • Can switch after testing

   • Disadvantages:

     • Temporarily requires double the resources
     • Implementation complexity
     • Increased cost

   • Implementation example:

     # 1. Create new node group
     eksctl create nodegroup \
       --cluster my-cluster \
       --name my-nodegroup-v2 \
       --node-type m5.large \
       --nodes 3 \
       --node-ami-family AmazonLinux2 \
       --node-labels "version=v2,color=green"
     
     # 2. Migrate workloads (update node selector)
     kubectl patch deployment my-app -p '{"spec":{"template":{"spec":{"nodeSelector":{"color":"green"}}}}}'
     
     # 3. Verify all workloads have moved to new nodes
     kubectl get pods -o wide
     
     # 4. Remove existing node group
     eksctl delete nodegroup --cluster my-cluster --name my-nodegroup-v1

2. Rolling Update:

   • How it works: Replace nodes one at a time (cordon → drain → terminate → add new node)

   • Advantages:

     • No additional resources needed
     • Default strategy for EKS managed node groups
     • Simple implementation

   • Disadvantages:

     • Difficult to rollback
     • Issues during update can affect entire cluster
     • Update time can be lengthy

   • Implementation example:

     # Update managed node group
     aws eks update-nodegroup-version \
       --cluster-name my-cluster \
       --nodegroup-name my-nodegroup
     
     # Modify update configuration
     aws eks update-nodegroup-config \
       --cluster-name my-cluster \
       --nodegroup-name my-nodegroup \
       --update-config '{"maxUnavailable": 1}'

3. Canary Deployment:

   • How it works: Create small new node group → Migrate some workloads → Validate → Complete migration

   • Advantages:

     • Minimizes risk
     • Allows gradual validation
     • Limits impact scope when issues occur

   • Disadvantages:

     • Implementation complexity
     • Requires additional resources
     • Time-consuming until complete migration

   • Implementation example:

     # 1. Create small canary node group
     eksctl create nodegroup \
       --cluster my-cluster \
       --name canary-nodegroup \
       --node-type m5.large \
       --nodes 1 \
       --node-labels "deployment=canary"
     
     # 2. Migrate some workloads
     kubectl patch deployment my-app -p '{"spec":{"template":{"spec":{"nodeSelector":{"deployment":"canary"}}}}}'
     
     # 3. Proceed with complete migration after validation

4. In-place Update:

   • How it works: Perform updates directly on existing nodes

   • Advantages:

     • No additional resources needed
     • Suitable for simple changes

   • Disadvantages:

     • High risk
     • Difficult rollback
     • Possible node damage if update fails
     • Not recommended in EKS

   • Implementation example:

     # SSH into node for direct update (not recommended)
     ssh ec2-user@node-ip
     sudo yum update -y

Why Blue/Green Deployment is the Safest:

1. Complete isolation:
   • New environment is completely separated from existing environment, minimizing impact
   • Existing environment is not affected even if issues occur during update
2. Immediate rollback:
   • Can immediately redirect traffic to existing environment when issues occur
   • Rollback possible without downtime
3. Validation opportunity:
   • Can thoroughly test new environment before switching to production traffic
   • Can validate under conditions identical to actual environment
4. Gradual transition:
   • Can gradually transition traffic to new environment
   • Limits impact scope when issues occur

Blue/Green Deployment Best Practices:

1. Automation:
   • Minimize human error by automating deployment process
   • Integrate with CI/CD pipeline
2. Enhanced monitoring:
   • Monitor performance and error metrics of new environment
   • Compare and analyze with existing environment
3. Gradual transition:
   • Gradually transition traffic to new environment
   • Roll back immediately when issues occur
4. Resource optimization:
   • Promptly clean up unnecessary resources after transition completes
   • Cost optimization

Blue/Green deployment has disadvantages of additional resources and implementation complexity, but it is the most excellent approach in terms of safety. It is especially recommended for important production environments or cases where business impact is significant if updates fail.

4. Which of the following is NOT a method for optimizing Auto Scaling of node groups in an Amazon EKS cluster? - A) Adjusting Cluster Autoscaler scan interval - B) Configuring pod priority and preemption - C) Tagging Auto Scaling groups per node group - D) Using the same instance type for all nodes

Show Answer

Answer: D) Using the same instance type for all nodes

Explanation: The option that is NOT a method for optimizing Auto Scaling of node groups in an Amazon EKS cluster is "Using the same instance type for all nodes." In practice, mixing various instance types is a more effective Auto Scaling strategy in terms of cost optimization and availability. Especially when using Spot Instances, specifying multiple instance types is recommended to increase capacity availability and reduce interruption risk.

Actual Methods for Optimizing Node Group Auto Scaling:

1. Adjusting Cluster Autoscaler Scan Interval:

   • Cluster Autoscaler periodically scans the cluster to determine if scaling up or down is needed.

   • You can balance response time and resource usage by adjusting the scan interval.

   • Example:

     # Cluster Autoscaler deployment configuration
     apiVersion: apps/v1
     kind: Deployment
     metadata:
       name: cluster-autoscaler
       namespace: kube-system
     spec:
       template:
         spec:
           containers:
           - name: cluster-autoscaler
             image: k8s.gcr.io/autoscaling/cluster-autoscaler:v1.23.0
             command:
             - ./cluster-autoscaler
             - --v=4
             - --stderrthreshold=info
             - --cloud-provider=aws
             - --scan-interval=30s  # Adjust scan interval (default: 10 seconds)
             - --max-node-provision-time=15m
             - --node-group-auto-discovery=asg:tag=k8s.io/cluster-autoscaler/enabled,k8s.io/cluster-autoscaler/my-cluster

2. Configuring Pod Priority and Preemption:

   • Use pod priority and preemption (PriorityClass) to ensure important workloads are scheduled first.

   • When resources are insufficient, lower priority pods are preempted to make room for higher priority pods.

   • Example:

     # Priority class definition
     apiVersion: scheduling.k8s.io/v1
     kind: PriorityClass
     metadata:
       name: high-priority
     value: 1000000
     globalDefault: false
     description: "High priority pods"
     ---
     # High priority pod
     apiVersion: v1
     kind: Pod
     metadata:
       name: high-priority-pod
     spec:
       priorityClassName: high-priority
       containers:
       - name: nginx
         image: nginx

3. Tagging Auto Scaling Groups per Node Group:

   • Tag Auto Scaling groups so that Cluster Autoscaler can identify and manage specific node groups.

   • Different Auto Scaling behaviors can be configured for each node group.

   • Example:

     # Tag Auto Scaling group
     aws autoscaling create-or-update-tags \
       --tags ResourceId=my-asg,ResourceType=auto-scaling-group,Key=k8s.io/cluster-autoscaler/enabled,Value=true,PropagateAtLaunch=true \
              ResourceId=my-asg,ResourceType=auto-scaling-group,Key=k8s.io/cluster-autoscaler/my-cluster,Value=owned,PropagateAtLaunch=true
     
     # Configure Auto Scaling settings per node group
     aws autoscaling update-auto-scaling-group \
       --auto-scaling-group-name my-asg \
       --min-size 2 \
       --max-size 10 \
       --desired-capacity 2

Benefits of Mixing Various Instance Types:

1. Cost Optimization:

   • Leverage price differences among various instance types

   • Improved availability when using Spot Instances

   • Example:

     # Node group using various instance types
     apiVersion: eksctl.io/v1alpha5
     kind: ClusterConfig
     metadata:
       name: my-cluster
       region: us-west-2
     managedNodeGroups:
       - name: mixed-instances
         instanceTypes: ["m5.large", "m5a.large", "m5d.large", "m5n.large"]
         minSize: 2
         maxSize: 10
         spot: true

2. Improved Availability:

   • Use alternative instance types when specific instance types have insufficient capacity

   • Can switch to other types when Spot Instances are interrupted

   • Example:

     # Capacity-optimized Spot allocation strategy
     apiVersion: eksctl.io/v1alpha5
     kind: ClusterConfig
     metadata:
       name: my-cluster
       region: us-west-2
     managedNodeGroups:
       - name: spot-nodes
         instanceTypes: ["c5.large", "c5a.large", "c5d.large", "c5n.large"]
         minSize: 2
         maxSize: 10
         spot: true
         spotAllocationStrategy: capacity-optimized

3. Selecting Instance Types Based on Workload Characteristics:

   • Provide instance types suitable for various workload requirements

   • Control workload placement using node selectors and taints

   • Example:

     # Node groups by workload characteristics
     apiVersion: eksctl.io/v1alpha5
     kind: ClusterConfig
     metadata:
       name: my-cluster
       region: us-west-2
     managedNodeGroups:
       - name: general-purpose
         instanceTypes: ["m5.large"]
         minSize: 2
         maxSize: 10
     
       - name: compute-intensive
         instanceTypes: ["c5.large"]
         minSize: 0
         maxSize: 10
         labels:
           workload-type: compute

Additional Strategies for Auto Scaling Optimization:

1. Overprovisioning:

   • Maintain a certain amount of spare resources to prepare for sudden scaling requests

   • Example:

     # Overprovisioning pod
     apiVersion: apps/v1
     kind: Deployment
     metadata:
       name: overprovisioning
       namespace: kube-system
     spec:
       replicas: 1
       selector:
         matchLabels:
           app: overprovisioning
       template:
         metadata:
           labels:
             app: overprovisioning
         spec:
           priorityClassName: overprovisioning
           containers:
           - name: reserve-resources
             image: k8s.gcr.io/pause:3.2
             resources:
               requests:
                 cpu: 1000m
                 memory: 1000Mi

2. Optimizing Scaling Policies:

   • Use target tracking scaling policies

   • Use step scaling policies

   • Enable predictive scaling

   • Example:

     # Configure target tracking scaling policy
     aws autoscaling put-scaling-policy \
       --auto-scaling-group-name my-asg \
       --policy-name cpu70-target-tracking-scaling-policy \
       --policy-type TargetTrackingScaling \
       --target-tracking-configuration '{"PredefinedMetricSpecification":{"PredefinedMetricType":"ASGAverageCPUUtilization"},"TargetValue":70.0,"DisableScaleIn":false}'

3. Using Karpenter:

   • Consider using Karpenter instead of Cluster Autoscaler

   • Faster node provisioning and more flexible instance type selection

   • Example:

     # Karpenter Provisioner
     apiVersion: karpenter.sh/v1alpha5
     kind: NodePool
     metadata:
       name: default
     spec:
       template:
         spec:
           requirements:
             - key: karpenter.sh/capacity-type
               operator: In
               values: ["spot", "on-demand"]
             - key: node.kubernetes.io/instance-type
               operator: In
               values: ["m5.large", "m5a.large", "m5d.large", "m5n.large"]
       limits:
         resources:
           cpu: 1000
           memory: 1000Gi

Using the same instance type for all nodes is not an Auto Scaling optimization strategy; rather, mixing various instance types is more effective in terms of cost optimization and availability. Therefore, "Using the same instance type for all nodes" is NOT a method for optimizing Auto Scaling of node groups.

5. Which of the following is NOT a security best practice to consider when creating node groups in an Amazon EKS cluster?
   • A) Requiring IMDSv2
   • B) Applying least privilege IAM policies
   • C) Assigning public IP addresses to all nodes
   • D) Restricting security group rules

Show Answer

Answer: C) Assigning public IP addresses to all nodes

Explanation: The option that is NOT a security best practice to consider when creating node groups in an Amazon EKS cluster is "Assigning public IP addresses to all nodes." The security best practice is actually the opposite: placing nodes in private subnets and not assigning public IP addresses. This reduces the attack surface by preventing nodes from being directly accessible from the internet.

Actual Security Best Practices for EKS Node Group Creation:

1. Requiring IMDSv2:

   • Require Instance Metadata Service version 2 (IMDSv2) to protect against SSRF (Server-Side Request Forgery) attacks

   • IMDSv2 uses session-based requests for enhanced security

   • Example:

     # eksctl configuration file
     apiVersion: eksctl.io/v1alpha5
     kind: ClusterConfig
     metadata:
       name: my-cluster
       region: us-west-2
     managedNodeGroups:
       - name: secure-nodes
         instanceType: m5.large
         minSize: 2
         maxSize: 5
         disableIMDSv1: true  # Disable IMDSv1
         metadataOptions:
           httpTokens: required  # Require IMDSv2
           httpPutResponseHopLimit: 1

2. Applying Least Privilege IAM Policies:

   • Grant only the minimum necessary permissions to node IAM roles

   • Create granular policies when additional permissions beyond default managed policies are needed

   • Example:

     # Least privilege policy for node IAM role
     apiVersion: eksctl.io/v1alpha5
     kind: ClusterConfig
     metadata:
       name: my-cluster
       region: us-west-2
     managedNodeGroups:
       - name: secure-nodes
         instanceType: m5.large
         minSize: 2
         maxSize: 5
         iam:
           attachPolicyARNs:
             - arn:aws:iam::aws:policy/AmazonEKSWorkerNodePolicy
             - arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly
             - arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy
           withAddonPolicies:
             imageBuilder: false
             autoScaler: false
             externalDNS: false
             certManager: false
             appMesh: false
             ebs: true
             fsx: false
             efs: false
             albIngress: false
             xRay: false
             cloudWatch: true

3. Restricting Security Group Rules:

   • Restrict inbound and outbound rules for node security groups

   • Open only the minimum necessary ports

   • Example:

     # Create security group
     aws ec2 create-security-group \
       --group-name eks-node-sg \
       --description "Security group for EKS nodes" \
       --vpc-id vpc-12345
     
     # Add only rules necessary for cluster communication
     aws ec2 authorize-security-group-ingress \
       --group-id sg-12345 \
       --protocol tcp \
       --port 443 \
       --source-group sg-cluster
     
     aws ec2 authorize-security-group-ingress \
       --group-id sg-12345 \
       --protocol tcp \
       --port 10250 \
       --source-group sg-cluster

Reasons for Not Assigning Public IP Addresses to Nodes:

1. Reduced Attack Surface:
   • Without public IPs, nodes cannot be directly accessed from the internet
   • Management tasks such as SSH access are performed through bastion hosts or AWS Systems Manager
2. Improved Security Architecture:
   • Place nodes in private subnets
   • Allow only outbound communication through NAT Gateway
   • Allow inbound traffic only through load balancers
3. Regulatory Compliance:
   • Many security standards and regulations require minimizing direct internet exposure
   • Helps with compliance for PCI DSS, HIPAA, and other regulations

Example of Placing Nodes in Private Subnets:

# eksctl configuration file
apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig
metadata:
  name: my-cluster
  region: us-west-2
vpc:
  subnets:
    private:
      us-west-2a: { id: subnet-private-a }
      us-west-2b: { id: subnet-private-b }
    public:
      us-west-2a: { id: subnet-public-a }
      us-west-2b: { id: subnet-public-b }
managedNodeGroups:
  - name: secure-nodes
    instanceType: m5.large
    minSize: 2
    maxSize: 5
    privateNetworking: true  # Place nodes in private subnets

Additional EKS Security Best Practices:

1. Enabling Encryption:

   • EBS volume encryption

   • Secrets encryption

   • Example:

     apiVersion: eksctl.io/v1alpha5
     kind: ClusterConfig
     metadata:
       name: my-cluster
       region: us-west-2
     secretsEncryption:
       keyARN: arn:aws:kms:us-west-2:123456789012:key/key-id
     nodeGroups:
       - name: secure-nodes
         volumeEncrypted: true
         volumeKmsKeyID: arn:aws:kms:us-west-2:123456789012:key/key-id

2. Container Security:

   • Disable privileged containers

   • Use read-only root filesystem

   • Example:

     apiVersion: v1
     kind: Pod
     metadata:
       name: secure-pod
     spec:
       containers:
       - name: secure-container
         image: nginx
         securityContext:
           privileged: false
           readOnlyRootFilesystem: true
           allowPrivilegeEscalation: false

3. Implementing Network Policies:

   • Restrict pod-to-pod communication

   • Apply default deny policies

   • Example:

     apiVersion: networking.k8s.io/v1
     kind: NetworkPolicy
     metadata:
       name: default-deny
       namespace: default
     spec:
       podSelector: {}
       policyTypes:
       - Ingress
       - Egress

4. Logging and Monitoring:

   • Enable CloudWatch Logs

   • Enable GuardDuty EKS Protection

   • Example:

     # Enable CloudWatch Logs
     aws eks update-cluster-config \
       --name my-cluster \
       --logging '{"clusterLogging":[{"types":["api","audit","authenticator","controllerManager","scheduler"],"enabled":true}]}'

Assigning public IP addresses to all nodes is not a security best practice; rather, it increases security risk. Therefore, "Assigning public IP addresses to all nodes" is NOT a security best practice to consider when creating node groups in an Amazon EKS cluster.