Kubernetes & Amazon EKS Training

English

한국어
中文
日本語
Español

English

한국어
中文
日本語
Español

Appearance

EKS Cluster Creation Quiz - Part 5 ​

This quiz tests your understanding of advanced configuration, optimization, and operational best practices for Amazon EKS clusters. It focuses on cost optimization, high availability design, security enhancement, and operational automation.

Multiple Choice Questions ​

1. What is the most effective way to optimize costs in an Amazon EKS cluster? ​

A. Run all worker nodes as On-Demand instances B. Use a mix of Spot instances and On-Demand instances C. Run all worker nodes as Reserved instances D. Run all worker nodes on Fargate

Answer and Explanation

Answer: B. Use a mix of Spot instances and On-Demand instances

Explanation: The most effective way to optimize costs in an EKS cluster is to use a mix of Spot instances and On-Demand instances:

  • Spot instances: Up to 90% cheaper than On-Demand pricing, but can be interrupted when AWS reclaims capacity. Suitable for fault-tolerant stateless workloads.
  • On-Demand instances: Higher price but stable, suitable for critical stateful workloads or applications sensitive to interruption.

Through this mixed approach:

  1. Run critical workloads on On-Demand instances
  2. Run fault-tolerant workloads on Spot instances
  3. Control workload placement using node affinity, tolerations, and taints

Additional cost optimization strategies include auto-scaling using Karpenter or Cluster Autoscaler, selecting appropriate instance sizes, using Graviton (ARM) instances, and utilizing Reserved Instances or Savings Plans.

2. What is the main reason for deploying nodes across multiple Availability Zones (AZs) in an EKS cluster? ​

A. Reduce network latency B. Increase data throughput C. Improve high availability and fault tolerance D. Enable data replication across AWS regions

Answer and Explanation

Answer: C. Improve high availability and fault tolerance

Explanation: The main reason for deploying nodes across multiple Availability Zones (AZs) in an EKS cluster is to improve high availability and fault tolerance:

  1. AZ Failure Response: If one AZ fails, nodes in other AZs continue operating, maintaining application availability.
  2. Infrastructure Redundancy: Distributing workloads across multiple AZs adds a protective layer against physical infrastructure failures.
  3. Automatic Recovery: Kubernetes automatically reschedules pods from failed nodes to healthy nodes, minimizing service disruption.
  4. Rolling Update Stability: Availability is maintained during updates as workloads are distributed across multiple AZs.

EKS deploys the control plane across multiple AZs by default, but deploying worker nodes across multiple AZs is also a best practice to ensure overall cluster high availability. You can specify multiple subnets (each located in different AZs) when creating node groups.

3. What is the default CNI plugin for pod networking in an EKS cluster? ​

A. Calico B. Flannel C. Amazon VPC CNI D. Weave Net

Answer and Explanation

Answer: C. Amazon VPC CNI

Explanation: The default CNI (Container Network Interface) plugin for pod networking in Amazon EKS clusters is Amazon VPC CNI. The main characteristics of this plugin are:

  1. Native VPC Networking: Each pod receives a unique IP address within the VPC, directly utilizing AWS VPC networking.
  2. Security Group Integration: AWS security groups can be applied at the pod level, enabling fine-grained network security control.
  3. IP Address Management: Each node is allocated secondary IP addresses from VPC subnets to provide to pods.
  4. Performance: Network performance is improved by not using overlay networks.
  5. AWS Service Integration: Seamlessly integrates with other AWS services like AWS Load Balancer Controller, AWS App Mesh.

Amazon VPC CNI is open source and managed on GitHub. It can be replaced with other CNI plugins like Calico or Cilium as needed, but Amazon VPC CNI is the default option for EKS and officially supported by AWS.

4. What is the name of the feature that links IAM roles to Kubernetes service accounts in an EKS cluster? ​

A. IAM for Service Accounts (IRSA) B. Pod Identity Webhook C. Kubernetes IAM Authenticator D. EKS Identity Manager

Answer and Explanation

Answer: A. IAM for Service Accounts (IRSA)

Explanation: The feature that links IAM roles to Kubernetes service accounts in an EKS cluster is IAM for Service Accounts (IRSA). The main characteristics of this feature are:

  1. Fine-grained Permission Control: Control access to AWS resources at the pod level, preventing broad permission grants at the node level.
  2. OIDC-based Authentication: EKS uses an OpenID Connect (OIDC) provider to establish trust relationships between Kubernetes service accounts and IAM roles.
  3. Enhanced Security: Enables implementing the principle of least privilege by granting only the minimum required permissions per application.
  4. Implementation Method:
    • Create OIDC provider for EKS cluster
    • Create IAM role that trusts the service account
    • Create Kubernetes service account with specific annotation
    • Deploy pod using that service account

With IRSA, applications using the AWS SDK can securely access AWS services using their own IAM role instead of relying on the node's IAM role.

5. What is the Kubernetes-native tool that manages node group Auto Scaling in an EKS cluster? ​

A. Horizontal Pod Autoscaler B. Vertical Pod Autoscaler C. Cluster Autoscaler D. Node Autoscaler

Answer and Explanation

Answer: C. Cluster Autoscaler

Explanation: The Kubernetes-native tool that manages node group Auto Scaling in an EKS cluster is Cluster Autoscaler. The main characteristics of this tool are:

  1. Automatic Scaling: Automatically adds nodes when pods cannot be scheduled due to resource shortage, and removes nodes when they are underutilized.
  2. AWS Auto Scaling Group Integration: Works integrated with AWS Auto Scaling Groups in EKS.
  3. How It Works:
    • Scale Out: Adds nodes when pods are in Pending state due to resource constraints
    • Scale In: Removes nodes when utilization is low and pods can be moved to other nodes
  4. Configuration Options:
    • Set scale up/down thresholds
    • Specify node group discovery method
    • Set scale down delay
    • Respect Pod Disruption Budgets (PDB)

Horizontal Pod Autoscaler (HPA) automatically adjusts pod count, and Vertical Pod Autoscaler (VPA) automatically adjusts pod resource requests, but adjusting node count is the role of Cluster Autoscaler.

Note that AWS also offers Karpenter as a new node provisioning tool, which provides faster and more flexible node provisioning capabilities.

Short Answer Questions ​

6. What configuration is needed to enable Kubernetes control plane logs and send them to CloudWatch Logs in an EKS cluster? ​

Answer and Explanation

To send Kubernetes control plane logs to CloudWatch Logs in an EKS cluster, you need to enable specific log types either during cluster creation or on an existing cluster.

Required Configuration:

  1. Enable Log Types: Enable one or more of the following log types:
    • api: Kubernetes API server logs
    • audit: Kubernetes audit logs
    • authenticator: AWS IAM authenticator logs
    • controllerManager: Controller manager logs
    • scheduler: Scheduler logs
  2. Enable via AWS Management Console:
    • Select cluster in EKS console
    • Select "Logging" tab
    • Enable desired log types
  3. Enable via AWS CLI:
bash
aws eks update-cluster-config \
    --region region-code \
    --name cluster-name \
    --logging '{"clusterLogging":[{"types":["api","audit","authenticator","controllerManager","scheduler"],"enabled":true}]}'
  1. Enable via eksctl:
bash
eksctl utils update-cluster-logging \
    --region=region-code \
    --cluster=cluster-name \
    --enable-types=api,audit,authenticator,controllerManager,scheduler

Enabled logs are automatically sent to the CloudWatch Logs log group /aws/eks/cluster-name/cluster. Each log type is stored as a separate log stream.

Notes:

  • Enabling logs incurs additional costs (CloudWatch Logs pricing applies).
  • Audit logs in particular can generate large amounts of data, so be mindful of cost management.
  • Set log retention periods to manage costs.

7. How do you send kubelet logs from worker nodes to CloudWatch Logs in an EKS cluster? ​

Answer and Explanation

To send kubelet logs from worker nodes to CloudWatch Logs in an EKS cluster, you need to install and configure the CloudWatch agent. Unlike control plane logs, worker node logs are not automatically sent to CloudWatch.

Implementation Steps:

  1. Install CloudWatch Agent: Deploy CloudWatch agent as a DaemonSet in Kubernetes.

  2. Configure Fluentd or Fluent Bit: Configure log collector to send kubelet logs to CloudWatch Logs.

  3. Recommended Method: Use Amazon EKS Add-on:

    bash
    # Create namespace for CloudWatch log collection
kubectl create namespace amazon-cloudwatch

# Create service account for AWS observability access
eksctl create iamserviceaccount \
    --name cloudwatch-agent \
    --namespace amazon-cloudwatch \
    --cluster my-cluster \
    --attach-policy-arn arn:aws:iam::aws:policy/CloudWatchAgentServerPolicy \
    --approve \
    --override-existing-serviceaccounts

# Create service account for Fluent Bit
eksctl create iamserviceaccount \
    --name fluent-bit \
    --namespace amazon-cloudwatch \
    --cluster my-cluster \
    --attach-policy-arn arn:aws:iam::aws:policy/CloudWatchAgentServerPolicy \
    --approve \
    --override-existing-serviceaccounts

# Install CloudWatch agent and Fluent Bit
kubectl apply -f https://raw.githubusercontent.com/aws-samples/amazon-cloudwatch-container-insights/latest/k8s-deployment-manifest-templates/deployment-mode/daemonset/container-insights-monitoring/quickstart/cwagent-fluent-bit-quickstart.yaml

  4. Customize Configuration: Modify ConfigMap to collect specific log paths and formats.

    yaml
    apiVersion: v1
kind: ConfigMap
metadata:
  name: fluent-bit-config
  namespace: amazon-cloudwatch
data:
  fluent-bit.conf: |
    [INPUT]
        Name tail
        Path /var/log/kubelet.log
        Tag kubelet
    [OUTPUT]
        Name cloudwatch
        Match kubelet
        region region-name
        log_group_name /aws/eks/my-cluster/nodes
        log_stream_prefix kubelet-
        auto_create_group true

  5. Verify Logs: Check log group /aws/eks/my-cluster/nodes in CloudWatch Logs console.

Key Log Collection Targets:

  • /var/log/kubelet.log: kubelet logs
  • /var/log/kube-proxy.log: kube-proxy logs
  • /var/log/aws-routed-eni/ipamd.log: VPC CNI logs
  • /var/log/containers/*.log: container logs

Alternative Methods:

  • Use AWS Distro for OpenTelemetry (ADOT)
  • Use Amazon OpenSearch with Fluent Bit combination
  • Build custom logging solution (e.g., ELK stack)

Best Practices:

  • Manage costs with log retention period settings
  • Selectively collect only necessary logs
  • Collect only important information through log filtering
  • Track costs by tagging log groups

8. Why is Pod Security Policy (PSP) no longer used in EKS clusters, and what are the alternatives? ​

Answer and Explanation

Pod Security Policy (PSP) has been deprecated since Kubernetes version 1.21 and was completely removed in Kubernetes 1.25. Accordingly, EKS no longer supports PSP.

Reasons for Deprecation:

  1. Complexity: PSP was difficult to configure and understand.
  2. Debugging Difficulty: Did not provide clear error messages when PSP violations occurred, making troubleshooting difficult.
  3. Limited Flexibility: Fine-grained control was difficult in certain scenarios.
  4. Lack of Consistency: Integration with other Kubernetes security mechanisms was not smooth.

Alternatives:

  1. Pod Security Standards (PSS) / Pod Security Admission (PSA):

    • Official alternative introduced since Kubernetes 1.22

    • Provides three security levels: Privileged, Baseline, Restricted

    • Applied through namespace labels

    • Example:

      yaml
      apiVersion: v1
kind: Namespace
metadata:
  name: my-namespace
  labels:
    pod-security.kubernetes.io/enforce: restricted
    pod-security.kubernetes.io/audit: restricted
    pod-security.kubernetes.io/warn: restricted
  2. Kyverno:

    • Policy engine with YAML-based policy definitions

    • Provides more flexible and powerful features than PSP

    • Supports validation, mutation, generation, cleanup policies

    • Example:

      yaml
      apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: restrict-privileged
spec:
  validationFailureAction: enforce
  rules:
  - name: privileged-containers
    match:
      resources:
        kinds:
        - Pod
    validate:
      message: "Privileged containers are not allowed"
      pattern:
        spec:
          containers:
            - name: "*"
              securityContext:
                privileged: false
  3. OPA Gatekeeper:

    • Policy controller based on Open Policy Agent

    • Policy definitions using Rego language

    • Uses ConstraintTemplate and Constraint concepts

    • Example:

      yaml
      apiVersion: templates.gatekeeper.sh/v1beta1
kind: ConstraintTemplate
metadata:
  name: k8spsprivilegedcontainer
spec:
  crd:
    spec:
      names:
        kind: K8sPSPPrivilegedContainer
  targets:
    - target: admission.k8s.gatekeeper.sh
      rego: |
        package k8spsprivilegedcontainer
        violation[{"msg": msg}] {
          c := input.review.object.spec.containers[_]
          c.securityContext.privileged
          msg := "Privileged containers are not allowed"
        }
  4. AWS Built-in Security Features:
    • Amazon GuardDuty for EKS Protection
    • AWS Security Hub's EKS security standards
    • Amazon Inspector for EKS

Migration Strategy:

  1. Analyze and document current PSP policies
  2. Select replacement solution (PSA, Kyverno, OPA Gatekeeper, etc.)
  3. Deploy new policies in audit mode to assess impact
  4. Gradually apply policies (transition to enforce mode)
  5. Track policy violations with monitoring and logging settings

It is important to migrate from PSP to an alternative solution before upgrading to EKS 1.25 or higher.

Hands-on Questions ​

9. Write a node group configuration that uses a mix of Spot instances and On-Demand instances for cost optimization in an EKS cluster. It should meet the following requirements: ​

  • On-Demand node group for critical workloads (2-5 nodes)
  • Spot node group for general workloads (2-10 nodes)
  • Appropriate node labels and taints
  • Example of node affinity and tolerations for workload placement
Answer and Explanation

A node group configuration using a mix of Spot instances and On-Demand instances for cost optimization in an EKS cluster is as follows:

1. On-Demand Node Group Configuration (for Critical Workloads) ​

Configuration using eksctl:

yaml
apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig
metadata:
  name: my-cluster
  region: us-west-2
nodeGroups:
  - name: critical-workloads
    instanceType: m5.xlarge
    desiredCapacity: 2
    minSize: 2
    maxSize: 5
    capacityType: ON_DEMAND
    labels:
      workload-type: critical
      node-lifecycle: on-demand
    tags:
      k8s.io/cluster-autoscaler/enabled: "true"
      k8s.io/cluster-autoscaler/my-cluster: "owned"
    iam:
      withAddonPolicies:
        autoScaler: true
    ssh:
      allow: false

Configuration using AWS CLI:

bash
aws eks create-nodegroup \
  --cluster-name my-cluster \
  --nodegroup-name critical-workloads \
  --scaling-config minSize=2,maxSize=5,desiredSize=2 \
  --instance-types m5.xlarge \
  --capacity-type ON_DEMAND \
  --subnets subnet-0a1b2c3d4e5f6g7h8 subnet-0a1b2c3d4e5f6g7h9 \
  --node-role arn:aws:iam::123456789012:role/EKS-NodeInstanceRole \
  --labels workload-type=critical,node-lifecycle=on-demand \
  --tags "k8s.io/cluster-autoscaler/enabled=true,k8s.io/cluster-autoscaler/my-cluster=owned"

2. Spot Node Group Configuration (for General Workloads) ​

Configuration using eksctl:

yaml
apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig
metadata:
  name: my-cluster
  region: us-west-2
nodeGroups:
  - name: general-workloads
    instanceTypes: ["m5.large", "m5a.large", "m5d.large", "m5ad.large"]
    desiredCapacity: 3
    minSize: 2
    maxSize: 10
    capacityType: SPOT
    labels:
      workload-type: general
      node-lifecycle: spot
    taints:
      - key: spot
        value: "true"
        effect: PreferNoSchedule
    tags:
      k8s.io/cluster-autoscaler/enabled: "true"
      k8s.io/cluster-autoscaler/my-cluster: "owned"
    iam:
      withAddonPolicies:
        autoScaler: true
    ssh:
      allow: false

Configuration using AWS CLI:

bash
aws eks create-nodegroup \
  --cluster-name my-cluster \
  --nodegroup-name general-workloads \
  --scaling-config minSize=2,maxSize=10,desiredSize=3 \
  --instance-types m5.large m5a.large m5d.large m5ad.large \
  --capacity-type SPOT \
  --subnets subnet-0a1b2c3d4e5f6g7h8 subnet-0a1b2c3d4e5f6g7h9 \
  --node-role arn:aws:iam::123456789012:role/EKS-NodeInstanceRole \
  --labels workload-type=general,node-lifecycle=spot \
  --taints "spot=true:PreferNoSchedule" \
  --tags "k8s.io/cluster-autoscaler/enabled=true,k8s.io/cluster-autoscaler/my-cluster=owned"

3. Node Affinity and Tolerations Example for Workload Placement ​

Critical Workload Deployment Example (prefer On-Demand nodes):

yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: critical-app
spec:
  replicas: 3
  selector:
    matchLabels:
      app: critical-app
  template:
    metadata:
      labels:
        app: critical-app
    spec:
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: workload-type
                operator: In
                values:
                - critical
              - key: node-lifecycle
                operator: In
                values:
                - on-demand
      containers:
      - name: critical-app
        image: my-critical-app:latest
        resources:
          requests:
            memory: "1Gi"
            cpu: "500m"
          limits:
            memory: "2Gi"
            cpu: "1000m"

General Workload Deployment Example (allow Spot nodes):

yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: general-app
spec:
  replicas: 5
  selector:
    matchLabels:
      app: general-app
  template:
    metadata:
      labels:
        app: general-app
    spec:
      affinity:
        nodeAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
          - weight: 1
            preference:
              matchExpressions:
              - key: node-lifecycle
                operator: In
                values:
                - spot
      tolerations:
      - key: "spot"
        operator: "Equal"
        value: "true"
        effect: "PreferNoSchedule"
      containers:
      - name: general-app
        image: my-general-app:latest
        resources:
          requests:
            memory: "512Mi"
            cpu: "250m"
          limits:
            memory: "1Gi"
            cpu: "500m"

4. Additional Optimization Settings ​

Cluster Autoscaler Deployment:

yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: cluster-autoscaler
  namespace: kube-system
  labels:
    app: cluster-autoscaler
spec:
  replicas: 1
  selector:
    matchLabels:
      app: cluster-autoscaler
  template:
    metadata:
      labels:
        app: cluster-autoscaler
    spec:
      serviceAccountName: cluster-autoscaler
      containers:
      - image: k8s.gcr.io/autoscaling/cluster-autoscaler:v1.23.0
        name: cluster-autoscaler
        command:
        - ./cluster-autoscaler
        - --v=4
        - --stderrthreshold=info
        - --cloud-provider=aws
        - --skip-nodes-with-local-storage=false
        - --expander=least-waste
        - --node-group-auto-discovery=asg:tag=k8s.io/cluster-autoscaler/enabled,k8s.io/cluster-autoscaler/my-cluster
        - --balance-similar-node-groups
        - --skip-nodes-with-system-pods=false

AWS Node Termination Handler (Spot instance interruption handling):

bash
helm repo add eks https://aws.github.io/eks-charts
helm install aws-node-termination-handler \
  --namespace kube-system \
  --set enableSpotInterruptionDraining=true \
  --set enableRebalanceMonitoring=true \
  --set enableRebalanceDraining=true \
  eks/aws-node-termination-handler

5. Best Practices and Considerations ​

  1. Use Various Instance Types: Using multiple instance types in Spot node groups distributes interruption risk.

  2. Set Pod Disruption Budgets (PDB): Set PDBs for critical applications to limit the number of pods disrupted simultaneously.

    yaml
    apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: critical-app-pdb
spec:
  minAvailable: 2
  selector:
    matchLabels:
      app: critical-app

  3. Set Appropriate Resource Requests and Limits: Set appropriate container resource requests and limits to efficiently utilize node resources.

  4. Utilize Horizontal Pod Autoscaler: Automatically adjust pod count based on workload demand.

    yaml
    apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: general-app-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: general-app
  minReplicas: 3
  maxReplicas: 10
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 70

  5. Cost Monitoring and Optimization: Monitor and optimize cluster costs using tools like AWS Cost Explorer and Kubecost.

Advanced Questions ​

10. Explain strategies for implementing multi-tenancy in an EKS cluster, and compare the advantages and disadvantages of each approach. ​

Answer and Explanation

Implementing multi-tenancy in an EKS cluster means ensuring appropriate isolation and resource management while multiple teams, applications, or customers share the same Kubernetes infrastructure. The following are the main strategies for implementing multi-tenancy in EKS and the advantages and disadvantages of each approach.

1. Cluster-Level Separation (Hard Multi-tenancy) ​

Description: Provisioning separate EKS clusters per tenant.

Implementation Method:

bash
# Create cluster for Tenant A
eksctl create cluster --name tenant-a-cluster --region us-west-2

# Create cluster for Tenant B
eksctl create cluster --name tenant-b-cluster --region us-west-2

Advantages:

  • Ensures complete isolation (security, networking, resources)
  • Customizable cluster versions and configurations per tenant
  • Issues with one tenant don't affect others
  • Suitable for environments with strict regulatory requirements

Disadvantages:

  • High operational overhead (managing multiple clusters)
  • Reduced resource utilization (duplicate control plane and system components per cluster)
  • Increased costs (control plane cost per cluster)
  • Difficulty in centralized management and policy enforcement

2. Namespace-Level Separation (Soft Multi-tenancy) ​

Description: Separating tenants using Kubernetes namespaces within a single EKS cluster.

Implementation Method:

yaml
# Create namespace for Tenant A
apiVersion: v1
kind: Namespace
metadata:
  name: tenant-a
  labels:
    tenant: a

# Create namespace for Tenant B
apiVersion: v1
kind: Namespace
metadata:
  name: tenant-b
  labels:
    tenant: b

Advantages:

  • Simplified management with single cluster
  • Improved resource utilization
  • Cost efficiency (shared control plane)
  • Easier centralized management and policy enforcement

Disadvantages:

  • Difficult to ensure complete isolation
  • Security risks due to cluster-level resource sharing
  • Excessive resource usage by one tenant can affect others
  • Cluster upgrades affect all tenants

3. Namespace-Level Separation + Additional Security Controls ​

Description: Applying additional security and resource control mechanisms to namespace separation.

Implementation Method:

  1. Network Policies:
yaml
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: deny-cross-tenant-traffic
  namespace: tenant-a
spec:
  podSelector: {}
  policyTypes:
  - Ingress
  - Egress
  ingress:
  - from:
    - namespaceSelector:
        matchLabels:
          tenant: a
  egress:
  - to:
    - namespaceSelector:
        matchLabels:
          tenant: a
  1. Resource Quotas:
yaml
apiVersion: v1
kind: ResourceQuota
metadata:
  name: tenant-quota
  namespace: tenant-a
spec:
  hard:
    requests.cpu: "10"
    requests.memory: 20Gi
    limits.cpu: "20"
    limits.memory: 40Gi
    pods: "50"
  1. RBAC Permission Control:
yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: tenant-admin
  namespace: tenant-a
subjects:
- kind: Group
  name: tenant-a-admins
  apiGroup: rbac.authorization.k8s.io
roleRef:
  kind: Role
  name: tenant-admin-role
  apiGroup: rbac.authorization.k8s.io

Advantages:

  • Maintains advantages of namespace separation
  • Enhanced security and resource isolation
  • Per-tenant access control and resource allocation
  • Maintains cost efficiency

Disadvantages:

  • Increased configuration and management complexity
  • Still lacks complete isolation of cluster-level resources
  • Additional effort required for policy setup and maintenance

4. Virtual Clusters ​

Description: Creating virtual Kubernetes control planes within a single physical EKS cluster, providing each tenant with their own "cluster".

Implementation Method:

bash
# Install vcluster
helm repo add vcluster https://charts.loft.sh
helm repo update

# Create virtual cluster for Tenant A
helm install vcluster-tenant-a vcluster/vcluster \
  --namespace tenant-a \
  --create-namespace \
  --set sync.nodes.enabled=true

# Create virtual cluster for Tenant B
helm install vcluster-tenant-b vcluster/vcluster \
  --namespace tenant-b \
  --create-namespace \
  --set sync.nodes.enabled=true

Advantages:

  • Combines advantages of cluster-level and namespace-level separation
  • Provides dedicated Kubernetes API server and control plane per tenant
  • Improved resource utilization and cost efficiency
  • Customizable cluster versions and configurations per tenant

Disadvantages:

  • Additional overhead and complexity
  • Limited maturity and support for virtual cluster technology
  • Limited support for some Kubernetes features
  • Complexity in debugging and troubleshooting

5. Multi-tenancy Utilizing AWS Service Integration ​

Description: Enhancing EKS cluster multi-tenancy using AWS services like AWS IAM, AWS Organizations, AWS Resource Access Manager.

Implementation Method:

  1. IAM Roles for Service Accounts (IRSA):
yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: tenant-a-sa
  namespace: tenant-a
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::123456789012:role/tenant-a-role
  1. AWS Organizations and SCP (Service Control Policies):
json
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "DenyAccessToOtherTenantsResources",
      "Effect": "Deny",
      "Action": ["s3:*"],
      "Resource": ["arn:aws:s3:::tenant-b-*"]
    }
  ]
}

Advantages:

  • Fine-grained access control for AWS services
  • Enhanced governance utilizing organizational structure and policies
  • Additional isolation layer at AWS service level
  • Integration with existing AWS security model

Disadvantages:

  • Increased dependency on AWS services
  • Increased configuration and management complexity
  • Limited portability as AWS-specific solution
  • Potential additional AWS service costs

Best Practices for Multi-tenancy Implementation ​

  1. Analyze Requirements:
    • Evaluate required level of isolation between tenants
    • Consider regulatory and compliance requirements
    • Consider operational overhead and cost constraints
  2. Consider Hybrid Approach:
    • Provide dedicated clusters for critical tenants
    • Group less critical tenants with namespace-level separation
  3. Automation and IaC (Infrastructure as Code):
    • Automate cluster and namespace provisioning using Terraform, AWS CDK, or eksctl
    • Manage configurations through GitOps workflows
  4. Monitoring and Cost Allocation:
    • Monitor resource usage per tenant
    • Track costs per tenant using cost allocation tags
    • Analyze costs using Kubecost or AWS Cost Explorer
  5. Security Enhancement:
    • Regular security audits and vulnerability scanning
    • Apply principle of least privilege
    • Utilize network policies and service mesh

Conclusion ​

The optimal strategy for implementing multi-tenancy in EKS varies depending on the organization's specific requirements, security needs, operational capabilities, and cost constraints. Many organizations adopt a hybrid approach combining multiple strategies rather than a single approach. For example, using dedicated clusters for critical or regulated workloads while applying namespace-level separation for development and test environments.

When selecting a multi-tenancy strategy, you should balance factors such as security, isolation, resource utilization, operational overhead, and cost. It's also important to evaluate whether the chosen strategy can adapt to the organization's changing requirements over time.