Kubernetes & Amazon EKS Training

English

한국어
中文
日本語
Español

English

한국어
中文
日本語
Español

Appearance

EKS Networking Quiz - Part 3 ​

This quiz tests your understanding of advanced networking concepts in Amazon EKS, including service mesh, VPC endpoints, multi-cluster networking, and network security.

Multiple Choice Questions ​

1. What is the main architectural change that occurs when implementing a service mesh (e.g., AWS App Mesh, Istio) in Amazon EKS? ​

A. All pod-to-pod communication is routed outside the VPC B. A sidecar proxy is added to each pod to mediate service-to-service communication C. Kubernetes Service objects are no longer used D. All network traffic is routed through AWS Transit Gateway

Show Answer

Answer: B. A sidecar proxy is added to each pod to mediate service-to-service communication

Explanation: The most significant architectural change when implementing a service mesh is that a sidecar proxy (typically Envoy) is added to each pod. This sidecar proxy intercepts and processes all inbound and outbound traffic for the pod, mediating service-to-service communication.

Key Features:

  1. Sidecar Pattern: A proxy container is deployed alongside each application container. This proxy handles all network communication.
  2. Traffic Flow Changes:
    • Traditional: Client → Service → Target Pod
    • Service Mesh: Client → Client Sidecar → Service → Target Sidecar → Target Pod
  3. Data Plane and Control Plane:
    • Data Plane: Collection of sidecar proxies
    • Control Plane: Central component that manages proxy configuration and applies policies
  4. No Application Code Changes: One of the main benefits of a service mesh is the ability to add advanced networking features without changing application code.

Service Mesh Implementation Example (AWS App Mesh):

yaml
# App Mesh sidecar injection example
apiVersion: apps/v1
kind: Deployment
metadata:
  name: example-app
  labels:
    app: example
spec:
  replicas: 3
  selector:
    matchLabels:
      app: example
  template:
    metadata:
      labels:
        app: example
      annotations:
        appmesh.k8s.aws/mesh: my-mesh  # App Mesh mesh name
        appmesh.k8s.aws/virtualNode: example-vn  # Virtual node name
    spec:
      containers:
      - name: example
        image: example:latest
        ports:
        - containerPort: 8080

Features Provided by Service Mesh:

  • Traffic management (routing, load balancing, circuit breaking)
  • Security (mTLS, authentication, authorization)
  • Observability (metrics, logs, distributed tracing)
  • Policy enforcement

Issues with other options:

  • A. All pod-to-pod communication is routed outside the VPC: Service mesh typically operates within the cluster and does not route traffic outside the VPC.
  • C. Kubernetes Service objects are no longer used: Service mesh complements rather than replaces Kubernetes Service objects.
  • D. All network traffic is routed through AWS Transit Gateway: Service mesh is unrelated to AWS Transit Gateway and manages service-to-service communication within the cluster.

2. What is the main benefit of using VPC endpoints to privately access AWS services in Amazon EKS? ​

A. Provides unlimited bandwidth to all AWS services B. Enables private access to AWS services without an internet gateway C. Reduces AWS service usage costs by 50% D. Provides automatic authentication to all AWS services

Show Answer

Answer: B. Enables private access to AWS services without an internet gateway

Explanation: The main benefit of using VPC endpoints in Amazon EKS is the ability to privately access AWS services without an internet gateway. This enhances security and reduces data transfer costs.

VPC Endpoint Types:

  1. Interface Endpoints (AWS PrivateLink):
    • Provides private connectivity to most AWS services
    • Creates endpoint network interfaces (ENIs) in each subnet
    • Examples: ECR, CloudWatch, SNS, SQS, etc.
  2. Gateway Endpoints:
    • Provides private connectivity to S3 and DynamoDB
    • Adds routes to route tables
    • No additional cost

VPC Endpoint Configuration Example for EKS:

yaml
# CloudFormation example
Resources:
  S3GatewayEndpoint:
    Type: AWS::EC2::VPCEndpoint
    Properties:
      ServiceName: !Sub com.amazonaws.${AWS::Region}.s3
      VpcId: !Ref VPC
      RouteTableIds:
        - !Ref PrivateRouteTable
      VpcEndpointType: Gateway

  ECRApiEndpoint:
    Type: AWS::EC2::VPCEndpoint
    Properties:
      ServiceName: !Sub com.amazonaws.${AWS::Region}.ecr.api
      VpcId: !Ref VPC
      SubnetIds:
        - !Ref PrivateSubnet1
        - !Ref PrivateSubnet2
      SecurityGroupIds:
        - !Ref EndpointSecurityGroup
      PrivateDnsEnabled: true
      VpcEndpointType: Interface

Key AWS Services Requiring VPC Endpoints for EKS:

  • Amazon ECR (pulling container images)
  • Amazon S3 (configuration files, backups, etc.)
  • AWS KMS (encryption keys)
  • Amazon CloudWatch (logging and monitoring)
  • AWS STS (assuming IAM roles)

Benefits of Using VPC Endpoints:

  1. Enhanced Security: Traffic does not traverse the public internet
  2. Reduced Network Costs: Decreased data transfer costs to AWS services
  3. Reduced Latency: Direct routing within the AWS network
  4. Compliance: Meets data sovereignty and regulatory requirements

EKS Node Configuration in Private Subnets:

bash
# Create node group in private subnets with eksctl
eksctl create nodegroup \
  --cluster my-cluster \
  --name private-ng \
  --node-private-networking \
  --vpc-private-subnets subnet-0123456789abcdef0,subnet-0123456789abcdef1

Issues with other options:

  • A. Provides unlimited bandwidth to all AWS services: VPC endpoints do not provide unlimited bandwidth; there may be bandwidth limits depending on the service and region.
  • C. Reduces AWS service usage costs by 50%: VPC endpoints can reduce data transfer costs, but they do not reduce AWS service usage costs by 50%.
  • D. Provides automatic authentication to all AWS services: VPC endpoints do not automate authentication; appropriate IAM permissions are still required.

3. What is the most effective method for implementing multi-cluster networking in Amazon EKS? ​

A. Use public load balancers on each cluster for inter-cluster communication B. Use AWS Transit Gateway to connect multiple VPCs and configure inter-cluster routing C. Deploy all clusters in a single VPC to reduce network complexity D. Use NAT gateways on each cluster for inter-cluster communication

Show Answer

Answer: B. Use AWS Transit Gateway to connect multiple VPCs and configure inter-cluster routing

Explanation: The most effective method for implementing multi-cluster networking in Amazon EKS is to use AWS Transit Gateway to connect multiple VPCs and configure inter-cluster routing. This approach provides scalability, security, and ease of management.

Multi-Cluster Networking with AWS Transit Gateway:

  1. Architecture Overview:

    • Each EKS cluster is deployed in a separate VPC
    • Transit Gateway connects all VPCs
    • Inter-cluster communication is routed through Transit Gateway

  2. Configuration Steps:

    bash
    # 1. Create Transit Gateway
aws ec2 create-transit-gateway --description "EKS Multi-Cluster TGW"

# 2. Attach VPC to Transit Gateway
aws ec2 create-transit-gateway-vpc-attachment \
  --transit-gateway-id tgw-0123456789abcdef0 \
  --vpc-id vpc-0123456789abcdef0 \
  --subnet-ids subnet-0123456789abcdef0 subnet-0123456789abcdef1

# 3. Update routing tables
aws ec2 create-route \
  --route-table-id rtb-0123456789abcdef0 \
  --destination-cidr-block 10.1.0.0/16 \
  --transit-gateway-id tgw-0123456789abcdef0

  3. CIDR Planning:

    • Assign non-overlapping CIDR blocks to each cluster/VPC
    • Example: Cluster1: 10.0.0.0/16, Cluster2: 10.1.0.0/16, Cluster3: 10.2.0.0/16

Multi-Cluster Service Discovery Options:

  1. AWS Cloud Map:

    bash
    # Create namespace
aws servicediscovery create-private-dns-namespace \
  --name multi-cluster.local \
  --vpc vpc-0123456789abcdef0

# Register service
aws servicediscovery register-instance \
  --service-id srv-0123456789abcdef0 \
  --instance-id api-service-cluster1 \
  --attributes AWS_INSTANCE_IPV4=10.0.1.123

  2. Custom CoreDNS Configuration:

    yaml
    apiVersion: v1
kind: ConfigMap
metadata:
  name: coredns
  namespace: kube-system
data:
  Corefile: |
    .:53 {
        errors
        health
        kubernetes cluster.local in-addr.arpa ip6.arpa {
           pods insecure
           upstream
           fallthrough in-addr.arpa ip6.arpa
        }
        forward . /etc/resolv.conf
        cache 30
        loop
        reload
        loadbalance
    }
    cluster2.svc.local:53 {
        errors
        cache 30
        forward . 10.1.0.2
    }

Multi-Cluster Networking Security Considerations:

  1. Inter-VPC Traffic Control:

    • Use Transit Gateway security groups and routing tables to restrict traffic
    • Allow only necessary ports and protocols

  2. Network Policies:

    yaml
    apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-cross-cluster
spec:
  podSelector:
    matchLabels:
      app: api-service
  ingress:
  - from:
    - ipBlock:
        cidr: 10.1.0.0/16  # Cluster2's CIDR
  egress:
  - to:
    - ipBlock:
        cidr: 10.1.0.0/16  # Cluster2's CIDR

Multi-Cluster Service Mesh Options:

  1. Istio Multi-Cluster:
    • Manage multiple clusters with a single control plane
    • Cross-cluster service discovery and load balancing
  2. AWS App Mesh:
    • Create mesh spanning multiple clusters
    • Service discovery through AWS Cloud Map

Cost Optimization Considerations:

  • Consider Transit Gateway hourly and data processing charges
  • Minimize cross-cluster data transfer
  • Communicate within the same availability zone when possible

Issues with other options:

  • A. Use public load balancers on each cluster for inter-cluster communication: This method increases security risks, incurs internet data transfer costs, and increases latency.
  • C. Deploy all clusters in a single VPC to reduce network complexity: Deploying multiple clusters in a single VPC can lead to IP address space limitations, lack of security boundaries, and scalability issues.
  • D. Use NAT gateways on each cluster for inter-cluster communication: NAT gateways are for outbound internet traffic and are not suitable for inter-cluster communication.

5. What is the most effective method for optimizing pod networking performance in Amazon EKS? ​

A. Use host network mode for all pods B. Enable prefix delegation feature of Amazon VPC CNI C. Use NodePort services for all pods D. Use AWS Global Accelerator for all intra-cluster communication

Show Answer

Answer: B. Enable prefix delegation feature of Amazon VPC CNI

Explanation: The most effective method for optimizing pod networking performance in Amazon EKS is to enable the prefix delegation feature of Amazon VPC CNI. This feature significantly increases the number of secondary IP addresses allocated to each node and reduces ENI (Elastic Network Interface) creation frequency, improving networking performance and scalability.

How Prefix Delegation Works:

  1. Default VPC CNI vs Prefix Delegation:

    • Default VPC CNI: Allocates individual secondary IP addresses to each ENI
    • Prefix Delegation: Allocates /28 CIDR blocks (16 IPs) to each ENI

  2. Enabling Method:

    bash
    # Enable prefix delegation
kubectl set env daemonset aws-node -n kube-system ENABLE_PREFIX_DELEGATION=true

# Verify prefix delegation
kubectl describe daemonset aws-node -n kube-system | grep ENABLE_PREFIX_DELEGATION

  3. Additional Configuration Options:

    bash
    # Set prefix allocation size (default: /28)
kubectl set env daemonset aws-node -n kube-system WARM_PREFIX_TARGET=1

# Threshold for requesting new prefix when available IPs are low
kubectl set env daemonset aws-node -n kube-system WARM_IP_TARGET=5

Benefits of Prefix Delegation:

  1. Improved Scalability:
    • Increased maximum pods per node (typically from 110 to 250+)
    • Reduced API throttling due to fewer ENI creation calls
  2. Faster Pod Startup Time:
    • Reduced API calls needed to allocate IP addresses to new pods
    • Improved performance for large-scale pod deployments
  3. IP Address Efficiency:
    • Support more pods with the same number of ENIs
    • Mitigates IP address exhaustion issues

Maximum Pods Per Instance Type Comparison:

Instance TypeDefault VPC CNIPrefix Delegation Enabled
t3.medium17110
m5.large29110
c5.xlarge58250
r5.2xlarge58250

Configuration Example (ConfigMap):

yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: amazon-vpc-cni
  namespace: kube-system
data:
  enable-prefix-delegation: "true"
  warm-prefix-target: "1"
  warm-ip-target: "5"

Considerations and Limitations:

  1. Subnet Size:
    • Prefix delegation requires sufficiently large subnets
    • Minimum /24 CIDR block recommended
  2. Security Group Rules:
    • Security group rules can be simplified with prefix delegation
    • Can reference CIDR blocks instead of individual IPs
  3. Compatibility:
    • Some legacy EC2 instance types do not support prefix delegation
    • Nitro-based instances recommended
  4. IP Address Management:
    • Prefix delegation uses IP addresses more efficiently, but proper CIDR planning is still required

Monitoring and Troubleshooting:

bash
# Check IP address allocation per node
kubectl exec -n kube-system aws-node-xxxxx -- curl -s http://localhost:61679/v1/enis | jq

# Check prefix delegation status
kubectl logs -n kube-system aws-node-xxxxx | grep -i prefix

Issues with other options:

  • A. Use host network mode for all pods: Host network mode causes pods to share the node's network namespace, leading to port conflicts and removing network isolation.
  • C. Use NodePort services for all pods: NodePort is a service exposure mechanism and is unrelated to pod networking performance optimization.
  • D. Use AWS Global Accelerator for all intra-cluster communication: AWS Global Accelerator is for global traffic management and is not suitable for intra-cluster communication optimization.

Short Answer Questions ​

7. What is the most commonly used open-source proxy for sidecar proxies when implementing a service mesh in Amazon EKS? ​

Show Answer

Answer: Envoy

Detailed Explanation:

The most commonly used sidecar proxy when implementing a service mesh in Amazon EKS is Envoy. Envoy is a high-performance C++-based proxy used as the data plane proxy in most major service mesh implementations (Istio, AWS App Mesh, Consul Connect, etc.).

Key Features of Envoy:

  1. High-Performance Architecture:
    • Written in C++ for low latency and high throughput
    • Event-driven, asynchronous networking model
  2. Rich Traffic Management Features:
    • Load balancing (round robin, weighted, least request, etc.)
    • Circuit breaking and outlier detection
    • Retry and timeout policies
    • Traffic splitting and mirroring
  3. Observability:
    • Detailed metrics and statistics
    • Distributed tracing integration (Zipkin, Jaeger, etc.)
    • Access logging
  4. Security Features:
    • TLS/mTLS termination
    • Authentication and authorization
    • Rate limiting

Envoy Deployment in Service Mesh:

  1. Sidecar Pattern:

    yaml
    apiVersion: apps/v1
kind: Deployment
metadata:
  name: example-app
spec:
  template:
    spec:
      containers:
      - name: app
        image: app:latest
      - name: envoy-proxy
        image: envoyproxy/envoy:v1.20.0
        ports:
        - containerPort: 15001
        volumeMounts:
        - name: envoy-config
          mountPath: /etc/envoy
      volumes:
      - name: envoy-config
        configMap:
          name: envoy-config

  2. Automatic Injection:

    • Istio: sidecar.istio.io/inject: "true" annotation
    • AWS App Mesh: appmesh.k8s.aws/sidecarInjectorWebhook: enabled label

Envoy Configuration Example:

yaml
static_resources:
  listeners:
  - address:
      socket_address:
        address: 0.0.0.0
        port_value: 15001
    filter_chains:
    - filters:
      - name: envoy.filters.network.http_connection_manager
        typed_config:
          "@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
          stat_prefix: ingress_http
          route_config:
            name: local_route
            virtual_hosts:
            - name: backend
              domains: ["*"]
              routes:
              - match:
                  prefix: "/"
                route:
                  cluster: service_backend
          http_filters:
          - name: envoy.filters.http.router
            typed_config:
              "@type": type.googleapis.com/envoy.extensions.filters.http.router.v3.Router
  clusters:
  - name: service_backend
    connect_timeout: 0.25s
    type: STRICT_DNS
    lb_policy: ROUND_ROBIN
    load_assignment:
      cluster_name: service_backend
      endpoints:
      - lb_endpoints:
        - endpoint:
            address:
              socket_address:
                address: backend-service
                port_value: 80

Envoy Integration by Service Mesh:

  1. Istio:
    • Uses Envoy as sidecar proxy
    • istiod dynamically manages Envoy configuration
    • Components like Pilot, Mixer, Citadel integrate with Envoy
  2. AWS App Mesh:
    • AWS App Mesh controller injects Envoy sidecar
    • Integrates with AWS Cloud Map for service discovery
    • Envoy Management Service (EMS) manages Envoy configuration
  3. Consul Connect:
    • Uses Envoy as data plane proxy
    • Consul provides service discovery and configuration management

Envoy Monitoring and Debugging:

bash
# Port forward Envoy admin interface
kubectl port-forward <pod-name> 19000:19000

# Check configuration and stats
curl localhost:19000/config_dump
curl localhost:19000/stats

# Check cluster status
curl localhost:19000/clusters

Performance Optimization Considerations:

  • Resource allocation: Allocate sufficient CPU and memory to Envoy
  • Connection pooling: Configure upstream connection pooling for improved performance
  • Buffer size: Set appropriate buffer sizes for memory usage optimization
  • Filter chain: Enable only necessary filters to minimize overhead

Envoy is a core component of modern service mesh architecture, playing a critical role in making microservice communication secure, reliable, and observable.

8. What is the name of the Kubernetes add-on responsible for internal DNS resolution in Amazon EKS clusters? ​

Show Answer

Answer: CoreDNS

Detailed Explanation:

The Kubernetes add-on responsible for internal DNS resolution in Amazon EKS clusters is CoreDNS. CoreDNS serves as the DNS server for service discovery within Kubernetes clusters, handling name resolution for pods and services.

Key Features of CoreDNS:

  1. Service Discovery:
    • Resolves DNS names in the format <service-name>.<namespace>.svc.cluster.local
    • Supports reverse DNS lookups for pod IP addresses
  2. Plugin Architecture:
    • Extends functionality through various plugins
    • Caching, metrics, logging, error handling, etc.
  3. Configuration Flexibility:
    • Declarative configuration through Corefile
    • Supports dynamic reload

CoreDNS Deployment in EKS:

  1. Default Deployment Configuration:

    • Automatically deployed when EKS cluster is created
    • Runs in the kube-system namespace
    • Typically deployed with 2 or more replicas

  2. Verification:

    bash
    # Check CoreDNS pods
kubectl get pods -n kube-system -l k8s-app=kube-dns

# Check CoreDNS version
kubectl describe deployment coredns -n kube-system | grep Image

CoreDNS Configuration (Corefile):

yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: coredns
  namespace: kube-system
data:
  Corefile: |
    .:53 {
        errors
        health {
            lameduck 5s
        }
        ready
        kubernetes cluster.local in-addr.arpa ip6.arpa {
            pods insecure
            fallthrough in-addr.arpa ip6.arpa
            ttl 30
        }
        prometheus :9153
        forward . /etc/resolv.conf
        cache 30
        loop
        reload
        loadbalance
    }

Key Plugin Descriptions:

  1. errors: Logs errors
  2. health: Provides health check endpoint
  3. ready: Provides readiness check endpoint
  4. kubernetes: Handles Kubernetes service discovery
  5. prometheus: Exposes Prometheus metrics
  6. forward: Forwards external DNS queries to upstream DNS servers
  7. cache: Caches DNS responses
  8. loop: Detects and prevents DNS loops
  9. reload: Automatically reloads on Corefile changes
  10. loadbalance: Load balances across multiple A/AAAA records

Custom Configuration Examples:

  1. Use Specific DNS Server for External Domain:

    example.com {
    forward . 10.0.0.1
}

  2. Stub Domain Configuration:

    internal.corp {
    file /etc/coredns/internal.db
}

  3. Conditional Forwarding:

    . {
    forward . 8.8.8.8 8.8.4.4 {
        policy sequential
    }
}

Performance Optimization and Scaling:

  1. Auto Scaling:

    yaml
    apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: coredns
  namespace: kube-system
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: coredns
  minReplicas: 2
  maxReplicas: 10
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 60

  2. Resource Allocation Optimization:

    yaml
    resources:
  limits:
    memory: 170Mi
  requests:
    cpu: 100m
    memory: 70Mi

  3. Cache Tuning:

    cache {
    success 10000
    denial 1000
    prefetch 10 10% 2m
}

Troubleshooting:

  1. DNS Resolution Test:

    bash
    # Create test pod
kubectl run dnsutils --image=gcr.io/kubernetes-e2e-test-images/dnsutils:1.3 -- sleep 3600

# Test DNS lookup
kubectl exec -it dnsutils -- nslookup kubernetes.default

  2. Check CoreDNS Logs:

    bash
    kubectl logs -n kube-system -l k8s-app=kube-dns

  3. Check DNS Policy:

    bash
    kubectl get pods <pod-name> -o jsonpath='{.spec.dnsPolicy}'

CoreDNS is a critical component of EKS clusters, providing core service discovery functionality for microservices architecture. Ensuring reliable DNS service through proper configuration and monitoring is essential.

Hands-on Questions ​

10. Explain how to implement a service mesh (e.g., AWS App Mesh) in an Amazon EKS cluster to secure and monitor microservice communication. Include implementation steps, key components, and monitoring methods. ​

Show Answer

Answer:

Here's how to implement AWS App Mesh in an Amazon EKS cluster to secure and monitor microservice communication:

1. AWS App Mesh Implementation Steps ​

1.1. Set Up Prerequisites ​

bash
# Set up required IAM permissions
eksctl create iamserviceaccount \
  --cluster=my-cluster \
  --namespace=appmesh-system \
  --name=appmesh-controller \
  --attach-policy-arn=arn:aws:iam::aws:policy/AWSCloudMapFullAccess,arn:aws:iam::aws:policy/AWSAppMeshFullAccess \
  --override-existing-serviceaccounts \
  --approve

# Add Helm repository
helm repo add eks https://aws.github.io/eks-charts
helm repo update

1.2. Install App Mesh Controller ​

bash
# Create App Mesh controller namespace
kubectl create ns appmesh-system

# Install App Mesh controller
helm install appmesh-controller eks/appmesh-controller \
  --namespace appmesh-system \
  --set region=${AWS_REGION} \
  --set serviceAccount.create=false \
  --set serviceAccount.name=appmesh-controller

1.3. Create Mesh ​

yaml
# mesh.yaml
apiVersion: appmesh.k8s.aws/v1beta2
kind: Mesh
metadata:
  name: my-mesh
spec:
  namespaceSelector:
    matchLabels:
      mesh: my-mesh
bash
kubectl apply -f mesh.yaml

1.4. Set Up Application Namespace ​

bash
# Create and label application namespace
kubectl create ns app-namespace
kubectl label namespace app-namespace mesh=my-mesh
kubectl label namespace app-namespace appmesh.k8s.aws/sidecarInjectorWebhook=enabled

1.5. Define Virtual Nodes and Services ​

yaml
# virtual-node.yaml
apiVersion: appmesh.k8s.aws/v1beta2
kind: VirtualNode
metadata:
  name: service-a
  namespace: app-namespace
spec:
  podSelector:
    matchLabels:
      app: service-a
  listeners:
    - portMapping:
        port: 8080
        protocol: http
      healthCheck:
        protocol: http
        path: "/health"
        port: 8080
        healthyThreshold: 2
        unhealthyThreshold: 2
        timeoutMillis: 2000
        intervalMillis: 5000
  backends:
    - virtualService:
        virtualServiceRef:
          name: service-b
  serviceDiscovery:
    dns:
      hostname: service-a.app-namespace.svc.cluster.local
yaml
# virtual-service.yaml
apiVersion: appmesh.k8s.aws/v1beta2
kind: VirtualService
metadata:
  name: service-a
  namespace: app-namespace
spec:
  awsName: service-a.app-namespace.svc.cluster.local
  provider:
    virtualRouter:
      virtualRouterRef:
        name: service-a-router
yaml
# virtual-router.yaml
apiVersion: appmesh.k8s.aws/v1beta2
kind: VirtualRouter
metadata:
  name: service-a-router
  namespace: app-namespace
spec:
  listeners:
    - portMapping:
        port: 8080
        protocol: http
  routes:
    - name: service-a-route
      httpRoute:
        match:
          prefix: /
        action:
          weightedTargets:
            - virtualNodeRef:
                name: service-a
              weight: 1

1.6. Deploy Application ​

yaml
# deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: service-a
  namespace: app-namespace
spec:
  replicas: 3
  selector:
    matchLabels:
      app: service-a
  template:
    metadata:
      labels:
        app: service-a
    spec:
      containers:
      - name: service-a
        image: service-a:latest
        ports:
        - containerPort: 8080
        livenessProbe:
          httpGet:
            path: /health
            port: 8080
        readinessProbe:
          httpGet:
            path: /health
            port: 8080

2. Secure Communication with mTLS Configuration ​

2.1. Set Up AWS Certificate Manager Private CA ​

bash
# Create private CA
aws acm-pca create-certificate-authority \
  --certificate-authority-configuration file://ca-config.json \
  --certificate-authority-type "ROOT" \
  --idempotency-token 1234567890 \
  --tags Key=Name,Value=AppMeshCA

# Save CA ARN
export CA_ARN=$(aws acm-pca list-certificate-authorities --query 'CertificateAuthorities[?Status==`ACTIVE`].Arn' --output text)

2.2. Add TLS Configuration ​

yaml
# virtual-node-with-tls.yaml
apiVersion: appmesh.k8s.aws/v1beta2
kind: VirtualNode
metadata:
  name: service-a
  namespace: app-namespace
spec:
  podSelector:
    matchLabels:
      app: service-a
  listeners:
    - portMapping:
        port: 8080
        protocol: http
      tls:
        mode: STRICT  # Enable mTLS
        certificate:
          acm:
            certificateArn: arn:aws:acm:region:account-id:certificate/certificate-id
  backends:
    - virtualService:
        virtualServiceRef:
          name: service-b
        clientPolicy:
          tls:
            enforce: true
            ports:
              - 8080
            validation:
              trust:
                acm:
                  certificateAuthorityArns:
                    - ${CA_ARN}
  serviceDiscovery:
    dns:
      hostname: service-a.app-namespace.svc.cluster.local

3. Set Up Monitoring and Observability ​

3.1. AWS X-Ray Integration ​

yaml
# mesh-with-xray.yaml
apiVersion: appmesh.k8s.aws/v1beta2
kind: Mesh
metadata:
  name: my-mesh
spec:
  namespaceSelector:
    matchLabels:
      mesh: my-mesh
  egressFilter:
    type: ALLOW_ALL
  tracing:
    awsXRay:
      logLevel: INFO
bash
# Deploy X-Ray daemon
kubectl apply -f https://github.com/aws/aws-app-mesh-controller-for-k8s/raw/master/config/samples/xray-daemon.yaml

3.2. Amazon CloudWatch Integration ​

yaml
# envoy-config.yaml
apiVersion: appmesh.k8s.aws/v1beta2
kind: Mesh
metadata:
  name: my-mesh
spec:
  namespaceSelector:
    matchLabels:
      mesh: my-mesh
  egressFilter:
    type: ALLOW_ALL
  serviceDiscovery:
    ipPreference: IPv4_PREFERRED
  logging:
    accessLog:
      file:
        path: /dev/stdout
        format:
          json:
            - key: "source"
              value: "%DOWNSTREAM_REMOTE_ADDRESS%"
            - key: "destination"
              value: "%UPSTREAM_REMOTE_ADDRESS%"
            - key: "protocol"
              value: "%PROTOCOL%"
bash
# Deploy CloudWatch agent
kubectl apply -f https://raw.githubusercontent.com/aws-samples/amazon-cloudwatch-container-insights/latest/k8s-deployment-manifest-templates/deployment-mode/daemonset/container-insights-monitoring/cloudwatch-namespace.yaml

kubectl apply -f https://raw.githubusercontent.com/aws-samples/amazon-cloudwatch-container-insights/latest/k8s-deployment-manifest-templates/deployment-mode/daemonset/container-insights-monitoring/cwagent/cwagent-serviceaccount.yaml

kubectl apply -f https://raw.githubusercontent.com/aws-samples/amazon-cloudwatch-container-insights/latest/k8s-deployment-manifest-templates/deployment-mode/daemonset/container-insights-monitoring/cwagent/cwagent-configmap.yaml

kubectl apply -f https://raw.githubusercontent.com/aws-samples/amazon-cloudwatch-container-insights/latest/k8s-deployment-manifest-templates/deployment-mode/daemonset/container-insights-monitoring/cwagent/cwagent-daemonset.yaml

3.3. Prometheus and Grafana Setup ​

bash
# Create Prometheus namespace
kubectl create namespace prometheus

# Install Prometheus
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update
helm install prometheus prometheus-community/prometheus \
  --namespace prometheus \
  --set alertmanager.persistentVolume.storageClass="gp2" \
  --set server.persistentVolume.storageClass="gp2"

# Install Grafana
helm repo add grafana https://grafana.github.io/helm-charts
helm repo update
helm install grafana grafana/grafana \
  --namespace prometheus \
  --set persistence.storageClassName="gp2" \
  --set persistence.enabled=true \
  --set adminPassword='EKS!sAWSome' \
  --values grafana.yaml \
  --set service.type=LoadBalancer
yaml
# grafana.yaml
datasources:
  datasources.yaml:
    apiVersion: 1
    datasources:
    - name: Prometheus
      type: prometheus
      url: http://prometheus-server.prometheus.svc.cluster.local
      access: proxy
      isDefault: true

4. Configure Traffic Management and Advanced Features ​

4.1. Canary Deployment Configuration ​

yaml
# virtual-router-canary.yaml
apiVersion: appmesh.k8s.aws/v1beta2
kind: VirtualRouter
metadata:
  name: service-a-router
  namespace: app-namespace
spec:
  listeners:
    - portMapping:
        port: 8080
        protocol: http
  routes:
    - name: service-a-route
      httpRoute:
        match:
          prefix: /
        action:
          weightedTargets:
            - virtualNodeRef:
                name: service-a-v1
              weight: 90
            - virtualNodeRef:
                name: service-a-v2
              weight: 10

4.2. Circuit Breaker Configuration ​

yaml
# virtual-node-circuit-breaker.yaml
apiVersion: appmesh.k8s.aws/v1beta2
kind: VirtualNode
metadata:
  name: service-a
  namespace: app-namespace
spec:
  # ... existing configuration ...
  listeners:
    - portMapping:
        port: 8080
        protocol: http
      outlierDetection:
        baseEjectionDuration:
          unit: s
          value: 30
        interval:
          unit: s
          value: 10
        maxEjectionPercent: 50
        maxServerErrors: 5

4.3. Retry Policy Configuration ​

yaml
# virtual-router-retry.yaml
apiVersion: appmesh.k8s.aws/v1beta2
kind: VirtualRouter
metadata:
  name: service-a-router
  namespace: app-namespace
spec:
  # ... existing configuration ...
  routes:
    - name: service-a-route
      httpRoute:
        match:
          prefix: /
        action:
          weightedTargets:
            - virtualNodeRef:
                name: service-a
              weight: 1
        retryPolicy:
          maxRetries: 3
          perRetryTimeout:
            unit: ms
            value: 2000
          httpRetryEvents:
            - server-error
            - gateway-error
            - client-error
            - stream-error

5. Monitoring and Troubleshooting ​

5.1. Check Envoy Proxy Logs ​

bash
# Check Envoy sidecar logs for specific pod
kubectl logs <pod-name> -c envoy -n app-namespace

# Stream all Envoy logs
kubectl logs -f -l app=service-a -c envoy -n app-namespace

5.2. Access Envoy Admin Interface ​

bash
# Set up port forwarding
kubectl port-forward <pod-name> -n app-namespace 9901:9901

# Access in browser
# http://localhost:9901/

5.3. Check X-Ray Traces ​

Navigate to the X-Ray service in the AWS Management Console to view service maps and traces.

5.4. Create CloudWatch Dashboard ​

bash
# Prepare JSON file for CloudWatch dashboard creation
cat > appmesh-dashboard.json << EOF
{
  "widgets": [
    {
      "type": "metric",
      "x": 0,
      "y": 0,
      "width": 12,
      "height": 6,
      "properties": {
        "metrics": [
          [ "AWS/AppMesh", "RequestCount", "MeshName", "my-mesh", "VirtualNodeName", "service-a", { "stat": "Sum" } ]
        ],
        "period": 60,
        "region": "${AWS_REGION}",
        "title": "Request Count"
      }
    },
    {
      "type": "metric",
      "x": 12,
      "y": 0,
      "width": 12,
      "height": 6,
      "properties": {
        "metrics": [
          [ "AWS/AppMesh", "Latency", "MeshName", "my-mesh", "VirtualNodeName", "service-a", { "stat": "Average" } ]
        ],
        "period": 60,
        "region": "${AWS_REGION}",
        "title": "Latency"
      }
    }
  ]
}
EOF

# Create dashboard using AWS CLI
aws cloudwatch put-dashboard --dashboard-name AppMeshDashboard --dashboard-body file://appmesh-dashboard.json

6. Best Practices and Considerations ​

6.1. Resource Requirements ​

  • Plan for node resources as Envoy sidecar is added to each pod
  • Typically allocate 100-200m CPU and 128-256Mi memory to each Envoy proxy

6.2. Gradual Implementation Strategy ​

  1. Phased Approach:
    • Start with non-business-critical services
    • Evaluate impact with traffic mirroring
    • Gradually expand after successful validation
  2. mTLS Implementation:
    • Start with PERMISSIVE mode
    • Verify all services are compatible
    • Switch to STRICT mode

6.3. Performance Optimization ​

  • Adjust Envoy resource limits
  • Set appropriate health check intervals
  • Minimize unnecessary logging and tracing

6.4. Security Hardening ​

  • Use least privilege IAM policies
  • Regular certificate rotation
  • Implement defense in depth with network policies

AWS App Mesh provides a powerful service mesh solution for securing and monitoring microservice communication in EKS clusters. Proper configuration and monitoring can significantly improve application reliability, security, and observability.