Kubernetes Cluster Administration

Supported Versions: Kubernetes 1.34 (Released 2025-11-24) Last Updated: February 23, 2026

Kubernetes cluster administration is an important task that includes cluster setup, maintenance, monitoring, troubleshooting, and upgrades. In this chapter, we will explore various aspects of Kubernetes cluster administration and best practices for cluster management in Amazon EKS.

Core Concepts

Cluster Lifecycle Management: The entire process from cluster creation to decommissioning
Control Plane Management: Managing core components such as API server, scheduler, and controller manager
Node Management: Adding, removing, and maintaining worker nodes
Resource Allocation: Setting resource allocation and limits for CPU, memory, storage, etc.
Upgrade Strategy: Cluster and application upgrade strategies to minimize downtime

Cluster Administration Overview
Cluster Component Management
Resource Management
Cluster Networking
Authentication and Authorization Management
Cluster Upgrades
Backup and Recovery
Monitoring and Logging
Troubleshooting
Amazon EKS Cluster Administration
Cluster Administration Best Practices
Conclusion

Environment Setup

The following tools are required for cluster administration:

bash

# Install kubectl (Linux)
curl -LO "https://dl.k8s.io/release/v1.33.3/bin/linux/amd64/kubectl"
chmod +x kubectl
sudo mv kubectl /usr/local/bin/

# Install kubeadm (for cluster creation and management)
sudo apt-get update && sudo apt-get install -y kubeadm=1.33.3-00

# Install Helm (for package management)
curl https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 | bash

# Install k9s (cluster management UI)
curl -sS https://webinstall.dev/k9s | bash

Cluster Administration Overview

Kubernetes cluster administration is the process of managing the entire lifecycle of a cluster. This includes the following main areas:

Cluster Setup and Configuration: Cluster creation, node addition, networking setup, storage configuration, etc.
Operations Management: Resource monitoring, performance optimization, capacity planning, troubleshooting
Security Management: Authentication, authorization, network policies, security contexts, etc.
Upgrades and Patches: Cluster version upgrades, security patch application
Backup and Recovery: Cluster data backup, disaster recovery planning

The following diagram shows the main areas of Kubernetes cluster administration and related tools:

Cluster Component Management

A Kubernetes cluster consists of control plane components and node components. Managing each component is critical for cluster stability and performance.

Control Plane Component Management

API Server Management

The API server is a core component of the control plane that exposes the Kubernetes API.

bash

# Check API server logs
kubectl logs -n kube-system kube-apiserver-<master-node-name>

# Check API server configuration (kubeadm cluster)
sudo cat /etc/kubernetes/manifests/kube-apiserver.yaml

# Check API server status
kubectl get --raw='/healthz'

etcd Management

etcd is a distributed key-value store that stores all cluster data for Kubernetes.

bash

# etcd backup
ETCDCTL_API=3 etcdctl --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key \
  snapshot save /backup/etcd-snapshot-$(date +%Y-%m-%d).db

# Check etcd status
ETCDCTL_API=3 etcdctl --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key \
  endpoint health

Node Management

Nodes are worker machines that run containerized applications.

bash

# List nodes
kubectl get nodes

# Check node detailed information
kubectl describe node <node-name>

# Add node label
kubectl label node <node-name> environment=production

# Set node to maintenance mode
kubectl drain <node-name> --ignore-daemonsets

# Return node after maintenance
kubectl uncordon <node-name>

Component Status Monitoring

bash

# Check control plane component status
kubectl get componentstatuses

# Check system pod status
kubectl get pods -n kube-system

# Check node resource usage
kubectl top nodes

Cluster Administration Tools

Various tools are available for Kubernetes cluster administration:

kubectl: Command-line tool for interacting with Kubernetes clusters
kubeadm: Tool for creating and managing Kubernetes clusters
kops: Tool for creating, upgrading, and managing Kubernetes clusters
eksctl: Tool for creating and managing Amazon EKS clusters
Helm: Kubernetes application package manager
Kubernetes Dashboard: Web-based Kubernetes user interface
Prometheus & Grafana: Monitoring and alerting tools
Fluentd & Elasticsearch: Logging tools

Cluster Component Management

A Kubernetes cluster consists of multiple components, and effectively managing these components is important.

Control Plane Components

Control plane components manage the overall state of the cluster:

kube-apiserver: Component that exposes the Kubernetes API
etcd: Key-value store that stores cluster data
kube-scheduler: Component that schedules pods to nodes
kube-controller-manager: Component that runs controllers
cloud-controller-manager: Component that interacts with cloud providers

The following diagram shows Kubernetes control plane components and their interactions:

Control Plane Component Monitoring

It is important to monitor the status of control plane components:

bash

# Check control plane component status
kubectl get componentstatuses

# Check API server logs
kubectl logs -n kube-system kube-apiserver-<node-name>

# Check etcd status
kubectl exec -it -n kube-system etcd-<node-name> -- etcdctl endpoint health

Control Plane Component Configuration

How to manage control plane component configuration:

yaml

# kube-apiserver configuration example
apiVersion: v1
kind: Pod
metadata:
  name: kube-apiserver
  namespace: kube-system
spec:
  containers:
  - command:
    - kube-apiserver
    - --advertise-address=192.168.1.10
    - --allow-privileged=true
    - --authorization-mode=Node,RBAC
    - --client-ca-file=/etc/kubernetes/pki/ca.crt
    - --enable-admission-plugins=NodeRestriction
    - --enable-bootstrap-token-auth=true
    - --etcd-cafile=/etc/kubernetes/pki/etcd/ca.crt
    - --etcd-certfile=/etc/kubernetes/pki/apiserver-etcd-client.crt
    - --etcd-keyfile=/etc/kubernetes/pki/apiserver-etcd-client.key
    - --etcd-servers=https://127.0.0.1:2379
    - --kubelet-client-certificate=/etc/kubernetes/pki/apiserver-kubelet-client.crt
    - --kubelet-client-key=/etc/kubernetes/pki/apiserver-kubelet-client.key
    - --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname
    - --secure-port=6443
    - --service-account-key-file=/etc/kubernetes/pki/sa.pub
    - --service-cluster-ip-range=10.96.0.0/12
    - --tls-cert-file=/etc/kubernetes/pki/apiserver.crt
    - --tls-private-key-file=/etc/kubernetes/pki/apiserver.key
    image: k8s.gcr.io/kube-apiserver:v1.21.0
    name: kube-apiserver

Node Components

Node components run on each node and manage pods:

kubelet: Agent running on each node that ensures pods and containers are running
kube-proxy: Maintains network rules and handles connection forwarding
Container Runtime: Software that runs containers (Docker, containerd, CRI-O, etc.)

Node Management

Key commands for node management:

bash

# List nodes
kubectl get nodes

# Check node detailed information
kubectl describe node <node-name>

# Add node label
kubectl label node <node-name> key=value

# Add node taint
kubectl taint node <node-name> key=value:NoSchedule

# Set node to maintenance mode
kubectl cordon <node-name>

# Drain node
kubectl drain <node-name> --ignore-daemonsets --delete-emptydir-data

Node Troubleshooting

Commands for node troubleshooting:

bash

# Check node status
kubectl describe node <node-name> | grep Conditions -A 10

# Check node resource usage
kubectl top node <node-name>

# Check kubelet logs
journalctl -u kubelet

# Check container runtime status
systemctl status docker  # When using Docker
systemctl status containerd  # When using containerd

Resource Management

Effectively managing resources in a Kubernetes cluster is important for maintaining cluster stability and performance.

Resource Quotas

Resource quotas limit resource usage per namespace:

yaml

apiVersion: v1
kind: ResourceQuota
metadata:
  name: compute-resources
  namespace: dev
spec:
  hard:
    requests.cpu: "1"
    requests.memory: 1Gi
    limits.cpu: "2"
    limits.memory: 2Gi
    pods: "10"

In the above example, the dev namespace can have a maximum of 10 pods, 1 CPU and 1Gi memory requests, and 2 CPU and 2Gi memory limits.

Limit Ranges

Limit ranges set defaults and limits for individual resources within a namespace:

yaml

apiVersion: v1
kind: LimitRange
metadata:
  name: limit-range
  namespace: dev
spec:
  limits:
  - default:
      cpu: 500m
      memory: 512Mi
    defaultRequest:
      cpu: 200m
      memory: 256Mi
    max:
      cpu: 1
      memory: 1Gi
    min:
      cpu: 100m
      memory: 128Mi
    type: Container

In the above example, all containers in the dev namespace have default limits of 500m CPU and 512Mi memory, default requests of 200m CPU and 256Mi memory, maximum of 1 CPU and 1Gi memory, and minimum of 100m CPU and 128Mi memory.

Horizontal Pod Autoscaler (HPA)

HPA automatically adjusts the number of pods based on CPU usage or custom metrics:

yaml

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: frontend-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: frontend
  minReplicas: 2
  maxReplicas: 10
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 80

In the above example, the frontend deployment automatically scales out when CPU utilization exceeds 80% and scales in when below 80%. It maintains a minimum of 2 and maximum of 10 replicas.

Vertical Pod Autoscaler (VPA)

VPA automatically adjusts pod CPU and memory requests:

yaml

apiVersion: autoscaling.k8s.io/v1
kind: VerticalPodAutoscaler
metadata:
  name: frontend-vpa
spec:
  targetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: frontend
  updatePolicy:
    updateMode: "Auto"

In the above example, pods in the frontend deployment have their CPU and memory requests automatically adjusted based on actual resource usage.

Cluster Networking

Kubernetes cluster networking manages communication between pods, services, and nodes.

Cluster Network Model

Basic requirements of the Kubernetes network model:

All pods can communicate with all other pods without NAT
Node agents (kubelet) can communicate with all pods on that node
Pods running in NAT mode can communicate with the outside

The following diagram shows Kubernetes networking components and communication flows:

CNI (Container Network Interface) Plugins

Kubernetes implements networking through CNI plugins. Common CNI plugins:

Calico: CNI with enhanced network policy and security features
Flannel: Provides simple overlay networking
Cilium: eBPF-based networking and security solution
AWS VPC CNI: CNI integrated with AWS VPC
Weave Net: Multi-host container networking solution

CNI Plugin Installation and Configuration

CNI plugin installation example (Calico):

bash

# Install Calico
kubectl apply -f https://docs.projectcalico.org/manifests/calico.yaml

# Check Calico status
kubectl get pods -n kube-system -l k8s-app=calico-node

Service Networking

Kubernetes services provide stable endpoints for pod sets:

ClusterIP: Service accessible only within the cluster
NodePort: Service accessible through a specific port on all nodes
LoadBalancer: Service accessible through an external load balancer
ExternalName: Provides CNAME record for external services

Service CIDR Configuration

Service CIDR defines the service IP address range:

bash

# Set service CIDR in kube-apiserver configuration
--service-cluster-ip-range=10.96.0.0/12

CoreDNS Management

CoreDNS provides DNS services for Kubernetes:

bash

# Check CoreDNS status
kubectl get pods -n kube-system -l k8s-app=kube-dns

# Check CoreDNS configuration
kubectl get configmap -n kube-system coredns -o yaml

CoreDNS configuration example:

yaml

apiVersion: v1
kind: ConfigMap
metadata:
  name: coredns
  namespace: kube-system
data:
  Corefile: |
    .:53 {
        errors
        health {
           lameduck 5s
        }
        ready
        kubernetes cluster.local in-addr.arpa ip6.arpa {
           pods insecure
           fallthrough in-addr.arpa ip6.arpa
           ttl 30
        }
        prometheus :9153
        forward . /etc/resolv.conf
        cache 30
        loop
        reload
        loadbalance
    }

Network Policies

Network policies control communication between pods:

yaml

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: db-network-policy
  namespace: default
spec:
  podSelector:
    matchLabels:
      role: db
  policyTypes:
  - Ingress
  - Egress
  ingress:
  - from:
    - podSelector:
        matchLabels:
          role: frontend
    ports:
    - protocol: TCP
      port: 3306
  egress:
  - to:
    - podSelector:
        matchLabels:
          role: monitoring
    ports:
    - protocol: TCP
      port: 9090

In the above example, pods with the role=db label only allow TCP port 3306 inbound traffic from pods with the role=frontend label and TCP port 9090 outbound traffic to pods with the role=monitoring label.

Authentication and Authorization Management

Kubernetes authentication and authorization management are core elements of cluster security.

The following diagram shows the Kubernetes authentication and authorization flow:

Authentication

Kubernetes supports various authentication methods:

X.509 Certificates: Authentication using client certificates
Service Account Tokens: JWT tokens associated with service accounts
OpenID Connect (OIDC): Authentication through external identity providers
Webhook Token Authentication: Token verification through external services
Authentication Proxy: Request processing through authentication proxy

X.509 Certificate Management

X.509 certificate creation and management:

bash

# Create Certificate Signing Request (CSR)
openssl req -new -key user.key -out user.csr -subj "/CN=user/O=group"

# Submit CSR to Kubernetes
cat <<EOF | kubectl apply -f -
apiVersion: certificates.k8s.io/v1
kind: CertificateSigningRequest
metadata:
  name: user-csr
spec:
  request: $(cat user.csr | base64 | tr -d '\n')
  signerName: kubernetes.io/kube-apiserver-client
  usages:
  - client auth
EOF

# Approve CSR
kubectl certificate approve user-csr

# Get certificate
kubectl get csr user-csr -o jsonpath='{.status.certificate}' | base64 --decode > user.crt

OIDC Authentication Configuration

OIDC authentication configuration example:

bash

# Add OIDC flags to kube-apiserver configuration
--oidc-issuer-url=https://accounts.google.com
--oidc-client-id=kubernetes
--oidc-username-claim=email
--oidc-groups-claim=groups

Authorization

Kubernetes supports various authorization modes:

RBAC (Role-Based Access Control): Role-based access control
ABAC (Attribute-Based Access Control): Attribute-based access control
Node: Node authorization
Webhook: Authorization through external services

RBAC Configuration

RBAC is the most common authorization mechanism:

yaml

# Role example
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  namespace: default
  name: pod-reader
rules:
- apiGroups: [""]
  resources: ["pods"]
  verbs: ["get", "watch", "list"]

# RoleBinding example
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: read-pods
  namespace: default
subjects:
- kind: User
  name: user
  apiGroup: rbac.authorization.k8s.io
roleRef:
  kind: Role
  name: pod-reader
  apiGroup: rbac.authorization.k8s.io

In the above example, user has permission to view pods in the default namespace.

ClusterRole and ClusterRoleBinding

Manages permissions for cluster-wide resources:

yaml

# ClusterRole example
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: node-reader
rules:
- apiGroups: [""]
  resources: ["nodes"]
  verbs: ["get", "watch", "list"]

# ClusterRoleBinding example
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: read-nodes
subjects:
- kind: User
  name: user
  apiGroup: rbac.authorization.k8s.io
roleRef:
  kind: ClusterRole
  name: node-reader
  apiGroup: rbac.authorization.k8s.io

In the above example, user has permission to view all nodes in the cluster.

Service Account Management

Service accounts are used by pods to communicate with the API server:

yaml

# Create service account
apiVersion: v1
kind: ServiceAccount
metadata:
  name: my-service-account
  namespace: default

# Grant permissions to service account
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: my-service-account-binding
  namespace: default
subjects:
- kind: ServiceAccount
  name: my-service-account
  namespace: default
roleRef:
  kind: Role
  name: pod-reader
  apiGroup: rbac.authorization.k8s.io

# Use service account in pod
apiVersion: v1
kind: Pod
metadata:
  name: my-pod
spec:
  serviceAccountName: my-service-account
  containers:
  - name: my-container
    image: nginx

Security Context

Security context defines permissions and access control for pods and containers:

yaml

apiVersion: v1
kind: Pod
metadata:
  name: security-context-pod
spec:
  securityContext:
    runAsUser: 1000
    runAsGroup: 3000
    fsGroup: 2000
  containers:
  - name: security-context-container
    image: nginx
    securityContext:
      allowPrivilegeEscalation: false
      capabilities:
        drop:
        - ALL
      readOnlyRootFilesystem: true

In the above example, the pod runs with UID 1000 and GID 3000, and the container cannot escalate privileges, has all Linux capabilities dropped, and has the root filesystem mounted as read-only.

Cluster Upgrades

Kubernetes cluster upgrades are necessary to apply new features, performance improvements, and security patches.

The following diagram shows the Kubernetes cluster upgrade process:

Upgrade Planning

Considerations when planning cluster upgrades:

Version Compatibility: Check compatibility between Kubernetes versions
Upgrade Path: Check supported upgrade paths
Downtime: Plan for expected downtime during upgrade
Rollback Plan: Develop a rollback plan in case of issues
Application Impact: Assess the impact of upgrades on applications

Control Plane Upgrade

Control plane upgrade using kubeadm:

bash

# Check upgrade plan
kubeadm upgrade plan

# Upgrade first control plane node
ssh control-plane-1
sudo apt-get update
sudo apt-get install -y kubeadm=1.22.0-00
sudo kubeadm upgrade apply v1.22.0

# Upgrade additional control plane nodes
ssh control-plane-2
sudo apt-get update
sudo apt-get install -y kubeadm=1.22.0-00
sudo kubeadm upgrade node

# Upgrade kubelet and kubectl
sudo apt-get install -y kubelet=1.22.0-00 kubectl=1.22.0-00
sudo systemctl daemon-reload
sudo systemctl restart kubelet

Worker Node Upgrade

Worker node upgrade process:

bash

# Drain node
kubectl drain <node-name> --ignore-daemonsets --delete-emptydir-data

# SSH to node
ssh <node-name>

# Upgrade kubeadm
sudo apt-get update
sudo apt-get install -y kubeadm=1.22.0-00
sudo kubeadm upgrade node

# Upgrade kubelet and kubectl
sudo apt-get install -y kubelet=1.22.0-00 kubectl=1.22.0-00
sudo systemctl daemon-reload
sudo systemctl restart kubelet

# Uncordon node
kubectl uncordon <node-name>

Upgrade Verification

Verify cluster status after upgrade:

bash

# Check node versions
kubectl get nodes

# Check component status
kubectl get componentstatuses

# Check pod status
kubectl get pods --all-namespaces

# Test cluster functionality
kubectl create deployment nginx --image=nginx
kubectl expose deployment nginx --port=80
kubectl get svc nginx

Backup and Recovery

Kubernetes cluster backup and recovery is an important part of disaster recovery planning.

The following diagram shows the Kubernetes cluster backup and recovery process:

etcd Backup

etcd stores all state information for the Kubernetes cluster, so regular backups are important:

bash

# Create etcd snapshot
ETCDCTL_API=3 etcdctl --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key \
  snapshot save /backup/etcd-snapshot-$(date +%Y-%m-%d-%H-%M-%S).db

# Check snapshot status
ETCDCTL_API=3 etcdctl --write-out=table snapshot status /backup/etcd-snapshot-2023-01-01-12-00-00.db

etcd Recovery

Restore from etcd snapshot:

bash

# Stop all Kubernetes services
sudo systemctl stop kubelet kube-apiserver kube-controller-manager kube-scheduler

# Backup etcd data directory
sudo mv /var/lib/etcd /var/lib/etcd.bak

# Restore from snapshot
ETCDCTL_API=3 etcdctl --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key \
  --data-dir=/var/lib/etcd \
  --initial-cluster=master-1=https://192.168.1.10:2380 \
  --initial-cluster-token=etcd-cluster-1 \
  --initial-advertise-peer-urls=https://192.168.1.10:2380 \
  snapshot restore /backup/etcd-snapshot-2023-01-01-12-00-00.db

# Set permissions
sudo chown -R etcd:etcd /var/lib/etcd

# Restart Kubernetes services
sudo systemctl start etcd
sudo systemctl start kubelet kube-apiserver kube-controller-manager kube-scheduler

Resource Backup

Backup Kubernetes resources as YAML files:

bash

# Backup all resources in all namespaces
for ns in $(kubectl get ns -o jsonpath='{.items[*].metadata.name}'); do
  mkdir -p /backup/resources/$ns
  for resource in $(kubectl api-resources --namespaced=true -o name); do
    kubectl get -n $ns $resource -o yaml > /backup/resources/$ns/$resource.yaml
  done
done

# Backup cluster-scoped resources
mkdir -p /backup/resources/cluster-scoped
for resource in $(kubectl api-resources --namespaced=false -o name); do
  kubectl get $resource -o yaml > /backup/resources/cluster-scoped/$resource.yaml
done

Backup Automation

Automate backup tasks with CronJob:

yaml

apiVersion: batch/v1
kind: CronJob
metadata:
  name: etcd-backup
  namespace: kube-system
spec:
  schedule: "0 0 * * *"  # Run daily at midnight
  jobTemplate:
    spec:
      template:
        spec:
          containers:
          - name: etcd-backup
            image: bitnami/etcd:latest
            command:
            - /bin/sh
            - -c
            - |
              ETCDCTL_API=3 etcdctl --endpoints=https://etcd-client:2379 \
                --cacert=/etc/kubernetes/pki/etcd/ca.crt \
                --cert=/etc/kubernetes/pki/etcd/server.crt \
                --key=/etc/kubernetes/pki/etcd/server.key \
                snapshot save /backup/etcd-snapshot-$(date +%Y-%m-%d-%H-%M-%S).db
            volumeMounts:
            - name: etcd-certs
              mountPath: /etc/kubernetes/pki/etcd
              readOnly: true
            - name: backup
              mountPath: /backup
          restartPolicy: OnFailure
          volumes:
          - name: etcd-certs
            hostPath:
              path: /etc/kubernetes/pki/etcd
              type: Directory
          - name: backup
            persistentVolumeClaim:
              claimName: etcd-backup-pvc

Monitoring and Logging

Effective monitoring and logging is a core element of cluster administration.

The following diagram shows the Kubernetes cluster monitoring and logging architecture:

Monitoring Tools

Tools for Kubernetes cluster monitoring:

Prometheus: Metric collection and storage
Grafana: Metric visualization
Alertmanager: Alert management
kube-state-metrics: Generate Kubernetes object metrics
metrics-server: Provide resource usage metrics

Prometheus and Grafana Installation

Install Prometheus and Grafana using Helm:

bash

# Add Helm repository
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update

# Install Prometheus stack
helm install prometheus prometheus-community/kube-prometheus-stack \
  --namespace monitoring \
  --create-namespace

Key Monitoring Metrics

Key metrics to monitor:

Node Metrics: CPU, memory, disk, network usage
Pod Metrics: CPU, memory usage, restart count
Container Metrics: CPU, memory usage, filesystem usage
API Server Metrics: Request latency, request count, error rate
etcd Metrics: Disk I/O, leader changes, commit latency

Logging Tools

Tools for Kubernetes cluster logging:

Elasticsearch: Log storage and search
Fluentd/Fluent Bit: Log collection and forwarding
Kibana: Log visualization
Loki: Log aggregation system
Grafana: Log visualization

EFK (Elasticsearch, Fluentd, Kibana) Stack Installation

Install EFK stack using Helm:

bash

# Install Elasticsearch
helm install elasticsearch elastic/elasticsearch \
  --namespace logging \
  --create-namespace

# Install Fluentd
helm install fluentd fluent/fluentd \
  --namespace logging

# Install Kibana
helm install kibana elastic/kibana \
  --namespace logging \
  --set service.type=LoadBalancer

Log Collection Configuration

Fluentd configuration example:

yaml

apiVersion: v1
kind: ConfigMap
metadata:
  name: fluentd-config
  namespace: logging
data:
  fluent.conf: |
    <source>
      @type tail
      path /var/log/containers/*.log
      pos_file /var/log/fluentd-containers.log.pos
      tag kubernetes.*
      read_from_head true
      <parse>
        @type json
        time_format %Y-%m-%dT%H:%M:%S.%NZ
      </parse>
    </source>

    <filter kubernetes.**>
      @type kubernetes_metadata
      kubernetes_url https://kubernetes.default.svc
      bearer_token_file /var/run/secrets/kubernetes.io/serviceaccount/token
      ca_file /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
    </filter>

    <match kubernetes.**>
      @type elasticsearch
      host elasticsearch-master
      port 9200
      logstash_format true
      logstash_prefix k8s
    </match>

Troubleshooting

Kubernetes cluster troubleshooting is an important part of cluster administration.

Pod Troubleshooting

Commands for pod troubleshooting:

bash

# Check pod status
kubectl get pod <pod-name> -o wide

# Check pod detailed information
kubectl describe pod <pod-name>

# Check pod logs
kubectl logs <pod-name>
kubectl logs <pod-name> -c <container-name>  # For multi-container pods
kubectl logs <pod-name> --previous  # Logs from previous container

# Execute command in pod
kubectl exec -it <pod-name> -- /bin/sh

Node Troubleshooting

Commands for node troubleshooting:

bash

# Check node status
kubectl get node <node-name> -o wide

# Check node detailed information
kubectl describe node <node-name>

# Check node resource usage
kubectl top node <node-name>

# SSH to node
ssh <node-name>

# Check node system logs
journalctl -u kubelet

# Check node resource usage
top
df -h
free -m

Networking Troubleshooting

Commands for networking troubleshooting:

bash

# Check service status
kubectl get svc <service-name>

# Check service detailed information
kubectl describe svc <service-name>

# Check endpoints
kubectl get endpoints <service-name>

# Check DNS
kubectl run -it --rm --restart=Never busybox --image=busybox -- nslookup <service-name>

# Test network connectivity
kubectl run -it --rm --restart=Never busybox --image=busybox -- wget -O- <service-name>:<port>

# Check network policies
kubectl get networkpolicy
kubectl describe networkpolicy <policy-name>

Control Plane Troubleshooting

Commands for control plane troubleshooting:

bash

# Check component status
kubectl get componentstatuses

# Check API server logs
kubectl logs -n kube-system kube-apiserver-<node-name>

# Check controller manager logs
kubectl logs -n kube-system kube-controller-manager-<node-name>

# Check scheduler logs
kubectl logs -n kube-system kube-scheduler-<node-name>

# Check etcd logs
kubectl logs -n kube-system etcd-<node-name>

Amazon EKS Cluster Administration

Amazon EKS is a managed Kubernetes service that automates many aspects of cluster administration.

The following diagram shows the Amazon EKS cluster architecture and management components:

EKS Cluster Configuration

EKS cluster configuration management:

bash

# Check EKS cluster information
aws eks describe-cluster --name my-cluster

# Update EKS cluster
aws eks update-cluster-config \
  --name my-cluster \
  --resources-vpc-config endpointPublicAccess=true,endpointPrivateAccess=true

# Update EKS cluster version
aws eks update-cluster-version \
  --name my-cluster \
  --kubernetes-version 1.22

EKS Node Group Management

EKS node group management:

bash

# Check node group information
aws eks describe-nodegroup \
  --cluster-name my-cluster \
  --nodegroup-name my-nodegroup

# Scale node group
aws eks update-nodegroup-config \
  --cluster-name my-cluster \
  --nodegroup-name my-nodegroup \
  --scaling-config minSize=2,maxSize=10,desiredSize=5

# Update node group
aws eks update-nodegroup-version \
  --cluster-name my-cluster \
  --nodegroup-name my-nodegroup

EKS Add-on Management

EKS add-on management:

bash

# Check available add-ons
aws eks describe-addon-versions \
  --kubernetes-version 1.22

# Install add-on
aws eks create-addon \
  --cluster-name my-cluster \
  --addon-name vpc-cni \
  --addon-version v1.10.1-eksbuild.1

# Update add-on
aws eks update-addon \
  --cluster-name my-cluster \
  --addon-name vpc-cni \
  --addon-version v1.10.2-eksbuild.1

# Delete add-on
aws eks delete-addon \
  --cluster-name my-cluster \
  --addon-name vpc-cni

EKS Cluster Upgrade

EKS cluster upgrade process:

Control Plane Upgrade:

bash

aws eks update-cluster-version \
  --name my-cluster \
  --kubernetes-version 1.22

Add-on Upgrade:

bash

aws eks update-addon \
  --cluster-name my-cluster \
  --addon-name vpc-cni \
  --addon-version v1.10.2-eksbuild.1

Node Group Upgrade:

bash

aws eks update-nodegroup-version \
  --cluster-name my-cluster \
  --nodegroup-name my-nodegroup

EKS Cluster Monitoring

EKS cluster monitoring tools:

Amazon CloudWatch: Metrics, logs, alerts
AWS CloudTrail: API call logging
Amazon Managed Grafana: Metric visualization
Amazon Managed Service for Prometheus: Metric collection and storage

Enable CloudWatch Container Insights:

bash

# Enable Container Insights
eksctl utils update-cluster-logging \
  --enable-types all \
  --cluster my-cluster \
  --approve

Cluster Administration Best Practices

Best practices for Kubernetes and EKS cluster administration:

Cluster Configuration Best Practices

Infrastructure as Code (IaC): Manage cluster configuration using Terraform, AWS CDK, eksctl, etc.
Version Control: Store cluster configuration in version control systems
Multiple Environments: Separate development, staging, and production environments
Network Separation: Configure appropriate network separation and security groups
Least Privilege Principle: Grant only the minimum necessary permissions

Operations Best Practices

Regular Backups: Regular backup of etcd and important resources
Monitoring and Alerting: Build comprehensive monitoring and alerting systems
Centralized Logging: Centralize and analyze logs
Automation: Automate repetitive tasks
Disaster Recovery Planning: Establish and test clear disaster recovery plans

Security Best Practices

Regular Updates: Regular updates of cluster and nodes
Network Policies: Configure appropriate network policies
Encryption: Encrypt data at rest and in transit
Security Context: Configure appropriate security contexts
Image Scanning: Scan container images for vulnerabilities

Resource Management Best Practices

Resource Requests and Limits: Set appropriate resource requests and limits for all pods
Namespace Separation: Separate workloads by namespace
Resource Quotas: Set resource quotas per namespace
HPA and VPA: Configure autoscaling
Node Affinity and Taints: Optimize workload placement

EKS-Specific Best Practices

Managed Node Groups: Use managed node groups when possible
Fargate: Use Fargate for serverless workloads
EKS Add-ons: Use official EKS add-ons
IAM Roles for Service Accounts (IRSA): Manage IAM permissions per pod
VPC CNI Customization: Configure VPC CNI according to networking requirements

Conclusion

Kubernetes cluster administration plays an important role in maintaining cluster stability, security, and performance. This chapter covered various aspects of cluster administration including cluster component management, resource management, networking, authentication and authorization management, upgrades, backup and recovery, monitoring and logging, and troubleshooting.

Using Amazon EKS reduces the complexity of Kubernetes control plane management and simplifies cluster administration through integration with AWS services. However, understanding fundamental Kubernetes concepts and best practices is still important for effective cluster management.

Cluster administration is an ongoing process that must be continuously adjusted according to cluster requirements and workload characteristics. It is important to use monitoring tools to track cluster status, minimize repetitive tasks through automation, and follow best practices to maintain cluster stability and security.

Cluster Networking

Kubernetes cluster networking manages pod-to-pod communication, service discovery, and external access.

Network Architecture

CNI Plugin Management

CNI (Container Network Interface) plugins handle networking for Kubernetes clusters.

bash

# Install Calico CNI
kubectl apply -f https://docs.projectcalico.org/manifests/calico.yaml

# Install Flannel CNI
kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml

# Install Cilium CNI (using Helm)
helm repo add cilium https://helm.cilium.io/
helm install cilium cilium/cilium --version 1.14.0 --namespace kube-system

CNI Plugin Comparison

CNI Plugin	Network Model	Network Policy Support	Performance	Features
Calico	BGP	Yes	High	Strong in network policies, routing-based
Flannel	VXLAN/host-gateway	No	Medium	Simple setup, limited features
Cilium	eBPF	Yes	Very High	L3-L7 policies, high performance
Weave Net	VXLAN	Yes	Medium	Encryption support, multi-cluster
AWS VPC CNI	AWS VPC	No	High	Optimized for AWS EKS

Network Troubleshooting

bash

# Test pod network connectivity
kubectl run -it --rm network-test --image=busybox -- sh
# Inside the container
ping <target-ip>
traceroute <target-ip>
wget -O- <service-name>

# DNS troubleshooting
kubectl run -it --rm dns-test --image=busybox -- sh
# Inside the container
nslookup kubernetes.default.svc.cluster.local
cat /etc/resolv.conf

# Check service endpoints
kubectl get endpoints <service-name>

# Check network policies
kubectl describe networkpolicy -n <namespace>

Authentication and Authorization Management

Kubernetes authentication and authorization management are core elements of cluster security. RBAC (Role-Based Access Control) is used to manage permissions for users and service accounts.

Authentication Methods

Kubernetes supports various authentication methods:

X.509 Certificates: Authentication using client certificates
Service Account Tokens: Used for API server access within pods
OpenID Connect (OIDC): Integration with external identity providers
Webhook Token Authentication: Integration with external authentication services
Authentication Proxy: Authentication through proxy

RBAC Configuration

yaml

# role.yaml - namespace-scoped role
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  namespace: default
  name: pod-reader
rules:
- apiGroups: [""]
  resources: ["pods"]
  verbs: ["get", "watch", "list"]

yaml

# rolebinding.yaml - binding role to user
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: read-pods
  namespace: default
subjects:
- kind: User
  name: jane
  apiGroup: rbac.authorization.k8s.io
roleRef:
  kind: Role
  name: pod-reader
  apiGroup: rbac.authorization.k8s.io

yaml

# clusterrole.yaml - cluster-scoped role
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: secret-reader
rules:
- apiGroups: [""]
  resources: ["secrets"]
  verbs: ["get", "watch", "list"]

yaml

# clusterrolebinding.yaml - binding cluster role to user
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: read-secrets-global
subjects:
- kind: Group
  name: manager
  apiGroup: rbac.authorization.k8s.io
roleRef:
  kind: ClusterRole
  name: secret-reader
  apiGroup: rbac.authorization.k8s.io

User Certificate Creation

bash

# Generate private key
openssl genrsa -out jane.key 2048

# Create Certificate Signing Request (CSR)
openssl req -new -key jane.key -out jane.csr -subj "/CN=jane/O=dev"

# Sign certificate with Kubernetes CA
sudo openssl x509 -req -in jane.csr \
  -CA /etc/kubernetes/pki/ca.crt \
  -CAkey /etc/kubernetes/pki/ca.key \
  -CAcreateserial \
  -out jane.crt -days 365

# Add user to kubeconfig
kubectl config set-credentials jane --client-certificate=jane.crt --client-key=jane.key
kubectl config set-context jane-context --cluster=kubernetes --user=jane

Service Account Management

bash

# Create service account
kubectl create serviceaccount app-service-account

# Bind role to service account
kubectl create rolebinding app-service-account-binding \
  --role=pod-reader \
  --serviceaccount=default:app-service-account

# Check service account token
kubectl describe serviceaccount app-service-account

Permission Verification

bash

# Check user permissions
kubectl auth can-i get pods --as jane

# Check permissions in a specific namespace
kubectl auth can-i create deployments --as jane --namespace production

Cluster Upgrades

Kubernetes cluster upgrades are necessary to apply new features, security patches, and bug fixes. Upgrades must be carefully planned and executed.

Upgrade Planning

Upgrade Strategy Comparison

Strategy	Description	Advantages	Disadvantages	Suitable Environment
In-place Upgrade	Directly upgrade existing cluster	Resource efficient, simple procedure	Complex rollback, potential downtime	Development, test environments
Blue/Green Deployment	Create new version cluster and switch	Safe rollback, verifiable	Resource duplication, increased cost	Production environments
Canary Deployment	Move only some workloads to new cluster	Gradual verification, reduced risk	Complex management, dual operation	Critical production environments

Upgrade Using kubeadm

bash

# Check current version
kubeadm version

# Check upgrade plan
sudo kubeadm upgrade plan

# Control plane upgrade
sudo apt-get update
sudo apt-get install -y kubeadm=1.33.3-00
sudo kubeadm upgrade apply v1.33.3

# kubelet upgrade
sudo apt-get install -y kubelet=1.33.3-00 kubectl=1.33.3-00
sudo systemctl daemon-reload
sudo systemctl restart kubelet

# Worker node upgrade (on each node)
# 1. Drain node
kubectl drain <node-name> --ignore-daemonsets

# 2. kubeadm upgrade
sudo apt-get update
sudo apt-get install -y kubeadm=1.33.3-00
sudo kubeadm upgrade node

# 3. kubelet upgrade
sudo apt-get install -y kubelet=1.33.3-00 kubectl=1.33.3-00
sudo systemctl daemon-reload
sudo systemctl restart kubelet

# 4. Uncordon node
kubectl uncordon <node-name>

Post-Upgrade Verification

bash

# Check cluster version
kubectl version

# Check node versions
kubectl get nodes

# Check component status
kubectl get componentstatuses

# Check workload status
kubectl get pods -A

Backup and Recovery

Kubernetes cluster backup and recovery is an important part of disaster recovery planning. Main backup targets are the etcd database, persistent volume data, and Kubernetes resource definitions.

etcd Backup and Recovery

etcd is a core component that stores all state information for the cluster.

bash

# etcd backup
ETCDCTL_API=3 etcdctl --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key \
  snapshot save /backup/etcd-snapshot-$(date +%Y-%m-%d).db

# etcd recovery
# 1. Stop cluster
sudo systemctl stop kubelet
sudo docker stop $(docker ps -q)

# 2. Restore etcd data
ETCDCTL_API=3 etcdctl --endpoints=https://127.0.0.1:2379 \
  snapshot restore /backup/etcd-snapshot-2025-11-24.db \
  --data-dir=/var/lib/etcd-restore \
  --name=master \
  --initial-cluster=master=https://127.0.0.1:2380 \
  --initial-cluster-token=etcd-cluster-1 \
  --initial-advertise-peer-urls=https://127.0.0.1:2380

# 3. Configure to use restored data directory
sudo mv /var/lib/etcd /var/lib/etcd.bak
sudo mv /var/lib/etcd-restore /var/lib/etcd

# 4. Restart cluster
sudo systemctl start kubelet

Kubernetes Resource Backup

bash

# Backup all resources in all namespaces
mkdir -p /backup/resources/$(date +%Y-%m-%d)
for ns in $(kubectl get ns -o jsonpath='{.items[*].metadata.name}'); do
  kubectl -n $ns get all -o yaml > /backup/resources/$(date +%Y-%m-%d)/$ns-all.yaml
done

# Backup specific resource types
for resource in deployments services configmaps secrets; do
  kubectl get $resource -A -o yaml > /backup/resources/$(date +%Y-%m-%d)/$resource.yaml
done

Backup and Recovery Using Velero

Velero is a tool for backing up and recovering Kubernetes cluster resources and persistent volumes.

bash

# Install Velero (using AWS S3 backup storage)
velero install \
  --provider aws \
  --plugins velero/velero-plugin-for-aws:v1.7.0 \
  --bucket velero-backup \
  --backup-location-config region=us-west-2 \
  --snapshot-location-config region=us-west-2 \
  --secret-file ./credentials-velero

# Full cluster backup
velero backup create full-cluster-backup --include-namespaces '*'

# Backup specific namespace
velero backup create production-backup --include-namespaces production

# Check backup status
velero backup describe full-cluster-backup

# Restore from backup
velero restore create --from-backup full-cluster-backup

Backup Strategy Comparison

Backup Method	Backup Target	Advantages	Disadvantages	Recovery Time
etcd Snapshot	Cluster state	Built-in feature, complete state preservation	Volume data not included, manual process	Medium
Resource YAML Backup	Kubernetes objects	Simple implementation, selective restore	Volume data not included, relationship complexity	Slow
Velero	Resources and volumes	Automation, scheduling, volume snapshots	Additional tool installation required	Fast
Cloud Provider Snapshots	Entire cluster	Complete recovery, cloud integration	Cloud dependency, cost	Very Fast

Monitoring and Logging

Effective cluster management requires a comprehensive monitoring and logging system. This allows problems to be detected and resolved early.

Monitoring Architecture

Prometheus and Grafana Installation

bash

# Install Prometheus and Grafana using Helm
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update

helm install prometheus prometheus-community/kube-prometheus-stack \
  --namespace monitoring \
  --create-namespace \
  --set grafana.enabled=true \
  --set prometheus.service.type=NodePort

# Check services
kubectl get svc -n monitoring

# Access Grafana (using port forwarding)
kubectl port-forward svc/prometheus-grafana 3000:80 -n monitoring
# Default username: admin, default password: prom-operator

EFK Stack Installation (Elasticsearch, Fluentd, Kibana)

bash

# Install Elasticsearch and Kibana
helm repo add elastic https://helm.elastic.co
helm repo update

helm install elasticsearch elastic/elasticsearch \
  --namespace logging \
  --create-namespace \
  --set replicas=1 \
  --set minimumMasterNodes=1

helm install kibana elastic/kibana \
  --namespace logging \
  --set service.type=NodePort

# Install Fluentd
kubectl apply -f https://raw.githubusercontent.com/fluent/fluentd-kubernetes-daemonset/master/fluentd-daemonset-elasticsearch.yaml

Key Monitoring Metrics

Metric Type	Description	Key Metrics	Monitoring Tools
Node Metrics	Node-level resource usage	CPU, memory, disk, network	node-exporter, Prometheus
Pod Metrics	Container resource usage	CPU, memory usage, limits	cAdvisor, Prometheus
Cluster Metrics	Cluster state and resources	Pod count, node status, events	kube-state-metrics
Application Metrics	Custom application metrics	Request count, latency, error rate	Prometheus client libraries

Log Collection and Analysis

bash

# Check logs for a specific pod
kubectl logs <pod-name> -n <namespace>

# Check logs from previous instance
kubectl logs <pod-name> -n <namespace> --previous

# Check logs for a specific container (multi-container pod)
kubectl logs <pod-name> -c <container-name> -n <namespace>

# Stream logs
kubectl logs -f <pod-name> -n <namespace>

# Check logs for all pods (using label selector)
kubectl logs -l app=nginx -n <namespace>

Alert Configuration

You can configure alerts using Prometheus Alertmanager:

yaml

# alertmanager-config.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: alertmanager-config
  namespace: monitoring
data:
  alertmanager.yml: |
    global:
      resolve_timeout: 5m
      slack_api_url: 'https://hooks.slack.com/services/T00000000/B00000000/XXXXXXXXXXXXXXXXXXXXXXXX'

    route:
      receiver: 'slack-notifications'
      group_wait: 30s
      group_interval: 5m
      repeat_interval: 4h
      group_by: ['alertname', 'cluster', 'service']

    receivers:
    - name: 'slack-notifications'
      slack_configs:
      - channel: '#alerts'
        send_resolved: true
        title: "{{ range .Alerts }}{{ .Annotations.summary }}\n{{ end }}"
        text: "{{ range .Alerts }}{{ .Annotations.description }}\n{{ end }}"

Troubleshooting

Kubernetes cluster troubleshooting is an important skill for system administrators and operators. A systematic approach is required for effective troubleshooting.

Troubleshooting Methodology

Common Problems and Solutions

Problem Type	Symptoms	Diagnostic Commands	Common Solutions
Pod Not Starting	Pod in Pending or ContainerCreating state	`kubectl describe pod <pod-name>`	Check resource constraints, image availability, volume mounts
Service Connection Issues	Cannot access pods through service	`kubectl describe svc <service-name>`, `kubectl get endpoints <service-name>`	Check label selectors, pod status, network policies
Node Issues	Node in NotReady state	`kubectl describe node <node-name>`, `kubectl get events`	Check kubelet status, system resources, network connectivity
DNS Issues	Cannot connect by service name	`kubectl exec -it <pod-name> -- nslookup kubernetes.default`	Check CoreDNS pods, kube-dns service, network policies
Authentication Issues	API server access denied	`kubectl auth can-i <verb> <resource>`	Check RBAC settings, certificate validity, service account

Pod Troubleshooting

bash

# Check pod status
kubectl get pod <pod-name> -o wide

# Check pod details
kubectl describe pod <pod-name>

# Check pod logs
kubectl logs <pod-name>
kubectl logs <pod-name> --previous  # Logs from previous container

# Execute command in pod
kubectl exec -it <pod-name> -- /bin/sh

# Check pod events
kubectl get events --field-selector involvedObject.name=<pod-name>

Node Troubleshooting

bash

# Check node status
kubectl get nodes
kubectl describe node <node-name>

# Check node resource usage
kubectl top node <node-name>

# Check node system logs (SSH required)
ssh <node-ip> 'sudo journalctl -u kubelet'

# Check kubelet status (SSH required)
ssh <node-ip> 'sudo systemctl status kubelet'

Networking Troubleshooting

bash

# Check service and endpoints
kubectl get svc <service-name>
kubectl get endpoints <service-name>

# DNS troubleshooting
kubectl run -it --rm dns-test --image=busybox -- sh
# Inside the container
nslookup kubernetes.default.svc.cluster.local
cat /etc/resolv.conf

# Network connectivity test
kubectl run -it --rm network-test --image=nicolaka/netshoot -- sh
# Inside the container
ping <target-ip>
traceroute <target-ip>
curl <service-name>:<port>

Amazon EKS Cluster Administration

Amazon EKS (Elastic Kubernetes Service) is a managed Kubernetes service on AWS where AWS manages the control plane. However, management of nodes, networking, security, etc. is the user's responsibility.

EKS Cluster Architecture

EKS Cluster Creation

bash

# Create cluster using eksctl
eksctl create cluster \
  --name my-cluster \
  --version 1.33 \
  --region us-west-2 \
  --nodegroup-name standard-workers \
  --node-type t3.medium \
  --nodes 3 \
  --nodes-min 1 \
  --nodes-max 5 \
  --managed

# Create cluster using AWS CLI
aws eks create-cluster \
  --name my-cluster \
  --role-arn arn:aws:iam::123456789012:role/eks-cluster-role \
  --resources-vpc-config subnetIds=subnet-12345,subnet-67890,securityGroupIds=sg-12345

Node Group Management

bash

# Create managed node group
eksctl create nodegroup \
  --cluster my-cluster \
  --region us-west-2 \
  --name my-nodegroup \
  --node-type t3.medium \
  --nodes 3 \
  --nodes-min 1 \
  --nodes-max 5

# Scale node group
eksctl scale nodegroup \
  --cluster my-cluster \
  --name my-nodegroup \
  --nodes 5 \
  --region us-west-2

# Update node group
eksctl update nodegroup \
  --cluster my-cluster \
  --name my-nodegroup \
  --region us-west-2 \
  --max-pods-per-node 110

EKS Cluster Upgrade

bash

# Check cluster version
aws eks describe-cluster --name my-cluster --query "cluster.version"

# Upgrade cluster control plane
aws eks update-cluster-version \
  --name my-cluster \
  --kubernetes-version 1.33

# Upgrade managed node group
aws eks update-nodegroup-version \
  --cluster-name my-cluster \
  --nodegroup-name my-nodegroup

EKS Cluster Authentication and Authorization

bash

# Map IAM user/role to cluster RBAC
eksctl create iamidentitymapping \
  --cluster my-cluster \
  --arn arn:aws:iam::123456789012:role/admin-role \
  --group system:masters \
  --username admin

# Check aws-auth ConfigMap
kubectl describe configmap aws-auth -n kube-system

EKS Cluster Monitoring

bash

# Enable CloudWatch Container Insights
eksctl utils update-cluster-logging \
  --enable-types all \
  --cluster my-cluster \
  --region us-west-2

# Install Prometheus and Grafana (using Amazon EKS add-on)
aws eks create-addon \
  --cluster-name my-cluster \
  --addon-name amazon-cloudwatch-observability \
  --addon-version v1.1.1-eksbuild.1

Cluster Administration Best Practices

Best practices for effective Kubernetes cluster management are important for ensuring stability, security, and performance.

Cluster Setup Best Practices

Multi-Availability Zone Configuration: Distribute nodes across multiple availability zones for high availability
Appropriate Sizing: Select node types and counts appropriate for workloads
Autoscaling Configuration: Enable cluster autoscaler and horizontal pod autoscaler
Apply Network Policies: Start with a default deny policy and allow only necessary communication
Set Resource Quotas: Set resource limits per namespace

Operations Best Practices

Use Declarative Configuration: Define all resources as YAML files and version control them
Adopt GitOps: Use Git as the single source of truth and build automated deployment pipelines
Regular Backups: Regular backup of etcd data and persistent volume data
Monitoring and Alerting: Build comprehensive monitoring systems and set alerts for key metrics
Centralized Logging: Collect all logs to a central logging system for easy analysis

Security Best Practices

Least Privilege Principle: Grant only the minimum necessary permissions using RBAC
Network Segmentation: Limit pod-to-pod communication using network policies
Image Scanning: Implement container image scanning for vulnerability detection
Secret Management: Use external secret management tools (e.g., AWS Secrets Manager, HashiCorp Vault)
Regular Security Audits: Conduct regular audits of cluster configuration and permissions

Upgrade Best Practices

Gradual Upgrades: Upgrade gradually rather than all at once
Test Environment First: Verify upgrades in test environments before production
Create Backups: Perform full backups before upgrades
Rollback Plan: Develop a plan to rollback to previous versions in case of issues
Set Upgrade Windows: Perform upgrades during low-usage periods

Cost Optimization Best Practices

Select Appropriate Node Sizes: Select optimal node types for workloads
Utilize Spot Instances: Use spot instances for non-critical workloads
Configure Autoscaling: Configure automatic scale up and down based on demand
Optimize Resource Requests and Limits: Set resource requests and limits based on actual usage
Identify Idle Resources: Regularly identify and remove idle resources

Documentation Best Practices

Document Architecture: Document cluster architecture, networking, and security settings
Document Operations Procedures: Document common operations tasks, troubleshooting procedures, and emergency response plans
Change Management: Record and track all cluster changes
Create Runbooks: Provide step-by-step guides for common scenarios
Knowledge Sharing: Conduct regular knowledge sharing and training sessions within the team

Conclusion

Kubernetes cluster administration is a complex task that includes various aspects. A systematic approach is required from cluster setup to operation, monitoring, troubleshooting, and upgrades.

For effective cluster administration, focus on the following key areas:

Cluster Component Management: Stable operation of control plane and node components
Resource Management: Efficient resource allocation and usage
Networking: Secure and efficient network configuration
Security: Appropriate authentication and authorization management
Backup and Recovery: Data loss prevention and disaster recovery planning
Monitoring and Logging: Cluster status and performance monitoring
Troubleshooting: Systematic troubleshooting approach

When using managed Kubernetes services like Amazon EKS, it is important to understand the shared responsibility model between the service provider and the user. While AWS manages the control plane, management of nodes, networking, security, etc. is still the user's responsibility.

By following best practices and utilizing appropriate tools, you can operate a stable, secure, and efficient Kubernetes cluster. Continuous learning and improvement to enhance cluster management capabilities is important.

References:
Kubernetes Official Documentation: Cluster Administration
Amazon EKS User Guide
Kubernetes Best Practices: Cluster Administration
etcd Documentation: Backup and Recovery
Prometheus Documentation

Quiz

To test what you learned in this chapter, try the Cluster Administration Quiz.

Traffic Management

Security

Observability

Resilience

Advanced

Comparison Guide

ArgoCD

Metrics

Logging

Tracing

Alerting

Observability End-to-End Labs

Cilium Deep Dive

Calico Deep Dive

Istio Detailed Quiz

Linkerd Detailed Quiz

Cilium Service Mesh Detailed Quiz

ArgoCD Detailed Quiz

Metrics

Logging

Tracing

Alerting

Observability Labs

Kubernetes Cluster Administration ​

Core Concepts ​

Table of Contents ​

Environment Setup ​

Cluster Administration Overview ​

Cluster Component Management ​

Control Plane Component Management ​

API Server Management ​

etcd Management ​

Node Management ​

Component Status Monitoring ​

Cluster Administration Tools ​

Cluster Component Management ​

Control Plane Components ​

Control Plane Component Monitoring ​

Control Plane Component Configuration ​

Node Components ​

Node Management ​

Node Troubleshooting ​

Resource Management ​

Resource Quotas ​

Limit Ranges ​

Horizontal Pod Autoscaler (HPA) ​

Vertical Pod Autoscaler (VPA) ​

Cluster Networking ​

Cluster Network Model ​

CNI (Container Network Interface) Plugins ​

CNI Plugin Installation and Configuration ​

Service Networking ​

Service CIDR Configuration ​

CoreDNS Management ​

Network Policies ​

Authentication and Authorization Management ​

Authentication ​

X.509 Certificate Management ​

OIDC Authentication Configuration ​

Authorization ​

RBAC Configuration ​

ClusterRole and ClusterRoleBinding ​

Service Account Management ​

Security Context ​

Cluster Upgrades ​

Upgrade Planning ​

Control Plane Upgrade ​

Worker Node Upgrade ​

Upgrade Verification ​

Backup and Recovery ​

etcd Backup ​

etcd Recovery ​

Resource Backup ​

Backup Automation ​

Monitoring and Logging ​

Monitoring Tools ​

Prometheus and Grafana Installation ​

Key Monitoring Metrics ​

Logging Tools ​

EFK (Elasticsearch, Fluentd, Kibana) Stack Installation ​

Kubernetes Cluster Administration

Core Concepts

Table of Contents

Environment Setup

Cluster Administration Overview

Cluster Component Management

Control Plane Component Management

API Server Management

etcd Management

Node Management

Component Status Monitoring

Cluster Administration Tools

Cluster Component Management

Control Plane Components

Control Plane Component Monitoring

Control Plane Component Configuration

Node Components

Node Management

Node Troubleshooting

Resource Management

Resource Quotas

Limit Ranges

Horizontal Pod Autoscaler (HPA)

Vertical Pod Autoscaler (VPA)

Cluster Networking

Cluster Network Model

CNI (Container Network Interface) Plugins

CNI Plugin Installation and Configuration

Service Networking

Service CIDR Configuration

CoreDNS Management

Network Policies

Authentication and Authorization Management

Authentication

X.509 Certificate Management

OIDC Authentication Configuration

Authorization

RBAC Configuration

ClusterRole and ClusterRoleBinding

Service Account Management

Security Context

Cluster Upgrades

Upgrade Planning

Control Plane Upgrade

Worker Node Upgrade

Upgrade Verification

Backup and Recovery

etcd Backup

etcd Recovery

Resource Backup

Backup Automation

Monitoring and Logging

Monitoring Tools

Prometheus and Grafana Installation

Key Monitoring Metrics

Logging Tools

EFK (Elasticsearch, Fluentd, Kibana) Stack Installation

Log Collection Configuration

Troubleshooting

Pod Troubleshooting

Node Troubleshooting

Networking Troubleshooting

Control Plane Troubleshooting

Amazon EKS Cluster Administration

EKS Cluster Configuration

EKS Node Group Management

EKS Add-on Management

EKS Cluster Upgrade

EKS Cluster Monitoring

Cluster Administration Best Practices

Cluster Configuration Best Practices