Kubernetes Cluster Administration

Supported Versions: EKS standard support for Kubernetes 1.34–1.37 (checked October 12, 2026) Last Updated: October 12, 2026

Kubernetes cluster administration is an important task that includes cluster setup, maintenance, monitoring, troubleshooting, and upgrades. In this chapter, we will explore various aspects of Kubernetes cluster administration and best practices for cluster management in Amazon EKS.

Self-managed kubeadm operations and EKS service operations are distinct. EKS does not expose control-plane hosts, static Pod manifests, or direct etcd access. Treat the blocks below as separate examples, not one script to run sequentially. Check upstream support and add-on compatibility for the actual cluster version.

Core Concepts

Cluster Lifecycle Management: The entire process from cluster creation to decommissioning
Control Plane Management: Managing core components such as API server, scheduler, and controller manager
Node Management: Adding, removing, and maintaining worker nodes
Resource Allocation: Setting resource allocation and limits for CPU, memory, storage, etc.
Upgrade Strategy: Cluster and application upgrade strategies to minimize downtime

Cluster Administration Overview
Cluster Component Management
Resource Management
Cluster Networking
Authentication and Authorization Management
Cluster Upgrades
Backup and Recovery
Monitoring and Logging
Troubleshooting
Amazon EKS Cluster Administration
Cluster Administration Best Practices
Conclusion

Environment Setup

The following tools are required for cluster administration:

Use the official kubectl installation guide, keeping kubectl within one minor version of the API server. For self-managed clusters, install kubeadm/kubelet from the target minor's pkgs.k8s.io repository; the old 1.x.y-00 package examples are obsolete. Choose an exact package version from the configured repository before installation.

Install Helm and k9s from their official Helm instructions and k9s releases, checking platform architecture and checksums. EKS administration additionally requires an authenticated AWS CLI and a compatible eksctl.

bash

kubectl version --client
helm version
k9s version
# Self-managed nodes only, after configuring the target minor repository:
apt-cache madison kubeadm

Cluster Administration Overview

Kubernetes cluster administration is the process of managing the entire lifecycle of a cluster. This includes the following main areas:

Cluster Setup and Configuration: Cluster creation, node addition, networking setup, storage configuration, etc.
Operations Management: Resource monitoring, performance optimization, capacity planning, troubleshooting
Security Management: Authentication, authorization, network policies, security contexts, etc.
Upgrades and Patches: Cluster version upgrades, security patch application
Backup and Recovery: Cluster data backup, disaster recovery planning

The following diagram shows the main areas of Kubernetes cluster administration and related tools:

Cluster Component Management

A Kubernetes cluster consists of control plane components and node components. Managing each component is critical for cluster stability and performance.

Control Plane Component Management

Tree diagram showing the Kubernetes control plane fanning out to its five components (API server, etcd, scheduler, controller manager, cloud controller manager), each paired with the operational concern it owns: authentication and authorization, data backup, scheduling policies, controller state monitoring, and cloud resource management.

API Server Management

The API server is a core component of the control plane that exposes the Kubernetes API.

bash

# Check API server logs
kubectl logs -n kube-system kube-apiserver-<master-node-name>

# Check API server configuration (kubeadm cluster)
sudo cat /etc/kubernetes/manifests/kube-apiserver.yaml

# Check API server status
kubectl get --raw='/readyz?verbose'

etcd Management

etcd is a distributed key-value store that stores Kubernetes API state.

bash

# etcd backup
ETCDCTL_API=3 etcdctl --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key \
  snapshot save /backup/etcd-snapshot-$(date +%Y-%m-%d).db

# Check etcd status
ETCDCTL_API=3 etcdctl --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key \
  endpoint health

Node Management

Nodes are worker machines that run containerized applications.

bash

# List nodes
kubectl get nodes

# Check node detailed information
kubectl describe node <node-name>

# Add node label
kubectl label node <node-name> environment=production

# Set node to maintenance mode
kubectl drain <node-name> --ignore-daemonsets

# Return node after maintenance
kubectl uncordon <node-name>

Component Status Monitoring

bash

# Check control plane component status
kubectl get --raw='/readyz?verbose'

# Check system pod status
kubectl get pods -n kube-system

# Check node resource usage
kubectl top nodes

Diagram showing a cluster administrator's five areas of responsibility, each mapped to the tools used to carry it out.

Diagram showing a cluster administrator's five areas of responsibility, each mapped to the tools used to carry it out.Open full screen ↗

Cluster Administration Tools

Various tools are available for Kubernetes cluster administration:

kubectl: Command-line tool for interacting with Kubernetes clusters
kubeadm: Tool for creating and managing Kubernetes clusters
kops: Tool for creating, upgrading, and managing Kubernetes clusters
eksctl: Tool for creating and managing Amazon EKS clusters
Helm: Kubernetes application package manager
Headlamp: Kubernetes web UI; the old Kubernetes Dashboard project is archived
Prometheus & Grafana: Monitoring and alerting tools
Fluentd & Elasticsearch: Logging tools

Cluster Component Management

A Kubernetes cluster consists of multiple components, and effectively managing these components is important.

Control Plane Components

Control plane components manage the overall state of the cluster:

kube-apiserver: Component that exposes the Kubernetes API
etcd: Key-value store that stores cluster data
kube-scheduler: Component that schedules pods to nodes
kube-controller-manager: Component that runs controllers
cloud-controller-manager: Component that interacts with cloud providers

The following diagram shows Kubernetes control plane components and their interactions:

Architecture diagram showing etcd, kube-scheduler, kube-controller-manager, and cloud-controller-manager communicating bidirectionally with the kube-apiserver at the center, and a worker node's kubelet communicating bidirectionally with the API server while using the container runtime; kube-proxy independently watches Service/EndpointSlice state.

Control Plane Component Monitoring

It is important to monitor the status of control plane components:

bash

# Check control plane component status
kubectl get --raw='/readyz?verbose'

# Check API server logs
kubectl logs -n kube-system kube-apiserver-<node-name>

# Check etcd status
kubectl exec -n kube-system etcd-<node-name> -- etcdctl \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/healthcheck-client.crt \
  --key=/etc/kubernetes/pki/etcd/healthcheck-client.key endpoint health

Control Plane Component Configuration

The manifest below is only a flag/configuration fragment. It omits the host networking, certificate mounts, and other kubeadm-generated settings; do not replace a running control-plane manifest with this fragment. Match the image to the cluster upgrade plan.

How to manage control plane component configuration:

yaml

# kube-apiserver configuration example
apiVersion: v1
kind: Pod
metadata:
  name: kube-apiserver
  namespace: kube-system
spec:
  containers:
  - command:
    - kube-apiserver
    - --advertise-address=192.168.1.10
    - --allow-privileged=true
    - --authorization-mode=Node,RBAC
    - --client-ca-file=/etc/kubernetes/pki/ca.crt
    - --enable-admission-plugins=NodeRestriction
    - --enable-bootstrap-token-auth=true
    - --etcd-cafile=/etc/kubernetes/pki/etcd/ca.crt
    - --etcd-certfile=/etc/kubernetes/pki/apiserver-etcd-client.crt
    - --etcd-keyfile=/etc/kubernetes/pki/apiserver-etcd-client.key
    - --etcd-servers=https://127.0.0.1:2379
    - --kubelet-client-certificate=/etc/kubernetes/pki/apiserver-kubelet-client.crt
    - --kubelet-client-key=/etc/kubernetes/pki/apiserver-kubelet-client.key
    - --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname
    - --secure-port=6443
    - --service-account-key-file=/etc/kubernetes/pki/sa.pub
    - --service-account-signing-key-file=/etc/kubernetes/pki/sa.key
    - --service-account-issuer=https://kubernetes.default.svc.cluster.local
    - --service-cluster-ip-range=10.96.0.0/12
    - --tls-cert-file=/etc/kubernetes/pki/apiserver.crt
    - --tls-private-key-file=/etc/kubernetes/pki/apiserver.key
    image: registry.k8s.io/kube-apiserver:v1.36.4
    name: kube-apiserver

Node Components

Node components run on each node and manage pods:

kubelet: Agent running on each node that ensures pods and containers are running
kube-proxy: Maintains network rules and handles connection forwarding
Container Runtime: Software that runs containers (containerd, CRI-O, or Docker Engine with an external CRI adapter)

Node Management

Key commands for node management:

bash

# List nodes
kubectl get nodes

# Check node detailed information
kubectl describe node <node-name>

# Add node label
kubectl label node <node-name> key=value

# Add node taint
kubectl taint node <node-name> key=value:NoSchedule

# Set node to maintenance mode
kubectl cordon <node-name>

# Drain node
kubectl drain <node-name> --ignore-daemonsets

Drain may stop on standalone Pods, PDBs, or local data. Investigate instead of adding --force or --delete-emptydir-data by default; the latter explicitly authorizes emptyDir data loss. Wait for drain and workload health before maintenance.

Node Troubleshooting

Commands for node troubleshooting:

bash

# Check node status
kubectl describe node <node-name> | grep Conditions -A 10

# Check node resource usage
kubectl top node <node-name>

# Check kubelet logs
journalctl -u kubelet

# Check container runtime status
systemctl status docker  # When using Docker
systemctl status containerd  # When using containerd

Resource Management

Effectively managing resources in a Kubernetes cluster is important for maintaining cluster stability and performance.

Namespace Management

Use a disposable namespace to practice lifecycle operations. Deleting a namespace deletes its namespaced resources; inspect and back up required data first. get all is only a workload subset.

bash

kubectl create namespace admin-demo
kubectl get all -n admin-demo
# Cleanup only the disposable exercise namespace:
kubectl delete namespace admin-demo

Resource Quotas

Resource quotas limit resource usage per namespace:

yaml

apiVersion: v1
kind: ResourceQuota
metadata:
  name: compute-resources
  namespace: dev
spec:
  hard:
    requests.cpu: "1"
    requests.memory: 1Gi
    limits.cpu: "2"
    limits.memory: 2Gi
    pods: "10"

In the above example, the dev namespace can have a maximum of 10 pods, 1 CPU and 1Gi memory requests, and 2 CPU and 2Gi memory limits.

Limit Ranges

Limit ranges set defaults and limits for individual resources within a namespace:

yaml

apiVersion: v1
kind: LimitRange
metadata:
  name: limit-range
  namespace: dev
spec:
  limits:
  - default:
      cpu: 500m
      memory: 512Mi
    defaultRequest:
      cpu: 200m
      memory: 256Mi
    max:
      cpu: 1
      memory: 1Gi
    min:
      cpu: 100m
      memory: 128Mi
    type: Container

In the above example, all containers in the dev namespace have default limits of 500m CPU and 512Mi memory, default requests of 200m CPU and 256Mi memory, maximum of 1 CPU and 1Gi memory, and minimum of 100m CPU and 128Mi memory.

Horizontal Pod Autoscaler (HPA)

HPA automatically adjusts the number of pods based on CPU usage or custom metrics:

yaml

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: frontend-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: frontend
  minReplicas: 2
  maxReplicas: 10
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 80

In the above example, the frontend deployment targets average CPU utilization of 80% of requested CPU, subject to tolerance, missing metrics, stabilization windows and scaling policies. It maintains a minimum of 2 and maximum of 10 replicas.

Vertical Pod Autoscaler (VPA)

VPA automatically adjusts pod CPU and memory requests:

yaml

apiVersion: autoscaling.k8s.io/v1
kind: VerticalPodAutoscaler
metadata:
  name: frontend-vpa
spec:
  targetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: frontend
  updatePolicy:
    updateMode: "Recreate"

In the above example, pods in the frontend deployment have their CPU and memory requests automatically adjusted based on actual resource usage.

Cluster Networking

Kubernetes cluster networking manages communication between pods, services, and nodes.

Cluster Network Model

Basic requirements of the Kubernetes network model:

All pods can communicate with all other pods without NAT
Node agents (kubelet) can communicate with all pods on that node
External connectivity depends on routing and egress policy; there is no universal Pod NAT-mode requirement

The following diagram shows Kubernetes networking components and communication flows:

Architecture diagram showing client traffic entering through Ingress to a Service that load-balances across two pods on separate nodes, which communicate with each other and reach an external service.

CNI (Container Network Interface) Plugins

Kubernetes implements networking through CNI plugins. Common CNI plugins:

Calico: CNI with enhanced network policy and security features
Flannel: Provides simple overlay networking
Cilium: eBPF-based networking and security solution
AWS VPC CNI: CNI integrated with AWS VPC
Weave Net (historical): Archived; choose a maintained alternative for new installations

CNI Plugin Installation and Configuration

CNI plugin installation example (Calico):

Choose one CNI or a documented chaining/migration setup. The Calico, Flannel, and Cilium alternatives must not be installed sequentially into the same running cluster. Use a supported, pinned release and the provider-specific instructions; on EKS, follow the VPC CNI or planned alternative-CNI procedure in the networking chapter.

bash

# Inspect the installed networking components before making changes
kubectl get daemonsets -A
kubectl get pods -A -l k8s-app=calico-node

Service Networking

Kubernetes services provide stable endpoints for pod sets:

ClusterIP: Service accessible only within the cluster
NodePort: Service accessible through a specific port on all nodes
LoadBalancer: Service accessible through an external load balancer
ExternalName: Provides CNAME record for external services

Service CIDR Configuration

Service CIDR defines the service IP address range:

bash

# Set service CIDR in kube-apiserver configuration
--service-cluster-ip-range=10.96.0.0/12

CoreDNS Management

CoreDNS provides DNS services for Kubernetes:

bash

# Check CoreDNS status
kubectl get pods -n kube-system -l k8s-app=kube-dns

# Check CoreDNS configuration
kubectl get configmap -n kube-system coredns -o yaml

CoreDNS configuration example:

yaml

apiVersion: v1
kind: ConfigMap
metadata:
  name: coredns
  namespace: kube-system
data:
  Corefile: |
    .:53 {
        errors
        health {
           lameduck 5s
        }
        ready
        kubernetes cluster.local in-addr.arpa ip6.arpa {
           pods insecure
           fallthrough in-addr.arpa ip6.arpa
           ttl 30
        }
        prometheus :9153
        forward . /etc/resolv.conf
        cache 30
        loop
        reload
        loadbalance
    }

Network Policies

Network policies control communication between pods:

yaml

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: db-network-policy
  namespace: default
spec:
  podSelector:
    matchLabels:
      role: db
  policyTypes:
  - Ingress
  - Egress
  ingress:
  - from:
    - podSelector:
        matchLabels:
          role: frontend
    ports:
    - protocol: TCP
      port: 3306
  egress:
  - to:
    - podSelector:
        matchLabels:
          role: monitoring
    ports:
    - protocol: TCP
      port: 9090

In the above example, pods with the role=db label only allow TCP port 3306 inbound traffic from pods with the role=frontend label and TCP port 9090 outbound traffic to pods with the role=monitoring label.

Authentication and Authorization Management

Kubernetes authentication and authorization management are core elements of cluster security.

The following diagram shows the Kubernetes authentication and authorization flow:

Architecture diagram showing a request moving through authentication, authorization, and admission control within the API server, with the concrete authentication methods and authorization modes each stage supports.

Authentication

Kubernetes supports various authentication methods:

X.509 Certificates: Authentication using client certificates
Service Account Tokens: JWT tokens associated with service accounts
OpenID Connect (OIDC): Authentication through external identity providers
Webhook Token Authentication: Token verification through external services
Authentication Proxy: Request processing through authentication proxy

X.509 Certificate Management

X.509 certificate creation and management:

bash

# Generate a protected private key and CSR
umask 077
openssl genrsa -out user.key 2048
openssl req -new -key user.key -out user.csr -subj "/CN=user/O=group"

# Submit CSR to Kubernetes
cat <<EOF | kubectl apply -f -
apiVersion: certificates.k8s.io/v1
kind: CertificateSigningRequest
metadata:
  name: user-csr
spec:
  request: $(cat user.csr | base64 | tr -d '\n')
  signerName: kubernetes.io/kube-apiserver-client
  usages:
  - client auth
EOF

# Approve CSR
kubectl certificate approve user-csr
kubectl wait --for=jsonpath='{.status.certificate}' csr/user-csr --timeout=60s

# Get certificate
kubectl get csr user-csr -o jsonpath='{.status.certificate}' | base64 --decode > user.crt

OIDC Authentication Configuration

OIDC authentication configuration example:

bash

# Add OIDC flags to kube-apiserver configuration
--oidc-issuer-url=https://accounts.google.com
--oidc-client-id=kubernetes
--oidc-username-claim=email
--oidc-groups-claim=groups

Authorization

Kubernetes supports various authorization modes:

RBAC (Role-Based Access Control): Role-based access control
ABAC (Attribute-Based Access Control): Attribute-based access control
Node: Node authorization
Webhook: Authorization through external services

RBAC Configuration

RBAC is the most common authorization mechanism:

yaml

# Role example
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  namespace: default
  name: pod-reader
rules:
- apiGroups: [""]
  resources: ["pods"]
  verbs: ["get", "watch", "list"]

# RoleBinding example
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: read-pods
  namespace: default
subjects:
- kind: User
  name: user
  apiGroup: rbac.authorization.k8s.io
roleRef:
  kind: Role
  name: pod-reader
  apiGroup: rbac.authorization.k8s.io

In the above example, user has permission to view pods in the default namespace.

ClusterRole and ClusterRoleBinding

Manages permissions for cluster-wide resources:

yaml

# ClusterRole example
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: node-reader
rules:
- apiGroups: [""]
  resources: ["nodes"]
  verbs: ["get", "watch", "list"]

# ClusterRoleBinding example
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: read-nodes
subjects:
- kind: User
  name: user
  apiGroup: rbac.authorization.k8s.io
roleRef:
  kind: ClusterRole
  name: node-reader
  apiGroup: rbac.authorization.k8s.io

In the above example, user has permission to view all nodes in the cluster.

Service Account Management

Service accounts are used by pods to communicate with the API server:

yaml

# Create service account
apiVersion: v1
kind: ServiceAccount
metadata:
  name: my-service-account
  namespace: default

# Grant permissions to service account
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: my-service-account-binding
  namespace: default
subjects:
- kind: ServiceAccount
  name: my-service-account
  namespace: default
roleRef:
  kind: Role
  name: pod-reader
  apiGroup: rbac.authorization.k8s.io

# Use service account in pod
---
apiVersion: v1
kind: Pod
metadata:
  name: my-pod
spec:
  serviceAccountName: my-service-account
  containers:
  - name: my-container
    image: nginx

Security Context

Security context defines permissions and access control for pods and containers:

yaml

apiVersion: v1
kind: Pod
metadata:
  name: security-context-pod
spec:
  securityContext:
    runAsUser: 1000
    runAsGroup: 3000
    fsGroup: 2000
    runAsNonRoot: true
    seccompProfile:
      type: RuntimeDefault
  containers:
  - name: security-context-container
    image: busybox:1.36
    command: ["sh", "-c", "sleep 3600"]
    securityContext:
      allowPrivilegeEscalation: false
      capabilities:
        drop:
        - ALL
      readOnlyRootFilesystem: true

In the above example, the pod runs with UID 1000 and GID 3000, and the container cannot escalate privileges, has all Linux capabilities dropped, and has the root filesystem mounted as read-only.

Cluster Upgrades

Kubernetes cluster upgrades are necessary to apply new features, performance improvements, and security patches.

The following diagram shows the Kubernetes cluster upgrade process:

Workflow diagram of a Kubernetes cluster upgrade: planning and version compatibility check, etcd backup, first control plane node upgrade and function test, remaining control plane and worker node upgrades, then cluster validation ending in upgrade complete, with a rollback path that restores from the backup when validation finds a problem.

Upgrade Planning

Considerations when planning cluster upgrades:

Version Compatibility: Check compatibility between Kubernetes versions
Upgrade Path: Check supported upgrade paths
Downtime: Plan for expected downtime during upgrade
Rollback Plan: Develop a rollback plan in case of issues
Application Impact: Assess the impact of upgrades on applications

Control Plane Upgrade

Control plane upgrade using kubeadm:

Use the version-specific kubeadm upgrade procedure. Enable the target minor's pkgs.k8s.io repository and select an exact published package version; upgrade one minor at a time.

Back up etcd and validate workload/add-on compatibility. On the first control-plane node, upgrade kubeadm, run kubeadm upgrade plan, then kubeadm upgrade apply <target-version>.
On additional control-plane nodes, upgrade kubeadm and run kubeadm upgrade node.
Drain each node before upgrading its kubelet. If drain fails, stop the procedure and resolve the cause. Upgrade kubelet/kubectl to compatible versions, reload systemd, restart kubelet, verify Ready and workloads, then uncordon from an administrative client.
On each worker, upgrade kubeadm and run kubeadm upgrade node, then perform the drain/kubelet/verification/uncordon sequence. Do not run a generic whole-system upgrade as a substitute for the Kubernetes version-specific procedure.

Run commands on the explicitly identified node or administrative client; a sequence of nested ssh sessions is not a multi-node automation script.

Worker Node Upgrade

Worker node upgrade process:

Use the version-specific kubeadm upgrade procedure. Enable the target minor's pkgs.k8s.io repository and select an exact published package version; upgrade one minor at a time.

Back up etcd and validate workload/add-on compatibility. On the first control-plane node, upgrade kubeadm, run kubeadm upgrade plan, then kubeadm upgrade apply <target-version>.
On additional control-plane nodes, upgrade kubeadm and run kubeadm upgrade node.
Drain each node before upgrading its kubelet. If drain fails, stop the procedure and resolve the cause. Upgrade kubelet/kubectl to compatible versions, reload systemd, restart kubelet, verify Ready and workloads, then uncordon from an administrative client.
On each worker, upgrade kubeadm and run kubeadm upgrade node, then perform the drain/kubelet/verification/uncordon sequence. Do not run a generic whole-system upgrade as a substitute for the Kubernetes version-specific procedure.

Run commands on the explicitly identified node or administrative client; a sequence of nested ssh sessions is not a multi-node automation script.

Upgrade Verification

Verify cluster status after upgrade:

bash

# Check node versions
kubectl get nodes

# Check component status
kubectl get --raw='/readyz?verbose'

# Check pod status
kubectl get pods --all-namespaces

# Test cluster functionality
kubectl create deployment nginx --image=nginx
kubectl expose deployment nginx --port=80
kubectl get svc nginx

Backup and Recovery

Kubernetes cluster backup and recovery is an important part of disaster recovery planning.

The following diagram shows the Kubernetes cluster backup and recovery process:

Architecture diagram showing scheduled etcd snapshots and resource-YAML backups both landing in backup storage, which later feeds an etcd restore and a resource restore during disaster recovery.

Architecture diagram showing scheduled etcd snapshots and resource-YAML backups both landing in backup storage, which later feeds an etcd restore and a resource restore during disaster recovery.Open full screen ↗

etcd Backup

etcd stores all state information for the Kubernetes cluster, so regular backups are important:

bash

# Create etcd snapshot
ETCDCTL_API=3 etcdctl --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key \
  snapshot save /backup/etcd-snapshot-$(date +%Y-%m-%d-%H-%M-%S).db

# Check snapshot status
etcdutl snapshot status --write-out=table /backup/etcd-snapshot-2023-01-01-12-00-00.db

etcd Recovery

Restore from etcd snapshot:

For self-managed disaster recovery, stop all API servers and the affected etcd processes using the distribution-specific runbook. Stopping kubelet alone leaves existing static Pod containers running. Restore into a new directory with a compatible etcdutl; preserve the original data until recovery is verified. This single-member command is an illustration, not a multi-member HA restore:

bash

etcdutl snapshot status "$SNAPSHOT_FILE" --write-out=table
etcdutl snapshot restore "$SNAPSHOT_FILE" \
  --data-dir=/var/lib/etcd-restore \
  --name=etcd-1 \
  --initial-cluster=etcd-1=https://127.0.0.1:2380 \
  --initial-cluster-token=restored-cluster \
  --initial-advertise-peer-urls=https://127.0.0.1:2380 \
  --bump-revision=1000000000 --mark-compacted

Set SNAPSHOT_FILE to the verified snapshot. For HA, restore the same snapshot on every member with its unique name/peer URL and the same full membership list. Choose a revision bump exceeding changes since the snapshot. Update the etcd manifest/service to the restored path with correct ownership and certificates, verify quorum/health, then restart API servers/controllers. See the official recovery guide. EKS users cannot restore managed control-plane etcd directly.

Resource Backup

These exports are protected inventories, not a complete portable restore plan. They include Secrets, need restricted permissions/encryption, and do not include PV data. Use umask 077 and check every command failure; restore CRDs/dependencies in order and remove server-owned metadata as appropriate. kubectl get all only returns a subset of resource kinds.

Backup Kubernetes resources as YAML files:

bash

# Export listable resources (includes sensitive Secrets)
set -eu
umask 077
for ns in $(kubectl get ns -o jsonpath='{.items[*].metadata.name}'); do
  mkdir -p /backup/resources/$ns
  for resource in $(kubectl api-resources --verbs=list --namespaced=true -o name); do
    kubectl get -n "$ns" "$resource" -o yaml > "/backup/resources/$ns/$resource.yaml"
  done
done

# Backup cluster-scoped resources
mkdir -p /backup/resources/cluster-scoped
for resource in $(kubectl api-resources --verbs=list --namespaced=false -o name); do
  kubectl get "$resource" -o yaml > "/backup/resources/cluster-scoped/$resource.yaml"
done

Backup Automation

Self-managed kubeadm example only: replace the tooling image with a verified image containing a compatible etcdctl, match the control-plane label/taint and certificate paths, and create the backup PVC. The selected host must expose etcd at the shown loopback address and allow the PVC mount. This does not run on the managed EKS control plane. Verify the snapshot and copy it to protected external storage; an in-cluster PVC alone is not disaster recovery.

Automate backup tasks with CronJob:

yaml

apiVersion: batch/v1
kind: CronJob
metadata:
  name: etcd-backup
  namespace: kube-system
spec:
  concurrencyPolicy: Forbid
  schedule: "0 0 * * *"  # Run daily at midnight
  jobTemplate:
    spec:
      template:
        spec:
          hostNetwork: true
          automountServiceAccountToken: false
          nodeSelector:
            node-role.kubernetes.io/control-plane: ""
          tolerations:
          - key: node-role.kubernetes.io/control-plane
            operator: Exists
            effect: NoSchedule
          containers:
          - name: etcd-backup
            image: example.invalid/etcd-backup-tools:replace-me
            command:
            - /bin/sh
            - -c
            - |
              set -eu
              umask 077
              ETCDCTL_API=3 etcdctl --endpoints=https://127.0.0.1:2379 \
                --cacert=/etc/kubernetes/pki/etcd/ca.crt \
                --cert=/etc/kubernetes/pki/etcd/healthcheck-client.crt \
                --key=/etc/kubernetes/pki/etcd/healthcheck-client.key \
                snapshot save /backup/etcd-snapshot-$(date +%Y-%m-%d-%H-%M-%S).db
            volumeMounts:
            - name: etcd-ca
              mountPath: /etc/kubernetes/pki/etcd/ca.crt
              readOnly: true
            - name: etcd-client-cert
              mountPath: /etc/kubernetes/pki/etcd/healthcheck-client.crt
              readOnly: true
            - name: etcd-client-key
              mountPath: /etc/kubernetes/pki/etcd/healthcheck-client.key
              readOnly: true
            - name: backup
              mountPath: /backup
          restartPolicy: OnFailure
          volumes:
          - name: etcd-ca
            hostPath:
              path: /etc/kubernetes/pki/etcd/ca.crt
              type: File
          - name: etcd-client-cert
            hostPath:
              path: /etc/kubernetes/pki/etcd/healthcheck-client.crt
              type: File
          - name: etcd-client-key
            hostPath:
              path: /etc/kubernetes/pki/etcd/healthcheck-client.key
              type: File
          - name: backup
            persistentVolumeClaim:
              claimName: etcd-backup-pvc

Monitoring and Logging

Effective monitoring and logging is a core element of cluster administration.

The following diagram shows the Kubernetes cluster monitoring and logging architecture:

Architecture diagram showing API server and node metrics flowing through kube-state-metrics and Node Exporter into Prometheus, then to Alertmanager and Grafana, while pod logs flow through Fluentd/Fluent Bit into Elasticsearch with Kibana and into Loki, with Loki also feeding Grafana.

Monitoring Tools

Tools for Kubernetes cluster monitoring:

Prometheus: Metric collection and storage
Grafana: Metric visualization
Alertmanager: Alert management
kube-state-metrics: Generate Kubernetes object metrics
metrics-server: Provide resource usage metrics

Prometheus and Grafana Installation

Install Prometheus and Grafana using Helm:

bash

# Add Helm repository
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update

# Install Prometheus stack
helm install prometheus prometheus-community/kube-prometheus-stack \
  --namespace monitoring \
  --create-namespace

Key Monitoring Metrics

Key metrics to monitor:

Node Metrics: CPU, memory, disk, network usage
Pod Metrics: CPU, memory usage, restart count
Container Metrics: CPU, memory usage, filesystem usage
API Server Metrics: Request latency, request count, error rate
etcd Metrics: Disk I/O, leader changes, commit latency

Logging Tools

Tools for Kubernetes cluster logging:

Elasticsearch: Log storage and search
Fluentd/Fluent Bit: Log collection and forwarding
Kibana: Log visualization
Loki: Log aggregation system
Grafana: Log visualization

EFK (Elasticsearch, Fluentd, Kibana) Stack Installation

Install EFK stack using Helm:

The standalone Elastic Stack Helm-chart repository is archived. For maintained deployments use Elastic Cloud on Kubernetes (ECK), then define Elasticsearch/Kibana resources and a compatible log collector. The operator installation alone does not create an EFK stack:

bash

helm repo add elastic https://helm.elastic.co
helm upgrade --install elastic-operator elastic/eck-operator \
  --namespace elastic-system --create-namespace \
  --version "${ECK_CHART_VERSION:?Select a supported ECK chart version}"

Keep dashboards behind ClusterIP/authenticated access; configure storage, TLS, credentials, collector parsing, and RBAC following the ECK guide.

Log Collection Configuration

This example parses the CRI log envelope rather than assuming Docker JSON. The collector image must include the metadata/output plugins, mount node logs and writable position-file storage, and receive scoped metadata RBAC. Configure TLS/authentication for the actual Elasticsearch service; the placeholder host alone is not a complete ECK integration. Handle partial/multiline records according to the selected collector.

Fluentd configuration example:

yaml

apiVersion: v1
kind: ConfigMap
metadata:
  name: fluentd-config
  namespace: logging
data:
  fluent.conf: |
    <source>
      @type tail
      path /var/log/containers/*.log
      pos_file /var/log/fluentd-containers.log.pos
      tag kubernetes.*
      read_from_head true
      <parse>
        @type regexp
        expression /^(?<time>[^ ]+) (?<stream>stdout|stderr) (?<logtag>[^ ]*) (?<log>.*)$/
        time_type string
        time_format %Y-%m-%dT%H:%M:%S.%N%:z
      </parse>
    </source>

    <filter kubernetes.**>
      @type kubernetes_metadata
      kubernetes_url https://kubernetes.default.svc
      bearer_token_file /var/run/secrets/kubernetes.io/serviceaccount/token
      ca_file /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
    </filter>

    <match kubernetes.**>
      @type elasticsearch
      host elasticsearch-master
      port 9200
      logstash_format true
      logstash_prefix k8s
    </match>

Troubleshooting

Kubernetes cluster troubleshooting is an important part of cluster administration.

Pod Troubleshooting

Commands for pod troubleshooting:

bash

# Check pod status
kubectl get pod <pod-name> -o wide

# Check pod detailed information
kubectl describe pod <pod-name>

# Check pod logs
kubectl logs <pod-name>
kubectl logs <pod-name> -c <container-name>  # For multi-container pods
kubectl logs <pod-name> --previous  # Logs from previous container

# Execute command in pod
kubectl exec -it <pod-name> -- /bin/sh

Node Troubleshooting

Commands for node troubleshooting:

bash

# Check node status
kubectl get node <node-name> -o wide

# Check node detailed information
kubectl describe node <node-name>

# Check node resource usage
kubectl top node <node-name>

# SSH to node
ssh <node-name>

# Check node system logs
journalctl -u kubelet

# Check node resource usage
top
df -h
free -m

Networking Troubleshooting

Commands for networking troubleshooting:

bash

# Check service status
kubectl get svc <service-name>

# Check service detailed information
kubectl describe svc <service-name>

# Check endpoints
kubectl get endpointslices -l kubernetes.io/service-name=<service-name>

# Check DNS
kubectl run -it --rm --restart=Never busybox --image=busybox -- nslookup <service-name>

# Test network connectivity
kubectl run -it --rm --restart=Never busybox --image=busybox -- wget -O- <service-name>:<port>

# Check network policies
kubectl get networkpolicy
kubectl describe networkpolicy <policy-name>

Control Plane Troubleshooting

Commands for control plane troubleshooting:

bash

# Check component status
kubectl get --raw='/readyz?verbose'

# Check API server logs
kubectl logs -n kube-system kube-apiserver-<node-name>

# Check controller manager logs
kubectl logs -n kube-system kube-controller-manager-<node-name>

# Check scheduler logs
kubectl logs -n kube-system kube-scheduler-<node-name>

# Check etcd logs
kubectl logs -n kube-system etcd-<node-name>

Amazon EKS Cluster Administration

Amazon EKS is a managed Kubernetes service that automates many aspects of cluster administration.

The following diagram shows the Amazon EKS cluster architecture and management components:

Architecture diagram showing a user managing Amazon EKS through the AWS console, CLI, or API, with EKS running a control plane plus managed node groups and Fargate, the control plane using AWS IAM, VPC, and CloudWatch, and the VPC CNI, CoreDNS, and kube-proxy add-ons.

EKS Cluster Configuration

Set the real administrator CIDR before changing endpoint access and verify private access remains reachable. A version upgrade must be the next supported minor with compatible add-ons/nodes; do not use the example as a downgrade or skip-minor operation. Poll the returned update ID and stop dependent changes if it fails.

EKS cluster configuration management:

bash

# Check EKS cluster information
aws eks describe-cluster --name my-cluster

# Update EKS cluster
aws eks update-cluster-config \
  --name my-cluster \
  --resources-vpc-config "endpointPublicAccess=true,endpointPrivateAccess=true,publicAccessCidrs=${ADMIN_CIDR:?Set an approved administrator public CIDR}"

# Update EKS cluster version
aws eks update-cluster-version \
  --name my-cluster \
  --kubernetes-version "${TARGET_VERSION:?Select the next EKS-supported minor version}"

EKS Node Group Management

EKS node group management:

bash

# Check node group information
aws eks describe-nodegroup \
  --cluster-name my-cluster \
  --nodegroup-name my-nodegroup

# Scale node group
aws eks update-nodegroup-config \
  --cluster-name my-cluster \
  --nodegroup-name my-nodegroup \
  --scaling-config minSize=2,maxSize=10,desiredSize=5

# Update node group
aws eks update-nodegroup-version \
  --cluster-name my-cluster \
  --nodegroup-name my-nodegroup

EKS Add-on Management

Read the current version using aws eks describe-cluster --name my-cluster --query cluster.version --output text; use it for add-on discovery and select a compatible pinned version. Review existing configuration/IAM before create/update and do not create an already managed add-on again. The removal example uses --preserve to leave the CNI running while removing EKS management; uninstalling live networking is a separate disruptive operation.

EKS add-on management:

bash

# Check available add-ons
aws eks describe-addon-versions --addon-name vpc-cni \
  --kubernetes-version "${CLUSTER_VERSION:?Set the actual cluster version}"

# Install add-on
aws eks create-addon \
  --cluster-name my-cluster \
  --addon-name vpc-cni \
  --addon-version "${CNI_ADDON_VERSION:?Select a compatible pinned VPC CNI add-on version}"

# Update add-on
aws eks update-addon \
  --cluster-name my-cluster \
  --addon-name vpc-cni \
  --addon-version "${CNI_ADDON_VERSION:?Select a compatible pinned VPC CNI add-on version}"

# Delete add-on
aws eks delete-addon \
  --cluster-name my-cluster \
  --addon-name vpc-cni --preserve

EKS Cluster Upgrade

EKS cluster upgrade process:

Control Plane Upgrade:

bash

aws eks update-cluster-version \
  --name my-cluster \
  --kubernetes-version "${TARGET_VERSION:?Select the next EKS-supported minor version}"

Add-on Upgrade:

bash

aws eks update-addon \
  --cluster-name my-cluster \
  --addon-name vpc-cni \
  --addon-version "${CNI_ADDON_VERSION:?Select a compatible pinned VPC CNI add-on version}"

Node Group Upgrade:

bash

aws eks update-nodegroup-version \
  --cluster-name my-cluster \
  --nodegroup-name my-nodegroup

EKS Cluster Monitoring

Control-plane logging exports api/audit/authenticator/controllerManager/scheduler logs. Container Insights requires the CloudWatch agent/add-on and scoped telemetry IAM permissions; it is not enabled by update-cluster-logging. The CloudWatch observability add-on installs CloudWatch/Fluent Bit components, not Prometheus/Grafana. See the official add-on setup.

EKS cluster monitoring tools:

Amazon CloudWatch: Metrics, logs, alerts
AWS CloudTrail: API call logging
Amazon Managed Grafana: Metric visualization
Amazon Managed Service for Prometheus: Metric collection and storage

Enable EKS control-plane logging:

bash

# Enable EKS control-plane logs
eksctl utils update-cluster-logging \
  --enable-types all \
  --cluster my-cluster \
  --approve

Cluster Administration Best Practices

Best practices for Kubernetes and EKS cluster administration:

Cluster Configuration Best Practices

Infrastructure as Code (IaC): Manage cluster configuration using Terraform, AWS CDK, eksctl, etc.
Version Control: Store cluster configuration in version control systems
Multiple Environments: Separate development, staging, and production environments
Network Separation: Configure appropriate network separation and security groups
Least Privilege Principle: Grant only the minimum necessary permissions

Operations Best Practices

Regular Backups: Regular backup of etcd and important resources
Monitoring and Alerting: Build comprehensive monitoring and alerting systems
Centralized Logging: Centralize and analyze logs
Automation: Automate repetitive tasks
Disaster Recovery Planning: Establish and test clear disaster recovery plans

Security Best Practices

Regular Updates: Regular updates of cluster and nodes
Network Policies: Configure appropriate network policies
Encryption: Encrypt data at rest and in transit
Security Context: Configure appropriate security contexts
Image Scanning: Scan container images for vulnerabilities

Resource Management Best Practices

Resource Requests and Limits: Set appropriate resource requests and limits for all pods
Namespace Separation: Separate workloads by namespace
Resource Quotas: Set resource quotas per namespace
HPA and VPA: Configure autoscaling
Node Affinity and Taints: Optimize workload placement

EKS-Specific Best Practices

Managed Node Groups: Use managed node groups when possible
Fargate: Use Fargate for serverless workloads
EKS Add-ons: Use official EKS add-ons
IAM Roles for Service Accounts (IRSA): Manage IAM permissions per pod
VPC CNI Customization: Configure VPC CNI according to networking requirements

Conclusion

Kubernetes cluster administration plays an important role in maintaining cluster stability, security, and performance. This chapter covered various aspects of cluster administration including cluster component management, resource management, networking, authentication and authorization management, upgrades, backup and recovery, monitoring and logging, and troubleshooting.

Using Amazon EKS reduces the complexity of Kubernetes control plane management and simplifies cluster administration through integration with AWS services. However, understanding fundamental Kubernetes concepts and best practices is still important for effective cluster management.

Cluster administration is an ongoing process that must be continuously adjusted according to cluster requirements and workload characteristics. It is important to use monitoring tools to track cluster status, minimize repetitive tasks through automation, and follow best practices to maintain cluster stability and security.

Cluster Networking

Kubernetes cluster networking manages pod-to-pod communication, service discovery, and external access.

Network Architecture

Architecture diagram showing cluster networking split into pod network, service network, ingress, and network policies, each implemented by a CNI plugin, Service types (ClusterIP, NodePort, LoadBalancer), an ingress controller, and network security respectively.

CNI Plugin Management

CNI (Container Network Interface) plugins handle networking for Kubernetes clusters.

bash

# Inspect the installed networking components before making changes
kubectl get daemonsets -A
kubectl get pods -A -l k8s-app=calico-node

CNI Plugin Comparison

CNI Plugin	Network Model	Network Policy Support	Performance	Features
Calico	BGP	Yes	High	Strong in network policies, routing-based
Flannel	VXLAN/host-gateway	No	Medium	Simple setup, limited features
Cilium	eBPF	Yes	Very High	L3-L7 policies, high performance
Weave Net	VXLAN	Yes	Medium	Encryption support, multi-cluster
AWS VPC CNI	AWS VPC	Yes, with supported version/configuration	Workload dependent	Native EKS integration

Network Troubleshooting

bash

# Test pod network connectivity
kubectl run -it --rm network-test --image=busybox -- sh
# Inside the container
ping <target-ip>
traceroute <target-ip>
wget -O- <service-name>

# DNS troubleshooting
kubectl run -it --rm dns-test --image=busybox -- sh
# Inside the container
nslookup kubernetes.default.svc.cluster.local
cat /etc/resolv.conf

# Check service endpoints
kubectl get endpointslices -l kubernetes.io/service-name=<service-name>

# Check network policies
kubectl describe networkpolicy -n <namespace>

Authentication and Authorization Management

Kubernetes authentication and authorization management are core elements of cluster security. RBAC (Role-Based Access Control) is used to manage permissions for users and service accounts.

Authentication Methods

Kubernetes supports various authentication methods:

X.509 Certificates: Authentication using client certificates
Service Account Tokens: Used for API server access within pods
OpenID Connect (OIDC): Integration with external identity providers
Webhook Token Authentication: Integration with external authentication services
Authentication Proxy: Authentication through proxy

RBAC Configuration

yaml

# role.yaml - namespace-scoped role
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  namespace: default
  name: pod-reader
rules:
- apiGroups: [""]
  resources: ["pods"]
  verbs: ["get", "watch", "list"]

yaml

# rolebinding.yaml - binding role to user
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: read-pods
  namespace: default
subjects:
- kind: User
  name: jane
  apiGroup: rbac.authorization.k8s.io
roleRef:
  kind: Role
  name: pod-reader
  apiGroup: rbac.authorization.k8s.io

yaml

# clusterrole.yaml - cluster-scoped role
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: namespace-reader
rules:
- apiGroups: [""]
  resources: ["namespaces"]
  verbs: ["get", "watch", "list"]

yaml

# clusterrolebinding.yaml - binding cluster role to user
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: read-namespaces-global
subjects:
- kind: Group
  name: namespace-viewers
  apiGroup: rbac.authorization.k8s.io
roleRef:
  kind: ClusterRole
  name: namespace-reader
  apiGroup: rbac.authorization.k8s.io

User Certificate Creation

For self-managed client-certificate authentication, submit a CSR and have an authorized approver verify the requested identity/groups. Do not distribute the cluster CA private key. EKS user access should use IAM/access entries.

bash

umask 077
openssl genrsa -out jane.key 2048
openssl req -new -key jane.key -out jane.csr -subj "/CN=jane/O=dev"
cat <<EOF | kubectl apply -f -
apiVersion: certificates.k8s.io/v1
kind: CertificateSigningRequest
metadata:
  name: jane-csr
spec:
  request: $(base64 < jane.csr | tr -d '\n')
  signerName: kubernetes.io/kube-apiserver-client
  expirationSeconds: 86400
  usages:
  - client auth
EOF
# Authorized approver only, after reviewing the CSR identity:
kubectl certificate approve jane-csr
kubectl wait --for=jsonpath='{.status.certificate}' csr/jane-csr --timeout=60s
kubectl get csr jane-csr -o jsonpath='{.status.certificate}' | base64 --decode > jane.crt
kubectl config set-credentials jane --client-certificate=jane.crt --client-key=jane.key
kubectl config set-context jane-context --cluster=kubernetes --user=jane

Service Account Management

bash

# Create service account
kubectl create serviceaccount app-service-account

# Bind role to service account
kubectl create rolebinding app-service-account-binding \
  --role=pod-reader \
  --serviceaccount=default:app-service-account

# Inspect ServiceAccount metadata (projected tokens are not listed here)
kubectl describe serviceaccount app-service-account

Permission Verification

bash

# Check user permissions
kubectl auth can-i get pods --as jane

# Check permissions in a specific namespace
kubectl auth can-i create deployments --as jane --namespace production

Cluster Upgrades

Kubernetes cluster upgrades are necessary to apply new features, security patches, and bug fixes. Upgrades must be carefully planned and executed.

Upgrade Planning

Tree diagram showing upgrade planning split into four tasks, checking version compatibility, creating a backup, choosing an upgrade strategy, and planning downtime, each leading to a concrete action: reviewing API changes, an etcd backup, in-place versus blue/green, and user communication.

Upgrade Strategy Comparison

Strategy	Description	Advantages	Disadvantages	Suitable Environment
In-place Upgrade	Directly upgrade existing cluster	Resource efficient, simple procedure	Complex rollback, potential downtime	Development, test environments
Blue/Green Deployment	Create new version cluster and switch	Safe rollback, verifiable	Resource duplication, increased cost	Production environments
Canary Deployment	Move only some workloads to new cluster	Gradual verification, reduced risk	Complex management, dual operation	Critical production environments

Upgrade Using kubeadm

Use the version-specific kubeadm upgrade procedure. Enable the target minor's pkgs.k8s.io repository and select an exact published package version; upgrade one minor at a time.

Back up etcd and validate workload/add-on compatibility. On the first control-plane node, upgrade kubeadm, run kubeadm upgrade plan, then kubeadm upgrade apply <target-version>.
On additional control-plane nodes, upgrade kubeadm and run kubeadm upgrade node.
Drain each node before upgrading its kubelet. If drain fails, stop the procedure and resolve the cause. Upgrade kubelet/kubectl to compatible versions, reload systemd, restart kubelet, verify Ready and workloads, then uncordon from an administrative client.
On each worker, upgrade kubeadm and run kubeadm upgrade node, then perform the drain/kubelet/verification/uncordon sequence. Do not run a generic whole-system upgrade as a substitute for the Kubernetes version-specific procedure.

Run commands on the explicitly identified node or administrative client; a sequence of nested ssh sessions is not a multi-node automation script.

Post-Upgrade Verification

bash

# Check cluster version
kubectl version

# Check node versions
kubectl get nodes

# Check component status
kubectl get --raw='/readyz?verbose'

# Check workload status
kubectl get pods -A

Backup and Recovery

Kubernetes cluster backup and recovery is an important part of disaster recovery planning. Main backup targets are the etcd database, persistent volume data, and Kubernetes resource definitions.

etcd Backup and Recovery

etcd is a core component that stores all state information for the cluster.

bash

etcdutl snapshot status "$SNAPSHOT_FILE" --write-out=table
etcdutl snapshot restore "$SNAPSHOT_FILE" \
  --data-dir=/var/lib/etcd-restore \
  --name=etcd-1 \
  --initial-cluster=etcd-1=https://127.0.0.1:2380 \
  --initial-cluster-token=restored-cluster \
  --initial-advertise-peer-urls=https://127.0.0.1:2380 \
  --bump-revision=1000000000 --mark-compacted

Kubernetes Resource Backup

bash

# Export selected resources, not a full cluster backup
set -eu
umask 077
mkdir -p /backup/resources/$(date +%Y-%m-%d)
for ns in $(kubectl get ns -o jsonpath='{.items[*].metadata.name}'); do
  kubectl -n $ns get all -o yaml > /backup/resources/$(date +%Y-%m-%d)/$ns-all.yaml
done

# Backup specific resource types
for resource in deployments services configmaps secrets; do
  kubectl get $resource -A -o yaml > /backup/resources/$(date +%Y-%m-%d)/$resource.yaml
done

Backup and Recovery Using Velero

Select compatible Velero/AWS plugin versions using the official matrix. The IRSA example assumes a configured cluster OIDC provider and a scoped role trusting the velero ServiceAccount. Configure the backup bucket, volume snapshot/file-backup support, encryption, and restore permissions before installation; not every PVC is automatically protected.

Velero is a tool for backing up and recovering Kubernetes cluster resources and persistent volumes.

bash

# Install Velero (using AWS S3 backup storage)
velero install \
  --provider aws \
  --plugins "${VELERO_AWS_PLUGIN_IMAGE:?Select a plugin compatible with your Velero release}" \
  --bucket velero-backup \
  --backup-location-config region=us-west-2 \
  --snapshot-location-config region=us-west-2 \
  --no-secret \
  --sa-annotations "eks.amazonaws.com/role-arn=${VELERO_ROLE_ARN:?Set the preconfigured IRSA role ARN}"

# Full cluster backup
velero backup create full-cluster-backup --include-namespaces '*'

# Backup specific namespace
velero backup create production-backup --include-namespaces production

# Check backup status
velero backup describe full-cluster-backup

# Restore from backup
velero restore create --from-backup full-cluster-backup

Backup Strategy Comparison

Backup Method	Backup Target	Advantages	Disadvantages	Recovery Time
etcd Snapshot	Cluster state	Built-in feature, complete state preservation	Volume data not included, manual process	Medium
Resource YAML Backup	Kubernetes objects	Simple implementation, selective restore	Volume data not included, relationship complexity	Slow
Velero	Resources and volumes	Automation, scheduling, volume snapshots	Additional tool installation required	Fast
Cloud Provider Snapshots	Supported disks/filesystems	Backend-native recovery points	Does not capture the whole Kubernetes/EKS cluster	Depends on data/backend

Monitoring and Logging

Effective cluster management requires a comprehensive monitoring and logging system. This allows problems to be detected and resolved early.

Monitoring Architecture

Architecture diagram showing Kubernetes monitoring split into metric collection, log collection, alerting, and visualization, handled by Prometheus with kube-state-metrics and node-exporter, Fluentd/Fluent Bit shipping logs to Elasticsearch and Kibana, Alertmanager, and Grafana respectively.

Prometheus and Grafana Installation

bash

# Install Prometheus and Grafana using Helm
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update

helm install prometheus prometheus-community/kube-prometheus-stack \
  --namespace monitoring \
  --create-namespace \
  --set grafana.enabled=true \
  --set prometheus.service.type=ClusterIP

# Check services
kubectl get svc -n monitoring

# Access Grafana (using port forwarding)
kubectl port-forward svc/prometheus-grafana 3000:80 -n monitoring
# Obtain credentials from the configured Grafana Secret; do not assume a published default password

EFK Stack Installation (Elasticsearch, Fluentd, Kibana)

bash

helm repo add elastic https://helm.elastic.co
helm upgrade --install elastic-operator elastic/eck-operator \
  --namespace elastic-system --create-namespace \
  --version "${ECK_CHART_VERSION:?Select a supported ECK chart version}"

Keep dashboards behind ClusterIP/authenticated access; configure storage, TLS, credentials, collector parsing, and RBAC following the ECK guide.

Key Monitoring Metrics

Metric Type	Description	Key Metrics	Monitoring Tools
Node Metrics	Node-level resource usage	CPU, memory, disk, network	node-exporter, Prometheus
Pod Metrics	Container resource usage	CPU, memory usage, limits	cAdvisor, Prometheus
Cluster Metrics	Cluster state and resources	Pod count, node/object status, desired/current replicas	kube-state-metrics
Application Metrics	Custom application metrics	Request count, latency, error rate	Prometheus client libraries

Log Collection and Analysis

bash

# Check logs for a specific pod
kubectl logs <pod-name> -n <namespace>

# Check logs from previous instance
kubectl logs <pod-name> -n <namespace> --previous

# Check logs for a specific container (multi-container pod)
kubectl logs <pod-name> -c <container-name> -n <namespace>

# Stream logs
kubectl logs -f <pod-name> -n <namespace>

# Check logs for all pods (using label selector)
kubectl logs -l app=nginx -n <namespace>

Alert Configuration

You can configure alerts using Prometheus Alertmanager:

Create a protected slack-webhook Secret with key url in monitoring, then merge these Helm values into the existing kube-prometheus-stack release. Do not commit the webhook URL. A standalone ConfigMap is not automatically consumed by the operator.

yaml

# alertmanager-values.yaml
alertmanager:
  alertmanagerSpec:
    secrets:
    - slack-webhook
  config:
    global:
      resolve_timeout: 5m
      slack_api_url_file: /etc/alertmanager/secrets/slack-webhook/url
    route:
      receiver: slack-notifications
      group_wait: 30s
      group_interval: 5m
      repeat_interval: 4h
      group_by: [alertname, cluster, service]
    receivers:
    - name: slack-notifications
      slack_configs:
      - channel: '#alerts'
        send_resolved: true
        title: '{{ range .Alerts }}{{ .Annotations.summary }}{{ end }}'
        text: '{{ range .Alerts }}{{ .Annotations.description }}{{ end }}'

Keep the current chart version and other values when upgrading; verify Alertmanager reload/status before relying on notifications.

Troubleshooting

Kubernetes cluster troubleshooting is an important skill for system administrators and operators. A systematic approach is required for effective troubleshooting.

Troubleshooting Methodology

Workflow diagram showing the six-step troubleshooting sequence from problem identification through documentation, with the information-gathering step branching into log, event, and resource status checks.

Common Problems and Solutions

Problem Type	Symptoms	Diagnostic Commands	Common Solutions
Pod Not Starting	Pod in Pending or ContainerCreating state	`kubectl describe pod <pod-name>`	Check resource constraints, image availability, volume mounts
Service Connection Issues	Cannot access pods through service	`kubectl describe svc <service-name>`, `kubectl get endpointslices -l kubernetes.io/service-name=<service-name>`	Check label selectors, pod status, network policies
Node Issues	Node in NotReady state	`kubectl describe node <node-name>`, `kubectl get events`	Check kubelet status, system resources, network connectivity
DNS Issues	Cannot connect by service name	`kubectl exec -it <pod-name> -- nslookup kubernetes.default`	Check CoreDNS pods, kube-dns service, network policies
Authentication/Authorization Issues	API server access denied	`kubectl auth can-i <verb> <resource>`	Check RBAC settings, certificate validity, service account

Pod Troubleshooting

bash

# Check pod status
kubectl get pod <pod-name> -o wide

# Check pod details
kubectl describe pod <pod-name>

# Check pod logs
kubectl logs <pod-name>
kubectl logs <pod-name> --previous  # Logs from previous container

# Execute command in pod
kubectl exec -it <pod-name> -- /bin/sh

# Check pod events
kubectl get events --field-selector involvedObject.name=<pod-name>

Node Troubleshooting

bash

# Check node status
kubectl get nodes
kubectl describe node <node-name>

# Check node resource usage
kubectl top node <node-name>

# Check node system logs (SSH required)
ssh <node-ip> 'sudo journalctl -u kubelet'

# Check kubelet status (SSH required)
ssh <node-ip> 'sudo systemctl status kubelet'

Networking Troubleshooting

bash

# Check service and endpoints
kubectl get svc <service-name>
kubectl get endpointslices -l kubernetes.io/service-name=<service-name>

# DNS troubleshooting
kubectl run -it --rm dns-test --image=busybox -- sh
# Inside the container
nslookup kubernetes.default.svc.cluster.local
cat /etc/resolv.conf

# Network connectivity test
kubectl run -it --rm network-test --image=nicolaka/netshoot -- sh
# Inside the container
ping <target-ip>
traceroute <target-ip>
curl <service-name>:<port>

Amazon EKS Cluster Administration

Amazon EKS (Elastic Kubernetes Service) is a managed Kubernetes service on AWS where AWS manages the control plane. Customer responsibilities for nodes depend on managed node groups, Auto Mode, Fargate, or self-managed compute; workload security and configuration remain customer responsibilities.

EKS Cluster Architecture

Architecture diagram showing an Amazon EKS cluster split into an AWS-managed control plane (API server, etcd, scheduler) and a customer-responsibility area covering the data plane (managed node groups with EC2 Auto Scaling groups, self-managed nodes, Fargate), networking (VPC CNI and AWS VPC), and security (IAM authentication with IAM roles and policies).

EKS Cluster Creation

bash

# Create cluster using eksctl
eksctl create cluster \
  --name my-cluster \
  --version 1.36 \
  --region us-west-2 \
  --nodegroup-name standard-workers \
  --node-type t3.medium \
  --nodes 3 \
  --nodes-min 1 \
  --nodes-max 5 \
  --managed

# Alternative: create control plane using AWS CLI (not after the eksctl example)
aws eks create-cluster \
  --name my-cluster \
  --role-arn arn:aws:iam::123456789012:role/eks-cluster-role \
  --kubernetes-version 1.36 \
  --resources-vpc-config "subnetIds=${EKS_SUBNET_IDS:?Set two or more appropriate subnets},securityGroupIds=${EKS_SECURITY_GROUP_ID:?Set the intended security group},endpointPrivateAccess=true,endpointPublicAccess=true,publicAccessCidrs=${ADMIN_CIDR:?Set an approved administrator public CIDR}"

Node Group Management

bash

# Create managed node group
eksctl create nodegroup \
  --cluster my-cluster \
  --region us-west-2 \
  --name my-nodegroup \
  --node-type t3.medium \
  --nodes 3 \
  --nodes-min 1 \
  --nodes-max 5

# Scale node group
eksctl scale nodegroup \
  --cluster my-cluster \
  --name my-nodegroup \
  --nodes 5 \
  --region us-west-2

# Update node group
aws eks update-nodegroup-version \
  --cluster-name my-cluster \
  --nodegroup-name my-nodegroup \
  --region us-west-2

EKS Cluster Upgrade

bash

# Check cluster version
aws eks describe-cluster --name my-cluster --query "cluster.version"

# Upgrade cluster control plane
aws eks update-cluster-version \
  --name my-cluster \
  --kubernetes-version "${TARGET_VERSION:?Select the next EKS-supported minor version}"

# Upgrade managed node group
aws eks update-nodegroup-version \
  --cluster-name my-cluster \
  --nodegroup-name my-nodegroup

EKS Cluster Authentication and Authorization

The cluster must use API or API_AND_CONFIG_MAP authentication mode. Plan migration from legacy aws-auth mappings while preserving existing administrator/node access. This example grants a viewer role access only to the default namespace:

bash

aws eks describe-cluster --name my-cluster --query cluster.accessConfig.authenticationMode
aws eks create-access-entry --cluster-name my-cluster \
  --principal-arn arn:aws:iam::123456789012:role/cluster-viewer --type STANDARD
aws eks associate-access-policy --cluster-name my-cluster \
  --principal-arn arn:aws:iam::123456789012:role/cluster-viewer \
  --policy-arn arn:aws:eks::aws:cluster-access-policy/AmazonEKSViewPolicy \
  --access-scope type=namespace,namespaces=default

See the EKS access-entry guide. Permissions to manage EKS access entries and Kubernetes workload permissions are separate.

EKS Cluster Monitoring

bash

# Enable EKS control-plane logs
eksctl utils update-cluster-logging \
  --enable-types all \
  --cluster my-cluster \
  --region us-west-2

# Install CloudWatch observability (not Prometheus/Grafana)
aws eks create-addon \
  --cluster-name my-cluster \
  --addon-name amazon-cloudwatch-observability \
  --addon-version "${CLOUDWATCH_ADDON_VERSION:?Select a compatible add-on version}"

Cluster Administration Best Practices

Best practices for effective Kubernetes cluster management are important for ensuring stability, security, and performance.

Cluster Setup Best Practices

Multi-Availability Zone Configuration: Distribute nodes across multiple availability zones for high availability
Appropriate Sizing: Select node types and counts appropriate for workloads
Autoscaling Configuration: Enable cluster autoscaler and horizontal pod autoscaler
Apply Network Policies: Start with a default deny policy and allow only necessary communication
Set Resource Quotas: Set resource limits per namespace

Operations Best Practices

Use Declarative Configuration: Define all resources as YAML files and version control them
Adopt GitOps: Use Git as the single source of truth and build automated deployment pipelines
Regular Backups: Regular backup of etcd data and persistent volume data
Monitoring and Alerting: Build comprehensive monitoring systems and set alerts for key metrics
Centralized Logging: Collect all logs to a central logging system for easy analysis

Security Best Practices

Least Privilege Principle: Grant only the minimum necessary permissions using RBAC
Network Segmentation: Limit pod-to-pod communication using network policies
Image Scanning: Implement container image scanning for vulnerability detection
Secret Management: Use external secret management tools (e.g., AWS Secrets Manager, HashiCorp Vault)
Regular Security Audits: Conduct regular audits of cluster configuration and permissions

Upgrade Best Practices

Gradual Upgrades: Upgrade gradually rather than all at once
Test Environment First: Verify upgrades in test environments before production
Create Backups: Perform full backups before upgrades
Rollback Plan: Develop a plan to rollback to previous versions in case of issues
Set Upgrade Windows: Perform upgrades during low-usage periods

Cost Optimization Best Practices

Select Appropriate Node Sizes: Select optimal node types for workloads
Utilize Spot Instances: Use spot instances for non-critical workloads
Configure Autoscaling: Configure automatic scale up and down based on demand
Optimize Resource Requests and Limits: Set resource requests and limits based on actual usage
Identify Idle Resources: Regularly identify and remove idle resources

Documentation Best Practices

Document Architecture: Document cluster architecture, networking, and security settings
Document Operations Procedures: Document common operations tasks, troubleshooting procedures, and emergency response plans
Change Management: Record and track all cluster changes
Create Runbooks: Provide step-by-step guides for common scenarios
Knowledge Sharing: Conduct regular knowledge sharing and training sessions within the team

Conclusion

Kubernetes cluster administration is a complex task that includes various aspects. A systematic approach is required from cluster setup to operation, monitoring, troubleshooting, and upgrades.

For effective cluster administration, focus on the following key areas:

Cluster Component Management: Stable operation of control plane and node components
Resource Management: Efficient resource allocation and usage
Networking: Secure and efficient network configuration
Security: Appropriate authentication and authorization management
Backup and Recovery: Data loss prevention and disaster recovery planning
Monitoring and Logging: Cluster status and performance monitoring
Troubleshooting: Systematic troubleshooting approach

When using managed Kubernetes services like Amazon EKS, it is important to understand the shared responsibility model between the service provider and the user. AWS manages the control plane, while compute responsibility varies by mode; customers still manage application configuration and security.

By following best practices and utilizing appropriate tools, you can operate a stable, secure, and efficient Kubernetes cluster. Continuous learning and improvement to enhance cluster management capabilities is important.

References:
Kubernetes Official Documentation: Cluster Administration
Amazon EKS User Guide
Kubernetes Best Practices: Cluster Administration
etcd Documentation: Backup and Recovery
Prometheus Documentation

Quiz

To test what you learned in this chapter, try the Cluster Administration Quiz.

Traffic Management

Security

Observability

Resilience

Advanced

Comparison Guide

ArgoCD

Metrics

Logging

Tracing

Alerting

Observability End-to-End Labs

Kubernetes Cluster Administration ​

Core Concepts ​

Table of Contents ​

Environment Setup ​

Cluster Administration Overview ​

Cluster Component Management ​

Control Plane Component Management ​

API Server Management ​

etcd Management ​

Node Management ​

Component Status Monitoring ​

Cluster Administration Tools ​

Cluster Component Management ​

Control Plane Components ​

Control Plane Component Monitoring ​

Control Plane Component Configuration ​

Node Components ​

Node Management ​

Node Troubleshooting ​

Resource Management ​

Namespace Management ​

Resource Quotas ​

Limit Ranges ​

Horizontal Pod Autoscaler (HPA) ​

Vertical Pod Autoscaler (VPA) ​

Cluster Networking ​

Cluster Network Model ​

CNI (Container Network Interface) Plugins ​

CNI Plugin Installation and Configuration ​

Service Networking ​

Service CIDR Configuration ​

CoreDNS Management ​

Network Policies ​

Authentication and Authorization Management ​

Authentication ​

X.509 Certificate Management ​

OIDC Authentication Configuration ​

Authorization ​

RBAC Configuration ​

ClusterRole and ClusterRoleBinding ​

Service Account Management ​

Security Context ​

Cluster Upgrades ​

Upgrade Planning ​

Control Plane Upgrade ​

Worker Node Upgrade ​

Upgrade Verification ​

Backup and Recovery ​

etcd Backup ​

etcd Recovery ​

Kubernetes Cluster Administration

Core Concepts

Table of Contents

Environment Setup

Cluster Administration Overview

Cluster Component Management

Control Plane Component Management

API Server Management

etcd Management

Node Management

Component Status Monitoring

Cluster Administration Tools

Cluster Component Management

Control Plane Components

Control Plane Component Monitoring

Control Plane Component Configuration

Node Components

Node Management

Node Troubleshooting

Resource Management

Namespace Management

Resource Quotas

Limit Ranges

Horizontal Pod Autoscaler (HPA)

Vertical Pod Autoscaler (VPA)

Cluster Networking

Cluster Network Model

CNI (Container Network Interface) Plugins

CNI Plugin Installation and Configuration

Service Networking

Service CIDR Configuration

CoreDNS Management

Network Policies

Authentication and Authorization Management

Authentication

X.509 Certificate Management

OIDC Authentication Configuration

Authorization

RBAC Configuration

ClusterRole and ClusterRoleBinding

Service Account Management

Security Context

Cluster Upgrades

Upgrade Planning

Control Plane Upgrade

Worker Node Upgrade

Upgrade Verification

Backup and Recovery

etcd Backup

etcd Recovery