Kubernetes Scheduling, Preemption, and Eviction

Supported Versions: Kubernetes 1.34 - 1.36 (Descheduler v0.36 example) Last Updated: September 17, 2026

In Kubernetes, scheduling is the process of placing pods on appropriate nodes. Preemption is the process of removing lower-priority pods to make room for higher-priority pods, and eviction terminates a Pod; its workload controller may create a replacement that the scheduler places separately. In this chapter, we will learn about Kubernetes scheduling mechanisms, node selection, preemption, eviction, and scheduling optimization methods in Amazon EKS.

Lab Environment Setup

To follow the examples in this document, you need the following tools and environment:

Required Tools

• kubectl within one minor version of the API server
• A working Kubernetes cluster (EKS, minikube, kind, etc.)
• A cluster with multiple nodes (for scheduling tests)

Scheduling Example Setup

# Create namespace
kubectl create namespace scheduling-demo

# Add labels to nodes (if you have multiple nodes)
kubectl label nodes <node-name> disktype=ssd
kubectl label nodes <node-name> gpu=true

# Create a pod using node affinity
kubectl -n scheduling-demo apply -f - <<EOF
apiVersion: v1
kind: Pod
metadata:
  name: nginx-ssd
  labels:
    app: nginx
spec:
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
        - matchExpressions:
          - key: disktype
            operator: In
            values:
            - ssd
  containers:
  - name: nginx
    image: nginx
EOF

# Create priority class
kubectl apply -f - <<EOF
apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: high-priority
value: 1000000
globalDefault: false
description: "This priority class should be used for critical service pods only."
EOF

# Create Pod Disruption Budget (PDB)
kubectl -n scheduling-demo apply -f - <<EOF
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: nginx-pdb
spec:
  minAvailable: 1
  selector:
    matchLabels:
      app: nginx
EOF

Kubernetes Scheduling Architecture

Kubernetes scheduling architecture: kube-scheduler runs pods through queueing, filtering, scoring and binding, constrained by placement policies, with priority-based preemption and eviction feeding back into the pipeline.Open full screen ↗

Scheduling Concept Comparison

Concept	Purpose	Use Cases	Kubernetes Version
Node Selector	Place pods on nodes with specific labels	Simple node selection	All versions
Node Affinity	Define complex node selection rules	Advanced node selection	1.6+
Pod Affinity	Place pods close to other pods	Co-locating related services	1.6+
Pod Anti-Affinity	Place pods away from other pods	Ensuring high availability	1.6+
Taints and Tolerations	Allow only specific pods on nodes	Dedicated nodes, node isolation	1.6+
Topology Spread Constraints	Spread pods across topology domains	Distribution across availability zones	1.16+ (GA in 1.19)
Priority and Preemption	Prioritize important workloads	Critical service guarantees	1.8+ (GA in 1.11)
Pod Disruption Budget	Limit simultaneously disrupted pods	Ensuring high availability	1.4+ (GA in 1.21)

Basic Scheduling Concepts

Key Concept: The Kubernetes scheduler is a control plane component that selects the optimal node to run pods, operating in two phases: filtering and scoring.

Scheduling Process

1. Filtering Phase (Predicates)

   • Identifies a suitable set of nodes that can run the pod
   • Considers resource requirements, node selectors, affinity rules, taints/tolerations, etc.
   • Excludes a node if any condition is not met

2. Scoring Phase (Priorities)

   • Assigns scores to nodes that passed filtering
   • Considers resource utilization, pod distribution, affinity preferences, etc.
   • Selects the node with the highest score

3. Binding Phase

   • Assigns the pod to the selected node
   • Updates binding information to the API server

Table of Contents

1. Scheduling Overview
2. How the Scheduler Works
3. Node Selection
4. Pod Affinity and Anti-Affinity
5. Taints and Tolerations
6. Node Affinity
7. Pod Priority and Preemption
8. Pod Eviction
9. Pod Disruption Budget (PDB)
10. Node Pressure Eviction
11. TopologySpreadConstraints
12. Pod Deletion Cost
13. Descheduler
14. Scheduling Optimization in Amazon EKS
15. Scheduling Best Practices
16. Conclusion

Scheduling Overview

The Kubernetes scheduler is a control plane component that places pods on appropriate nodes. The scheduler considers various factors to determine the optimal node to place pods:

1. Resource Requirements: CPU, memory, and other resources requested by the pod
2. Hardware/Software/Policy Constraints: Node selectors, node affinity, taints, etc.
3. Affinity/Anti-Affinity Specifications: Placement relationships with other pods
4. Data Locality: Placing pods close to data
5. Inter-Workload Interference: Minimizing interference between different workloads
6. Custom objectives: Deadline-aware or workload-interference-aware scheduling requires suitable custom logic; the default scheduler does not infer application deadlines

Scheduling Process

The scheduling process is broadly divided into two phases:

1. Filtering: Identifies a set of nodes that can run the pod

   • Checks whether resource requirements are met
   • Checks constraints such as node selectors, affinity, taints

2. Scoring: Scores filtered nodes to select the optimal node

   • Resource utilization balance
   • Inter-pod affinity/anti-affinity
   • Data locality
   • Taints/tolerations

How the Scheduler Works

The Kubernetes scheduler operates through the following process:

Pipeline diagram showing a pod creation event moving through the scheduling queue, the kube-scheduler, filter plugins, score plugins, best-node selection, and a binding request to the API server until the pod lands on a node.Open full screen ↗

1. Pod Queue Watching: The scheduler watches the API server for unscheduled pods.
2. Node Filtering: Identifies a set of nodes that can run the pod.
3. Node Scoring: Scores the filtered nodes.
4. Node Selection: Selects the node with the highest score.
5. Binding: Binds the pod to the selected node.

Scheduling Plugins

The Kubernetes scheduler is designed to be extensible using a plugin architecture. Various plugins operate at different stages of the scheduling process:

1. Filter Plugins: Filter out nodes where the pod cannot run

   • NodeResourcesFit: Checks node resource capacity
   • NodeName: Checks the pod's nodeName field
   • NodeUnschedulable: Checks node schedulability
   • TaintToleration: Checks taints and tolerations

2. Score Plugins: Assign scores to nodes

   • NodeResourcesBalancedAllocation: Considers resource usage balance
   • ImageLocality: Considers image locality
   • InterPodAffinity: Considers inter-pod affinity
   • NodeAffinity: Considers node affinity

NodeResourcesFit Scoring Strategy: LeastAllocated vs. MostAllocated

NodeResourcesFit is both a filter plugin (does the node have enough allocatable CPU/memory for the pod?) and a score plugin. Its score plugin behavior is controlled by scoringStrategy.type, configured through KubeSchedulerConfiguration:

• LeastAllocated (default): scores nodes higher the less allocated they already are. New pods are pulled toward the emptiest nodes, spreading load evenly.
• MostAllocated: scores nodes higher the more allocated they already are (as long as the pod still fits). New pods are packed onto the busiest nodes first, leaving other nodes untouched or empty.
• RequestedToCapacityRatio: a configurable curve between the two, useful for extended resources like GPUs where you want a custom shape rather than a pure min/max.

apiVersion: kubescheduler.config.k8s.io/v1
kind: KubeSchedulerConfiguration
profiles:
- schedulerName: most-allocated-scheduler
  pluginConfig:
  - name: NodeResourcesFit
    args:
      scoringStrategy:
        type: MostAllocated
        resources:
        - name: cpu
          weight: 1
        - name: memory
          weight: 1

Amazon EKS's control plane is fully managed, so you cannot edit the default kube-scheduler's KubeSchedulerConfiguration directly. To use MostAllocated, run a second scheduler (a Deployment running the kube-scheduler binary with this config, bound to system:kube-scheduler/system:volume-scheduler via a dedicated ServiceAccount) and point specific pods at it with spec.schedulerName, as shown in Multiple Schedulers below — the same pattern used in Building a Custom Scheduler.

Verified test — spread vs. bin-packing behavior. To confirm the practical difference, a disposable 3-worker kind cluster (kubectl version v1.37.0, isolated, torn down after the test — no shared infrastructure was touched) was seeded with uneven baseline load using nodeName-pinned filler pods (16 CPU allocatable per node): worker=12 pods (75%), worker2=6 pods (37%), worker3=0 pods (0%). Six identical 1-CPU pods were then scheduled twice — once with the default scheduler (LeastAllocated) and once with a second scheduler configured with MostAllocated — against the same starting state:

Scheduler / strategy	Where the 6 new pods landed	Final CPU utilization (worker / worker2 / worker3)
Default (LeastAllocated)	All 6 on worker3 (the emptiest node)	75% / 37% / 37% — all three nodes now occupied
MostAllocated	3 on worker, 3 on worker2 (the two busiest nodes)	93% / 56% / 0% — worker3 stayed completely idle

LeastAllocated leveled worker3 up until it matched worker2, so every node ends up partially used and none of them can be safely scaled down. MostAllocated kept concentrating load on the already-busy nodes and left worker3 untouched — exactly the node a cluster autoscaler or Karpenter consolidation pass would then terminate.

This is why MostAllocated is commonly recommended for batch/short-lived job workloads running alongside Karpenter or Cluster Autoscaler: bin-packing jobs onto fewer nodes maximizes the number of nodes that go fully idle and become eligible for consolidation, directly reducing compute cost. LeastAllocated remains the better default for long-running, latency-sensitive services, since spreading load leaves headroom on every node to absorb traffic bursts or a kubectl drain without cascading pressure onto a single packed node.

Reproduce this exact comparison yourself, step by step, in Scheduler Scoring Strategy Lab.

Multiple Schedulers

Kubernetes can run multiple schedulers simultaneously. This allows implementing custom scheduling logic for specific workloads.

apiVersion: v1
kind: Pod
metadata:
  name: custom-scheduled-pod
spec:
  schedulerName: my-custom-scheduler
  containers:
  - name: container
    image: nginx

In the example above, the schedulerName field specifies the scheduler to schedule the pod.

Node Selection

Kubernetes provides several mechanisms to place pods on specific nodes.

Diagram comparing three node-placement mechanisms: nodeSelector matching a node label, nodeName pinning to a specific node, and nodeAffinity evaluating an expression against candidate zones.Open full screen ↗

Node Selector

Node selector is the simplest way to restrict pods to only be placed on nodes with specific labels.

apiVersion: v1
kind: Pod
metadata:
  name: gpu-pod
spec:
  nodeSelector:
    gpu: "true"
  containers:
  - name: gpu-container
    image: busybox:1.36
    command: ["sh", "-c", "sleep 3600"]
    resources:
      limits:
        nvidia.com/gpu: 1

In the example above, the pod is only placed on nodes with the gpu=true label.

The GPU example tests scheduling only: the node must actually have a GPU and a working device plugin that advertises nvidia.com/gpu. A gpu=true label alone does not allocate GPU resources.

nodeName

You can use the nodeName field to directly place a pod on a specific node. This method bypasses the scheduler and is generally not recommended.

apiVersion: v1
kind: Pod
metadata:
  name: specific-node-pod
spec:
  nodeName: worker-node-1
  containers:
  - name: container
    image: nginx

In the example above, the pod is directly placed on the node named worker-node-1.

Pod Affinity and Anti-Affinity

Pod affinity and anti-affinity provide ways to place pods based on relationships between pods.

Diagram contrasting pod affinity, which co-locates a web pod with a cache pod on the same node, against pod anti-affinity, which separates two web pod replicas across different nodes, both configurable as hard or soft requirements.Open full screen ↗

Pod Affinity

Pod affinity causes pods to be placed on the same node or topology domain as pods with specific labels.

apiVersion: v1
kind: Pod
metadata:
  name: frontend
spec:
  affinity:
    podAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
      - labelSelector:
          matchExpressions:
          - key: app
            operator: In
            values:
            - cache
        topologyKey: kubernetes.io/hostname
  containers:
  - name: frontend
    image: nginx

In the example above, the frontend pod is placed on the same host as pods with the app=cache label.

Pod Anti-Affinity

Pod anti-affinity causes pods to be placed on a different node or topology domain than pods with specific labels.

apiVersion: v1
kind: Pod
metadata:
  name: frontend
  labels:
    app: frontend
spec:
  affinity:
    podAntiAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
      - labelSelector:
          matchExpressions:
          - key: app
            operator: In
            values:
            - frontend
        topologyKey: kubernetes.io/hostname
  containers:
  - name: frontend
    image: nginx

In the example above, the frontend pod is placed on a different host than other pods with the app=frontend label. This is useful for distributing instances of the same application across multiple nodes for high availability.

Affinity Types

Pod affinity and anti-affinity have two types:

1. requiredDuringSchedulingIgnoredDuringExecution: Hard requirement that must be met during scheduling
2. preferredDuringSchedulingIgnoredDuringExecution: Soft requirement that is preferred but not required

# preferredDuringSchedulingIgnoredDuringExecution example
affinity:
  podAffinity:
    preferredDuringSchedulingIgnoredDuringExecution:
    - weight: 100
      podAffinityTerm:
        labelSelector:
          matchExpressions:
          - key: app
            operator: In
            values:
            - cache
        topologyKey: kubernetes.io/hostname

In the example above, the weight field indicates the weight of this preference. When there are multiple preferences, the higher weight preferences are considered more important.

Taints and Tolerations

Taints and tolerations are mechanisms that allow nodes to reject specific pods.

Diagram showing a node taint rejecting pods unless they carry a matching toleration, the three taint effects NoSchedule, PreferNoSchedule and NoExecute, and a worked example where a GPU node tainted key=gpu:NoSchedule rejects a regular pod but admits a GPU pod with a matching toleration.Open full screen ↗

Taints

Taints are applied to nodes to restrict pods from being scheduled on them.

# Add taint to node
kubectl taint nodes node1 key=value:NoSchedule

There are three taint effects:

1. NoSchedule: Pods without tolerations are not scheduled on the node
2. PreferNoSchedule: Prefer not to schedule pods without tolerations on the node
3. NoExecute: Pods without tolerations are evicted from the node

Tolerations

Tolerations are applied to pods to allow them to be scheduled on nodes with taints.

apiVersion: v1
kind: Pod
metadata:
  name: nginx
spec:
  tolerations:
  - key: "key"
    operator: "Equal"
    value: "value"
    effect: "NoSchedule"
  containers:
  - name: nginx
    image: nginx

In the example above, the pod can be scheduled on nodes with the key=value:NoSchedule taint.

Use Cases

Common use cases for taints and tolerations:

1. Dedicated Nodes: Designate nodes to run only specific workloads
2. Special Hardware: Manage nodes with special hardware like GPUs
3. Node Maintenance: Prevent new pod scheduling on nodes under maintenance
4. Node Issues: Evict pods from nodes with issues

Default Taints

Kubernetes applies default taints to some nodes:

• node.kubernetes.io/not-ready: Node is not ready
• node.kubernetes.io/unreachable: Node is unreachable
• node.kubernetes.io/memory-pressure: Node has memory pressure
• node.kubernetes.io/disk-pressure: Node has disk pressure
• node.kubernetes.io/pid-pressure: Node has PID pressure
• node.kubernetes.io/network-unavailable: Node network is unavailable
• node.kubernetes.io/unschedulable: Node is unschedulable

Node Affinity

Node affinity provides a more expressive way to place pods on specific sets of nodes. It allows specifying more complex conditions than node selector.

Node Affinity Types

Node affinity has two types:

1. requiredDuringSchedulingIgnoredDuringExecution: Hard requirement that must be met during scheduling
2. preferredDuringSchedulingIgnoredDuringExecution: Soft requirement that is preferred but not required

apiVersion: v1
kind: Pod
metadata:
  name: with-node-affinity
spec:
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
        - matchExpressions:
          - key: topology.kubernetes.io/zone
            operator: In
            values:
            - us-west-2a
            - us-west-2b
      preferredDuringSchedulingIgnoredDuringExecution:
      - weight: 1
        preference:
          matchExpressions:
          - key: another-node-label-key
            operator: In
            values:
            - another-node-label-value
  containers:
  - name: with-node-affinity
    image: nginx

In the example above, the pod is only placed on nodes where the topology.kubernetes.io/zone label is us-west-2a or us-west-2b. Additionally, it is preferably placed on nodes with the another-node-label-key=another-node-label-value label.

Operators

Node affinity supports various operators:

• In: Label value matches one of the specified values
• NotIn: Label value does not match the specified values
• Exists: A label with the specified key exists
• DoesNotExist: A label with the specified key does not exist
• Gt: Label value is greater than the specified value
• Lt: Label value is less than the specified value

Pod Priority and Preemption

Kubernetes provides pod priority and preemption features to ensure important workloads can secure cluster resources.

Diagram showing a PriorityClass assigning priority to a pod, triggering preemption of lower-priority pods when resources are insufficient, alongside the four-step preemption process from scheduling failure to scheduling the higher-priority pod and example built-in priority classes.Open full screen ↗

PriorityClass

PriorityClass defines the relative importance of pods. The higher the priority value, the more important the pod.

apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: high-priority
value: 1000000
globalDefault: false
description: "This priority class should be used for critical workloads."

In the example above, the value field indicates the priority value. The higher the value, the higher the priority. If the globalDefault field is set to true, this priority class is applied to pods without a specified priority class.

Applying PriorityClass to Pods

To apply a priority class to a pod, use the priorityClassName field.

apiVersion: v1
kind: Pod
metadata:
  name: high-priority-pod
spec:
  priorityClassName: high-priority
  containers:
  - name: container
    image: nginx

Preemption

Preemption is the process of removing lower-priority pods to schedule higher-priority pods. When the scheduler cannot find a node to schedule a higher-priority pod, it preempts lower-priority pods to secure resources.

Preemption process:

1. Scheduler cannot find a node to schedule a higher-priority pod
2. Scheduler selects a node to remove lower-priority pods through preemption
3. Requests deletion of selected lower-priority Pods through the API; kubelet/runtime perform termination
4. When pods terminate gracefully, schedules the higher-priority pod on that node

Preemption Considerations

Things to consider when using preemption:

1. Graceful Termination Period: Preempted pods go through the graceful termination process for the time specified in terminationGracePeriodSeconds
2. PodDisruptionBudget: The scheduler tries to avoid violations, but preemption may violate a PDB when no suitable victims avoid it
3. System Priority Classes: Kubernetes provides priority classes for system components
   • system-cluster-critical: Pods critical for cluster operation
   • system-node-critical: Pods critical for node operation

Pod Eviction

Pod eviction terminates a Pod; its workload controller may create a replacement that the scheduler places separately. Eviction can happen for various reasons.

Diagram grouping pod eviction into three sources -- the controller manager evicting pods from NotReady or Unreachable nodes, kubelet evicting pods on resource shortage or hardware issues while monitoring the memory, nodefs, imagefs, and pid eviction signals, and users draining nodes for maintenance.Open full screen ↗

Eviction Types

1. Eviction by kube-controller-manager:

   • The taint-eviction-controller handles NoExecute taints. Pods normally receive 300-second not-ready/unreachable tolerations; eviction follows their toleration settings
   • When a node is in Unreachable state

2. Eviction by kubelet:

   • Node resource shortage (memory, disk, etc.)
   • Hardware failures can lead to node unavailability; they are not a generic kubelet pressure-eviction signal

3. Eviction by user:

   • Executing kubectl drain command
   • Node maintenance tasks

kubelet Eviction Signals

kubelet monitors the following eviction signals:

1. memory.available: Available memory
2. nodefs.available: Available space in the node file system
3. nodefs.inodesFree: Available inodes in the node file system
4. imagefs.available: Available space in the image file system
5. imagefs.inodesFree: Available inodes in the image file system
6. pid.available: Available process IDs

Soft and hard thresholds can be set for each signal:

• Soft Threshold: Evict pods after grace-period when threshold is exceeded
• Hard Threshold: Evict pods immediately when threshold is exceeded

# kubelet configuration example
evictionHard:
  memory.available: "100Mi"
  nodefs.available: "10%"
  nodefs.inodesFree: "5%"
  imagefs.available: "15%"
  imagefs.inodesFree: "5%"
evictionSoft:
  memory.available: "200Mi"
  nodefs.available: "15%"
evictionSoftGracePeriod:
  memory.available: "1m"
  nodefs.available: "2m"
evictionMaxPodGracePeriod: 30
evictionPressureTransitionPeriod: "30s"

Eviction Priority

The kubelet ranks candidates by whether usage exceeds requests, then Pod priority, then usage relative to requests. It does not simply evict all BestEffort, then all Burstable, then all Guaranteed Pods. Disk/PID pressure has different accounting constraints; QoS is not a universal eviction ordering.

Pod Disruption Budget (PDB)

Pod Disruption Budget (PDB) is a way to maintain application availability during voluntary disruptions. PDB limits the number of pods that can be simultaneously disrupted.

Diagram showing a PodDisruptionBudget's minAvailable, maxUnavailable, and selector settings gating a voluntary disruption such as node drain, allowing or denying eviction, with an example deployment where equivalent minAvailable and maxUnavailable settings produce the same effect.Open full screen ↗

PDB Definition

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: frontend-pdb
spec:
  minAvailable: 2
  selector:
    matchLabels:
      app: frontend

or

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: frontend-pdb
spec:
  maxUnavailable: 1
  selector:
    matchLabels:
      app: frontend

In the examples above:

• minAvailable: Minimum number of pods that must always be available
• maxUnavailable: Maximum number of pods that can be unavailable at the same time
• selector: Label selector that selects pods to which the PDB applies

PDB Operation

1. When voluntary disruptions like node drain occur, Kubernetes checks the PDB
2. If PDB conditions are met, proceed with pod eviction
3. If PDB conditions are not met, deny pod eviction

PDBs gate Eviction API requests such as normal drain/descheduler operations. Direct Pod deletion, controller rollouts, and node-pressure eviction bypass this gate. minAvailable: 2 and maxUnavailable: 1 are equivalent only for a workload with three desired replicas; neither creates replacement capacity.

PDB Best Practices

1. Set PDB for all critical workloads: Set PDB for all workloads requiring high availability
2. Choose appropriate values: Select minAvailable or maxUnavailable values appropriate for workload characteristics
3. Consider replica count: minAvailable may equal replicas to block voluntary evictions, but maintenance can then stall; configure a deliberate disruption allowance
4. Regular testing: Test PDB operation through node drain and similar tasks

Node Pressure Eviction

Node pressure eviction is a mechanism where pods are evicted due to node resource shortage.

Node Condition Status

kubelet reports the following node condition statuses:

1. MemoryPressure: Node is low on memory
2. DiskPressure: Node is low on disk space
3. PIDPressure: Node is low on process IDs

When these conditions occur, kubelet evicts pods to secure resources.

Eviction Policy Configuration

Eviction policies can be set in kubelet configuration:

# kubelet configuration example
evictionHard:
  memory.available: "100Mi"
  nodefs.available: "10%"
  nodefs.inodesFree: "5%"
  imagefs.available: "15%"
  imagefs.inodesFree: "5%"
evictionSoft:
  memory.available: "200Mi"
  nodefs.available: "15%"
evictionSoftGracePeriod:
  memory.available: "1m"
  nodefs.available: "2m"
evictionMinimumReclaim:
  memory.available: "50Mi"
  nodefs.available: "5%"
evictionMaxPodGracePeriod: 30
evictionPressureTransitionPeriod: "30s"

In the example above:

• evictionMinimumReclaim: Minimum resources that must be reclaimed after eviction
• evictionPressureTransitionPeriod: Wait time between pressure state transitions

TopologySpreadConstraints

TopologySpreadConstraints provide fine-grained control over how pods are distributed across topology domains such as availability zones, nodes, or regions. This feature offers more flexibility than Pod anti-affinity for achieving high availability and efficient resource utilization.

Diagram showing TopologySpreadConstraints controlling pod spread across availability zones through maxSkew, topologyKey, whenUnsatisfiable and the usual labelSelector, the DoNotSchedule and ScheduleAnyway options of whenUnsatisfiable, and an EKS example where a new pod with maxSkew=1 lands in ap-northeast-2b, the zone holding the fewest pods.Open full screen ↗

Key Fields

Field	Description	Required
maxSkew	For DoNotSchedule, allowed difference between a target domain and the global minimum; ScheduleAnyway uses skew as a preference	Yes
topologyKey	Node label key that defines topology domains	Yes
whenUnsatisfiable	Action when constraints cannot be satisfied: DoNotSchedule or ScheduleAnyway	Yes
labelSelector	Selects Pods to count; normally specify it and matching Pod labels	No (null matches no Pods)
minDomains	Minimum eligible-domain count for skew calculation (stable since v1.30)	No
matchLabelKeys	Pod label keys to match for spread calculation (1.27+)	No

whenUnsatisfiable Options

• DoNotSchedule: Scheduler will not schedule the pod if the constraint cannot be satisfied (hard constraint)
• ScheduleAnyway: Scheduler still schedules the pod, giving higher priority to nodes that minimize skew (soft constraint)

EKS Availability Zone Spread Example

apiVersion: apps/v1
kind: Deployment
metadata:
  name: web-app
spec:
  replicas: 6
  selector:
    matchLabels:
      app: web
  template:
    metadata:
      labels:
        app: web
    spec:
      topologySpreadConstraints:
      - maxSkew: 1
        topologyKey: topology.kubernetes.io/zone
        whenUnsatisfiable: DoNotSchedule
        labelSelector:
          matchLabels:
            app: web
      - maxSkew: 1
        topologyKey: kubernetes.io/hostname
        whenUnsatisfiable: ScheduleAnyway
        labelSelector:
          matchLabels:
            app: web
      containers:
      - name: web
        image: nginx:1.30.4
        resources:
          requests:
            cpu: 100m
            memory: 128Mi

This configuration ensures:

1. Pods are evenly distributed across availability zones (hard constraint)
2. Pods are preferably distributed across nodes within each zone (soft constraint)

minDomains and matchLabelKeys

apiVersion: apps/v1
kind: Deployment
metadata:
  name: app-with-min-domains
spec:
  replicas: 4
  selector:
    matchLabels:
      app: distributed-app
  template:
    metadata:
      labels:
        app: distributed-app
        version: v1
    spec:
      topologySpreadConstraints:
      - maxSkew: 1
        topologyKey: topology.kubernetes.io/zone
        whenUnsatisfiable: DoNotSchedule
        labelSelector:
          matchLabels:
            app: distributed-app
        minDomains: 3
        matchLabelKeys:
        - version
      containers:
      - name: app
        image: myapp:v1

• minDomains: If fewer than 3 eligible domains exist, the global minimum becomes zero. With maxSkew 1, one matching Pod per eligible domain can still schedule; further Pods can remain Pending. It does not block every Pod immediately.
• matchLabelKeys: Automatically uses the pod's version label value in the selector, enabling per-revision spread without modifying the selector.

Advantages Over Pod Anti-Affinity

Aspect	TopologySpreadConstraints	Pod Anti-Affinity
Flexibility	Allows controlled skew (maxSkew > 1)	Binary: either same or different domain
Soft constraints	ScheduleAnyway for best-effort	preferredDuringScheduling but less control
Multi-level	Multiple constraints with different topologyKeys	Requires complex nested rules
Performance	Better scheduler performance at scale	Can slow scheduling with many pods
Use case	Even distribution with tolerance	Strict separation

Pod Deletion Cost

Pod Deletion Cost is a best-effort preference used by the ReplicaSet controller when scaling down. HPA changes the desired replica count; it does not choose individual victim Pods. The annotation does not protect Jobs/StatefulSets, prevent eviction, or guarantee deletion order.

How It Works

When a controller (like HPA or manual scale-down) needs to reduce replicas, it considers:

1. Pods with lower deletion cost are removed first
2. Default deletion cost is 0
3. Valid range: -2147483648 to 2147483647

Basic Example

apiVersion: v1
kind: Pod
metadata:
  name: worker-pod
  annotations:
    controller.kubernetes.io/pod-deletion-cost: "100"
spec:
  containers:
  - name: worker
    image: worker:latest

HPA Scale-Down Priority Control

Use deletion cost to protect important pods during HPA scale-down:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: web-service
spec:
  replicas: 5
  selector:
    matchLabels:
      app: web
  template:
    metadata:
      labels:
        app: web
      # Lower cost pods are deleted first during scale-down
      annotations:
        controller.kubernetes.io/pod-deletion-cost: "0"
    spec:
      containers:
      - name: web
        image: nginx:1.30.4

Cache Protection Pattern

Run the cache with explicit CPU requests if it will be scaled using CPU-utilization HPA. The following is a scheduling example, not a complete Redis production configuration:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: cache-service
spec:
  replicas: 3
  selector:
    matchLabels:
      app: cache
  template:
    metadata:
      labels:
        app: cache
    spec:
      automountServiceAccountToken: false
      containers:
      - name: cache
        image: redis:7
        resources:
          requests:
            cpu: 100m
            memory: 128Mi

After measuring actual cache warmth, an authorized operator/controller can annotate a selected ReplicaSet-owned Pod once before scale-down:

kubectl -n default annotate pod "$CACHE_POD" \
  controller.kubernetes.io/pod-deletion-cost="1000" --overwrite

Set CACHE_POD to an actual cache Pod. Frequent annotation writes create API load. A custom updater would need both Redis and Kubernetes client tooling plus narrowly scoped Pod patch permissions; elapsed time alone is not proof of a warm cache. No updater is installed by this example.

Practical Use Cases

1. Stateful caches managed by a ReplicaSet: Prefer retaining warm replicas
2. Leader election: Keep leader pods running longer
3. Connection draining: Give time for long-running connections
4. Cache warming: Preserve pods with warm caches
5. Limitations: Job and StatefulSet controllers do not use this preference

Descheduler

The Descheduler is a Kubernetes component that evicts pods from nodes to allow the scheduler to reschedule them to more appropriate nodes. Unlike the scheduler which only places new pods, the descheduler helps maintain optimal pod placement over time.

Diagram showing how the Descheduler restores balance when node additions or removals or pod changes break an evenly distributed cluster, by evicting running pods so the scheduler re-places them, alongside six representative Descheduler strategies such as RemoveDuplicates, LowNodeUtilization, and PodLifeTime.Open full screen ↗

Why Descheduling Is Needed

1. Cluster changes: New nodes added, node labels changed
2. Pod drift: Initial placement becomes suboptimal over time
3. Affinity violations: Rules violated after cluster changes
4. Resource imbalance: Some nodes overutilized, others underutilized
5. Failed pods: Pods stuck in restart loops

Key Strategies

Strategy	Description	Use Case
RemoveDuplicates	Removes duplicate pods from the same node	Ensure HA after node failures
LowNodeUtilization	Moves pods from overutilized to underutilized nodes	Balance cluster resources
RemovePodsHavingTooManyRestarts	Evicts pods with excessive restarts	Clean up problematic pods
PodLifeTime	Evicts pods older than specified age	Force fresh scheduling
RemovePodsViolatingInterPodAntiAffinity	Evicts pods violating anti-affinity rules	Restore affinity compliance
RemovePodsViolatingNodeAffinity	Evicts pods violating node affinity	Restore affinity compliance
RemovePodsViolatingTopologySpreadConstraint	Evicts pods violating spread constraints	Restore even distribution

Helm Installation

Descheduler v0.36.0 is the verified example release, targeting Kubernetes v1.36 and the preceding two minor versions in its test window. Check the compatibility matrix before applying it to another release. Save reviewed Helm values in descheduler-values.yaml with schedule and deschedulerPolicy.profiles (the policy profiles shown below); the old strategies.*.enabled values do not configure this API.

helm repo add descheduler https://kubernetes-sigs.github.io/descheduler/
helm upgrade --install descheduler descheduler/descheduler \
  --version 0.36.0 --namespace kube-system \
  --values descheduler-values.yaml

DeschedulerPolicy Configuration

apiVersion: descheduler/v1alpha2
kind: DeschedulerPolicy
profiles:
- name: default
  pluginConfig:
  - name: DefaultEvictor
    args:
      nodeFit: true
  - name: RemoveDuplicates
    args:
      excludeOwnerKinds: [StatefulSet]
  - name: LowNodeUtilization
    args:
      thresholds:
        cpu: 20
        memory: 20
        pods: 20
      targetThresholds:
        cpu: 50
        memory: 50
        pods: 50
  - name: RemovePodsHavingTooManyRestarts
    args:
      podRestartThreshold: 100
      includingInitContainers: true
  - name: PodLifeTime
    args:
      maxPodLifeTimeSeconds: 86400
      labelSelector:
        matchLabels:
          app.kubernetes.io/lifecycle: ephemeral
  - name: RemovePodsViolatingNodeAffinity
    args:
      nodeAffinityType: [requiredDuringSchedulingIgnoredDuringExecution]
  - name: RemovePodsViolatingTopologySpreadConstraint
    args:
      constraints: [DoNotSchedule]
  plugins:
    balance:
      enabled:
      - RemoveDuplicates
      - LowNodeUtilization
      - RemovePodsViolatingTopologySpreadConstraint
    deschedule:
      enabled:
      - RemovePodsHavingTooManyRestarts
      - PodLifeTime
      - RemovePodsViolatingNodeAffinity

PDB Respect

The descheduler respects Pod Disruption Budgets (PDBs). If evicting a pod would violate a PDB, the descheduler will not evict that pod:

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: web-pdb
spec:
  minAvailable: 2
  selector:
    matchLabels:
      app: web

With this PDB in place, the descheduler will ensure at least 2 pods with app: web label remain available during descheduling operations.

The policy above is a Descheduler configuration file, not an API object for kubectl apply. It uses Balance plugins for group redistribution and Deschedule plugins for per-Pod decisions. LowNodeUtilization normally evaluates resource requests rather than live CPU usage, and eviction does not guarantee the replacement Pod lands elsewhere. Review safeguards and test in dry-run before enabling recurring eviction.

Descheduler CronJob Example

apiVersion: batch/v1
kind: CronJob
metadata:
  name: descheduler
  namespace: kube-system
spec:
  schedule: "*/30 * * * *"
  concurrencyPolicy: Forbid
  jobTemplate:
    spec:
      template:
        spec:
          serviceAccountName: descheduler
          containers:
          - name: descheduler
            image: registry.k8s.io/descheduler/descheduler:v0.36.0
            args:
            - --policy-config-file=/policy/policy.yaml
            - --v=3
            volumeMounts:
            - name: policy
              mountPath: /policy
          volumes:
          - name: policy
            configMap:
              name: descheduler-policy
          restartPolicy: OnFailure

The standalone CronJob is an alternative to Helm, not an additional installation. It requires the descheduler ServiceAccount/RBAC and a descheduler-policy ConfigMap with key policy.yaml; use the official chart/manifests to supply these prerequisites.

Deep Dive: For detailed information on custom schedulers, see:

• Custom Scheduler Part 1: Basic Concepts
• Custom Scheduler Part 2: Implementation
• Custom Scheduler Part 3: Advanced Features

Scheduling Optimization in Amazon EKS

In Amazon EKS, you can optimize workloads using Kubernetes scheduling features.

Diagram showing four EKS scheduling optimization levers -- node group and instance type choice, availability-zone distribution, Karpenter auto scaling, and resource request and limit tuning -- each connected to the mechanism or automation tool that implements it: Cluster Autoscaler, multi-AZ deployment, Karpenter NodePool, and Vertical Pod Autoscaler.Open full screen ↗

Node Groups and Instance Types

In EKS, you can provide resources appropriate for workloads by utilizing various node groups and instance types:

1. Various Instance Types: Compute optimized, memory optimized, storage optimized, etc.
2. Spot Instances: Spot instances for cost-effective workloads
3. GPU Instances: GPU instances for AI/ML workloads

You can use node labels and taints to place specific workloads on specific node groups:

Use a reviewed eksctl configuration for the existing cluster, with matching Region, supported GPU instance/AMI, and required IAM permissions:

# gpu-nodegroup.yaml
apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig
metadata:
  name: my-cluster
  region: us-west-2
managedNodeGroups:
- name: gpu-nodes
  instanceType: p3.2xlarge
  desiredCapacity: 1
  privateNetworking: true
  labels:
    workload-type: gpu
  taints:
  - key: gpu
    value: "true"
    effect: NoSchedule

eksctl create nodegroup --config-file=gpu-nodegroup.yaml

Availability Zone Distribution

In EKS, you can distribute workloads across multiple availability zones using pod anti-affinity and topology spread constraints:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: web-server
spec:
  replicas: 3
  selector:
    matchLabels:
      app: web
  template:
    metadata:
      labels:
        app: web
    spec:
      topologySpreadConstraints:
      - maxSkew: 1
        topologyKey: topology.kubernetes.io/zone
        whenUnsatisfiable: DoNotSchedule
        labelSelector:
          matchLabels:
            app: web
      containers:
      - name: web
        image: nginx

In the example above, topologySpreadConstraints distributes pods evenly across multiple availability zones.

Auto Scaling with Karpenter

In Amazon EKS, you can use Karpenter to automatically provision nodes appropriate for workloads:

apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: default
spec:
  template:
    spec:
      requirements:
        - key: karpenter.sh/capacity-type
          operator: In
          values: ["spot", "on-demand"]
        - key: kubernetes.io/arch
          operator: In
          values: ["amd64", "arm64"]
      nodeClassRef:
        group: karpenter.k8s.aws
        kind: EC2NodeClass
        name: default-class
  limits:
    cpu: 1000
    memory: 1000Gi
  disruption:
    consolidationPolicy: WhenEmpty
    consolidateAfter: 30s
---
apiVersion: karpenter.k8s.aws/v1
kind: EC2NodeClass
metadata:
  name: default-class
spec:
  role: KarpenterNodeRole-my-cluster
  amiSelectorTerms:
    - alias: al2023@latest
  subnetSelectorTerms:
    - tags:
        karpenter.sh/discovery: my-cluster
  securityGroupSelectorTerms:
    - tags:
        karpenter.sh/discovery: my-cluster

Karpenter optimizes costs by selecting the optimal instance type for pod resource requirements.

Resource Request and Limit Optimization

Optimizing workload resource requests and limits in EKS is important:

1. Vertical Pod Autoscaler (VPA): Optimize resource requests based on actual workload resource usage
2. Goldilocks: Visualize VPA recommendations to support resource request optimization
3. Resource Quotas: Limit resource usage per namespace

# VPA example
apiVersion: autoscaling.k8s.io/v1
kind: VerticalPodAutoscaler
metadata:
  name: frontend-vpa
spec:
  targetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: frontend
  updatePolicy:
    updateMode: "Recreate"

Scheduling Best Practices

Best practices for optimizing scheduling in Kubernetes and EKS:

1. Set appropriate resource requests and limits:

   • Set resource requests based on actual workload resource usage
   • Set appropriate resource limits for important workloads
   • Use VPA to automatically optimize resource requests

2. Workload distribution:

   • Use pod anti-affinity to distribute important workloads across multiple nodes
   • Use topology spread constraints to distribute workloads across multiple availability zones
   • Use node affinity to place specific workloads on specific nodes

3. Node resource optimization:

   • Use various instance types to provide appropriate resources for workloads
   • Use spot instances for cost optimization
   • Use Karpenter for automatic node provisioning appropriate for workloads

4. PDB configuration:

   • Set PDB for important workloads
   • Select minAvailable or maxUnavailable values appropriate for workload characteristics
   • Regularly test PDB operation

5. Priority and preemption configuration:

   • Set high priority classes for important workloads
   • Use system-cluster-critical or system-node-critical priority classes for system components
   • Understand and test preemption impact

6. Node taints and tolerations:

   • Set dedicated nodes for specialized workloads
   • Apply taints to nodes under maintenance
   • Set appropriate tolerations

Conclusion

Kubernetes scheduling, preemption, and eviction mechanisms play important roles in efficiently managing cluster resources and maintaining workload availability. By understanding and utilizing these features, you can optimize and reliably operate workloads in Amazon EKS clusters.

Scheduling optimization is an ongoing process, and adjustments should be continuously made according to workload characteristics and cluster state. It is important to track cluster resource usage using monitoring tools and adjust scheduling policies as needed.

Quiz

To test what you learned in this chapter, try the Scheduling, Preemption, and Eviction Quiz.