Node Bootstrap

< Previous: Air-Gap Setup | Table of Contents | Next: GPU Integration >

Supported Versions: EKS 1.31+, nodeadm 0.1+ Last Updated: February 23, 2026

This document covers the process of bootstrapping on-premises servers as EKS Hybrid Nodes using nodeadm.

Bootstrap Workflow Overview

The following steps outline the complete node bootstrap process from IAM credential setup to a fully ready hybrid node.

Bootstrap Steps

1. Prepare IAM credentials — Create SSM Hybrid Activation or configure IAM Roles Anywhere
2. Download nodeadm — Download the CLI binary for your architecture
3. Run nodeadm install — Install Kubernetes components and dependencies
4. Write NodeConfig YAML — Configure cluster details, credentials, kubelet, and containerd
5. Install CA certificates — Add private registry CA certs to system trust store (if using a private registry)
6. Run nodeadm init — Initialize the node and register with EKS cluster
7. Install CNI — Deploy Cilium via Helm for pod networking
8. Verify registration — Confirm node shows Ready in kubectl get nodes

nodeadm CLI Download and Installation

nodeadm is the CLI tool for initializing and managing EKS Hybrid Nodes.

Step 1: Download nodeadm

# Download nodeadm (Linux x86_64)
curl -Lo nodeadm https://hybrid-assets.eks.amazonaws.com/releases/latest/bin/linux/amd64/nodeadm
chmod +x nodeadm
sudo mv nodeadm /usr/local/bin/

# For ARM64 architecture:
# curl -Lo nodeadm https://hybrid-assets.eks.amazonaws.com/releases/latest/bin/linux/arm64/nodeadm

# Check version
nodeadm version

Step 2: Run nodeadm install

The nodeadm install command installs Kubernetes components (kubelet, kubectl, etc.) and system dependencies. This must run before nodeadm init.

# Install with SSM credential provider
sudo nodeadm install 1.31 --credential-provider ssm

# Install with IAM Roles Anywhere credential provider
sudo nodeadm install 1.31 --credential-provider iam-ra

# Custom timeout for slow networks
sudo nodeadm install 1.31 --credential-provider ssm --timeout 20m0s

Note: Replace 1.31 with your target Kubernetes version. The version must match your EKS cluster version.

Installation File Paths

Component	Ubuntu/AL2023 Path	RHEL Path
kubelet	/usr/bin/kubelet	/usr/bin/kubelet
kubectl	/usr/bin/kubectl	/usr/bin/kubectl
SSM Agent	/snap/amazon-ssm-agent (Ubuntu) / systemd (AL2023)	/usr/bin/amazon-ssm-agent
containerd	/usr/bin/containerd	/usr/bin/containerd
nodeadm	/usr/local/bin/nodeadm	/usr/local/bin/nodeadm

Writing NodeConfig YAML

# nodeconfig.yaml
apiVersion: node.eks.aws/v1alpha1
kind: NodeConfig
spec:
  cluster:
    name: my-hybrid-cluster
    region: ap-northeast-2
    apiServerEndpoint: https://XXXXXXXXXXXXXXXX.gr7.ap-northeast-2.eks.amazonaws.com
    certificateAuthority: |
      -----BEGIN CERTIFICATE-----
      MIIDxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
      xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
      -----END CERTIFICATE-----
    cidr: 10.100.0.0/16  # Service CIDR

  # Credential method selection (SSM or IAM Roles Anywhere)
  hybrid:
    # Method 1: SSM Hybrid Activations
    ssm:
      activationCode: <activation-code>
      activationId: <activation-id>

    # Method 2: IAM Roles Anywhere (uncomment to use)
    # iamRolesAnywhere:
    #   nodeName: hybrid-node-001  # Must match certificate CN
    #   trustAnchorArn: arn:aws:rolesanywhere:ap-northeast-2:123456789012:trust-anchor/xxxxx
    #   profileArn: arn:aws:rolesanywhere:ap-northeast-2:123456789012:profile/xxxxx
    #   roleArn: arn:aws:iam::123456789012:role/EKSHybridNodeRole
    #   certificatePath: /etc/iam/pki/server.pem
    #   privateKeyPath: /etc/iam/pki/server.key

  kubelet:
    config:
      maxPods: 110
      shutdownGracePeriod: 30s
      shutdownGracePeriodCriticalPods: 10s
    flags:
      - --node-labels=node.kubernetes.io/instance-type=on-prem-gpu
      - --register-with-taints=eks.amazonaws.com/compute-type=hybrid:NoSchedule

  containerd:
    config: |
      version = 2

      [plugins."io.containerd.grpc.v1.cri".registry]
        config_path = "/etc/containerd/certs.d"

      # Private registry TLS configuration (uncomment and adjust for your registry)
      # [plugins."io.containerd.grpc.v1.cri".registry.configs."registry.internal.company.io".tls]
      #   ca_file = "/etc/ssl/certs/registry-ca.crt"
      # [plugins."io.containerd.grpc.v1.cri".registry.configs."registry.internal.company.io".auth]
      #   username = "pull-robot"
      #   password = "<token>"

Create SSM Hybrid Activation

# Create SSM Hybrid Activation
aws ssm create-activation \
  --default-instance-name "eks-hybrid-node" \
  --iam-role "service-role/AmazonEC2RunCommandRoleForManagedInstances" \
  --registration-limit 100 \
  --region ap-northeast-2 \
  --tags "Key=Environment,Value=Production" "Key=NodeType,Value=Hybrid"

# Enter the output ActivationCode and ActivationId in nodeconfig.yaml

IAM Roles Anywhere Setup (Alternative)

If using IAM Roles Anywhere instead of SSM, configure the trust anchor, profile, and certificates:

# Create Trust Anchor
TRUST_ANCHOR_ARN=$(aws rolesanywhere create-trust-anchor \
  --name "eks-hybrid-trust-anchor" \
  --source "sourceType=CERTIFICATE_BUNDLE,sourceData={x509CertificateData=$(cat ca.pem)}" \
  --enabled \
  --query 'trustAnchor.trustAnchorArn' --output text)

# Create Profile
PROFILE_ARN=$(aws rolesanywhere create-profile \
  --name "eks-hybrid-profile" \
  --role-arns arn:aws:iam::123456789012:role/EKSHybridNodeRole \
  --enabled \
  --query 'profile.profileArn' --output text)

echo "Trust Anchor ARN: $TRUST_ANCHOR_ARN"
echo "Profile ARN: $PROFILE_ARN"
# Enter these values in nodeconfig.yaml under spec.hybrid.iamRolesAnywhere

NodeConfig YAML for IAM Roles Anywhere:

spec:
  hybrid:
    iamRolesAnywhere:
      nodeName: hybrid-node-001  # Must match certificate CN
      trustAnchorArn: arn:aws:rolesanywhere:ap-northeast-2:123456789012:trust-anchor/xxxxx
      profileArn: arn:aws:rolesanywhere:ap-northeast-2:123456789012:profile/xxxxx
      roleArn: arn:aws:iam::123456789012:role/EKSHybridNodeRole
      certificatePath: /etc/iam/pki/server.pem
      privateKeyPath: /etc/iam/pki/server.key

Note: When using IAM Roles Anywhere, enable acceptRoleSessionName on the IAM RA profile and set MaxSessionDuration on the IAM role to at least 1 hour (recommended: 12 hours) to avoid frequent credential refresh.

Install CA Certificate on System (Private Registry)

If you use a private container registry with a self-signed or internal CA certificate, install the CA cert on each node:

# Install CA certificate on system (Ubuntu)
sudo cp ca.crt /usr/local/share/ca-certificates/registry-ca.crt
sudo update-ca-certificates

# RHEL/CentOS
sudo cp ca.crt /etc/pki/ca-trust/source/anchors/registry-ca.crt
sudo update-ca-trust extract

# Configure directory for containerd to find certificate
sudo mkdir -p /etc/containerd/certs.d/<REGISTRY_HOST>
cat <<EOF | sudo tee /etc/containerd/certs.d/<REGISTRY_HOST>/hosts.toml
server = "https://<REGISTRY_HOST>"

[host."https://<REGISTRY_HOST>"]
  capabilities = ["pull", "resolve"]
  ca = "/usr/local/share/ca-certificates/registry-ca.crt"
EOF

Node Initialization

Configuration Validation

It is recommended to validate the configuration file before initializing the node:

# Validate configuration (recommended before node initialization)
nodeadm config check --config-source file://nodeconfig.yaml

Run Initialization

# Initialize node using nodeadm
sudo nodeadm init -c file://nodeconfig.yaml

# Check initialization logs
sudo journalctl -u kubelet -f

# Check node status (from EKS cluster)
kubectl get nodes -o wide

Verify Node Registration

# Check node list
kubectl get nodes --show-labels

# Expected output:
# NAME                STATUS   ROLES    AGE   VERSION   LABELS
# ip-10-0-1-100       Ready    <none>   1d    v1.31.0   topology.kubernetes.io/zone=ap-northeast-2a
# ip-10-0-2-100       Ready    <none>   1d    v1.31.0   topology.kubernetes.io/zone=ap-northeast-2b
# hybrid-node-001     Ready    <none>   5m    v1.31.0   eks.amazonaws.com/compute-type=hybrid

# Check node details
kubectl describe node hybrid-node-001

# Filter Hybrid Nodes
kubectl get nodes -l eks.amazonaws.com/compute-type=hybrid

---

Automated Bootstrap with systemd

For large-scale deployments, you can configure a systemd service to automatically run nodeadm install and nodeadm init when a node boots. Marker files ensure that install only runs on the first boot and is skipped on subsequent boots.

Prerequisites

The following files must be pre-placed on the node before automatic bootstrap:

• /etc/eks/nodeconfig.yaml — NodeConfig configuration file
• /etc/eks/bootstrap.env — Bootstrap environment variables
• /usr/local/bin/nodeadm — nodeadm binary

Note: These files can be pre-placed via VM image builds (Packer, etc.), cloud-init, or configuration management tools (Ansible, etc.).

Environment Configuration File

# /etc/eks/bootstrap.env
K8S_VERSION="1.31"
CREDENTIAL_PROVIDER="ssm"          # ssm or iam-ra
NODECONFIG_PATH="/etc/eks/nodeconfig.yaml"

Bootstrap Script

#!/bin/bash
# /usr/local/bin/eks-hybrid-bootstrap.sh
set -euo pipefail

LOG_TAG="eks-hybrid-bootstrap"
MARKER_DIR="/var/lib/eks"
INSTALL_MARKER="${MARKER_DIR}/.nodeadm-installed"
INIT_MARKER="${MARKER_DIR}/.nodeadm-initialized"

# Load environment variables
source /etc/eks/bootstrap.env

log() { logger -t "$LOG_TAG" "$1"; echo "[$(date '+%H:%M:%S')] $1"; }

mkdir -p "$MARKER_DIR"

# --- install phase (first boot only) ---
if [ -f "$INSTALL_MARKER" ]; then
  log "nodeadm install already completed — skipping"
else
  log "Starting nodeadm install ${K8S_VERSION} (credential-provider: ${CREDENTIAL_PROVIDER})"
  nodeadm install "${K8S_VERSION}" --credential-provider "${CREDENTIAL_PROVIDER}"
  touch "$INSTALL_MARKER"
  log "nodeadm install completed"
fi

# --- init phase (first boot only) ---
if [ -f "$INIT_MARKER" ]; then
  log "nodeadm init already completed — skipping"
else
  log "Starting nodeadm init"
  nodeadm init -c "file://${NODECONFIG_PATH}"
  touch "$INIT_MARKER"
  log "nodeadm init completed — node registered with EKS cluster"
fi

sudo chmod +x /usr/local/bin/eks-hybrid-bootstrap.sh

systemd Service Unit

# /etc/systemd/system/eks-hybrid-bootstrap.service
[Unit]
Description=EKS Hybrid Node Bootstrap (install + init)
After=network-online.target
Wants=network-online.target
ConditionPathExists=!/var/lib/eks/.nodeadm-initialized

[Service]
Type=oneshot
EnvironmentFile=/etc/eks/bootstrap.env
ExecStart=/usr/local/bin/eks-hybrid-bootstrap.sh
RemainAfterExit=true
StandardOutput=journal
StandardError=journal

[Install]
WantedBy=multi-user.target

ConditionPathExists: The ! prefix means the service only runs when the file does not exist. Once init completes and creates the marker file, the service is automatically skipped on subsequent boots.

Enable the Service

sudo systemctl daemon-reload
sudo systemctl enable eks-hybrid-bootstrap.service

Verify Operation

# Check service status
sudo systemctl status eks-hybrid-bootstrap.service

# View bootstrap logs
sudo journalctl -u eks-hybrid-bootstrap.service

# Check marker files
ls -la /var/lib/eks/.nodeadm-*

Reinstallation

To start fresh from scratch, remove the marker files and reboot:

# Clean up existing state
sudo nodeadm uninstall
sudo rm -f /var/lib/eks/.nodeadm-installed /var/lib/eks/.nodeadm-initialized

# Reboot triggers automatic install + init
sudo reboot

Frequently Asked Questions

Q: What does each setting in the systemd unit file mean?

Setting	Meaning
Type=oneshot	Runs once at boot and exits
After=network-online.target	Runs only after network is fully ready
ConditionPathExists=!/var/lib/eks/.nodeadm-initialized	! prefix — runs only when the marker file does not exist
RemainAfterExit=true	Service stays in active state after process exits (allows status checks)
WantedBy=multi-user.target	Starts automatically during normal boot

Q: Do I need a new SSM activation code every time the node is rebooted?

No. The SSM Hybrid Activation activationCode/activationId is used only once during nodeadm init to register the SSM agent with AWS. After registration, the SSM agent renews its own credentials automatically, so activation codes are not needed on normal reboots.

However, if you run nodeadm uninstall, SSM artifacts are deleted and re-registration is required. You can reuse the same activation code if the registration-limit has not been reached.

Q: Does nodeadm init join the node to the cluster?

Yes. nodeadm init performs the following steps in order:

1. Generates kubelet configuration files (/etc/kubernetes/)
2. Registers SSM or IAM Roles Anywhere credentials
3. Starts the kubelet systemd service
4. kubelet registers (joins) the node with the EKS API server

In other words, nodeadm init is the actual cluster join command.

Q: Does SSM activation registration happen during install or init?

Phase	SSM-Related Action
nodeadm install --credential-provider ssm	Installs SSM Agent binary only
nodeadm init	Registers the SSM Agent with AWS using activationCode/activationId from nodeconfig.yaml

SSM activation (registration) happens during the init phase.

Q: How do I remove and re-register a node while keeping the SSM activation?

kubectl delete node <NODE_NAME> does not affect SSM registration (SSM operates at the OS level; node registration is at the Kubernetes level). If kubelet is still running, the node will automatically re-register:

# Remove node from cluster
kubectl delete node hybrid-node-001

# If kubelet is running, it auto-registers
# If stopped, restart manually
sudo systemctl restart kubelet

Q: After drain → delete → shutdown, will the node auto-register on reboot via systemd?

The node will re-register, but it is handled by the kubelet service itself, not the systemd bootstrap service:

1. nodeadm init installs kubelet as a systemd service
2. On reboot, kubelet starts automatically and re-registers with the API server
3. The bootstrap service is skipped because the marker file exists (this is the expected behavior)

# No need to delete marker files in this workflow
kubectl drain hybrid-node-001 --ignore-daemonsets --delete-emptydir-data
kubectl delete node hybrid-node-001
# Shutdown and reboot → kubelet auto-registers

Note: Only when nodeadm uninstall has been run should you delete the marker files and rely on the bootstrap service for reinstallation.

---

Cilium CNI Installation

Cilium is the AWS-supported CNI for EKS Hybrid Nodes. Hybrid nodes appear with status Not Ready until a CNI is installed. Amazon VPC CNI is not compatible with hybrid nodes.

Supported versions: Cilium v1.17.x and v1.18.x for all Kubernetes versions supported in Amazon EKS Helm repository: oci://public.ecr.aws/eks/cilium/cilium

Prerequisites:

• Kernel version: Cilium requires Linux kernel 5.10 or higher. Ubuntu 20.04 and RHEL 8 default kernels are below 5.10 — you must upgrade the kernel before installing Cilium v1.18.x.
• Hybrid nodes only: Cilium affinity must be set to run only on hybrid nodes (eks.amazonaws.com/compute-type: hybrid). Do not run Cilium on cloud nodes that use VPC CNI.
• IPAM settings are immutable: The clusterPoolIPv4PodCIDRList and clusterPoolIPv4MaskSize values cannot be changed after initial deployment. Plan your pod CIDR allocation carefully before installing.

Create Cilium Values YAML

# cilium-values.yaml
affinity:
  nodeAffinity:
    requiredDuringSchedulingIgnoredDuringExecution:
      nodeSelectorTerms:
      - matchExpressions:
        - key: eks.amazonaws.com/compute-type
          operator: In
          values:
          - hybrid

ipam:
  mode: cluster-pool
  operator:
    clusterPoolIPv4MaskSize: 25
    clusterPoolIPv4PodCIDRList:
    - <POD_CIDR>  # Same as your EKS cluster's remote pod networks

loadBalancer:
  serviceTopology: true

operator:
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
        - matchExpressions:
          - key: eks.amazonaws.com/compute-type
            operator: In
            values:
            - hybrid
  unmanagedPodWatcher:
    restart: false

envoy:
  enabled: false

kubeProxyReplacement: "false"

Install Cilium

helm install cilium oci://public.ecr.aws/eks/cilium/cilium \
  --version 1.18.3-0 \
  --namespace kube-system \
  --values cilium-values.yaml

Verify Installation

# Check Cilium pods are running
kubectl get pods -n kube-system -l app.kubernetes.io/name=cilium

# Nodes should now show Ready
kubectl get nodes -o wide

Cilium Upgrade

Procedure for upgrading Cilium to a new version:

# 1. Preflight check (validate compatibility before upgrade)
helm install cilium-preflight oci://public.ecr.aws/eks/cilium/cilium \
  --version NEW_VERSION \
  --namespace kube-system \
  --set preflight.enabled=true \
  --set agent=false --set operator.enabled=false

# 2. Upgrade while preserving existing values
helm upgrade cilium oci://public.ecr.aws/eks/cilium/cilium \
  --version NEW_VERSION \
  --namespace kube-system \
  --reuse-values

# 3. Verify status
kubectl get pods -n kube-system -l app.kubernetes.io/name=cilium

# 4. Rollback (if issues occur)
helm rollback cilium --namespace kube-system

Cilium Uninstall

Procedure for completely removing Cilium:

# 1. Helm uninstall
helm uninstall cilium --namespace kube-system

# 2. Delete CRDs
kubectl get crds -o name | grep cilium | xargs kubectl delete

# 3. On-disk cleanup (run on each node)
sudo rm -rf /var/run/cilium /var/lib/cilium /etc/cni/net.d/05-cilium.conflist
sudo rm -f /opt/cni/bin/cilium-cni

Calico Deprecation Notice

Note: Calico is no longer officially supported for EKS Hybrid Nodes and has been moved to the eks-hybrid-examples repository. For new deployments, Cilium is recommended. Existing Calico deployments will continue to work but with limited official support from AWS.

---

Bottlerocket Configuration

Bottlerocket is supported only in VMware vSphere environments (v1.37.0+) and only for x86_64 architecture. Bottlerocket does not use nodeadm and is bootstrapped via TOML-based configuration and user data.

SSM Hybrid Activation Configuration (settings.toml)

[settings.kubernetes]
cluster-name = "CLUSTER_NAME"
api-server = "API_SERVER_ENDPOINT"
cluster-certificate = "BASE64_CA_CERT"
service-cidr = "SERVICE_CIDR"

[settings.hybrid]
enable-credentials-file = true  # Required for Pod Identity

[settings.hybrid.ssm]
activation-id = "ACTIVATION_ID"
activation-code = "ACTIVATION_CODE"

IAM Roles Anywhere Configuration (settings.toml)

[settings.hybrid.iam-roles-anywhere]
trust-anchor-arn = "TRUST_ANCHOR_ARN"
profile-arn = "PROFILE_ARN"
role-arn = "ROLE_ARN"
node-name = "NODE_NAME"  # Must match certificate CN
certificate-path = "/PATH/TO/CERT"
private-key-path = "/PATH/TO/KEY"

VMware Deployment with govc

# Clone from VM template
govc vm.clone -vm "/PATH/TO/TEMPLATE" -ds="DATASTORE" \
  -on=false -template=false -folder=/FOLDER "VM_NAME"

# Configure user data
govc vm.change -dc="DC" -vm "VM_NAME" \
  -e guestinfo.userdata="${USER_DATA}" \
  -e guestinfo.userdata.encoding=gzip+base64

# Start VM
govc vm.power -on "VM_NAME"

Note: USER_DATA is the settings.toml content gzip-compressed and base64-encoded.

---

Webhook and Add-on Placement Guidance

Some EKS add-ons use webhooks that require the API server to reach pods directly. If your on-premises pod CIDR is not routable, these add-ons must run on cloud nodes only.

Cloud-Only Add-ons (Unroutable Pod CIDR)

Use nodeAffinity to restrict webhook-based add-ons to cloud nodes:

affinity:
  nodeAffinity:
    requiredDuringSchedulingIgnoredDuringExecution:
      nodeSelectorTerms:
      - matchExpressions:
        - key: eks.amazonaws.com/compute-type
          operator: NotIn
          values:
          - hybrid

Add-ons requiring this treatment: AWS Load Balancer Controller, CloudWatch Observability Agent, ADOT, cert-manager.

CoreDNS Mixed Mode

CoreDNS should run on both cloud and hybrid nodes for DNS resilience. Use topologySpreadConstraints with at least 4 replicas (2 per side). See Network Configuration - CoreDNS Dual-Location Deployment.

EKS Pod Identity Agent

EKS Pod Identity Agent requires the eks-auth VPC endpoint in private/air-gap environments. Install as an EKS managed add-on:

aws eks create-addon \
  --cluster-name my-hybrid-cluster \
  --addon-name eks-pod-identity-agent \
  --addon-version v1.3.3-eksbuild.1

---

Node Upgrade

Hybrid nodes follow the same Kubernetes version skew policy as upstream Kubernetes — they cannot be newer than the control plane and may be up to three minor versions older.

Cutover Migration (Recommended)

When spare capacity is available, create new nodes on the target version and gracefully migrate workloads:

# 1. Install nodeadm on new hosts with target version
nodeadm install K8S_VERSION --credential-provider CREDS_PROVIDER

# 2. Cordon old nodes
kubectl cordon NODE_NAME

# 3. Scale CoreDNS for resiliency
kubectl scale deployments/coredns --replicas=2 -n kube-system

# 4. Drain old nodes
kubectl drain NODE_NAME --ignore-daemonsets --delete-emptydir-data

# 5. Uninstall old nodes
sudo nodeadm uninstall

# 6. Delete old node resource
kubectl delete node NODE_NAME

In-Place Upgrade

When spare capacity is unavailable, upgrade nodes in-place (results in downtime):

# 1. Cordon the node
kubectl cordon NODE_NAME

# 2. Drain workloads
kubectl drain NODE_NAME --ignore-daemonsets --delete-emptydir-data

# 3. Run nodeadm upgrade
sudo nodeadm upgrade K8S_VERSION -c file://nodeConfig.yaml

# 4. Uncordon after upgrade completes
kubectl uncordon NODE_NAME

# 5. Monitor
kubectl get nodes -o wide -w

---

Troubleshooting

nodeadm debug

The nodeadm debug command validates network access, credentials, and cluster connectivity:

sudo nodeadm debug -c file://nodeConfig.yaml

This validates:

• Network access to AWS APIs
• AWS credentials retrieval for Hybrid Nodes IAM role
• Network access to EKS Kubernetes API endpoint
• Node authentication with EKS cluster

Common Issues and Fixes

Installation Issues

Issue	Symptom	Fix
Must run as root	"msg":"Command failed","error":"must run as root"	Run nodeadm with sudo
Unable to connect to dependencies	max retries achieved for http request	Verify network access to dependency repositories
Package manager failure	failed to run update using package manager	Run apt update or dnf update first
Timeout	context deadline exceeded	Use --timeout 20m0s flag

Connection Issues

Issue	Symptom	Fix
Node IP not in CIDR	node IP is not in any of the remote network CIDR blocks	Verify RemoteNodeNetworks includes node IP range
API server unreachable	Unable to connect to the server / dial tcp: i/o timeout	Check VPN/DX tunnel, firewall port 443, VPC routes to TGW/VGW
Unauthorized	Failed to ensure lease exists: Unauthorized	Verify IAM role, EKS access entry with HYBRID_LINUX type
Node stays NotReady	Node registered but NotReady	Install CNI (Cilium), check VXLAN port 8472
DNS resolution failure	no such host for EKS API endpoint	Configure Route 53 Resolver Inbound Endpoint, update on-prem DNS
Image pull failure	ErrImagePull on system pods	Verify ECR VPC endpoints, containerd registry config, CA certs
Certificate error	x509: certificate signed by unknown authority	Install CA cert in system trust store, run update-ca-certificates
Hybrid profile exists	hybrid profile already exists	Run nodeadm uninstall then nodeadm install then nodeadm init

SSM Credential Issues

Issue	Symptom	Fix
Invalid activation	InvalidActivation	Verify region, activationCode, activationId in nodeConfig.yaml
Expired activation	ActivationExpired	Create new SSM hybrid activation, update nodeConfig.yaml
Expired token	ExpiredTokenException	Restart SSM agent: systemctl restart amazon-ssm-agent

IAM Roles Anywhere Issues

Issue	Symptom	Fix
Certificate not found	open /etc/iam/pki/server.pem: no such file or directory	Create /etc/iam/pki/ directory, copy certificate and key
Not authorized	not authorized to perform: sts:AssumeRole	Verify trust policy, trust anchor ARN, IAM RA profile

Diagnostic Commands

# Check kubelet status and logs
sudo systemctl status kubelet
sudo journalctl -u kubelet -f

# Check containerd
sudo systemctl status containerd

# Validate credentials
sudo aws sts get-caller-identity

# Check SSM agent (AL2023/RHEL)
sudo systemctl status amazon-ssm-agent

# Check SSM agent (Ubuntu)
sudo systemctl status snap.amazon-ssm-agent.amazon-ssm-agent

# Run nodeadm diagnostics
sudo nodeadm debug -c file://nodeConfig.yaml

Node Reset

If bootstrap fails and you need to start fresh:

# Basic uninstall
sudo nodeadm uninstall

# Force uninstall (cleans all state, skips confirmation prompt)
sudo nodeadm uninstall --force

# Re-run initialization
sudo nodeadm init -c file://nodeconfig.yaml

Paths deleted by nodeadm uninstall:

• /etc/kubernetes - Kubernetes configuration files
• /etc/eks - EKS-related configuration
• SSM/IAM Roles Anywhere artifacts

v1.0.9+ Changes:

• /var/lib/kubelet is preserved by default (data protection improvement)
• --force option removes all artifacts including those normally preserved

---

< Previous: Air-Gap Setup | Table of Contents | Next: GPU Integration >