---------------------------------------- Source: https://www.atomai.click/kubernetes-docs/en/networking/ ---------------------------------------- # Kubernetes Networking > **Last Updated**: September 13, 2026. Feature references include Cilium 1.20.1, Calico Open Source 3.32, Flannel 0.28.9 and AWS VPC CNI 1.23.0. Check each product's Kubernetes/platform matrix before installation; these are not a jointly tested cluster configuration. ## Overview Kubernetes networking is the core infrastructure layer that enables communication between containerized applications. This section covers everything from basic Kubernetes networking concepts to advanced CNI (Container Network Interface) solutions and networking patterns in AWS EKS environments. ## Kubernetes Networking Model The current Kubernetes model provides a Pod network in which Pods can communicate directly across nodes without address translation or proxies, **subject to intentional network segmentation**. Node agents such as kubelet must be able to reach Pods on their own node. Network policy, routing and application listeners still determine whether a particular connection succeeds. Ordinary Pods have their own network namespace and cluster-wide addresses; containers in one Pod share that namespace and localhost. Host-network Pods share the node network, and dual-stack or multi-network configurations need more precise address handling. Recreating a Pod may assign a different IP; restarting a container inside the same Pod does not necessarily recreate its network sandbox. | Component | Role | |---|---| | Pod network | Addressing and connectivity between workload network namespaces | | Service/discovery | Stable service names or virtual addresses over changing endpoints | | Ingress/Gateway implementation | Configured external entry and application routing | | Network policy engine | Enforces the policies supported by the selected implementation | These roles do not form a mandatory serial packet path. Service translation, an L7 proxy and workload policy can change how a particular request traverses the network. ### Pod Networking Pod networking supplies the addressing and routes for Pod communication. The illustration below shows ordinary IPv4 Pods; its connections assume that applicable policies and network controls permit them. ![Illustrative direct IPv4 Pod paths across two nodes, with connectivity subject to the configured policy and routing.](https://raw.githubusercontent.com/Atom-oh/kubernetes-docs/main/en/.gitbook/assets/en-networking-readme-1.png) [🔍 View interactive diagram](https://www.atomai.click/kubernetes-docs/archmaps/en-networking-readme-1.html) The addresses are illustrative ordinary Pod addresses. Intentional isolation and host-network or multi-network configurations require their own interpretation. #### Pod Networking Implementation Methods | Method | Description | Example CNI | |--------|-------------|-------------| | **Overlay Network** | Encapsulates traffic over the existing network | Flannel VXLAN, Calico VXLAN/IPIP, Cilium VXLAN/Geneve | | **Native Routing** | Uses routes in the underlying network without that overlay encapsulation | AWS VPC CNI, Calico routing/BGP, Cilium native routing | | **Conditional Encapsulation** | Uses direct paths or encapsulation according to configured topology | Supported Calico/Flannel/Cilium modes, with different prerequisites | ### Service Networking Services describe a logical set of endpoints, usually Pods, and how to reach them. ClusterIP supplies a stable virtual IP by default; headless Services omit that virtual IP, and ExternalName uses DNS CNAME mapping. A Service can also have endpoints managed without a Pod selector. ![Typical entry mechanisms for ClusterIP, NodePort, LoadBalancer and ExternalName Services; DNS mapping is distinguished from packet forwarding.](https://raw.githubusercontent.com/Atom-oh/kubernetes-docs/main/en/.gitbook/assets/en-networking-readme-2.png) [🔍 View interactive diagram](https://www.atomai.click/kubernetes-docs/archmaps/en-networking-readme-2.html) These are typical exposure mechanisms, not security guarantees. NodePort range and accessible node addresses are configurable; LoadBalancers can be internal. ExternalName returns a DNS alias and does not create a forwarding proxy. #### Service Type Characteristics Create matching `app: my-app` Pods in `default`, listening on the shown target ports. The NodePort default allocation range is 30000–32767 and can be configured. External reachability still depends on addresses, routes and access controls. The LoadBalancer example explicitly selects **AWS Load Balancer Controller**, with EC2 instance targets and allocated NodePorts. Install/configure that controller and its IAM/subnet prerequisites first. EKS Auto Mode uses a different controller/class. Port 443 merely selects a TCP port here; TLS must be served by the backend on 8443 or configured separately on the load balancer. These port mappings illustrate the general Kubernetes Service API. AWS currently documents additional native EKS network-policy requirements: the Service port must match the container port, and controller-managed Pods with `metadata.ownerReferences` provide reliable enforcement. Adapt the examples to those requirements before testing that policy implementation. ```yaml apiVersion: v1 kind: Service metadata: name: my-service namespace: default spec: type: ClusterIP selector: app: my-app ports: - protocol: TCP port: 80 targetPort: 8080 --- apiVersion: v1 kind: Service metadata: name: my-nodeport-service namespace: default spec: type: NodePort selector: app: my-app ports: - protocol: TCP port: 80 targetPort: 8080 nodePort: 30080 --- apiVersion: v1 kind: Service metadata: name: my-loadbalancer-service annotations: service.beta.kubernetes.io/aws-load-balancer-scheme: internet-facing service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: instance namespace: default spec: type: LoadBalancer selector: app: my-app ports: - protocol: TCP port: 443 targetPort: 8443 loadBalancerClass: service.k8s.aws/nlb allocateLoadBalancerNodePorts: true ``` ### Ingress Networking An Ingress resource needs a controller and its data plane. This HTTP example uses AWS LBC with `spec.ingressClassName: alb` and IP targets. The referenced `api-v1`, `api-v2` and `web-frontend` Services must exist in `default`, expose port 80 and have ready, VPC-routable Pod endpoints. Configure HTTPS/certificates separately when required. See the [LBC guide](https://www.atomai.click/kubernetes-docs/llms/en/networking/03-aws-lb-controller.md) for its installation and target prerequisites. Ingress defines rules for routing HTTP/HTTPS traffic to internal cluster Services. ![Logical Ingress host/path routing to Service backends and Pods.](https://raw.githubusercontent.com/Atom-oh/kubernetes-docs/main/en/.gitbook/assets/en-networking-readme-3.png) [🔍 View interactive diagram](https://www.atomai.click/kubernetes-docs/archmaps/en-networking-readme-3.html) The box represents the Ingress data-plane function. AWS LBC programs ALB; application traffic does not traverse the controller reconciliation process. Depending on target mode, the data plane can reach Pod IPs or NodePorts instead of traversing a Service virtual IP as a literal extra hop. ```yaml apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: my-ingress annotations: alb.ingress.kubernetes.io/scheme: internet-facing alb.ingress.kubernetes.io/target-type: ip namespace: default spec: rules: - host: api.example.com http: paths: - path: /v1 pathType: Prefix backend: service: name: api-v1 port: number: 80 - path: /v2 pathType: Prefix backend: service: name: api-v2 port: number: 80 - host: web.example.com http: paths: - path: / pathType: Prefix backend: service: name: web-frontend port: number: 80 ingressClassName: alb ``` ## CNI (Container Network Interface) CNI standardizes the interface through which a runtime configures a container network. For current Kubernetes, kubelet requests Pod-sandbox operations through CRI and the **container runtime manages CNI**. Kubelet's old direct CNI management flags were removed in Kubernetes 1.24. ### Runtime and Plugin Responsibilities | Actor | Responsibility | |---|---| | kubelet | Requests sandbox creation/removal through the container runtime interface | | Container runtime | Selects the network configuration and invokes the CNI plugin chain | | CNI plugin | Receives configuration, performs ADD/DEL and other supported operations, and returns results | | IPAM implementation | Allocates/releases addresses; may be a delegated plugin or part of a provider-specific agent | | Optional node agent | Maintains provider-specific routes, policy, IP pools or datapath state | The runtime passes configuration to the plugin through the CNI interface; a separate long-running agent or IPAM binary is not mandatory for every plugin. Interface types also vary: veth pairs are common, but are not the only implementation. ## CNI Comparison | Project / scope | Networking and policy | Features and limits to distinguish | |---|---|---| | **Cilium 1.20.1** | eBPF networking; Envoy for relevant L7 functions; Cilium network policies and Hubble | Linux worker dataplane, with AMD64/Arm64 requirements. Windows CLI availability is not Windows CNI support. WireGuard/IPsec and Beta ztunnel mTLS have distinct scopes. | | **Calico Open Source 3.32** | Routing/encapsulation choices; iptables, nftables and eBPF options; ordered policy tiers and host/workload policy | Windows has separate limits, including no Linux eBPF or WireGuard dataplane. Whisker/Goldmane flow observability is available as Tech Preview. Consult the edition matrix for paid capabilities. | | **Flannel 0.28.9** | Host subnet allocation and inter-node transport; VXLAN, host-gw and other backends | `flanneld` itself does not enforce NetworkPolicy; the chart's optional `netpol.enabled` deploys a SIGs policy controller. WireGuard is a documented backend; IPsec is experimental. Windows VXLAN has specific settings/limits. | | **AWS VPC CNI 1.23.0 / EKS** | VPC address allocation and EC2 ENIs/prefixes; EKS standard and Admin network policy capabilities on supported Linux EC2 nodes | EKS Auto Mode is a managed networking implementation with additional DNS policy capabilities. Windows, Fargate, custom networking, prefix delegation and multi-NIC support have separate conditions. | | **Original Weave Net project** | Historical overlay networking implementation | The original `weaveworks/weave` repository is archived. Do not describe it as an active, supported default for a new cluster. | ### Policy, Encryption and Observability - Cilium provides HTTP/DNS-aware policy through the applicable L7 components, and cluster-wide/host policy. Its deny/allow semantics are not Calico's ordered Tier API. - Calico Open Source includes hierarchical policy tiers and host policy. The current product matrix assigns application-layer policy, DNS/FQDN policy and Cluster Mesh to Cloud/Enterprise; those must not be silently attributed to the open-source edition. Calico's documented in-transit encryption uses WireGuard. - Amazon EKS provides `ClusterNetworkPolicy` Admin/Baseline controls for Auto Mode and supported EC2/VPC-CNI installations. The DNS/FQDN `ApplicationNetworkPolicy` feature described by AWS is for **Auto Mode**. Its name does not imply current HTTP-method/body inspection. - Flannel's optional policy controller has its own requirements; selecting a networking backend alone does not enable enforcement. - Node-to-node encryption, authenticated workload identity and application mTLS are different controls. Network flow visibility also differs from application tracing or process/file enforcement. ### Routing and Performance Calico and Cilium can advertise routes using BGP; that does not by itself provide multi-cluster service discovery, policy synchronization or encryption. Flannel host-gw uses direct routes and requires suitable layer-2 connectivity. An overlay adds encapsulation and MTU considerations, but a universal performance ranking cannot be inferred from the CNI name. The former 100/98/95/85/80/75 percent throughput figure had no reproducible workload, versions or measurement source. Use comparable hardware, kernel, packet/request sizes, concurrency, encryption/policy settings, throughput, loss and tail latency. The separate [Pod benchmark](https://www.atomai.click/kubernetes-docs/llms/en/networking/06-pod-network-benchmark.md) retains its own historical environment and measurements. ## CNI Selection Guide Choose the required routing, policy, operating-system and support model first, then test that combination. | Need | Evaluation path | |---|---| | Standard EKS VPC addressing and supported network policies | Evaluate AWS VPC CNI/EKS capabilities before adding a second policy engine. | | Ordered policy tiers, host policy or infrastructure BGP | Evaluate the relevant Calico edition/dataplane and routing prerequisites. | | Cilium policy, Hubble or selected mesh features | Check Linux/kernel/platform compatibility and the [Cilium mesh guide](https://www.atomai.click/kubernetes-docs/llms/en/service-mesh/cilium-service-mesh/README.md). Envoy remains part of applicable L7 paths. | | A small network with a limited feature set | Evaluate Flannel's backend and optional policy controller against actual requirements. | | Process, syscall or file enforcement | Evaluate a runtime-security component such as Tetragon separately from network policy. | ### EKS Managed Add-on Configuration The following is an example **configuration payload**, not an instruction to install both Calico and the VPC CNI policy engine on the same workloads: ```json { "enableNetworkPolicy": "true" } ``` The string `"true"` is the documented type for this setting. Select a compatible EKS add-on build for the existing Kubernetes version and inspect that build's configuration schema: ```bash EKS_REGION=ap-northeast-2 KUBERNETES_MINOR=1.35 # Replace with the existing cluster's minor version aws eks describe-addon-versions --region "$EKS_REGION" --addon-name vpc-cni \ --kubernetes-version "$KUBERNETES_MINOR" : "${VPC_CNI_ADDON_VERSION:?Set the compatible eksbuild version selected from metadata}" aws eks describe-addon-configuration --region "$EKS_REGION" --addon-name vpc-cni \ --addon-version "$VPC_CNI_ADDON_VERSION" ``` The upstream 1.23.0 release number and an EKS `eksbuild` version are different identifiers. Merge changes with the intended managed add-on configuration; do not blindly select `latest` or replace unrelated values. A migration from a third-party policy implementation also needs removal of its existing enforcement state and a tested node/workload transition plan. ## EKS Networking Fundamentals ### EKS Default Networking Architecture | Location / component | Responsibility | |---|---| | EKS-managed VPC | AWS runs the managed Kubernetes control plane across Availability Zones. | | Customer cluster VPC | Worker networking, selected subnets and EKS-managed cross-account ENIs provide the configured paths to the control plane. | | ALB/NLB in selected customer VPC subnets | Provides the chosen public or internal application entry point; an internet gateway/NAT gateway is not a substitute for that routing configuration. | | NAT gateway or private service endpoints | Supplies the particular outbound paths the workload design requires. | The former figure put the control plane inside the customer VPC and load balancers outside it; it has been replaced by these ownership boundaries. ### DNS and Networking by Compute Mode | Compute mode | DNS / component placement | |---|---| | Standard EC2 nodes | Normally use the configured CoreDNS Deployment and installed networking components; replacements need their own supported configuration. | | Pure EKS Auto Mode | CoreDNS, VPC CNI and kube-proxy functions run as managed node systemd services. A CoreDNS Deployment/add-on is unnecessary for these nodes. | | Auto Mode mixed with non-Auto nodes | Retain the CoreDNS Deployment for the non-Auto nodes; they cannot use another node's Auto Mode DNS service. | Auto Mode's first DNS resolver is node-local. Upstream forwarding and control-plane communication can still require network access; this is not a guarantee that every DNS-related packet stays on the node. AWS documents both Admin and DNS policies for Auto Mode, while standard EC2 VPC-CNI Admin policy has its own version/enabling requirements. ### How VPC CNI Works AWS VPC CNI gives ordinary Pods VPC-routable addresses using the selected IPAM mode. Secondary IPv4 addresses, delegated prefixes, branch ENIs and multi-NIC configurations differ; host-network Pods share the node network. ![Illustrative secondary-IPv4 allocation from EC2 ENIs to Pods, including an optional warm interface.](https://raw.githubusercontent.com/Atom-oh/kubernetes-docs/main/en/.gitbook/assets/en-networking-readme-9.png) [🔍 View interactive diagram](https://www.atomai.click/kubernetes-docs/archmaps/en-networking-readme-9.html) This depicts secondary-IP mode only. A warm ENI is a configurable allocation strategy, not a requirement that every node always reserves exactly one. Prefix delegation, custom networking and branch ENIs have different allocation rules. #### ENI and IP Limits | Instance Type | Max ENIs | IPv4 slots per ENI | Legacy secondary-IP bootstrap value | |---------------|----------|--------------|------------------------| | t3.medium | 3 | 6 | 17 | | t3.large | 3 | 12 | 35 | | m5.large | 3 | 10 | 29 | | m5.xlarge | 4 | 15 | 58 | | m5.2xlarge | 4 | 15 | 58 | | c5.4xlarge | 8 | 30 | 234 | These values are verified against the VPC CNI 1.23.0 instance limits and legacy max-Pods table. The historical calculation is `ENIs × (IPv4 slots per ENI − 1) + 2`; it is not a current universal recommendation. Prefix delegation, custom networking, branch ENIs and multiple network cards change address capacity. Kubernetes scheduling is also bounded by kubelet `maxPods` and resources. EKS managed node groups cap `maxPods` at 110 for instances with fewer than 30 vCPUs and 250 otherwise; available IP count alone does not override that cap. ### EKS Networking Considerations #### IP Address Management For **Linux VPC CNI**, configure the documented environment variables through the selected add-on/Helm/DaemonSet management mechanism. The following is an EKS add-on configuration fragment. The old `amazon-vpc-cni` ConfigMap with `enable-prefix-delegation` does not configure Linux IPAMD this way. Preserve other intended add-on values when applying a change. ```json { "env": { "ENABLE_PREFIX_DELEGATION": "true", "WARM_PREFIX_TARGET": "1" } } ``` Alternatively, tune the total allocation floor and free-IP target. When either `MINIMUM_IP_TARGET` or `WARM_IP_TARGET` is configured, it takes precedence over `WARM_PREFIX_TARGET`; these are alternative policies rather than four independent additive targets. Allocation still occurs in prefix-sized units. Nitro support, contiguous `/28` space for IPv4 and a suitable kubelet Pod limit are separate prerequisites. Windows prefix allocation is a different configuration path: AWS documents `enable-windows-prefix-delegation` and its warm-target keys in the `amazon-vpc-cni` ConfigMap. Do not copy the Linux environment-variable procedure unchanged to Windows. ```json { "env": { "ENABLE_PREFIX_DELEGATION": "true", "MINIMUM_IP_TARGET": "5", "WARM_IP_TARGET": "2" } } ``` #### Custom Networking These IPv4 examples require real subnet/security-group IDs in the intended AZ and VPC. Enable custom networking and select each node's ENIConfig through its zone label. An explicit ENIConfig node annotation takes precedence over that label. The example names below use the same region in both languages; replace them with the actual node zones. Installing ENIConfig objects alone does not activate custom networking. ```json { "env": { "AWS_VPC_K8S_CNI_CUSTOM_NETWORK_CFG": "true", "ENI_CONFIG_LABEL_DEF": "topology.kubernetes.io/zone" } } ``` ```yaml apiVersion: crd.k8s.amazonaws.com/v1alpha1 kind: ENIConfig metadata: name: ap-northeast-2a spec: securityGroups: - sg-0123456789abcdef0 subnet: subnet-0123456789abcdef0 --- apiVersion: crd.k8s.amazonaws.com/v1alpha1 kind: ENIConfig metadata: name: ap-northeast-2b spec: securityGroups: - sg-0123456789abcdef0 subnet: subnet-fedcba9876543210f ``` ## Advanced Networking Concepts The items below get named in passing elsewhere in this overview. Full setup procedures and measured numbers live in the linked deep-dive pages; this section organizes how these pieces differ by layer and where each one fits. ### L2–L7 and the Difference Between Routers and Load Balancers "Router" and "load balancer" often show up in the same sentence, but they answer different questions. A router picks (generally) one path to a single destination; a load balancer picks one target out of several equivalent candidates using a distribution algorithm. | Layer | Device/function | Decision basis | Kubernetes/AWS mapping | |---|---|---|---| | L2 (link) | Switch, bridge | Destination MAC address | veth pairs and Linux bridges created by the CNI, the virtual NIC an ENI exposes | | L3 (network) | Router or transparent appliance insertion | Destination IP for routing; flow identity for appliance selection | The VPC's implicit router, TGW; GWLB encapsulates IP packets for appliances | | L4 (transport) | L4 load balancer | Connection/flow identity, commonly the 5-tuple | NLB; kube-proxy (iptables, IPVS, nftables); separate eBPF Service implementations | | L7 (application) | L7 load balancer/reverse proxy | Per-request host, path, headers; protocol-aware | ALB, Ingress/Gateway API implementations, service-mesh sidecars (Envoy) | The key difference is the **unit of distribution**. An L4 load balancer normally selects a target for a TCP connection or tracked UDP flow. An L7 proxy can select a target for each supported application request, including requests sharing a connection. GWLB distributes encapsulated IP flows across security appliances rather than parsing application requests. Flow stickiness depends on configured timeout, health and failover behavior; it is not a guarantee that a flow can never be reassigned or interrupted. > 📎 Protocol-level definitions of L2/L3 concepts are in [Network Fundamentals Part 1](https://www.atomai.click/kubernetes-docs/llms/en/basics/06-network-fundamentals-part1.md); ALB/NLB target types and real configuration are in [AWS Load Balancer Controller](https://www.atomai.click/kubernetes-docs/llms/en/networking/03-aws-lb-controller.md). ### Cross-Account/VPC Connectivity: TGW, VPC Peering, GWLB, PrivateLink, Lattice These five connectivity options differ in layer and traffic model. Measured latency across TGW RAM sharing, VPC Peering, PrivateLink, TGW Peering and VPC Lattice is in [Cross-Org VPC Connectivity](https://www.atomai.click/kubernetes-docs/llms/en/networking/05-cross-org-vpc-connectivity.md). This section adds GWLB, which isn't in that comparison table, and reframes all five by layer. | Connectivity | Layer/model | Characteristics | |---|---|---| | VPC Peering | L3, bidirectional IP routing | Not transitive; can't be configured across overlapping CIDRs | | Transit Gateway (TGW) | L3, hub-and-spoke IP routing | Uses attachment associations and propagation across one or more TGW route tables; shared cross-account via RAM | | Gateway Load Balancer (GWLB) | L3, transparent appliance insertion | Encapsulates the original packet in GENEVE (UDP 6081); a VPC endpoint service model connects consumer traffic to the provider's appliance fleet | | PrivateLink | Private endpoint connectivity | An NLB-backed endpoint service is one model; resource endpoints also exist. Consumer/provider CIDRs may overlap | | VPC Lattice | Application and resource networking | HTTP/HTTPS services support L7 routing and optional IAM authorization; TLS passthrough and resource configurations have different capabilities | GWLB inserts inspection appliances such as firewalls and IDS/IPS into an IP path through a Gateway Load Balancer endpoint. Its default flow stickiness uses five fields; supported configurations can instead use two or three. Validate forward and return routes, appliance health, encapsulation MTU, NACLs and the security groups of the actual workloads/appliances. GWLB itself does not have an ALB-style security group, and flow stickiness does not replace failure testing. > 📎 The full EKS/VPC Lattice integration (Gateway API Controller, IAM authorization, routing) is in [VPC Lattice](https://www.atomai.click/kubernetes-docs/llms/en/networking/02-vpc-lattice.md). ### How DNS Resolver and Route Tables Actually Behave **DNS resolver:** AmazonProvidedDNS **is Route 53 Resolver**. Its addresses include the primary VPC IPv4 network address plus two (`10.0.0.2` for `10.0.0.0/16`) and `169.254.169.253`; it resolves associated private zones and public names according to Resolver rules. CoreDNS normally serves the configured Kubernetes cluster domain, often `cluster.local`; `kube-dns` is its Service name, not a namespace or DNS zone. External forwarding follows the Corefile and the resolver file visible to the DNS Pod. Inspect those settings instead of assuming the node's resolver file is used unchanged. In a Resolver endpoint design, inbound endpoints accept on-premises queries, while outbound endpoints and associated rules forward selected VPC queries to on-premises DNS. Auto Mode's node-local resolver does not eliminate upstream dependencies. **Route tables:** VPC route evaluation generally uses longest-prefix matching. AWS permits replacing a `local` route's target and adding supported more-specific subnet routes for appliance routing; `local` is not unconditionally the most specific route. For identical destinations, static VPC routes take precedence over routes propagated from a virtual private gateway. A VPC route targeting a TGW is static; propagation inside a TGW belongs to its separate route tables. Invalid targets can leave `blackhole` entries that drop traffic, so inspect route state as well as the destination. A subnet without an explicit route-table association uses the VPC's main route table. > 📎 TGW/Peering route priority and static-route configuration examples are in [Cross-Org VPC Connectivity's operational findings](https://www.atomai.click/kubernetes-docs/llms/en/networking/05-cross-org-vpc-connectivity.md#operational-findings). ### The Kernel Data Plane: iptables, IPVS, eBPF and Packet Filtering Linux Service forwarding and network-policy enforcement can use different mechanisms. Netfilter provides packet-path hooks used by iptables and nftables. eBPF implementations can attach at XDP, tc or socket hooks and perform Service selection there. This does not mean every packet in an eBPF-enabled cluster bypasses Netfilter or connection tracking; the path depends on the CNI, kernel, routing and feature configuration. | Implementation | Where it sits | Characteristics | |---|---|---| | iptables | Sequential rule chains on netfilter hooks | Evaluation time scales with rule count (O(n)); kube-proxy's long-standing default mode | | IPVS | Kernel-native L4 load balancer, a netfilter extension | Hash-based lookup (near O(1)); deprecated as a kube-proxy mode starting with Kubernetes 1.35 | | nftables | netfilter's successor framework to iptables | kube-proxy's stable mode since 1.33; check kernel/CNI compatibility first | | eBPF (e.g., Cilium) | Configured XDP, tc and socket hooks | Can replace kube-proxy Service handling; it is a separate implementation, with path-specific Netfilter/conntrack behavior | Switching implementations can leave kernel rules and active connections behind. Follow the distribution/CNI migration procedure, drain workloads as required, and plan for node restarts where cleanup requires them. Replacing kube-proxy with an eBPF-based CNI also requires a supported cutover order so the implementations do not compete for the same Service traffic. > 📎 The IPVS deprecation timeline and the nftables stable transition are covered in [Introduction to Kubernetes](https://www.atomai.click/kubernetes-docs/llms/en/basics/04-kubernetes-introduction.md); Cilium's eBPF kube-proxy replacement is in [Cilium eBPF](https://www.atomai.click/kubernetes-docs/llms/en/networking/cilium/02-ebpf.md); Calico's eBPF data plane and its migration procedure are in [Calico eBPF](https://www.atomai.click/kubernetes-docs/llms/en/networking/calico/06-ebpf-dataplane.md). ### Compute-Intensive Networking: ENI, EFA, NVLink and Optical Transceivers ENI, EFA and NVLink serve different paths. An **ENI** is a virtual network interface attached to an EC2 instance in one Availability Zone; its normal IP traffic can reach other AZs and connected VPCs when routing and policy allow it (see [VPC CNI](https://www.atomai.click/kubernetes-docs/llms/en/networking/01-vpc-cni.md)). **EFA** provides an OS-bypass device used through libfabric by compatible MPI/NCCL software. **EFA device traffic is non-routable and cannot cross VPC/AZ boundaries**; normal IP traffic through the ENA device of an EFA-with-ENA interface remains routable. EFA-only interfaces have no ENA device or IP addressing. **NVLink** connects GPUs within supported systems, including supported rack-scale NVLink domains. Measure the selected hardware, collective operations and placement rather than assuming a fixed speedup over EFA. **Optical transceivers** are a general data-center networking concept. Copper DAC (Direct Attach Copper) cables suit short runs; optical modules and fiber support other reach and bandwidth requirements. QSFP and OSFP describe module form factors, not a guarantee of optical media. Treat this as general background: it does not establish the physical cabling of a particular AWS workload. > 📎 NVLink/IMEX topology-aware scheduling and GPU Pod placement examples are in [AI/ML Infrastructure](https://www.atomai.click/kubernetes-docs/llms/en/ai-ml/06-ai-infrastructure.md); EFA's VPC/AZ boundary constraint and measurements are in [Cross-Org VPC Connectivity](https://www.atomai.click/kubernetes-docs/llms/en/networking/05-cross-org-vpc-connectivity.md). ### What Next-Generation Protocols Mean for Kubernetes: HTTP/3, gRPC, QUIC The protocol mechanics of HTTP/3 (RFC 9114) and its QUIC transport (RFC 9000) are covered in [Network Fundamentals Part 2](https://www.atomai.click/kubernetes-docs/llms/en/basics/06-network-fundamentals-part2.md) and [Part 3](https://www.atomai.click/kubernetes-docs/llms/en/basics/06-network-fundamentals-part3.md). Here we cover only what actually affects Kubernetes traffic distribution. - **gRPC and L4 load balancers:** gRPC multiplexes requests over HTTP/2 connections. An L4 balancer normally keeps an established TCP connection on its selected endpoint; if that endpoint is a proxy, it can make further routing decisions. Adding Pods alone does not redistribute existing connections. Per-RPC distribution requires a compatible L7 proxy or client-side policy. A streaming RPC remains one call; its individual messages are not independently balanced. - **Gateway API's GRPCRoute:** Ingress has no gRPC-specific resource, but Gateway API standardizes service/method-level routing with `GRPCRoute`. Support varies by implementation (how many header matches, retry policies, etc.), so check the controller's own documentation. - **How far HTTP/3/QUIC actually reaches into the cluster:** HTTP/3 support between a client and the edge (a CDN, a load balancer) is a separate question from HTTP/3 support inside the cluster or on an Ingress's backend connection. Many Ingress/Gateway implementations still speak HTTP/1.1 or HTTP/2 to the backend, and whether end-to-end HTTP/3 is supported varies by implementation and version — don't generalize; check the documentation for the controller actually in use. ## Networking Sub-pages This section covers the following topics in detail: ### [VPC CNI](https://www.atomai.click/kubernetes-docs/llms/en/networking/01-vpc-cni.md) EKS networking with VPC addresses for ordinary Pods and mode-specific IPAM/policy prerequisites. ### [Cilium Deep Dive](https://www.atomai.click/kubernetes-docs/llms/en/networking/cilium/README.md) High-performance eBPF-based CNI solution. Provides advanced features like L7 Network Policy, Service Mesh, and observability (Hubble). ### [Calico Deep Dive](https://www.atomai.click/kubernetes-docs/llms/en/networking/calico/README.md) One of the most widely used CNIs. Powerful Network Policy, BGP support, and enterprise features. Covers introduction, architecture, networking modes, BGP deep dive, Network Policy, eBPF, advanced topics, EKS integration, and operations guide. ### [VPC Lattice](https://www.atomai.click/kubernetes-docs/llms/en/networking/02-vpc-lattice.md) AWS managed application networking service. Cross-VPC, cross-account service-to-service communication. ### [AWS Load Balancer Controller](https://www.atomai.click/kubernetes-docs/llms/en/networking/03-aws-lb-controller.md) Integrates Kubernetes Services and Ingress with AWS ELB (ALB/NLB). ### [Gateway API](https://www.atomai.click/kubernetes-docs/llms/en/networking/04-gateway-api.md) Next-generation Kubernetes ingress API. Standardized resource model and role-based configuration. ### [Pod Network Benchmark](https://www.atomai.click/kubernetes-docs/llms/en/networking/06-pod-network-benchmark.md) Pod-to-pod RTT, HTTP latency and throughput measured on EKS for the same node, same AZ and cross-AZ, plus DNS `ndots:5` query amplification. ## Network Troubleshooting ### Common Issues and Solutions #### Pod-to-Pod Communication Failure ```bash NAMESPACE=default POD_NAME=iperf-client # An existing diagnostic Pod with nslookup/curl SERVICE_NAME=my-service kubectl -n "$NAMESPACE" get pods -o wide kubectl -n "$NAMESPACE" exec "$POD_NAME" -- nslookup "$SERVICE_NAME" kubectl -n "$NAMESPACE" exec "$POD_NAME" -- \ curl --connect-timeout 3 --max-time 5 -v "http://$SERVICE_NAME:80/" kubectl -n kube-system logs -l k8s-app=aws-node -c aws-node --tail=100 kubectl -n kube-system logs -l k8s-app=cilium -c cilium-agent --tail=100 ``` Run diagnostics from an existing Pod with the named tools. Query only the CNI installed in the cluster; Auto Mode system services are not those DaemonSets. DNS success, TCP reachability and an application HTTP response are different checks. ICMP may be blocked or require extra privileges, so a failed ping alone does not prove a TCP service is unreachable. #### Service Unreachable ```bash NAMESPACE=default SERVICE_NAME=my-service kubectl -n "$NAMESPACE" get service "$SERVICE_NAME" -o yaml kubectl -n "$NAMESPACE" get endpointslices \ -l "kubernetes.io/service-name=$SERVICE_NAME" -o yaml kubectl -n kube-system logs -l k8s-app=kube-proxy --tail=100 ``` Use EndpointSlice for current endpoint diagnosis. Check Service selectors, target ports, endpoint readiness, address family and applicable policy. Inspect kube-proxy logs only if that component actually owns Service forwarding; an eBPF replacement or Auto Mode needs its own diagnostics. #### Network Policy Debugging ```bash kubectl get networkpolicies.networking.k8s.io -A kubectl -n kube-system exec ds/cilium -c cilium-agent -- cilium-dbg policy get kubectl -n kube-system exec ds/cilium -c cilium-agent -- cilium-dbg endpoint list # For a Calico installation using its standard CRD datastore: kubectl get networkpolicies.crd.projectcalico.org -A kubectl get globalnetworkpolicies.crd.projectcalico.org ``` The Cilium commands inspect one Agent selected by the DaemonSet reference; choose the affected node's Agent when tracing an incident. Calico native API installations can expose a different API group, so inspect the installation's served resources. Kubernetes, Calico and AWS extension policies are distinct resources and may have different precedence. ### Network Performance Testing This bounded TCP exercise uses the publisher's pinned Netshoot v0.16 image index, which contains Linux AMD64 and Arm64 images; its Dockerfile includes `iperf3`. Create these Pods in a test environment where TCP 5201 is permitted. It is an illustrative workload, not a measured CNI comparison. ```yaml apiVersion: v1 kind: Pod metadata: name: iperf-server namespace: default labels: app: iperf-server spec: restartPolicy: Never automountServiceAccountToken: false nodeSelector: kubernetes.io/os: linux containers: - name: netshoot image: nicolaka/netshoot:v0.16@sha256:b09d9b21381f47a79b3cbcb30da25266dc17186ea00ae65e99fdc51396f48e70 command: - iperf3 - -s workingDir: /tmp resources: requests: cpu: 100m memory: 64Mi limits: cpu: 500m memory: 256Mi securityContext: runAsNonRoot: true runAsUser: 1000 allowPrivilegeEscalation: false capabilities: drop: - ALL seccompProfile: type: RuntimeDefault ports: - containerPort: 5201 protocol: TCP --- apiVersion: v1 kind: Pod metadata: name: iperf-client namespace: default labels: app: iperf-client spec: restartPolicy: Never automountServiceAccountToken: false nodeSelector: kubernetes.io/os: linux containers: - name: netshoot image: nicolaka/netshoot:v0.16@sha256:b09d9b21381f47a79b3cbcb30da25266dc17186ea00ae65e99fdc51396f48e70 command: - sleep - '3600' workingDir: /tmp resources: requests: cpu: 100m memory: 64Mi limits: cpu: 500m memory: 256Mi securityContext: runAsNonRoot: true runAsUser: 1000 allowPrivilegeEscalation: false capabilities: drop: - ALL seccompProfile: type: RuntimeDefault ``` ```bash kubectl -n default wait --for=condition=Ready pod/iperf-server pod/iperf-client --timeout=120s IPERF_SERVER_IP="$(kubectl -n default get pod iperf-server -o jsonpath='{.status.podIP}')" test -n "$IPERF_SERVER_IP" kubectl -n default exec iperf-client -- iperf3 -c "$IPERF_SERVER_IP" -t 10 -b 10M ``` The client sleeps for one hour and the command caps offered traffic at 10 Mbit/s for ten seconds. This tests the selected path, not maximum throughput. Record actual Pod/node/AZ placement, resource limits and policy before interpreting results. Choose Windows-specific tools for Windows nodes. Remove only the test resources you created when finished. These standalone diagnostic Pods are for connectivity tests. For native EKS network-policy enforcement tests, use Deployment/Job-managed Pods and the documented Service/container-port requirements. ## Best Practices ### 1. IP Address Planning - Design CIDR blocks large enough - Separate Pod network from Service network - Design subnets with future expansion in mind ### 2. Apply Network Policies Create the isolated `networking-demo` namespace before using this example. It selects every Pod there and isolates both ingress and egress under standard Kubernetes NetworkPolicy semantics; required DNS and application flows need explicit allow rules. Enforcement requires a supporting policy engine. Additional cluster/admin policy APIs can alter precedence, and this one manifest is not a complete zero-trust architecture. - Apply default deny policies (Zero Trust) - Explicitly allow only required traffic - Isolate namespaces ```yaml apiVersion: networking.k8s.io/v1 kind: NetworkPolicy metadata: name: default-deny-all namespace: networking-demo spec: podSelector: {} policyTypes: - Ingress - Egress ``` ### 3. Performance Optimization - Choose appropriate CNI (matching workload) - MTU optimization - Kernel parameter tuning ### 4. Security Hardening - Select supported transport encryption and verify which traffic it covers. - Configure workload/application identity and mTLS where required; keep these separate from DNS/IP-based allowlists. - Review policy, certificate and access-control changes regularly. ### 5. Ensure Observability - Collect network metrics - Enable flow logs - Implement distributed tracing ## Next Steps 1. [VPC CNI](https://www.atomai.click/kubernetes-docs/llms/en/networking/01-vpc-cni.md) - Default EKS CNI 2. [Cilium Deep Dive](https://www.atomai.click/kubernetes-docs/llms/en/networking/cilium/README.md) - eBPF-based networking 3. [Calico Deep Dive](https://www.atomai.click/kubernetes-docs/llms/en/networking/calico/README.md) - Routing, policy and dataplanes 4. [VPC Lattice](https://www.atomai.click/kubernetes-docs/llms/en/networking/02-vpc-lattice.md) - AWS managed networking 5. [AWS Load Balancer Controller](https://www.atomai.click/kubernetes-docs/llms/en/networking/03-aws-lb-controller.md) - ELB integration 6. [Gateway API](https://www.atomai.click/kubernetes-docs/llms/en/networking/04-gateway-api.md) - Next-generation ingress 7. [Cross-Org VPC Connectivity](https://www.atomai.click/kubernetes-docs/llms/en/networking/05-cross-org-vpc-connectivity.md) - Connecting VPCs across AWS Organizations (field-verified) 8. [Pod Network Benchmark](https://www.atomai.click/kubernetes-docs/llms/en/networking/06-pod-network-benchmark.md) - Measured latency and throughput per node/AZ boundary --- ## References - [Kubernetes network model](https://kubernetes.io/docs/concepts/services-networking/) - [Kubernetes Services](https://kubernetes.io/docs/concepts/services-networking/service/) - [Container runtime and CNI](https://kubernetes.io/docs/concepts/extend-kubernetes/compute-storage-net/network-plugins/) - [Kubernetes NetworkPolicy](https://kubernetes.io/docs/concepts/services-networking/network-policies/) - [CNI specification](https://raw.githubusercontent.com/containernetworking/cni/main/SPEC.md) - [Calico product editions](https://docs.tigera.io/calico/latest/about) - [Calico policy tiers](https://docs.tigera.io/calico/latest/network-policy/policy-tiers/tiered-policy) - [Calico Whisker flow logs](https://docs.tigera.io/calico/latest/observability/view-flow-logs) - [Calico Windows limitations](https://docs.tigera.io/calico/latest/getting-started/kubernetes/windows-calico/limitations) - [Flannel 0.28.9 networking and policy](https://raw.githubusercontent.com/flannel-io/flannel/v0.28.9/README.md) - [Flannel backends](https://raw.githubusercontent.com/flannel-io/flannel/v0.28.9/Documentation/backends.md) - [Original Weave repository status](https://api.github.com/repos/weaveworks/weave) - [AWS VPC CNI 1.23.0](https://raw.githubusercontent.com/aws/amazon-vpc-cni-k8s/v1.23.0/README.md) - [EKS network policy configuration](https://docs.aws.amazon.com/eks/latest/userguide/cni-network-policy-configure.html) - [EKS standard and Admin network policies](https://docs.aws.amazon.com/eks/latest/userguide/cni-network-policy.html) - [EKS prefix delegation and maxPods](https://docs.aws.amazon.com/eks/latest/userguide/cni-increase-ip-addresses-procedure.html) - [EKS Admin and DNS policy deployment models](https://aws.amazon.com/blogs/containers/enhance-amazon-eks-network-security-posture-with-dns-and-admin-network-policies/) - [EKS Auto Mode networking](https://docs.aws.amazon.com/eks/latest/userguide/auto-networking.html) - [EKS add-on requirements](https://docs.aws.amazon.com/eks/latest/userguide/workloads-add-ons-available-eks.html) - [EKS control plane architecture](https://docs.aws.amazon.com/eks/latest/best-practices/control-plane.html) - [Netshoot v0.16 image metadata](https://hub.docker.com/v2/repositories/nicolaka/netshoot/tags/v0.16) - [Netshoot v0.16 Dockerfile](https://raw.githubusercontent.com/nicolaka/netshoot/v0.16/Dockerfile) - [Tetragon runtime security](https://tetragon.io/docs/overview/) - [AWS LBC 3.5 NLB configuration](https://github.com/kubernetes-sigs/aws-load-balancer-controller/blob/v3.5.0/docs/guide/service/nlb.md) - [AWS LBC 3.5 Ingress configuration](https://github.com/kubernetes-sigs/aws-load-balancer-controller/blob/v3.5.0/docs/guide/ingress/annotations.md) - [Gateway Load Balancer concepts](https://docs.aws.amazon.com/vpc/latest/privatelink/gateway-load-balancers.html) - [GENEVE encapsulation (RFC 8926)](https://www.rfc-editor.org/rfc/rfc8926) - [VPC DNS resolver](https://docs.aws.amazon.com/vpc/latest/userguide/vpc-dns.html) - [Route 53 Resolver endpoints and rules](https://docs.aws.amazon.com/Route53/latest/DeveloperGuide/resolver.html) - [VPC route table evaluation order](https://docs.aws.amazon.com/vpc/latest/userguide/VPC_Route_Tables.html) - [Local routes and more-specific subnet routes](https://docs.aws.amazon.com/vpc/latest/userguide/subnet-route-tables.html) - [Static and propagated route priority](https://docs.aws.amazon.com/vpc/latest/userguide/route-tables-priority.html) - [AmazonProvidedDNS addresses and behavior](https://docs.aws.amazon.com/vpc/latest/userguide/AmazonDNS-concepts.html) - [GWLB flow stickiness and failover](https://docs.aws.amazon.com/elasticloadbalancing/latest/gateway/edit-target-group-attributes.html) - [Kubernetes Service virtual IPs and kube-proxy modes](https://kubernetes.io/docs/reference/networking/virtual-ips/) - [CoreDNS Service names and forwarding configuration](https://kubernetes.io/docs/tasks/administer-cluster/dns-custom-nameservers/) - [PrivateLink resource endpoints](https://docs.aws.amazon.com/vpc/latest/privatelink/privatelink-access-resources.html) - [Netfilter/iptables project documentation](https://www.netfilter.org/documentation/index.html) - [EC2 Elastic Fabric Adapter](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/efa.html) - [QUIC transport protocol (RFC 9000)](https://www.rfc-editor.org/rfc/rfc9000) - [HTTP/3 (RFC 9114)](https://www.rfc-editor.org/rfc/rfc9114) - [gRPC over HTTP/2 and load balancing](https://grpc.io/blog/grpc-load-balancing/) - [Gateway API GRPCRoute](https://gateway-api.sigs.k8s.io/guides/user-guides/grpc-routing/) ---------------------------------------- Source: https://www.atomai.click/kubernetes-docs/en/basics/06-network-fundamentals-part1 ---------------------------------------- # Network Fundamentals Part 1 — The Layer Model, Link and Routing Layers > **Last Updated**: September 11, 2026 ::: tip This is a four-part series **Part 1: The Layer Model, Link and Routing Layers** *(this document)* · [Part 2: The Transport Layer and TLS](https://www.atomai.click/kubernetes-docs/llms/en/basics/06-network-fundamentals-part2.md) · [Part 3: Application Protocols](https://www.atomai.click/kubernetes-docs/llms/en/basics/06-network-fundamentals-part3.md) · [Part 4: A Request's Journey and the Cloud](https://www.atomai.click/kubernetes-docs/llms/en/basics/06-network-fundamentals-part4.md) ::: A browser request relies on several cooperating protocols. The exact sequence depends on caches, connection reuse, IP version and HTTP version, so troubleshooting needs to examine more than HTTP alone. This series walks through 25 networking protocols and mechanisms, **layer by layer, from the bottom up**. The reason for building from the bottom is simple: every upper layer is designed on the assumption that the layers below it already work. Read top-down and you keep hitting "but how does *that* part work?" Each entry follows the same shape: **one-line definition → how it works → where it bites in practice**. --- ## 0. The Layer Map on One Page | Layer | Job | Protocols covered here | |---|---|---| | Application | Actual service semantics | HTTP/3, WebSocket, WebRTC, gRPC, DNS, DoH, DHCP, MQTT, SSH, SMTP | | Security | Encryption and authentication (rides on transport) | TLS | | Transport | End-to-end data delivery | TCP, UDP, QUIC | | Internet / Routing | Choosing paths between networks | IPv4, IPv6, ICMP, BGP, OSPF, NAT | | Link | Delivery within one physical segment | Ethernet, Wi-Fi, VLAN, PPP, ARP | A few entries refuse to respect clean layer boundaries. TLS sits wedged between transport and application, QUIC rides on UDP while doing a transport layer's job, and ARP bridges IP and the link layer. NAT is less a protocol than a function. These "exceptions" account for most real-world troubleshooting. --- ![Shows the link/routing-layer path from a laptop through an L2 switch and home router to the ISP edge, the BGP-driven internet core, an OSPF data-center router, and finally the server, with each segment's protocol and MTU.](https://raw.githubusercontent.com/Atom-oh/kubernetes-docs/main/en/.gitbook/assets/en-basics-06-network-fundamentals-part1-0.png) [🔍 View interactive diagram](https://www.atomai.click/kubernetes-docs/archmaps/en-basics-06-network-fundamentals-part1-0.html) --- ## 1. Link Layer — Moving Bits Within One Segment The link layer cares about exactly one thing: **how to hand bits to the device sitting right next to you.** Whether the final destination is the next rack or the other side of the planet, this layer is only responsible for the next hop. ### Ethernet **Definition:** The link-layer standard that carries frames on wired local networks. **How it works:** Data is wrapped into frames, with destination and source MAC addresses in front. For known unicast, a switch uses its MAC table to select the destination port. Broadcast and unknown-unicast frames are typically flooded within the VLAN; multicast behavior depends on configuration. Early Ethernet relied on collision detection (CSMA/CD), but in modern switched full-duplex networks collisions have essentially disappeared. **In practice:** For Ethernet IP traffic, an MTU of 1500 means a 1500-byte IP packet inside the frame, excluding Ethernet header/FCS. Jumbo MTUs are device/path-specific; 9001 is an EC2-supported value, not a universal Ethernet size. In the cloud, layering a VPN or overlay network on top adds encapsulation headers that shrink the effective MTU, and failed MTU discovery can produce a black hole that shows up as "ping works, but large responses hang." It is one of the failure modes that takes the longest to diagnose. **MTU vs MSS:** MSS limits TCP data bytes, not the full frame. With MTU 1500, the base-header calculation gives 1460 for IPv4 (1500−20−20) or 1440 for IPv6 (1500−40−20). The sender further reduces actual data for any IP/TCP options it includes. TCP exchanges MSS during the handshake, so when MTU problems keep recurring across a tunnel, MSS clamping on the router (forcing a lower TCP MSS) is a widely used workaround. ### Wi-Fi **Definition:** The link-layer standard that carries LAN frames over a wireless segment (IEEE 802.11). **How it works:** Because the air is a shared medium, Wi-Fi is fundamentally different from Ethernet. Wi-Fi avoids relying on collision detection while transmitting and uses CSMA/CA: check that the channel is clear before sending, then use ACK/retry for ordinary unicast traffic; broadcast/multicast behavior differs. In other words, retransmission is already built into the link layer. **In practice:** Link-layer retransmission stacked on top of TCP retransmission inflates latency variance (jitter). Real-time quality problems get reported as "the server's fault" when the actual culprit is the client's wireless segment. Server RTT alone cannot locate the cause; correlate it with application processing time and client/AP retry, signal and queue metrics. ### VLAN **Definition:** A technique for segmenting shared switch infrastructure into logical L2 networks (IEEE 802.1Q). **How it works:** An 802.1Q-tagged frame carries a 4-byte VLAN tag. Access ports can carry untagged frames that the switch assigns to a VLAN. Broadcasts only reach hosts in the same VLAN, so you can segment a network without touching the cabling. Traffic between VLANs must pass through an L3 device (a router or L3 switch). **In practice:** VLANs provide logical L2 segmentation, not physical or cryptographic isolation. Routing and firewall controls determine permitted inter-segment traffic. VPCs, subnets and security groups serve different cloud networking roles; they are not one-for-one replacements for VLANs. > 📎 For how EKS structures its VPC, see [EKS Networking Fundamentals](https://www.atomai.click/kubernetes-docs/llms/en/eks/03-eks-networking-part1.md). ### PPP **Definition:** A protocol that carries packets over a point-to-point link connecting exactly two nodes. **How it works:** Unlike Ethernet, no addressing is needed — there is only one node at each end of the link. Instead, PPP provides link establishment, optional authentication, and upper-protocol negotiation (LCP/NCP). **In practice:** It looks like a relic of the dial-up era, but it survives as PPPoE on a large share of residential internet lines. With standard 1500-byte Ethernet payloads, the usual 6-byte PPPoE header plus 2-byte PPP protocol field leaves 1492 bytes for IP; negotiated larger underlays can preserve 1500. An unaccounted-for reduction to 1492 can cause the MTU problems described above. ### ARP **Definition:** The protocol that resolves an on-link IPv4 next-hop address to a MAC address. **How it works:** The IP layer says "send this to 10.0.1.5," but Ethernet only understands MAC addresses. So the host broadcasts "who has 10.0.1.5?" and the owning host replies. The result is cached with OS-specific neighbor states/timeouts. For an off-link destination, the host resolves its gateway’s MAC rather than the remote host’s MAC. **In practice:** ARP has no authentication. Anyone can answer "that IP is mine," which is exactly what makes ARP spoofing possible. The same property is also used legitimately: on failover, the new active node broadcasts a Gratuitous ARP to announce the VIP-to-MAC mapping to neighbors; switches also learn source-MAC location from the frame. When a VIP-based HA setup fails over slowly, delayed cache refresh is a prime suspect. > 📎 For how Cilium integrates L2/routing behavior with eBPF, see [Cilium Networking](https://www.atomai.click/kubernetes-docs/llms/en/networking/cilium/03-networking.md). --- ## 2. Internet and Routing Layer — Crossing Networks If the link layer gets you "next door," this layer gets you "to the other side of the planet." The central question is: **where should this packet go next?** ### IPv4 **Definition:** The internet-layer protocol built on 32-bit addresses. **How it works:** Every packet carries source and destination IPs; each router finds the most specific route (longest prefix match) in its routing table and forwards to the next hop. Delivery is best-effort — no guarantees, no ordering. Those guarantees are the job of the layer above (TCP). **In practice:** IPv4 has about 4.3 billion possible addresses, and scarcity made NAT widely used and turned the private ranges (10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16) into the internal-network standard. The first wall large organizations hit during cloud migration is overlap in these private ranges: overlapping on-premises/VPC CIDRs prevent straightforward routing over VPN or Direct Connect without a designed renumbering, translation or proxy solution. IP address design is something to lock down at project kickoff. ### IPv6 **Definition:** The next-generation internet-layer protocol with 128-bit addresses. **How it works:** With 128-bit addresses, exhaustion is a non-issue. The base header has a fixed 40-byte layout and no header checksum; routers do not fragment IPv6 packets. SLAAC lets hosts self-configure addresses without DHCP, and ARP is replaced by NDP (Neighbor Discovery Protocol). **In practice:** IPv6 is not backward-compatible with IPv4, so real deployments run dual stack — which means maintaining two sets of firewall rules and security policies. Missing rules on the IPv6 path is a common security gap. A global IPv6 address does not by itself make a workload internet-reachable. AWS still requires routing and permitted security-group/NACL traffic; an egress-only internet gateway can allow outbound IPv6 without unsolicited inbound connections. **Transition mechanisms:** There are three practical ways to coexist with IPv4: **dual stack** (run both side by side — most common, at the cost of duplicated policy), **tunneling** (wrap IPv6 packets in IPv4 to cross v4-only segments), and **NAT64/DNS64** (translate so IPv6-only clients can reach IPv4 servers — mobile carriers use this at scale as 464XLAT). Kubernetes supports dual-stack Services too, so cluster CIDR design can account for an IPv6 range from the start. ### ICMP **Definition:** The control protocol that reports network errors and state. **How it works:** ICMP carries control information and can include Echo payloads or quoted original-packet data: destination unreachable, TTL exceeded, fragmentation needed, and so on. `ping` uses Echo Request/Reply; `traceroute` increments TTL (or IPv6 Hop Limit) one hop at a time and reads the returning Time Exceeded messages. **In practice:** Blanket-blocking ICMP "for security" is common — and it is the direct cause of the MTU black hole mentioned earlier. Classical IPv4 PMTUD uses ICMP Type 3 Code 4, while IPv6 uses ICMPv6 Packet Too Big Type 2. Blocking required messages can cause black holes; PLPMTUD can instead probe packet sizes without relying on ICMP. Preserve required error/discovery traffic according to the IP version and policy. > 📎 For how this failure shows up in EKS, see [EKS Networking Deep Dive](https://www.atomai.click/kubernetes-docs/llms/en/eks/03-eks-networking-part3.md). ### OSPF **Definition:** A link-state routing protocol that computes optimal paths inside a single autonomous system. **How it works:** Every router floods its link state across the area, so routers in the same area converge on consistent link-state information, then each runs Dijkstra's algorithm to compute shortest paths. Interface costs are configured (often derived from bandwidth), and networks are split into areas to scale. **In practice:** OSPF is an IGP — for internal networks. Convergence is fast and paths are found automatically, but each router maintains link-state information for its attached areas, so at scale, area design determines performance. ### BGP **Definition:** A path-vector routing protocol that exchanges reachability between autonomous systems (ASes). **How it works:** BGP's goal differs from OSPF's: it picks not "the fastest path" but "the path policy prefers." Each AS advertises the prefixes it can reach along with the AS path; receivers rank routes by attributes such as AS_PATH length, Local Preference, and MED. Routing for the entire internet rests on this. **In practice:** BGP trusts advertisements by default, which is why bad prefix advertisements can cause widespread outages. RPKI origin validation checks whether the prefix origin is authorized; it does not validate the entire AS path or stop all route leaks. From a cloud perspective, Direct Connect uses BGP; Site-to-Site VPN can use BGP or supported static routing, so AS numbers, advertised prefix design, and path preference for redundancy (AS_PATH prepending and friends) become real design items. > 📎 For how Calico uses BGP inside a cluster, see [Calico BGP Deep Dive](https://www.atomai.click/kubernetes-docs/llms/en/networking/calico/04-bgp-deep-dive.md). ### NAT **Definition:** A function that translates IP addresses and, for NAPT/PAT, transport ports. **How it works:** A common case is many private hosts sharing a public address through PAT/NAPT. Translation can also be private-to-private; it is not always public-internet address sharing. A translation table keeps per-session mappings so return packets find their way back to the right internal host. **In practice:** NAT is the poster child for layering violations: an L3 device that rewrites L4 ports, and it breaks end-to-end connectivity — the internet's original premise. As a result P2P becomes hard, and workarounds such as STUN/TURN become necessary (see WebRTC below). In the cloud, NAT Gateway port exhaustion and data processing charges are the practical issues. For outbound-heavy workloads, VPC endpoints can reduce NAT processing for supported AWS services; compare their hourly/data charges and traffic path before assuming savings. --- **Next:** [Part 2: The Transport Layer and TLS](https://www.atomai.click/kubernetes-docs/llms/en/basics/06-network-fundamentals-part2.md) ## Verification References - https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/network_mtu.html - https://www.rfc-editor.org/rfc/rfc894 - https://www.rfc-editor.org/rfc/rfc6691 - https://www.rfc-editor.org/rfc/rfc4638 - https://www.rfc-editor.org/rfc/rfc5227 - https://www.rfc-editor.org/rfc/rfc792 - https://www.rfc-editor.org/rfc/rfc8899 - https://www.rfc-editor.org/rfc/rfc2328 - https://www.rfc-editor.org/rfc/rfc6811 - https://docs.kernel.org/networking/bridge.html - https://docs.aws.amazon.com/vpc/latest/userguide/VPC_Internet_Gateway.html - https://docs.aws.amazon.com/vpc/latest/userguide/egress-only-internet-gateway.html - https://docs.aws.amazon.com/vpn/latest/s2svpn/VPNRoutingTypes.html - https://docs.aws.amazon.com/vpc/latest/userguide/nat-gateway-scenarios.html - https://docs.aws.amazon.com/vpc/latest/userguide/nat-gateway-pricing.html ---------------------------------------- Source: https://www.atomai.click/kubernetes-docs/en/basics/06-network-fundamentals-part2 ---------------------------------------- # Network Fundamentals Part 2 — The Transport Layer and TLS > **Last Updated**: September 11, 2026 ::: tip This is a four-part series [Part 1: The Layer Model, Link and Routing Layers](https://www.atomai.click/kubernetes-docs/llms/en/basics/06-network-fundamentals-part1.md) · **Part 2: The Transport Layer and TLS** *(this document)* · [Part 3: Application Protocols](https://www.atomai.click/kubernetes-docs/llms/en/basics/06-network-fundamentals-part3.md) · [Part 4: A Request's Journey and the Cloud](https://www.atomai.click/kubernetes-docs/llms/en/basics/06-network-fundamentals-part4.md) ::: Part 1 delivered packets to the destination host. This part compares reliable streams (TCP and QUIC) with UDP datagrams, then explains how TLS protects communication. UDP itself does not supply reliability or TLS; applications choose an appropriate security protocol, such as DTLS, or use a transport such as QUIC that integrates TLS 1.3. One picture summarizes the heart of this part: ![Typical fresh connection: TCP plus a full TLS 1.3 handshake takes about 2 RTTs before a request, while QUIC combines these into about 1 RTT. Eligible resumption can send 0-RTT early data before handshake completion.](https://raw.githubusercontent.com/Atom-oh/kubernetes-docs/main/en/.gitbook/assets/en-basics-06-network-fundamentals-part2-0.png) [🔍 View interactive diagram](https://www.atomai.click/kubernetes-docs/archmaps/en-basics-06-network-fundamentals-part2-0.html) --- ## 3. Transport Layer — End-to-End Delivery From here on, your conversation partner is not a "network" but a "process." That is why port numbers appear. ### TCP **Definition:** A connection-oriented transport protocol providing a reliable, ordered byte stream. **How it works:** A 3-way handshake (SYN → SYN+ACK → ACK) establishes the connection. Sequence numbers preserve order, ACKs and retransmission recover losses, sliding windows control flow, and congestion control adapts to network load. To the application, TCP presents a clean abstraction: a gapless stream of bytes. **In practice:** Ordered delivery causes **head-of-line (HOL) blocking**: a missing TCP segment prevents delivery of bytes beyond the gap. With HTTP/2, this can delay multiple streams sharing that connection; data already delivered and independent application work can still progress. QUIC removes this particular cross-stream transport ordering dependency. For a typical fresh connection without optimizations, TCP setup costs about 1 RTT, followed by a full TLS 1.3 handshake of 1 RTT (typically 2 RTTs for TLS 1.2). Connection reuse, resumption, early data and TCP Fast Open change the timing; retries can add delay. Assess connection pooling against the actual workload. **A short lineage of congestion control:** Congestion control influences throughput along with bandwidth, latency, buffers and application behavior. Classic **Reno** reduces its congestion window on loss. **CUBIC**, a common Linux default, uses a cubic window-growth function to improve scalability on high-bandwidth paths. **BBR** models bottleneck bandwidth and propagation RTT to guide sending; its use of loss and ECN also depends on the implementation/version. No algorithm guarantees higher throughput on every long-haul or mobile path. `sysctl net.ipv4.tcp_congestion_control` only **reads** the configured default; changing the default affects new connections and requires separate configuration and measurement. **TIME_WAIT and port exhaustion:** In a normal graceful close, the active closer generally enters TIME_WAIT; simultaneous close can put both peers there. High connection churn can contribute to exhaustion of available connection tuples, ephemeral ports or NAT mappings, depending on the implementation and destination pattern. A TIME_WAIT entry does not universally reserve that port against every remote endpoint. Diagnose the actual limit and consider connection reuse before changing kernel settings; TIME_WAIT also protects against delayed packets from an old connection. ### UDP **Definition:** A minimal transport protocol that sends datagrams with no connection setup. **How it works:** The 8-byte header carries only source port, destination port, length, and checksum. No handshake, no retransmission, no ordering, no congestion control. It is essentially "IP with port numbers." **In practice:** Real-time applications can prefer timely delivery over retransmitting stale data, and DNS commonly uses UDP for small exchanges. Applications must implement any required reliability and appropriate congestion control themselves, or use a protocol such as QUIC that provides them. The caveat: being stateless makes UDP easy to abuse for spoofing and amplification attacks. When exposing UDP services externally, plan for response-size limits and request-rate control. ### QUIC **Definition:** A secure, multiplexed transport protocol implemented on top of UDP. **How it works:** QUIC redesigns, from scratch on UDP, everything TCP+TLS used to do. Four key properties: 1. **Independent stream delivery** — each stream has its own byte ordering, so loss on one stream need not prevent delivery of another stream’s available data. Packet recovery and congestion control still operate across the connection/path; a stream can block on its own missing bytes, and application or HTTP/3 QPACK dependencies can also cause blocking. 2. **Built-in encryption** — QUIC integrates the TLS 1.3 handshake and uses its own packet protection rather than TLS records. A normal full handshake takes about 1 RTT. Eligible, accepted resumption can carry **0-RTT early data**, but the handshake still completes later; Retry or additional handshake exchanges can increase latency. 3. **Connection IDs** — these support connection continuity through address changes, with path validation and endpoint support. Migration restrictions or unavailable paths can still interrupt a Wi-Fi-to-cellular transition; continuity is not guaranteed. 4. **Implementation flexibility** — QUIC is commonly implemented in user space, allowing transport changes to ship with an application or library. User-space implementation is not a protocol requirement. **In practice:** HTTP/3 usually uses UDP 443. If that path is blocked, an HTTP client can try HTTP/2 or HTTP/1.1 over TCP when the server supports them; QUIC itself does not turn into TCP. Check reachability, negotiated protocol and implementation/offload behavior before attributing a performance result to QUIC. CPU cost depends on the implementation and workload. One security caveat: **0-RTT data can be replayed.** Replaying an early-data exchange can make an application process a request more than once; transport packet deduplication alone does not provide application replay protection. Permit only operations the application has explicitly assessed as replay-safe. A GET name or an idempotency claim alone is insufficient. Servers can reject early data; HTTP servers can use `425 Too Early` so the client retries after the handshake. Configure this policy across the client, CDN and origin. --- ## 4. Security — TLS ### TLS **Definition:** The protocol providing confidentiality, integrity, and authentication for data in transit. **How it works:** TLS negotiates cryptographic parameters and establishes keys during a handshake. Certificate-based handshakes authenticate the server using a certificate and proof of key possession; PSK-based handshakes can authenticate using a previously established or externally provisioned key instead. TLS records protect application data. TLS 1.3 uses authenticated encryption (AEAD) for confidentiality and integrity. A normal full TLS 1.3 handshake takes about 1 RTT; resumption permits optional early data under additional conditions. TLS 1.3 removed static RSA key exchange and legacy cipher suites, but RSA certificate signatures are still supported. Ephemeral (EC)DHE key exchange provides forward secrecy, including when combined with a PSK. **PSK-only key exchange and 0-RTT data do not provide the same forward-secrecy guarantee.** **In practice:** Three things go wrong over and over. - **Certificate expiry** — automate renewal and separately monitor expiry and successful certificate deployment. - **SNI exposure** — TLS 1.3 alone leaves the ClientHello SNI visible. ECH (Encrypted Client Hello, RFC 9849) can protect the inner ClientHello when supported and configured by both endpoints. QUIC Initial packet keys are publicly derivable, so Initial encryption alone does not hide SNI. ECH also does not hide the destination IP or all traffic metadata. - **Termination point design** — document every hop: client to load balancer, load balancer to application, and any service-to-service connection. TLS termination does not automatically encrypt the next hop. Use TLS there when required; use mTLS when both peers must authenticate with certificates. A service mesh can automate this, but is not required for every design. **Certificate chains and OCSP stapling:** Typical X.509 validation builds a path from the leaf certificate, through any required intermediates, to a configured trust anchor. The server should send the needed intermediate certificates; the root is normally already trusted by the client. Missing intermediates can cause client-dependent failures, although not every valid chain contains an intermediate. Revocation handling depends on the issuer and client. Where OCSP is supported, stapling lets the server attach a signed status response and can reduce direct client lookups. It is not universal: Let’s Encrypt ended OCSP service in August 2025 and uses CRLs. Match certificate and revocation configuration to the actual CA and clients. > 📎 For how Istio automates this, see [Istio mTLS](https://www.atomai.click/kubernetes-docs/llms/en/service-mesh/istio/security/01-mtls.md). **Primary references**: [TLS 1.3](https://www.rfc-editor.org/rfc/rfc8446.html), [QUIC transport](https://www.rfc-editor.org/rfc/rfc9000.html), [QUIC/TLS](https://www.rfc-editor.org/rfc/rfc9001.html), [HTTP early data](https://www.rfc-editor.org/rfc/rfc8470.html), [ECH](https://www.rfc-editor.org/rfc/rfc9849.html), [Linux TCP settings](https://docs.kernel.org/networking/ip-sysctl.html), [Let’s Encrypt OCSP retirement](https://letsencrypt.org/2025/08/06/ocsp-service-has-reached-end-of-life/). --- **Next:** [Part 3: Application Protocols](https://www.atomai.click/kubernetes-docs/llms/en/basics/06-network-fundamentals-part3.md) ---------------------------------------- Source: https://www.atomai.click/kubernetes-docs/en/basics/06-network-fundamentals-part3 ---------------------------------------- # Network Fundamentals Part 3 — Ten Application Protocols > **Last Updated**: September 11, 2026 ::: tip This is a four-part series [Part 1: The Layer Model, Link and Routing Layers](https://www.atomai.click/kubernetes-docs/llms/en/basics/06-network-fundamentals-part1.md) · [Part 2: The Transport Layer and TLS](https://www.atomai.click/kubernetes-docs/llms/en/basics/06-network-fundamentals-part2.md) · **Part 3: Application Protocols** *(this document)* · [Part 4: A Request's Journey and the Cloud](https://www.atomai.click/kubernetes-docs/llms/en/basics/06-network-fundamentals-part4.md) ::: Transport protocols provide streams or datagrams; application protocols turn them into services. This part covers name resolution (DNS and DoH), bootstrapping (DHCP), operational access (SSH), mail (SMTP), and HTTP/3, WebSocket, WebRTC, gRPC and MQTT. --- ## 5. Application Layer — Actual Services ### DNS **Definition:** The distributed directory system that resolves domain names into IP addresses and other records. **How it works:** DNS uses hierarchical delegation. On a cache miss, a recursive resolver follows referrals from root to TLD to authoritative nameservers, or forwards to another resolver. Cached answers can avoid some or all of that work. A/AAAA records contain addresses, CNAME records aliases, MX records mail servers, and TXT records text used by several protocols. **In practice:** DNS is distributed, but a resolver, provider or configuration can become a shared dependency. For DNS failover, account for failure detection, record updates, the TTL of answers already cached, application caching and existing connections. Reducing TTL now does not shorten the TTL of an old cached answer. Some resolvers also serve stale answers under defined failure conditions (RFC 8767). Measure each stage; short TTL alone is not a failover-time guarantee. Load balancers and anycast can complement DNS, with their own health detection and convergence limits. **Common record types at a glance:** | Type | Purpose | Field note | |---|---|---| | A / AAAA | Domain → IPv4 / IPv6 | The basics | | CNAME | Alias → canonical name | Cannot coexist with apex SOA/NS; provider-specific ALIAS/ANAME or Route 53 Alias can offer apex mapping to supported targets | | MX | Mail-receiving server | Lower priority number wins | | TXT | Arbitrary strings | SPF/DKIM/DMARC, domain-ownership verification | | NS | Delegated nameservers | Sub-zone delegation | | SRV | Service location (host+port) | Discovery for some protocols | | CAA | Restrict authorized certificate issuers | Requires CA enforcement; does not itself prevent every mis-issuance | **DNSSEC and DoH solve different problems.** DNSSEC authenticates signed DNS data and its integrity through a validated trust chain; it does not encrypt queries. DoH uses HTTPS to authenticate the chosen resolver and protect confidentiality and integrity on the client–resolver hop. It does not prove that a malicious or mistaken resolver returned authoritative data. They can be used together. ![Shows recursive DNS resolution: the stub resolver's query walks through the recursive resolver down the root, TLD, and authoritative nameservers, with the answer cached for its TTL.](https://raw.githubusercontent.com/Atom-oh/kubernetes-docs/main/en/.gitbook/assets/en-basics-06-network-fundamentals-part3-0.png) [🔍 View interactive diagram](https://www.atomai.click/kubernetes-docs/archmaps/en-basics-06-network-fundamentals-part3-0.html) ### DoH **Definition:** DNS queries wrapped in and transported over HTTPS. **How it works:** Traditional DNS commonly uses plaintext UDP **and TCP** port 53. DoH carries DNS messages over HTTPS, protecting them from passive inspection and modification on that hop. Resolver endpoints and traffic metadata may still identify DoH use; the chosen resolver can see the queries. **In practice:** An independently selected public DoH resolver can bypass filtering/logging at the organization’s resolver and fail to resolve private names. DoH does not inherently disable policy: a managed DoH resolver can apply logging/filtering, and browser/OS policies can select approved resolvers. Test split DNS and endpoint policy instead of assuming that disabling encryption is always necessary. ### DHCP **Definition:** The protocol that automatically assigns hosts an IP address and network configuration. **How it works:** A common initial **DHCPv4** exchange is DORA: Discover → Offer → Request → Acknowledge. A local broadcast or a DHCP relay locates a server. The lease can include IPv4 address, subnet mask, gateway and DNS configuration. Renewal can use a shorter exchange. DHCPv6 uses different messages; IPv6 default-router information normally comes from Router Advertisements, and SLAAC is another address-configuration mechanism. **In practice:** In the cloud it is mostly abstracted away, but you meet it again in the VPC DHCP option set, which is where DNS servers and domain names are configured. When name resolution breaks in a hybrid setup that uses on-premises DNS, this is the setting to check. ### SSH **Definition:** The protocol providing encrypted remote shell access and tunneling. **How it works:** The server authenticates itself with its host key, a key exchange derives session keys, and then the user authenticates (public key or password). All subsequent traffic is encrypted. Beyond remote shells, SSH supports port forwarding, SFTP, and agent forwarding. **In practice:** Restrict forwarding according to the access policy and validate server host keys. A compromised host with access to a forwarded agent socket can request signatures/authentication from the agent; forwarding does not ordinarily copy the private key material there. Prefer a jump host (`ProxyJump`) when agent forwarding is unnecessary. Raw keys have no intrinsic expiry, but OpenSSH supports certificate validity periods and `authorized_keys` expiry restrictions. Remove departed users’ access and rotate or revoke credentials. AWS Systems Manager Session Manager can provide shell access without inbound SSH ports or distributing SSH keys, provided the managed node, IAM permissions and service connectivity are configured. CloudTrail records API activity; shell-content logging to CloudWatch Logs/S3 requires configuration. **Session content logging is unavailable for Session Manager SSH and port-forwarding sessions.** IAM-based access by itself does not imply that every command is recorded. ### SMTP **Definition:** The protocol that relays messages between mail servers. **How it works:** Clients submit mail to a submission server, and SMTP servers relay and receive messages, commonly using MX lookup for routing. IMAP and POP3 let users retrieve or access messages already stored in a mailbox; they do not replace SMTP’s server-side receipt. **In practice:** SMTP authentication and TLS secure submission/transport, but do not by themselves prove the visible sender domain. Three complementary domain mechanisms matter: - **SPF** — authorize sending hosts for the envelope MAIL FROM or HELO identity; this is not automatically the visible From header. - **DKIM** — verify a signature over covered message content using the signing domain’s DNS key; the signing domain can differ from the visible From domain. - **DMARC** — require the visible From domain to align with a passing SPF **or** DKIM identity, and publish requested handling/reporting policy. Configure SPF, DKIM and DMARC together where appropriate, monitor reports, and account for forwarding/mailing-list behavior. DMARC can pass with one aligned mechanism. These controls neither guarantee delivery nor eliminate display-name or lookalike-domain impersonation; receivers also apply local policy. ### HTTP/3 **Definition:** The third major version of HTTP, running on QUIC. **How it works:** HTTP semantics are shared across versions, but HTTP/3 uses QUIC streams and its own framing and mapping. It removes TCP’s cross-stream ordering dependency; within-stream loss, QPACK dependencies and shared congestion control can still delay work. A typical full handshake takes about 1 RTT, and supported migration can preserve a connection through an address change. QPACK replaces HPACK to accommodate independently delivered streams. **In practice:** Clients can discover HTTP/3 through `Alt-Svc`, prior knowledge or HTTPS DNS records advertising a supported protocol. `Alt-Svc` may be learned through an earlier TCP connection; a client that supports the HTTPS record can discover HTTP/3 before that exchange. Neither method guarantees reachability or a particular latency saving. Independent delivery and integrated handshakes can help on lossy or high-latency paths. Actual latency, throughput and CPU cost depend on implementation, offloads, workload and network conditions. Measure representative mobile and data-center traffic rather than assuming a universal win or loss. **The three generations side by side:** | | HTTP/1.1 | HTTP/2 | HTTP/3 | |---|---|---|---| | Transport | TCP | TCP | QUIC (UDP) | | Requests per connection | Sequential, or pipelined with ordered responses | Multiplexed | Multiplexed | | HOL blocking | Ordered responses and TCP delivery | TCP ordering across streams | No TCP cross-stream ordering; other blocking remains | | Header compression | None | HPACK | QPACK | | Encryption | Optional (HTTPS) | TLS for HTTPS; cleartext HTTP/2 also exists | TLS 1.3 integrated into QUIC | Multiplexing changes where ordering dependencies arise; HTTP/3 reduces one source of blocking without eliminating all scheduling, flow-control or application dependencies. ### WebSocket **Definition:** An application protocol for bidirectional messaging over a single connection. **How it works:** The HTTP/1.1 handshake uses `Upgrade` and a successful `101` response. HTTP/2 and HTTP/3 use Extended CONNECT instead (RFCs 8441 and 9220), when supported. Once established, either peer can send WebSocket messages without repeated HTTP polling. **In practice:** Plan for long-lived connections: heartbeat traffic within the relevant idle timeout, graceful draining during deployment, and reconnect backoff with jitter. Each socket remains on its owning instance. Shared application state or messaging, such as Redis Pub/Sub, can deliver events across instances but does not transfer live sockets or provide durable delivery by itself. Verify the handshake used by the negotiated HTTP version and the proxy’s support. ### WebRTC **Definition:** APIs and protocols for real-time media and data between compatible endpoints, including browsers and media servers. **How it works:** NAT can obstruct direct reachability, but two peers behind NAT may still connect. ICE exchanges and tests host, server-reflexive (learned with STUN) and relayed (TURN) candidates. Application signaling carries session descriptions and candidates. The selected path depends on connectivity checks and policy. Media uses SRTP, commonly with DTLS-SRTP key establishment; data channels use SCTP over DTLS. **In practice:** TURN relay usage contributes bandwidth and infrastructure cost; signaling, STUN and other service costs remain even with a direct media path. NAT mapping/filtering and firewall behavior influence connectivity, so the label “symmetric NAT” alone is not a universal proof that relay is unavoidable. Budget for TURN fallback and test actual networks. An SFU is a common multiparty design that trades server bandwidth/compute for reduced client upload compared with a full peer mesh. ### gRPC **Definition:** An RPC framework whose standard native transport uses HTTP/2, commonly with Protocol Buffers service and message schemas. **How it works:** Protocol Buffers definitions can generate client/server code and support unary, server-streaming, client-streaming and bidirectional-streaming RPCs. Binary encoding can be compact, but size and speed relative to JSON depend on data, implementation and compression; they are not protocol guarantees. **In practice:** Native gRPC works well for many service APIs. Browser APIs do not expose everything native gRPC requires, so browser clients commonly use gRPC-Web with a compatible server or translating proxy; available streaming modes depend on that implementation. Use schema-aware tools for inspection and debugging. A gRPC channel can use **zero or more HTTP/2 connections**, and many RPCs can share a long-lived connection. L4 balancing selects a backend per connection, so a small connection pool can concentrate RPC traffic; it does not guarantee per-RPC distribution. Consider a suitable client-side policy or a gRPC-aware L7 proxy (which may be part of a service mesh). Established streams still remain with their selected backend. For schema evolution, reserve deleted Protocol Buffers field numbers/names and never reuse their numbers. > 📎 For gRPC handling in Istio, see [Istio gRPC Advanced](https://www.atomai.click/kubernetes-docs/llms/en/service-mesh/istio/advanced/05-grpc.md). ### MQTT **Definition:** A lightweight publish-subscribe messaging protocol. **How it works:** Clients connect to a broker and publish/subscribe to topics. A fixed header can be as small as 2 bytes, but real packets can also need variable headers, properties and payloads. QoS 0/1/2 provide at-most-once, at-least-once and exactly-once **protocol delivery on the relevant sender–receiver leg**. Publisher-to-broker and broker-to-subscriber delivery are separate. A configured Will can be published on specified disconnection conditions; MQTT 5 Will Delay and reconnect behavior affect when it appears. **In practice:** Choose QoS according to loss/duplicate tolerance and cost. Successful QoS 2 delivery normally exchanges PUBLISH, PUBREC, PUBREL and PUBCOMP; it does not make an application’s database side effects or an entire business workflow exactly once. QoS 1 plus application deduplication is one possible trade-off. Plan broker availability, durable session/message state and recovery for the chosen product. Use TLS and an appropriate device authentication/authorization scheme; client certificates are one option, with provisioning and rotation requirements. **Primary references**: [DoH](https://www.rfc-editor.org/rfc/rfc8484.html), [DNS serve-stale](https://www.rfc-editor.org/rfc/rfc8767.html), [OpenSSH](https://man.openbsd.org/ssh), [Session Manager](https://docs.aws.amazon.com/systems-manager/latest/userguide/session-manager.html), [DMARC](https://www.rfc-editor.org/rfc/rfc7489.html), [HTTP/3](https://www.rfc-editor.org/rfc/rfc9114.html), [ICE](https://www.rfc-editor.org/rfc/rfc8445.html), [gRPC performance](https://grpc.io/docs/guides/performance/), [MQTT 5.0](https://docs.oasis-open.org/mqtt/mqtt/v5.0/os/mqtt-v5.0-os.html). --- **Next:** [Part 4: A Request's Journey and the Cloud](https://www.atomai.click/kubernetes-docs/llms/en/basics/06-network-fundamentals-part4.md) ---------------------------------------- Source: https://www.atomai.click/kubernetes-docs/en/basics/06-network-fundamentals-part4 ---------------------------------------- # Network Fundamentals Part 4 — A Request's Journey and the Cloud Mapping > **Last Updated**: September 11, 2026 ::: tip This is a four-part series [Part 1: The Layer Model, Link and Routing Layers](https://www.atomai.click/kubernetes-docs/llms/en/basics/06-network-fundamentals-part1.md) · [Part 2: The Transport Layer and TLS](https://www.atomai.click/kubernetes-docs/llms/en/basics/06-network-fundamentals-part2.md) · [Part 3: Application Protocols](https://www.atomai.click/kubernetes-docs/llms/en/basics/06-network-fundamentals-part3.md) · **Part 4: A Request's Journey and the Cloud** *(this document)* ::: This part connects the series’ 25 protocols and mechanisms through an illustrative request, then maps related responsibilities in AWS and Kubernetes. These are functional comparisons, not one-to-one replacements. ![Illustrative request path: address configuration and DNS, local delivery and routing, optional NAT, then TCP plus TLS for HTTP/1.1 or HTTP/2, or QUIC with integrated TLS for HTTP/3. Cache reuse and network configuration can skip steps.](https://raw.githubusercontent.com/Atom-oh/kubernetes-docs/main/en/.gitbook/assets/en-basics-06-network-fundamentals-part4-0.png) [🔍 View interactive diagram](https://www.atomai.click/kubernetes-docs/archmaps/en-basics-06-network-fundamentals-part4-0.html) --- ## 6. Following One Request All the Way Through For a new connection to `https://example.com`, the following is a conceptual dependency guide. It is not a packet trace: DNS queries and handshake packets themselves use the link and routing layers, and cached state or an existing connection can skip work. 1. **Address configuration** — the host already has addresses, routes and resolver settings, from DHCP, static configuration, IPv6 SLAAC/Router Advertisements or another managed mechanism. 2. **DNS (or DoH)** — resolve the destination when needed. A recursive resolver can use its cache, follow delegations or forward queries; the application need not contact the root. HTTPS records may also advertise connection parameters. 3. **Neighbor resolution** — for IPv4 over Ethernet, resolve the selected next hop’s MAC with ARP if it is not cached; IPv6 uses Neighbor Discovery. The next hop can be a local destination or a router. 4. **Ethernet / Wi-Fi** — send frames to that next hop. A default gateway is used only when the selected route calls for it. 5. **IP routing** — routers forward using their forwarding tables. Routes may be static, connected or learned through BGP, OSPF or another control plane; routing protocols do not run a fresh negotiation for every request. 6. **Optional NAT** — an IPv4 internet egress path may translate a private address to a public address. Many internal and IPv6 paths do not use NAT; private-to-private NAT also exists. 7. **TCP or QUIC** — establish or reuse transport. HTTP/1.1 and HTTP/2 commonly use TCP; HTTP/3 uses QUIC over UDP. 8. **TLS** — authenticate the peer and establish traffic keys according to the handshake mode. For QUIC, TLS 1.3 is integrated with step 7; resumption differs from a fresh certificate-based handshake. 9. **HTTP** — exchange the request and response using the negotiated version. HTTP/3 requires the QUIC branch; it does not run over the TCP branch. 10. **Optional application features** — WebSocket, browser-compatible gRPC or WebRTC may create additional connections or reuse/multiplex existing transports, depending on implementation. **ICMP** can report certain IP-layer errors, such as an unreachable destination or a packet too large for a path. It does not report every failure: packets or ICMP errors can be filtered, and TLS/application failures use their own mechanisms. Combine allowed, relevant ICMP with transport/application logs and measurements; absence of an ICMP error is not proof of success. --- ## 7. Where These Concepts Go in the Cloud Cloud networking retains addressing, routing, filtering and transport responsibilities, but the boundaries differ from traditional appliances. In AWS: | Traditional concept | AWS counterpart | |---|---| | Segmentation and filtering | VPC/subnets for logical network boundaries; security groups and NACLs for filtering, not VLAN equivalents | | Routing tables | VPC route tables, Transit Gateway | | BGP peering | Direct Connect virtual interfaces; dynamically routed Site-to-Site VPN (static VPN routing is also possible) | | NAT / private service access | NAT Gateway translates addresses; VPC endpoints provide private paths to supported services | | DNS servers | Route 53, Resolver endpoints | | DHCP | VPC DHCP option sets | | TLS termination / certificates | ALB HTTPS listeners, NLB TLS listeners or CloudFront; ACM manages supported certificates rather than forwarding traffic | | L7 load balancing | ALB; application proxies such as Istio/Envoy in a separately managed service mesh | | SSH access | Systems Manager Session Manager | | Internal-segment encryption | TLS/mTLS in applications or proxies; network-layer encryption is a separate design option | AWS App Mesh is a historical example, not a new-design default: AWS has announced support ends on **September 30, 2026**. Plan migration for existing deployments. **Three decisions to make first** when designing: 1. **IP address plan** — plan CIDRs for networks that must interconnect, including on-premises, Pod and Service ranges. Overlap can require translation or redesign; renumbering has operational cost, but no universal cost ranking applies. 2. **Outbound path** — match each destination to an internet or private-service path. Compare NAT hourly/data charges, interface-endpoint hourly/data charges, cross-AZ transfer and availability requirements. S3/DynamoDB gateway endpoints have no additional endpoint charge, but endpoints do not replace every internet destination. 3. **Encryption termination point** — document encryption and authentication on each hop, including the backend connection after a load balancer. Check the workload’s requirements and applicable policy; TLS termination alone does not secure the next hop. --- ## 8. Who Does This Work in Kubernetes Inside a cluster the same concepts repeat with new component names. This table is the bridge from this series to the deep-dive documents that follow. | Traditional concept | Kubernetes counterpart | |---|---| | Pod IP assignment | CNI/IPAM integration (VPC CNI, Cilium, …); this does not necessarily use DHCP per Pod | | Local delivery / forwarding | Host interfaces, neighbor handling and CNI datapath; implementation may use veth, routes, tunnels or eBPF | | DNS | Cluster DNS, often CoreDNS; `service.namespace.svc.` (`cluster.local` is a common configured domain) | | Service virtual IP / L4 balancing | kube-proxy on Linux: iptables or nftables; IPVS is deprecated since 1.35. An eBPF implementation can replace kube-proxy; it is not a kube-proxy mode | | Pod traffic policy | NetworkPolicy, enforced only by a networking controller/plugin that supports it | | L7 routing / TLS termination | Ingress/Gateway API resources plus an implementing controller and dataplane | | Service-to-service mTLS | Application TLS or a configured mesh such as Istio/Linkerd; not enabled merely by installing any CNI | | BGP routing / advertisement | For example, Calico BGP routes or MetalLB BGP advertisement of Service addresses; roles differ | For an ordinary ClusterIP Service backed by Pods, cluster DNS usually resolves the Service IP, then kube-proxy or its replacement selects an eligible endpoint using Service/EndpointSlice state. The datapath forwards to that endpoint on the same or another node. Headless Services instead expose endpoint addresses through DNS, and ExternalName Services return a CNAME. mTLS applies only when the relevant peers and policies are configured. These variations are why “every layer always runs” is not a valid packet-flow assumption. --- ## Wrapping Up After walking through these 25 protocols and mechanisms, examine the trade-offs and the scope of each guarantee. TCP supplies ordered, reliable delivery with recovery and ordering costs. UDP leaves these functions to higher layers. QUIC supplies reliable streams and integrated security over UDP. NAT conserves public IPv4 addresses while complicating unsolicited reachability, which ICE/STUN/TURN help address. DoH protects a resolver hop; organizational visibility depends on the resolver and endpoint policy. When investigating a failure, use each layer’s actual guarantees and observable evidence to narrow the fault domain. A plausible protocol-level explanation is a hypothesis until logs, traces or measurements distinguish it from alternatives. --- ## Next Documents From this foundation, move on to cluster networking: - [eBPF Fundamentals](https://www.atomai.click/kubernetes-docs/llms/en/basics/05-ebpf-fundamentals.md) — how packets are processed in the kernel - [Cilium Networking](https://www.atomai.click/kubernetes-docs/llms/en/networking/cilium/03-networking.md) — the eBPF-based CNI - [Calico BGP Deep Dive](https://www.atomai.click/kubernetes-docs/llms/en/networking/calico/04-bgp-deep-dive.md) — BGP routing inside the cluster - [Amazon VPC CNI](https://www.atomai.click/kubernetes-docs/llms/en/networking/01-vpc-cni.md) — the VPC CNI and IP allocation ## References The protocol list was seeded by ByteByteGo's "What Keeps the Internet Running?" infographic; the explanations and practical commentary were written independently. Primary references: [Kubernetes Service proxy modes](https://kubernetes.io/docs/reference/networking/virtual-ips/), [Service DNS](https://kubernetes.io/docs/concepts/services-networking/dns-pod-service/), [NetworkPolicy](https://kubernetes.io/docs/concepts/services-networking/network-policies/), [NAT Gateway cost guidance](https://docs.aws.amazon.com/vpc/latest/userguide/nat-gateway-pricing.html), [ECR VPC endpoints](https://docs.aws.amazon.com/AmazonECR/latest/userguide/vpc-endpoints.html), [App Mesh lifecycle](https://docs.aws.amazon.com/app-mesh/latest/userguide/what-is-app-mesh.html). ---------------------------------------- Source: https://www.atomai.click/kubernetes-docs/en/networking/01-vpc-cni ---------------------------------------- # Amazon VPC CNI > **Review baseline**: VPC CNI / Helm chart 1.23.0; network policy agent 1.4.1. > **Last reviewed**: September 11, 2026. Select an EKS add-on build compatible with the actual cluster version and Region. Upstream, Helm and EKS `eksbuild` versions are separate identifiers. ## Table of Contents - [VPC CNI Overview](#vpc-cni-overview) - [Networking Model](#networking-model) - [Installation and Configuration](#installation-and-configuration) - [IP Address Management](#ip-address-management) - [Network Policy Support](#network-policy-support) - [Advanced Features](#advanced-features) - [Troubleshooting](#troubleshooting) - [Best Practices](#best-practices) ## VPC CNI Overview Amazon VPC CNI supplies VPC-native Pod networking on standard EKS EC2 nodes. This guide's `aws-node` DaemonSet commands target that installation. Auto Mode runs managed networking components as node services; Fargate and Windows use different management paths. See the [overview](https://www.atomai.click/kubernetes-docs/llms/en/networking/README.md) before applying EC2/Linux instructions to another compute mode. Pods use VPC-routable addresses without an overlay encapsulation requirement. Routing, security groups, network policies, ENI/IP limits and application behavior still determine connectivity and performance. EKS supports **IPv4 or IPv6 Pod/Service addressing**, selected when creating the cluster; it does not support dual-stacked Pods or Services. A dual-stack VPC or an IPv6 Pod's IPv4 egress helper is a different concept. ### Architecture | Component | Responsibility | |---|---| | Container runtime / CNI binary | The runtime invokes CNI for the Pod sandbox; the AWS plugin requests an address and configures its network namespace. | | IPAMD | Maintains address pools and manages the required ordinary ENIs/IPs on Linux EC2 nodes. | | EKS network policy controller / node agent | The managed controller resolves policy endpoints; `aws-eks-nodeagent` enforces supported policies using eBPF when enabled. | | VPC resource controller | Manages features such as branch/trunk interfaces and Windows address allocation under their own prerequisites. | The former diagram labeled the CNI binary as directly called by kubelet. Current Kubernetes delegates CNI management to the container runtime. ### IP Allocation Modes | Property | Secondary IPv4 addresses | Prefix delegation | |---|---|---| | Allocation | Individual secondary addresses on an ENI | IPv4 `/28` prefixes with 16 addresses; IPv6 uses `/80` prefixes | | Capacity | Constrained by interface/address slots and kubelet settings | More addresses per slot, subject to supported hardware, free prefixes and kubelet/resource limits | | Allocation tradeoff | Fine-grained address allocation | Allocates a block at once; warm targets can reserve unused addresses | | Selection | Use according to compatibility and measured demand | Verify Nitro support, subnet fragmentation, workload churn and feature combinations | Prefix delegation is not a universal requirement for a large cluster, and does not manufacture address space in an exhausted subnet. ## Networking Model ### ENI Architecture For ordinary secondary-IPv4 mode, each ENI has a primary address and additional addresses available to the CNI. The primary ENI also carries the node's primary address. Additional ENIs can supply more secondary Pod addresses. Custom networking changes which interfaces/subnets supply Pod addresses; prefix and branch-ENI modes have different allocation rules. ```text Linux EC2 node — ordinary secondary-IPv4 illustration ├── Primary ENI: node primary IP + secondary IPs for Pods ├── Additional ENI: its primary IP + secondary IPs for Pods └── Additional ENI: its primary IP + secondary IPs for Pods ``` ### Instance Type ENI/IP Limits | Instance type | Max ENIs | IPv4 slots per ENI | Legacy secondary-IP bootstrap maxPods | |---|---|---|---| | t3.medium | 3 | 6 | 17 | | t3.large | 3 | 12 | 35 | | m5.large | 3 | 10 | 29 | | m5.xlarge | 4 | 15 | 58 | | m5.2xlarge | 4 | 15 | 58 | | c5.4xlarge | 8 | 30 | 234 | | m5.8xlarge | 8 | 30 | 234 | The historical calculation is **`ENIs × (IPv4 slots per ENI − 1) + 2`**. The `+2` accounts for the two host-network system Pods in that bootstrap calculation; for m5.large, `3 × 9 + 2 = 29`. It does not mean all current deployments always have exactly two host-network Pods. These are legacy bootstrap values, not current universal Pod-density recommendations. Prefix delegation, custom networking, branch interfaces, multiple network cards, CPU/memory and kubelet `maxPods` all matter. EKS managed node groups cap `maxPods` at 110 for instances with fewer than 30 vCPUs and 250 otherwise. Inspect the actual node's allocatable capacity. ### Prefix Delegation An EKS managed add-on configuration fragment for Linux IPv4 prefix mode is: ```json { "env": { "ENABLE_PREFIX_DELEGATION": "true", "WARM_PREFIX_TARGET": "1" } } ``` Merge this with the intended add-on configuration using the management procedure below. For a Helm-owned installation, the equivalent `env` mapping belongs in Helm values. Direct `kubectl set env` edits may be reconciled by the chosen manager. IPv4 allocation needs suitable contiguous `/28` blocks, not merely a positive `AvailableIpAddressCount`. Verify subnet reservations/fragmentation and supported Nitro instances. Enabling prefixes does not automatically raise every existing kubelet's Pod limit or increase the branch-ENI Pod limit. ## Installation and Configuration ### Establish Ownership and Compatibility Use configured AWS CLI credentials and a Kubernetes context for the intended cluster. Start with reads: ```bash EKS_REGION=ap-northeast-2 CLUSTER_NAME=my-cluster KUBERNETES_MINOR="$(aws eks describe-cluster --region "$EKS_REGION" \ --name "$CLUSTER_NAME" --query cluster.version --output text)" aws eks describe-addon-versions --region "$EKS_REGION" \ --addon-name vpc-cni --kubernetes-version "$KUBERNETES_MINOR" aws eks describe-addon --region "$EKS_REGION" \ --cluster-name "$CLUSTER_NAME" --addon-name vpc-cni kubectl -n kube-system get daemonset aws-node -o yaml ``` An EKS `ResourceNotFoundException` for `describe-addon` does not prove that no CNI is installed: it may be self-managed. Inspect the existing DaemonSet, ServiceAccount, Helm releases, configuration and IAM model before choosing **one** manager. Auto Mode networking is not installed through this workflow. ### Existing EKS Managed Add-on Export the existing settings and inspect the selected compatible build's schema: ```bash umask 077 aws eks describe-addon --region "$EKS_REGION" \ --cluster-name "$CLUSTER_NAME" --addon-name vpc-cni > vpc-cni-before.json jq -r '.addon.configurationValues // "{}"' vpc-cni-before.json > vpc-cni-config.json : "${VPC_CNI_ADDON_VERSION:?Select a compatible EKS add-on build from the metadata}" aws eks describe-addon-configuration --region "$EKS_REGION" \ --addon-name vpc-cni --addon-version "$VPC_CNI_ADDON_VERSION" ``` Review required intermediate upgrade versions and release changes. Edit `vpc-cni-config.json` to retain the intended existing configuration and incorporate only the selected changes. Do not assume a partial payload or a conflict flag preserves every setting automatically. Confirm the CNI's IAM permissions and its configured IRSA/Pod Identity role; IPv6 needs the corresponding permissions. For an already managed add-on, a reviewed update can use: ```bash set -eu VPC_CNI_UPDATE_ID="$(aws eks update-addon --region "$EKS_REGION" \ --cluster-name "$CLUSTER_NAME" --addon-name vpc-cni \ --addon-version "$VPC_CNI_ADDON_VERSION" \ --configuration-values file://vpc-cni-config.json --resolve-conflicts PRESERVE \ --query update.id --output text)" aws eks describe-update --region "$EKS_REGION" --name "$CLUSTER_NAME" \ --addon-name vpc-cni --update-id "$VPC_CNI_UPDATE_ID" ``` `PRESERVE` is an explicit conflict-handling choice; verify the resulting environment, images and behavior. Recheck the captured update ID until its status is `Successful`; if it is `Failed` or `Cancelled`, inspect its errors before proceeding. Only then check the resulting add-on and DaemonSet: ```bash aws eks describe-addon --region "$EKS_REGION" \ --cluster-name "$CLUSTER_NAME" --addon-name vpc-cni kubectl -n kube-system rollout status daemonset/aws-node --timeout=10m ``` An accepted API request or an earlier DaemonSet's ready state does not prove this update and network validation have completed. For an absent managed add-on after installation/ownership preparation, the create operation is separate: ```bash aws eks create-addon --region "$EKS_REGION" \ --cluster-name "$CLUSTER_NAME" --addon-name vpc-cni \ --addon-version "$VPC_CNI_ADDON_VERSION" \ --configuration-values file://vpc-cni-config.json ``` Do not repeatedly call `create-addon` to turn on individual features, and do not force `OVERWRITE` over an existing customized installation without a migration plan. ### Helm-owned Installation Pin the whole chart, which also selects the matching init and policy-agent components. Overriding only two image tags does not update the rest of the chart: ```bash helm repo add eks https://aws.github.io/eks-charts helm repo update eks helm show values eks/aws-vpc-cni --version 1.23.0 > chart-defaults.yaml helm template aws-vpc-cni eks/aws-vpc-cni --namespace kube-system \ --version 1.23.0 -f helm-values.yaml > rendered-cni.yaml ``` Prepare `helm-values.yaml` for the actual IP family, CNI ServiceAccount/IAM and selected features. The basic Linux examples in this guide use IPv4. Review the rendered resources before applying: ```bash helm upgrade --install aws-vpc-cni eks/aws-vpc-cni --namespace kube-system \ --version 1.23.0 -f helm-values.yaml --wait --timeout 10m ``` These commands assume a clean or Helm-owned installation. Existing EKS-managed or bootstrap-owned resources need a planned ownership migration. EKS partition/registry access and image-pull prerequisites must also match the environment. ### Important Configuration Values | Setting | Meaning | Baseline/default distinction | |---|---|---| | `WARM_IP_TARGET` | Desired free addresses for new ordinary Pod assignments | Unset by default; not a hard maximum | | `MINIMUM_IP_TARGET` | Floor for total allocated addresses | Unset by default; pair with a positive warm-IP target when used | | `WARM_ENI_TARGET` | Desired warm ENI capacity | Released default: 1; IP targets override it | | `WARM_PREFIX_TARGET` | Desired free IPv4 prefixes | Released chart/manifest sets 1; bare daemon documentation says unset | | `ENABLE_PREFIX_DELEGATION` | Select prefix allocation | Linux chart default: `"false"` | | `AWS_VPC_K8S_CNI_CUSTOM_NETWORK_CFG` | Select custom-networking behavior | Default: `"false"` | | `ENI_CONFIG_LABEL_DEF` | Node label key selecting ENIConfig | Daemon default: `k8s.amazonaws.com/eniConfig`; zone-based examples override it | | `ENABLE_POD_ENI` | Enable the EC2 Pod-ENI integration | Default: `"false"`; other SGPP prerequisites still apply | | `POD_SECURITY_GROUP_ENFORCING_MODE` | SGPP routing/SNAT/security-group behavior | Default: `strict` | | `NETWORK_POLICY_ENFORCING_MODE` | Network-policy behavior while a new Pod's rules are being configured | Default: `standard` | The two enforcing-mode settings control different systems. Environment values in EKS configuration payloads are strings. ### Custom Networking (ENIConfig) Create actual Pod subnets and security groups in the intended VPC/AZ, then reference their IDs: ```yaml apiVersion: crd.k8s.amazonaws.com/v1alpha1 kind: ENIConfig metadata: name: ap-northeast-2a spec: subnet: subnet-0123456789abcdef0 securityGroups: - sg-0123456789abcdef0 --- apiVersion: crd.k8s.amazonaws.com/v1alpha1 kind: ENIConfig metadata: name: ap-northeast-2b spec: subnet: subnet-0abcdef0123456789 securityGroups: - sg-0123456789abcdef0 ``` Enable custom networking and use the node's actual zone label: ```json { "env": { "AWS_VPC_K8S_CNI_CUSTOM_NETWORK_CFG": "true", "ENI_CONFIG_LABEL_DEF": "topology.kubernetes.io/zone" } } ``` An explicit ENIConfig node annotation takes precedence over the label. ENIConfig objects alone do not enable custom networking. Plan routing, DNS/security rules, address capacity and workload/node transition; current Pods do not move to a new subnet simply because a new CIDR or ENIConfig exists. ## IP Address Management ### Warm-pool Tuning Use measured Pod demand and churn, available address space and EC2 API limits. These are alternative example targets, not prescriptions based only on cluster size: ```json { "env": { "WARM_IP_TARGET": "2", "MINIMUM_IP_TARGET": "4" } } ``` ```json { "env": { "WARM_IP_TARGET": "5", "MINIMUM_IP_TARGET": "10" } } ``` `MINIMUM_IP_TARGET` is a total allocation floor; `WARM_IP_TARGET` targets free addresses. They take precedence over the ENI/prefix warm-target strategy. With prefix delegation, allocations still happen in prefix-sized units. More warm capacity can reduce allocation waits but consumes addresses, and aggressive changes can increase API calls. ### Adding a Secondary CIDR First review existing associations, connected-network overlap, VPC CIDR restrictions and subnet/routing requirements: ```bash VPC_ID=vpc-0123456789abcdef0 aws ec2 describe-vpcs --region "$EKS_REGION" --vpc-ids "$VPC_ID" \ --query 'Vpcs[0].CidrBlockAssociationSet' ``` The following uses illustrative IDs and address space; perform it only as part of the reviewed VPC plan: ```bash aws ec2 associate-vpc-cidr-block --region "$EKS_REGION" \ --vpc-id "$VPC_ID" --cidr-block 100.64.0.0/16 aws ec2 describe-vpcs --region "$EKS_REGION" --vpc-ids "$VPC_ID" \ --query 'Vpcs[0].CidrBlockAssociationSet' ``` Confirm that the new CIDR association is **associated**, rather than still associating, before creating a subnet in it: ```bash aws ec2 create-subnet --region "$EKS_REGION" --vpc-id "$VPC_ID" \ --cidr-block 100.64.0.0/19 --availability-zone ap-northeast-2a ``` The subnet also needs its intended route table, security rules and CNI selection. Existing Pods keep their current networking until the planned transition. RFC 6598 `100.64.0.0/10` is shared address space, not globally unique private capacity; check overlaps with every connected environment. ### IPv6 Cluster Configuration IP family is selected at cluster creation and cannot be changed afterward. The official `eksctl` interface uses a **configuration file**, not an `--ip-family` flag. A schema example is: ```yaml apiVersion: eksctl.io/v1alpha5 kind: ClusterConfig metadata: name: ipv6-example region: ap-northeast-2 version: '1.35' kubernetesNetworkConfig: ipFamily: IPv6 iam: withOIDC: true addons: - name: vpc-cni - name: coredns - name: kube-proxy managedNodeGroups: - name: linux-nitro amiFamily: AmazonLinux2023 instanceType: m5.large desiredCapacity: 2 privateNetworking: true ``` Replace the example version/Region, choose supported node images and review VPC endpoint/access and IAM settings before creation. Add-on versions omitted here resolve through the supported EKS/eksctl path; inspect and pin the required compatible builds for a controlled deployment. `iam.withOIDC` and the managed add-ons/node group reflect the documented eksctl IPv6 prerequisites. ```bash eksctl create cluster --config-file ipv6-cluster.yaml ``` This audit did not create a cluster. Local CLI help/schema checks used eksctl 0.229.0; they are not a live validation of networking, IAM or the Region's available builds. IPv6 requires supported Linux Nitro/Fargate paths and prefix allocation; Windows is unsupported. IPv6 Pods may have an egress-only IPv4 helper interface. The policy agent documents that IPv6 policy on the primary interface does not protect that helper's IPv4 traffic. If the design requires removing that path, review `ENABLE_V4_EGRESS`, dependencies and Pod rollout rather than assuming IPv6 policy alone blocks it. ## Network Policy Support ### Native Enforcement Standard native eBPF policy support was introduced in VPC CNI 1.14. Current EKS documentation lists newer prerequisites for standard/Admin policies; the reviewed 1.23 baseline must still be matched to the cluster and platform. ```json { "enableNetworkPolicy": "true" } ``` `"enableNetworkPolicy": "true"` is the documented string-valued configuration. Supported EC2 Linux nodes can use this implementation; Fargate and Windows do not use its enforcement. Auto Mode has its own managed implementation. EKS `ClusterNetworkPolicy` Admin/Baseline controls and Auto Mode DNS `ApplicationNetworkPolicy` are extensions, not aliases of standard `NetworkPolicy`. In standard mode, new Pods initially allow traffic while policy rules are resolved. A stricter startup behavior can be selected deliberately: ```json { "enableNetworkPolicy": "true", "env": { "NETWORK_POLICY_ENFORCING_MODE": "strict" } } ``` Strict mode requires correct policy coverage for DNS and other required traffic before workloads start. It does not configure SGPP's separate `POD_SECURITY_GROUP_ENFORCING_MODE`. ### NetworkPolicy Example Prerequisites: the `app` namespace contains controller-managed frontend/backend workloads; backend Pods listen on TCP 8080. AWS currently documents `metadata.ownerReferences` as important for reliable enforcement and requires matching Service/container port numbers (and matching names for named ports). ```yaml apiVersion: networking.k8s.io/v1 kind: NetworkPolicy metadata: name: allow-frontend-to-backend namespace: app spec: podSelector: matchLabels: app: backend policyTypes: - Ingress ingress: - from: - podSelector: matchLabels: app: frontend ports: - protocol: TCP port: 8080 ``` The source Pod selector is limited to the same namespace. This rule isolates selected backend ingress and permits matching frontend traffic to 8080; it does not deny backend egress or authenticate an application user. Consider other matching policies and Admin-tier behavior. Validate both allowed and denied flows with Deployment/Job-managed Pods rather than relying on standalone diagnostic Pods. ### Verification and Diagnostics ```bash kubectl -n kube-system logs -l k8s-app=aws-node -c aws-eks-nodeagent --tail=200 kubectl get networkpolicy -A kubectl get policyendpoints.networking.k8s.aws -A ``` These are **node-agent** logs; the policy controller runs in the EKS-managed control plane. PolicyEndpoint objects are generated state, not objects to edit/delete casually. On an authorized Linux node with the installed policy CLI: ```bash sudo /opt/cni/bin/aws-eks-na-cli ebpf progs sudo /opt/cni/bin/aws-eks-na-cli ebpf maps ``` The tool is `aws-eks-na-cli`, not `ebpf-sdk list-maps`. Inspect the affected node and distinguish process logs from configured policy-event logs and CloudWatch delivery. Enabling external log delivery also requires the appropriate IAM/configuration. ## Advanced Features ### Security Groups for Pods The selected security groups must already exist with appropriate DNS, API, application and return-path rules: ```yaml apiVersion: vpcresources.k8s.aws/v1beta1 kind: SecurityGroupPolicy metadata: name: my-security-group-policy namespace: app spec: podSelector: matchLabels: app: database securityGroups: groupIds: - sg-0123456789abcdef0 - sg-0abcdef0123456789 ``` An example **EC2** configuration choosing standard SGPP behavior is: ```json { "env": { "ENABLE_POD_ENI": "true", "POD_SECURITY_GROUP_ENFORCING_MODE": "standard" } } ``` This is not a complete SGPP installation. Verify supported trunking instance types, the cluster role's VPC resource-controller permissions, CNI permissions, subnet capacity and the relevant EKS prerequisites. T-family instances are not supported for trunking merely because they are Nitro-based. Newly created/recreated selected Pods receive the intended setup. The managed resource controller attaches an **additional trunk ENI** and associates branch ENIs with it. The trunk is not the node's primary `eth0` ENI. A selected Pod uses a branch interface with its security groups; prefix delegation does not increase the branch-Pod limit. Fargate security groups use their separate managed path. | Mode / feature | Consequence to verify | |---|---| | SGPP `strict` | Branch security-group behavior and no Pod source NAT; NodeLocal DNSCache and instance-target LoadBalancer/NodePort with `externalTrafficPolicy: Local` have documented restrictions | | SGPP `standard` | Supports the documented combined policy/DNS paths; with default external-SNAT behavior, out-of-VPC traffic uses the node primary address/security groups | | Custom networking plus SGPP | The Pod security groups take precedence over ENIConfig security groups | | IPv6 | Supported by the EKS service guide under its version/platform conditions, including EC2 CNI 1.16+; the older README feature-table “No” cell must not override that detailed guidance | | Windows / Auto Mode | This SGPP mechanism is unsupported; Auto Mode has separate node-class networking controls | Mode changes affect newly launched Pods; plan recreation and verify the traffic path. Do not assume the same security groups govern every packet after SNAT. ### Multiple Interfaces and Multus VPC CNI 1.20+ has native multi-NIC support for suitable instances with multiple network cards. Its `ENABLE_MULTI_NIC` and Pod NIC configuration are distinct from Multus, and applications must use the additional interfaces to gain their benefits. Multus is a meta-plugin. AWS's supported Multus arrangement uses VPC CNI as the **primary delegate**; using VPC CNI for higher-order interfaces is unsupported. Additional interfaces need their own compatible plugin, address assignment and lifecycle management. | Additional-interface requirement | Why it matters | |---|---| | Dedicated, identified interface | A hard-coded `eth1` can refer to an interface managed by IPAMD | | `node.k8s.amazonaws.com/no_manage=true` on the additional ENI | Prevents VPC CNI from managing the Multus interface | | AWS-assigned/routable addresses and correct subnet/SG/routes | An arbitrary `192.168.1.0/24` allocation is not automatically valid on an EC2 ENI | | Coordinated IPAM | A shared `host-local` range can allocate duplicates on different nodes | | Interface-specific policy tests | Extra interfaces and IPv4 helper paths are not automatically covered by every primary-interface policy | A NetworkAttachmentDefinition's `spec.config` contains the chosen CNI JSON, including its supported version, plugin, actual parent interface and IPAM configuration. The former generic `ipvlan`/`eth1`/`host-local` manifest omitted the prerequisites above and has been replaced by these implementation requirements. This guide does not claim a deployed Multus/IPAM solution. ### Windows Windows uses the VPC resource-controller IPAM path. Prepare the cluster role permissions, Windows node-role authentication/access entry (`EC2_WINDOWS` where applicable), and Linux/Fargate capacity for CoreDNS. Windows Fargate, Auto Mode, Hybrid Nodes, IPv6, custom networking, SGPP and native VPC-CNI network policy have documented restrictions. The controller's resulting ConfigMap must include the following Windows IPAM entry. This shows the required data, not an instruction to overwrite a manager-owned ConfigMap: ```yaml apiVersion: v1 kind: ConfigMap metadata: name: amazon-vpc-cni namespace: kube-system data: enable-windows-ipam: 'true' ``` For a **Helm-owned** installation, Windows prefix targets use different keys from Linux: ```yaml enableWindowsIpam: 'true' enableWindowsPrefixDelegation: 'true' warmWindowsPrefixTarget: 1 warmWindowsIPTarget: 0 minimumWindowsIPTarget: 0 ``` Review the matching build's schema, field ownership and resulting ConfigMap. Do not assume a Helm value is accepted unchanged as an EKS add-on configuration property. The Windows chart flags map to `enable-windows-ipam` and `enable-windows-prefix-delegation`; the warm-target fields are also Windows-specific here. After those prerequisites and AMI/version checks, a node-group command can use: ```bash eksctl create nodegroup --region "$EKS_REGION" --cluster "$CLUSTER_NAME" \ --name windows-example --managed --node-type m5.large --nodes 2 \ --node-ami-family WindowsServer2022FullContainer ``` Windows secondary-IP mode normally uses one ENI and its address-slot limit, not the Linux multi-ENI formula. Prefix delegation and actual kubelet limits need separate sizing. ## Troubleshooting ### IP Allocation and Scheduling Check Pod events to distinguish scheduling failure from sandbox/CNI allocation failure: ```bash kubectl -n kube-system logs -l k8s-app=aws-node -c aws-node --tail=300 kubectl get nodes -o json | jq '.items[] | {name: .metadata.name, allocatablePods: .status.allocatable.pods}' SUBNET_ID=subnet-0123456789abcdef0 aws ec2 describe-subnets --region "$EKS_REGION" --subnet-ids "$SUBNET_ID" \ --query 'Subnets[].{SubnetId:SubnetId,AvailableIPs:AvailableIpAddressCount}' ``` `allocatablePods` is kubelet scheduling capacity, not current IP utilization. Subnet available-address count does not show whether a contiguous `/28` is available. Check IPAMD logs, allocation mode, warm targets, ENI limits, API errors and the affected node before choosing a remedy. ### ENI Count ```bash INSTANCE_ID=i-0123456789abcdef0 aws ec2 describe-instances --region "$EKS_REGION" --instance-ids "$INSTANCE_ID" \ --query 'Reservations[].Instances[].{InstanceId:InstanceId,AttachedENIs:length(NetworkInterfaces)}' aws ec2 describe-instance-types --region "$EKS_REGION" --instance-types m5.large \ --query 'InstanceTypes[].NetworkInfo.{MaxENI:MaximumNetworkInterfaces,IPv4PerENI:Ipv4AddressesPerInterface}' ``` The original projection counted a nested list of interfaces rather than the interfaces themselves. The corrected query reports a count per instance. Multiple cards, unmanaged/trunk interfaces and instance-specific limits still need interpretation. ### Introspection and Metrics Select the affected node's actual `aws-node` Pod and keep this forwarding session open: ```bash kubectl -n kube-system get pods -l k8s-app=aws-node -o wide AWS_NODE_POD=aws-node-example kubectl -n kube-system port-forward "pod/$AWS_NODE_POD" 61678:61678 61679:61679 ``` From another local terminal: ```bash curl --fail http://127.0.0.1:61679/v1/enis curl --fail http://127.0.0.1:61678/metrics ``` IPAMD introspection defaults to loopback **61679**; Prometheus metrics use **61678**. `/v1/enis` is not a metrics endpoint. These commands use local curl through the Kubernetes forwarding path; they do not require a curl binary inside the CNI image. ### Classify Errors Before Changing the Cluster | Observation | Investigate before acting | |---|---| | `InsufficientFreeAddressesInSubnet` | Actual free addresses, warm allocation, selected subnets and planned capacity expansion | | `InsufficientCidrBlocks` | Contiguous prefix availability/fragmentation and subnet reservations | | ENI/SG limit error | The specific quota, instance/interface type and objects in use; avoid removing unrelated security groups | | ENI creation failure | Detailed AWS error, CNI credential role, permissions/conditions, quota and API connectivity | | Waiting for a Pod IP | IPAMD state, controller/API delays, throttling, sandbox events and address readiness | Restarting IPAMD, enlarging an instance or granting more node-role permissions is not a universal remedy. Capture evidence first and apply a reviewed change to the component that actually owns the failing operation. ## Best Practices Plan subnet capacity from expected Pods, warm pools, growth and failure/replacement overlap. A `/19` or RFC 6598 range is an example design choice, not a universal requirement. Associate new CIDRs, create the necessary subnets/routes and plan CNI/workload adoption together. Choose one warm-pool strategy. An IPv4 prefix example using free-IP and total-IP targets is: ```json { "env": { "ENABLE_PREFIX_DELEGATION": "true", "WARM_IP_TARGET": "5", "MINIMUM_IP_TARGET": "10" } } ``` Do not interpret an additional `WARM_PREFIX_TARGET` as an independent effective target when those IP targets are set. Monitor allocation failures and actual resource constraints rather than inferring safety solely from cluster size. ### Metrics and Alerts The released IPAMD code exports `awscni_total_ip_addresses`, `awscni_assigned_ip_addresses` and the counter `awscni_no_available_ip_addresses`. Total/assigned gauges describe the **IPAMD allocated pool**, not total VPC subnet space. Small warm targets can legitimately produce a high assigned/total ratio; cooldown, branch interfaces, IP family and kubelet capacity need additional context. Prometheus Operator CRDs and selectors must already be configured. For a Helm-owned CNI, the chart can create a PodMonitor with this **IPv4-cluster example**: ```yaml podMonitor: create: true labels: release: prometheus interval: 30s relabelings: - sourceLabels: - __meta_kubernetes_pod_node_name targetLabel: node - targetLabel: cluster replacement: example-cluster - targetLabel: ip_family replacement: ipv4 - targetLabel: job replacement: aws-vpc-cni ``` Replace the example cluster label, adapt the `release` selector and set IP-family labels truthfully; labels do not detect the cluster's family. The chart scrapes the Agent's named `metrics` port and the enabled policy agent's `agentmetrics` port. An EKS-managed add-on needs an independently configured scraper/PodMonitor instead of installing a second CNI Helm release. ```yaml apiVersion: monitoring.coreos.com/v1 kind: PrometheusRule metadata: name: vpc-cni-signals namespace: monitoring labels: release: prometheus spec: groups: - name: vpc-cni rules: - record: vpc_cni:allocated_ipv4_pool_utilization:ratio expr: (awscni_assigned_ip_addresses{job="aws-vpc-cni",ip_family="ipv4"} / awscni_total_ip_addresses{job="aws-vpc-cni",ip_family="ipv4"}) and (awscni_total_ip_addresses{job="aws-vpc-cni",ip_family="ipv4"} > 0) - alert: CniHighAllocatedIPv4PoolUtilization expr: vpc_cni:allocated_ipv4_pool_utilization:ratio > 0.9 for: 5m labels: severity: info annotations: summary: Most currently allocated IPAMD IPv4 addresses are assigned description: This is allocated-pool utilization, not subnet exhaustion. Check warm targets, assignment failures and available subnet space. - alert: CniIPAssignmentFailures expr: increase(awscni_no_available_ip_addresses{job="aws-vpc-cni"}[5m]) > 0 for: 1m labels: severity: warning annotations: summary: IPAMD could not assign an available IP address - alert: CniMetricsScrapeFailed expr: up{job="aws-vpc-cni"} == 0 for: 5m labels: severity: warning annotations: summary: A known CNI metrics endpoint cannot be scraped ``` The pool ratio is limited to IPv4 and a positive observed denominator. It is an informational tuning signal, not proof of subnet exhaustion. The assignment-failure counter signals an actual failed allocation. A failed scrape is different from a disappeared target; compare expected node/component inventory separately. No data is not healthy zero. Tune thresholds, labels and notification routing in the actual monitoring environment. ## References - [VPC CNI 1.23.0 documentation](https://raw.githubusercontent.com/aws/amazon-vpc-cni-k8s/v1.23.0/README.md) - [VPC CNI Helm values](https://raw.githubusercontent.com/aws/amazon-vpc-cni-k8s/v1.23.0/charts/aws-vpc-cni/values.yaml) - [Chart version metadata](https://raw.githubusercontent.com/aws/amazon-vpc-cni-k8s/v1.23.0/charts/aws-vpc-cni/Chart.yaml) - [Released CNI manifest](https://raw.githubusercontent.com/aws/amazon-vpc-cni-k8s/v1.23.0/config/master/aws-k8s-cni.yaml) - [IPAMD implementation](https://raw.githubusercontent.com/aws/amazon-vpc-cni-k8s/v1.23.0/pkg/ipamd/ipamd.go) - [IPAMD introspection server](https://raw.githubusercontent.com/aws/amazon-vpc-cni-k8s/v1.23.0/pkg/ipamd/introspect.go) - [IPAM datastore](https://raw.githubusercontent.com/aws/amazon-vpc-cni-k8s/v1.23.0/pkg/ipamd/datastore/data_store.go) - [IPAMD metric definitions](https://raw.githubusercontent.com/aws/amazon-vpc-cni-k8s/v1.23.0/utils/prometheusmetrics/prometheusmetrics.go) - [Network policy agent 1.4.1](https://github.com/aws/aws-network-policy-agent/blob/v1.4.1/README.md) - [AWS Helm chart index](https://aws.github.io/eks-charts/index.yaml) - [EKS security groups for Pods](https://docs.aws.amazon.com/eks/latest/userguide/security-groups-for-pods.html) - [SGPP operating considerations](https://docs.aws.amazon.com/eks/latest/best-practices/sgpp.html) - [EKS Multus support boundaries](https://docs.aws.amazon.com/eks/latest/userguide/pod-multus.html) - [EKS Windows networking](https://docs.aws.amazon.com/eks/latest/userguide/windows-support.html) - [EKS IPv6 support](https://docs.aws.amazon.com/eks/latest/userguide/cni-ipv6.html) - [eksctl IPv6 configuration](https://docs.aws.amazon.com/eks/latest/eksctl/vpc-ip-family.html) - [CNI IAM configuration](https://docs.aws.amazon.com/eks/latest/userguide/cni-iam-role.html) - [EKS VPC CNI management](https://docs.aws.amazon.com/eks/latest/userguide/managing-vpc-cni.html) - [EKS network policy conditions](https://docs.aws.amazon.com/eks/latest/userguide/cni-network-policy.html) - [Enable EKS network policy](https://docs.aws.amazon.com/eks/latest/userguide/cni-network-policy-configure.html) - [Prefix allocation and Pod-capacity limits](https://docs.aws.amazon.com/eks/latest/userguide/cni-increase-ip-addresses-procedure.html) ## Quiz Check your understanding with the [VPC CNI Quiz](https://www.atomai.click/kubernetes-docs/en/quizzes/networking/01-vpc-cni-quiz). ---------------------------------------- Source: https://www.atomai.click/kubernetes-docs/en/networking/cilium/ ---------------------------------------- # Cilium Deep Dive: The Future of Cloud Native Networking ## Overview and Reviewed Baseline This section covers Cilium networking, policy and observability. Examples are reviewed against **Cilium/Helm chart 1.20.1**, Cilium CLI **0.20.0** and Hubble CLI **1.19.4**. The Cilium 1.20 Kubernetes compatibility page lists **1.33–1.36** as tested; an upstream 1.37 release does not extend that matrix automatically. Supported hosts are AMD64/AArch64 Linux with kernel **5.10+**, or the documented distribution equivalent such as RHEL 8.10's backported 4.18 kernel. Individual features have additional requirements. > **Last Updated**: September 12, 2026 ### Historical Release Notes The dates below are GitHub publication dates in UTC and describe those releases, not current installation pins. Feature backports differ between release lines. | Date | Release | Verified highlights | | --- | --- | --- | | July 14, 2026 | [1.20.0-rc.0](https://github.com/cilium/cilium/releases/tag/v1.20.0-rc.0) | First 1.20 release candidate | | July 16, 2026 | [1.19.6](https://github.com/cilium/cilium/releases/tag/v1.19.6), [1.18.12](https://github.com/cilium/cilium/releases/tag/v1.18.12), [1.17.18](https://github.com/cilium/cilium/releases/tag/v1.17.18) | Gateway access-log configuration is listed for 1.19.6/1.18.12; the restart-policy and ClusterMesh affinity fixes cited here are listed in 1.19.6, not all three releases | | July 21, 2026 | [1.20.0-rc.1](https://github.com/cilium/cilium/releases/tag/v1.20.0-rc.1) | Second 1.20 release candidate | | July 29, 2026 | [1.20.0](https://github.com/cilium/cilium/releases/tag/v1.20.0) | GA release; selected changes below | | August 3, 2026 | [1.21.0-pre.0](https://github.com/cilium/cilium/releases/tag/v1.21.0-pre.0) | Next-cycle prerelease, not this guide's deployment baseline | | August 18, 2026 | [1.20.1](https://github.com/cilium/cilium/releases/tag/v1.20.1) | ClusterMesh documentation and bug fixes, including restart/CIDR-policy handling | | August 18, 2026 | [1.19.7](https://github.com/cilium/cilium/releases/tag/v1.19.7) | Includes ENI interface timing, Service/LB and other fixes | | August 18, 2026 | [1.18.13](https://github.com/cilium/cilium/releases/tag/v1.18.13) | VRRP/IGMP host-firewall support and related fixes | The 1.20.0 announcement reports **2,660+ new commits**, supported by a **community of 1,100+ contributors**. The latter is community size, not a count of authors in this release. Highlights include: - Gateway API **1.6.1**, TCPRoute/UDPRoute, BackendTLSPolicy, ListenerSets, ExternalAuth and CORS support, subject to each feature's configuration and API maturity. - Datapath plugins and opt-in `bpf.datapathMode=auto`; the announced default remains veth. Dual-stack clusters can configure an IPv6 egress gateway address. - **Beta** IPv6 ENI IPAM and migration from cluster-pool to multi-pool without rebuilding the cluster. In-place migration does not guarantee no interruption. - Traffic distribution hints, weighted Maglev backends and stable MCS integration; Kubernetes ClusterNetworkPolicy support and **beta** ztunnel-based workload identity. - A reported `cilium-cni` binary reduction from roughly **77 MB to 16 MB**. ADS/Delta xDS improvements are in the 1.20 announcement; do not attribute them to the 1.18.13 patch notes. These are upstream release claims, not measurements repeated in this audit. Review the [1.20 upgrade notes](https://docs.cilium.io/en/v1.20/operations/upgrade/#upgrade-notes) for removed/replaced legacy Mutual Authentication, Envoy Go extensions, Kafka-aware policies, the old CiliumNodeConfig API, libnetwork integration and custom CNI configuration changes. ### NetworkPolicy Security Advisory [GHSA-fm8w-2m5w-9j7r / CVE-2026-56743](https://github.com/cilium/cilium/security/advisories/GHSA-fm8w-2m5w-9j7r) has a project advisory publication date of **July 6, 2026**. The project API and GitHub global advisory API expose different dates: the latter records September 3. Use the project disclosure date for this release chronology; the global record date is not a later fix release. It affects **1.19.0–1.19.4** under the advisory's custom-cluster-name conditions: a standard Kubernetes NetworkPolicy peer containing only `ipBlock` can unintentionally permit ingress from workloads in the selected Pod's namespace. **1.19.5** fixes this issue; use an appropriate current patched release for the deployment. The advisory says CiliumNetworkPolicy/ClusterwideNetworkPolicy and releases below 1.19.0 are not affected by this particular bug. ## Introduction Cilium provides networking, security and observability for supported Linux Kubernetes environments. Routing, IPAM, encryption and Service handling are separate choices; selecting eBPF alone does not establish every feature or a performance guarantee. The old Docker libnetwork integration was removed in 1.20, so Docker/Mesos should not be listed here as interchangeable current installation targets. ### eBPF and Key Capabilities The kernel verifies eBPF programs before loading them and can JIT-compile them for execution at supported hooks. This enables packet processing and observability without a custom kernel module; the verifier does not prove application or policy correctness. Actual throughput, latency and memory depend on the programs, platform and workload. Cilium offers L3/L4 policy, L7 policy through Envoy/DNS proxy integration, optional WireGuard/IPsec, Service load balancing, Hubble flow visibility, ClusterMesh and BGP advertisement. XDP acceleration is optional and device/configuration dependent. L7 features may use per-node Envoy; workload identity through ztunnel has separate beta configuration. Installing the agent alone does not enable all mesh, encryption or multi-cluster behavior. ### Comparison with Other Networking Projects | Project | Connectivity / IPAM | Policy and related capabilities | | --- | --- | --- | | Cilium | Native or overlay routing; IPAM modes including cloud ENI | eBPF dataplane, Cilium/Kubernetes policies, L7 integration, Hubble and optional encryption | | Calico | Native/IPIP/VXLAN profiles with Calico or external IPAM | Linux Iptables/Nftables/BPF, supported Windows HNS; OSS WireGuard, staged policy and separate L7 integration | | Flannel | Pod connectivity through selected backends such as VXLAN/host-gw/WireGuard | The routing daemon does not enforce NetworkPolicy; its chart can deploy the SIGs network-policy controller with `netpol.enabled`, or it can pair with another policy implementation | | AWS VPC CNI | VPC ENI address allocation/networking | Native network policy on supported EC2 Linux nodes and separate SG-for-Pods functionality; EKS Auto Mode is a different managed implementation | A routing mode is not the same category as a packet-processing implementation. Calico is not limited to iptables/IPVS, Flannel can use an encrypted backend, and AWS policy is not synonymous with security groups. Service meshes are optional layers, and cross-cluster VPC connectivity is not restricted to Transit Gateway. Use a measured workload and an explicit support matrix instead of universal performance rankings. ## Architecture The **Kubernetes API server** stores Kubernetes/Cilium resources. Cilium agents watch the relevant state and program each node's dataplane; the Cilium Operator handles cluster-level responsibilities such as the selected IPAM and identity/controller work. There is no separate mandatory cluster-wide “Cilium API Server” deployment in this basic architecture. Agents have local APIs, and the optional ClusterMesh API server serves a different purpose. | Component | Role | | --- | --- | | Cilium Agent | Node-local endpoint, policy, routing/Service state and eBPF management | | Cilium Operator | Cluster-level reconciliation and mode-dependent allocation/controller work | | Envoy | Userspace proxy for enabled L7 policy, ingress/Gateway and related features | | Hubble server | Node-local flow API integrated with the agent | | Hubble Relay / UI | Aggregate flow streams / display service maps and flows | | Prometheus metrics endpoints | Separate statistics collection; Relay/UI is not the metrics scraping pipeline | | cilium / cilium-dbg / hubble | Cluster management CLI / agent diagnostics / flow client respectively | ### Networking and Packet Paths Native routing needs a reachable underlay; tunneling uses VXLAN or Geneve. AWS ENI and Azure IPAM are allocation/integration choices with their own platform requirements. Cilium's BGP Control Plane advertises reachability to routers and **does not program the datapath or provide internal cluster routing**. There is no universal XDP→TC→Pod sequence. Socket load balancing can act before packets exist, TC/netkit hooks depend on the datapath, optional XDP accelerates selected traffic, and L7 traffic may pass through Envoy. Return traffic also depends on NAT, conntrack and DSR choices. See the networking and eBPF chapters for the chosen profile. ## Integration with Amazon EKS Choose the actual networking and compute profile before installing anything. The example addon name/version `cilium` / `v1.17.0-eksbuild.1` was not a verified AWS distribution and is not an installation command here. Inspect the Region's actual add-on catalog, publisher, license and supported compute types if considering a packaged vendor add-on. | EKS profile | What to verify | | --- | --- | | Ordinary EC2 nodes, Cilium ENI replaces VPC CNI | Upstream/partner-managed CNI; AWS's supported EC2 CNI is VPC CNI. Plan CNI ownership, IAM, addressing, routes, bootstrap and node migration | | Ordinary EC2 nodes, AWS VPC CNI chaining | VPC CNI owns interfaces/IPAM; Cilium attaches its dataplane afterwards. Existing Pods need recreation, and L7/IPsec have documented limitations | | Hybrid Nodes | Follow AWS's specialized CNI guide and AWS-maintained Cilium build matrix; upstream 1.20.1 is not automatically the supported AWS build | | Auto Mode | Alternate CNI/policy plugins are unsupported; use the managed NodeClass/networking features | | Fargate | Alternate CNI/DaemonSet installation is unsupported | | Windows | The Cilium agent requirements are Linux; do not apply this recipe to Windows workers | AWS's general alternate-CNI page and specialized Hybrid guide differ in their Calico support wording; an example moving repositories does not establish support termination. For Hybrid Nodes, confirm the exact distribution, capability set and support owner. For Auto Mode, node-local CoreDNS/system networking is also different from the ordinary EC2 setup in this guide; mixed non-Auto nodes still need the traditional DNS Deployment. ### Prepared EC2 Cluster with Cilium ENI These are **Cilium Helm values for a prepared IPv4 EC2 cluster**, not a complete cluster-creation or in-place migration recipe. Before using them: 1. Choose a supported EKS/Kubernetes version and Linux AMI, and establish one CNI owner. Do not delete `aws-node` from an existing workload cluster as a shortcut. 2. Prepare node taints/scheduling so workloads wait until Cilium manages the node. Upstream EKS guidance uses `node.cilium.io/agent-not-ready=true:NoExecute`; assess eviction and bootstrap effects in the actual node lifecycle. 3. Prepare subnet capacity, ENI quotas/security groups, node metadata access and the operator's required EC2 permissions. The role ARN below is a placeholder for a correctly trusted **cilium-operator ServiceAccount** role, not a role created by the values file. 4. Retain working kube-proxy and DNS for this `kubeProxyReplacement: false` example. For replacement mode, follow the separate direct API/bootstrap-DNS requirements. Select max-Pods from the actual instance/IPAM capacity, not a universal 110. Save as `cilium-eni-values.yaml` and replace the role/interface choices with reviewed values: ```yaml eni: enabled: true ipam: mode: eni routingMode: native kubeProxyReplacement: false ipv4: enabled: true ipv6: enabled: false egressMasqueradeInterfaces: eth0 serviceAccounts: operator: annotations: eks.amazonaws.com/role-arn: arn:aws:iam::111122223333:role/CiliumOperatorENI ``` ```bash helm repo add cilium https://helm.cilium.io/ helm repo update cilium helm template cilium cilium/cilium --version 1.20.1 \ --namespace kube-system -f cilium-eni-values.yaml > cilium-eni-rendered.yaml # After preparing the cluster and reviewing the rendered configuration: helm install cilium cilium/cilium --version 1.20.1 \ --namespace kube-system -f cilium-eni-values.yaml ``` Manage these settings through the installation owner. A replacement `cilium-config` containing only a few keys can remove other required settings; the old `tunnel=disabled` value is replaced by `routingMode: native`. ENI allocation/permissions and SNAT behavior still require runtime validation; a successful render is not proof of usable EC2 networking. **IPv6 qualification:** the 1.20.1 ENI IPAM reference describes IPv6 as beta, while its EKS prerequisites page still states IPv4-only ENI integration. This guide keeps an IPv4 example and records that documentation inconsistency rather than treating either statement as proof of production EKS IPv6 compatibility. Review the current ENI/dual-stack subnet requirements before a separate IPv6 design. ### VPC CNI Chaining Alternative The upstream chaining guide requires VPC CNI 1.11.2+ and documents this profile: ```yaml cni: chainingMode: aws-cni exclusive: false enableIPv4Masquerade: false routingMode: native kubeProxyReplacement: false ``` Use it as a **different configuration**, not an overlay on the ENI-replacement values. VPC CNI remains the allocator. Upgrade the actual managed add-on through its owner rather than applying a historical upstream DaemonSet. Avoid competing policy engines on the same endpoints. Existing Pods are not retroactively attached to Cilium when the CNI chain changes; recreate them under a planned rollout and verify endpoint management. Chaining has documented L7 policy/IPsec limitations, so do not assume every example later in this page works in that profile. ### ClusterMesh ClusterMesh needs unique cluster identities, compatible versions, reachable/nonoverlapping Pod networks, authenticated API connectivity and an appropriate exposure model. A LoadBalancer Service can create cloud resources and needs a deliberate network/security design. Creating two public endpoints is not sufficient to connect clusters safely. Follow the maintained [ClusterMesh guide](https://www.atomai.click/kubernetes-docs/llms/en/service-mesh/cilium-service-mesh/01-architecture.md) and the advanced chapter for the chosen topology. ## Installation and Configuration ### Client Tools Use the appropriate official Cilium CLI 0.20.0 and Hubble CLI 1.19.4 assets for the workstation OS/architecture, and verify the supplied checksums before extraction. Linux ARM64 and AMD64 differ; macOS uses the corresponding Darwin assets. CLI versions are separate from the Cilium agent/chart version. See the [verified CLI installation guidance](https://www.atomai.click/kubernetes-docs/llms/en/service-mesh/cilium-service-mesh/README.md). ```bash cilium version --client hubble version ``` ### Non-Cloud Cluster-Pool Example The following **alternative** is for a prepared ordinary Linux cluster with kube-proxy and DNS already working. Ensure the example `10.244.0.0/16` Pod range is compatible with the cluster and does not overlap Service, node, VPC or connected-network ranges. Do not use this pool configuration for ENI mode. ```yaml routingMode: tunnel tunnelProtocol: vxlan kubeProxyReplacement: false ipv4: enabled: true ipv6: enabled: false ipam: mode: cluster-pool operator: clusterPoolIPv4PodCIDRList: - 10.244.0.0/16 clusterPoolIPv4MaskSize: 24 hubble: enabled: true relay: enabled: true ui: enabled: true metrics: enabled: - dns - drop - tcp - flow - icmp - httpV2 ``` Save as `cilium-values.yaml`, render the pinned chart, then install only on the prepared cluster: ```bash helm template cilium cilium/cilium --version 1.20.1 \ --namespace kube-system -f cilium-values.yaml > cilium-rendered.yaml helm install cilium cilium/cilium --version 1.20.1 \ --namespace kube-system -f cilium-values.yaml cilium status --wait ``` For an existing release, use its upgrade/GitOps process, preserve owned values and follow the version-specific upgrade procedure. Repeated `cilium install` examples are not a general way to change individual settings. | Choice | Current configuration and prerequisite | | --- | --- | | VXLAN/Geneve | `routingMode: tunnel` plus `tunnelProtocol`; permit the chosen encapsulation and set MTU for the path | | Native routing | `routingMode: native`; underlay must route the Pod addresses. `autoDirectNodeRoutes` needs suitable direct connectivity, not arbitrary multi-subnet routing | | kube-proxy replacement | `kubeProxyReplacement: true` or `false`, not legacy `strict`; replacement requires reachable `k8sServiceHost`/`k8sServicePort` and the documented bootstrap plan | | WireGuard | Enable supported encryption mode after checking kernel/platform and peer paths; it does not encrypt every possible traffic path automatically | | IPsec | Requires the documented key Secret, key distribution/rotation and compatible mode; the Helm enable flag alone is incomplete | | XDP/DSR/BBR | Separate device/kernel/topology-dependent choices, not a universal install preset | ## Network Policies Kubernetes `networking.k8s.io/v1` NetworkPolicy and Cilium `cilium.io/v2` policies are distinct APIs. Multiple allow policies can combine. These examples use **separate prepared test namespaces** so the L4 allow does not silently bypass the L7 restriction. Inspect all policies selecting the actual endpoints before drawing conclusions. ### L4 Example In `cilium-l4-demo`, this selects backend Pods and permits ingress from same-namespace frontend Pods on TCP 8080: ```yaml apiVersion: networking.k8s.io/v1 kind: NetworkPolicy metadata: name: allow-frontend-to-backend namespace: cilium-l4-demo spec: podSelector: matchLabels: app: backend policyTypes: - Ingress ingress: - from: - podSelector: matchLabels: app: frontend ports: - port: 8080 protocol: TCP ``` ### HTTP Example In a separate `cilium-l7-demo`, this selects backend Pods and restricts plaintext HTTP on TCP 8080 to the stated method/path from frontend Pods in that namespace. L7 proxy support must be available in the chosen CNI mode; encrypted HTTP is not automatically inspected without a supported termination configuration. ```yaml apiVersion: cilium.io/v2 kind: CiliumNetworkPolicy metadata: name: allow-product-read namespace: cilium-l7-demo spec: endpointSelector: matchLabels: k8s:app: backend ingress: - fromEndpoints: - matchLabels: k8s:app: frontend k8s:io.kubernetes.pod.namespace: cilium-l7-demo toPorts: - ports: - port: '8080' protocol: TCP rules: http: - method: GET path: ^/api/v1/products$ ``` Do not add a matching unrestricted L4 allow for the same peers/port: Cilium documents that such an allow removes the effect of the narrower L7 restrictions. L7 denial can return an HTTP 403 rather than a packet drop. Test allowed GET requests and denied methods/paths with real endpoint identities. ### DNS/FQDN Example In `cilium-dns-demo`, this permits DNS queries to ordinary CoreDNS Pods and TCP 443 to addresses learned for `api.example.com`. The domain is an example; replace it with an approved destination. The broad `*.amazonaws.com` wildcard is not an account/resource boundary and is omitted. ```yaml apiVersion: cilium.io/v2 kind: CiliumNetworkPolicy metadata: name: allow-api-domain namespace: cilium-dns-demo spec: endpointSelector: matchLabels: k8s:app: web egress: - toEndpoints: - matchLabels: k8s:k8s-app: kube-dns k8s:io.kubernetes.pod.namespace: kube-system toPorts: - ports: - port: '53' protocol: ANY rules: dns: - matchPattern: '*' - toFQDNs: - matchName: api.example.com toPorts: - ports: - port: '443' protocol: TCP ``` The DNS wildcard permits queries to the selected resolver, not connections to every returned address. It can still carry arbitrary DNS names; restrict query names where required, accounting for DNS search suffixes. NodeLocal DNS or a different resolver needs the correct destination selection. FQDN policy is DNS-derived IP authorization, not TLS hostname verification or HTTP URL authorization; shared IPs and application TLS/authentication still matter. ## Observability with Hubble The cluster-pool values above enable Relay/UI and component metrics. For an existing installation, apply the intended Hubble values through its configuration owner; `cilium hubble enable --ui` is a supported convenience command, but `cilium hubble enable --metrics=...` is not a supported 0.20.0 CLI flag. Configure `hubble.metrics.enabled` in Helm values instead. Do not enable legacy `http` and `httpV2` handlers together. Keep the Relay port-forward running in one terminal: ```bash cilium hubble port-forward --port-forward 4245 ``` In another terminal with Hubble CLI installed: ```bash hubble observe --server 127.0.0.1:4245 --namespace cilium-l7-demo hubble observe --server 127.0.0.1:4245 --protocol http hubble observe --server 127.0.0.1:4245 --from-label k8s:app=frontend --to-label k8s:app=backend hubble observe --server 127.0.0.1:4245 --verdict DROPPED hubble observe --server 127.0.0.1:4245 --http-status 403 ``` This local example assumes the default Relay server configuration; a TLS-enabled Relay needs the corresponding client trust/authentication. HTTP events require the traffic to traverse the configured L7 proxy. `DROPPED` is a datapath verdict, not every failed application request. Use `cilium hubble ui` for the UI port-forward, and configure Prometheus target discovery separately for metrics. Hubble flow streaming is not distributed application tracing by itself. ## Testing and Operations Connectivity/performance commands create test workloads and may change policy or generate substantial traffic. Use a reviewed test namespace/environment and permissions; this audit did not execute them against a cluster. ```bash cilium connectivity test --help cilium connectivity perf --help ``` The performance subcommand is `cilium connectivity perf`; `connectivity test --test=performance` merely supplies a test-name filter and is not the performance runner. Record software versions, topology, traffic and raw results before comparing throughput/latency. For inspection, distinguish the management CLI from **agent-side `cilium-dbg`**: ```bash cilium status --verbose kubectl get cnp,ccnp -A kubectl get pods -n kube-system -l k8s-app=cilium -o wide # Choose the agent Pod on the affected node. CILIUM_POD=replace-with-actual-cilium-pod kubectl exec -n kube-system "$CILIUM_POD" -c cilium-agent -- cilium-dbg endpoint list kubectl exec -n kube-system "$CILIUM_POD" -c cilium-agent -- cilium-dbg map list kubectl exec -n kube-system "$CILIUM_POD" -c cilium-agent -- cilium-dbg metrics list kubectl logs -n kube-system "$CILIUM_POD" -c cilium-agent --since=15m --tail=200 --timestamps ``` `cilium endpoint list`, `cilium bpf maps list` and `cilium metrics list` are not equivalent commands in the management CLI. `cilium sysdump` can collect diagnostic material; protect the resulting infrastructure/log data. A Ready agent or successful scrape is not a substitute for application and negative policy tests. ### Operational Priorities - Measure before enabling map preallocation, XDP, DSR, BBR or a fixed device pattern. These consume resources or change packet paths and require feature-specific checks. - Introduce default-deny in a selected scope with DNS, API, identity and application dependencies explicitly allowed. Check new Pods and upgrade transitions as well as established connections. - Keep encryption, certificate/key rotation, policy enforcement and observability as separate acceptance checks. Preserve a usable management/recovery path. - Review the current platform matrix and supported upgrade path. Historical release announcements do not establish current deployment compatibility. ## Deep Dive Table of Contents **[Introduction to Cilium and Basic Concepts](https://www.atomai.click/kubernetes-docs/llms/en/networking/cilium/01-introduction.md)** - Cilium Overview and History - Container Networking Basics - Understanding CNI (Container Network Interface) - Cilium's Differentiating Features **[eBPF Technology Deep Dive](https://www.atomai.click/kubernetes-docs/llms/en/networking/cilium/02-ebpf.md)** - Introduction to eBPF Technology and History - How eBPF Works Inside the Kernel - eBPF Program Types and Maps - Utilizing eBPF in Cilium **[Networking Models and VXLAN](https://www.atomai.click/kubernetes-docs/llms/en/networking/cilium/03-networking.md)** - Comparison of Container Networking Models - VXLAN Technology Deep Dive - Cilium's Overlay Networking - Performance Optimization Techniques - Routing Mechanisms (Encapsulation vs Native-Routing) - Cloud Provider Networking (AWS ENI, Google Cloud) **[IPAM and Network Policies](https://www.atomai.click/kubernetes-docs/llms/en/networking/cilium/04-ipam-policy.md)** - IP Address Management (IPAM) Strategies - Kubernetes and Cilium IPAM Integration - Network Policy Design and Implementation - Multi-Cluster Scenarios - IPAM Mode Deep Dive (Cluster Scope, Kubernetes Host Scope, Multi-Pool) - Cloud Provider IPAM (Azure IPAM, AWS ENI, GKE) - CRD-based IPAM **[L2-L7 Networking and Load Balancing](https://www.atomai.click/kubernetes-docs/llms/en/networking/cilium/05-l2-l7-networking.md)** - Understanding OSI Model Layers (L2, L3, L4, L7) - Cilium's Layer-specific Features - Service Mesh Integration - Load Balancing Architecture - Masquerading Configuration and Implementation Modes - IPv4 Fragment Handling **[Security and Visibility](https://www.atomai.click/kubernetes-docs/llms/en/networking/cilium/06-security-visibility.md)** - Cilium's Security Features - Network Visibility and Monitoring - Hubble Architecture and Usage - Real-time Threat Detection **[Advanced Topics and Real-World Cases](https://www.atomai.click/kubernetes-docs/llms/en/networking/cilium/07-advanced-topics.md)** - Performance Tuning and Troubleshooting - Large-Scale Deployment Strategies - Real-World Use Case Studies - Future Roadmap and Development Direction ## Additional Resources - [Networking Concepts Deep Dive](https://www.atomai.click/kubernetes-docs/llms/en/networking/cilium/networking-concepts.md) - [Glossary and Abbreviations](https://www.atomai.click/kubernetes-docs/llms/en/networking/cilium/glossary.md) ## References - [Cilium 1.20 Kubernetes compatibility](https://github.com/cilium/cilium/blob/v1.20.1/Documentation/network/kubernetes/compatibility.rst) - [Cilium system requirements](https://github.com/cilium/cilium/blob/v1.20.1/Documentation/operations/system_requirements.rst) - [EKS prerequisites](https://github.com/cilium/cilium/blob/v1.20.1/Documentation/installation/requirements-eks.rst) - [Cilium ENI IPAM](https://github.com/cilium/cilium/blob/v1.20.1/Documentation/network/concepts/ipam/eni.rst) - [AWS alternate CNI support](https://docs.aws.amazon.com/eks/latest/userguide/alternate-cni-plugins.html) - [AWS Hybrid Nodes CNI](https://docs.aws.amazon.com/eks/latest/userguide/hybrid-nodes-cni.html) - [Cilium L7 policy semantics](https://github.com/cilium/cilium/blob/v1.20.1/Documentation/security/policy/layer7.rst) - [Hubble project](https://github.com/cilium/hubble) - [Flannel networking and policy integration](https://github.com/flannel-io/flannel) - [Calico comparison terminology](https://www.atomai.click/kubernetes-docs/llms/en/networking/calico/glossary.md) ## Quiz To test what you've learned in this section, try the [Cilium Deep Dive Quiz](https://www.atomai.click/kubernetes-docs/en/quizzes/networking/cilium/01-introduction-quiz). ---------------------------------------- Source: https://www.atomai.click/kubernetes-docs/en/networking/cilium/01-introduction ---------------------------------------- # Part 1: Introduction > **Reviewed baseline**: Cilium 1.20.1 / Cilium CLI 0.20.0 / Hubble CLI 1.19.4. **Last Updated**: September 12, 2026 ## Lab Environment Setup Use an isolated, prepared Kubernetes environment. Cilium 1.20 lists Kubernetes **1.33–1.36** as tested. Nodes require AMD64/AArch64 Linux with kernel **5.10+**, or the documented equivalent such as RHEL 8.10's backported 4.18 kernel. A kind/minikube VM or container uses its node/VM kernel; the workstation's OS name alone does not establish compatibility. Feature-specific requirements still apply. Use kubectl within one minor version of the API server. Install the correct OS/architecture Cilium CLI and Hubble CLI assets with checksum verification as described in [the main guide](https://www.atomai.click/kubernetes-docs/llms/en/networking/cilium/README.md). Helm can render/review configuration; the audit used Helm 3.21.3. Do not repeat unchecked `latest` AMD64 downloads or reinstall the release in every chapter. ### Install Once with the Selected Profile EKS ENI, VPC CNI chaining and ordinary cluster-pool configurations have different prerequisites; see [the main guide](https://www.atomai.click/kubernetes-docs/llms/en/networking/cilium/README.md). The following lab values are the **ordinary IPv4 cluster-pool alternative**, with kube-proxy and DNS already working and one prepared CNI owner. They are not an EKS migration recipe. Verify that the Pod CIDR does not overlap Service, node or connected-network ranges. Save as `cilium-lab-values.yaml`: ```yaml routingMode: tunnel tunnelProtocol: vxlan kubeProxyReplacement: false ipv4: enabled: true ipv6: enabled: false ipam: mode: cluster-pool operator: clusterPoolIPv4PodCIDRList: - 10.244.0.0/16 clusterPoolIPv4MaskSize: 24 hubble: enabled: true relay: enabled: true ui: enabled: true metrics: enabled: - dns - drop - tcp - flow - icmp - httpV2 ``` ```bash CILIUM_LAB_CONTEXT=replace-with-nonproduction-context kubectl --context "$CILIUM_LAB_CONTEXT" get nodes -o wide cilium version --client # Fresh installation only, after preparing the cluster/CNI ownership and values. cilium install --context "$CILIUM_LAB_CONTEXT" --version 1.20.1 \ --values cilium-lab-values.yaml cilium status --context "$CILIUM_LAB_CONTEXT" --wait ``` If Cilium is already installed, inspect its version/configuration and follow its owner’s upgrade process. Installation/status commands and the later connectivity tests are not evidence of production compatibility; no cluster was provisioned for this audit. ## What is Cilium? Cilium provides networking, security and observability through a Linux eBPF dataplane and Kubernetes integration. It offers endpoint policy, mode-dependent IPAM/routing, Service handling and Hubble flow visibility. The Docker libnetwork integration was removed in 1.20; Kubernetes, Docker and Mesos should not be presented as interchangeable current installation targets. ### Core Components and Capabilities | Component / feature | Role and qualification | | --- | --- | | Cilium Agent | Per-node endpoint and dataplane management; it does not own every host/networking function | | Cilium Operator | Cluster-level allocation/identity/controller work; multiple replicas are supported, with leader election where applicable. The reviewed chart defaults to two replicas | | eBPF | Programs/maps at kernel hooks, subject to verification and feature requirements; performance must be measured | | L3/L4 and L7 policy | L7 needs the supported Envoy/DNS proxy path; Kafka-aware L7 policy was removed in 1.20, while L4 rules can still govern Kafka connections | | kube-proxy replacement | Optional Service handling; DSR, Maglev and XDP have their own configuration/topology constraints | | Encryption | `encryption.type` selects `ipsec`, `wireguard`, or beta `ztunnel`; ztunnel workload mTLS has its own enrollment, bootstrap, traffic and policy prerequisites | | Hubble | Network/proxy flow observations and service maps, not automatic end-to-end application tracing | | ClusterMesh / BGP | ClusterMesh needs identity, trust and network reachability; BGP advertises routes but does not program internal cluster routing | The component relationship is: kubelet requests Pod sandbox operations through **CRI**; the container runtime invokes the configured **CNI plugin**; Cilium coordinates endpoint setup and the agent programs the dataplane. CNI is not a per-packet forwarding hop. Envoy handles configured L7 proxy traffic, and Hubble exposes flow events separately from Prometheus metric scraping. ### Security Identity A security identity is an allocated numeric identifier for an endpoint's **security-relevant label set**, within the relevant allocation scope. These labels are filtered/configured and can include namespace-derived labels. Same `app` labels alone do not guarantee the same identity across namespaces or clusters. The numeric ID is not a permanent globally meaningful hash or a Pod IP. Other endpoint types also use identities. ## Container Networking Basics Host networking shares the host network namespace. A bridge connects interfaces on a host; an overlay encapsulates traffic across an underlay. Native routing relies on the underlay reaching Pod addresses. These concepts can coexist and should not be confused with choosing an eBPF versus Netfilter implementation. Operational questions include address capacity, routing/MTU, Service behavior, tenant policy, observability and failure recovery. A network model alone does not determine performance or security. ## Understanding CNI CNI is the CNCF specification/library/plugin ecosystem for container network configuration. Plugins exchange configuration/results in JSON, and can delegate address allocation to an IPAM plugin. CNI's setup/removal contract is distinct from CRI, which kubelet uses to communicate with the container runtime. Since Kubernetes 1.24, kubelet no longer owns the removed `--network-plugin`/`--cni-bin-dir` configuration flags. | Project | Relevant distinction | | --- | --- | | Cilium | Linux eBPF dataplane, several routing/IPAM modes, proxy-assisted L7 policy and Hubble | | Calico | Linux Iptables/Nftables/BPF and supported Windows HNS; OSS WireGuard, staged policy and separately configured L7 integration | | Flannel | Connectivity backends such as VXLAN/host-gw/WireGuard; policy can be supplied by its optional chart controller or another implementation | | AWS VPC CNI | VPC address allocation/networking; native policy on supported EC2 Linux nodes and separate SG-for-Pods controls | | Weave Net | The original `weaveworks/weave` repository is archived; treat it as a historical option and check any proposed maintained distribution separately | See [the current comparison](https://www.atomai.click/kubernetes-docs/llms/en/networking/cilium/README.md) for support boundaries. Do not use unbounded Kubernetes compatibility or “very high/high/medium” performance rankings as deployment evidence. Service meshes are optional integrations, not a prerequisite for ordinary Pod networking. ## Lab: A Scoped L4 Policy Use the chosen context and a dedicated namespace. This example defines policy; it does **not** deploy an application server or client images. Prepare controller-managed test workloads with approved images/tools: - A backend Pod labeled `app=backend`, listening on TCP 8080. - A frontend Pod labeled `app=frontend` and another client with a different label, both with a suitable test client. - Ordinary managed Pod interfaces, default label handling, known DNS/Service configuration and no other matching allow policies that invalidate the intended isolation. Save the namespace definition as `cilium-intro-namespace.yaml`, apply it, then prepare the test workloads: ```yaml apiVersion: v1 kind: Namespace metadata: name: cilium-intro-demo ``` ```bash kubectl --context "$CILIUM_LAB_CONTEXT" apply -f cilium-intro-namespace.yaml ``` Save the policy below as `cilium-intro-policy.yaml`: ```yaml apiVersion: cilium.io/v2 kind: CiliumNetworkPolicy metadata: name: allow-frontend-backend namespace: cilium-intro-demo spec: endpointSelector: matchLabels: k8s:app: backend ingress: - fromEndpoints: - matchLabels: k8s:app: frontend k8s:io.kubernetes.pod.namespace: cilium-intro-demo toPorts: - ports: - port: '8080' protocol: TCP ``` ```bash kubectl --context "$CILIUM_LAB_CONTEXT" get pods -n cilium-intro-demo --show-labels kubectl --context "$CILIUM_LAB_CONTEXT" apply -f cilium-intro-policy.yaml kubectl --context "$CILIUM_LAB_CONTEXT" get cnp -n cilium-intro-demo ``` With those prerequisites and no additional matching allow, expected **regular Pod-to-Pod ingress** results are: | Source / destination | Expected result | | --- | --- | | Same-namespace frontend → backend TCP 8080 | Allowed | | Other client label → backend TCP 8080 | Denied | | Frontend → backend another port/protocol | Not allowed by this policy | | Same app label from another namespace | Not allowed by this policy | Verify both positive and negative connections after policy realization. Other allow/deny policies, host traffic and probes can change the effective result. This ingress rule does not restrict frontend egress or provide HTTP method/path filtering. Inspect actual traffic and endpoint state rather than assuming API acceptance equals enforcement. `cilium connectivity test` is an additional test runner that creates workloads and policies. Run it only in a reviewed test environment with the needed permissions; it is not a read-only status command. Use `cilium connectivity perf` for the separate performance runner and preserve actual version/topology/results when benchmarking. ## References and Next Steps - [Cilium requirements and setup](https://www.atomai.click/kubernetes-docs/llms/en/networking/cilium/README.md) - [CNI project](https://github.com/containernetworking/cni) - [Kubernetes network plugins and runtime ownership](https://kubernetes.io/docs/concepts/extend-kubernetes/compute-storage-net/network-plugins/) - [Kubernetes client version skew](https://kubernetes.io/releases/version-skew-policy/) - [Cilium identities and terminology](https://github.com/cilium/cilium/blob/v1.20.1/Documentation/gettingstarted/terminology.rst) - [Original Weave repository metadata](https://api.github.com/repos/weaveworks/weave) Continue to [eBPF](https://www.atomai.click/kubernetes-docs/llms/en/networking/cilium/02-ebpf.md) or test your understanding with the [Introduction Quiz](https://www.atomai.click/kubernetes-docs/en/quizzes/networking/cilium/01-introduction-quiz). ---------------------------------------- Source: https://www.atomai.click/kubernetes-docs/en/networking/cilium/02-ebpf ---------------------------------------- # eBPF Technology Deep Dive > **Review baseline**: Cilium 1.20.1, Linux 5.10+ or a documented equivalent backport (for example RHEL 8.10's 4.18 kernel), tested Kubernetes 1.33–1.36. Individual BPF features have separate requirements. > **Last reviewed**: September 12, 2026 ## Lab Environment Setup Use a disposable Linux development VM with a maintained distribution, the tracepoints used below, and permission to load tracing programs. This is separate from installing Cilium; do not load experimental programs on cluster nodes. Linux 6.12 is the source reference for the verifier discussion, not a claim that all 6.12 distributions enable every feature. Required tools are Clang with the BPF backend, target-architecture UAPI headers, libbpf 1.x development headers/libraries, libelf, zlib, a C compiler and bpftool. BCC and bpftrace are optional alternatives. Debian/Ubuntu package names commonly include `clang`, `libbpf-dev`, `libelf-dev`, `zlib1g-dev`, `build-essential` and `pkg-config`; bpftool packaging depends on the distribution/kernel. Do not assume `linux-tools-generic` is available or appropriate on every Debian system. Cilium documents AMD64/AArch64 hosts; running Cilium natively outside its container image additionally requires Clang/LLVM 18.1+, which is a separate requirement from this small tracing lab. ```bash uname -r clang --version clang --print-targets pkg-config --modversion libbpf bpftool version test -r /sys/kernel/tracing/events/syscalls/sys_enter_execve/format test -r /sys/kernel/tracing/events/sched/sched_process_exec/format # Active feature probing: run only on the prepared lab VM. sudo bpftool feature probe kernel ``` Tracefs must be mounted and accessible; some systems expose it under `/sys/kernel/debug/tracing`. Kernel configuration, capabilities, lockdown/LSM policy and container restrictions can prevent loading or attachment even as container root. `CAP_BPF` alone is not a universal tracing permission; requirements depend on kernel, program type and BPF-token delegation. **Validation boundary:** these examples passed host C syntax checks against libbpf 1.7, userspace linking and deterministic helper simulations. Clang BPF-target compilation, the running kernel verifier and live tracepoint attachment still require validation in the prepared VM. They are not production-tested or lossless tracing recipes. ## Introduction to eBPF Technology and Historical Background eBPF lets approved programs execute at supported Linux hooks to observe or influence kernel behavior. Verification restricts memory access and execution, but kernel, verifier, JIT and helper bugs remain possible. Acceptance does not prove that a host cannot crash or that the program implements the intended policy. ### From BPF to eBPF: History of Evolution McCanne and Jacobson's *The BSD Packet Filter: A New Architecture for User-level Packet Capture* has a December 19, 1992 preprint date and identifies its presentation at Winter USENIX, January 25–29, 1993. Preserve that distinction when citing the year. Classic BPF uses the 32-bit A/X registers and scratch memory for filtering, avoiding unnecessary packet copies to userspace. Its restricted instruction set does not mean it cannot execute on modern CPUs. Extended BPF added a 64-bit instruction set with eleven registers R0–R10 (R10 is the read-only frame pointer), a commonly limited 512-byte stack, maps and more program types. This is not a historical change from ten to eleven general-purpose registers. Function/tail-call combinations can impose additional stack limits. ### Technical Evolution of eBPF: Key Features by Kernel Version These selected upstream milestones were checked against versioned source. They are not a distribution support matrix; backports, build options, architectures and helpers differ. | Kernel | Selected milestone | |---|---| | [3.15](https://github.com/torvalds/linux/blob/v3.15/include/linux/filter.h) | Extended instruction set; internal classic-BPF translation | | [3.16](https://github.com/torvalds/linux/blob/v3.16/arch/x86/net/bpf_jit_comp.c) | x86 extended-BPF JIT | | [3.18](https://github.com/torvalds/linux/blob/v3.18/include/uapi/linux/bpf.h) | BPF syscall/verification infrastructure; no usable HASH/ARRAY map types yet | | [3.19](https://github.com/torvalds/linux/blob/v3.19/include/uapi/linux/bpf.h) | HASH/ARRAY maps and socket-filter program type | | [4.1](https://github.com/torvalds/linux/blob/v4.1/include/uapi/linux/bpf.h) | KPROBE and TC SCHED_CLS/SCHED_ACT | | [4.2](https://github.com/torvalds/linux/blob/v4.2/include/uapi/linux/bpf.h) | PROG_ARRAY and tail calls | | [4.8](https://github.com/torvalds/linux/blob/v4.8/include/uapi/linux/bpf.h) | XDP program type | | [4.10](https://github.com/torvalds/linux/blob/v4.10/include/uapi/linux/bpf.h) | LRU hash maps | | [4.16](https://github.com/torvalds/linux/blob/v4.16/include/uapi/linux/bpf.h) | BPF-to-BPF function calls | | [4.17](https://github.com/torvalds/linux/blob/v4.17/include/uapi/linux/bpf.h) | Raw tracepoints | | [4.18](https://github.com/torvalds/linux/blob/v4.18/include/uapi/linux/bpf.h) | BTF load API | | [5.2](https://github.com/torvalds/linux/blob/v5.2/include/uapi/linux/bpf.h) | Direct map-value access used for global data | | [5.7](https://github.com/torvalds/linux/blob/v5.7/include/uapi/linux/bpf.h) | BPF link API and BPF LSM | | [5.8](https://github.com/torvalds/linux/blob/v5.8/include/uapi/linux/bpf.h) | BPF ring buffer | | [5.10](https://github.com/torvalds/linux/blob/v5.10/include/uapi/linux/bpf.h) | Sleepable programs for supported attachment types | | [5.15](https://github.com/torvalds/linux/blob/v5.15/include/uapi/linux/bpf.h) | BPF timer helpers | | [5.19](https://github.com/torvalds/linux/blob/v5.19/include/uapi/linux/bpf.h) | Dynamic-pointer helpers | | [6.2](https://github.com/torvalds/linux/blob/v6.2/kernel/bpf/helpers.c) | Typed object-allocation kfuncs, not unrestricted malloc | Bounded loops arrived in Linux 5.3; the [upstream verifier change](https://github.com/torvalds/linux/commit/2589726d12a1b12eaaa93c7f1ea64287e383c7a5) explains loop analysis and state pruning. An old “loops are not implemented” paragraph remaining in the design FAQ is not current feature guidance. Bounded loops can still exceed verifier complexity limits. ### Growth and Application Areas Cilium's public repository was created in December 2015; “project started in 2017” is inaccurate. Repository creation is not an exact product-launch date or proof of “first major project” status. | Area | Examples and boundaries | |---|---| | Networking | Cilium/Calico datapaths, Katran load balancing and XDP filtering | | Runtime security | Falco, Tracee and Tetragon use kernel events; enforcement depends on product and hooks | | Tracing | BCC (including Python/Lua frontends), bpftrace, storage and block-I/O tracing | | Network observability | Hubble flow/proxy events; flow graphs are not distributed application-span tracing | | Service mesh | Cilium combines kernel forwarding with userspace proxies for supported L7 features | | Community | The eBPF Foundation supports the ecosystem; funding and project maturity are not compatibility criteria | `seccomp-bpf` uses the classic BPF filter interface for system-call decisions. Linux may internally translate classic filters, but this is not the general eBPF program/map/helper API. ### eBPF vs Traditional Kernel Modules: Paradigm Shift | Characteristic | eBPF | Kernel module | |---|---|---| | Safety | Verifier-constrained; implementation bugs and operational risk remain | Broader native kernel access; bugs can destabilize the host | | Deployment | Supported programs can be loaded/attached without reboot | Many modules can also load/unload without reboot when dependencies and usage permit | | Compatibility | Instruction/helper ABI and feature requirements; CO-RE can relocate supported type accesses | Kernel/module ABI, configuration and distribution support | | Performance | Often JIT-compiled; hook, program and workload determine overhead | Native execution also has workload-dependent costs | | Development | Restricted context, helpers/kfuncs and verifier limits | Kernel API and ordinary kernel development constraints | | Permissions | Appropriate privileges or delegation for loading/attachment | Privileged loading; signing/lockdown may restrict it | Both require operational testing. Modules are not limited to vendor implementations, and eBPF does not inherently make production rollout safe. ## In-depth Analysis of eBPF Architecture Inside the Kernel ### Detailed Description of eBPF Architecture Components In userspace, Clang compiles C to BPF ELF; Rust uses its own compiler/toolchain ecosystem. libbpf handles ELF sections, maps, relocations, loading and supported attachment APIs. BCC provides higher-level APIs; bpftrace provides a tracing language. CO-RE uses BTF and relocations to adapt supported type/field accesses. It does not supply missing helpers, program types or kernel configuration, nor guarantee arbitrary cross-architecture/kernel compatibility. Kernel internal structures, tracepoint formats and kfuncs are not stable ABI merely because a program uses BTF. In the kernel, the verifier checks a program for its type, context, helpers and permissions. JIT can translate accepted BPF into native instructions; an interpreter is another execution mechanism where supported. JIT output does not then pass through a mandatory second VM stage. Attachment connects the loaded program to a hook; loading alone does not subscribe to tracepoints. ### Detailed Analysis of eBPF Program Lifecycle 1. **Develop:** select hook/context and define maps/license metadata. Not all programs require GPL compatibility, but GPL-only helpers and certain types/kfuncs impose restrictions. These tracing samples use GPL metadata for their helpers. 2. **Compile:** create BPF ELF and required debug/BTF information with the target toolchain and headers. 3. **Open/load:** parse ELF, create or explicitly reuse maps, relocate and invoke the BPF load API. Verification and optional JIT occur during loading. 4. **Attach:** use the appropriate API. libbpf can infer these tracepoints from `SEC("tracepoint/...")`. Keep the link/attachment alive. 5. **Run/observe:** events invoke the program; userspace reads maps/buffers. Sampling and capacity limits can lose observations. 6. **Update/unload:** keep compatible maps/links/pins only deliberately. Destroy this lab's link and close its object to release resources. In Linux 6.12, program length is limited to up to 1,000,000 instructions for the BPF-capable loading path and 4,096 for the unprivileged path. The verifier separately has a 1,000,000-instruction **analysis complexity** limit. Smaller programs can fail verification. Unprivileged BPF is often disabled; token/capability and program-type checks still apply. ### eBPF Program Types and Characteristics | Hook / program type | Purpose and return-value boundary | |---|---| | XDP / `BPF_PROG_TYPE_XDP` | Native driver XDP runs before skb allocation; generic/offloaded modes differ. `XDP_DROP`, `PASS`, `TX`, `REDIRECT` are actions, not throughput guarantees | | TC / `SCHED_CLS`, `SCHED_ACT` | Ingress/egress packet classification/actions. Classifier `TC_ACT_*` semantics require appropriate direct-action setup | | Socket filter / `SOCKET_FILTER` | Socket packet delivery: zero drops, positive capture length may truncate. Creation/connect policies use other hooks | | kprobe/uprobe / `KPROBE` | Kernel/userspace probes; there is no separate `BPF_PROG_TYPE_UPROBE`. Inlining, blacklists and symbol availability constrain attachment | | Tracepoint / `TRACEPOINT` | Statically declared event context; inspect the target format. Not a guaranteed stable kernel ABI | | Perf event / `PERF_EVENT` | Performance sampling; return behavior depends on its perf-event integration | | cgroup / `CGROUP_SKB`, `CGROUP_SOCK`, `CGROUP_SOCK_ADDR`, etc. | Network/socket control; context and allow/deny conventions vary | | LSM / `LSM` | MAC-style programs normally preserve earlier errors and return zero/error; cgroup-LSM has different grant semantics | | Socket operations / `SOCK_OPS` | TCP callbacks; operation, reply fields and helper support matter | | fentry/fexit / `TRACING` | BTF-based function tracing where supported; target and attachment constraints remain | Select hooks by visibility/control needs. XDP lacks some later-stack context; TC handles skb-backed traffic; tracepoint/probe observation does not automatically enforce network policy. Do not copy context structs or return codes across program types. ### eBPF Maps: Core of Data Sharing and State Storage Maps live while references remain, such as FDs, loaded programs or explicit bpffs pins. Pins are not disk persistence and do not preserve map contents across reboot. Reloading does not automatically reuse the old map. | Type | Use and constraint | |---|---| | `HASH` | Bounded key/value table; insertion can fail when full. Expected constant-time lookup is not a latency guarantee | | `ARRAY` | Preallocated, zero-initialized values at valid indices; zero is not a missing hash entry | | `LRU_HASH` | Bounded cache with LRU-style eviction, not a lossless cumulative counter | | `RINGBUF` | Multiple producers/single consumer across CPUs; key/value sizes zero, power-of-two byte capacity; failed reservations do not block | | `PERF_EVENT_ARRAY` | Per-CPU perf channels; userspace must provision/consume events and track lost records | | `PROG_ARRAY` | Tail-call program references; compatible targets and call limits apply | | `PERCPU_HASH` / `PERCPU_ARRAY` | Less cross-CPU contention, not universally race-free. Userspace reads all possible-CPU slots with required padding | | `SOCKMAP` / `SOCKHASH` | Socket references for supported redirection/programs, not arbitrary socket-operation hooks | libbpf 1.x removed `struct bpf_map_def SEC("maps")`. These BTF-style definitions illustrate eight map categories using actual types. Combine needed maps with a suitable program; the declarations alone are not an event pipeline. **`map_types.bpf.c`** ```c #include #include /* Definitions only; combine the needed maps with a suitable program. */ struct { __uint(type, BPF_MAP_TYPE_HASH); __uint(max_entries, 1024); __type(key, __u32); __type(value, __u64); } hash_counts SEC(".maps"); struct { __uint(type, BPF_MAP_TYPE_ARRAY); __uint(max_entries, 1); __type(key, __u32); __type(value, __u64); } total SEC(".maps"); struct { __uint(type, BPF_MAP_TYPE_LRU_HASH); __uint(max_entries, 1024); __type(key, __u32); __type(value, __u64); } cache SEC(".maps"); struct { __uint(type, BPF_MAP_TYPE_RINGBUF); __uint(max_entries, 256 * 1024); } events SEC(".maps"); struct { __uint(type, BPF_MAP_TYPE_PERF_EVENT_ARRAY); __type(key, __u32); __type(value, __u32); /* libbpf determines max_entries from the number of possible CPUs. */ } perf_events SEC(".maps"); struct { __uint(type, BPF_MAP_TYPE_PROG_ARRAY); __uint(max_entries, 10); __type(key, __u32); __type(value, __u32); } jump_table SEC(".maps"); struct { __uint(type, BPF_MAP_TYPE_PERCPU_ARRAY); __uint(max_entries, 1); __type(key, __u32); __type(value, __u64); } cpu_counts SEC(".maps"); struct { __uint(type, BPF_MAP_TYPE_SOCKMAP); __uint(max_entries, 1024); __type(key, __u32); __type(value, __u32); } sockets SEC(".maps"); ``` Shared counters need atomic increments. New hash keys need `BPF_NOEXIST` insertion followed by lookup/increment of the winning entry; `BPF_ANY` initialization can overwrite another CPU's count. Array entries already exist at valid indices. ## Utilizing eBPF in Cilium: Innovation in Container Networking ### Cilium Architecture and the Role of eBPF ![Cilium logical roles: Kubernetes state and Operator, per-node agents, kernel programs/maps and Hubble flow observations.](https://raw.githubusercontent.com/Atom-oh/kubernetes-docs/main/en/.gitbook/assets/en-networking-cilium-02-ebpf-1.png) [🔍 View interactive diagram](https://www.atomai.click/kubernetes-docs/archmaps/en-networking-cilium-02-ebpf-1.html) The diagram shows logical responsibilities, not mandatory locations or a single compilation pipeline. CLI can run outside the cluster; agents run on eligible managed nodes. Program build/load details vary by version/feature. Hubble Relay aggregates flows; Prometheus metrics use separate endpoints. The agent reconciles endpoints, identities, policy and datapath state. The Operator handles configured cluster-wide tasks such as identity/IPAM lifecycle; it is not the packet-forwarding path. Hubble combines BPF flow information with userspace proxy events. ### Detailed Analysis of Cilium's eBPF Datapath These are cooperating functions, not a fixed order for every packet: 1. **Entry:** socket hooks can resolve Service backends before a packet exists; TC handles packet paths; optional XDP acceleration handles supported external traffic. 2. **Identity/policy:** IP/identity and endpoint policy control L3/L4 access. Supported HTTP/gRPC policy uses Envoy and DNS policy uses the DNS proxy; L7 parsing/enforcement is not entirely BPF. 3. **State/translation:** conntrack, service/backend, reverse-NAT and affinity maps have different roles. Not every packet repeats backend selection. 4. **Forwarding:** native routing or configured overlay carries traffic. DSR dispatch/return paths require the selected mode's network prerequisites. 5. **Observation:** datapath counters/events and proxy events have configuration and collection-loss limits. Backend readiness comes from control-plane state and applicable health mechanisms, not a universal BPF application probe. Maglev, affinity, DSR and acceleration are feature choices, not universal defaults. ### Detailed Description of Cilium's Major eBPF Programs | Cilium 1.20.1 source | Role | |---|---| | `bpf/bpf_lxc.c` | Endpoint packet path, policy, conntrack and forwarding | | `bpf/bpf_overlay.c` | Overlay packet path | | `bpf/bpf_host.c` | Host/device path and supported host-firewall processing | | `bpf/bpf_xdp.c` | XDP path, including configured load-balancer acceleration | | `bpf/bpf_sock.c` | Socket-address hooks including connect/sendmsg/recvmsg service translation | | `bpf/lib/lb.h` | Shared load-balancing helpers | | `bpf/lib/policy.h` | Shared policy helpers | There are no top-level `bpf_lb.c` or `bpf_network.c` files in this release. Function names and feature gates change; inspect the exact release instead of treating conceptual names as source files. ### Cilium's eBPF Map Usage These examples are not a stable map-layout API: | Name / family | Key and role | |---|---| | `cilium_lxc` | Address/family → endpoint forwarding metadata; not simply endpoint ID | | `cilium_ipcache_v2` | Prefix, address family, cluster context → identity/tunnel metadata | | `cilium_policy_v3_` | Identity, direction, protocol, destination port and prefix → policy entry | | `cilium_ct4_global`, `cilium_ct6_global`, `cilium_ct_any4_global`, etc. | Connection-tuple state; actual maps depend on protocol/family/configuration | | `cilium_lb4_services_v2` / `cilium_lb6_services_v2` | Address/port, protocol, scope and backend slot → service metadata/backend reference; backend records are separate maps | | `cilium_metrics` | Reason, direction and source-location key → packet/byte counters | `cilium-dbg map get` displays userspace-cached content, not necessarily a fresh kernel dump. Use matching `cilium-dbg bpf ...` decoders or bpftool for their supported kernel views. Do not write raw bytes into Cilium maps as a troubleshooting shortcut. ### eBPF-based Features and Their Boundaries - **Policy:** Kubernetes NetworkPolicy has L3/L4 semantics; Cilium resources add supported capabilities. Unrestricted L4 allow can bypass an overlapping L7-restricted allow; inspect combined policy. - **Encryption:** in WireGuard/IPsec modes, BPF steers traffic into those kernel facilities; cryptography is not solely BPF instructions. Configure keys, ports, MTU and traffic coverage for the chosen mode. IPsec and WireGuard key operations differ. - **Service mesh:** userspace proxies supply supported L7 processing. Kafka L7 policy is removed. Beta workload mTLS/ztunnel has separate prerequisites and is not implied by node encryption. - **Bandwidth:** EDT/bandwidth-manager and congestion control do not guarantee end-to-end QoS or throughput. - **Multi-cluster:** Cluster Mesh requires connectivity, identities, addressing and compatible configuration; it does not automatically synchronize every policy object or solve routing. ## Lab: eBPF Program Development and Debugging ### 1. Basic eBPF Program Development Save the named files in a new lab directory. This program records `execve` **attempts**, including later failures. `execveat` has a different syscall-entry tracepoint. Debug print is shared/noisy and is not a production event transport. `SEC()` supplies the intended type/hook; the GPL metadata suits the helper used here. **`hello.bpf.c`** ```c #include #include SEC("tracepoint/syscalls/sys_enter_execve") int hello_execve(void *ctx) { (void)ctx; char message[] = "execve attempt\n"; bpf_trace_printk(message, sizeof(message)); return 0; } char LICENSE[] SEC("license") = "GPL"; ``` ### 2. Advanced eBPF Program Using Maps `sched_process_exec` is emitted after a successful execution transition in the referenced kernel. Count by `comm`, a short task name of at most 16 bytes including termination, not a unique executable path/process identity. Names can collide/change. This observes the host, not automatically one Pod. The map holds at most 1,024 names. `lost_events[0]` counts name-read failures and `[1]` events without a usable counter entry, including capacity exhaustion. These do not cover every possible collection failure; 64-bit counters can wrap. The lab does not delete entries while counting. **`exec_shared.h`** ```c #ifndef EXEC_SHARED_H #define EXEC_SHARED_H #define COMM_BYTES 16 #define MAX_COMMANDS 1024 struct comm_key { char comm[COMM_BYTES]; }; #endif ``` **`exec_count.bpf.c`** ```c #include #include #include "exec_shared.h" struct { __uint(type, BPF_MAP_TYPE_HASH); __uint(max_entries, MAX_COMMANDS); __type(key, struct comm_key); __type(value, __u64); } exec_counts SEC(".maps"); struct { __uint(type, BPF_MAP_TYPE_ARRAY); __uint(max_entries, 2); __type(key, __u32); __type(value, __u64); } lost_events SEC(".maps"); static __always_inline void record_loss(__u32 reason) { __u64 *lost = bpf_map_lookup_elem(&lost_events, &reason); if (lost) __sync_fetch_and_add(lost, 1); } SEC("tracepoint/sched/sched_process_exec") int count_exec(void *ctx) { (void)ctx; struct comm_key key = {}; __u64 zero = 0; if (bpf_get_current_comm(key.comm, sizeof(key.comm)) != 0) { record_loss(0); return 0; } __u64 *count = bpf_map_lookup_elem(&exec_counts, &key); if (!count) { /* A competing CPU may insert first; never overwrite its count. */ bpf_map_update_elem(&exec_counts, &key, &zero, BPF_NOEXIST); count = bpf_map_lookup_elem(&exec_counts, &key); } if (count) __sync_fetch_and_add(count, 1); else record_loss(1); return 0; } char LICENSE[] SEC("license") = "GPL"; ``` #### User-space Application and Attachment Lifetime This loader accepts either sample object, loads exactly one program, attaches it and retains the link until Ctrl-C/SIGTERM. It reads counter map FDs from the same object rather than assuming pins exist. Iteration starts with NULL and produces a bounded, non-atomic live sample. **`run_bpf.c`** ```c #define _POSIX_C_SOURCE 200809L #include #include #include #include #include #include #include #include #include "exec_shared.h" static volatile sig_atomic_t stopping; static void stop(int signal_number) { (void)signal_number; stopping = 1; } static int dump_counts(int map_fd, int lost_fd) { struct comm_key current, next; const struct comm_key *previous = NULL; unsigned int seen = 0; while (seen < MAX_COMMANDS) { if (bpf_map_get_next_key(map_fd, previous, &next) != 0) { if (errno == ENOENT) break; perror("get next key"); return -1; } __u64 value; if (bpf_map_lookup_elem(map_fd, &next, &value) == 0) printf("%.*s: %" PRIu64 "\n", COMM_BYTES, next.comm, (uint64_t)value); else if (errno != ENOENT) { perror("lookup count"); return -1; } current = next; previous = ¤t; seen++; } for (__u32 reason = 0; reason < 2; reason++) { __u64 value; if (bpf_map_lookup_elem(lost_fd, &reason, &value) != 0) { perror("lookup loss"); return -1; } printf("lost[%u]: %" PRIu64 "\n", reason, (uint64_t)value); } if (fflush(stdout) != 0) { perror("flush output"); return -1; } return 0; } int main(int argc, char **argv) { struct bpf_object *object = NULL; struct bpf_link *link = NULL; int result = 1; if (argc != 2) { fprintf(stderr, "usage: %s OBJECT.bpf.o\n", argv[0]); return 2; } struct sigaction action = {.sa_handler = stop}; sigemptyset(&action.sa_mask); if (sigaction(SIGINT, &action, NULL) || sigaction(SIGTERM, &action, NULL)) { perror("sigaction"); return 1; } object = bpf_object__open_file(argv[1], NULL); if (!object) { perror("open BPF object"); return 1; } struct bpf_program *program = bpf_object__next_program(object, NULL); if (!program || bpf_object__next_program(object, program)) { fprintf(stderr, "expected exactly one program\n"); goto cleanup; } if (bpf_object__load(object) != 0) { fprintf(stderr, "load failed; inspect libbpf/verifier diagnostics\n"); goto cleanup; } int counts = bpf_object__find_map_fd_by_name(object, "exec_counts"); int losses = bpf_object__find_map_fd_by_name(object, "lost_events"); if (counts >= 0 && losses < 0) { fprintf(stderr, "counter object is missing lost_events\n"); goto cleanup; } link = bpf_program__attach(program); if (!link) { perror("attach tracepoint"); goto cleanup; } fprintf(stderr, "Attached; Ctrl-C detaches. Counts are live samples.\n"); result = 0; while (!stopping) { if (counts >= 0 && dump_counts(counts, losses) != 0) { result = 1; break; } sleep(2); } cleanup: bpf_link__destroy(link); bpf_object__close(object); return result; } ``` #### Compile and Run On Debian/Ubuntu multiarch installations, GCC's multiarch directory supplies UAPI `asm/` headers; adjust paths for other distributions. `-g` supplies BTF for `.maps`. Compile and start the loader in terminal A on the prepared VM: ```bash MULTIARCH=$(gcc -print-multiarch) test -n "$MULTIARCH" clang -O2 -g -target bpf -I"/usr/include/$MULTIARCH" \ -c hello.bpf.c -o hello.bpf.o clang -O2 -g -target bpf -I"/usr/include/$MULTIARCH" \ -c exec_count.bpf.c -o exec_count.bpf.o cc -O2 -Wall -Wextra run_bpf.c -o run_bpf \ $(pkg-config --cflags --libs libbpf) sudo ./run_bpf hello.bpf.o ``` In terminal B read `sudo cat /sys/kernel/tracing/trace_pipe`; run an external executable such as `/usr/bin/true` in terminal C. Stop the hello loader with Ctrl-C, then run `sudo ./run_bpf exec_count.bpf.o`. Observe changing name/count pairs while executing commands in another terminal. The observer and other host activity also generate events, so no fixed total/PID is promised. Failed `execve` attempts can appear in hello output but should not emit `sched_process_exec`. `bpftool prog load OBJECT PIN` alone does not attach this tracepoint. The example deliberately owns a link. Explicit pinning/reuse is a separate lifecycle decision: `pinmaps` and `map ... pinned ...` are not interchangeable syntax. Closing this loader releases its unpinned resources. ### 3. Exploring and Debugging Cilium eBPF Programs Use an already prepared cluster and correct kubeconfig context. Select the agent on the affected Pod's node; endpoint IDs are node-local. Replace the explicit placeholders: ```bash kubectl config current-context kubectl -n kube-system get pods -l k8s-app=cilium -o wide export CILIUM_POD=cilium-REPLACE-WITH-ACTUAL-POD export ENDPOINT_ID=REPLACE-WITH-NODE-LOCAL-ID kubectl -n kube-system exec "$CILIUM_POD" -c cilium-agent -- cilium-dbg status --verbose kubectl -n kube-system exec "$CILIUM_POD" -c cilium-agent -- cilium-dbg endpoint list kubectl -n kube-system exec "$CILIUM_POD" -c cilium-agent -- cilium-dbg endpoint get "$ENDPOINT_ID" kubectl -n kube-system exec "$CILIUM_POD" -c cilium-agent -- cilium-dbg map list kubectl -n kube-system exec "$CILIUM_POD" -c cilium-agent -- cilium-dbg service list kubectl -n kube-system exec "$CILIUM_POD" -c cilium-agent -- cilium-dbg bpf lb list --frontends kubectl -n kube-system exec "$CILIUM_POD" -c cilium-agent -- cilium-dbg bpf lb list --backends ``` Inspect desired policies with `kubectl get networkpolicy,ciliumnetworkpolicy -n YOUR_NAMESPACE` and applicable cluster-wide policies separately. Compare endpoint realized state with actual flows. Removed `policy trace` and deprecated `policy get` are not substitutes. Run one monitor at a time, stopping with Ctrl-C: ```bash kubectl -n kube-system exec "$CILIUM_POD" -c cilium-agent -- cilium-dbg monitor --related-to "$ENDPOINT_ID" --type drop ``` Use `--type policy-verdict` for emitted policy decisions or `--type l7` for available proxy events. Visibility depends on configuration. HTTP rejection may be HTTP 403 rather than a network DROPPED verdict. For enabled Hubble Relay, keep `cilium hubble port-forward` running, then use: ```bash hubble status hubble observe --namespace default --last 20 hubble observe --protocol http --last 20 hubble observe --namespace default --last 20 --output json ``` JSON piped to `jq` is not a service-dependency graph. Enabled Hubble UI supplies a service map (`cilium hubble ui`). HTTP visibility needs a supported proxy/L7 path; encrypted application content is not automatically decoded. ### 4. Performance Analysis and Optimization On a controlled node with profiling permissions/support, inspect actual program IDs. These commands target local kernel state and were not executed by this audit: ```bash sudo bpftool prog show export PROG_ID=REPLACE-WITH-ACTUAL-ID sudo bpftool prog show id "$PROG_ID" sudo bpftool prog dump xlated id "$PROG_ID" sudo bpftool prog profile id "$PROG_ID" duration 10 cycles instructions ``` Profiling needs metric names and suitable kernel/PMU support. `bpftool -p map dump ...` pretty-prints content, not lookup latency. `perf`/bpftrace can profile workloads, but verify target symbols, probe availability and arguments. A kretprobe does not reliably expose entry `arg0` without explicit correlation. Record protocol, packet size, concurrency, policy, encryption, proxy and routing when measuring the complete workload. Inspect installed Helm values and `cilium-dbg status --verbose` before changing XDP/native routing. A faster hook or synthetic result does not prove lower application latency. ### 5. Troubleshooting Tips | Symptom | Check | |---|---| | C build fails | Correct UAPI/libbpf headers, BPF compiler target, `__u32`/`__u64`, `-g` for BTF | | Verifier rejection | Loader stderr/verifier log, bounds, stack initialization, helpers, license and complexity | | Loaded but no events | Attachment/link lifetime, exact tracepoint, trigger and permissions | | Missing map data | Same map instance, insertion errors/capacity, key meaning, reference/pin lifetime | | Wrong Cilium flow | Correct node/endpoint, combined desired/realized policy, route/backend state, L7 proxy behavior | | Missing Hubble records | Relay, filters, configured visibility and lost-event reporting | `trace_pipe` contains trace output, not verifier diagnostics. For an intentional load test in the isolated VM, bpftool `-d` gives loader/verifier diagnostics; loading still does not prove attachment or behavior. Do not disable policy, expand production privileges or rewrite maps to make a test pass. ## Sources - [Original BPF paper](https://www.tcpdump.org/papers/bpf-usenix93.pdf), [Linux BPF design Q&A](https://docs.kernel.org/bpf/bpf_design_QA.html), [verifier](https://docs.kernel.org/bpf/verifier.html), [ring buffer](https://docs.kernel.org/bpf/ringbuf.html), [licensing](https://docs.kernel.org/bpf/bpf_licensing.html), [seccomp](https://docs.kernel.org/userspace-api/seccomp_filter.html) - [Linux 6.12 BPF loading](https://github.com/torvalds/linux/blob/v6.12/kernel/bpf/syscall.c), [exec event placement](https://github.com/torvalds/linux/blob/v6.12/fs/exec.c), [libbpf 1.7](https://github.com/libbpf/libbpf/tree/v1.7.0), [bpftool 7.7](https://github.com/libbpf/bpftool/releases/tag/v7.7.0) - [Cilium 1.20.1 BPF source](https://github.com/cilium/cilium/tree/v1.20.1/bpf), [maps](https://github.com/cilium/cilium/tree/v1.20.1/pkg/maps), [load-balancer maps](https://github.com/cilium/cilium/tree/v1.20.1/pkg/loadbalancer/maps), [command reference](https://github.com/cilium/cilium/tree/v1.20.1/Documentation/cmdref) - [Cilium system requirements](https://docs.cilium.io/en/v1.20/operations/system_requirements/), [Kubernetes compatibility](https://docs.cilium.io/en/v1.20/network/kubernetes/compatibility/), [kube-proxy replacement](https://docs.cilium.io/en/v1.20/network/kubernetes/kubeproxy-free/), [encryption](https://docs.cilium.io/en/v1.20/security/network/encryption/) ## Quiz [Check your understanding](https://www.atomai.click/kubernetes-docs/en/quizzes/networking/cilium/02-ebpf-quiz). ---------------------------------------- Source: https://www.atomai.click/kubernetes-docs/en/networking/cilium/03-networking ---------------------------------------- # Networking Models and VXLAN > **Review baseline**: Cilium 1.20.1, tested Kubernetes 1.33–1.36, Linux 5.10+ or documented equivalent backports such as RHEL 8.10's 4.18 kernel. > **Last reviewed**: September 12, 2026 ## Lab Environment Setup Use the [installation guide](https://www.atomai.click/kubernetes-docs/llms/en/networking/cilium/README.md) to prepare a disposable cluster and architecture-appropriate Cilium CLI. Keep kubectl within one minor version of the API server; “v1.31 or higher” is not a compatibility rule. The generic mode examples below require at least two schedulable Linux nodes, no competing Pod CNI, working kube-proxy and a non-overlapping Pod CIDR. The native-routing example additionally requires the nodes to share an L2 segment. These are not EKS ENI, GKE Dataplane V2, AKS managed-Cilium or in-place CNI migration recipes. ### Network Analysis Tools Install tcpdump/Wireshark through the analysis host's supported package source. Node packet capture must run on the relevant node/network namespace, not merely on the laptop that runs kubectl. An agent monitor reports emitted BPF events; it is not a full packet capture. ```bash kubectl config current-context kubectl -n kube-system get pods -l k8s-app=cilium -o wide export CILIUM_POD=cilium-REPLACE-WITH-AGENT-ON-TARGET-NODE kubectl -n kube-system exec "$CILIUM_POD" -c cilium-agent -- \ cilium-dbg monitor --type trace -v ``` For the explicit VXLAN profile below, capture a bounded sample on the relevant worker: ```bash sudo tcpdump -nn -i any -c 50 'udp port 8472' ``` Use the configured tunnel port if it differs. Generate cross-node Pod traffic: same-node traffic need not traverse the overlay. An empty capture can mean the wrong node, interface, port or traffic path rather than a network failure. ## Container Networking Model Comparison Host namespaces, bridges and inter-node transports describe different aspects of networking and can coexist. They are not a universal performance or security ranking. | Model | Mechanism | Important tradeoff | |---|---|---| | Host network | A Pod shares the node network namespace | Port conflicts and reduced network-namespace isolation; not automatically the best application performance | | Bridge | A virtual L2 bridge connects interfaces | Inter-node communication still needs routing/transport; Cilium does not require a Linux bridge for every endpoint | | Overlay | Encapsulated traffic crosses an IP underlay | Extra headers and processing, but the underlay need not route every Pod prefix | | Native/underlay routing | The network can route workload addresses | Requires correct forwarding/return routes and address planning; does not inherently provide or remove policy/encryption | ### Cilium Networking Modes Cilium's `routingMode` is `tunnel` or `native`. VXLAN/Geneve choose the tunnel protocol. Cloud IPAM integrations are another configuration dimension, often paired with a native datapath; they are not a third `routingMode` value. BGP is a route-advertisement mechanism, not a separate packet-forwarding mode. ## VXLAN Technology Deep Dive VXLAN carries an inner Ethernet frame in UDP over an IP network. A VTEP encapsulates/decapsulates traffic; the 24-bit VNI offers a theoretical space of 2^24 identifiers. This does not promise that a Kubernetes deployment supports 16 million tenants. The standardized VXLAN destination port is UDP 4789. **Cilium defaults to UDP 8472** for VXLAN and UDP 6081 for Geneve; both are configurable. Cilium can carry security-identity metadata in encapsulation, so do not equate generic VXLAN segment counts with Cilium tenant/policy boundaries. ### VXLAN Packet Structure ```text Outer Ethernet Outer IP (IPv4 or IPv6) Outer UDP (Cilium VXLAN default destination 8472; standard 4789) VXLAN header (8 bytes, including VNI) Inner Ethernet Inner IP packet and transport/application payload ``` IP carries UDP; an outer IP header is not itself carried inside the outer UDP header. VXLAN segmentation does not provide encryption, integrity or automatic NetworkPolicy isolation. Restrict the underlay path appropriately and configure policy/encryption separately. ### MTU Budget For ordinary VXLAN with no additional encapsulation/options, the reduction in the inner IP budget is: | Underlay IP family | Outer IP + UDP + VXLAN + inner Ethernet | Inner IP budget for a 1,500-byte underlay IP MTU | |---|---|---| | IPv4 | 20 + 8 + 8 + 14 = 50 bytes | 1,450 bytes | | IPv6 | 40 + 8 + 8 + 14 = 70 bytes | 1,430 bytes | The outer Ethernet header is outside that underlay IP MTU. Encryption, Geneve options and other paths can change the budget. The effective route MTU and a Pod veth's device MTU need not be identical. In Cilium 1.20.1, Helm **`MTU` overrides the underlying-network MTU**; Cilium then calculates route overhead. `MTU: 0` selects detection. Setting `MTU: 1450` as if it meant “the final Pod payload MTU” can subtract the tunnel overhead again. Local interface detection also does not prove the smallest MTU across the entire path. ### VXLAN vs Other Encapsulations | Technology | Carrier / identifier | Protocol or port | Boundary | |---|---|---|---| | VXLAN | Ethernet in UDP; 24-bit VNI | UDP 4789 standard; Cilium 8472 default | Fixed base header; not encryption | | Geneve | Generic network virtualization with extensible options; 24-bit VNI | UDP 6081 | Option length changes overhead | | GRE | Generic encapsulation; base GRE has no VXLAN-style VNI | IP protocol 47, not TCP/UDP port 47 | Optional extensions must be considered; “unlimited networks” is not a defined capacity | | NVGRE | Ethernet over GRE; 24-bit VSID within the GRE key | IP protocol 47 | Different identifier/flow semantics; support depends on implementation | ## Cilium's Overlay Networking Without an overriding platform/profile configuration, Cilium uses tunnel routing with VXLAN. Cross-node Pod transport needs reachable node addresses, permitted tunnel UDP traffic and a usable MTU. Overlay does not fix overlapping Pod address ranges or make disconnected nodes reachable. ![Cross-node overlay flow: endpoint processing, source VTEP encapsulation, underlay transit, destination decapsulation and endpoint delivery.](https://raw.githubusercontent.com/Atom-oh/kubernetes-docs/main/en/.gitbook/assets/en-networking-cilium-03-networking-2.png) [🔍 View interactive diagram](https://www.atomai.click/kubernetes-docs/archmaps/en-networking-cilium-03-networking-2.html) The figure is a conceptual flow. Its addresses are not a per-node IPAM allocation plan; real node Pod blocks must be allocated consistently without overlap. Policy is applied where configured, and source/destination hooks can differ. 1. Cilium identifies the remote endpoint/node from control-plane and datapath state. 2. The source encapsulates the relevant Pod packet and sends it using underlay node addresses. 3. The destination decapsulates and processes/delivers the inner packet. 4. Datapath events, route state and captures help locate failures; one missing event alone does not identify the cause. ## Routing Mechanisms ### Encapsulation The underlay only needs the node/tunnel path, rather than a route for every Pod prefix. The cost includes headers and processing. Larger frames can reduce the relative overhead only if the whole path supports the chosen MTU. ### Native Routing The node and underlay must route the Pod addresses, including return traffic. Routes can come from a cloud network, a router, static configuration or another routing component. Enabling native mode does not automatically start BGP or advertise every Pod CIDR. `autoDirectNodeRoutes: true` installs direct PodCIDR routes for nodes sharing an L2 network. With multiple L2 segments, `directRoutingSkipUnreachable` may skip unreachable direct routes while an independently working routed path handles them. It does not fall back to overlay tunnels. **Do not combine tunnel routing with `autoDirectNodeRoutes: true`: Cilium 1.20.1 explicitly rejects this combination at startup.** The old “hybrid mode” recipe was invalid. Native routing can still coexist with feature-specific encapsulation, such as a configured Geneve DSR path; that is a separate service feature. The Cilium BGP Control Plane advertises configured Pod/Service prefixes to peers. It does **not program the local datapath** and must not be treated as the component that automatically supplies missing intra-cluster routes. ## Performance Optimization Techniques Measure with the same protocol, payload sizes, concurrency, node placement, policy, encryption and proxy settings before comparing modes. Removing one encapsulation header does not guarantee lower application latency. - **Datapath:** socket load balancing, supported XDP acceleration and DSR apply to particular paths. They are not automatically enabled by VXLAN or native routing. - **Connection tracking:** Cilium BPF connection tracking and Linux netfilter conntrack are different state mechanisms. Bypassing a netfilter path does not mean all established traffic stops using Cilium connection state. - **Maps:** size maps against actual capacity and memory pressure. LRU eviction is useful for caches, not a universal optimization for every map. - **Host tuning:** CPU/NUMA placement, IRQ distribution/coalescing and queue configuration can help or hurt particular workloads. Huge pages are not a general Cilium speed switch; require evidence for the actual consumer and environment. ## Cloud Provider-specific Networking | Environment | Correct distinction | |---|---| | AWS ENI IPAM | Cilium allocates VPC-routable ENI addresses with operator IAM/API/subnet/instance-capacity requirements. ENI security groups and Cilium policy complement each other; this is not automatically the AWS VPC CNI's per-Pod branch-ENI feature | | EKS platforms | Alternate CNI on ordinary EC2 nodes has separate support responsibilities. Fargate and EKS Auto Mode do not support replacing their CNI with this generic lab profile. Hybrid Nodes have a separate supported installation path | | Google Cloud | Self-managed upstream Cilium can use Kubernetes host-scope IPAM and routable alias ranges. Managed GKE Dataplane V2 uses Google-managed Cilium/`anetd`; do not install another upstream dataplane over it or assume identical exposed features | | Azure | Azure CNI Powered by Cilium is managed by AKS with delegated IPAM. Upstream Azure IPAM targets self-managed Azure VM/VMSS clusters; AKS BYOCNI is another explicitly selected deployment model | Cloud firewall/security-group configuration is not automatically created by every Cilium policy. Select the platform guide and support model first. ## Lab: Cilium Networking Mode Configuration and Performance Testing ### Select One Mode on a Fresh Prepared Cluster Save this common file, first replacing the Pod CIDR if it overlaps node, Service, VPC or connected-network ranges. Keep the chosen range consistent with cluster/kube-proxy configuration and the native profile's `ipv4NativeRoutingCIDR`; changing only one file is insufficient. `kubeProxyReplacement: false` deliberately assumes working kube-proxy. These are Helm values, not a ConfigMap to apply with kubectl. **`lab-common.yaml`** ```yaml kubeProxyReplacement: false ipv4: enabled: true ipv6: enabled: false ipam: mode: cluster-pool operator: clusterPoolIPv4PodCIDRList: - 10.244.0.0/16 clusterPoolIPv4MaskSize: 24 MTU: 0 hubble: enabled: true relay: enabled: true ui: enabled: true ``` Choose exactly one of the following mode files. Use separate disposable clusters for comparisons rather than reinstalling the CNI repeatedly on a live cluster. **`mode-vxlan.yaml`** ```yaml routingMode: tunnel tunnelProtocol: vxlan tunnelPort: 8472 autoDirectNodeRoutes: false ``` **`mode-geneve.yaml`** ```yaml routingMode: tunnel tunnelProtocol: geneve tunnelPort: 6081 autoDirectNodeRoutes: false ``` **`mode-native.yaml`** ```yaml routingMode: native ipv4NativeRoutingCIDR: 10.244.0.0/16 autoDirectNodeRoutes: true ``` For the VXLAN example: ```bash kubectl config current-context cilium install --version 1.20.1 --values lab-common.yaml --values mode-vxlan.yaml cilium status --wait ``` Select `mode-geneve.yaml` or `mode-native.yaml` instead only when its network prerequisites hold. Do not apply the obsolete `tunnel: vxlan`, `ipv4-range` or `ipv4-service-range` ConfigMap examples; configure IPAM through the supported Helm fields. ### Network Performance Testing Use the CLI's maintained performance workloads rather than assuming an unrelated manifest creates `netperf-client` and `netperf-server`. On the prepared disposable cluster: ```bash cilium connectivity perf --test-namespace cilium-net-perf \ --namespace-labels docs-audit-lab=cilium-networking-03 \ --duration 10s --samples 2 --crr --udp \ --host-net=false --pod-net=true --same-node=true --other-node=true \ --report-dir ./cilium-net-perf-results ``` The duration is per test case/sample, not a ten-second total run. This creates test workloads and network load. CLI 0.20.0 appends a sequence suffix to the namespace (`cilium-net-perf-1` for the default single suite). Save versions, placement and settings with results; no throughput/latency number is guaranteed. TCP request/response, connection-rate and stream tests answer different questions. If using an independently prepared iperf3 setup, UDP testing still needs the TCP control connection and a UDP data path; a Service exposing only TCP 5201 is insufficient. Offered UDP rate is not measured achieved throughput. These examples were checked against current official values, API schemas and CLI source. This audit did not render Helm templates, deploy a cluster or run a network benchmark after the host restart; validate the complete platform/lab environment before relying on results. ## Sources - [Cilium 1.20.1 routing](https://github.com/cilium/cilium/blob/v1.20.1/Documentation/network/concepts/routing.rst), [Helm values](https://github.com/cilium/cilium/blob/v1.20.1/install/kubernetes/cilium/values.yaml), [startup validation](https://github.com/cilium/cilium/blob/v1.20.1/daemon/cmd/daemon_main.go), [MTU calculation](https://github.com/cilium/cilium/blob/v1.20.1/pkg/mtu/mtu.go), [MTU option](https://github.com/cilium/cilium/blob/v1.20.1/pkg/mtu/cell.go) - [VXLAN RFC 7348](https://www.rfc-editor.org/rfc/rfc7348.txt), [Geneve RFC 8926](https://www.rfc-editor.org/rfc/rfc8926.txt), [GRE RFC 2784](https://www.rfc-editor.org/rfc/rfc2784.txt), [NVGRE RFC 7637](https://www.rfc-editor.org/rfc/rfc7637.txt) - [BGP Control Plane](https://github.com/cilium/cilium/blob/v1.20.1/Documentation/network/bgp-control-plane/bgp-control-plane.rst), [AWS ENI](https://github.com/cilium/cilium/blob/v1.20.1/Documentation/network/concepts/ipam/eni.rst), [EKS alternate CNI](https://docs.aws.amazon.com/eks/latest/userguide/alternate-cni-plugins.html), [GKE Dataplane V2](https://docs.cloud.google.com/kubernetes-engine/docs/concepts/dataplane-v2), [Azure IPAM](https://github.com/cilium/cilium/blob/v1.20.1/Documentation/network/concepts/ipam/azure.rst) - [CLI 0.20.0 connectivity/perf options](https://github.com/cilium/cilium-cli/blob/v0.20.0/vendor/github.com/cilium/cilium/cilium-cli/cli/connectivity.go), [iperf3 invocation](https://software.es.net/iperf/invoking.html), [kubectl version skew](https://kubernetes.io/releases/version-skew-policy/) [Return to Main Page](https://www.atomai.click/kubernetes-docs/llms/en/networking/cilium/README.md) ## Quiz Work through the [networking validation exercises and expected results](https://www.atomai.click/kubernetes-docs/en/quizzes/networking/cilium/03-networking-quiz). ---------------------------------------- Source: https://www.atomai.click/kubernetes-docs/en/networking/cilium/04-ipam-policy ---------------------------------------- # IPAM and Network Policies > **Review baseline**: Cilium 1.20.1; tested Kubernetes 1.33–1.36. Resource API versions and platform requirements are checked separately. > **Last reviewed**: September 12, 2026 ## Lab Environment Setup Use the [installation profiles](https://www.atomai.click/kubernetes-docs/llms/en/networking/cilium/README.md) and [networking prerequisites](https://www.atomai.click/kubernetes-docs/llms/en/networking/cilium/03-networking.md) to prepare a disposable Linux cluster. Keep kubectl within the supported API-server version skew. IPAM profiles below are alternatives for installation or a documented migration, not ConfigMap toggles to apply sequentially to a running cluster. The policy lab uses its own namespace and matching workloads below, replacing the old Cilium 1.14 Star Wars manifest whose labels did not match these policies. Inspect existing Kubernetes/Cilium cluster-wide policy and admission settings first; a new namespace does not override them. ```bash cilium status --wait kubectl -n kube-system get configmap cilium-config -o yaml kubectl get ciliumnodeconfigs --all-namespaces -o yaml helm -n kube-system get values cilium --all ``` Use the actual namespace/release name if different. Desired Helm values and ConfigMaps, node overrides, operator settings and realized agent state answer different questions. A grep result containing `ipam` is not a complete effective-configuration check. ## IP Address Management Strategies Cilium Pod IPAM allocates workload addresses. Kubernetes allocates Service **ClusterIPs**; Cilium **LoadBalancer IPAM** is a separate facility for LoadBalancer addresses. A `CiliumPodIPPool` does not allocate Service ClusterIPs. ### Allocators and Authoritative State | Configuration | Who allocates node capacity / Pod IPs? | State to inspect | |---|---|---| | `cluster-pool` (generic default) | Cilium Operator assigns per-node CIDRs; each agent allocates local addresses | `CiliumNode.spec.ipam.podCIDRs` and operator status | | `kubernetes` / host scope | Kubernetes supplies node PodCIDRs; each agent allocates within them | Kubernetes `Node.spec.podCIDRs` / `spec.podCIDR` and supported provider annotations | | `multi-pool` | Operator assigns blocks from named pools in response to agent demand; agent allocates Pod IPs | `CiliumPodIPPool`, `CiliumNode.spec.ipam.pools.requested` / `allocated` | | `crd` | An external allocator supplies available addresses; agent consumes/releases them | `CiliumNode.spec.ipam.pool` / `status.ipam.used` (and applicable IPv6 fields) | | `eni` | Operator manages AWS ENIs/IPs/prefixes; agent translates interface state into its multi-pool allocator | `CiliumNode.status.eni.enis` and mode-specific pool/demand state | | `azure` | Upstream operator/agent integration for self-managed Azure VM/VMSS clusters | Azure/CiliumNode allocation state | | `delegated-plugin` | Cilium CNI invokes another IPAM plugin, such as managed AKS's Azure IPAM | Provider/plugin state; do not substitute upstream Azure IPAM configuration | | GKE integration | Upstream GKE integration uses host-scope `kubernetes` IPAM; managed Dataplane V2 has its own ownership | Provider configuration and Node CIDRs; not a separate `gke` IPAM value | Both cluster-pool and Kubernetes host scope allocate individual Pod addresses locally. The difference is who allocates the **node prefixes**. Neither eliminates coordination or guarantees that a user-supplied pool cannot overlap a VPC, node network, Service range or another cluster. The released CRD/type definitions use `pool` and `used` for generic CRD-backed allocation. Some explanatory documentation still says `available`/`inuse`; use the installed schema and mode-specific fields rather than copying those older names. ENI in 1.20.1 uses the interface/multi-pool path, so generic CRD-backed fields are not its universal source of truth. ### Kubernetes/CNI Integration The kubelet requests Pod sandbox operations through the container runtime; a CNI-capable runtime invokes the Cilium CNI plugin. Allocation depends on the selected backend, after which the plugin/agent configures the endpoint network. In host-scope mode, Kubernetes must supply the required address-family CIDRs, for example through a correctly configured node-CIDR allocator. ## IPAM Configuration These are **Helm value fragments** to merge into the matching installation profile. Review Pod/Service/node/external address ranges first. ### Cluster Pool **`cluster-pool-values.yaml`** ```yaml ipam: mode: cluster-pool operator: clusterPoolIPv4PodCIDRList: - 10.244.0.0/16 clusterPoolIPv4MaskSize: 24 ipv4: enabled: true ipv6: enabled: false ``` The Operator coordinates node blocks, not a central request for each Pod address. Multiple CIDRs in `clusterPoolIPv4PodCIDRList` expand the common allocation space; this is different from selecting named pools per workload. Do not replace existing pool-list entries to grow a live cluster. Add a non-conflicting CIDR through the documented expansion procedure. The node mask size is not a routine mutable setting. Address counts per block are not identical to usable Pod capacity because addresses are reserved or used by node-local facilities. For a cluster already prepared for Kubernetes/Cilium dual stack: **`dual-stack-values.yaml`** ```yaml ipam: mode: cluster-pool operator: clusterPoolIPv4PodCIDRList: - 10.244.0.0/16 clusterPoolIPv4MaskSize: 24 clusterPoolIPv6PodCIDRList: - fd00:10:244::/104 clusterPoolIPv6MaskSize: 120 ipv4: enabled: true ipv6: enabled: true ``` Enabling these two Cilium address-family flags does not configure Kubernetes Service CIDRs, underlay IPv6 connectivity or cloud support by itself. ### Multi-Pool and CiliumPodIPPool The documented mode is `multi-pool`; the resource API remains `cilium.io/v2alpha1`. Do not infer a feature's maturity solely from that API suffix. On a fresh cluster using this mode, provide a default pool for ordinary allocations: **`multi-pool-values.yaml`** ```yaml ipam: mode: multi-pool operator: autoCreateCiliumPodIPPools: default: ipv4: cidrs: - 10.244.0.0/16 maskSize: 24 ``` This additional named pool uses the current `cidrs` and `maskSize` fields: **`blue-pool.yaml`** ```yaml apiVersion: cilium.io/v2alpha1 kind: CiliumPodIPPool metadata: name: blue-pool spec: ipv4: cidrs: - 10.245.0.0/16 maskSize: 24 namespaceSelector: matchLabels: ipam-pool: blue podSelector: matchLabels: role: blue ``` The pool resource is cluster-scoped. `podSelector` and `namespaceSelector` are separate fields, and both must match when both are configured. The old `ipv4.cidr`, `blockSize` and generic `selector` example was invalid. For the selector example: **`blue-namespace.yaml`** ```yaml apiVersion: v1 kind: Namespace metadata: name: ipam-selection-demo labels: ipam-pool: blue annotations: ipam.cilium.io/require-pool-match: 'true' ``` **`blue-pod.yaml`** ```yaml apiVersion: v1 kind: Pod metadata: name: blue-client namespace: ipam-selection-demo labels: role: blue spec: automountServiceAccountToken: false containers: - name: client image: quay.io/cilium/alpine-curl:v1.10.0@sha256:913e8c9f3d960dde03882defa0edd3a919d529c2eb167caa7f54194528bde364 command: - /usr/bin/pause ``` Apply these only on a prepared multi-pool installation. The namespace's `require-pool-match` annotation prevents automatic fallback to the default pool when a non-default selector match is required. Pool choice follows explicit Pod/namespace `ipam.cilium.io/ip-pool` or address-family pool annotations, then automatic selectors, then the default pool. Automatic selection must match exactly one pool for the address family; overlapping selectors cause allocation failure. Pool annotations affect **new allocations**, not already-running Pod IPs. Node-specific defaults can also be configured through `CiliumNodeConfig`. Pool selection is not a substitute for network authorization: control who can change workload labels/annotations and enforce traffic policy separately. Pools must not have overlapping CIDRs. In-use ranges/pools must not be removed casually; `maskSize`, `allowFirstIP` and `allowLastIP` are immutable. The first/last address reservation has documented small-prefix exceptions. A current documented online migration exists from **cluster-pool to multi-pool**. That does not authorize arbitrary live IPAM changes or an unplanned reverse migration. Follow its prerequisites and workload/capacity checks; no migration is executed by this chapter. ### AWS ENI Use the full EKS/ENI installation profile for routing, operator IAM, subnet/instance capacity and node preparation. The following fragment illustrates the current keys, including optional IPv4 prefix delegation: **`eni-values-fragment.yaml`** ```yaml ipam: mode: eni eni: enabled: true eniTags: team: platform awsEnablePrefixDelegation: true routingMode: native endpointRoutes: enabled: true ipv4: enabled: true ipv6: enabled: false ``` `eni.awsEnablePrefixDelegation` requires the instance/subnet setup to support the requested prefixes. An IPv4 `/28` contains 16 addresses, not a guarantee of 16 additional schedulable Pods in every configuration. The default is disabled. Do not add the invented `eni-prefix-delegation-enabled` key. The Operator makes the EC2 API calls; pre-allocation reduces per-Pod delays but cannot eliminate quota, API or subnet exhaustion. `eni.eniTags` tags managed interfaces. Let the SDK resolve the appropriate EC2 endpoint unless there is a deliberate, validated `eni.ec2APIEndpoint` override. This example is IPv4. The ENI reference documents IPv6 as beta with different prefix/allocation behavior; platform validation is separate from enabling `ipv6.enabled`. AWS VPC CNI chaining keeps address ownership with AWS VPC CNI. Ordinary EC2, Hybrid Nodes, Fargate and Auto Mode have different installation/support boundaries; Fargate and Auto Mode cannot use this replacement profile. ## Querying Per-Node Allocation State ### CiliumNode Example This is an **illustrative read-only object shape**, not a manifest to apply over Operator-owned state: **`ciliumnode-example.yaml`** ```yaml apiVersion: cilium.io/v2 kind: CiliumNode metadata: name: hybrid-node-001 spec: addresses: - ip: 10.85.0.1 type: CiliumInternalIP - ip: 10.80.1.10 type: InternalIP ipam: podCIDRs: - 10.85.0.0/25 ``` The first address can be `CiliumInternalIP`, not the underlay node's `InternalIP`. Multiple InternalIPs/address families and multiple allocation entries can exist. Even a typed InternalIP is only a candidate for a network design, not an automatically valid next hop from every router. ### Mode-specific Inventory Save this query as `ciliumnode-inventory.jq`: **`ciliumnode-inventory.jq`** ```text .items[] | { name: .metadata.name, internalNodeIPs: ([.spec.addresses[]? | select(.type == "InternalIP") | .ip] | unique), clusterPoolPodCIDRs: (.spec.ipam.podCIDRs // []), multiPoolAllocations: (.spec.ipam.pools.allocated // []), eniInterfaceIDs: ((.status.eni.enis // {}) | keys), operatorStatus: (.status.ipam["operator-status"] // {}) } ``` ```bash kubectl get ciliumnodes -o json | jq -f ciliumnode-inventory.jq ``` An absent/empty field can be normal for a different IPAM mode. For Kubernetes host scope, inspect the Kubernetes Node instead: **`kubernetes-node-inventory.jq`** ```text .items[] | { name: .metadata.name, internalNodeIPs: ([.status.addresses[]? | select(.type == "InternalIP") | .address] | unique), podCIDRs: (.spec.podCIDRs // []), legacyPodCIDR: (.spec.podCIDR // null) } ``` ```bash kubectl get nodes -o json | jq -f kubernetes-node-inventory.jq ``` These queries retain all relevant addresses/CIDRs instead of silently selecting `[0]`. They produce inventory, not `ip route add` commands. Select interfaces/next hops and verify forwarding/return paths through the network's routing procedure. Do not assume every CIDR should be installed via the first address on an unrelated router. The inventory can inform [EKS Hybrid Nodes network planning](https://www.atomai.click/kubernetes-docs/llms/en/eks-hybrid-nodes/02-network-configuration.md), but it does not replace that environment's routing and reachability checks. ## Network Policy Design and Implementation ### Resource and Rule Semantics - A namespaced `CiliumNetworkPolicy` selects endpoints in its namespace; `CiliumClusterwideNetworkPolicy` provides cluster-wide scope. Host-firewall `nodeSelector` is supported only in the latter, with host firewall configured. - `endpointSelector` identifies subjects; ingress/egress describe traffic relative to them. Cilium rule `spec.labels` stores optional identification/metadata, not references that inherit another policy. Kubernetes `metadata.labels` labels the resource itself. - Allowed traffic from applicable policies is combined; explicit deny rules have their documented precedence. Another unrestricted L4 allow can bypass an overlapping L7-restricted allow. - Default deny is direction-specific. Preserve required DNS/application paths deliberately, and inspect realized policy plus actual flows. Merely disabling default deny is not a universal L7 dry-run mode. ### Matched Policy Lab Run these steps in one shell in the disposable cluster: ```bash set -euo pipefail kubectl create namespace cilium-ipam-policy-demo kubectl label namespace cilium-ipam-policy-demo docs-audit-lab=cilium-ipam-policy-04 ``` Stop if the namespace already exists and choose a fresh name consistently. These workloads use the official CLI's test images and labels used by the policies: **`policy-app.yaml`** ```yaml apiVersion: v1 kind: Pod metadata: name: frontend namespace: cilium-ipam-policy-demo labels: app: frontend spec: automountServiceAccountToken: false containers: - name: client image: quay.io/cilium/alpine-curl:v1.10.0@sha256:913e8c9f3d960dde03882defa0edd3a919d529c2eb167caa7f54194528bde364 command: - /usr/bin/pause --- apiVersion: v1 kind: Pod metadata: name: outsider namespace: cilium-ipam-policy-demo labels: app: outsider spec: automountServiceAccountToken: false containers: - name: client image: quay.io/cilium/alpine-curl:v1.10.0@sha256:913e8c9f3d960dde03882defa0edd3a919d529c2eb167caa7f54194528bde364 command: - /usr/bin/pause --- apiVersion: apps/v1 kind: Deployment metadata: name: backend namespace: cilium-ipam-policy-demo spec: replicas: 1 selector: matchLabels: app: backend template: metadata: labels: app: backend spec: automountServiceAccountToken: false affinity: podAntiAffinity: requiredDuringSchedulingIgnoredDuringExecution: - labelSelector: matchLabels: app: frontend topologyKey: kubernetes.io/hostname containers: - name: http image: quay.io/cilium/json-mock:v1.4.1@sha256:6a66df90808a39c02e7a9d58af7bf0e54d8f8b7d4bc528f48c891969a7049195 ports: - containerPort: 8080 name: http readinessProbe: httpGet: path: / port: http --- apiVersion: v1 kind: Service metadata: name: backend namespace: cilium-ipam-policy-demo spec: selector: app: backend ports: - name: http port: 8080 targetPort: http protocol: TCP --- apiVersion: v1 kind: Pod metadata: name: client namespace: cilium-ipam-policy-demo labels: app: client spec: automountServiceAccountToken: false containers: - name: client image: quay.io/cilium/alpine-curl:v1.10.0@sha256:913e8c9f3d960dde03882defa0edd3a919d529c2eb167caa7f54194528bde364 command: - /usr/bin/pause ``` ```bash kubectl apply -f policy-app.yaml kubectl -n cilium-ipam-policy-demo wait --for=condition=Ready \ pod/frontend pod/outsider pod/client --timeout=120s kubectl -n cilium-ipam-policy-demo rollout status deployment/backend --timeout=120s kubectl -n cilium-ipam-policy-demo get pods -o wide --show-labels BACKEND_IP=$(kubectl -n cilium-ipam-policy-demo get service backend -o jsonpath='{.spec.clusterIP}') test -n "$BACKEND_IP" kubectl -n cilium-ipam-policy-demo exec frontend -- \ curl --fail --silent --show-error --max-time 5 "http://$BACKEND_IP:8080/" ``` First confirm baseline connectivity from the outsider as well. Backend anti-affinity requires another eligible node. The database rule below illustrates an additional application dependency; this lab does not deploy or validate a database server. ### L3/L4 Policy **`backend-l4.yaml`** ```yaml apiVersion: cilium.io/v2 kind: CiliumNetworkPolicy metadata: name: backend-access namespace: cilium-ipam-policy-demo spec: endpointSelector: matchLabels: app: backend ingress: - fromEndpoints: - matchLabels: k8s:io.kubernetes.pod.namespace: cilium-ipam-policy-demo k8s:app: frontend toPorts: - ports: - port: '8080' protocol: TCP egress: - toEndpoints: - matchLabels: k8s:io.kubernetes.pod.namespace: cilium-ipam-policy-demo k8s:app: database toPorts: - ports: - port: '3306' protocol: TCP - toEndpoints: - matchLabels: k8s:io.kubernetes.pod.namespace: kube-system k8s:k8s-app: kube-dns toPorts: - ports: - port: '53' protocol: UDP - port: '53' protocol: TCP ``` Apply `backend-l4.yaml`, wait for policy realization on the backend's agent and check fresh requests. Frontend should retain access; outsider denial requires flow evidence, not just a nonzero curl exit. ### L7 HTTP Policy This is an **alternative definition of the same `backend-access` resource**, not a second overlapping allow policy. Applying it replaces this lab's L4 version; still inspect other applicable policies. **`backend-http.yaml`** ```yaml apiVersion: cilium.io/v2 kind: CiliumNetworkPolicy metadata: name: backend-access namespace: cilium-ipam-policy-demo spec: endpointSelector: matchLabels: app: backend ingress: - fromEndpoints: - matchLabels: k8s:io.kubernetes.pod.namespace: cilium-ipam-policy-demo k8s:app: frontend toPorts: - ports: - port: '8080' protocol: TCP rules: http: - method: ^GET$ path: ^/$ - method: ^POST$ path: ^/$ headerMatches: - name: content-type value: application/json egress: - toEndpoints: - matchLabels: k8s:io.kubernetes.pod.namespace: cilium-ipam-policy-demo k8s:app: database toPorts: - ports: - port: '3306' protocol: TCP - toEndpoints: - matchLabels: k8s:io.kubernetes.pod.namespace: kube-system k8s:k8s-app: kube-dns toPorts: - ports: - port: '53' protocol: UDP - port: '53' protocol: TCP ``` ```bash kubectl apply -f backend-http.yaml kubectl -n cilium-ipam-policy-demo get cnp backend-access -o yaml ``` After realization, the policy permits GET `/`, and POST `/` only with the exact `content-type: application/json` value. A request to another path/method should be denied by the proxy. The application must itself implement an allowed operation; policy permission does not guarantee application success. The demo server's known readiness path is GET `/`. HTTP method/path fields are regular expressions. Keep examples anchored and escape metacharacters when adapting them. HTTP/gRPC rules require Envoy and visible application traffic; TLS termination/interception must be configured when needed. gRPC service/method information is carried in the HTTP/2 path and metadata in headers, not a separate `rules.grpc` field or arbitrary protobuf-payload filtering. ### Kafka Policy Boundary Current Cilium does not provide the removed `rules.kafka` API. Do not apply the old topic/API-key/client-ID YAML. Use appropriate L4 connectivity rules to the broker's **actual listener port**, and configure authentication/authorization such as topic access in the broker. NetworkPolicy neither replaces broker authorization nor enables encryption. ### DNS/FQDN Policy This example assumes the selected resolver is a CoreDNS/kube-dns Pod in `kube-system` on TCP/UDP 53. Inspect the Pod's actual resolver first; NodeLocal DNS, OpenShift and managed DNS paths need their own matching configuration. **`dns-egress.yaml`** ```yaml apiVersion: cilium.io/v2 kind: CiliumNetworkPolicy metadata: name: dns-egress namespace: cilium-ipam-policy-demo spec: endpointSelector: matchLabels: app: client egress: - toEndpoints: - matchLabels: k8s:io.kubernetes.pod.namespace: kube-system k8s:k8s-app: kube-dns toPorts: - ports: - port: '53' protocol: UDP - port: '53' protocol: TCP rules: dns: - matchPattern: '*' - toFQDNs: - matchName: api.example.com - matchPattern: '*.googleapis.com' toPorts: - ports: - port: '443' protocol: TCP ``` The DNS L7 rule redirects matching resolver traffic through Cilium's DNS proxy so it can learn name-to-IP responses. Permitting port 53 alone does not provide that observation. `matchPattern: "*"` permits DNS queries to the selected resolver; HTTPS egress is separately limited to IPs learned for the `toFQDNs` names. Replace `api.example.com` with a resolvable, controlled test name before claiming a successful external test. `*.googleapis.com` matches one label beneath that suffix; it does not match the apex or multiple labels. In this release, `**.googleapis.com` can match one or more subdomain levels, still excluding the apex. DNS TTL/cache state and fresh lookups matter. Name-to-IP allowance is not an HTTP hostname, URL or application-user authorization check. The default DNS proxy runs in the agent; a separate standalone DNS proxy is documented as alpha. DNS policy does not universally require Envoy. For OpenShift, the official example uses the `openshift-dns` resolver configuration and port 5353 rather than blindly copying this rule. ### CIDRs, Services and Entities | Rule | Boundary | |---|---| | `toCIDR` / `toCIDRSet` | Select IP prefixes, primarily external peers. By default they do not substitute for selectors of Cilium-managed Pods/nodes; documented opt-in CIDR matching for `pods`/`nodes` is beta and consumes identities | | `toServices` | Resolves Service selectors or selectorless EndpointSlice addresses into policy selectors; it does not create a Service or route. Selectorless cases inherit CIDR-mode limitations | | `world` | Broad outside-cluster identity category, not “public Internet only” or a named remote-cluster selector | | `cluster` / `cluster-mesh` | `cluster` covers local cluster endpoints plus documented reserved entities/remote nodes; `cluster-mesh` additionally selects meshed-cluster endpoints | | `all` | Broad combination including cluster/mesh and external peers; not a least-privilege shortcut | For API-server access, use the documented `kube-apiserver` entity behavior rather than assuming `toServices: default/kubernetes` has ordinary workload-selector semantics. ## Multi-cluster Scenarios Cluster Mesh shares state while keeping Kubernetes clusters and network namespaces separate. Remote nodes do not become local Kubernetes Node objects, and policy resources are not automatically distributed. ```text State: cluster A Cluster Mesh control plane <-- mTLS --> cluster B control plane Data: Pod A --> node A datapath --> reachable network --> node B datapath --> Pod B ``` The Cluster Mesh API server synchronizes state; Pod packets do not need to pass through it. Control-plane mTLS does not by itself encrypt Pod-to-Pod traffic. ### Setup Prerequisites and Partial Sequence Prepare separate clusters with non-overlapping Pod CIDRs, reachable node InternalIPs, allowed network paths, the same datapath mode and Cilium versions within the documented one-minor difference. Native routing additionally needs all remote Pod ranges reachable and covered by the configured native-routing CIDR. Assign unique Cilium names/IDs at installation (for example `cluster-a`/1 and `cluster-b`/2) and configure peer certificate trust. The following is a partial sequence **after** those prerequisites and the private NodePort control-plane path are prepared. Kubeconfig context names need not equal Cilium cluster names: ```bash export CTX_A=prepared-context-a export CTX_B=prepared-context-b cilium clustermesh enable --context "$CTX_A" --service-type NodePort cilium clustermesh enable --context "$CTX_B" --service-type NodePort cilium clustermesh connect --context "$CTX_A" --destination-context "$CTX_B" cilium clustermesh status --context "$CTX_A" --wait cilium clustermesh status --context "$CTX_B" --wait ``` This does not provision VPC peering/VPNs, routes, firewalls, private endpoints or certificate trust. Follow the full platform-specific setup; do not change a live cluster's name/ID casually. ### Global Services and Cross-cluster Policy Create matching namespaces and actual backend workloads in each prepared cluster, then use the **same Service name and namespace** with compatible ports: **`global-service.yaml`** ```yaml apiVersion: v1 kind: Service metadata: name: global-service namespace: mesh-demo annotations: service.cilium.io/global: 'true' spec: type: ClusterIP selector: app: global-app ports: - name: http port: 80 targetPort: 8080 protocol: TCP ``` `service.cilium.io/global` is the current annotation. A global service shares local backends by default; `service.cilium.io/shared: "false"` stops sharing them to peers without necessarily preventing local clients from using remote backends. Local ClusterIPs do not have to be identical. Do not promise automatic failover solely from this annotation. By default, unreachable-cluster state is retained (`clustermesh.cacheTTL: 0s`); a configured positive TTL can revoke stale remote data after control-plane disconnection. That is not an application health probe or a zero-downtime guarantee. Apply this ingress policy in the destination cluster's `mesh-demo` namespace when the named source workload exists in `cluster-a`: **`cross-cluster-policy.yaml`** ```yaml apiVersion: cilium.io/v2 kind: CiliumNetworkPolicy metadata: name: allow-cluster-a-frontend namespace: mesh-demo spec: endpointSelector: matchLabels: app: global-app ingress: - fromEndpoints: - matchLabels: k8s:app: frontend k8s:io.kubernetes.pod.namespace: frontend-ns k8s:io.cilium.k8s.policy.cluster: cluster-a toPorts: - ports: - port: '8080' protocol: TCP ``` The cluster label uses the configured **Cilium cluster name**, not a kubeconfig context. In current Cilium, endpoint selectors default to the local cluster unless peers are explicitly selected. Install the required policies independently in each cluster and test both traffic directions. ## Validation and Cleanup The examples were checked against release-specific schemas, configuration/source contracts and bounded local fixtures. This audit does not claim live IP allocation, kernel policy enforcement, cloud provisioning, a running database or successful cross-cluster traffic. Inspect desired and realized state, verify a successful baseline, then correlate expected denials with the relevant flow. Clean up only this run's namespaced policy workloads after verifying ownership. For the separate multi-pool exercise, release workloads and verify pool allocations are no longer in use before considering pool removal; never delete active CiliumNode/pool state as a shortcut. ## Sources - [IPAM modes/migration](https://github.com/cilium/cilium/blob/v1.20.1/Documentation/network/concepts/ipam/index.rst), [cluster pool](https://github.com/cilium/cilium/blob/v1.20.1/Documentation/network/concepts/ipam/cluster-pool.rst), [host scope](https://github.com/cilium/cilium/blob/v1.20.1/Documentation/network/concepts/ipam/kubernetes.rst), [multi-pool](https://github.com/cilium/cilium/blob/v1.20.1/Documentation/network/concepts/ipam/multi-pool.rst), [migration procedure](https://github.com/cilium/cilium/blob/v1.20.1/Documentation/network/concepts/ipam/cluster-pool-to-multi-pool.rst) - [PodIPPool schema](https://github.com/cilium/cilium/blob/v1.20.1/pkg/k8s/apis/cilium.io/client/crds/v2alpha1/ciliumpodippools.yaml), [CiliumNode schema](https://github.com/cilium/cilium/blob/v1.20.1/pkg/k8s/apis/cilium.io/client/crds/v2/ciliumnodes.yaml), [ENI IPAM](https://github.com/cilium/cilium/blob/v1.20.1/Documentation/network/concepts/ipam/eni.rst), [Helm values](https://github.com/cilium/cilium/blob/v1.20.1/install/kubernetes/cilium/values.yaml), [EKS CNI boundaries](https://docs.aws.amazon.com/eks/latest/userguide/alternate-cni-plugins.html) - [Policy rule API](https://github.com/cilium/cilium/blob/v1.20.1/pkg/policy/api/rule.go), [L3 rules](https://github.com/cilium/cilium/blob/v1.20.1/Documentation/security/policy/layer3.rst), [L7 rules](https://github.com/cilium/cilium/blob/v1.20.1/Documentation/security/policy/layer7.rst), [DNS policies](https://github.com/cilium/cilium/blob/v1.20.1/Documentation/security/dns.rst), [wildcard implementation](https://github.com/cilium/cilium/blob/v1.20.1/pkg/fqdn/matchpattern/matchpattern.go) - [Cluster Mesh setup](https://github.com/cilium/cilium/blob/v1.20.1/Documentation/network/clustermesh/setup.rst), [architecture](https://github.com/cilium/cilium/blob/v1.20.1/Documentation/network/clustermesh/intro.rst), [global services](https://github.com/cilium/cilium/blob/v1.20.1/Documentation/network/clustermesh/global-services.rst), [cross-cluster policy](https://github.com/cilium/cilium/blob/v1.20.1/Documentation/network/clustermesh/policy.rst) [Return to Main Page](https://www.atomai.click/kubernetes-docs/llms/en/networking/cilium/README.md) ## Quiz [Check your IPAM and policy understanding](https://www.atomai.click/kubernetes-docs/en/quizzes/networking/cilium/04-ipam-policy-quiz). ---------------------------------------- Source: https://www.atomai.click/kubernetes-docs/en/networking/cilium/05-l2-l7-networking ---------------------------------------- # L2–L7 Networking and Load Balancing > **Review baseline**: Cilium 1.20.1, CLI 0.20.0; Istio examples use the 1.31 API. > **Last reviewed**: September 12, 2026 ## Lab Environment Setup Use a disposable cluster prepared through the [installation](https://www.atomai.click/kubernetes-docs/llms/en/networking/cilium/README.md) and [networking](https://www.atomai.click/kubernetes-docs/llms/en/networking/cilium/03-networking.md) guides. Check platform/kernel support and kubectl version skew. The HTTP lab needs two schedulable Linux nodes. The DSR/Maglev experiment additionally needs a prepared kube-proxy-free cluster, a reachable real API-server endpoint and a supported network path. Choose complete installation values for the experiment. Repeated `cilium install --config ...` commands are not a live feature-migration procedure, and deleting kube-proxy after an arbitrary install is not a safe shortcut. ## Understanding the OSI Layers OSI is a conceptual model, not seven Cilium processes or a fixed sequence of policy hooks. | Layer | Role / examples | Cilium relationship | |---|---|---| | L1 physical | Bits, media, transceivers, repeaters | Underlying hardware/network requirements | | L2 data link | Ethernet frames, MAC addressing, bridges/switches | Packet handling and explicitly configured L2 service announcements | | L3 network | IP packets, routing, ICMP | Routing, identity/CIDR policy and supported fragment handling | | L4 transport | TCP segments and reliable streams; UDP datagrams without delivery/order guarantees | Port/protocol policy, connection state, service translation | | L5 session | Session/dialog organization | Conceptual functionality often implemented inside applications/protocols | | L6 presentation | Representation, encoding and cryptographic transformations | TLS is often mapped here conceptually, not a universal separate Linux layer | | L7 application | HTTP, DNS, gRPC and other application protocols | Supported proxy policies; an application protocol's existence does not imply a Cilium policy parser | ### Actual Layer-specific Features - **L2:** L2 Announcements is a beta, configured ARP/NDP response mechanism for eligible Service IPs. It needs kube-proxy replacement and appropriate devices/local-network reachability. The elected node receives that Service's traffic; this is not arbitrary MAC/VLAN ACL support or a general L2 bridge/promise to capture every packet. `externalTrafficPolicy: Local` has a documented incompatibility. - **L3:** IP/identity policy and routing have mode-specific requirements. Multicast is a separately enabled beta feature requiring VXLAN; the documented kernel minimum is 5.10 on AMD64 and 6.0 on AArch64. Do not assume it works in every routing mode. - **L4:** TCP/UDP port policy, connection tracking, socket/packet service load balancing and supported affinity operate at different hooks. A socket decision can occur before packet construction. - **L7:** Current built-in policy groups are HTTP and DNS. gRPC uses the supported HTTP/2 path. Kafka L7 rules are removed. TLS/SNI features require their documented proxy configuration; encrypted application content is not automatically inspectable. HTTP/gRPC policy uses Envoy; DNS policy uses Cilium's DNS proxy. Envoy may be an agent-managed process or a dedicated `cilium-envoy` DaemonSet according to values/upgrade compatibility. Fresh 1.20 chart defaults with L7 enabled favor the DaemonSet, and the profile below sets it explicitly. Adding a policy does not override every installation setting. ## HTTP Policy Lab Create a fresh namespace and matching workloads. The images/digests and known server readiness path come from the official CLI test deployment definitions. ```bash set -euo pipefail kubectl create namespace cilium-l2l7-demo kubectl label namespace cilium-l2l7-demo docs-audit-lab=cilium-l2l7-05 ``` Stop if the namespace already exists; choose a new name consistently instead of reusing another run's resources. **`l7-app.yaml`** ```yaml apiVersion: v1 kind: Pod metadata: name: client namespace: cilium-l2l7-demo labels: app: client spec: automountServiceAccountToken: false containers: - name: client image: quay.io/cilium/alpine-curl:v1.10.0@sha256:913e8c9f3d960dde03882defa0edd3a919d529c2eb167caa7f54194528bde364 command: - /usr/bin/pause --- apiVersion: v1 kind: Pod metadata: name: outsider namespace: cilium-l2l7-demo labels: app: outsider spec: automountServiceAccountToken: false containers: - name: client image: quay.io/cilium/alpine-curl:v1.10.0@sha256:913e8c9f3d960dde03882defa0edd3a919d529c2eb167caa7f54194528bde364 command: - /usr/bin/pause --- apiVersion: apps/v1 kind: Deployment metadata: name: app1 namespace: cilium-l2l7-demo spec: replicas: 1 selector: matchLabels: app: app1 template: metadata: labels: app: app1 spec: automountServiceAccountToken: false affinity: podAntiAffinity: requiredDuringSchedulingIgnoredDuringExecution: - labelSelector: matchLabels: app: client topologyKey: kubernetes.io/hostname containers: - name: http image: quay.io/cilium/json-mock:v1.4.1@sha256:6a66df90808a39c02e7a9d58af7bf0e54d8f8b7d4bc528f48c891969a7049195 ports: - containerPort: 8080 name: http readinessProbe: httpGet: path: / port: http --- apiVersion: v1 kind: Service metadata: name: app1-service namespace: cilium-l2l7-demo spec: selector: app: app1 ports: - name: http port: 80 targetPort: http protocol: TCP ``` ```bash kubectl apply -f l7-app.yaml kubectl -n cilium-l2l7-demo wait --for=condition=Ready pod/client pod/outsider --timeout=120s kubectl -n cilium-l2l7-demo rollout status deployment/app1 --timeout=120s kubectl -n cilium-l2l7-demo get pods,services -o wide kubectl -n cilium-l2l7-demo exec client -- \ curl --fail --silent --show-error --max-time 5 http://app1-service/ ``` Verify outsider baseline connectivity too. The Service exposes port 80, but its backend listens on **8080**; the Pod ingress policy uses the backend port. This replaces the nonexistent/unmatched older application setup and a client Pod whose default labels/entrypoint did not match the example. **`app1-http.yaml`** ```yaml apiVersion: cilium.io/v2 kind: CiliumNetworkPolicy metadata: name: app1-http namespace: cilium-l2l7-demo spec: endpointSelector: matchLabels: app: app1 ingress: - fromEndpoints: - matchLabels: k8s:io.kubernetes.pod.namespace: cilium-l2l7-demo k8s:app: client toPorts: - ports: - port: '8080' protocol: TCP rules: http: - method: ^GET$ path: ^/$ - method: ^POST$ path: ^/api/v1$ headerMatches: - name: x-demo-tenant value: team-a ``` ```bash kubectl apply -f app1-http.yaml kubectl -n cilium-l2l7-demo get cnp app1-http -o yaml ``` After policy realization, the named client can make GET `/` requests. POST `/api/v1` is permitted by this rule only with the exact `x-demo-tenant: team-a` header; the backend must still implement that API operation. Other methods/paths/peers are denied only insofar as no other applicable policy allows them. Verify realized policy and flow evidence as well as application responses. The header is a **demonstration filter, not authentication**. The old 32-character “token” condition did not validate identity, signature, issuer, expiry or authorization. In the released translator, a `headers` string containing a value matches that value literally: `X-Auth-Token: ^[a-zA-Z0-9]{32}$` is not a regex token validator. Use explicit `headerMatches` for the intended exact/presence requirement, and authenticate users in the application/appropriate authentication layer. Methods and paths support regular expressions. Built-in Cilium HTTP policy has no arbitrary request-body predicate. Header presence, exact equality, URL filtering and application authorization are different controls. ## Service Mesh Integration Cilium supplies networking and supported network policy while Istio owns its configured proxies and mesh behavior. Integration does not automatically bypass Istio sidecars, remove their mTLS cost, unify all traces or guarantee faster requests. ```text Configuration: istiod --> Istio Envoy proxies Request: app --> source sidecar --> Cilium/network --> destination sidecar --> app ``` ### Preserve Istio's Traffic Interception The current Cilium integration guide offers kube-proxy coexistence and a carefully configured full-replacement option. A coexistence fragment is: **`istio-cilium-values.yaml`** ```yaml kubeProxyReplacement: false socketLB: hostNamespaceOnly: true cni: exclusive: false ``` For an intentionally prepared full-replacement setup, `kubeProxyReplacement: true` additionally requires a reachable API endpoint and the replacement prerequisites. Keep `socketLB.hostNamespaceOnly: true` to avoid Pod socket translation bypassing Istio interception, and `cni.exclusive: false` when sharing the node CNI configuration. Istio sidecar redirection can use an init container or the Istio CNI node agent; ambient uses its corresponding node/CNI path. Follow the [maintained Istio installation guide](https://www.atomai.click/kubernetes-docs/llms/en/service-mesh/istio/01-installation.md), selecting one mode. The Kubernetes API server must reach Istio's admission webhook. Managed-control-plane/overlay networks may need a documented routing or host-network solution; do not prescribe `istiod hostNetwork: true` for every overlay cluster. ### Retain mTLS and Assign L7 Responsibility Do not apply plaintext Cilium HTTP inspection to Istio-encrypted workload traffic. This example keeps Istio mTLS and L7 routing in Istio, and uses Cilium **L3/L4-only** policy. Disabling mTLS just to make the old combined L7 example pass would change the security design. The following is a **sidecar-mode** configuration for an already prepared `istio-cilium-demo` namespace with `productpage` and `reviews` workloads, a reviews Service on port 9080, and reviews Pods labeled `version: v1`/`v2`. It is not a complete Bookinfo deployment. **`istio-reviews.yaml`** ```yaml apiVersion: networking.istio.io/v1 kind: VirtualService metadata: name: reviews-route namespace: istio-cilium-demo spec: hosts: - reviews.istio-cilium-demo.svc.cluster.local http: - match: - headers: end-user: exact: jason route: - destination: host: reviews.istio-cilium-demo.svc.cluster.local subset: v2 port: number: 9080 - route: - destination: host: reviews.istio-cilium-demo.svc.cluster.local subset: v1 port: number: 9080 --- apiVersion: networking.istio.io/v1 kind: DestinationRule metadata: name: reviews-subsets namespace: istio-cilium-demo spec: host: reviews.istio-cilium-demo.svc.cluster.local subsets: - name: v1 labels: version: v1 - name: v2 labels: version: v2 --- apiVersion: security.istio.io/v1 kind: PeerAuthentication metadata: name: default namespace: istio-cilium-demo spec: mtls: mode: STRICT --- apiVersion: cilium.io/v2 kind: CiliumNetworkPolicy metadata: name: reviews-l4 namespace: istio-cilium-demo spec: endpointSelector: matchLabels: app: reviews ingress: - fromEndpoints: - matchLabels: k8s:io.kubernetes.pod.namespace: istio-cilium-demo k8s:app: productpage toPorts: - ports: - port: '9080' protocol: TCP ``` The DestinationRule defines the subsets referenced by the VirtualService. The `end-user: jason` header selects a demo route; it is not authenticated identity. Verify successful sidecar injection, actual endpoint labels, subset readiness and mesh telemetry. Do not reuse the 9080 Cilium rule as an ambient policy recipe: ambient HBONE uses encrypted tunneling on port 15008 and changes the visible traffic/identity boundary. Apply the appropriate Istio policy and platform-specific Cilium network guards for that topology. ## Load Balancing Architecture Cilium's service maps, backend maps, reverse-NAT state and connection tracking support different forwarding stages. L7 Envoy load balancing is another component; do not merge its algorithm list with the BPF service datapath. ### BPF Forwarding Modes and Algorithms | Setting / mechanism | Meaning and limits | |---|---| | `loadBalancer.mode: snat` | Default forwarding mode; applicable external service paths use source translation/reverse state rather than a direct-return path | | `dsr` | Remote backend replies can bypass the ingress load-balancing node. The network must permit that return path; it cannot recover a client IP already translated by an upstream proxy | | `hybrid` | TCP uses DSR and UDP uses SNAT. This is valid load-balancer behavior, distinct from the invalid tunnel/auto-direct-routing combination | | Annotation-based forwarding | Supported opt-in per-Service behavior; forwarding annotations are creation-time choices and changing them can break connections | | `loadBalancer.algorithm: random` | Default BPF backend-selection algorithm | | `maglev` | Consistent selection for supported external N–S paths, including supported XDP acceleration; ordinary socket-LB E–W connections are not subject to Maglev | | `sessionAffinity: ClientIP` | Separate Kubernetes Service affinity. The key is the external source IP or, for applicable in-cluster socket-LB traffic, the client's network-namespace cookie | Maglev is not a guarantee that sessions survive backend removal. Nodes need consistent backend state, table size and seed. The default table size is 16381; 65521 used below is an allowed value, not a universal recommendation. Larger tables cost memory. Affinity expiry and connection state are separate from hashing. DSR dispatch can use native-routing IP options, Geneve under documented native/Geneve-overlay configurations, or the documented native-only IPIP/IP6IP6 path. VXLAN overlay is not interchangeable with Geneve DSR dispatch. IPIP has its own port/translation constraints; verify the release guide before selecting it. XDP acceleration needs supported devices/drivers. `native` expects the selected devices to support it; `best-effort` enables it where supported. It is not enabled simply by writing the old `enable-xdp-acceleration` key, and early XDP forwarding may not be visible at tcpdump's later capture point. ### Cilium and kube-proxy | Aspect | Correct comparison | |---|---| | Linux service implementation | Cilium uses BPF hooks/maps; kube-proxy has iptables and nftables modes, plus IPVS deprecated since Kubernetes 1.35 | | Platform | Cilium's stated Linux/kernel requirements apply; Windows kernelspace kube-proxy is a different implementation | | Connection state | Cilium BPF connection/NAT state is distinct from Linux netfilter conntrack; “optional versus always” is too broad | | L7 | Cilium integrates supported proxies; kube-proxy's Service forwarding is not an HTTP policy engine | | Performance | Measure the same workload and configuration; neither product name establishes a fixed rank | Do not infer kube-proxy DSR configuration merely because the underlying Linux IPVS subsystem has direct-routing capabilities. ## Prepared DSR/Maglev Lab This profile is for a fresh, prepared kube-proxy-free **IPv4 Geneve-overlay** test cluster. It does not migrate an existing CNI, remove kube-proxy or configure cloud anti-spoofing/routing controls. Use a non-conflicting Pod CIDR and validate the external return path. **`lb-values.yaml`** ```yaml kubeProxyReplacement: true routingMode: tunnel tunnelProtocol: geneve ipv4: enabled: true ipv6: enabled: false ipam: mode: cluster-pool operator: clusterPoolIPv4PodCIDRList: - 10.244.0.0/16 clusterPoolIPv4MaskSize: 24 loadBalancer: mode: dsr dsrDispatch: geneve algorithm: maglev acceleration: disabled maglev: tableSize: 65521 bpf: masquerade: true enableIPv4Masquerade: true enableIPv6Masquerade: false l7Proxy: true envoy: enabled: true hubble: enabled: true relay: enabled: true ``` Supply the real API endpoint and one persisted per-cluster Maglev seed. The seed is a base64 encoding of 12 random bytes; generate it once, store it with the cluster values and reuse it, rather than regenerating it on each upgrade. ```bash : "${API_SERVER_HOST:?Set the reachable real API server host, not its ClusterIP}" : "${API_SERVER_PORT:?Set the actual API server port}" : "${MAGLEV_SEED:?Set the persisted base64 encoding of 12 random bytes}" helm repo add cilium https://helm.cilium.io/ helm repo update cilium helm install cilium cilium/cilium --version 1.20.1 --namespace kube-system \ --values lb-values.yaml \ --set-string k8sServiceHost="$API_SERVER_HOST" \ --set k8sServicePort="$API_SERVER_PORT" \ --set-string maglev.hashSeed="$MAGLEV_SEED" cilium status --wait ``` Use the namespace created by this guide in the selected cluster; if this is a different disposable cluster, repeat the namespace creation/label steps there first. The HTTP-policy application and this external-LB backend use different labels, so the earlier client-only HTTP rule does not accidentally block this experiment. **`lb-echo.yaml`** ```yaml apiVersion: apps/v1 kind: Deployment metadata: name: lb-echo namespace: cilium-l2l7-demo spec: replicas: 1 selector: matchLabels: app: lb-echo template: metadata: labels: app: lb-echo spec: automountServiceAccountToken: false containers: - name: http image: quay.io/cilium/json-mock:v1.4.1@sha256:6a66df90808a39c02e7a9d58af7bf0e54d8f8b7d4bc528f48c891969a7049195 ports: - containerPort: 8080 name: http readinessProbe: httpGet: path: / port: http --- apiVersion: v1 kind: Service metadata: name: lb-echo namespace: cilium-l2l7-demo spec: type: NodePort selector: app: lb-echo ports: - name: http port: 80 targetPort: http protocol: TCP ``` ```bash kubectl apply -f lb-echo.yaml kubectl -n cilium-l2l7-demo rollout status deployment/lb-echo --timeout=120s kubectl -n cilium-l2l7-demo get pods -l app=lb-echo -o wide kubectl -n cilium-l2l7-demo get service lb-echo -o wide NODEPORT=$(kubectl -n cilium-l2l7-demo get service lb-echo -o jsonpath='{.spec.ports[0].nodePort}') ``` Use a reachable ingress node **different from the backend node**, and an external client not subject to Cilium's in-cluster socket LB. From that client, request `http://ENTRY_NODE_IP:NODEPORT/` using the actual values. A Pod-to-ClusterIP curl proves neither external DSR nor Maglev behavior. Verify request/response paths and backend/connection state; do not infer them from a successful HTTP response alone. ## Masquerading Pod egress masquerading changes a source address when required for the configured external path. It is not encryption or a firewall, and it is distinct from Service DNAT and DSR forwarding. The following fragment excludes the example `10.0.0.0/8` destination range from source masquerading **only when the network really supports those Pod source/return routes**: **`masquerade-values.yaml`** ```yaml bpf: masquerade: true enableIPv4Masquerade: true enableIPv6Masquerade: false ipv4NativeRoutingCIDR: 10.0.0.0/8 ``` `ipv4NativeRoutingCIDR` expresses the assumed routable range and corresponding masquerade exclusion. It does not install routes or switch the entire datapath to native routing. A broad exclusion without working return routes can break connectivity. - BPF masquerading depends on the BPF NodePort feature in this release and only applies on devices carrying the BPF program. Inspect selected devices; use the documented `devices` configuration when needed. - The iptables implementation uses its documented `egressMasqueradeInterfaces` behavior. Do not treat that field or the removed generic `masquerade-interfaces`/`masquerade-all` examples as universal BPF controls. - IPv6 BPF masquerading is documented as beta. Neither implementation removes Cilium's platform/kernel requirements, and both ultimately process traffic in the kernel. - Node-address exceptions, ip-masq-agent exclusions and later cloud/NAT gateways can affect the observed source. Use a controlled observer plus node-side state/captures; reaching an arbitrary public site does not prove a particular NAT implementation. On the relevant agent: ```bash kubectl -n kube-system get pods -l k8s-app=cilium -o wide export CILIUM_POD=cilium-REPLACE-WITH-AGENT-ON-TARGET-NODE kubectl -n kube-system exec "$CILIUM_POD" -c cilium-agent -- cilium-dbg status --verbose kubectl -n kube-system exec "$CILIUM_POD" -c cilium-agent -- cilium-dbg bpf nat list ``` ## Fragment Handling and MTU The released fragment tracker stores datagram identity and L4 source/destination ports in bounded LRU maps. It can recover port context for later fragments that lack an L4 header; it is **not a BPF payload-reassembly engine** or a guarantee against fragment attacks. The documented feature includes IPv4 and IPv6 tracking, enabled by default through the corresponding flags, and is marked beta. The valid IPv4 flag is still `enable-ipv4-fragment-tracking`. `bpf-fragments-map-max` controls tracked datagram map capacity; the old `fragment-tracking-timeout` and `max-fragments-per-flow` settings are not the released configuration contract. An explicit example using the chart's extra configuration map: **`fragment-values.yaml`** ```yaml extraConfig: enable-ipv4-fragment-tracking: 'true' bpf-fragments-map-max: '8192' ``` 8192 is a capacity example, not a maximum number of fragments per flow. Inspect `cilium_ipv4_frag_datagrams` / `cilium_ipv6_frag_datagrams` and their pressure when diagnosing capacity; pressure is not a reassembly-success or attack-prevention counter. Prefer correct packet sizing and a working path MTU discovery path. PMTUD depends on the relevant error signaling and network behavior; it does not guarantee automatic optimal sizing everywhere. Cilium's `MTU` is the **underlying-network override**. As explained in the [networking guide](https://www.atomai.click/kubernetes-docs/llms/en/networking/cilium/03-networking.md), ordinary VXLAN overhead is 50 bytes for IPv4 underlay and 70 for IPv6; blindly setting the base to 1450 for a 1500-byte path can subtract overhead twice. ## Observability and Troubleshooting Inspect the correct node, actual Envoy deployment mode, desired policy, realized endpoint state and fresh traffic. Use `cilium-dbg` for agent-local operations and the standalone CLI for cluster operations. Removed `policy trace` commands and forced endpoint regeneration are not the starting point for this diagnosis. Keep an enabled Hubble Relay port-forward running in a separate terminal, then observe relevant flows: ```bash cilium hubble port-forward ``` ```bash hubble observe --namespace cilium-l2l7-demo --protocol http --last 20 hubble observe --namespace cilium-l2l7-demo --verdict DROPPED --last 20 ``` HTTP policy rejection may appear as HTTP 403 rather than a packet DROPPED verdict. A timeout alone can also mean readiness, DNS, routing, TLS or observation problems. Correlate layers instead of interpreting every error as policy success. ## Validation Limits and Sources These are release-source/schema-checked examples with bounded local fixtures, not a production-tested platform or live-cluster benchmark. Image execution, webhook reachability, mTLS traffic, DSR return paths, NAT behavior and fragmentation still require validation in the prepared environment. Clean up only this run's labeled application resources; do not remove the cluster CNI as lab cleanup. - [Cilium kube-proxy replacement/DSR/Maglev](https://github.com/cilium/cilium/blob/v1.20.1/Documentation/network/kubernetes/kubeproxy-free.rst), [masquerading](https://github.com/cilium/cilium/blob/v1.20.1/Documentation/network/concepts/masquerading.rst), [fragment handling](https://github.com/cilium/cilium/blob/v1.20.1/Documentation/network/concepts/fragmentation.rst), [fragment map implementation](https://github.com/cilium/cilium/blob/v1.20.1/pkg/maps/fragmap/fragmap.go) - [L2 Announcements](https://github.com/cilium/cilium/blob/v1.20.1/Documentation/network/l2-announcements.rst), [multicast](https://github.com/cilium/cilium/blob/v1.20.1/Documentation/network/multicast.rst), [HTTP rule translator](https://github.com/cilium/cilium/blob/v1.20.1/pkg/envoy/policy/envoy_l7_rules_translator.go), [Envoy chart defaults](https://github.com/cilium/cilium/blob/v1.20.1/install/kubernetes/cilium/templates/_helpers.tpl) - [Cilium/Istio integration](https://github.com/cilium/cilium/blob/v1.20.1/Documentation/network/servicemesh/istio.rst), [Istio CNI/init-container modes](https://istio.io/latest/docs/setup/additional-setup/cni/), [webhook requirements](https://istio.io/latest/docs/ops/configuration/mesh/webhook/), [Istio 1.31 schemas](https://github.com/istio/istio/blob/1.31.0/manifests/charts/base/files/crd-all.gen.yaml) - [Kubernetes Service proxy modes/affinity](https://kubernetes.io/docs/reference/networking/virtual-ips/), [Cilium 1.20.1 values](https://github.com/cilium/cilium/blob/v1.20.1/install/kubernetes/cilium/values.yaml) [Return to Main Page](https://www.atomai.click/kubernetes-docs/llms/en/networking/cilium/README.md) ## Quiz [Review the L2–L7 and load-balancing questions](https://www.atomai.click/kubernetes-docs/en/quizzes/networking/cilium/05-l2-l7-networking-quiz). ---------------------------------------- Source: https://www.atomai.click/kubernetes-docs/en/networking/cilium/06-security-visibility ---------------------------------------- # Security and Visibility > **Review baseline**: Cilium 1.20.1; Cilium CLI 0.20.0; Hubble CLI 1.19.4. > **Last reviewed**: September 12, 2026. Kubernetes 1.33–1.36 is the Cilium 1.20 compatibility range; choose kubectl within the API server's supported version skew. ## Lab Environment Setup Use an existing Cilium 1.20.1 test cluster with at least two schedulable Linux nodes, working DNS, and policy enforcement enabled. Follow [the installation and platform prerequisites](https://www.atomai.click/kubernetes-docs/llms/en/networking/cilium/README.md), including the EKS restrictions and verified CLI downloads. These examples do not install or replace a CNI. They require Helm, kubectl, the Cilium and Hubble CLIs, and jq. The lab assumes CoreDNS Pods labeled `k8s-app=kube-dns` in `kube-system`. Verify the actual resolver path; NodeLocal DNS or a different distribution needs different destinations/ports. Policies, proxy configuration and platform controls already present in the cluster can affect the results. ### Hubble Installation and Setup Save `hubble-values.yaml`. This fragment enables the local servers, Relay, UI and selected metric plugins. Apply it to an **existing release at the same chart version**; review the retained installation values first. For a version upgrade, use the upgrade procedure instead of blindly reusing old values. ```yaml # hubble-values.yaml hubble: enabled: true relay: enabled: true ui: enabled: true metrics: enabled: - dns - drop - tcp - flow - httpV2 serviceMonitor: enabled: false ``` ```bash helm upgrade cilium cilium/cilium --namespace kube-system \ --version 1.20.1 --reuse-values --values hubble-values.yaml --wait cilium status --wait # Terminal 1: keep this process running; stop it with Ctrl-C. cilium hubble port-forward ``` In another terminal, verify API connectivity. Port forwarding stays local; it does not publish Relay or UI through a public LoadBalancer. ```bash # Terminal 2 hubble status hubble observe --last 20 # Optional UI; keep its local forwarding process running while using it. cilium hubble ui ``` ## Cilium's Security Features Cilium combines network policy with endpoint identities and optional encryption. Hubble makes the resulting network events observable. Their boundaries matter when evaluating a security requirement. ### Cilium Security Architecture | Responsibility | Component and scope | | --- | --- | | Network microsegmentation | Cilium L3/L4 policy selects identities, addresses, ports and directions. A namespaced CiliumNetworkPolicy selects endpoints in its namespace. | | HTTP policy | Cilium's Envoy integration filters methods, paths and headers when the proxy can see HTTP. DNS policy uses the DNS proxy. | | DNS/FQDN control | DNS rules control queries; `toFQDNs` permits destination IPs learned from observed DNS answers. This is not an automatic malicious-domain reputation feed. | | Node transport encryption | IPsec or WireGuard protects supported traffic between nodes. Coverage depends on mode and configuration. | | Network investigation | Hubble records flow metadata and policy verdicts; Relay, CLI and UI expose it. | | Process and syscall security | **Tetragon** is a separate project for runtime events and configured enforcement. Enabling Hubble does not install it. | | Threat detection and response | External alert rules, SIEM/WAF and response controllers must be configured for the intended detection and action. | ### Network and Application Security Use least-privilege policy to limit lateral movement and explicit egress to limit dependencies. Security identities derive from security-relevant labels; they are not necessarily unique per Pod and do not authenticate an end user. Current Cilium policy supports HTTP and DNS L7 rules. gRPC can use HTTP method/path/header rules where its HTTP/2 traffic is visible. Kafka topic policies are no longer supported. HTTP `headers` entries with values are literal matches, not regular-expression authentication. An `Authorization` header's presence or shape does not validate a JWT, its issuer, signature or authorization claims. Use the application's authentication layer or a configured gateway for that. An HTTP rule on port 8443 does not decrypt HTTPS. TLS termination or a separately supported inspection arrangement is required before HTTP policy can evaluate encrypted application data. Do not bypass a service mesh's encryption merely to make L7 inspection work. ### Identity, Authentication and Encryption Cilium's SPIRE-based **mutual authentication remains Beta**: it performs an out-of-band handshake for security identities. That handshake alone does not encrypt application traffic. Its documented limitations include no ClusterMesh support and no interoperability with arbitrary external mTLS systems. The separate **ztunnel workload mTLS feature is also Beta**; it has its own enrollment and certificate prerequisites. Neither feature should be presented as the automatic consequence of an identity selector. ### Encryption Configuration The following are **alternative Helm fragments**, not two settings to enable together. Select one for a planned installation/change and validate the kernel, routing and platform prerequisites. ```yaml # wireguard-values.yaml encryption: enabled: true type: wireguard nodeEncryption: false ``` ```yaml # ipsec-values.yaml encryption: enabled: true type: ipsec nodeEncryption: false ipsec: secretName: cilium-ipsec-keys ``` WireGuard requires kernel support and the node-to-node UDP path on port 51871. IPsec requires a correctly formatted, securely managed `cilium-ipsec-keys` Secret in Cilium's namespace before enabling it; follow the official key creation and rotation procedure. Merely naming a key file in a ConfigMap does not provision that key or its volume. By default, these modes protect supported Cilium-managed Pod traffic crossing nodes; same-node traffic is not encrypted by these node tunnels. Traffic to arbitrary external destinations is not automatically covered. WireGuard node-to-node encryption is a separate Beta option; control-plane nodes are excluded from that extension by default, while their Cilium-managed cross-node Pod traffic can still be encrypted. Verify the actual packet path and use application TLS where required. Host firewall compatibility also depends on the encryption mode. ## Network Visibility with Hubble Hubble receives datapath, proxy and agent events and enriches them with Kubernetes metadata. It is not simply a reader that periodically polls all eBPF maps. ```text Kernel/datapath events + proxy/agent events | v Hubble server in each Cilium agent | | | bounded flow metric endpoint optional file exporter buffer TCP 9965 | | ^ log collector/storage Relay query | scrape ^ Prometheus <--- Grafana queries | CLI / UI ``` The server maintains bounded in-memory history. Relay queries multiple servers; it is not a durable database. UI offers flow exploration and service dependency maps, while the CLI supports explicit filters. No matching records can mean no traffic, the wrong filter, unavailable peers, missing L7 visibility or overwritten/lost events. ### Hubble CLI Usage Examples ```bash hubble observe --namespace cilium-security-demo --last 100 hubble observe --from-pod cilium-security-demo/frontend \ --to-service cilium-security-demo/backend --last 100 hubble observe --namespace cilium-security-demo --protocol http \ --http-status '4+' --http-status '5+' --last 100 hubble observe --namespace cilium-deny-demo --verdict DROPPED \ --drop-reason-desc POLICY_DENIED --last 100 hubble observe --pod cilium-security-demo/frontend --follow ``` `--pod namespace/name` matches either endpoint; use `--from-pod`/`--to-pod` for direction. Pod names are not label selectors: use `--from-label` or `--to-label` when selecting labels. Do not combine `--namespace` with `--from-pod`/`--to-pod`; the CLI rejects that combination. `DROPPED` is a verdict. `POLICY_DENIED` is a drop reason, selected with `--drop-reason-desc`. HTTP status prefixes use `4+` and `5+`, not `4..` and `5..`. HTTP filters require proxy-derived L7 events; a dropped TCP connection need not have an HTTP status. ## Network Visibility and Monitoring ### Hubble Metrics The enabled plugins expose different observations: | Plugin | Example metric | Meaning and limit | | --- | --- | --- | | `flow` | `hubble_flows_processed_total` | Processed flow events by protocol/type/verdict; not unique requests or packets on every path. | | `drop` | `hubble_drop_total{reason="POLICY_DENIED"}` | Observed drops; the reason label is the enum name. | | `tcp` | `hubble_tcp_flags_total` | Observed TCP flags; not a general RTT, retransmission or concurrent-connection metric. | | `dns` | `hubble_dns_queries_total`, `hubble_dns_responses_total` | Observed DNS queries/responses and response codes; not a generic DNS latency histogram. | | `httpV2` | `hubble_http_requests_total`, `hubble_http_request_duration_seconds` | HTTP response-flow-derived request counts/status and duration in seconds. Requires HTTP visibility. | Do not enable `http` and `httpV2` together. Choose source/destination labels carefully to control cardinality, and avoid adding request headers or sensitive identities without a reason. Check `hubble_lost_events_total` and peer availability before treating a missing event as proof that traffic did not occur. ### Prometheus Integration The chart creates the headless `hubble-metrics` Service in the Cilium namespace, exposing port **9965** by default. Its Service label `k8s-app=hubble` is used for discovery; the Service selects agent Pods labeled `k8s-app=cilium`. Prometheus should discover the individual endpoints rather than rely on one static DNS target. With an existing Prometheus Operator and ServiceMonitor CRD, merge this fragment into the release values. `release: monitoring` is an example: it must match your Prometheus `serviceMonitorSelector`, and its namespace selector must include the ServiceMonitor's namespace. A resource that Prometheus does not select will not be scraped. ```yaml # hubble-servicemonitor-values.yaml hubble: metrics: serviceMonitor: enabled: true labels: release: monitoring ``` The chart's ServiceMonitor uses the named port `hubble-metrics` and the Cilium namespace's endpoints. Without the Operator, configure equivalent Kubernetes service discovery in your actual Prometheus configuration. Creating an unrelated ConfigMap does not configure Prometheus. `*.hubble-metrics.cilium.io` is used in metrics TLS identity configuration; it is not a public scrape target on port 9091. Grafana dashboards query Prometheus metrics; Hubble UI's service map queries Relay. Import dashboards that match the enabled plugins and labels. HTTP dashboards will be empty for traffic whose HTTP payload is not observable. ### Flow Export and Retention For node-local rotated files, optionally merge `hubble-export-values.yaml`. The field mask deliberately keeps network metadata; it does not export full HTTP headers. ```yaml # hubble-export-values.yaml hubble: export: static: enabled: true filePath: /var/run/cilium/hubble/events.log fileMaxSizeMb: 10 fileMaxBackups: 5 fieldMask: - time - source.namespace - source.pod_name - destination.namespace - destination.pod_name - l4 - IP - node_name - is_reply - verdict - drop_reason_desc ``` The static exporter writes on each node and rotates according to its file settings. Arrange a separate collector, access controls and storage retention if events must survive node loss. Static configuration changes require agent rollout; the dynamic exporter supports different update behavior. Exporter filters and field masks can intentionally omit events/fields, and finite buffers can still lose observations. ## Real-time Threat Detection Hubble supplies evidence for investigations; it does not include a switch that makes it a complete IDS, WAF or automatic quarantine system. There are no supported Cilium settings named `enable-threat-detection`, `enable-anomaly-detection` or `alert-to-slack`. Repeated denied destinations may suggest scanning; a traffic spike may justify investigation. These observations are not proof of an attack. Correlate them with application authentication logs, workload changes, API audit events and, where deployed, Tetragon runtime events. SQL injection, XSS and command injection require suitable application/WAF/detection rules; a normal HTTP flow record alone does not classify them. For alerts, define and test external Prometheus/SIEM rules against normal traffic, missing data and event loss. Rate limits, firewall isolation and response automation are separately configured controls. Bound the response scope and provide a recovery path; do not automatically isolate every Pod that records a dropped packet. ## Lab: Hubble Installation and Usage This lab uses two **fresh, separate namespaces** so a broad allow rule cannot invalidate the default-deny exercise. It creates no database or external API. Commands are examples for your test cluster; the documentation audit checked schemas and local fixtures, not a live deployment. ### 1. Create and Check the Workloads Save `visibility-app.yaml`. The client and server images match Cilium CLI's versioned test defaults. Backend anti-affinity requires a second schedulable node, making the frontend-to-backend path cross-node. ```yaml # visibility-app.yaml apiVersion: v1 kind: Pod metadata: name: frontend labels: app: frontend spec: automountServiceAccountToken: false containers: - name: client image: quay.io/cilium/alpine-curl:v1.10.0@sha256:913e8c9f3d960dde03882defa0edd3a919d529c2eb167caa7f54194528bde364 command: - /usr/bin/pause --- apiVersion: v1 kind: Pod metadata: name: outsider labels: app: outsider spec: automountServiceAccountToken: false containers: - name: client image: quay.io/cilium/alpine-curl:v1.10.0@sha256:913e8c9f3d960dde03882defa0edd3a919d529c2eb167caa7f54194528bde364 command: - /usr/bin/pause --- apiVersion: apps/v1 kind: Deployment metadata: name: backend spec: replicas: 1 selector: matchLabels: app: backend template: metadata: labels: app: backend spec: automountServiceAccountToken: false affinity: podAntiAffinity: requiredDuringSchedulingIgnoredDuringExecution: - labelSelector: matchLabels: app: frontend topologyKey: kubernetes.io/hostname containers: - name: http image: quay.io/cilium/json-mock:v1.4.1@sha256:6a66df90808a39c02e7a9d58af7bf0e54d8f8b7d4bc528f48c891969a7049195 ports: - containerPort: 8080 name: http readinessProbe: httpGet: path: / port: http --- apiVersion: v1 kind: Service metadata: name: backend spec: selector: app: backend ports: - name: http port: 8080 targetPort: http protocol: TCP ``` ```bash set -eu for ns in cilium-security-demo cilium-deny-demo; do kubectl create namespace "$ns" kubectl label namespace "$ns" audit-lab=security-visibility kubectl --namespace "$ns" apply -f visibility-app.yaml kubectl --namespace "$ns" wait --for=condition=Ready pod/frontend pod/outsider --timeout=120s kubectl --namespace "$ns" rollout status deployment/backend --timeout=120s for client in frontend outsider; do kubectl --namespace "$ns" exec "$client" -- \ curl --fail --silent --show-error --max-time 5 http://backend:8080/ done done ``` Both clients in both namespaces must reach the backend before applying policy. If not, resolve readiness, scheduling, DNS and network issues first. Do not infer a policy denial from an arbitrary curl error. ### 2. Apply and Observe HTTP Policy Save `backend-http.yaml`. It allows the `frontend` identity to issue `GET /` to backend TCP 8080. No broad ingress rule should overlap this example: an L4 allow can bypass the intended L7 restriction. ```yaml # backend-http.yaml apiVersion: cilium.io/v2 kind: CiliumNetworkPolicy metadata: name: backend-http namespace: cilium-security-demo spec: endpointSelector: matchLabels: app: backend ingress: - fromEndpoints: - matchLabels: k8s:io.kubernetes.pod.namespace: cilium-security-demo k8s:app: frontend toPorts: - ports: - port: '8080' protocol: TCP rules: http: - method: GET path: / ``` ```bash kubectl apply -f backend-http.yaml # After the endpoint has realized the policy: kubectl -n cilium-security-demo exec frontend -- \ curl --fail --silent --show-error --max-time 5 http://backend:8080/ # Display the HTTP response code; do not use --fail here. kubectl -n cilium-security-demo exec frontend -- \ curl --silent --show-error --max-time 5 --output /dev/null \ --write-out '%{http_code}\n' --request POST http://backend:8080/ # A separate client is not in the allowed identity selector. kubectl -n cilium-security-demo exec outsider -- \ curl --silent --show-error --max-time 5 http://backend:8080/ hubble observe --namespace cilium-security-demo --verdict DROPPED --last 100 ``` After policy realization, expect frontend `GET /` to succeed and its `POST /` to receive the proxy's HTTP 403. The outsider's new connection should be denied at L3/L4. Correlate the request time, endpoints and Hubble event; DNS errors, missing containers and unrelated HTTP errors are not a successful denial test. Use Hubble UI to inspect the generated dependency edge and drops. For a real backend that needs a database and an external API, the following **optional dependency policy** illustrates egress. It is not applied by this lab: `database` and `api.example.com` must be replaced with real dependencies. Verify DNS endpoints first. ```yaml # backend-dependencies.yaml apiVersion: cilium.io/v2 kind: CiliumNetworkPolicy metadata: name: backend-dependencies namespace: cilium-security-demo spec: endpointSelector: matchLabels: app: backend egress: - toEndpoints: - matchLabels: k8s:io.kubernetes.pod.namespace: kube-system k8s:k8s-app: kube-dns toPorts: - ports: - port: '53' protocol: UDP - port: '53' protocol: TCP rules: dns: - matchPattern: '*' - toEndpoints: - matchLabels: k8s:io.kubernetes.pod.namespace: cilium-security-demo k8s:app: database toPorts: - ports: - port: '3306' protocol: TCP - toFQDNs: - matchName: api.example.com toPorts: - ports: - port: '443' protocol: TCP ``` TCP and UDP DNS are allowed, and DNS proxy rules let Cilium observe answers used by `toFQDNs`. Permitting DNS queries is distinct from permitting subsequent connections to their resolved IPs. This policy's DNS `*` allows all query names; it is not a domain blocklist. ### 3. Verify Default Deny Separately Save `deny-except-dns.yaml`. The explicit `policyTypes` activate isolation in both directions; `ingress: []` contains **no ingress allow rules**. The single egress exception permits DNS to the matched resolver Pods only. ```yaml # deny-except-dns.yaml apiVersion: networking.k8s.io/v1 kind: NetworkPolicy metadata: name: deny-except-dns namespace: cilium-deny-demo spec: podSelector: {} policyTypes: - Ingress - Egress ingress: [] egress: - to: - namespaceSelector: matchLabels: kubernetes.io/metadata.name: kube-system podSelector: matchLabels: k8s-app: kube-dns ports: - protocol: UDP port: 53 - protocol: TCP port: 53 ``` ```bash kubectl apply -f deny-except-dns.yaml kubectl -n cilium-deny-demo exec frontend -- \ curl --silent --show-error --max-time 5 http://backend:8080/ hubble observe --namespace cilium-deny-demo --verdict DROPPED \ --drop-reason-desc POLICY_DENIED --last 100 ``` Do not replace the empty ingress list with `ingress: [{}]`: that is an allow-all ingress rule. Standard NetworkPolicy allows are additive, so another policy can open traffic. Cilium deny rules and cluster policies can impose further restrictions. This namespace exercise does not claim to isolate host-network traffic or every host-originated path. ### 4. Inspect JSON and Export a Local Summary Save the following as `flow-summary.jq`. `--output jsonpb` gives the protobuf response envelope with `.flow`; this avoids relying on the CLI's legacy `json` compatibility setting. ```text [.[] | select(.flow != null) | .flow] as $flows | { flow_records: ($flows | length), other_records: (length - ($flows | length)), policy_denied_records: ( [$flows[] | select(.verdict == "DROPPED" and .drop_reason_desc == "POLICY_DENIED")] | length ), dropped_by_reason: ( [$flows[] | select(.verdict == "DROPPED")] | group_by(.drop_reason_desc // "UNKNOWN") | map({reason: (.[0].drop_reason_desc // "UNKNOWN"), records: length}) ) } ``` ```bash set -eu hubble observe --namespace cilium-deny-demo --last 100 --output jsonpb > flows.jsonl jq --slurp --from-file flow-summary.jq flows.jsonl ``` The result counts **flow records in this finite sample**, not unique attacks, connections or all cluster packets. It separately counts non-flow records; inspect them for loss/status information. For a live local filter: ```bash hubble observe --namespace cilium-deny-demo --follow --output jsonpb | jq --unbuffered -c 'select(.flow.verdict == "DROPPED" and .flow.drop_reason_desc == "POLICY_DENIED")' ``` This pipeline prints locally. Notifications require a separately configured integration, credentials, retry/deduplication policy and handling of stream failures. ### 5. Clean Up the Test Namespaces After reviewing the namespace names, remove only this lab's workloads and policies. The ownership check stops on lookup failure or a different label. ```bash set -eu for ns in cilium-security-demo cilium-deny-demo; do LAB_OWNER=$(kubectl get namespace "$ns" -o jsonpath='{.metadata.labels.audit-lab}') test "$LAB_OWNER" = security-visibility kubectl delete namespace "$ns" done ``` ## Primary References - [Cilium policy enforcement](https://raw.githubusercontent.com/cilium/cilium/v1.20.1/Documentation/security/policy/intro.rst) - [Kubernetes NetworkPolicy](https://kubernetes.io/docs/concepts/services-networking/network-policies/) - [HTTP policy](https://raw.githubusercontent.com/cilium/cilium/v1.20.1/Documentation/security/policy/layer7.rst) - [DNS policy](https://raw.githubusercontent.com/cilium/cilium/v1.20.1/Documentation/security/dns.rst) - [Hubble setup](https://raw.githubusercontent.com/cilium/cilium/v1.20.1/Documentation/observability/hubble/setup.rst) - [Metrics](https://raw.githubusercontent.com/cilium/cilium/v1.20.1/Documentation/observability/metrics.rst) - [Flow exporter](https://raw.githubusercontent.com/cilium/cilium/v1.20.1/Documentation/observability/hubble/configuration/export.rst) - [WireGuard](https://raw.githubusercontent.com/cilium/cilium/v1.20.1/Documentation/security/network/encryption-wireguard.rst) - [IPsec](https://raw.githubusercontent.com/cilium/cilium/v1.20.1/Documentation/security/network/encryption-ipsec.rst) - [Mutual authentication](https://raw.githubusercontent.com/cilium/cilium/v1.20.1/Documentation/network/servicemesh/mutual-authentication/mutual-authentication.rst) - [ztunnel](https://raw.githubusercontent.com/cilium/cilium/v1.20.1/Documentation/security/network/encryption-ztunnel.rst) - [Tetragon](https://raw.githubusercontent.com/cilium/tetragon/main/README.md) - [Hubble CLI filters](https://raw.githubusercontent.com/cilium/hubble/v1.19.4/vendor/github.com/cilium/cilium/hubble/cmd/observe/flows.go) [Return to Main Page](https://www.atomai.click/kubernetes-docs/llms/en/networking/cilium/README.md) ## Quiz Test the policy, encryption and observability boundaries in the [topic quiz](https://www.atomai.click/kubernetes-docs/en/quizzes/networking/cilium/06-security-visibility-quiz). ---------------------------------------- Source: https://www.atomai.click/kubernetes-docs/en/networking/cilium/07-advanced-topics ---------------------------------------- # Advanced Topics and Real-World Cases > **Review baseline**: Cilium 1.20.1, Cilium CLI 0.20.0 and Hubble CLI 1.19.4. > **Last reviewed**: September 12, 2026. Cilium 1.20's Kubernetes compatibility range is 1.33–1.36; historical measurements below retain their original environment. ## Lab Environment Setup Use [the installation prerequisites](https://www.atomai.click/kubernetes-docs/llms/en/networking/cilium/README.md) and a disposable test cluster with at least two schedulable Linux nodes. Keep the OS, kernel, Cilium settings, topology, MTU, policies, encryption and test workload in the result record. Use kubectl within its supported API-server version skew. Helm, jq and the Cilium/Hubble CLIs are needed for the commands below; `kubectl top` additionally needs a metrics API. ### Performance Testing Environment Setup Record the existing deployment before changing it: ```bash cilium version cilium status --verbose kubectl version kubectl get nodes -o wide kubectl -n kube-system get pods -l k8s-app=cilium -o wide helm get values cilium --namespace kube-system -o yaml > cilium-current-values.yaml ``` The current CLI can create its own matching test workloads. Run this only when generating network load and creating test resources are acceptable: ```bash cilium connectivity perf --test-namespace cilium-advanced-perf \ --namespace-labels docs-audit-lab=cilium-advanced-07 \ --duration 10s --samples 2 --crr --udp \ --host-net=false --pod-net=true --same-node=true --other-node=true \ --report-dir ./cilium-advanced-perf-results ``` The default single-concurrency run uses **`cilium-advanced-perf-1`**, even though the argument lacks that suffix. Ensure that namespace is unused before the test. `--duration` applies to each scenario/sample, not to the complete run; scheduling, setup and the combination of cases add time. The command is an active workload test, not a read-only diagnostic. Compare same-node and cross-node results, TCP request/response and UDP, and the actual CPU/memory/packet-loss behavior. Repeat with one planned change at a time. A successful run does not prove application SLOs, a production capacity limit or every policy path. The audit validated these command contracts without running the test or provisioning a cluster. ## Performance Tuning and Troubleshooting ### Performance Tuning Architecture ![Four areas to investigate for Cilium performance: kernel behavior, eBPF maps, resource allocation and the selected networking path.](https://raw.githubusercontent.com/Atom-oh/kubernetes-docs/main/en/.gitbook/assets/en-networking-cilium-07-advanced-topics-0.png) [View interactive diagram](https://www.atomai.click/kubernetes-docs/archmaps/en-networking-cilium-07-advanced-topics-0.html) The figure groups investigation areas; it does not prescribe increasing every setting or disabling security controls. ### Performance Tuning Areas | Area | What to measure and what the setting controls | | --- | --- | | Socket backlogs | `net.core.somaxconn` limits the socket listen backlog; `net.ipv4.tcp_max_syn_backlog` concerns pending SYN_RECV requests per listener. Check the server's accept behavior and the relevant network namespace. | | Neighbor entries | `net.ipv4.neigh.default.gc_thresh1`, `gc_thresh2` and `gc_thresh3` are distinct garbage-collection thresholds. There is no single `gc_thresh` setting. | | Connection tracking | Netfilter's `nf_conntrack_max` and Cilium's BPF CT maps are separate mechanisms. Increasing the former does not resize the latter. | | BPF maps | Inspect map pressure, insertion failures, churn and memory. CT, NAT, LB and per-endpoint policy maps have different scopes and sizing rules. | | CPU and memory | Measure agent, operator, Envoy and Hubble separately. Requests affect scheduling; CPU limits can throttle and memory limits can cause OOM termination. Raising all limits is not a diagnosis. | | Network path | Identify native/tunnel routing, MTU, masquerading, encryption and service forwarding. XDP acceleration applies to supported paths and drivers; it is not required for kube-proxy replacement. | Keep security and routing requirements constant when comparing performance. A faster result obtained by removing required encryption or policy is not an equivalent configuration. ### Map Sizing Save the following only as a **candidate change** after measuring pressure. Merge it into the complete, versioned release values; do not replace the whole Cilium ConfigMap with a small snippet. ```yaml # map-sizing-values.yaml bpf: mapDynamicSizeRatio: 0.005 ``` `0.005` is a nominal **0.5% of node memory** input to dynamic sizing, not 5% and not a hard limit on all Cilium memory. For example, 32 GiB × 0.005 is 163.84 MiB before map limits, rounding and other allocations. The affected large maps include CT, NAT, neighbor and socket reverse-NAT maps; other maps and userspace memory remain separate. Explicit `bpf.ctTcpMax`, `bpf.ctAnyMax` or `bpf.natMax` values override dynamic sizing for those maps. Keep NAT capacity compatible with CT capacity and verify the resolved startup sizes. Increasing maps can consume more memory; replacing maps can disrupt existing connections. Distributed LRU (`bpf.distributedLRU.enabled`) uses per-CPU pools to reduce contention, with memory/eviction tradeoffs. It requires dynamic sizing and recreates maps when enabled. The official high-performance profile is not a universal live toggle: introduce such datapath changes on prepared new nodes or use the documented migration procedure. The old `proxy-max-memory-percentage`, `proxy-max-threads`, `enable-xdp`, `tunnel: disabled` and `kube-proxy-replacement: strict` examples are not current configuration recipes. Use supported chart settings such as `envoy.resources`, `routingMode`, boolean `kubeProxyReplacement` and `loadBalancer.acceleration`, with their prerequisites. ### Hubble Cost and Event Loss A larger event queue can absorb a burst, but it uses memory and does not solve a sustained processing deficit or reduce CPU consumption: ```yaml # hubble-queue-values.yaml hubble: eventQueueSize: 32768 ``` If repeated trace events dominate processing, separately evaluate a longer aggregation interval: ```yaml # hubble-aggregation-values.yaml bpf: monitorAggregation: medium monitorInterval: 10s ``` The actual chart field is **`bpf.monitorInterval`**, not `bpf.events.monitorInterval`. Inspect the rendered `monitor-aggregation-interval` before deployment. Increased aggregation, event-rate limits and disabled event classes all reduce observations available to monitor, Hubble metrics and export. A lost Hubble event is not itself a dropped application packet; check both observability loss and datapath drops. ### Advanced Datapath Prerequisites | Feature | Prerequisites and limits to check | | --- | --- | | netkit | Beta in this baseline; Linux 6.8+ and BPF host routing. Existing veth Pods cannot simply switch device type on agent restart. | | BIG TCP | Family-specific kernel/NIC requirements; the combined tuning profile requires Linux 6.8+ and supported NICs. It is not a generic MTU increase. | | BPF host routing | Requires compatible kube-proxy replacement and BPF masquerading. Bypasses host netfilter hooks; check Istio and other integrations that rely on those hooks. | | XDP service acceleration | Native XDP-capable devices and a supported external service-forwarding path. Use the driver/platform guidance and verify the running status. | | Bandwidth Manager | Per-Pod egress uses EDT; ingress uses an eBPF token bucket. `10M` in the bandwidth annotation means 10 Mbit/s, not 10 MB/s. | | BBR for Pods | Bandwidth Manager, Linux 5.18+ and BPF host routing; newly created Pods adopt the setting. Host-only BBR is a separate option. | Bandwidth enforcement has documented limitations with egress L7 Cilium policies and nested network namespaces such as kind. These conditions differ from ordinary Cilium installation requirements. Do not use one tuning profile as evidence that all service-mesh, cloud or kernel combinations work. ### Targeted Troubleshooting Commands Use the standalone `cilium` CLI for cluster operations and **`cilium-dbg` inside the relevant agent** for local endpoints, policies and maps. Select the node being investigated instead of arbitrarily querying the first agent: ```bash set -euo pipefail : "${NODE_NAME:?Set NODE_NAME to the node being investigated}" CILIUM_POD=$(kubectl -n kube-system get pods -l k8s-app=cilium \ --field-selector "spec.nodeName=$NODE_NAME,status.phase=Running" -o json | jq -er 'if (.items | length) == 1 then .items[0].metadata.name else error("expected exactly one running Cilium Pod on the selected node") end') kubectl -n kube-system exec "$CILIUM_POD" -c cilium-agent -- cilium-dbg status --verbose kubectl -n kube-system exec "$CILIUM_POD" -c cilium-agent -- cilium-dbg endpoint list kubectl -n kube-system exec "$CILIUM_POD" -c cilium-agent -- cilium-dbg policy get kubectl -n kube-system exec "$CILIUM_POD" -c cilium-agent -- cilium-dbg policy selectors kubectl -n kube-system exec "$CILIUM_POD" -c cilium-agent -- cilium-dbg map list kubectl -n kube-system exec "$CILIUM_POD" -c cilium-agent -- cilium-dbg bpf metrics list kubectl -n kube-system logs "$CILIUM_POD" -c cilium-agent --since=10m --tail=200 ``` Endpoint IDs are local to that agent. Use an ID from its endpoint list: ```bash : "${CILIUM_POD:?Select the owning Cilium Pod first}" : "${ENDPOINT_ID:?Read the endpoint ID from the selected agent endpoint list}" kubectl -n kube-system exec "$CILIUM_POD" -c cilium-agent -- \ cilium-dbg endpoint get "$ENDPOINT_ID" # Stream local BPF drop events; stop with Ctrl-C. kubectl -n kube-system exec "$CILIUM_POD" -c cilium-agent -- \ cilium-dbg monitor --type drop ``` `cilium-dbg map list` lists open maps known to the agent's map manager; it is not an inventory of every kernel BPF map. `cilium-dbg monitor` displays emitted BPF events and optional captured traces, not a lossless tcpdump of every packet. Large CT-map dumps can be expensive; inspect metrics and the affected node before running `cilium-dbg bpf ct list global`. For Hubble, first establish the local Relay connection described in [Security and Visibility](https://www.atomai.click/kubernetes-docs/llms/en/networking/cilium/06-security-visibility.md): ```bash hubble status hubble observe --protocol tcp --verdict DROPPED --since 1h hubble observe --protocol dns --from-label k8s:app=frontend --last 100 hubble observe --http-status '5+' --from-namespace production --last 100 ``` `--type` selects event types, not DNS record type A. Use the DNS flow fields if you need to distinguish query types. HTTP status filters use `5+`, not `5xx`. Retained history and event availability bound `--since 1h`; it cannot recover an hour of overwritten events. ### Common Troubleshooting Scenarios | Symptom | Evidence to gather | Next decision | | --- | --- | --- | | CT/NAT pressure | Map pressure, insertion/drop reasons, connection churn, actual configured sizes | Investigate churn/timeouts and memory headroom before resizing. | | OOM or CPU saturation | Container termination reason, memory/CPU history, throttling, proxy and Hubble load | Identify the responsible component and change its budget or workload. | | Unexpected policy result | Correct namespace/labels, endpoint policy revision, proxy errors and Hubble verdict | Check direction, additive allow rules and deny precedence; ordinary allows are not a priority-ordered firewall list. | | Cross-node failure | DNS, node/pod routes, MTU, tunnel/encryption ports and platform firewall | Verify forward and return paths; a BGP session alone does not prove datapath reachability. | | Upgrade regression | Old/new values, version notes, all component versions, proxy reconnections | Use the prepared supported rollback path and investigate feature compatibility. | ## Large-Scale Deployment Strategies Capacity planning must include node/Pod density, Services and backends, identities, policy expansion, API watch traffic, IPAM allocation and flow volume. A policy's object count alone does not determine its per-endpoint map cost. ### Large-Scale Deployment Architecture ```text Management / GitOps / shared monitoring | config and collected telemetry +-----------------------+ v v Workload cluster A Workload cluster B - Cilium Operator - Cilium Operator - agent on each node - agent on each node - local Hubble servers - local Hubble servers - Relay/export setup - Relay/export setup | | +--- optional ClusterMesh metadata/data paths ``` A central management cluster does not replace the operator in every managed cluster. Size each cluster's operator and agents; the chart's operator replicas and anti-affinity need enough eligible nodes. Aggregate metrics/logs through configured collectors. ClusterMesh requires its own addressing, identity, trust and reachability design; it does not automatically replicate every policy resource. `ciliumEndpointSlice.enabled` is an opt-in Cilium feature, distinct from Kubernetes EndpointSlice. It is not the old `enable-endpoint-slice` ConfigMap flag. Evaluate version and feature compatibility before enabling it; Egress Gateway currently cannot be combined with CiliumEndpointSlice or ClusterMesh. Egress Gateway **SNATs** selected traffic to a predictable gateway address; it does not preserve the original Pod source address. Gateway interfaces/IPs and routing must already be provisioned, including platform-specific requirements on AWS. It needs BPF masquerading, kube-proxy replacement and CRD identity allocation. Newly started Pods can briefly send traffic before the egress policy takes effect; do not treat it as an immediate fail-closed source-IP guarantee. ### Rollout and Recovery Keep desired values, policies, address-pool definitions and necessary trust/key material under appropriate versioning and backup controls. Rehearse recovery; a ConfigMap backup is not a complete IPAM or cryptographic recovery plan. For minor upgrades, first reach the current minor's latest patch, run the required preflight, then move **one minor at a time**. Preserve initial `upgradeCompatibility` as instructed by the upgrade guide and migrate renamed/removed values. Do not use `--reuse-values` across minor versions. Agents, operator and other Cilium components should converge on the same version. Traffic through userspace proxies can reconnect during upgrades; buffered monitoring events can be lost. New features/resources may need removal or migration before a rollback is valid. A generic application blue/green deployment or Helm rollback is not a guarantee of a reversible, zero-downtime CNI migration. ## Real-World Use Case Studies ### A Documented Historical Scalability Experiment The official scalability report describes **1,000 worker nodes**, three controller nodes and kernel **5.4.0-1009-gcp** on Google Cloud. Its setup does not identify a Cilium version, so these results must not be relabeled as a Cilium 1.20.1 benchmark. The report discusses resource consumption and convergence under its own workload; it is not a current support matrix or a capacity promise. Its health-check changes and large rollout concurrency were experiment choices. Preserve the actual test conditions when citing it and validate an operational configuration separately. ### Design Scenario 1: Large-Scale E-commerce For many services and high request volume, evaluate eBPF service forwarding, identity/L7 policy, Hubble and optional ClusterMesh. Measure p95/p99 latency, throughput, errors, CPU per request and policy convergence under the same topology and protection requirements. No named implementation or reproducible measurements support a universal percentage improvement here. ### Design Scenario 2: Financial Services Combine least-privilege policy, appropriately scoped transport/application encryption and controlled flow export with application/API audit records. Test key rotation, event loss, retention and cross-cluster trust. Hubble flows alone do not constitute complete regulatory audit evidence or guarantee a shorter audit. ### Design Scenario 3: Telecommunications and Edge Assess NIC/driver support, CPU scheduling, service-forwarding paths, packet sizes, loss and latency under realistic traffic. XDP may benefit a qualifying forwarding path. It does not by itself implement a 5G user-plane function or prove a fixed packets-per-second rate on arbitrary hardware. Remote sites still require an underlay and explicit failure/recovery testing. ## Future Roadmap and Development Direction The community roadmap explicitly makes **no date commitments**. Track release notes, accepted designs and issues for a specific capability rather than treating a list of desired integrations as a delivery promise. | Area | Questions to investigate | | --- | --- | | eBPF and kernels | Which kernel feature, backport, NIC and architecture does the proposed path require? CO-RE does not supply missing kernel capabilities. | | Networking and IPv6 | Which IPAM, routing, policy and external-integration combinations are supported by the selected release? | | Security and observability | Is this Cilium network policy, Beta workload authentication/encryption, Tetragon runtime enforcement, or an external detector/storage system? | | Cloud, mesh and serverless | Does the managed platform permit the intended CNI/host hooks? Are mesh interception and authentication preserved? | | Edge, IoT, 5G and AI/ML | What extra device, transport, runtime or accelerator integration is required? A Kubernetes CNI does not establish all application-specific capabilities. | Participate through project issues, design proposals, documentation and community discussions. Commercial support and managed distributions have their own feature/support contracts. ## Current BGP Configuration The former `CiliumBGPPeeringPolicy` example is obsolete for this baseline. Current configuration separates cluster/node selection, peer settings and advertised prefixes into three **`cilium.io/v2`** resources. This is a configuration model for an isolated routing lab, not a complete cloud-router setup. It assumes cluster-pool or Kubernetes host-scope IPAM for `PodCIDR` advertisements, a reachable peer router, the corresponding router configuration, and nodes intentionally labeled `cilium-bgp=lab`. `192.0.2.1` is a documentation address; replace it with the real peer IP, without `/32`. Multi-pool IPAM uses a different advertisement type and pool selection. ```yaml # bgp-values.yaml bgpControlPlane: enabled: true ``` ```yaml # bgp-lab.yaml apiVersion: cilium.io/v2 kind: CiliumBGPClusterConfig metadata: name: lab-bgp spec: nodeSelector: matchLabels: cilium-bgp: lab bgpInstances: - name: asn-64512 localASN: 64512 peers: - name: router-64513 peerASN: 64513 peerAddress: 192.0.2.1 peerConfigRef: name: lab-peer --- apiVersion: cilium.io/v2 kind: CiliumBGPPeerConfig metadata: name: lab-peer spec: timers: connectRetryTimeSeconds: 120 holdTimeSeconds: 90 keepAliveTimeSeconds: 30 gracefulRestart: enabled: true restartTimeSeconds: 120 families: - afi: ipv4 safi: unicast advertisements: matchLabels: advertise: lab --- apiVersion: cilium.io/v2 kind: CiliumBGPAdvertisement metadata: name: lab-pod-cidrs labels: advertise: lab spec: advertisements: - advertisementType: PodCIDR ``` The peer's advertisement selector matches `advertise: lab`, and `peerConfigRef` resolves to `lab-peer`. Graceful Restart requires compatible peer behavior and appropriate timers; it cannot retain a failed datapath or guarantee application availability. BGP advertises reachability but does **not install the local datapath routes** or create DNS records. After preparing and applying an appropriate configuration in your routing lab, inspect both Cilium's state and the external router's received routes: ```bash kubectl get ciliumbgpclusterconfigs,ciliumbgppeerconfigs,ciliumbgpadvertisements cilium bgp peers cilium bgp routes advertised ipv4 unicast ``` Test actual forward/return traffic separately. Do not describe long-existing HTTP policy or arbitrary percentage gains as new features of an old release. Consult the exact release notes when migrating from Cilium 1.18 or another earlier minor. ## Next Steps Use the [IPAM and policy](https://www.atomai.click/kubernetes-docs/llms/en/networking/cilium/04-ipam-policy.md), [L2–L7 networking](https://www.atomai.click/kubernetes-docs/llms/en/networking/cilium/05-l2-l7-networking.md) and [security/visibility](https://www.atomai.click/kubernetes-docs/llms/en/networking/cilium/06-security-visibility.md) chapters to validate individual paths. Keep benchmark conditions and limitations with each result, then rehearse a staged rollout and recovery. After reviewing the exact namespace created by the performance test, remove its resources: ```bash set -eu PERF_NS=cilium-advanced-perf-1 LAB_OWNER=$(kubectl get namespace "$PERF_NS" -o jsonpath='{.metadata.labels.docs-audit-lab}') test "$LAB_OWNER" = cilium-advanced-07 kubectl delete namespace "$PERF_NS" ``` ## Primary References - [Tuning guide](https://raw.githubusercontent.com/cilium/cilium/v1.20.1/Documentation/operations/performance/tuning.rst) - [Chart values](https://raw.githubusercontent.com/cilium/cilium/v1.20.1/install/kubernetes/cilium/values.yaml) - [Chart ConfigMap template](https://raw.githubusercontent.com/cilium/cilium/v1.20.1/install/kubernetes/cilium/templates/cilium-configmap.yaml) - [Map sizing](https://raw.githubusercontent.com/cilium/cilium/v1.20.1/Documentation/network/ebpf/maps.rst) - [Map sizing implementation](https://raw.githubusercontent.com/cilium/cilium/v1.20.1/pkg/option/config.go) - [Upgrade guide](https://raw.githubusercontent.com/cilium/cilium/v1.20.1/Documentation/operations/upgrade.rst) - [Upgrade limitations](https://raw.githubusercontent.com/cilium/cilium/v1.20.1/Documentation/operations/upgrade-warning.rst) - [BGP configuration](https://raw.githubusercontent.com/cilium/cilium/v1.20.1/Documentation/network/bgp-control-plane/bgp-control-plane-configuration.rst) - [BGP operation](https://raw.githubusercontent.com/cilium/cilium/v1.20.1/Documentation/network/bgp-control-plane/bgp-control-plane-operation.rst) - [Bandwidth Manager](https://raw.githubusercontent.com/cilium/cilium/v1.20.1/Documentation/network/kubernetes/bandwidth-manager.rst) - [Egress Gateway](https://raw.githubusercontent.com/cilium/cilium/v1.20.1/Documentation/network/egress-gateway/egress-gateway.rst) - [Historical scalability report](https://raw.githubusercontent.com/cilium/cilium/v1.20.1/Documentation/operations/performance/scalability/report.rst) - [Community roadmap](https://raw.githubusercontent.com/cilium/cilium/v1.20.1/Documentation/community/roadmap.rst) - [Linux IP sysctls](https://www.kernel.org/doc/html/latest/networking/ip-sysctl.html) [Return to Main Page](https://www.atomai.click/kubernetes-docs/llms/en/networking/cilium/README.md) ## Quiz Review the operational boundaries and diagnostic commands in the [topic quiz](https://www.atomai.click/kubernetes-docs/en/quizzes/networking/cilium/07-advanced-topics-quiz). ---------------------------------------- Source: https://www.atomai.click/kubernetes-docs/en/networking/cilium/networking-concepts ---------------------------------------- # Deep Dive into Networking Concepts > **Review baseline**: Cilium 1.20.1. > **Last reviewed**: September 12, 2026. This document provides in-depth explanations of core networking concepts needed to understand Cilium. It explores container networking, overlays, NAT, routing, DNS, load balancing and policy. Examples are conceptual or partial Helm/API configurations for prepared test environments, not complete installation or migration recipes. Verify platform and version prerequisites in [the Cilium overview](https://www.atomai.click/kubernetes-docs/llms/en/networking/cilium/README.md); managed platforms do not all permit the same CNI features. ## Learning Objectives Through this document, you will understand: - The basic structure of the OSI model and TCP/IP stack and the role of each layer - Basic principles and implementation methods of container networking - Differences between overlay networks and underlay networks - How core networking concepts such as NAT, routing, and DNS are utilized in Cilium ## Table of Contents 1. [OSI Model and TCP/IP Stack](#osi-model-and-tcp-ip-stack) 2. [Container Networking Basics](#container-networking-basics) 3. [Overlay Networks](#overlay-networks) 4. [Network Address Translation (NAT)](#network-address-translation-nat) 5. [Routing Protocols](#routing-protocols) 6. [DNS and Service Discovery](#dns-and-service-discovery) 7. [Load Balancing Concepts](#load-balancing-concepts) 8. [Network Security Basics](#network-security-basics) ## OSI Model and TCP/IP Stack > **Key Concept**: The OSI model is a conceptual framework that classifies network communication into 7 abstract layers, making complex networking processes easier to understand. The OSI (Open Systems Interconnection) model is a conceptual framework that classifies network communication into 7 abstract layers. Each layer is responsible for specific networking functions, allowing complex networking processes to be broken down for easier understanding. ### OSI Model and TCP/IP Model Comparison ![Diagram mapping the seven OSI reference layers to the four TCP/IP stack layers, with representative protocols shown under each OSI layer.](https://raw.githubusercontent.com/Atom-oh/kubernetes-docs/main/en/.gitbook/assets/en-networking-cilium-networking-concepts-0.png) [🔍 View interactive diagram](https://www.atomai.click/kubernetes-docs/archmaps/en-networking-cilium-networking-concepts-0.html) The mapping is a teaching approximation, not a protocol implementation specification. SSL is a legacy label in the figure; use supported TLS versions for current systems. ### OSI 7-Layer Model 1. **Physical Layer** - Converts bit streams into electrical, optical, or wireless signals - Includes cables, transceivers and physical signaling; a switch also implements functions at higher layers - Data unit: Bit 2. **Data Link Layer** - Responsible for data transfer between nodes on a physical network - Device identification using MAC (Media Access Control) addresses - Error detection and, where the link protocol provides it, recovery; Ethernet error detection does not itself correct damaged frames - Data unit: Frame - Ethernet and Wi-Fi protocols operate at this layer 3. **Network Layer** - Responsible for packet routing between different networks - Logical addressing (IP addresses) - Path determination and packet forwarding - Data unit: Packet - IP (Internet Protocol) is the core protocol of this layer 4. **Transport Layer** - End-to-end communication control - Data segmentation and reassembly - TCP provides flow control and retransmission; UDP does not provide those guarantees - Data unit: TCP segment or UDP datagram - TCP (Transmission Control Protocol) and UDP (User Datagram Protocol) are the main protocols of this layer 5. **Session Layer** - Establishment, maintenance, and termination of communication sessions - Synchronization and dialog control - Checkpoint setting and recovery - Session management can be discussed at this layer; real RPC implementations do not necessarily map to a single OSI layer 6. **Presentation Layer** - Data format conversion and encryption - Character encoding, data compression, encryption/decryption - Encoding/compression illustrate this responsibility; TLS is an Internet protocol, not a literal OSI presentation-layer implementation 7. **Application Layer** - Provides network services used by applications, not necessarily a graphical user interface - Services such as email, file transfer, web browsing - HTTP, FTP, SMTP, DNS are examples of this layer ### Relationship Between Cilium and OSI Model Cilium operates at multiple OSI layers: | OSI Layer | Cilium Feature | Example | |-----------|----------------|---------| | L2 (Data Link) | Link-level reachability; optional L2 Announcements | ARP/NDP responses for configured Service VIPs | | L3 (Network) | IP routing, CIDR-based policy | IP routing between pods | | L4 (Transport) | Port-based filtering, connection tracking | Service port access control | | L7 (Application) | Supported HTTP/gRPC and DNS proxy rules | HTTP path policy or DNS query policy | L2 Announcements remains Beta in this baseline and requires its controller/device configuration. It is distinct from a general MAC-address security-policy interface. ### TCP/IP Stack The TCP/IP stack is a set of protocols that form the foundation of the Internet, an architecture often described with four layers and compared with OSI; it is not a direct implementation of the seven-layer model. 1. **Network Interface Layer** - Corresponds to the Physical and Data Link layers of the OSI model - Responsible for interface with physical network media - Includes protocols like Ethernet and Wi-Fi 2. **Internet Layer** - Corresponds to the Network layer of the OSI model - Packet routing using IP (Internet Protocol) - Includes ICMP (Internet Control Message Protocol); ARP resolves IPv4 next-hop link-layer addresses at the link boundary, while IPv6 uses Neighbor Discovery 3. **Transport Layer** - Same as the Transport layer of the OSI model - Includes TCP and UDP protocols - Provides connection-oriented (TCP) and connectionless (UDP) communication 4. **Application Layer** - Integrates the Session, Presentation, and Application layers of the OSI model - Includes protocols like HTTP, SMTP, FTP, DNS - Provides interface between user applications and the network ### Cilium Features by Layer Cilium provides features at various network layers: - **L2 (Data Link Layer)**: Link reachability and optional L2 service announcements; this is not a general MAC-address NetworkPolicy API or universal ARP-spoofing protection - **L3 (Network Layer)**: IP address-based routing and filtering, IPAM - **L4 (Transport Layer)**: Port-based filtering, load balancing, connection tracking - **L7 (Application Layer)**: Configured HTTP/gRPC proxy functions and DNS policy; the former Kafka L7 policy API is removed ## Container Networking Basics Container networking is a mechanism that allows containerized applications to communicate with each other and with the outside world. Container orchestration platforms like Kubernetes use various networking models and solutions. ### Container Network Interface (CNI) CNI (Container Network Interface) defines a standard interface between container runtimes and network plugins. Current Kubernetes uses a CRI container runtime to invoke CNI plugins. This interface allows multiple networking implementations; the CNI specification does not require every plugin to implement Kubernetes NetworkPolicy. #### Key Components of CNI: 1. **Plugins**: Executables responsible for creating and configuring network interfaces 2. **Configuration Files**: JSON format files that define plugin behavior 3. **IPAM (IP Address Management)**: Module responsible for IP address allocation and management #### Main Responsibilities of CNI Plugins: - Adding/removing interfaces to/from container network namespaces - Allocating and releasing IP addresses - Configuring routing tables - A networking implementation may separately provide policy controllers/datapath enforcement; policy is not a mandatory CNI execution operation ### Container Networking Models There are several container networking models, each suitable for different use cases and requirements. #### 1. Bridge Networking - Creates a virtual bridge on the host to connect containers - Each container connects to the bridge through virtual ethernet (veth) pairs - Efficient communication between containers on the same host - The default bridge is an example from standalone Linux Docker; it is not the Kubernetes or Cilium networking model ![Diagram showing two containers each connected through a veth pair to the docker0 Linux bridge on the Docker host, which forwards traffic onto the host network via eth0.](https://raw.githubusercontent.com/Atom-oh/kubernetes-docs/main/en/.gitbook/assets/en-networking-cilium-networking-concepts-1.png) [🔍 View interactive diagram](https://www.atomai.click/kubernetes-docs/archmaps/en-networking-cilium-networking-concepts-1.html) This is a Linux Docker bridge example with illustrative addresses. The host's routing/NAT path is simplified; it is not Cilium's default bridge topology. #### 2. Host Networking - Container directly uses the host's network namespace - No separate network isolation - Avoids a separate container network namespace; performance still depends on the actual workload and path - Potential for port conflicts ![Diagram showing two containers inside one host sharing the host network stack (eth0, 192.168.1.10) directly, with no separate network namespace or isolation layer between them.](https://raw.githubusercontent.com/Atom-oh/kubernetes-docs/main/en/.gitbook/assets/en-networking-cilium-networking-concepts-2.png) [🔍 View interactive diagram](https://www.atomai.click/kubernetes-docs/archmaps/en-networking-cilium-networking-concepts-2.html) “No isolation” here means sharing the network namespace. It does not mean that every process, filesystem or other container isolation boundary is removed. #### 3. Overlay Networking - Supports communication between containers across multiple hosts - Uses encapsulation protocols like VXLAN and GENEVE - Suitable for large-scale clusters - Supported by Cilium, Calico, Flannel, etc. ![Diagram showing two hosts, each running a container on the 10.0.0.0/24 overlay network, with packets encapsulated at Host A's eth0, carried over a VXLAN tunnel across the physical network, and decapsulated at Host B's eth0 for delivery to the peer container.](https://raw.githubusercontent.com/Atom-oh/kubernetes-docs/main/en/.gitbook/assets/en-networking-cilium-networking-concepts-3.png) [🔍 View interactive diagram](https://www.atomai.click/kubernetes-docs/archmaps/en-networking-cilium-networking-concepts-3.html) This generic VXLAN illustration uses standard UDP 4789 and a shared L2 subnet. Cilium's default VXLAN port is 8472; its Pod CIDR allocations must follow the selected IPAM mode rather than copying this drawing. #### 4. Underlay Networking (Direct Routing) - Directly utilizes physical network infrastructure - No encapsulation overhead - Requires control over network infrastructure - Can integrate with routing protocols like BGP ![Diagram showing two hosts, each routing container traffic through a local routing table directly onto the physical network with no encapsulation.](https://raw.githubusercontent.com/Atom-oh/kubernetes-docs/main/en/.gitbook/assets/en-networking-cilium-networking-concepts-4.png) [🔍 View interactive diagram](https://www.atomai.click/kubernetes-docs/archmaps/en-networking-cilium-networking-concepts-4.html) The entries illustrate host routes. An actual deployment still needs reachable next hops and valid forward/return routes for its Pod addresses. ### Kubernetes Networking Model The Kubernetes model provides direct Pod connectivity **barring intentional network segmentation**: 1. Pods can communicate with other Pods without a mandatory proxy or NAT in the Pod network. 2. Node agents must be able to communicate with Pods **on that node**. This is not a blanket requirement that every host reach every Pod. 3. External connectivity follows the cluster's routing and security policy; unrestricted internet access is not required. NetworkPolicy enforcement depends on a capable network implementation. The API can exist even when the installed plugin does not enforce it. #### Kubernetes Network Components: 1. **Pod Network**: Network connecting all pods in the cluster 2. **Service Network**: Provides stable endpoints for sets of pods 3. **Cluster DNS**: DNS service for service discovery 4. **Ingress/Egress**: Manages communication with outside the cluster ### Cilium's Container Networking Approach Cilium leverages eBPF to provide a high-performance, scalable container networking solution: 1. **eBPF-based Data Path**: Direct packet processing within the kernel 2. **Support for Various Networking Modes**: Overlay (VXLAN, Geneve) and native routing; the generic Helm default is tunnel mode with VXLAN, subject to platform overrides 3. **Advanced Load Balancing**: kube-proxy replacement functionality 4. **Network Policies**: Granular policies at L3-L7 levels 5. **Integrated IPAM**: Support for various IP address allocation strategies A generic Helm installation without platform overrides defaults to cluster-pool IPAM: the operator allocates node CIDRs and agents allocate Pod IPs from their node's pool. `ipam.mode: kubernetes` uses the Node's `spec.podCIDR`/`spec.podCIDRs`. ENI mode uses EC2 interfaces and VPC addresses; it is not a universal recommendation for every EKS compute mode. See [IPAM and policies](https://www.atomai.click/kubernetes-docs/llms/en/networking/cilium/04-ipam-policy.md). ## Overlay Networks Overlay networks are a technology that builds a virtual network layer on top of existing network infrastructure. This technology allows virtual network topologies to be created independently of physical network topology. In container environments, it is widely used to enable communication between containers across multiple hosts. ### How Overlay Networks Work Overlay networks work using encapsulation technology. Original packets are encapsulated inside other packets and transmitted through the physical network. 1. **Packet Encapsulation**: The original packet (inner packet) is wrapped with new headers and sometimes new trailers. 2. **Tunneling**: Encapsulated packets are transmitted through the physical network to the destination host. 3. **Packet Decapsulation**: At the destination host, the outer header is removed and the original packet is extracted. 4. **Packet Forwarding**: The original packet is forwarded to the destination container. ### Major Overlay Network Protocols #### VXLAN (Virtual Extensible LAN) VXLAN is one of the most widely used overlay protocols in container networking. - **VXLAN Tunnel Endpoint (VTEP)**: Responsible for encapsulation and decapsulation of packets - **VXLAN Network Identifier (VNI)**: A 24-bit field with 16,777,216 possible values; this is not Cilium's supported tenant/endpoint capacity - **UDP Encapsulation**: Standard VXLAN uses UDP 4789; Cilium's default VXLAN tunnel port is UDP 8472 - **MAC-in-UDP Encapsulation**: Encapsulates original L2 frames into UDP packets VXLAN Packet Structure: ![Diagram of a VXLAN-encapsulated packet, showing the outer Ethernet, IP, and UDP headers wrapping a VXLAN header, which itself wraps the original Ethernet frame, IP header, TCP/UDP header, and payload.](https://raw.githubusercontent.com/Atom-oh/kubernetes-docs/main/en/.gitbook/assets/en-networking-cilium-networking-concepts-5.png) [🔍 View interactive diagram](https://www.atomai.click/kubernetes-docs/archmaps/en-networking-cilium-networking-concepts-5.html) This is the generic VXLAN wire format. UDP 4789 and the 24-bit VNI describe the standard; Cilium defaults to UDP 8472 and uses overlay metadata for identity. Field width is not a cluster-capacity guarantee. #### GENEVE (Generic Network Virtualization Encapsulation) GENEVE is a more flexible overlay protocol designed to overcome VXLAN limitations. - **Extensible Option Headers**: Supports various metadata - **Protocol Independent**: Can be used with various virtualization technologies - **UDP Encapsulation**: Transmitted via UDP port 6081 - **Flexible Tunneling**: Supports various network virtualization requirements #### IPsec IPsec is a protocol suite that provides security services at the IP packet level. - **Authentication and Encryption**: IPsec provides mechanisms for integrity/authentication and, with the appropriate mode, confidentiality - **Transport and Tunnel Modes**: Supports various deployment scenarios - **Security Association (SA)**: Defines security parameters between communicating parties - **Internet Key Exchange (IKE)**: A general IPsec negotiation mechanism. Cilium's IPsec setup instead uses an administrator-provided key Secret and its documented rotation procedure ### Advantages and Disadvantages of Overlay Networks #### Advantages: - **Flexibility**: Can configure virtual networks independently of physical network topology - **Scalability**: Supports large network segments and numerous endpoints - **Isolation**: Logical segments can separate traffic when configured correctly; encapsulation alone is not authentication, encryption or a complete policy boundary - **Compatibility**: Can work with existing network infrastructure #### Disadvantages: - **Overhead**: Increased packet size and processing overhead due to encapsulation - **MTU Considerations**: Reduced Maximum Transmission Unit (MTU) due to encapsulation - **Complexity**: Troubleshooting and debugging can be more complex - **Latency**: Encapsulation adds processing work; measure the actual effect with the selected implementation and offloads ### Overlay Networks in Cilium Cilium supports overlay protocols like VXLAN and Geneve, leveraging eBPF to provide efficient packet processing. - **eBPF-based VXLAN Processing**: Direct packet encapsulation and decapsulation within the kernel - **Efficient Routing**: Packet forwarding through optimized paths - **Encryption Options**: Encrypted overlay via IPsec or WireGuard - **Mode Selection**: Choose a supported routing mode. Enabling automatic direct node routes together with tunnel mode is rejected; it is not a fallback mechanism #### Cilium VXLAN Configuration Example: Helm values for a **new, prepared IPv4 test installation**; choose non-overlapping Pod CIDRs. This is not a live IPAM migration or a replacement ConfigMap. ```yaml # vxlan-values.yaml routingMode: tunnel tunnelProtocol: vxlan tunnelPort: 8472 autoDirectNodeRoutes: false ipv4: enabled: true ipv6: enabled: false ipam: mode: cluster-pool operator: clusterPoolIPv4PodCIDRList: - 10.244.0.0/16 clusterPoolIPv4MaskSize: 24 ``` ## Network Address Translation (NAT) Network Address Translation (NAT) is the process of modifying the source or destination addresses of IP packets. NAT is primarily used to enable devices on private networks to communicate with the public internet, or to enable communication between two networks with overlapping network address spaces. ### Main Types of NAT #### 1. Source NAT (SNAT) Source NAT modifies the source IP address of packets. It is typically used when devices on private networks access the internet. - **How It Works**: Rewrites a source address, and sometimes its port; private-to-public translation is one common use - **Use Cases**: Internet access, outbound connections - **Tracking**: Stores connection state in NAT table ![Diagram showing a client on an internal network (10.0.0.2:1234) sending traffic through a NAT router whose SNAT rewrites the source address to public IP 198.51.100.1:5678 before it reaches a server on the internet (203.0.113.5).](https://raw.githubusercontent.com/Atom-oh/kubernetes-docs/main/en/.gitbook/assets/en-networking-cilium-networking-concepts-6.png) [🔍 View interactive diagram](https://www.atomai.click/kubernetes-docs/archmaps/en-networking-cilium-networking-concepts-6.html) The documentation-range addresses illustrate one private-to-public SNAT case. SNAT means source translation; the replacement need not always be a public address. #### 2. Destination NAT (DNAT) Destination NAT modifies the destination IP address of packets. It is typically used when accessing services on private networks from the public internet. - **How It Works**: Rewrites a destination address/port; public-to-private forwarding is one example - **Use Cases**: Port forwarding, load balancing, inbound connections - **Configuration**: Defines mappings for specific ports or port ranges ![Diagram showing a client on the internet sending traffic through a NAT router that rewrites the destination address, reaching a server on the internal network at its private IP.](https://raw.githubusercontent.com/Atom-oh/kubernetes-docs/main/en/.gitbook/assets/en-networking-cilium-networking-concepts-7.png) [🔍 View interactive diagram](https://www.atomai.click/kubernetes-docs/archmaps/en-networking-cilium-networking-concepts-7.html) This shows one public-to-private DNAT case with documentation addresses. DNAT is destination translation and also appears in other address realms. #### 3. Port Address Translation (PAT) PAT modifies both IP addresses and port numbers. This allows multiple internal hosts to share a single public IP address. - **How It Works**: Translates IP:port combinations of internal hosts to different ports of a single public IP - **Use Cases**: IP address conservation, support for many internal hosts - **Limitations**: Finite port and state resources; the number of simultaneous flows also depends on protocol, destination tuples and mapping reuse, not a universal 65,000-connection ceiling ![Diagram showing two internal hosts sharing a single public IP (198.51.100.1) through a PAT router, which maps each host to a distinct public port (5000 and 5001) when reaching a server on the internet.](https://raw.githubusercontent.com/Atom-oh/kubernetes-docs/main/en/.gitbook/assets/en-networking-cilium-networking-concepts-8.png) [🔍 View interactive diagram](https://www.atomai.click/kubernetes-docs/archmaps/en-networking-cilium-networking-concepts-8.html) The example uses different translated ports for the same remote server. The roughly 65,000-port space is not a universal cap on all NAT connections; protocol, destination tuple, mapping behavior and state capacity matter. #### 4. Twice NAT and Bi-directional NAT **Twice NAT** changes both source and destination addresses as traffic crosses address realms, and can help reconcile overlapping address spaces. The address mappings, DNS/application assumptions and return path must all be designed together. **Bi-directional NAT** in RFC 2663 instead describes allowing sessions to be initiated from either realm. It does not, by definition, mean changing both addresses in each packet. ### Advantages and Disadvantages of NAT #### Advantages: - **IP Address Conservation**: Supports many internal hosts with a limited number of public IP addresses - **Address Hiding**: May conceal internal addresses, but NAT is not a substitute for firewall policy or authentication - **Address-Realm Reconciliation**: Appropriate translation can connect overlapping realms; that does not provide isolation by itself - **Flexible Network Design**: Can change ISP without reconfiguring internal network #### Disadvantages: - **Connection Tracking Overhead**: Resources needed for state table maintenance - **Certain Protocol Issues**: Some protocols may not be compatible with NAT - **Loss of End-to-End Connectivity**: Difficulty with direct peer-to-peer communication - **Complex Troubleshooting**: NAT-related problem debugging can be complex ### NAT in Kubernetes and Cilium #### NAT in Kubernetes Kubernetes uses NAT in various scenarios: 1. **Communication Outside the Cluster**: SNAT may be used depending on address reachability, masquerading exclusions and the chosen datapath 2. **Service Implementation**: Packet-based implementations can translate a Service destination; socket-level load balancing may choose a backend before such a packet exists 3. **NodePort Services**: The implementation forwards a node IP:port to selected backends; return-path behavior depends on SNAT/DSR and traffic policy 4. **LoadBalancer Services**: Provider/controller behavior varies and need not be a single DNAT step from a public address to a Pod #### NAT in Cilium Cilium leverages eBPF to provide efficient NAT implementation: 1. **eBPF-based NAT**: Performs NAT directly within the kernel 2. **High-Performance Connection Tracking**: Connection state tracking using optimized BPF maps 3. **NAT Controls**: Supported masquerading exclusions, service forwarding and Egress Gateway features have distinct configuration and prerequisites 4. **Masquerading**: Conditional source translation on configured paths/devices; excluded CIDRs and supported modes affect the result Egress Gateway is a separate feature that directs matching outbound traffic through selected nodes and SNATs it to configured gateway addresses. It changes the original source IP; interfaces, addresses and return paths must be prepared. New Pods can send traffic before policy convergence, so it is not an immediate fail-closed source-IP guarantee. #### Cilium NAT Configuration Example: This Helm fragment assumes a prepared kube-proxy replacement/BPF masquerading environment with a reachable API endpoint. Review the actual attached devices and routes first. Excluding a CIDR from SNAT does not create a return route; the NAT size is an example, not a recommended universal capacity. ```yaml # masquerade-values.yaml enableIPv4Masquerade: true kubeProxyReplacement: true bpf: masquerade: true natMax: 262144 ipMasqAgent: enabled: true config: nonMasqueradeCIDRs: - 10.0.0.0/8 - 172.16.0.0/12 - 192.168.0.0/16 masqLinkLocal: false ``` ## Routing Protocols Routing protocols define the rules and procedures that determine the optimal path for packets to travel from source to destination in a network. These protocols play an important role in adapting to network topology changes, efficiently forwarding traffic, and bypassing network failures. ### Classification of Routing Protocols #### 1. Interior Gateway Protocols (IGP) Interior Gateway Protocols are used to exchange routing information within a single Autonomous System (AS). ##### Distance Vector Protocols - **RIP (Routing Information Protocol)** - Uses hop count as metric - Valid metrics reach 15 hops; metric 16 represents unreachable - Simple implementation, suitable for small networks - Periodic updates are approximately every 30 seconds, with timer randomization and triggered updates for changes - **EIGRP (Enhanced Interior Gateway Routing Protocol)** - Configurable composite metric; default coefficients use throughput/bandwidth and delay, not load or reliability - Sends only partial updates - Fast convergence - Cisco-origin protocol documented in Informational RFC 7868; that publication is not an IETF Standards Track designation ##### Link State Protocols - **OSPF (Open Shortest Path First)** - Calculates shortest path using Dijkstra's algorithm - Area-based hierarchy - Fast convergence - Supports large-scale networks - Exchanges topology information through Link State Advertisements (LSAs) - **IS-IS (Intermediate System to Intermediate System)** - Link state protocol similar to OSPF - Widely used in large service provider networks - Supports multiple network layers - Efficient routing updates #### 2. Exterior Gateway Protocols (EGP) Exterior Gateway Protocols are used to exchange routing information between different Autonomous Systems. - **BGP (Border Gateway Protocol)** - Core routing protocol of the Internet - Path vector protocol - Policy-based routing decisions - Reliable sessions over TCP - Path selection through path attributes (AS path, local preference, etc.) - iBGP (internal BGP) and eBGP (external BGP) variants ### Routing Protocols in Container Networking In container environments, traditional routing protocols are used alongside container-specific routing mechanisms. #### 1. Container Networking with BGP BGP is gaining popularity in container networking for the following reasons: - **Reachability Advertisement**: Advertises Pod or Service prefixes to routers; the local forwarding implementation still determines how traffic travels - **Scalability**: Supports large-scale clusters and multi-cluster environments - **Existing Network Integration**: Integration with data center network infrastructure - **Availability**: Multipath and convergence depend on router policy, timers and a functioning datapath; a session alone does not guarantee fast failover #### 2. Container Network Routing Mechanisms - **Host-based Routing**: Hosts maintain Pod routes and may participate in a separately configured route-advertisement mechanism - **Centralized Routing**: Controller manages routing decisions centrally - **Distributed Routing**: Direct routing information exchange between nodes - **Policy-based Routing**: Routing decisions based on traffic characteristics ### Routing in Cilium Cilium implements routing with eBPF and supports different datapath modes. **Host routing is a separate axis**: BPF host routing optimizes forwarding inside the node and can bypass parts of the host stack/netfilter. It requires compatible kube-proxy replacement/BPF masquerading and has integration constraints. It does not mean selecting native rather than tunnel routing between nodes. #### 1. Native Routing (Direct Routing) In native routing mode, Cilium routes pod IPs directly without overlay encapsulation. - **How It Works**: Pod traffic uses underlay routes without overlay encapsulation; enabling native mode does not automatically enable BGP - **Advantages**: Avoids overlay encapsulation overhead; actual performance requires measurement - **Requirements**: Valid routes for the relevant Pod addresses and their return traffic, not merely reachability between node IPs - **Use Cases**: Performance-critical workloads, single-subnet clusters Native routing needs valid Pod routes, but choosing `routingMode: native` does not automatically advertise them with BGP. Provision the underlay/return routes or configure the appropriate route-distribution mechanism. The earlier host-route illustration shows the forwarding principle. #### 2. BGP Routing Cilium supports BGP routing to integrate pod IPs with physical network infrastructure. - **How It Works**: Cilium advertises pod CIDRs through BGP peering - **Advantages**: Integration with existing network infrastructure, high availability - **Components**: BGP peering, route filtering, community attributes - **Use Cases**: Integration with data center networks, multi-cluster environments #### 3. Overlay Routing Cilium can route pod traffic between nodes using overlay protocols like VXLAN or Geneve. - **How It Works**: Encapsulates pod packets for transmission between nodes - **Advantages**: Minimizes network infrastructure requirements, flexible deployment - **Use Cases**: Cloud environments, complex network topologies #### 4. Hybrid Routing Do not assume Cilium automatically uses native routes when reachable and otherwise falls back to an overlay. Current tunnel mode cannot be combined with `autoDirectNodeRoutes: true`; the agent rejects that configuration. Choose a supported datapath and provision its underlay. The valid load-balancer mode called `hybrid` is a different feature: TCP uses DSR while UDP uses SNAT. It is not an overlay/native routing fallback. ### Cilium Routing Configuration Examples #### Native Routing Configuration: This native-routing fragment assumes the intended Pod CIDR and nodes reachable on a shared L2 network for automatic direct routes. Other topologies need an appropriate routing mechanism. It must not be combined with tunnel mode as an automatic fallback. ```yaml # native-values.yaml routingMode: native autoDirectNodeRoutes: true ipv4NativeRoutingCIDR: 10.244.0.0/16 ``` #### BGP Routing Configuration: The feature flag below is only one prerequisite. Configure the current `CiliumBGPClusterConfig`, `CiliumBGPPeerConfig` and `CiliumBGPAdvertisement` resources and the external router as described in [Advanced Topics](https://www.atomai.click/kubernetes-docs/llms/en/networking/cilium/07-advanced-topics.md). BGP advertisement and the datapath routing mode are independent choices. ```yaml # bgp-values.yaml bgpControlPlane: enabled: true ``` #### Overlay Routing Configuration: Use the complete VXLAN values example above. Verify the running mode and port in agent status; do not overwrite installation settings with a small ConfigMap. ## DNS and Service Discovery DNS (Domain Name System) and service discovery play a critical role in modern network applications, especially in dynamic container environments. These mechanisms abstract service locations and allow applications to adapt to network topology changes. ### DNS (Domain Name System) DNS is a distributed system that translates human-readable domain names into IP addresses. #### How DNS Works 1. **Hierarchical Namespace**: Domain names are organized in a hierarchical structure separated by dots (e.g., www.example.com) 2. **Distributed Database**: Network of DNS servers distributed worldwide 3. **Iterative and Recursive Queries**: Two main methods of processing client requests 4. **Caching**: Temporary storage of results for performance improvement #### DNS Record Types - **A Record**: Maps domain name to IPv4 address - **AAAA Record**: Maps domain name to IPv6 address - **CNAME Record**: Alias (canonical name) for domain name - **MX Record**: Specifies mail server - **SRV Record**: Specifies server providing specific service - **TXT Record**: Stores text information (primarily used for verification and policies) - **PTR Record**: Reverse mapping of IP address to domain name (reverse DNS) #### DNS Resolution Process A common uncached lookup separates the application's stub resolver from a recursive resolver: | Step | Query/response | | --- | --- | | 1 | The stub asks its configured recursive resolver for `www.example.com`. | | 2 | The resolver asks a root server and receives a referral to `.com` servers. | | 3 | The resolver asks a `.com` server and receives a referral to `example.com` authoritative servers. | | 4 | The resolver asks the authoritative server and obtains the relevant answer. | | 5 | The resolver caches according to TTL and returns the answer to the stub. | The authoritative servers do not normally forward this sequence among themselves. Caches, aliases and configured forwarders can change the exact exchanges. ### Service Discovery in Container Environments Service discovery is the process of automatically detecting available services and locating them on a network. In container environments, it is particularly important for effectively managing dynamically created and removed services. #### Service Discovery Approaches 1. **DNS-based Service Discovery** - Creates DNS records when services are registered - Clients discover services through standard DNS lookups - Simple and widely supported - Examples: Kubernetes DNS, CoreDNS 2. **Key-Value Store-based Service Discovery** - Stores service information in centralized key-value stores - Clients query the store to discover services - Rich metadata support - Examples: etcd, Consul, ZooKeeper 3. **API-based Service Discovery** - Provides service information through dedicated APIs - Clients call APIs to discover services - Complex querying and filtering support - Example: Kubernetes API Server 4. **Mesh-based Service Discovery** - Service mesh infrastructure handles service discovery - Supports client-side load balancing and routing - Advanced traffic management features - Examples: Istio, Linkerd ### DNS and Service Discovery in Kubernetes Kubernetes provides built-in mechanisms for service discovery within the cluster. #### Kubernetes Services Kubernetes Services provide stable endpoints for sets of pods: - **ClusterIP**: A Service virtual IP, normally used inside the cluster; any external routability is an explicit network design, not an intrinsic security boundary - **NodePort**: A node port exposed on eligible node addresses, subject to traffic policy, routing and firewall rules - **LoadBalancer**: Requests a provider/controller implementation, which may be public or internal - **ExternalName**: DNS alias for external service #### Kubernetes DNS Kubernetes runs a cluster DNS service (typically CoreDNS) to support service discovery: - **Service DNS**: `..svc.`; `cluster.local` is a common configured domain, not a universal constant - **Pod DNS**: The old address-based `pod.` form is implementation-dependent/legacy. Stable Pod names commonly use hostname/subdomain with a corresponding headless Service - **Headless Services**: DNS can return endpoint addresses rather than a VIP; readiness and `publishNotReadyAddresses` affect which records are published DNS lookup and Service forwarding are separate: | Phase | Responsibility | | --- | --- | | DNS lookup | CoreDNS resolves an ordinary Service name to its ClusterIP using Kubernetes object state. It does not choose the application backend for that connection. | | Connection | The client sends traffic to the returned Service address. | | Forwarding | The Service implementation, such as Cilium's datapath, selects an eligible backend using its Service/EndpointSlice-derived state. | | Headless Service | DNS returns endpoint addresses instead of a Service VIP; client-side behavior determines which address is used. | Object watches and datapath updates are asynchronous; a DNS response is not a backend health probe. #### Kubernetes Service Discovery Mechanisms 1. **Environment Variables**: Service links can reflect Services present when the Pod is created; they are not a live discovery feed and can be disabled 2. **DNS**: Service name resolution through cluster DNS 3. **API Server**: Retrieve service information by directly querying Kubernetes API 4. **EndpointSlice Objects**: Provide backend address, port and readiness information for Service implementations ### DNS and Service Discovery in Cilium Cilium integrates with Kubernetes service discovery mechanisms and provides additional features. #### Cilium's DNS-based Policies Cilium can define network policies based on DNS names: - **DNS Name-based Filtering**: Access control for specific domain names - **Wildcard Support**: `*.example.com` matches one subdomain level; `**.example.com` supports multiple levels in this version, and neither includes the apex without an explicit match - **FQDN Policies**: Policies based on Fully Qualified Domain Names (FQDNs) This policy-only example requires the namespace, labeled workload and verified resolver path to exist. DNS observation and TCP 443 destination allowances are separate. The DNS `*` permits all query names; toFQDNs is not hostname authentication. ```yaml # dns-policy.yaml apiVersion: cilium.io/v2 kind: CiliumNetworkPolicy metadata: name: dns-policy namespace: cilium-fqdn-demo spec: endpointSelector: matchLabels: app: myapp egress: - toEndpoints: - matchLabels: k8s:io.kubernetes.pod.namespace: kube-system k8s:k8s-app: kube-dns toPorts: - ports: - port: '53' protocol: UDP - port: '53' protocol: TCP rules: dns: - matchPattern: '*' - toFQDNs: - matchName: api.example.com - matchPattern: '*.api.example.com' toPorts: - ports: - port: '443' protocol: TCP ``` #### Cilium's Service Discovery Enhancements Cilium provides several features that enhance Kubernetes service discovery: 1. **eBPF-based Service Implementation**: - kube-proxy replacement - Direct service load balancing within the kernel - Improved performance and features 2. **Global Services**: - Service discovery across multiple clusters - Cross-cluster load balancing - Matching Service names/namespaces and explicit sharing/ClusterMesh configuration 3. **Service Affinity**: - Session affinity support - ClientIP affinity is separate from the load-balancing algorithm; socket-level paths can use a network-namespace cookie - Stateful connection support 4. **Health Check Integration**: - Backend state follows Kubernetes readiness/EndpointSlice information and configured proxy checks - Changes are propagated asynchronously - Do not assume every Cilium Service performs active application probes or instantaneous failover #### Cilium Service Configuration Example: Session affinity is configured on the Service; Global Services use an annotation and a working ClusterMesh. Peer Services must have the same name and namespace. This example does not create the application, ClusterMesh or an external load balancer. ```yaml # global-service.yaml apiVersion: v1 kind: Service metadata: name: api namespace: cilium-service-demo annotations: service.cilium.io/global: 'true' spec: type: ClusterIP selector: app: api ports: - name: http port: 80 targetPort: 8080 sessionAffinity: ClientIP sessionAffinityConfig: clientIP: timeoutSeconds: 10800 ``` ## Load Balancing Concepts Load balancing is a technology that distributes network traffic across multiple servers or backend services to optimize resource utilization, support throughput, latency and availability goals when combined with suitable capacity and backend health handling. In container environments, effectively distributing traffic among dynamically changing backend instances is particularly important. ### Types of Load Balancing #### 1. L4 (Transport Layer) Load Balancing L4 load balancing distributes traffic based on transport layer information such as IP addresses and port numbers. - **How It Works**: Routing decisions based on TCP/UDP header information - **Advantages**: Fast processing, low overhead, can handle encrypted traffic - **Disadvantages**: Cannot perform advanced routing based on application layer information - **Use Cases**: TCP/UDP-based services, high-performance requirements ![Diagram showing a client request routed by a transport-layer load balancer to one of two backend servers, based only on TCP/UDP header information.](https://raw.githubusercontent.com/Atom-oh/kubernetes-docs/main/en/.gitbook/assets/en-networking-cilium-networking-concepts-12.png) [🔍 View interactive diagram](https://www.atomai.click/kubernetes-docs/archmaps/en-networking-cilium-networking-concepts-12.html) The branches represent possible backend choices, not broadcasting a connection to both servers. L4 forwarding can carry TLS without inspecting the encrypted HTTP payload. #### 2. L7 (Application Layer) Load Balancing L7 load balancing distributes traffic based on application layer information such as HTTP headers, URLs, and cookies. - **How It Works**: Routing decisions by inspecting HTTP/HTTPS request contents - **Advantages**: Content-based routing, advanced traffic management, security features - **Disadvantages**: Proxy processing cost; HTTP content inspection of HTTPS needs appropriate TLS termination - **Use Cases**: Web applications, microservices, API gateways ![Diagram showing a client HTTP request routed by an application-layer load balancer to one of two backend services, based on URL path and header inspection.](https://raw.githubusercontent.com/Atom-oh/kubernetes-docs/main/en/.gitbook/assets/en-networking-cilium-networking-concepts-13.png) [🔍 View interactive diagram](https://www.atomai.click/kubernetes-docs/archmaps/en-networking-cilium-networking-concepts-13.html) The selected route depends on the request attributes. HTTP content routing over HTTPS requires an appropriate TLS termination/inspection path. ### Load Balancing Algorithms Load balancing algorithms determine how traffic is distributed to backend servers. #### 1. Round Robin - **How It Works**: Distributes requests to each backend server sequentially - **Advantages**: Simple sequencing; equal request counts do not imply equal backend work - **Disadvantages**: Does not consider server capacity differences or current load - **Variants**: Weighted Round Robin (applies weights based on server capacity) #### 2. Least Connections - **How It Works**: Forwards new requests to server with fewest active connections - **Advantages**: Considers server load, effective for long connections - **Disadvantages**: Connection count does not always accurately reflect load - **Variants**: Weighted Least Connections (applies weights based on server capacity) #### 3. IP Hash - **How It Works**: Hashes client IP address for consistent backend server selection - **Advantages**: Can provide stable selection while inputs/backend membership remain stable; it is not permanent session storage - **Disadvantages**: Possible uneven distribution, potential overload on specific servers - **Variants**: Source-Destination IP Hash (considers both source and destination IPs) #### 4. Least Response Time - **How It Works**: Forwards requests to server with shortest response time - **Advantages**: Considers performance and availability, suitable for latency-sensitive applications - **Disadvantages**: Response time measurement overhead, affected by network variability - **Variants**: Weighted Response Time (considers both server capacity and response time) #### 5. Random Selection - **How It Works**: Randomly selects backend server - **Advantages**: Simple implementation, no special state tracking required - **Disadvantages**: Possible uneven distribution - **Variants**: Weighted Random Selection (adjusts probability based on server capacity) ### Load Balancer Deployment Models #### 1. Hardware Load Balancers - **Characteristics**: Dedicated physical equipment - **Advantages**: High performance, reliability, dedicated hardware acceleration - **Disadvantages**: Cost, limited scalability, lack of flexibility - **Examples**: Application delivery controller appliances; some product families also offer virtual/software editions #### 2. Software Load Balancers - **Characteristics**: Software running on general-purpose servers - **Advantages**: Flexibility, cost efficiency, programmability - **Disadvantages**: Capacity depends on implementation, hardware and workload; software is not inherently slower than every appliance - **Examples**: NGINX, HAProxy, Envoy #### 3. Cloud Load Balancers - **Characteristics**: Services managed by cloud providers - **Advantages**: Reduced management overhead, auto-scaling, high availability - **Disadvantages**: Vendor lock-in, limited customization - **Examples**: AWS ELB/ALB/NLB, Google Cloud Load Balancing, Azure Load Balancer #### 4. Container-Native Load Balancers - **Characteristics**: Load balancing optimized for container environments - **Advantages**: Integration with container orchestration, dynamic service discovery - **Disadvantages**: Specialized for container environments - **Examples**: Kubernetes Services, Istio, Cilium ### Load Balancing in Kubernetes Kubernetes provides multiple levels of load balancing: #### 1. Service Load Balancing - **ClusterIP**: Internal cluster load balancing - **NodePort**: External access through node ports - **LoadBalancer**: External load balancer provisioning - **ExternalName**: DNS alias for external services #### 2. Ingress Controllers - L7 load balancing and routing - URL-based routing and TLS termination; authentication capabilities depend on the controller and configuration - Implementations include Traefik, HAProxy and Istio-based controllers. Community `ingress-nginx` retired in March 2026; remaining artifacts are not a maintained installation recommendation #### 3. Service Mesh - Advanced traffic management between microservices - Granular routing, traffic splitting, fault injection - Examples: Istio, Linkerd and Consul service mesh; their traffic-management and security feature sets differ ### Load Balancing in Cilium Cilium implements efficient load balancing using eBPF: #### 1. eBPF-based Load Balancing - **kube-proxy Replacement**: Direct service load balancing within the kernel - **Performance**: Supported BPF paths can avoid parts of the conventional stack; quantify the result for the actual workload - **Scalability**: Supports large-scale services and endpoints - **Connection Tracking Optimization**: Efficient state management ![Diagram of Cilium eBPF-based load balancing: a packet Pod A sends to a Service IP passes through a four-step eBPF pipeline in the kernel — packet intercept, service map lookup, backend selection, packet forwarding — and is delivered straight to Pod B without kube-proxy.](https://raw.githubusercontent.com/Atom-oh/kubernetes-docs/main/en/.gitbook/assets/en-networking-cilium-networking-concepts-14.png) [🔍 View interactive diagram](https://www.atomai.click/kubernetes-docs/archmaps/en-networking-cilium-networking-concepts-14.html) This depicts a packet-path Service translation. Socket-level load balancing can instead select a backend before a Service-IP packet exists. Latency improvements require measurement for the actual path. #### 2. Load Balancing Algorithms The BPF Service algorithms are **random** (the default) and **Maglev**. Maglev hashes flow information; it is not simply source-IP affinity. Ordinary socket-level east-west selection is a different path from the external packet paths where Maglev is applied. `ClientIP` session affinity is configured independently on a Service. Its timeout is not a “Maglev timeout”; Maglev has no timer that periodically rebalances sessions. Membership, seed or table changes can remap selection, and a removed backend cannot continue serving a connection merely because hashing is consistent. #### 3. L7 Load Balancing Cilium also supports L7 (Application Layer) load balancing: - **HTTP Header-based Routing**: Routing based on specific header values - **URL Path-based Routing**: Traffic distribution based on URL patterns - **gRPC Routing**: Routing based on gRPC methods and metadata - **Kafka**: Current Cilium does not provide the former Kafka topic L7 policy/routing feature; use broker-appropriate controls #### 4. Global Service Load Balancing Cilium supports load balancing across multiple clusters: - **Cross-cluster Load Balancing**: Traffic distribution among backends across multiple clusters - **Locality Preference**: Configured local/remote affinity is not automatic measurement of network latency - **Failure Handling**: Depends on endpoint state and remote-cache behavior; the default zero cache TTL can retain stale remote state, so application failover must be tested #### Cilium Load Balancing Configuration Example: These are Helm values for a prepared installation. The shown hash seed is a valid **12-byte base64 demonstration value**. For deployment, generate and persist a common random seed for the participating nodes, and review a seed/table change as a connection-impacting operation. See the [prepared load-balancing profile](https://www.atomai.click/kubernetes-docs/llms/en/networking/cilium/05-l2-l7-networking.md). ```yaml # load-balancing-values.yaml kubeProxyReplacement: true loadBalancer: algorithm: maglev maglev: tableSize: 16381 hashSeed: AAECAwQFBgcICQoL ``` ## Network Security Basics Network security is the practice of protecting network infrastructure, applications, and data from unauthorized access, misuse, failure, or modification. In container environments, network security is even more important due to their dynamic and distributed nature. ### Core Network Security Concepts #### 1. Defense in Depth Defense in depth combines controls to reduce the impact of an individual failure. Shared dependencies or a common misconfiguration can still affect multiple layers. - **Multiple Security Layers**: Protection at network, host, application, and data levels - **Redundant Controls**: Combination of various security mechanisms - **Failure Isolation**: Design and test boundaries; independence of failures is not automatic - **Threat Detection and Response**: Monitoring and response at each layer #### 2. Principle of Least Privilege The principle of least privilege is a security practice that grants users, processes, or applications only the minimum privileges necessary to perform their tasks. - **Granular Access Control**: Restricting access to only necessary resources - **Privilege Separation**: Separation of privileges for various functions - **Default Deny**: Denying all access not explicitly allowed - **Regular Review**: Regular auditing and adjustment of privileges #### 3. Network Segmentation Network segmentation is a technique that divides a network into smaller segments or zones to enhance security and limit lateral movement of threats. - **Security Zones**: Grouping systems with similar security requirements - **Microsegmentation**: Granular control at the workload level - **Perimeter Protection**: Control and monitoring of traffic between zones - **Threat Isolation**: Limiting the scope of impact of a breach #### 4. Encryption Encryption is the process of transforming data so that it cannot be read by unauthorized parties. - **Encryption in Transit**: Protecting data moving over the network (e.g., supported TLS) - **Encryption at Rest**: Protecting data stored on disk or in databases - **End-to-End Encryption**: Protecting data across the entire communication path - **Key Management**: Secure generation, storage, and rotation of encryption keys ### Container Networking Security Threats Container environments present unique security challenges: #### 1. Network-based Attacks - **DDoS (Distributed Denial of Service) Attacks**: Large volumes of traffic to disrupt service availability - **Port Scanning**: Exploring open ports and vulnerabilities - **ARP Spoofing**: Manipulating Address Resolution Protocol to intercept network traffic - **DNS Poisoning**: Redirecting DNS lookups to malicious destinations #### 2. Application Layer Attacks - **SQL Injection**: Inserting malicious SQL code - **XSS (Cross-Site Scripting)**: Inserting client-side scripts - **CSRF (Cross-Site Request Forgery)**: Performing malicious actions through authenticated users - **Command Injection**: Malicious input to execute system commands #### 3. Container-Specific Threats - **Image Vulnerabilities**: Container images containing vulnerable components - **Privilege Escalation**: Gaining permissions within or across a boundary; it is not always the same event as a container escape - **Lateral Movement**: Unauthorized access from one container to another - **Volume Mount Exploitation**: Access to sensitive host paths ### Network Security Controls #### 1. Firewalls Firewalls are network security systems that filter network traffic based on defined security rules. - **Packet Filtering**: Filtering based on IP addresses, ports, protocols - **Stateful Inspection**: Context-based decisions tracking connection state - **Application Layer Filtering**: Understanding and inspecting application protocols - **Next-Generation Firewalls (NGFW)**: Advanced threat detection and prevention features #### 2. Intrusion Detection and Prevention Systems (IDS/IPS) IDS/IPS are systems that monitor network traffic and detect or block malicious activity. - **Signature-based Detection**: Matching known attack patterns - **Anomaly Detection**: Identifying activities that deviate from normal behavior - **Behavior Monitoring**: Analysis of suspicious activity patterns - **Automated Response**: Real-time response to detected threats #### 3. Network Policies Network policies are sets of rules that define allowed communication within a network. - **Ingress Control**: Restricting incoming traffic - **Egress Control**: Restricting outgoing traffic - **Granular Policies**: Communication control at workload level - **Label-based Policies**: Flexible policy application in dynamic environments #### 4. Encryption Protocols Encryption protocols provide secure communication over networks. - **TLS**: Protecting web traffic and API communication; SSL protocols are obsolete - **IPsec**: Network layer encryption - **WireGuard**: Modern and efficient VPN protocol - **mTLS (mutual TLS)**: Authentication of both client and server ### Network Security in Kubernetes Kubernetes provides several mechanisms for network security of containerized applications: #### 1. Network Policies Kubernetes NetworkPolicy specifies L3/L4 allowances for selected Pods. It needs an enforcing network implementation; allows from applicable policies combine, and existing/host-network paths require their own semantics. - **Pod Selectors**: Selecting pods to which policies apply based on labels - **Ingress Rules**: Controlling incoming traffic - **Egress Rules**: Controlling outgoing traffic - **CIDR-based Rules**: Filtering based on IP ranges This L4 example uses its own namespace and assumes matching frontend/API/database workloads and the stated CoreDNS labels. It includes DNS egress. Keep it separate from the later L7 example: a broad L4 allow can bypass an overlapping L7 restriction. ```yaml # api-l4-policy.yaml apiVersion: networking.k8s.io/v1 kind: NetworkPolicy metadata: name: api-allow namespace: cilium-policy-l4-demo spec: podSelector: matchLabels: app: api policyTypes: - Ingress - Egress ingress: - from: - podSelector: matchLabels: app: frontend ports: - protocol: TCP port: 8080 egress: - to: - podSelector: matchLabels: app: database ports: - protocol: TCP port: 5432 - to: - namespaceSelector: matchLabels: kubernetes.io/metadata.name: kube-system podSelector: matchLabels: k8s-app: kube-dns ports: - protocol: UDP port: 53 - protocol: TCP port: 53 ``` #### 2. Service Mesh Security Service mesh is an infrastructure layer that manages and protects communication between microservices. - **mTLS**: Encrypted communication between services - **Authentication and Authorization**: Service identity verification and access control - **Traffic Policies**: Granular routing and access control - **Observability**: Visibility into service-to-service communication #### 3. Security Contexts Security contexts define privilege and access control settings for pods and containers. - **Privilege Restriction**: Running as non-root user - **Capability Restriction**: Allowing only necessary Linux capabilities - **Read-only Root Filesystem**: Restricts writes to the container root filesystem; mounted volumes can remain writable - **seccomp and AppArmor**: Restricting system calls and application behavior ### Cilium's Network Security Features Cilium leverages eBPF to provide powerful network security features: #### 1. Identity-based Security Cilium supports workload-identity policy derived from security-relevant labels, alongside explicit CIDR/IP controls where configured. - **Label-based Policies**: Consistent security in dynamic environments - **Service Account-based Policies**: Access control based on Kubernetes service accounts - **DNS-based Policies**: Egress control based on FQDNs - **API-aware Security**: Filtering based on HTTP methods and paths `toCIDR` selects destination ranges, but by default CIDR selectors do not match managed in-cluster Pods/nodes; this version has an explicit Beta opt-in for those cases. The `world` entity covers external endpoints rather than all known cluster/ClusterMesh identities. Use the appropriate identity/entity scope instead of treating `world` as an allow-all-clusters synonym. #### 2. Transparent Encryption Cilium can encrypt supported paths without application changes. Node tunnels do not cover same-node traffic or every external destination. The separate SPIRE mutual-authentication handshake does not itself encrypt application traffic; Beta ztunnel workload mTLS has its own prerequisites. - **IPsec**: Network layer encryption for inter-node traffic - **WireGuard**: Modern and efficient encryption protocol - **Transparent Integration**: Encryption applied without application changes - **Key Rotation**: Follow the chosen mode's key lifecycle; Cilium IPsec requires provisioned key material and its documented Secret rotation procedure #### 3. Threat Detection and Visibility Cilium/Hubble provides network observations that can support investigation. A complete IDS/WAF, runtime enforcement or alert/response workflow requires the appropriate separate configuration or integration. - **Hubble**: Network flow monitoring and analysis - **Flow Logs**: Detailed logs of pod-to-pod communication - **Anomaly Detection**: External detection rules can analyze observed patterns; Hubble does not automatically classify every attack - **Security Event Alerts**: Configure an alerting/SIEM integration and account for event loss, noise and incomplete observations #### 4. L3-L7 Policy Enforcement Cilium provides comprehensive policy enforcement from network layer to application layer. - **L3/L4 Policies**: IP and port-based filtering - **L7 HTTP Filtering**: URL, method, header-based control - **L7 gRPC Filtering**: gRPC method and metadata-based control - **DNS Policy**: Query filtering and DNS observation for FQDN rules; no current Kafka topic L7 policy #### Cilium Network Security Configuration Example: This alternative L7 policy uses a different namespace from the L4 example. It requires visible plaintext HTTP or an appropriate TLS inspection path, real labeled dependencies and resolver reachability. The `.example` external name is a placeholder; no working external service is provisioned. ```yaml # api-l7-policy.yaml apiVersion: cilium.io/v2 kind: CiliumNetworkPolicy metadata: name: secure-api namespace: cilium-policy-l7-demo spec: endpointSelector: matchLabels: app: api ingress: - fromEndpoints: - matchLabels: k8s:io.kubernetes.pod.namespace: cilium-policy-l7-demo k8s:app: frontend toPorts: - ports: - port: '8080' protocol: TCP rules: http: - method: GET path: /api/v1/products egress: - toEndpoints: - matchLabels: k8s:io.kubernetes.pod.namespace: kube-system k8s:k8s-app: kube-dns toPorts: - ports: - port: '53' protocol: UDP - port: '53' protocol: TCP rules: dns: - matchPattern: '*' - toEndpoints: - matchLabels: k8s:io.kubernetes.pod.namespace: cilium-policy-l7-demo k8s:app: database toPorts: - ports: - port: '5432' protocol: TCP - toFQDNs: - matchName: api.external-service.example toPorts: - ports: - port: '443' protocol: TCP ``` ### Network Security Best Practices #### 1. Default Deny Policy - Implement default deny policy that only allows explicitly permitted traffic - Open only necessary communication paths - Regular policy review and removal of unnecessary rules - Maintain audit trail for policy changes #### 2. Defense in Depth Approach - Implement multiple security layers - Combine network, host, and application-level protection - Redundant controls with various security mechanisms - Eliminate single points of failure #### 3. Least Privilege Networking - Allow only minimum necessary network access - Define granular policies per service - Block unnecessary ports and protocols - Regular access review and adjustment #### 4. Continuous Monitoring and Auditing - Monitor network traffic and policy violations - Detect anomalies and potential threats - Alerts and response to security events - Regular security audits and vulnerability assessments ## Primary References - [Cilium routing](https://raw.githubusercontent.com/cilium/cilium/v1.20.1/Documentation/network/concepts/routing.rst) - [Cilium chart values](https://raw.githubusercontent.com/cilium/cilium/v1.20.1/install/kubernetes/cilium/values.yaml) - [Kube-proxy replacement](https://raw.githubusercontent.com/cilium/cilium/v1.20.1/Documentation/network/kubernetes/kubeproxy-free.rst) - [Masquerading](https://raw.githubusercontent.com/cilium/cilium/v1.20.1/Documentation/network/concepts/masquerading.rst) - [BGP Control Plane](https://raw.githubusercontent.com/cilium/cilium/v1.20.1/Documentation/network/bgp-control-plane/bgp-control-plane.rst) - [Global Services](https://raw.githubusercontent.com/cilium/cilium/v1.20.1/Documentation/network/clustermesh/global-services.rst) - [Policy language](https://raw.githubusercontent.com/cilium/cilium/v1.20.1/Documentation/security/policy/layer3.rst) - [DNS policy](https://raw.githubusercontent.com/cilium/cilium/v1.20.1/Documentation/security/dns.rst) - [IPsec](https://raw.githubusercontent.com/cilium/cilium/v1.20.1/Documentation/security/network/encryption-ipsec.rst) - [WireGuard](https://raw.githubusercontent.com/cilium/cilium/v1.20.1/Documentation/security/network/encryption-wireguard.rst) - [Kubernetes network model](https://kubernetes.io/docs/concepts/services-networking/) - [Services](https://kubernetes.io/docs/concepts/services-networking/service/) - [DNS for Services and Pods](https://kubernetes.io/docs/concepts/services-networking/dns-pod-service/) - [NetworkPolicy](https://kubernetes.io/docs/concepts/services-networking/network-policies/) - [CNI specification](https://raw.githubusercontent.com/containernetworking/cni/main/SPEC.md) - [Docker bridge networking](https://docs.docker.com/engine/network/drivers/bridge/) - [Docker host networking](https://docs.docker.com/engine/network/drivers/host/) - [Ingress NGINX retirement](https://kubernetes.io/blog/2025/11/11/ingress-nginx-retirement/) - [Internet architecture / RFC 1122](https://www.rfc-editor.org/rfc/rfc1122.txt) - [DNS / RFC 1034](https://www.rfc-editor.org/rfc/rfc1034.txt) - [NAT terminology / RFC 2663](https://www.rfc-editor.org/rfc/rfc2663.txt) - [NAT mapping behavior / RFC 4787](https://www.rfc-editor.org/rfc/rfc4787.txt) - [RIP v2 / RFC 2453](https://www.rfc-editor.org/rfc/rfc2453.txt) - [EIGRP / RFC 7868](https://www.rfc-editor.org/rfc/rfc7868.txt) ## Quiz To test what you learned in this chapter, try the [Topic Quiz](https://www.atomai.click/kubernetes-docs/en/quizzes/networking/cilium/networking-concepts-quiz). ---------------------------------------- Source: https://www.atomai.click/kubernetes-docs/en/networking/cilium/glossary ---------------------------------------- # Glossary and Abbreviations > **Review baseline**: Cilium 1.20.1. > **Last reviewed**: September 12, 2026. An alphabetical reference for Cilium, eBPF, Kubernetes and networking. Repeated entries are consolidated. ## A **API (Application Programming Interface)** - General - A set of interface definitions that enable communication between applications **ARP (Address Resolution Protocol)** - Networking - Resolves an IPv4 address to a link-layer address on the local link, commonly an Ethernet MAC address. - For a remote destination, a host resolves its next hop. IPv6 uses Neighbor Discovery rather than ARP. **AWS ENI (Elastic Network Interface)** - Networking - Virtual network interface provided by Amazon Web Services - Used in Cilium's AWS ENI IPAM mode ## B **BGP (Border Gateway Protocol)** - Networking - An inter-domain routing protocol used to advertise reachability between peers. - Cilium BGP Control Plane advertises selected prefixes; it is not a native-routing mode and does not program the local datapath routes. **BPF (Berkeley Packet Filter)** - eBPF - Technology for packet filtering, predecessor to eBPF - Originally developed for network packet capture **BPF Maps** - eBPF - Kernel-managed data structures used by BPF programs and userspace to share state or events. - Many types use keys and values; ring buffers, queues and stacks have different operations. A BPF ring buffer does not support map lookup/update/delete. ## C **CGroup (Control Group)** - Kubernetes - Linux control groups organize processes and account for or control resources such as CPU and memory. - Container runtimes use cgroups; they are not, by themselves, process/network namespace isolation. **CIDR (Classless Inter-Domain Routing)** - Networking - Method for IP address allocation and routing aggregation - Example: 192.168.1.0/24 represents IP address range from 192.168.1.0 to 192.168.1.255 **Cilium** - Cilium - Open source networking, security, and observability solution based on eBPF - Used as a Kubernetes CNI implementation **Cilium Agent** - Cilium - The node-local Cilium component that manages endpoints, BPF programs and policy/datapath state. It runs on Cilium-managed eligible nodes. **Cilium Operator** - Cilium - The cluster-level controller for tasks such as CRD registration, mode-dependent IPAM/LB IPAM, garbage collection and enabled Ingress/Gateway controllers. - Replica count is configurable. Optional identity management and ClusterMesh synchronization depend on enabled features; it is not the node packet-forwarding component. **ClusterMesh** - Cilium - Cilium's multi-cluster network metadata/connectivity features for service discovery, load balancing and remote-identity policy. - Requires compatible addressing, trust and reachable paths; it neither supplies shared storage nor automatically replicates every policy resource. **CNI (Container Network Interface)** - Kubernetes - Container Network Interface: a specification and plugins for configuring container network connectivity. - In current Kubernetes, the CRI container runtime loads/invokes CNI plugins. kubelet's former direct CNI-management flags were removed in Kubernetes 1.24. **CoreDNS** - Kubernetes - DNS server commonly used in Kubernetes clusters - Plays an important role in service discovery **CRD (Custom Resource Definition)** - Kubernetes - Method to define custom resources by extending the Kubernetes API - Cilium uses CRDs to define network policies, etc. ## D **DaemonSet** - A Kubernetes controller that runs daemon Pods on eligible nodes selected by its scheduling constraints; it need not cover every node. **DNAT (Destination Network Address Translation)** - Networking - NAT type that modifies the destination IP address of packets - Used for load balancing and port forwarding **DNS (Domain Name System)** - Networking - A distributed naming system that publishes records such as A/AAAA addresses, CNAME aliases and SRV service information. - Cilium DNS policy and learned-IP FQDN policy are related but distinct controls. ## E **eBPF (extended Berkeley Packet Filter)** - eBPF - Extended Berkeley Packet Filter: programmable kernel hooks and associated infrastructure used by Cilium. - The verifier checks program properties before acceptance. It does not guarantee that kernel or verifier implementations are free of vulnerabilities. **Endpoint** - Cilium - A Cilium-managed network endpoint, commonly a Pod, with local datapath/policy state. - Its endpoint ID is local to the agent and is distinct from a security identity shared by multiple endpoints. **Envoy** - Cilium - An open-source proxy used by Cilium for configured HTTP/gRPC policy, L7 visibility and proxy-based service routing. - DNS policy uses Cilium's DNS proxy. Kafka L7 policy is no longer supported; not every L7 rule automatically deploys an Envoy instance. ## F **FQDN (Fully Qualified Domain Name)** - Fully Qualified Domain Name: an absolute name identifying its full position in the DNS tree, often written with the final root dot, for example `www.example.com.`. - Cilium `toFQDNs` permits learned destination IPs. It does not by itself authenticate an HTTPS server or constrain all HTTP Host values on a shared IP. ## G **GENEVE (Generic Network Virtualization Encapsulation)** - Encapsulation protocol for network virtualization **gRPC (gRPC Remote Procedure Call)** - High-performance RPC (Remote Procedure Call) framework developed by Google ## H **Hubble** - Cilium - Cilium's network observability layer: flow events, supported protocol metadata, metrics and query interfaces. - History is bounded and observations can be lost or filtered. Alerts, durable storage and automated response require configured integrations. ## I **Identity** - Cilium - A numeric security identifier derived from security-relevant labels. Multiple endpoints can share it within the applicable allocation scope. - CiliumIdentity's `security-labels` field is the source of truth in CRD allocation mode. Reserved and node-local identities are not all represented by these cluster-scoped objects. **IPAM (IP Address Management)** - Networking - IP Address Management: address allocation, tracking and reclamation. - Cilium modes have different allocation owners and data sources, including cluster-pool, multi-pool, Kubernetes host-scope and cloud-specific modes. A platform name is not necessarily a separate `ipam.mode` value. **IPsec** - Networking - Internet Protocol Security: a suite of mechanisms for IP-layer authentication/integrity and, with the appropriate configuration, confidentiality. - Cilium uses IPsec for supported inter-node traffic encryption; key management and path-specific limitations still apply. **Istio** - Open source platform that implements service mesh ## K **Kafka** - A distributed event-streaming platform. It remains a possible workload, but current Cilium does not provide the former Kafka topic L7 policy API. **kube-proxy** - Kubernetes - A Kubernetes component that implements Service virtual-IP/port forwarding through supported node networking mechanisms. - Cilium can replace this function with eBPF; XDP acceleration is optional and the platform must support the chosen configuration. **Kubernetes** - Open source platform that automates deployment, scaling, and management of containerized applications ## L **L2 (Layer 2)** - Data link layer of OSI model **L3 (Layer 3)** - Network layer of OSI model **L4 (Layer 4)** - Transport layer of OSI model **L7 (Layer 7)** - Application layer of OSI model **LoadBalancer** - A traffic-distribution function. Kubernetes `type: LoadBalancer` requests an implementation from a controller/provider; an external load balancer is not guaranteed without one. ## M **MAC (Media Access Control) Address** - Media Access Control address: a link-layer address associated with an interface. - Addresses may be locally administered or changed; uniqueness and authenticity must not be assumed. **mTLS (mutual TLS)** - Mutual TLS: TLS in which both peers authenticate, typically by validating each other's certificates. - Peer authentication is distinct from application authorization. Cilium's out-of-band mutual authentication and Beta ztunnel workload mTLS are separate features with different traffic-protection properties. **MTU (Maximum Transmission Unit)** - Maximum Transmission Unit: the largest network-layer packet carried on a link/interface without fragmentation, including its IP header but not the link-layer header. - Path MTU is constrained by the path; tunnel/encryption overhead affects the usable inner packet size. It is not TCP MSS or application payload size. ## N **NAT (Network Address Translation)** - Process of modifying IP address information in IP packets **NodePort** - A Kubernetes Service exposure method using an allocated node port on eligible node addresses. - Address selection, traffic policy, firewall and platform routing determine reachability; declaring a NodePort does not guarantee public access. ## O **OSI (Open Systems Interconnection) Model** - Conceptual model that classifies network communication into 7 abstract layers **Overlay Network** - Virtual network built on top of an existing network ## P **Pod** - Smallest deployable computing unit in Kubernetes **Proxy** - A component that mediates communication between peers; it need not be a separate physical server. ## R **RBAC (Role-Based Access Control)** - Method for controlling access to system resources based on roles ## S **Service** - A Kubernetes abstraction for reaching a logical set of backends, often Pods selected by labels. - Ordinary ClusterIP Services have a virtual IP; headless Services do not. ExternalName uses DNS aliasing, and selectorless Services can use manually managed EndpointSlices. **SNAT (Source Network Address Translation)** - NAT type that modifies the source IP address of packets **Socket** - An operating-system communication endpoint used for network or local inter-process communication. ## T **TCP (Transmission Control Protocol)** - Connection-oriented transport protocol that provides reliable byte streams **TLS (Transport Layer Security)** - Cryptographic protocol that protects communication over networks ## U **UDP (User Datagram Protocol)** - Connectionless transport protocol ## V **VETH (Virtual Ethernet)** - Virtual ethernet device, typically created in pairs **VNI (VXLAN Network Identifier)** - VXLAN Network Identifier: a 24-bit field in the VXLAN header. - Cilium can carry identity information in overlay metadata; the field width is not a promise of millions of independently configured tenant networks. **VTEP (VXLAN Tunnel Endpoint)** - Endpoint responsible for encapsulation and decapsulation of VXLAN packets **VXLAN (Virtual Extensible LAN)** - Networking - Network virtualization technology that overlays Layer 2 networks over Layer 3 networks - One of Cilium's overlay networking modes ## W **WireGuard** - Networking - A VPN tunnel protocol used by Cilium for supported cross-node traffic. - Same-node Pod traffic is not encrypted by its node tunnel; external traffic and optional node encryption have separate limits. Performance relative to IPsec requires a comparable measurement. ## X **XDP (eXpress Data Path)** - eBPF - eXpress Data Path: a packet-processing hook; native XDP runs in a supporting network driver's receive path. - PASS continues into the networking stack; other actions can drop, transmit or redirect. Cilium's supported XDP acceleration is optional, not a universal throughput or DDoS-protection guarantee. ## Primary References - [Cilium identities](https://raw.githubusercontent.com/cilium/cilium/v1.20.1/Documentation/internals/security-identities.rst) - [CiliumIdentity schema](https://raw.githubusercontent.com/cilium/cilium/v1.20.1/pkg/k8s/apis/cilium.io/client/crds/v2/ciliumidentities.yaml) - [Cilium Operator](https://raw.githubusercontent.com/cilium/cilium/v1.20.1/Documentation/internals/cilium_operator.rst) - [Identity management modes](https://raw.githubusercontent.com/cilium/cilium/v1.20.1/Documentation/network/kubernetes/identity-management-mode.rst) - [WireGuard](https://raw.githubusercontent.com/cilium/cilium/v1.20.1/Documentation/security/network/encryption-wireguard.rst) - [BGP](https://raw.githubusercontent.com/cilium/cilium/v1.20.1/Documentation/network/bgp-control-plane/bgp-control-plane.rst) - [Kubernetes CNI/CRI](https://kubernetes.io/docs/concepts/extend-kubernetes/compute-storage-net/network-plugins/) - [Kubernetes Services](https://kubernetes.io/docs/concepts/services-networking/service/) - [DaemonSet](https://kubernetes.io/docs/concepts/workloads/controllers/daemonset/) - [BPF ring buffer](https://docs.kernel.org/bpf/ringbuf.html) - [ARP / RFC 826](https://www.rfc-editor.org/rfc/rfc826.txt) - [IPv6 Neighbor Discovery / RFC 4861](https://www.rfc-editor.org/rfc/rfc4861.txt) - [VXLAN / RFC 7348](https://www.rfc-editor.org/rfc/rfc7348.txt) - [MAC addressing / RFC 7042](https://www.rfc-editor.org/rfc/rfc7042.txt) ## Quiz [Topic quiz](https://www.atomai.click/kubernetes-docs/en/quizzes/networking/cilium/glossary-quiz) ---------------------------------------- Source: https://www.atomai.click/kubernetes-docs/en/networking/calico/ ---------------------------------------- # Calico Deep Dive: Kubernetes Networking and Policy > **Review baseline**: Calico Open Source 3.32.2 · **Last Updated**: September 12, 2026 > Calico 3.32 is tested against Kubernetes 1.34–1.36. This is not an open-ended `3.29+ / Kubernetes 1.28+` compatibility guarantee. ## Overview Calico provides networking and network policy for Kubernetes, with additional host and VM capabilities that depend on the deployment and product edition. This series covers architecture, encapsulation and routing, BGP, policy, eBPF, EKS integration and operations. Choose a configuration using the [current requirements](https://docs.tigera.io/calico/latest/getting-started/kubernetes/requirements), not an undated maturity or resource-usage ranking. ### July 2026: Calico for VMs on Kubernetes Tigera's [official announcement](https://www.tigera.io/news/tigera-launches-ebpf-powered-calico-for-vms-on-kubernetes-vm-migration-that-doesnt-require-rebuilding-the-network/) is dated **July 23, 2026**. It describes VM/container networking, IP continuity, L2 bridge extension, policy and observability for VMware migrations. This is a product announcement, not a promise that every advertised capability is included in Calico Open Source. Check the exact edition, topology and feature status: the [Enterprise 3.23 release notes](https://docs.tigera.io/calico-enterprise/latest/release-notes/) still mark KubeVirt live migration as tech preview. Marketing availability does not remove that feature-specific limitation. ## Compatibility and feature boundaries - Calico 3.32.2 was released on August 30, 2026. Its tested Kubernetes minor versions are 1.34, 1.35 and 1.36; Kubernetes 1.37 being available does not establish compatibility. - The general Linux requirement is kernel 5.10 or later with the required modules. Consult the eBPF guide for supported architectures, vendor backports and higher requirements for individual features. - Linux data planes include iptables, nftables and eBPF. Defaults depend on installer/platform; current self-managed kubeadm operator installations can default to eBPF. There is no blanket feature-parity guarantee. - [Calico for Windows](https://docs.tigera.io/calico/latest/getting-started/kubernetes/windows-calico/limitations) supports specified IPv4 VXLAN and BGP configurations, but not Linux eBPF, IPIP, IPv6/dual stack, WireGuard or every Linux policy feature. - Open Source includes tiered policies, Goldmane flow aggregation and the Whisker UI. DNS/FQDN policy, application-layer policy and other advanced capabilities have edition boundaries in the [product comparison](https://docs.tigera.io/calico/latest/about/calico-product-editions). ## Calico and Cilium | Requirement | Calico | Cilium | |---|---|---| | Linux data plane | iptables / nftables / eBPF, depending on configuration | eBPF, with Envoy for applicable L7 functions | | Kubernetes NetworkPolicy | Supported, plus Calico policies and tiers | Supported, plus Cilium policies | | L7 / DNS policy | Check Enterprise/Cloud licensing and feature status | HTTP and DNS policy available; protocol-specific limits apply | | BGP | BIRD-based routing in the applicable networking mode | BGP control-plane advertisement; assess the required routes and topology | | Observability | Open Source Goldmane/Whisker and metrics; paid features add capabilities | Hubble and metrics | | Windows | Supported configurations with significant limitations | Cilium 1.20 agents require Linux; not a Windows beta dataplane | | kube-proxy replacement | Available with the eBPF data plane | Available when configured | | Multi-cluster / mesh | Separate features and integrations; edition-dependent | Cluster Mesh and optional service-mesh features; not all enabled by installation | Both can be production choices. Resource usage and operational complexity depend on rules, traffic, platform and tuning. Validate the required features on the target environment. Do not install two primary CNIs on one cluster merely because they work in separate environments. Cilium's [versioned requirements](https://github.com/cilium/cilium/blob/v1.20.1/Documentation/operations/system_requirements.rst) and this site's [Cilium service-mesh guide](https://www.atomai.click/kubernetes-docs/llms/en/service-mesh/cilium-service-mesh/README.md) describe its platform and mesh boundaries. ## Architecture ![Schematic Calico BGP deployment with Kubernetes datastore, optional Typha, Felix, confd and BIRD.](https://raw.githubusercontent.com/Atom-oh/kubernetes-docs/main/en/.gitbook/assets/en-networking-calico-readme-0.png) [🔍 View interactive diagram](https://www.atomai.click/kubernetes-docs/archmaps/en-networking-calico-readme-0.html) The figure is a schematic BGP deployment, not a mandatory component layout. The EKS policy-only example below uses the Kubernetes datastore and omits BIRD/confd. “Control plane” describes a logical role, not placement on EKS managed control-plane machines. Typha is a separate Deployment rather than a per-node process. | Component | Role and scope | |---|---| | Felix | Programs policy and applicable routes on workload nodes | | BIRD / confd | BGP and its configuration when that backend is enabled; absent in policy-only mode | | Typha | Optional datastore update cache/fan-out; operator scales replicas with the installation, not necessarily three | | kube-controllers | Kubernetes resource reconciliation, synchronization and cleanup | | Calico CNI / IPAM | Interface and Pod-address management when Calico owns networking; Amazon VPC CNI/IPAM retains these roles in the EKS example | | Calico API server | Aggregated `projectcalico.org/v3` API over internal CRDs in the default model; native v3 CRDs are a separate tech preview | Use the [architecture reference](https://docs.tigera.io/calico/latest/reference/architecture/overview) and the actual rendered workloads to identify enabled components. This guide uses the Kubernetes API datastore; an etcd-backed design has separate installation and feature constraints. ## Networking modes and MTU | Mode | Encapsulation and routing | Example Pod MTU with a 1500-byte IPv4 underlay | |---|---|---| | IPIP | IPv4-in-IPv4, usually with BGP route distribution | 1480 | | VXLAN | UDP 4789 by default; VXLAN Pod routing does not require BGP | 1450 | | Unencapsulated | Underlay must route Pod addresses; BGP is one way to distribute routes | 1500 | | CrossSubnet | An IPIP or VXLAN setting that encapsulates only across node subnets | Still reserve the required tunnel overhead for paths that need it | These MTUs are examples, not universal constants. IPv6 VXLAN overhead, jumbo underlays, WireGuard and cloud path limits change the calculation. IPIP supports IPv4 only, and IPv4 VXLAN is also usable where IPIP is unsuitable. Check [MTU configuration](https://docs.tigera.io/calico/latest/networking/configuring/mtu) and [overlay requirements](https://docs.tigera.io/calico/latest/networking/configuring/vxlan-ipip). BGP availability alone does not prove that every underlay hop can route Pod CIDRs; same-L2 adjacency is not a universal prerequisite for an unencapsulated routed fabric. Plan the underlay, ports, address family and platform before selecting a mode. ## EKS: retain Amazon VPC CNI and add Calico policy This example is for Linux EC2 nodes with an existing, supported Amazon VPC CNI installation. It does not replace Pod networking. It is not an Auto Mode or Fargate installation recipe. The [official EKS guide](https://docs.tigera.io/calico/latest/getting-started/kubernetes/managed-public-cloud/eks) requires: 1. Disable Amazon VPC CNI's native network-policy enforcement before selecting Calico as the policy engine; running both conflicts. For an existing protected cluster, plan and validate the policy handover rather than creating an unprotected transition. 2. Set VPC CNI `ANNOTATE_POD_IP=true` and grant its `aws-node` ServiceAccount `patch` access to Pods. Manage these settings through the installed add-on/configuration owner so reconciliation does not revert them. Check the actual ServiceAccount name before applying the additive RBAC example below. 3. Do not claim coverage for IPv6 Pods with `ENABLE_V4_EGRESS=true`: the Calico EKS guide explicitly excludes enforcement for that combination. 4. Choose **one** installation method below. These are fresh-install examples, not commands for taking over an existing operator or migrating an active CNI. ```yaml apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: calico-vpc-cni-pod-ip-patch rules: - apiGroups: [""] resources: ["pods"] verbs: ["patch"] --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: calico-vpc-cni-pod-ip-patch roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: calico-vpc-cni-pod-ip-patch subjects: - kind: ServiceAccount name: aws-node namespace: kube-system ``` ### Method A: pinned operator manifests ```bash set -euo pipefail CALICO_VERSION=v3.32.2 kubectl create -f "https://raw.githubusercontent.com/projectcalico/calico/$CALICO_VERSION/manifests/v1_crd_projectcalico_org.yaml" kubectl create -f "https://raw.githubusercontent.com/projectcalico/calico/$CALICO_VERSION/manifests/tigera-operator.yaml" kubectl -n tigera-operator rollout status deployment/tigera-operator --timeout=300s kubectl apply -f - <<'YAML' apiVersion: operator.tigera.io/v1 kind: Installation metadata: name: default spec: kubernetesProvider: EKS cni: type: AmazonVPC calicoNetwork: bgp: Disabled linuxDataplane: Iptables --- apiVersion: operator.tigera.io/v1 kind: APIServer metadata: name: default spec: {} YAML ``` ### Method B: pinned Helm installation Complete the same VPC CNI prerequisites. Calico 3.32 separates the CRD installation from the operator chart; installing only the small operator chart is insufficient for a fresh cluster. Save these values as `calico-eks-values.yaml`: ```yaml installation: kubernetesProvider: EKS cni: type: AmazonVPC calicoNetwork: bgp: Disabled linuxDataplane: Iptables apiServer: enabled: true ``` ```bash set -euo pipefail helm repo add projectcalico https://docs.tigera.io/calico/charts helm repo update projectcalico helm template calico-crds projectcalico/crd.projectcalico.org.v1 --version v3.32.2 | kubectl apply --server-side -f - helm install calico projectcalico/tigera-operator --version v3.32.2 --namespace tigera-operator --create-namespace -f calico-eks-values.yaml ``` The pinned chart also enables Goldmane and Whisker by default. Review those components and access controls in the rendered manifests. Native `projectcalico.org/v3` CRDs are a separate tech preview; the examples here use the conventional internal CRDs plus aggregated API server. ### Verify, then test policy behavior ```bash kubectl get tigerastatus kubectl -n calico-system get pods -o wide kubectl -n calico-system rollout status daemonset/calico-node --timeout=300s kubectl wait --for=condition=Available apiservice/v3.projectcalico.org --timeout=300s kubectl get felixconfigurations.projectcalico.org ``` Inspect degraded/progressing status and test both allowed and denied flows using disposable workloads before relying on enforcement. A Ready DaemonSet is not a policy proof. In AmazonVPC policy-only mode, empty Calico IPPools or absent BIRD sessions are not necessarily faults: AWS still supplies Pod IPAM and networking. ### Full Calico networking and other installation methods Full Calico networking on EKS is a separate new-cluster design. The official procedure starts without workload nodes and changes the CNI before adding them; do not apply a `cni.type: Calico` fragment over a running VPC CNI cluster. See [EKS integration](https://www.atomai.click/kubernetes-docs/llms/en/networking/calico/08-eks-integration.md) and the official EKS procedure. For self-managed clusters without an existing CNI, use the [on-premises guide](https://docs.tigera.io/calico/latest/getting-started/kubernetes/self-managed-onprem/onpremises). Direct manifests remain an alternative, but their namespace, Typha configuration and lifecycle differ from operator installs. Choose one owner rather than layering Helm, operator and `calico.yaml` installations. ## Policy examples with explicit scope Use a dedicated `calico-demo` namespace. The following ingress and egress examples select only that namespace; they are not a cluster-wide zero-trust rollout. Existing Calico tiers and earlier policies can still change the result. ```yaml apiVersion: v1 kind: Namespace metadata: name: calico-demo --- apiVersion: networking.k8s.io/v1 kind: NetworkPolicy metadata: name: allow-frontend-to-backend namespace: calico-demo spec: podSelector: matchLabels: app: backend policyTypes: [Ingress] ingress: - from: - podSelector: matchLabels: app: frontend ports: - protocol: TCP port: 8080 ``` The peer `podSelector` means frontend Pods in the **same namespace**. It does not authenticate users, allow all same-namespace traffic, or set egress policy. The next independent example restricts egress from the demo namespace to selected CoreDNS Pods on UDP/TCP 53 and denies other egress: ```yaml apiVersion: projectcalico.org/v3 kind: GlobalNetworkPolicy metadata: name: calico-demo-dns-only spec: namespaceSelector: kubernetes.io/metadata.name == 'calico-demo' selector: all() order: 100 types: [Egress] egress: - action: Allow protocol: UDP destination: namespaceSelector: kubernetes.io/metadata.name == 'kube-system' selector: k8s-app == 'kube-dns' ports: [53] - action: Allow protocol: TCP destination: namespaceSelector: kubernetes.io/metadata.name == 'kube-system' selector: k8s-app == 'kube-dns' ports: [53] - action: Deny ``` Confirm the real DNS endpoints and labels first. This selector-based example targets ordinary CoreDNS Pods; it is not a NodeLocal DNSCache or Auto Mode system-resolver policy. Port 53 alone does not identify an authorized DNS server. If you also need application egress, design and test those explicit allowances before enabling the final deny. A separate later allow cannot override a matching earlier Calico Deny. ### FQDN policy is edition-specific The `destination.domains` field used in Calico Enterprise/Cloud DNS policies is **not in the Open Source 3.32.2 NetworkPolicy schema**. Do not apply it to this Open Source installation. For an entitled deployment, use the [domain-based policy guide](https://docs.tigera.io/calico-enterprise/latest/network-policy/domain-based-policy), configure trusted DNS servers and permit the DNS path. Restrict domains deliberately: `*.amazonaws.com` would be a broad allowance, not authorization to one AWS account or service. DNS-to-IP authorization is not equivalent to validating HTTP Host or TLS identity. ## Monitoring and health ```yaml apiVersion: projectcalico.org/v3 kind: FelixConfiguration metadata: name: default spec: prometheusMetricsEnabled: true prometheusMetricsPort: 9091 ``` Metrics are disabled by default in Felix. Enabling this listener does not create a Prometheus scrape job or make it publicly safe; configure private discovery and access controls using the [metrics guide](https://docs.tigera.io/calico/latest/operations/monitor/monitor-component-metrics). `flowLogsFileEnabled` is not an Open Source FelixConfiguration field. Use the supported [Goldmane/Whisker flow-log path](https://docs.tigera.io/calico/latest/observability/view-flow-logs) instead of copying Enterprise file-log settings. | Metric | Meaning | |---|---| | `felix_active_local_endpoints` | Active local workload and host endpoints | | `felix_active_local_policies` | Policies active for endpoints on this node | | `felix_iptables_rules` | Active iptables rules; data-plane-specific | | `felix_int_dataplane_failures` | Failed dataplane updates that will be retried | | `felix_cluster_num_hosts` | Felix's cluster-wide host count; do not sum it across every Felix instance | | `typha_connections_accepted` | Cumulative accepted connections, not the current connection count | | `typha_connections_active` | Currently open client connections | See the [Felix](https://docs.tigera.io/calico/latest/reference/felix/prometheus) and [Typha](https://docs.tigera.io/calico/latest/reference/typha/prometheus) metric references. They are component health/configuration metrics, not a universal denied-packet counter. Felix health defaults to localhost:9099; Typha health commonly uses 9098 when enabled. Read the deployed probes before checking them: `curl localhost` on your laptop does not inspect a node's health server. ## Troubleshooting ```bash kubectl -n calico-system get pods -o wide kubectl -n calico-system logs -l k8s-app=calico-node -c calico-node --tail=100 kubectl get installations.operator.tigera.io default -o yaml kubectl get networkpolicies.networking.k8s.io -A kubectl get networkpolicies.projectcalico.org -A kubectl get globalnetworkpolicies.projectcalico.org kubectl get ippools.projectcalico.org -o wide ``` Operator installs normally use `calico-system`; direct manifests can use `kube-system`. Use fully qualified API resource names to distinguish Kubernetes and Calico NetworkPolicies. `kubectl get nodes ...status.conditions` is not a Calico routing-status command. BIRD status commands only apply when BGP is enabled, and `calicoctl node status` needs the appropriate Calico node environment rather than an arbitrary administrator laptop. | Symptom | Investigate before changing configuration | |---|---| | Pod has no IP | Identify the IPAM owner first: VPC CNI logs/capacity in policy-only EKS, Calico IPAM otherwise | | Cross-node failure | Routes, underlay/firewall permissions, MTU and the chosen encapsulation; enabling a tunnel blindly can worsen the outage | | Policy mismatch | Endpoint labels, namespaces, direction, tiers/order, existing policies and actual dataplane | | High CPU | Traffic/rule scale and metrics/profile evidence; an eBPF migration is a planned change, not an immediate generic fix | Use a matching-version [calicoctl](https://docs.tigera.io/calico/latest/reference/calicoctl/) only when needed, selecting the actual operating system/CPU architecture and verifying the release artifact. Never infer a policy-only failure solely from missing BGP or Calico IPAM state. ## Deep-dive contents | Part | Topic | |---|---| | [1](https://www.atomai.click/kubernetes-docs/llms/en/networking/calico/01-introduction.md) | Introduction, project history and lab setup | | [2](https://www.atomai.click/kubernetes-docs/llms/en/networking/calico/02-architecture.md) | Components, datastore and packet flow | | [3](https://www.atomai.click/kubernetes-docs/llms/en/networking/calico/03-networking-modes.md) | Encapsulation, direct routing and MTU | | [4](https://www.atomai.click/kubernetes-docs/llms/en/networking/calico/04-bgp-deep-dive.md) | BGP, route reflectors and external integration | | [5](https://www.atomai.click/kubernetes-docs/llms/en/networking/calico/05-network-policy.md) | NetworkPolicy, tiers and policy design | | [6](https://www.atomai.click/kubernetes-docs/llms/en/networking/calico/06-ebpf-dataplane.md) | eBPF setup, limitations and troubleshooting | | [7](https://www.atomai.click/kubernetes-docs/llms/en/networking/calico/07-advanced-topics.md) | Advanced networking/security topics | | [8](https://www.atomai.click/kubernetes-docs/llms/en/networking/calico/08-eks-integration.md) | EKS and VPC CNI integration | | [9](https://www.atomai.click/kubernetes-docs/llms/en/networking/calico/09-operations.md) | Operations and diagnostics | | [Glossary](https://www.atomai.click/kubernetes-docs/llms/en/networking/calico/glossary.md) | Terminology | [Calico introduction quiz](https://www.atomai.click/kubernetes-docs/en/quizzes/networking/calico/01-introduction-quiz) · [Official documentation](https://docs.tigera.io/calico/latest/about/) · [Release 3.32.2](https://github.com/projectcalico/calico/releases/tag/v3.32.2) ---------------------------------------- Source: https://www.atomai.click/kubernetes-docs/en/networking/calico/01-introduction ---------------------------------------- # Part 1: Introduction to Calico > **Review baseline**: Calico Open Source 3.32.2, kind 0.33.0, Kubernetes 1.36.4 > **Last Updated**: September 12, 2026. Calico 3.32 is tested against Kubernetes 1.34–1.36. ## Lab environment This disposable local lab selects iptables, VXLAN and Calico IPAM explicitly. It does not replace an existing CNI or configure EKS. The audit checked published artifacts and configuration without creating the cluster or testing live traffic. | Tool/environment | Requirement | |---|---| | kind | 0.33.0; pin the 1.36.4 image below instead of accepting an unpinned default | | Docker | A supported working runtime with capacity for three kind nodes | | Node OS | Linux kernel/modules meeting [Calico requirements](https://docs.tigera.io/calico/latest/getting-started/kubernetes/requirements); on macOS this is the container VM's kernel | | kubectl | Within one minor of API server 1.36; a matching 1.36 client is convenient | | calicoctl | Optional matching 3.32.2 client for the actual CLI host OS/architecture | | curl / Python 3 | Optional client download and SHA-256 verification below | | Helm | Optional alternative in the [overview](https://www.atomai.click/kubernetes-docs/llms/en/networking/calico/README.md), not needed for this lab | The [Kubernetes skew policy](https://kubernetes.io/releases/version-skew-policy/) does not support an arbitrary `kubectl 1.28+` with every later server. Check Pod/Service CIDRs against your container network, host LAN and VPN before creating the lab. ### Optional: a matching calicoctl Choose one platform, verify the exact release asset's published digest and keep the binary in the lab directory. These commands do not require global installation or home-directory configuration. ```bash set -euo pipefail CALICO_VERSION=v3.32.2 case "$(uname -s)" in Linux) CALICO_OS=linux ;; Darwin) CALICO_OS=darwin ;; *) echo "Select a supported calicoctl OS" >&2; exit 1 ;; esac case "$(uname -m)" in x86_64|amd64) CALICO_ARCH=amd64 ;; aarch64|arm64) CALICO_ARCH=arm64 ;; *) echo "Select a supported calicoctl architecture" >&2; exit 1 ;; esac CALICO_ASSET="calicoctl-$CALICO_OS-$CALICO_ARCH" curl --fail --location --retry 3 \ "https://api.github.com/repos/projectcalico/calico/releases/tags/$CALICO_VERSION" \ --output calico-release.json curl --fail --location --retry 3 \ "https://github.com/projectcalico/calico/releases/download/$CALICO_VERSION/$CALICO_ASSET" \ --output calicoctl python3 - "$CALICO_ASSET" <<'PY' import hashlib import json import pathlib import sys release = json.loads(pathlib.Path("calico-release.json").read_text()) if release["tag_name"] != "v3.32.2": raise SystemExit("Unexpected release") asset = next(a for a in release["assets"] if a["name"] == sys.argv[1]) expected = asset.get("digest") or "" actual = "sha256:" + hashlib.sha256(pathlib.Path("calicoctl").read_bytes()).hexdigest() if not expected.startswith("sha256:") or actual != expected: raise SystemExit("Digest mismatch or missing published digest") print("Verified", asset["name"], actual) PY chmod +x calicoctl ./calicoctl --help ``` Run `./calicoctl version` after configuring the lab datastore to see client and cluster information. The documented `version` command has no `--client` flag. Once the aggregated API server is ready, `kubectl` can also manage Calico resources; calicoctl is not mandatory for every operation. ### Create a separate kind cluster Use an unused cluster name and a new local kubeconfig. The [kind 0.33.0 release](https://github.com/kubernetes-sigs/kind/releases/tag/v0.33.0) publishes this 1.36.4 image within Calico's tested minor range. The registry digest and amd64/arm64 manifest were checked; node-image layers were not downloaded during the audit. ```bash set -euo pipefail CALICO_LAB_KUBECONFIG="$PWD/calico-lab.kubeconfig" test ! -e "$CALICO_LAB_KUBECONFIG" cat > kind-calico.yaml <<'YAML' kind: Cluster apiVersion: kind.x-k8s.io/v1alpha4 networking: disableDefaultCNI: true kubeProxyMode: iptables podSubnet: 10.244.0.0/16 nodes: - role: control-plane - role: worker - role: worker YAML kind create cluster --name calico-lab --config kind-calico.yaml \ --kubeconfig "$CALICO_LAB_KUBECONFIG" \ --image kindest/node:v1.36.4@sha256:099e049362a1526b2db71494e1947aae99bd16290d7c895f2b7ea312e3cbfaed export KUBECONFIG="$CALICO_LAB_KUBECONFIG" export DATASTORE_TYPE=kubernetes kubectl config current-context kubectl cluster-info ``` Nodes and ordinary Pods may remain unready until the CNI is installed. Do not install a second CNI to clear that condition. If the Pod CIDR conflicts, change it in both kind and the Installation before creating the cluster. ```bash CALICO_VERSION=v3.32.2 kubectl create -f "https://raw.githubusercontent.com/projectcalico/calico/$CALICO_VERSION/manifests/v1_crd_projectcalico_org.yaml" kubectl create -f "https://raw.githubusercontent.com/projectcalico/calico/$CALICO_VERSION/manifests/tigera-operator.yaml" kubectl -n tigera-operator rollout status deployment/tigera-operator --timeout=300s kubectl apply -f - <<'YAML' apiVersion: operator.tigera.io/v1 kind: Installation metadata: name: default spec: kubernetesProvider: Kind cni: type: Calico calicoNetwork: linuxDataplane: Iptables bgp: Disabled ipPools: - cidr: 10.244.0.0/16 blockSize: 26 encapsulation: VXLAN natOutgoing: Enabled nodeSelector: all() --- apiVersion: operator.tigera.io/v1 kind: APIServer metadata: name: default spec: {} YAML kubectl get tigerastatus kubectl -n calico-system get pods -o wide ``` Wait for the operator-created workloads to appear, then check their rollouts and conditions. An empty label selection or one controller's availability does not establish that all node networking works. ```bash kubectl -n calico-system rollout status daemonset/calico-node --timeout=300s kubectl -n calico-system rollout status deployment/calico-kube-controllers --timeout=300s kubectl wait --for=condition=Available apiservice/v3.projectcalico.org --timeout=300s kubectl wait --for=condition=Ready nodes --all --timeout=300s kubectl get ippools.projectcalico.org -o wide kubectl get installations.operator.tigera.io default -o yaml # Optional, if the matching local client was downloaded: ./calicoctl version ./calicoctl get nodes ``` BGP is disabled here, so BIRD sessions and `calicoctl node status` are not readiness criteria. That command also needs the appropriate node environment rather than only a laptop kubeconfig. Observe actual component counts; CSI/Typha replicas are not fixed. Use disposable workloads to check Pod, Service and DNS connectivity and both permitted and denied policy flows. ## What Calico provides Calico combines Kubernetes networking, IPAM and policy enforcement. In policy-only integrations, another CNI retains networking and IPAM. Features vary by operating system, data plane and product edition; a platform listing does not promise identical behavior. ## Project history and governance Project Calico began at Metaswitch in 2014; Tigera was established in 2016 and is its primary maintainer. The release records below correct the earlier 3.0/3.29 dates and distinguish the original eBPF preview from later feature availability. | Date | Primary release record | |---|---| | December 21, 2017 | [Calico 3.0.0](https://github.com/projectcalico/calico/releases/tag/v3.0.0), a historical release, not an installation recommendation | | February 25, 2020 | [eBPF introduction](https://www.tigera.io/blog/introducing-the-calico-ebpf-dataplane/): announced as a **3.13 tech preview**, not GA | | October 29, 2024 | [Calico 3.29.0](https://github.com/projectcalico/calico/releases/tag/v3.29.0) | | August 30, 2026 | [Calico 3.32.2](https://github.com/projectcalico/calico/releases/tag/v3.32.2), this review's baseline | The old timeline's “full eBPF parity” and “Windows eBPF” claims were incorrect. Current [Windows limitations](https://docs.tigera.io/calico/latest/getting-started/kubernetes/windows-calico/limitations) still exclude Linux eBPF, IPIP, IPv6/dual stack and WireGuard. Calico uses Apache-2.0 licensing with Tigera and community maintenance. A CNCF Landscape listing is not CNCF ownership, incubation or graduation. Enterprise is a commercial self-managed product; Cloud is a SaaS offering. Open Source is not restricted to small or non-production clusters. ![Calico ecosystem and commercial product relationships.](https://raw.githubusercontent.com/Atom-oh/kubernetes-docs/main/en/.gitbook/assets/en-networking-calico-01-introduction-4.png) [🔍 View interactive diagram](https://www.atomai.click/kubernetes-docs/archmaps/en-networking-calico-01-introduction-4.html) The CNCF box represents Landscape/ecosystem participation only. Tigera maintains the open-source project as well as its products; the figure's grouping does not confer governance authority on CNCF. ## Core capabilities ### 1. Networking and data planes Encapsulation and implementation are separate choices. Calico can use IPIP, VXLAN or a routed underlay. CrossSubnet is a conditional IPIP/VXLAN setting, not a WAN connection service. Linux data planes include iptables, nftables and eBPF. eBPF runs **inside the kernel** and can bypass parts of its conventional packet-processing path; it does not bypass the kernel. Unencapsulated routing avoids tunnel headers only when the underlay has the required Pod routes, without guaranteeing the lowest latency for every workload. ### 2. Kubernetes and Calico policy Kubernetes NetworkPolicy is namespaced and additive. Calico adds explicit actions, ordered policies and tiers, including tiers in Open Source. GlobalNetworkPolicy has cluster resource scope but can select one namespace. HostEndpoint describes a host endpoint to protect; it is not a third policy type below NetworkPolicy in a fixed hierarchy. These are **independent examples** in a dedicated namespace. Consider existing tiers and higher-priority policies; neither is a complete security baseline. ```yaml apiVersion: v1 kind: Namespace metadata: name: calico-demo --- apiVersion: networking.k8s.io/v1 kind: NetworkPolicy metadata: name: default-deny-ingress namespace: calico-demo spec: podSelector: {} policyTypes: [Ingress] ingress: [] ``` ```yaml apiVersion: projectcalico.org/v3 kind: GlobalNetworkPolicy metadata: name: calico-demo-trusted-ingress spec: namespaceSelector: kubernetes.io/metadata.name == 'calico-demo' selector: app == 'backend' order: 100 types: [Ingress] ingress: - action: Allow protocol: TCP source: namespaceSelector: kubernetes.io/metadata.name == 'calico-demo' selector: trusted == 'true' destination: ports: [8080] - action: Deny ``` The Calico example allows TCP 8080 to selected backends from matching demo-namespace endpoints, then denies other ingress. Protect label-writing permissions: `trusted` is not cryptographic identity. These examples do not configure egress or DNS. CIDR/port rules are supported, but a large private CIDR is not an identity boundary. DNS/FQDN and application-layer policy require the appropriate Enterprise/Cloud features; see the [edition matrix](https://docs.tigera.io/calico/latest/about/calico-product-editions). ### 3. IP address management When Calico owns IPAM, pools and blocks control allocation. An IPv4 /26 block contains 64 addresses, not 64 guaranteed usable Pod addresses on every platform; Windows reserves addresses and IPv6 has different defaults. In VPC CNI policy-only mode, AWS owns IPAM. This illustrates the [IPPool API](https://docs.tigera.io/calico/latest/reference/resources/ippool). **Do not create it beside an overlapping operator-managed pool.** The kind lab already has its pool; encapsulation/IPAM changes are separate planned exercises. ```yaml apiVersion: projectcalico.org/v3 kind: IPPool metadata: name: example-ipv4-pool spec: cidr: 10.244.0.0/16 blockSize: 26 ipipMode: Never vxlanMode: Always natOutgoing: true nodeSelector: all() ``` Multiple non-overlapping pools and node selectors can separate allocations. `natOutgoing` normally applies to traffic leaving Calico pools; it is not a firewall or encryption setting. Neither direct routing nor CrossSubnet connects separate sites without an underlay design. ### 4. BGP routing BGP distributes routes; application packets do not flow through the BIRD process, and BGP does not encrypt them. BGP can support direct routing or coexist with IPIP. Full mesh, route reflectors and external peers are topology choices. The following belongs to a **separate routed lab**, not the BGP-disabled kind example. Replace the documentation address, ASNs and node labels with a designed topology and matching router configuration. Do not disable the node mesh before replacement route distribution works. ```yaml apiVersion: projectcalico.org/v3 kind: BGPConfiguration metadata: name: default spec: logSeverityScreen: Info nodeToNodeMeshEnabled: true asNumber: 64512 --- apiVersion: projectcalico.org/v3 kind: BGPPeer metadata: name: example-rack-tor spec: peerIP: 192.0.2.1 asNumber: 64513 nodeSelector: rack == 'rack-1' ``` BGPPeer supports `password.secretKeyRef` for session authentication. The Secret belongs in the Calico node component's namespace and the router must use matching credentials; this does not encrypt workload traffic. Service-CIDR advertisement and mesh removal require additional testing; see [BGP deep dive](https://www.atomai.click/kubernetes-docs/llms/en/networking/calico/04-bgp-deep-dive.md). ### 5. Platform and scale boundaries | Environment | Boundary | |---|---| | EKS | VPC CNI + Calico policy is one integration; full Calico CNI is a separate new-cluster design | | AKS | Check the provider's supported CNI/policy combination and current installation procedure | | GKE | Dataplane V2 uses **Cilium**; Calico applies to the relevant legacy configuration, not an installation over V2 | | Self-managed Kubernetes | Check distribution, kernel, CNI ownership, routes and privileges | | Windows | Specified IPv4 configurations; no Linux eBPF, IPIP, IPv6/dual-stack or WireGuard parity | | Hosts / VMs | Separate installation and feature requirements; KubeVirt/Enterprise status differs from basic host protection | [GKE's documentation](https://cloud.google.com/kubernetes-engine/docs/concepts/dataplane-v2) explicitly distinguishes Cilium in V2 from the legacy Calico path. Typha caches and distributes updates through a separate set of Pods, reducing direct Felix datastore watches. Three replicas are an example, not a universal minimum. Capacity depends on policies, endpoints, Service churn, hardware, datastore and data plane. This introduction has no reproducible evidence for a fixed “5,000 nodes / 100,000 Pods / millions of rules” limit. ## Calico, kube-proxy and performance kube-proxy implements Service forwarding, not CNI networking or NetworkPolicy. Calico's standard data planes can work alongside it, as in this lab; the eBPF data plane can replace Service handling when configured. | Concern | Compare | |---|---| | Pod networking/IPAM | CNI/IPAM implementations with the same topology | | Service forwarding | Selected kube-proxy backend or an eBPF replacement | | Policy | Equivalent rules and enforcement coverage | | Scale | Services/endpoints, selectors, churn and connection reuse | | CPU/memory/latency | Hardware, kernel, versions, workload, warm-up, repeats and errors | kube-proxy is not iptables-only: current Kubernetes also offers nftables and version-dependent legacy backends. An IP-set lookup does not make the entire Calico packet path O(1). Initial iptables Service NAT selection also differs from later packets' conntrack fast path. The earlier unsourced 1,000-node/50,000-Pod rule-count, latency and memory example was not a reproducible benchmark and should not be used for sizing. Traditional VM networks can also be automated and distributed. Calico's declarative policy does not imply unlimited IP capacity or guaranteed second-level convergence. ## Deployment scenarios - **On-premises**: coordinate Pod routes, BGP peers/filters, return paths and host protection. Disabling encapsulation alone does not create underlay routes. - **EKS**: to retain AWS networking, select `cni.type: AmazonVPC` and follow the [reviewed overview](https://www.atomai.click/kubernetes-docs/llms/en/networking/calico/README.md), including policy-engine ownership and Pod-IP annotations. Do not apply an EKS Installation to this Kind lab or run two policy engines. - **Hybrid/multi-cluster**: connectivity, discovery and policy administration are separate functions. A CrossSubnet IPPool does not establish VPNs, shared identity or cross-cluster discovery. Evaluate the appropriate cluster-mesh/multi-cluster product features and underlay separately; “Calico Federation” is not a universal built-in link. - **Regulated workloads**: Enterprise/Cloud can add reports, logs and security features; installing them does not establish compliance. API audit logs record API changes and flow logs record network observations, not automatically every enforcement decision. WireGuard is also available in supported Open Source Linux configurations. ## Community and source development Use the [community page](https://www.tigera.io/project-calico/community/) for current Slack/meeting links, the [issue tracker](https://github.com/projectcalico/calico/issues) for reproducible reports, and the [contributor guide](https://github.com/projectcalico/calico/blob/v3.32.2/CONTRIBUTING.md). Do not assume an undated biweekly schedule or old forum URL is current. For source study, the [developer guide](https://github.com/projectcalico/calico/blob/v3.32.2/DEVELOPER_GUIDE.md) describes a Linux/Docker/git/make environment and component-specific tests. There is no root `make dev-environment` target. This optional source workflow is separate from the networking lab and was not executed during the audit: ```bash git clone --depth 1 --branch v3.32.2 https://github.com/projectcalico/calico.git calico-source-study cd calico-source-study # Read prerequisites and the selected component's Makefile before running tests. cat DEVELOPER_GUIDE.md make -C calicoctl test ``` Open Source provides community-supported networking and policy for production as well as labs. Enterprise adds commercial capabilities/support; Cloud delivers SaaS management. Select by the [feature matrix](https://docs.tigera.io/calico/latest/about/calico-product-editions), not a blanket “small versus large cluster” rule. ## Clean up the disposable lab After saving results, remove only the `calico-lab` cluster created for this exercise with `kind delete cluster --name calico-lab`. Keep any unrelated clusters and kubeconfigs. This local cleanup is not an EKS deletion procedure. [Next: Calico architecture](https://www.atomai.click/kubernetes-docs/llms/en/networking/calico/02-architecture.md) · [Calico overview](https://www.atomai.click/kubernetes-docs/llms/en/networking/calico/README.md) · [Introduction quiz](https://www.atomai.click/kubernetes-docs/en/quizzes/networking/calico/01-introduction-quiz) ---------------------------------------- Source: https://www.atomai.click/kubernetes-docs/en/networking/calico/02-architecture ---------------------------------------- # Part 2: Architecture > **Review baseline**: Calico Open Source 3.32.2 / operator 1.42.6; Calico 3.32 is tested against Kubernetes 1.34–1.36. > **Last Updated**: September 12, 2026. Examples are configuration references, not a live-cluster validation. ## Overview This section provides an in-depth exploration of Calico's architecture. Understanding how each component works and interacts is essential for effective deployment, troubleshooting, and optimization of Calico in production environments. ## Full Architecture Diagram ![Simplified Kubernetes API and Typha state fan-out toward Felix and the BGP configuration path, with intermediate components omitted.](https://raw.githubusercontent.com/Atom-oh/kubernetes-docs/main/en/.gitbook/assets/en-networking-calico-02-architecture-0.png) [🔍 View interactive diagram](https://www.atomai.click/kubernetes-docs/archmaps/en-networking-calico-02-architecture-0.html) This is a simplified control-state diagram. The BIRD edge omits confd, which renders its configuration; Typha is not a direct BIRD configuration API. BIRD/confd and Typha depend on the installation mode, and control-plane components are not all shown. ## Felix: The Calico Agent Felix runs in the Calico node agent on selected workload nodes and programs applicable routes, interface settings and policy in the kernel. In the Linux full-networking path, the container runtime invokes the CNI chain, and the CNI/IPAM plugins create interfaces and allocate addresses. Felix observes endpoint changes asynchronously; it is not the handler of a direct CNI ADD call. Operator, platform and networking mode determine the exact components. ### Felix Responsibilities Linux CNI/IPAM creates Pod interfaces and addresses. Felix reconciles endpoint state and kernel policy. HTTP health serving and datastore status reporting are separate functions. ### Core Functions 1. **Route Programming**: Reconciles applicable workload and tunnel routes; BIRD's kernel protocol also installs learned routes in BGP mode 2. **ACL Enforcement**: Programs iptables/nftables/eBPF rules for network policies 3. **Interface Management**: Reconciles endpoint interface state and relevant kernel settings; Linux veth creation belongs to the CNI path 4. **Health Reporting**: Reports node and endpoint health to the datastore 5. **Endpoint reconciliation**: Watches workload endpoint state and programs applicable policy/routes; the CNI/IPAM plugins allocate addresses and create Linux Pod interfaces ### Felix Data Plane Options Felix supports multiple data plane backends: | Data Plane | Description | Best For | | ------------ | -------------------------- | ------------------------------------------- | | **iptables** | Traditional Linux firewall | Compatibility, mature deployments | | **nftables** | Native nftables implementation | Check supported kernel, platform and feature set | | **eBPF** | In-kernel programmable | Optional Service handling; requires a coordinated migration and supported features | ### FelixConfiguration Resource ```yaml apiVersion: projectcalico.org/v3 kind: FelixConfiguration metadata: name: default spec: logSeverityScreen: Info healthEnabled: true healthPort: 9099 prometheusMetricsEnabled: true prometheusMetricsPort: 9091 reportingInterval: 30s reportingTTL: 90s ``` This minimal example uses fields accepted by Calico 3.32.2. Apply changes through the configuration owner; it is not a data-plane migration or performance-tuning recipe. Felix's health host defaults to localhost. Enabling metrics does not configure a Prometheus scrape or make public exposure appropriate. | Configuration concern | Correct owner / interpretation | |---|---| | Linux data plane | Operator `Installation.spec.calicoNetwork.linuxDataplane` selects `Iptables`, `Nftables` or `BPF` for the supported configuration | | `bpfEnabled` | Low-level Felix setting; coordinate an operator-managed transition and kube-proxy/API reachability rather than patching this alone | | `iptablesBackend: NFT` | Selects the iptables-nft tool backend, not the native Calico nftables data plane | | Connect-time load balancing | Current field is `bpfConnectTimeLoadBalancing: TCP`, `Enabled` or `Disabled`; the older boolean `bpfConnectTimeLoadBalancingEnabled` is still accepted but deprecated | | Node address detection | Operator `calicoNetwork.nodeAddressAutodetectionV4` / `V6`, or the node startup environment in a manifest-managed install; not Felix fields named `ipAutoDetectionMethod` or `ipv6AutoDetectionMethod` | | Flow visibility | Use the supported Goldmane/Whisker configuration; Open Source does not accept the Enterprise file-log fields shown in the previous example | | MTU and tunnel modes | Derive from the underlay, encapsulation and encryption; coordinate Installation/IPPool settings rather than arbitrarily setting 1440/1410/1420 or enabling every tunnel | | Host failsafe ports | Review actual API/BGP/etcd/administrative reachability before replacing the default lists; the old shortened lists could remove needed exceptions | | Durations | Use current names such as `reportingInterval`, `reportingTTL`, `iptablesPostWriteCheckInterval` and `iptablesLockProbeInterval`; do not mechanically append `Secs`/`Millis` | The released schema rejects the old `iptablesLockFilePath`, `iptablesLockTimeoutSecs`, `iptablesLockProbeIntervalMillis`, `iptablesPostWriteCheckIntervalSecs`, `reportingIntervalSecs` and `reportingTTLSecs` names. Consult the [Felix resource reference](https://docs.tigera.io/calico/latest/reference/resources/felixconfig) and [operator API](https://docs.tigera.io/calico/latest/reference/installation/api). Address or data-plane changes require their own rollout checks. ### Felix iptables Rule Structure The following are selected prefixes from the [released rule definitions](https://github.com/projectcalico/calico/blob/v3.32.2/felix/rules/rule_defs.go), not the complete chain graph. They describe the iptables data plane; inspect actual rules for the installed mode and configuration. | Chain/prefix | Role | |---|---| | `cali-FORWARD` | Calico forwarding hook | | `cali-from-wl-dispatch` | Dispatch from workload interfaces | | `cali-to-wl-dispatch` | Dispatch to workload interfaces | | `cali-fw-…` / `cali-tw-…` | Per-workload directional chains | | `cali-pi-…` / `cali-po-…` | Inbound/outbound policy chains | ### Felix Data Flow On Pod creation, the runtime invokes the CNI/IPAM chain, which configures the network and records endpoint state. Felix observes relevant changes and programs policy/routes; BGP configuration follows its own confd/BIRD path when enabled. Pod Running does not prove routing or policy convergence. ## BIRD: BGP Routing Daemon BIRD (BIRD Internet Routing Daemon) exchanges BGP routes when Calico's BGP backend is enabled. BIRD/confd are not mandatory in a policy-only or BGP-disabled VXLAN installation. The following topology examples require an appropriately designed BGP-enabled cluster; they are not additions to the BGP-disabled introductory kind lab. ### BIRD in Calico Architecture ![Diagram showing BIRD on each of three nodes forming a full iBGP mesh to exchange pod routes, then peering over eBGP with the top-of-rack switch, which passes those routes on to the core router.](https://raw.githubusercontent.com/Atom-oh/kubernetes-docs/main/en/.gitbook/assets/en-networking-calico-02-architecture-3.png) [🔍 View interactive diagram](https://www.atomai.click/kubernetes-docs/archmaps/en-networking-calico-02-architecture-3.html) The lines represent BGP sessions, not application-packet transit through BIRD. The size labels are illustrative guidance, not a protocol requirement or a fixed threshold for route reflectors. ### BGP Session Types | Session Type | Use Case | Configuration | | --------------------- | --------------------------- | ---------------------- | | **Node-to-Node Mesh** | Default for small clusters | Automatic, full mesh | | **Route Reflector** | Reduce mesh session count as topology requires | Configure and verify replacement peers first | | **External Peering** | On-premises integration | Manual BGP peer config | ### BGP Configuration Examples #### Node-to-Node Mesh (Default) ```yaml apiVersion: projectcalico.org/v3 kind: BGPConfiguration metadata: name: default spec: logSeverityScreen: Info nodeToNodeMeshEnabled: true asNumber: 64512 ``` #### Route Reflector Configuration Use the [official BGP transition procedure](https://docs.tigera.io/calico/latest/networking/configuring/bgp). Assigning a route-reflector cluster ID immediately removes that node from the existing node mesh and can disrupt workloads. Prepare dedicated nodes without application workloads, or plan an explicit maintenance migration. Do not replace an existing Calico Node with a partial object that omits its other settings. For the Kubernetes API datastore, the documented node annotation preserves the existing Node fields. Replace these example names with the prepared nodes: ```bash # Existing, prepared RR nodes with no application workloads. kubectl get nodes rr-1 rr-2 -o yaml > rr-nodes-before.yaml kubectl get bgpconfiguration.projectcalico.org default -o yaml > bgp-before.yaml kubectl annotate node rr-1 projectcalico.org/RouteReflectorClusterID=244.0.0.1 --overwrite kubectl annotate node rr-2 projectcalico.org/RouteReflectorClusterID=244.0.0.2 --overwrite kubectl label nodes rr-1 rr-2 route-reflector=true --overwrite kubectl apply -f - <<'YAML' apiVersion: projectcalico.org/v3 kind: BGPPeer metadata: name: nodes-to-route-reflectors spec: nodeSelector: all() peerSelector: route-reflector == 'true' YAML ``` `all()` to the RR selector covers clients and RR-to-RR peering; verify both reflectors and the client routes. Wait for established sessions and confirm actual reachability before disabling the old node mesh. An Established session alone does not prove that the required routes were accepted. ```bash # Only after replacement sessions, routes and test traffic have been verified. kubectl patch bgpconfiguration.projectcalico.org default --type merge \ -p '{"spec":{"nodeToNodeMeshEnabled":false}}' ``` This is an ordered transition, not an instruction to apply every block at once or a guarantee of no disruption. Keep the saved configuration and a tested recovery path. Addresses, ASNs and any reused AS numbers in external-fabric examples require deliberate route-policy and AS-loop handling. #### External BGP Peering Replace the example peer address, ASNs and rack selector with the planned topology. The password reference requires a matching Secret/key in the Calico node component's namespace and matching router configuration. It authenticates the BGP session, not the workload payload. ```yaml apiVersion: projectcalico.org/v3 kind: BGPPeer metadata: name: tor-switch-peer spec: peerIP: 10.0.0.1 asNumber: 65001 nodeSelector: rack == 'rack-1' password: secretKeyRef: name: bgp-passwords key: tor-password sourceAddress: UseNodeIP keepOriginalNextHop: false ``` ### Route Propagation Process ![Diagram showing Felix adding a route to the kernel routing table, BIRD picking up that route info through its BGP session management, and its route exchange function advertising the Pod CIDR to other nodes and external routers via a BGP UPDATE, with Route Reflector support for large clusters and export-filter-based route filtering shown as further BIRD functions.](https://raw.githubusercontent.com/Atom-oh/kubernetes-docs/main/en/.gitbook/assets/en-networking-calico-02-architecture-4.png) [🔍 View interactive diagram](https://www.atomai.click/kubernetes-docs/archmaps/en-networking-calico-02-architecture-4.html) This shows one route-information path. BIRD's kernel protocol can also install learned routes, while confd/IPAM data contributes to generated routing configuration. BGP route filters are routing policy, not Kubernetes NetworkPolicy enforcement. ### BIRD Status Commands Select a node where BIRD is running. The released [startup script](https://github.com/projectcalico/calico/blob/v3.32.2/node/filesystem/etc/service/available/bird/run) sets the IPv4 control socket below. A manifest-managed installation may use another namespace. ```bash CALICO_NODE=worker-node-name CALICO_POD=$(kubectl -n calico-system get pods -l k8s-app=calico-node \ --field-selector "spec.nodeName=$CALICO_NODE" -o jsonpath='{.items[0].metadata.name}') : "${CALICO_POD:?No Calico Pod on the selected node}" kubectl -n calico-system exec "$CALICO_POD" -c calico-node -- \ birdcl -s /var/run/calico/bird.ctl show protocols kubectl -n calico-system exec "$CALICO_POD" -c calico-node -- \ birdcl -s /var/run/calico/bird.ctl show route ``` Use the actual protocol names and prefixes from the output for more detailed queries. Commands and sample console output are different things; the former `birdcl>` prompts were not Bash commands. These read-only checks do not configure routing. ## confd: Configuration Management confd is a lightweight configuration management tool that watches the Calico datastore and generates BIRD configuration files. ### confd Workflow confd watches the relevant BGP configuration, renders its template, checks the candidate and signals BIRD to reload. ### confd Template Processing Use the [released template](https://github.com/projectcalico/calico/blob/v3.32.2/confd/etc/calico/confd/templates/bird.cfg.template), not an invented `.NodeIP` / `.BGPPeers` data structure. This excerpt illustrates kernel synchronization; its filter and surrounding configuration are defined elsewhere, so it is not a complete `bird.cfg`. ```text protocol kernel { learn; persist; scan time 2; import all; export filter calico_kernel_programming; graceful restart; merge paths on; } ``` The [confd template definition](https://github.com/projectcalico/calico/blob/v3.32.2/confd/etc/calico/confd/conf.d/bird.toml) writes `/etc/calico/confd/config/bird.cfg`, validates the candidate with `bird -p -c {{.src}}`, and uses `sv hup bird || true` as its configured reload action. This establishes that BIRD can export selected learned routes to the kernel; it is not merely receiving every route from Felix. Reload and graceful-restart behavior still need status and traffic checks. Manage BGP settings through their API owner rather than editing the generated file. ## Typha: Scaling Component Typha is a fan-out proxy that sits between the Kubernetes API server and Felix agents. It reduces load on the API server by caching and distributing datastore updates. ### Why Typha? Typha reduces repeated datastore update processing by caching state and streaming changes to multiple clients. Installation ownership, TLS and actual scaling logic matter as well as node count. ### Typha scaling in operator 1.42.6 The operator deploys and scales Typha; there is no universal “only above 50 nodes” rule. The pinned [autoscaler implementation](https://github.com/tigera/operator/blob/v1.42.6/pkg/controller/installation/typha_autoscaler.go) counts nodes that are not marked unschedulable, excludes AKS virtual nodes, and separately checks for enough Linux nodes to place the desired replicas. Taints and other placement constraints still matter. The actual [scale function](https://github.com/tigera/operator/blob/v1.42.6/pkg/common/autoscale.go), rather than its abbreviated comment, returns: - 1 replica for 1–2 counted nodes. - 2 replicas for 3–4 counted nodes. - `max(3, floor(N / 200) + 2)` for 5 or more counted nodes. | Counted nodes | Desired replicas in this version | |---|---| | 50 | 3 | | 200 | 3 | | 500 | 4 | | 1,000 | 7 | | 2,000 | 12 | This is a version-specific desired count, not a per-replica capacity guarantee or a recommendation for every installation. Non-cluster-host mode uses a separate eligible HostEndpoint count. The former `max(3, ceil(N / 200))` table did not describe this operator. ### Operator-managed Typha configuration Keep the operator's Deployment, ServiceAccount/RBAC, Service, disruption budget and TLS configuration together. The hand-written Deployment formerly shown here omitted essential dependencies and could overwrite operator-managed settings. Felix-to-Typha TLS uses a trusted CA, Typha server certificate/key and the expected Felix client identity. Port 5473 is the default sync port, not a user-traffic proxy. ```bash # Change the operator's supported setting through its API. kubectl patch installation.operator.tigera.io default --type merge \ -p '{"spec":{"typhaMetricsPort":9093}}' kubectl -n calico-system get deployment calico-typha -o yaml kubectl -n calico-system get service calico-typha -o yaml kubectl -n calico-system get pdb ``` Typha's health endpoint defaults to localhost:9098. This operator derives the health port as the configured Felix health port minus one and configures probes accordingly. A Pod-network Deployment whose probe targets the Pod IP will not reach a listener bound only to localhost; copying probes without their network/bind settings is unsafe. The operator source supplies TLS mounts and client-identity settings that are absent from the old standalone example. ### Typha Fan-out Architecture Each Typha maintains cached state for its client streams. Client grouping in a diagram is not a fixed per-instance capacity specification. ## kube-controllers: Kubernetes Integration calico-kube-controllers runs selected reconciliation functions. Which controllers run depends on the datastore, edition and installation configuration. Policy/namespace/service-account projection into an etcd datastore is different from Kubernetes API datastore handling. ### Available controller roles | Controller | Purpose | | ------------------------------- | ------------------------------------------------- | | **Node Controller** | Syncs Kubernetes nodes with Calico node resources | | **Policy Controller** | Syncs Kubernetes NetworkPolicy with Calico policy | | **Namespace Controller** | Syncs namespace labels for profile management | | **ServiceAccount Controller** | Projects service-account labels into Calico profiles; does not grant Kubernetes RBAC | | **WorkloadEndpoint Controller** | Updates workload endpoint metadata such as Pod labels on the applicable datastore path | ### Controller Reconciliation Loop ![Sequence diagram showing kube-controllers repeatedly listing Kubernetes and Calico resources, diffing them, and either writing changes to the Calico datastore or taking no action when the two are already in sync.](https://raw.githubusercontent.com/Atom-oh/kubernetes-docs/main/en/.gitbook/assets/en-networking-calico-02-architecture-8.png) [🔍 View interactive diagram](https://www.atomai.click/kubernetes-docs/archmaps/en-networking-calico-02-architecture-8.html) This is a logical desired-versus-observed reconciliation sketch, not a trace proving two remote LIST calls every interval. Real controllers use watches/caches, and their enabled roles depend on the datastore and installation. ### kube-controllers Configuration For the operator installation, configure the real [KubeControllersConfiguration API](https://docs.tigera.io/calico/latest/reference/resources/kubecontrollersconfig). An arbitrary ConfigMap named `calico-kube-controllers-config` is not consumed by the Deployment shown in this guide. ```bash kubectl get kubecontrollersconfiguration.projectcalico.org default -o yaml kubectl patch kubecontrollersconfiguration.projectcalico.org default --type merge \ -p '{"spec":{"logSeverityScreen":"Info","healthChecks":"Enabled","prometheusMetricsPort":9094}}' ``` This merge patch preserves the existing `controllers` configuration. If GitOps manages the resource, make the equivalent change in its desired state instead. A replacement manifest with empty controller objects can alter existing reconciliation or allocation settings. Operator 1.42.6 selects `ENABLED_CONTROLLERS=node,loadbalancer` for its standard Open Source deployment. The broader list above describes available controller roles, not five controllers necessarily running with every datastore. Its [renderer](https://github.com/tigera/operator/blob/v1.42.6/pkg/render/kubecontrollers/kube-controllers.go) specifies one replica and a `Recreate` strategy; the previous leader-election claim was not supported by that configuration. Keep this workload under the installation owner rather than replacing or manually scaling it. ## Datastore Options The operator examples here use the Kubernetes API datastore. Calico state can involve Calico CRDs and native Kubernetes objects; not every logical Calico resource is a separate CRD. The usual aggregated API server exposes `projectcalico.org/v3` over the internal representation. Native v3 CRDs are a separate Calico 3.32 tech preview and have their own migration procedure. Typha distributes read/watch updates; it is not a general write proxy for Felix. Components that update status or resources use their own datastore access. Kubernetes persists its API state in its backing store, but Calico users do not need a separate Calico etcd cluster for this mode. Direct etcdv3 access is a different installation choice with explicit support and feature constraints. Do not infer that it is faster, unlimited, or required above 5,000 nodes. The eBPF data plane requires the Kubernetes datastore. A direct-etcd deployment also needs its own TLS trust, credentials, availability and consistent backup/restore design. | Concern | Kubernetes API datastore | Direct etcdv3 | |---|---|---| | Access control | Kubernetes authentication/RBAC plus the appropriate Calico API path | etcd authentication/TLS and access controls | | Operations | Reuse the cluster API; follow provider-specific backup procedures | Operate and back up the selected etcd deployment | | Host/VM support | Check the specific installation and edition | Check the specific installation and edition | | Selection | Used by this operator guide | A separately validated design, not a node-count shortcut | On managed Kubernetes, “Kubernetes backup” does not mean users can take direct control-plane etcd snapshots. Back up supported resources using the platform's procedure. ## Component Interaction Sequence Kubelet requests sandbox creation through the container runtime, which invokes CNI/IPAM. Endpoint and policy data reaches Felix through the selected datastore/watch path. In BGP mode, confd and BIRD handle routing configuration separately. These components converge asynchronously; verify actual connectivity and enforcement. ## Packet Flow Analysis ### Ingress Packet Flow (Pod-to-Pod, Same Node) ![Diagram showing a packet crossing from one pod to another on the same node through their veth interfaces and the host's iptables/eBPF policy check.](https://raw.githubusercontent.com/Atom-oh/kubernetes-docs/main/en/.gitbook/assets/en-networking-calico-02-architecture-12.png) [🔍 View interactive diagram](https://www.atomai.click/kubernetes-docs/archmaps/en-networking-calico-02-architecture-12.html) The policy box summarizes the applicable source-egress and destination-ingress kernel checks. The veth interfaces belong to the two Pods' network paths; packets are not sent through the Felix process. ### Egress Packet Flow (Pod-to-Pod, Different Nodes with IPIP) ![Sequence diagram showing a packet from Pod A passing the Felix/iptables egress policy check on Node 1, reaching Node 2 either IPIP/VXLAN-encapsulated or forwarded directly via a BGP route, then passing the ingress policy check and reaching Pod B.](https://raw.githubusercontent.com/Atom-oh/kubernetes-docs/main/en/.gitbook/assets/en-networking-calico-02-architecture-13.png) [🔍 View interactive diagram](https://www.atomai.click/kubernetes-docs/archmaps/en-networking-calico-02-architecture-13.html) Read the two paths as alternatives. “Felix/iptables” means kernel rules programmed by Felix, not daemon packet forwarding. BIRD supplies routing control information in BGP mode; it does not carry application packets. ### Packet Structure Comparison ``` Original Pod-to-Pod Packet: ┌─────────────────────────────────────────────────────────────┐ │ Ethernet │ IP Header │ TCP/UDP │ Payload │ │ Header │ Src: 192.168.1.10 │ Header │ │ │ │ Dst: 192.168.2.10 │ │ │ └─────────────────────────────────────────────────────────────┘ IPIP Encapsulated Packet: ┌───────────────────────────────────────────────────────────────────────────────┐ │ Ethernet │ Outer IP │ Inner IP │ TCP/UDP │ Payload │ │ Header │ Src: 10.0.1.10 │ Src: 192.168.1.10 │ Header │ │ │ │ Dst: 10.0.1.11 │ Dst: 192.168.2.10 │ │ │ │ │ Proto: 4 (IPIP)│ │ │ │ └───────────────────────────────────────────────────────────────────────────────┘ ``` ## Summary Calico's architecture is designed for scalability, performance, and operational simplicity: 1. **Felix**: The workhorse agent on every node, programming routes and ACLs 2. **BIRD**: Distributes routes via BGP, enabling native routing integration 3. **confd**: Bridges the datastore to BIRD configuration 4. **Typha**: Scales the system by reducing API server load 5. **kube-controllers**: Keeps Kubernetes and Calico in sync 6. **Datastore**: Kubernetes API (recommended) or etcd for configuration storage Understanding these components and their interactions is essential for: * Troubleshooting connectivity issues * Optimizing performance at scale * Planning capacity and architecture * Integrating with existing network infrastructure [Previous: Part 1 - Introduction to Calico](https://www.atomai.click/kubernetes-docs/llms/en/networking/calico/01-introduction.md) [Next: Part 3 - Networking Modes](https://www.atomai.click/kubernetes-docs/llms/en/networking/calico/03-networking-modes.md) [Return to Calico Overview](https://www.atomai.click/kubernetes-docs/llms/en/networking/calico/README.md) ## Quiz To test what you've learned in this chapter, try the [Architecture Quiz](https://www.atomai.click/kubernetes-docs/en/quizzes/networking/calico/02-architecture-quiz). ---------------------------------------- Source: https://www.atomai.click/kubernetes-docs/en/networking/calico/03-networking-modes ---------------------------------------- # Part 3: Networking Modes > **Review baseline**: Calico Open Source 3.32.2 / operator 1.42.6; Kubernetes 1.34–1.36 is Calico 3.32's tested range. > **Last Updated**: September 12, 2026. Historical benchmark values below are retained as unverified reports, not new measurements. ## Scope and mode selection This chapter concerns Calico-owned Linux Pod networking and IPAM. In EKS policy-only mode, Amazon VPC CNI still owns Pod networking; creating Calico IPPools does not switch that installation to an overlay. The examples are alternative designs, not manifests to apply together or over the already-created pool in the introductory lab. The audit performed no cluster migration or network benchmark. | Choice | Meaning | Important boundary | |---|---|---| | IPIP | IPv4-in-IPv4, IP protocol 4 | Calico IPIP is IPv4-only; underlay must permit it | | VXLAN | Inner Ethernet carried in UDP, Calico default port 4789 | Outer IPv4 and IPv6 have different overhead; port/VNI are configurable | | Direct / unencapsulated | Pod IP packets routed without a Pod-network overlay | Underlay and return paths must route Pod addresses | | CrossSubnet | A setting of IPIP or VXLAN | Encapsulate inter-node traffic only when the relevant node addresses lie in different configured subnets | `Always` concerns eligible inter-node traffic to addresses in the configured pool; same-node traffic does not need a physical tunnel. `Never` disables that encapsulation, not all networking. CrossSubnet is not an AZ, Region or WAN-link detector: two subnets in one AZ can still require encapsulation. Inspect the node address and subnet mask used by Calico. Defaults depend on installation/provider and data plane. There is no universal “IPIP is the default for all clouds” rule or guarantee that Direct is always fastest. The [overlay guide](https://docs.tigera.io/calico/latest/networking/configuring/vxlan-ipip) describes the supported routing paths. ### Routing and encapsulation are separate choices By default, Felix programs routes for VXLAN pools, while confd/BIRD program cluster routes for IPIP and unencapsulated pools. Calico 3.32 also supports `Installation.spec.calicoNetwork.clusterRoutingMode: Felix` for those non-VXLAN routes. The corresponding lower-level settings are Felix `programClusterRoutes: Enabled` and BGP `programClusterRoutes: Disabled`; use the operator setting when it owns the installation. External BGP advertisements still require BGP. Static routes or a suitable routed fabric can also provide underlay reachability, so BGP and same-L2 adjacency are not universal requirements for every unencapsulated design. ## Packet structure and overhead The following uses an **underlay IP MTU**. The outer Ethernet header is outside that IP MTU. Assume no IPv4 options or extra inner VLAN tags; TCP options and other encapsulations can reduce payload further. ```text Direct: outer Ethernet | Pod IP | TCP or UDP | payload IPIP: outer Ethernet | outer IPv4 | Pod IPv4 | TCP or UDP | payload VXLAN: outer Ethernet | outer IP | UDP | VXLAN | inner Ethernet | Pod IP | TCP or UDP | payload ``` | Transport | Overhead above the Pod IP packet | Pod IP MTU when underlay IP MTU is 1500 | |---|---|---| | Direct, no other tunnel | 0 | 1500 | | IPIP, outer IPv4 | 20 | 1480 | | VXLAN, outer IPv4 | 20 + 8 + 8 + 14 = 50 | 1450 | | VXLAN, outer IPv6 | 40 + 8 + 8 + 14 = 70 | 1430 | | WireGuard, outer IPv4 | 60 | 1440 | | WireGuard, outer IPv6 | 80 | 1420 | For VXLAN, the 14 bytes in the MTU overhead are the **inner Ethernet header**, not the outer Ethernet header. Plain TCP has a minimum 20-byte header; UDP has an 8-byte header. Thus a 1500-byte IPv4 IP packet can contain up to 1460 bytes of TCP payload or 1472 bytes of UDP payload under these assumptions. The original shared “TCP/UDP = 20 bytes” label was incorrect. IPIP's protocol number is 4, not TCP/UDP port 4. Calico's usual VXLAN VNI is 4096 and its default UDP port is 4789, but both can be configured. Other current VXLAN implementations may use 8472; that is not limited to obsolete software. See [IP-in-IP](https://www.rfc-editor.org/rfc/rfc2003) and [VXLAN](https://www.rfc-editor.org/rfc/rfc7348). ### Illustrated packet paths ![IPv4 packets pass through the source kernel's IPIP tunnel and the destination kernel's decapsulation path.](https://raw.githubusercontent.com/Atom-oh/kubernetes-docs/main/en/.gitbook/assets/en-networking-calico-03-networking-modes-2.png) [🔍 View interactive diagram](https://www.atomai.click/kubernetes-docs/archmaps/en-networking-calico-03-networking-modes-2.html) The “Felix” columns represent kernel routing/policy programmed by Felix; packets do not traverse the Felix daemon. This is the IPv4 inter-node path, not same-node traffic or an encryption mechanism. ![Two Calico VTEPs encapsulate and decapsulate an inner frame over UDP.](https://raw.githubusercontent.com/Atom-oh/kubernetes-docs/main/en/.gitbook/assets/en-networking-calico-03-networking-modes-3.png) [🔍 View interactive diagram](https://www.atomai.click/kubernetes-docs/archmaps/en-networking-calico-03-networking-modes-3.html) 4789 and VNI 4096 are the illustrated defaults. The 50-byte overhead and 1450 MTU apply to an outer-IPv4 1500-byte path with the stated header assumptions, not every network or address family. ### CrossSubnet example With node addresses 10.0.1.10/24 and 10.0.1.11/24, the same-subnet path can be unencapsulated. A peer at 10.0.2.20/24 needs encapsulation in the CrossSubnet design. Incorrect node masks can therefore change the result even when the cloud subnet names look right. CrossSubnet does not establish inter-VPC/Region connectivity or provide encryption. ![Same-subnet nodes use an unencapsulated path while IPIP carries traffic between two configured node subnets.](https://raw.githubusercontent.com/Atom-oh/kubernetes-docs/main/en/.gitbook/assets/en-networking-calico-03-networking-modes-1.png) [🔍 View interactive diagram](https://www.atomai.click/kubernetes-docs/archmaps/en-networking-calico-03-networking-modes-1.html) The subnet addresses/masks drive this choice. The figure's 1500/1480 values assume an IPv4 1500-byte underlay; workloads should still use the minimum MTU across their possible paths. Their interface MTU does not increase dynamically for a same-subnet flow. ### Node diagnostics Run these read-only commands in an authorized **Linux node network namespace**, not an ordinary application Pod. Interfaces are present only for the enabled mode. The values shown by the commands depend on the actual installation. ```bash ip link show tunl0 ip link show vxlan.calico bridge fdb show dev vxlan.calico ip route show ``` A typical local Pod route is a host route such as `10.244.1.5/32 dev cali…`; do not route an entire /24 or /26 into one Pod's veth. An aggregate block may instead have a blackhole route plus more-specific Pod routes. Remote blocks can use a tunnel or next-hop node/router, and the route protocol label depends on BIRD versus Felix programming. ![Direct, IPIP and VXLAN show different packet-wrapper paths between Pods.](https://raw.githubusercontent.com/Atom-oh/kubernetes-docs/main/en/.gitbook/assets/en-networking-calico-03-networking-modes-5.png) [🔍 View interactive diagram](https://www.atomai.click/kubernetes-docs/archmaps/en-networking-calico-03-networking-modes-5.html) These are IPv4 1500-byte-underlay examples. The diagram compares packet wrappers, not measured speed or identical control-plane behavior. Other encapsulations and Service paths must be included when selecting the workload MTU. ## Configure a pool through its owner The `kubectl …projectcalico.org` examples assume the aggregated Calico API used in the introduction (or the appropriate native-v3 setup). Without it, use a matching calicoctl for logical Calico resources. Operator commands apply only to operator installations. Pool ranges must also avoid conflicting Service and node/underlay ranges. Use one configuration owner. Pools listed in `Installation.spec.calicoNetwork.ipPools` are reconciled by the operator; edit that desired list through its owner rather than applying competing IPPool objects. Standalone pools use the Calico IPPool API. In both cases, verify the actual cluster Pod CIDR, non-overlap, IPAM type and existing allocations first. ```bash kubectl get installation.operator.tigera.io default -o yaml kubectl get ippools.projectcalico.org -o yaml calicoctl ipam show --show-blocks ``` For an operator-owned pool, this is an **entry fragment** for the existing `ipPools` list. Preserve other entries and Installation fields. Do not create it over the introductory lab's already-allocated /16 pool: ```yaml - name: mode-demo-pool cidr: 10.244.0.0/16 blockSize: 26 encapsulation: VXLAN natOutgoing: Enabled nodeSelector: all() ``` For a standalone, newly planned pool, the equivalent IPv4 resource is below. This is an alternative to the operator entry, not an additional overlapping pool. The CIDR is an example that must fit the real cluster and not collide with any existing pool. ```yaml apiVersion: projectcalico.org/v3 kind: IPPool metadata: name: mode-demo-pool spec: cidr: 10.244.0.0/16 blockSize: 26 ipipMode: Never vxlanMode: Always natOutgoing: true nodeSelector: all() ``` Select one row, not multiple resources with the same CIDR: | IPv4 design | IPPool `ipipMode` | IPPool `vxlanMode` | Operator `encapsulation` | |---|---|---|---| | IPIP Always | Always | Never | IPIP | | IPIP CrossSubnet | CrossSubnet | Never | IPIPCrossSubnet | | VXLAN Always | Never | Always | VXLAN | | VXLAN CrossSubnet | Never | CrossSubnet | VXLANCrossSubnet | | Direct | Never | Never | None | IPIP and VXLAN cannot both be enabled in one pool. `encapsulation` is an operator pool field; it is not the standalone IPPool field name. In the normal aggregated-API installation, overlapping pool creation is rejected. With native v3 CRDs (tech preview), overlap validation is asynchronous and a created pool can receive a Disabled condition; creation success is not proof of usable allocation. Calico 3.32's Installation schema permits a pool list (up to 25 entries), with controller validation and platform constraints. Older examples claiming exactly one IPv4 pool should not be used as a universal current limit. ### Direct routing with external BGP If the design uses external BGP, configure the real peers and return routes before removing an overlay. A peer declaration alone does not configure the physical router or prove route acceptance. This separate topology example is not an addition to a BGP-disabled VXLAN lab: ```yaml apiVersion: projectcalico.org/v3 kind: BGPPeer metadata: name: example-rack-tor spec: peerIP: 192.0.2.1 asNumber: 65001 nodeSelector: rack == 'rack1' ``` Replace the documentation address and AS number, label the intended nodes and validate route filters/AS-loop handling for each rack. `natOutgoing: false` is appropriate only when return routing and any required external NAT are designed; BGP does not make private Pod addresses internet-routable by itself. Use the [BGP transition guidance](https://www.atomai.click/kubernetes-docs/llms/en/networking/calico/02-architecture.md) and [BGP deep dive](https://www.atomai.click/kubernetes-docs/llms/en/networking/calico/04-bgp-deep-dive.md) for mesh/RR changes. ![An unencapsulated Pod packet crosses a routed underlay whose routes are provided by BGP in this example.](https://raw.githubusercontent.com/Atom-oh/kubernetes-docs/main/en/.gitbook/assets/en-networking-calico-03-networking-modes-4.png) [🔍 View interactive diagram](https://www.atomai.click/kubernetes-docs/archmaps/en-networking-calico-03-networking-modes-4.html) This illustrates a BGP-based design, not a requirement that every Direct design use BGP. The 1500 value assumes that usable path MTU and no other tunnel; eBPF Service handoff or encryption can impose a lower workload MTU. ## NAT and pool selection With `natOutgoing: true`, the usual Calico behavior is SNAT for source addresses in that pool when the destination is outside **all Calico IPPools**. It is not simply a “leaving the cluster” test. Even a disabled pool can identify a no-NAT destination range; removing it can change NAT behavior. Additional Felix settings can also exclude host IPs. NAT does not grant NetworkPolicy permission. See [outgoing NAT](https://docs.tigera.io/calico/latest/networking/configuring/workloads-outside-cluster). ### Topology-based automatic allocation This is a separate planning example with two disjoint /18 pools inside a /16 cluster range. It must not coexist with an allocated parent /16 pool. Do not delete an in-use parent pool just to make the example fit. ```yaml apiVersion: projectcalico.org/v3 kind: IPPool metadata: name: zone-a-pool spec: cidr: 10.244.0.0/18 ipipMode: Never vxlanMode: CrossSubnet natOutgoing: true nodeSelector: topology.kubernetes.io/zone == 'ap-northeast-2a' --- apiVersion: projectcalico.org/v3 kind: IPPool metadata: name: zone-b-pool spec: cidr: 10.244.64.0/18 ipipMode: Never vxlanMode: CrossSubnet natOutgoing: true nodeSelector: topology.kubernetes.io/zone == 'ap-northeast-2b' ``` For automatic allocation, ensure every intended node matches an eligible pool; the selector does not schedule Pods. The zone labels above choose allocation pools, while CrossSubnet still uses node addresses/masks to decide encapsulation. ### Explicit namespace or Pod pool requests Create and verify a suitable pool before requesting it. This fragment adds an annotation through the namespace's existing owner; it is not a complete namespace replacement. Pod annotations take precedence over namespace annotations, which override CNI pool configuration. ```yaml metadata: annotations: cni.projectcalico.org/ipv4pools: '["production-pool"]' ``` `production-pool` must be an existing enabled pool with sufficient addresses. `assignmentMode: Manual` can exclude a pool from automatic selection while allowing explicit requests. **Neither a pool selector nor this annotation is a security boundary.** The released [IPAM implementation](https://github.com/projectcalico/calico/blob/v3.32.2/libcalico-go/lib/ipam/ipam.go) deliberately ignores node/namespace pool selectors when an enabled pool is explicitly requested. Control who can request pools if address ranges carry trust implications. Existing Pods keep their addresses; changing annotations does not renumber them. ## Cloud and platform boundaries | Environment | Guidance | |---|---| | Self-managed AWS EC2 | Check IP protocol 4 or VXLAN UDP reachability, routes, source/destination checks and return paths for the chosen mode | | EKS with Amazon VPC CNI | Default Pod networking is VPC CNI, not Calico VXLAN; policy-only Calico does not own these pools | | EKS with full Calico networking | Separate planned installation with Calico CNI/IPAM; use the [official EKS procedure](https://docs.tigera.io/calico/latest/getting-started/kubernetes/managed-public-cloud/eks) | | Azure with Calico-owned networking | The Calico overlay guide supports VXLAN where IPIP is unsupported; UDR configuration is not a fix for unsupported IPIP encapsulation | | AKS | Use the specific supported Azure CNI/policy integration, not a generic Calico-overlay assumption | | GCE / GKE | Self-managed GCE routing differs from managed GKE; [GKE Dataplane V2 uses Cilium](https://cloud.google.com/kubernetes-engine/docs/concepts/dataplane-v2) | | On-premises | Direct, static/BGP routing or overlay depends on underlay reachability; there is no universal fastest choice | | OpenStack Neutron integration | The cited Calico overlay guide excludes this integration; do not copy Kubernetes overlay guidance without its platform procedure | This chapter does not provide a custom-CNI recipe for EKS Auto Mode or Fargate. Disabling BGP in a VXLAN example means that configuration does not need it; it does not mean AWS has no BGP-capable services. Windows also has separate limitations, including no Calico IPIP or VXLAN CrossSubnet support. ## MTU configuration and validation Use the minimum usable MTU across paths the workload may take, including encryption and Service paths. The [Calico MTU guide](https://docs.tigera.io/calico/latest/networking/configuring/mtu) explains automatic detection and operator/manifest ownership. `mtuIfacePattern` selects interfaces considered during detection; it is not an on/off switch and does not prove the end-to-end path MTU. **Do not add IPIP and WireGuard overhead blindly.** In Calico's normal mixed deployment, WireGuard is used between enabled peers; IPIP/VXLAN is used on other paths. Choose the smallest applicable MTU. With a real 1500-byte path, IPv4 WireGuard plus IPIP means `min(1440, 1480) = 1440`, not `1500 − 60 − 20 = 1420`. Outer-IPv6 WireGuard separately has an 80-byte overhead. AKS has a documented WireGuard exception: the underlying path can be 1400 even when the interface shows 1500, giving 1340 for IPv4 WireGuard or 1320 for IPv6. The eBPF NodePort path also uses VXLAN, so an unencapsulated Pod pool alone does not imply a 1500-byte workload MTU. For an operator installation, after determining that **1450 is appropriate for this particular IPv4 VXLAN path**, merge it into the existing desired state: ```bash kubectl patch installation.operator.tigera.io default --type merge -p '{"spec":{"calicoNetwork":{"mtu":1450}}}' ``` For a manifest-managed installation, the documented setting is `calico-config.data.veth_mtu`; update that ConfigMap and roll the Calico node DaemonSet according to its procedure. Do not apply the manifest procedure to an operator-owned Deployment. **The updated workload MTU applies to new workloads.** Restarting calico-node does not by itself recreate application Pods or prove their MTU changed. | Underlay IP MTU example | IPIP IPv4 | VXLAN IPv4 | VXLAN IPv6 | WireGuard IPv4 | WireGuard IPv6 | |---|---|---|---|---|---| | 9000 | 8980 | 8950 | 8930 | 8940 | 8920 | | 9001, where the AWS path really supports it | 8981 | 8951 | 8931 | 8941 | 8921 | Jumbo support must hold across the whole path; an interface setting alone is insufficient. Check from a diagnostic workload when validating the workload path, rather than only from the node. These bounded checks assume an approved Linux diagnostic Pod with iputils and the required permissions. Set real Pod names/addresses. The payload sizes below are **IPv4 ICMP** examples: add 20 bytes of IPv4 and 8 of ICMP. IPv6 needs different accounting; successful probes do not prove every ECMP path is safe. ```bash CHECK_NS=calico-demo CHECK_POD=diagnostic-client CHECK_TARGET=diagnostic-server DEST_IPV4=$(kubectl -n "$CHECK_NS" get pod "$CHECK_TARGET" -o jsonpath='{.status.podIP}') case "$DEST_IPV4" in ""|*:*) echo "Select a ready target Pod with an IPv4 address" >&2; exit 1 ;; esac kubectl -n "$CHECK_NS" exec "$CHECK_POD" -- ip link show eth0 kubectl -n "$CHECK_NS" exec "$CHECK_POD" -- ping -4 -c 3 -W 2 -M do -s 1472 "$DEST_IPV4" kubectl -n "$CHECK_NS" exec "$CHECK_POD" -- ping -4 -c 3 -W 2 -M do -s 1452 "$DEST_IPV4" kubectl -n "$CHECK_NS" exec "$CHECK_POD" -- ping -4 -c 3 -W 2 -M do -s 1422 "$DEST_IPV4" ``` The three payloads test IP packet sizes 1500, 1480 and 1450. Failures can reflect policy/ICMP filtering as well as MTU. For packet capture, inspect IPv4 fragmentation-needed and IPv6 Packet Too Big messages with the appropriate capture permissions; the original IPv4-only filter did not cover IPv6. ## Change modes or migrate addresses deliberately Changing encapsulation is not the same as changing the Pod CIDR or block size. Calico supports changing the encapsulation configuration, but in-progress connections can be disrupted. Validate underlay permissions, routes, actual MTU, data-plane support and recovery before a maintenance change. Do not restart every node or every Deployment in a namespace as a generic migration step. For an operator-managed pool, change its `encapsulation` in the existing desired Installation list, preserving all other pools/settings. For a **standalone IPv4 IPPool only**, this mode-only example preserves its CIDR and allocation settings and changes the two encapsulation fields together: ```bash POOL_NAME=mode-demo-pool kubectl get ippool.projectcalico.org "$POOL_NAME" -o yaml > pool-before.yaml kubectl patch ippool.projectcalico.org "$POOL_NAME" --type merge -p '{"spec":{"ipipMode":"Never","vxlanMode":"Always"}}' ``` This is not a no-disruption guarantee. If a planned Direct-to-IPIP-CrossSubnet transition is appropriate, its field pair is `ipipMode: CrossSubnet` / `vxlanMode: Never`; changing it does not require replacing the pool CIDR. Recreate selected application workloads only as needed for the validated MTU/address plan, using their own rollout and readiness strategy. [PodDisruptionBudgets](https://kubernetes.io/docs/concepts/workloads/pods/disruptions/) do not limit a Deployment controller's rolling update. ### A separate IPPool/CIDR migration Use the [pool migration procedure](https://docs.tigera.io/calico/latest/networking/ipam/migrate-pools) only when Calico owns IPAM and the orchestrator/network design supports it. 1. Inventory existing pools, the Kubernetes/kube-proxy cluster CIDR, explicit pool requests and all allocations. A new pool outside the cluster CIDR can change NAT or break traffic; the old example's 10.245/16 is not automatically compatible with the introductory 10.244/16 cluster. 2. Add a verified non-overlapping pool through its owner and test new allocations before withdrawing the old one. Preserve the existing pool for old workloads. 3. Stop new old-pool allocations through the appropriate owner. A standalone `spec.disabled: true` excludes the pool from IPAM. An operator's `nodeSelector: "!all()"` disables **automatic selection**, but explicit old-pool requests bypass selectors; remove those requests too. 4. Migrate selected workloads in controlled batches, checking addresses, MTU, routes, policy and application readiness. Recreating Pods can interrupt applications and change IP addresses; a new pool does not guarantee seamless rollback. 5. Retire the old pool only after its remaining allocations and dependencies have been accounted for, including tunnel or LoadBalancer uses where applicable. Pod listings alone are insufficient. Keep its NAT/routing effects in mind when removing it from the owner. Useful read-only checks are: ```bash kubectl get ippools.projectcalico.org -o yaml calicoctl ipam show --show-blocks calicoctl ipam show --show-borrowed kubectl get pods --all-namespaces -o wide ``` A pool's block size is a separate migration concern; do not change an existing pool's immutable allocation structure by replacing a tutorial manifest. The copied old example that restarted calico-node “for immediate mode application” did not prove workload MTU or application recovery. ## Earlier benchmark reports — unverified provenance The earlier English and Korean pages contained different numbers and did not supply raw results, complete software versions, placement or a reproducible harness. Both records are preserved below; they cannot be treated as one experiment or as validated performance guarantees. This audit did not rerun them. ### Record A: earlier English page Reported environment: **3 × c5.xlarge on AWS**, a stated 10 Gbps network, iperf3 TCP, **one stream for 60 seconds**. No placement-group, Calico/kernel version or latency-collection method was supplied. | Reported metric | Direct | IPIP | VXLAN | |---|---|---|---| | Throughput, Gbps | 9.41 | 9.12 | 8.89 | | p99 latency, µs | 45 | 52 | 61 | | CPU, % per Gbps | 2.1 | 2.8 | 3.4 | AWS documents an ordinary 5 Gbps single-flow limit outside a cluster placement group, with specified exceptions. This report's values above 9 Gbps therefore need the missing placement/path conditions before they can be used to predict a new deployment. “Up to 10 Gbps” also does not establish sustained baseline bandwidth. See [EC2 bandwidth](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-instance-network-bandwidth.html). ### Record B: earlier Korean page | Reported metric | Direct | IPIP | VXLAN | Stated method | |---|---|---|---|---| | Throughput, Gbps | 9.8 | 9.2 | 8.5 | iperf3, MTU 1500 | | Latency, µs (statistic unspecified) | 35 | 42 | 55 | netperf TCP_RR | | CPU utilization | Low | Medium | Medium-high | At 10 Gbps | | PPS, millions per second | 1.8 | 1.5 | 1.2 | 64-byte packets | The hardware, sample count and exact interpretation of “64 bytes” were not supplied. netperf's TCP_RR test normally reports **transactions per second**; an explicitly justified reciprocal can estimate average request/response cycle time, but is not p99 or isolated one-way network latency. The raw output/conversion for the reported microsecond values is missing. Header size alone does not determine which mode is faster. NIC offloads, kernel/data plane, packet size, CPU, routes, connection reuse and offered load can change the result. Preserve these records as unverified history and measure the target environment rather than ranking modes from them. ### Bounded client-side probes for a new experiment Prepare dedicated test Pods containing matching iperf3/netperf versions, running server listeners and the required policy permissions. These commands are only client probes, not a complete reproduction of either record. Capture versions, node/AZ placement, MTU, request/response sizes, raw output and repeated runs. Confirm that the selected server has an IPv4 address for this example. ```bash set -euo pipefail BENCH_NS=calico-demo CLIENT_POD=benchmark-client SERVER_POD=benchmark-server SERVER_IP=$(kubectl -n "$BENCH_NS" get pod "$SERVER_POD" -o jsonpath='{.status.podIP}') : "${SERVER_IP:?Server Pod has no address}" case "$SERVER_IP" in *:*) echo "This example requires an IPv4 server Pod" >&2; exit 1 ;; esac kubectl -n "$BENCH_NS" get pods "$CLIENT_POD" "$SERVER_POD" -o wide kubectl -n "$BENCH_NS" exec "$CLIENT_POD" -- iperf3 -c "$SERVER_IP" -t 30 -P 4 -J > iperf3-result.json kubectl -n "$BENCH_NS" exec "$CLIENT_POD" -- netperf -H "$SERVER_IP" -t TCP_RR -l 60 > netperf-result.txt ``` The iperf3 example uses four streams and is therefore not the single-stream Record A method. The [netperf manual](https://github.com/HewlettPackard/netperf/blob/master/doc/netperf.txt) defines its reported units and optional latency outputs. Keep the test load isolated, stop only owned test servers/resources afterward, and do not alter production network modes to reproduce an uncited chart. [Calico overview](https://www.atomai.click/kubernetes-docs/llms/en/networking/calico/README.md) · [Architecture](https://www.atomai.click/kubernetes-docs/llms/en/networking/calico/02-architecture.md) · [Next: BGP deep dive](https://www.atomai.click/kubernetes-docs/llms/en/networking/calico/04-bgp-deep-dive.md) · [Networking modes quiz](https://www.atomai.click/kubernetes-docs/en/quizzes/networking/calico/03-networking-modes-quiz) ---------------------------------------- Source: https://www.atomai.click/kubernetes-docs/en/networking/calico/04-bgp-deep-dive ---------------------------------------- # Part 4: BGP Deep Dive > **Review baseline**: Calico 3.32.2; Calico 3.32 tests Kubernetes 1.34–1.36. **Last Updated**: September 12, 2026. > > Configuration examples assume a Linux Calico cluster with BGP enabled and the standard Calico API server installed (`projectcalico.org/v3`). They are separate topology alternatives, not one manifest to apply in sequence. Retain the installation's operator/GitOps ownership and merge intended fields into its existing configuration. The [installation guide](https://www.atomai.click/kubernetes-docs/llms/en/networking/calico/01-introduction.md) covers API prerequisites; the [networking modes guide](https://www.atomai.click/kubernetes-docs/llms/en/networking/calico/03-networking-modes.md) covers BGP-free routing alternatives. Router addresses, ASNs and CIDRs must match a network you control. No live fabric or cluster failover was tested for this review. ## Introduction Border Gateway Protocol (BGP) exchanges reachability information. Calico can use it to distribute workload routes and integrate with an existing routed fabric. BGP is a control-plane protocol: it can accompany unencapsulated routing or IP-in-IP, and it does not itself guarantee better performance. Calico 3.32 also supports Felix-managed cluster routing without BGP; external BGP advertisement still requires a BGP speaker. This deep dive covers BGP fundamentals, Calico's BGP architecture options, configuration resources, and advanced deployment patterns for enterprise environments. *** ## BGP Fundamentals ### What is BGP? BGP (Border Gateway Protocol) is a path-vector routing protocol designed to exchange routing information between autonomous systems. In Calico, BGP distributes pod IP routes across cluster nodes and optionally to external network infrastructure. ### Key BGP Concepts | Concept | Description | | -------------------------- | -------------------------------------------------------------------- | | **Autonomous System (AS)** | A collection of IP networks under a single administrative domain | | **AS Number (ASN)** | 16-bit or 32-bit identifier; allocation excludes special/reserved ranges | | **iBGP** | Internal BGP - sessions between routers in the same AS | | **eBGP** | External BGP - sessions between routers in different ASes | | **NLRI** | Network Layer Reachability Information - the routes being advertised | | **BGP Speaker** | A router or software that participates in BGP | ### Private AS Number Ranges For internal use within organizations, IANA reserves the following private ASN ranges: ``` 16-bit Private ASN Range: 64512 - 65534 32-bit Private ASN Range: 4200000000 - 4294967294 ``` Calico's default cluster ASN is `64512`. Private ASNs must be removed from AS paths before those routes reach the global Internet; they are identifiers, not inherently unroutable IP addresses. Other special ranges include documentation ASNs `64496–64511` and `65536–65551`, and `23456` (AS_TRANS). Consult the [IANA registry](https://www.iana.org/assignments/as-numbers/as-numbers.xhtml) rather than treating every other integer as an allocated public ASN. ### BGP Route Selection Process Compare the actual implementation and routing policy. Cisco `Weight` and administrative distances 20/200 are not universal BGP properties or Calico BIRD defaults. Calico 3.32.2 pins its BIRD fork to `v0.3.3-211-g9111ec3c`. For comparable eligible BGP routes, its selection function checks higher LOCAL_PREF, shorter AS_PATH (when enabled), lower ORIGIN, lower MED under the applicable neighbor-AS policy, eBGP over iBGP, and lower IGP metric. Remaining ties use router/ORIGINATOR_ID, CLUSTER_LIST length and peer IP; optional older-route preference changes the tie break. Suppression, next-hop reachability, stale-route handling and BIRD route preference also matter. This is not a universal eleven-step ladder. Calico 3.32 translates its route priorities into LOCAL_PREF and kernel metrics. Therefore, do not assume every locally exported route retains the upstream BIRD default LOCAL_PREF of 100. ### iBGP vs eBGP Behavior | Attribute | iBGP | eBGP | | --- | --- | --- | | AS relationship | Same AS | Different ASes | | AS_PATH | Normally preserved | Normally prepends the local AS | | Route propagation | An iBGP-learned route is normally not sent to another iBGP peer; RR is an exception | Export depends on policy and loop prevention | | Next hop | Often preserved; must remain reachable | Often changed; `nextHopMode` and topology affect this | | TTL and administrative distance | Implementation/configuration dependent | Implementation/configuration dependent | Locally originated or eBGP-learned routes can be sent to iBGP peers. Calico's generated external-peer configuration uses BIRD multihop; do not diagnose it from a generic “eBGP TTL 1” table. Inspect the generated configuration and negotiated session state. *** ## Calico BGP Architecture ### BIRD: Calico's BGP Implementation When BGP is enabled, Calico runs its BIRD fork in `calico-node`; confd renders its configuration. BIRD is not required in a BGP-disabled deployment. Both BIRD and Felix have routing responsibilities depending on the selected mode. ![BGP control-plane relationships: confd configures BIRD, which exchanges routes with peers, while Felix programs the dataplane.](https://raw.githubusercontent.com/Atom-oh/kubernetes-docs/main/en/.gitbook/assets/en-networking-calico-04-bgp-deep-dive-1.png) [🔍 View interactive diagram](https://www.atomai.click/kubernetes-docs/archmaps/en-networking-calico-04-bgp-deep-dive-1.html) > The boundary is schematic: the Calico API server is a separate component, not a process inside each calico-node Pod. Felix also manages local workload routes and, in the selected mode, cluster routes. BIRD is present only when enabled. ### BGP Topology Options Common internal BGP topology choices are: 1. **Node-to-Node Mesh (Full Mesh)** - Default configuration 2. **Route Reflectors** - Recommended for larger clusters *** ## Full-Mesh Topology ### How Full-Mesh Works With BGP and the default node mesh enabled, participating non-RR nodes peer with each other. Nodes marked as route reflectors are excluded from the automatic mesh. ![Ten sessions connect every pair of five nodes in a full mesh.](https://raw.githubusercontent.com/Atom-oh/kubernetes-docs/main/en/.gitbook/assets/en-networking-calico-04-bgp-deep-dive-3.png) [🔍 View interactive diagram](https://www.atomai.click/kubernetes-docs/archmaps/en-networking-calico-04-bgp-deep-dive-3.html) > The arrows enumerate bidirectional sessions, not one-way traffic. This counts one session per pair for the address family under discussion. ### Session Count Formula The number of BGP sessions in a full-mesh topology grows quadratically: ``` Sessions = N × (N - 1) / 2 Examples: - 10 nodes: 10 × 9 / 2 = 45 sessions - 50 nodes: 50 × 49 / 2 = 1,225 sessions - 100 nodes: 100 × 99 / 2 = 4,950 sessions - 500 nodes: 500 × 499 / 2 = 124,750 sessions ``` ### Full-Mesh Scaling and Transition The formula assumes one session per node pair for the address family being counted. Each node has `N−1` peers. CPU and memory depend on route count, update churn, policy, hardware and convergence targets; the former per-node memory table and fixed 50/200-node limits were not measured capacity limits. Check the existing configuration: ```bash kubectl get bgpconfiguration.projectcalico.org default -o yaml ``` An absent `default` resource means defaults may be in use. Prepare and validate replacement RR or fabric sessions before disabling the automatic mesh. Follow the transition order below; merely creating an RR label does not provide a working replacement. *** ## Route Reflector Topology ### Route Reflector Concepts Route Reflectors (RRs) solve the iBGP scalability problem by allowing a subset of nodes to reflect routes to other nodes. This eliminates the need for a full mesh. ![Six clients each peer with two mutually peered route reflectors.](https://raw.githubusercontent.com/Atom-oh/kubernetes-docs/main/en/.gitbook/assets/en-networking-calico-04-bgp-deep-dive-4.png) [🔍 View interactive diagram](https://www.atomai.click/kubernetes-docs/archmaps/en-networking-calico-04-bgp-deep-dive-4.html) > This drawing contains six clients plus two RRs: 13 sessions. In the figure’s 2N+1 expression N counts clients, while full-mesh N counts total nodes. Automatic mesh is disabled only after the explicit replacement topology is verified. ### Route Reflector Key Attributes | Attribute | Description | | -------------------- | ------------------------------------------------------------- | | **Cluster ID** | Identifies a set of RRs serving the same clients | | **Originator ID** | Prevents routing loops (set to the router ID of originator) | | **Route Reflection** | RR re-advertises routes learned from clients to other clients | ### Session Count with Route Reflectors Let `T` be the total node count, `R` the number of reflectors, and `C=T−R` the number of clients. If every client peers with every RR and the RRs peer with each other: ```text RR sessions = C×R + R×(R−1)/2 T=100, R=2: 98×2 + 1 = 197 (full mesh of the same 100 nodes: 4,950) T=500, R=2: 498×2 + 1 = 997 (full mesh of the same 500 nodes: 124,750) ``` If “100 nodes” instead means 100 clients plus two additional RRs, the count is 201, but that topology has 102 nodes. The two meanings must not be mixed. ### Configuring Route Reflector Nodes Use prepared, workload-free RR nodes for this transition. Setting a cluster ID immediately removes that node from the automatic mesh; changing a busy node in place can interrupt connectivity. This Kubernetes-datastore example preserves existing node IPs and other fields. **1. Label and annotate the prepared RR nodes** ```bash kubectl label node rr-node-1 rr-node-2 route-reflector=true kubectl annotate node rr-node-1 rr-node-2 projectcalico.org/RouteReflectorClusterID=244.0.0.1 ``` The shared ID identifies this redundant RR cluster, not the Kubernetes cluster. Other RR clusters/hierarchy levels need an intentional ID design. **2. Create explicit peerings** ```yaml apiVersion: projectcalico.org/v3 kind: BGPPeer metadata: name: peer-to-rr spec: nodeSelector: "!has(route-reflector)" peerSelector: "has(route-reflector)" --- apiVersion: projectcalico.org/v3 kind: BGPPeer metadata: name: rr-mesh spec: nodeSelector: "has(route-reflector)" peerSelector: "has(route-reflector)" ``` `peerSelector` selects Calico nodes, and reverse peering is automatic unless `reversePeering: Manual` is selected. It does not discover arbitrary external routers. **3. Verify before removing the old path** Verify Established sessions on both RRs and their clients, expected advertised/received workload prefixes, reachable next hops, and representative cross-node traffic. Confirm forwarding survives the planned loss of either RR. Ordinary client mesh sessions can remain during this transition. **4. Disable automatic mesh only after those checks** Update the owned `BGPConfiguration/default` manifest, preserving its ASN, communities and other settings. The equivalent merge patch for an existing resource is: ```bash kubectl patch bgpconfiguration.projectcalico.org default --type=merge -p '{"spec":{"nodeToNodeMeshEnabled":false}}' ``` If `default` does not exist, create it through the installation's configuration owner after the same checks. Recheck routes and traffic after the change; keep a rollback plan for the original topology. ### Route Reflector Redundancy Patterns **Pattern 1: Dual Route Reflectors (Small/Medium Clusters)** ![Each zone’s clients peer with both route reflectors placed in separate zones.](https://raw.githubusercontent.com/Atom-oh/kubernetes-docs/main/en/.gitbook/assets/en-networking-calico-04-bgp-deep-dive-11.png) [🔍 View interactive diagram](https://www.atomai.click/kubernetes-docs/archmaps/en-networking-calico-04-bgp-deep-dive-11.html) > This provides a redundant route-distribution path for surviving clients when one RR is lost, provided transport, forwarding and remaining capacity are healthy. It does not preserve workloads located in a failed zone. **Pattern 2: Hierarchical Route Reflectors** Rack-level RRs can peer with global RRs to reduce per-node session fan-out. Total sessions still grow with clients and racks. A single RR per rack remains a failure point even if global RRs are redundant; evaluate each tier's redundancy, cluster IDs, reflection rules, reachability and convergence before adopting a hierarchy. *** ## BGPPeer Resource The `BGPPeer` resource defines BGP peering relationships between Calico nodes and external BGP speakers. ### BGPPeer Scope Types | Type | Description | Use Case | | ----------------- | -------------------- | ----------------------- | | **Global** | Applies to all nodes | External router peering | | **Node-specific** | Uses nodeSelector | Rack-local peering | | **Per-node** | Specifies exact node | Special configurations | ### Global BGPPeer Example Peer all nodes with external ToR switches: ```yaml apiVersion: projectcalico.org/v3 kind: BGPPeer metadata: name: peer-to-tor-switches spec: peerIP: 10.0.0.1 asNumber: 65001 # No nodeSelector means all nodes peer with this address ``` ### Node-Specific BGPPeer Example Peer nodes in specific racks with their local ToR switch: ```yaml apiVersion: projectcalico.org/v3 kind: BGPPeer metadata: name: rack1-tor-peer spec: nodeSelector: rack == 'rack1' peerIP: 10.0.1.1 asNumber: 65001 --- apiVersion: projectcalico.org/v3 kind: BGPPeer metadata: name: rack2-tor-peer spec: nodeSelector: rack == 'rack2' peerIP: 10.0.2.1 asNumber: 65002 ``` ### BGPPeer with peerSelector Use `peerSelector` to dynamically select Calico nodes as peers: ```yaml apiVersion: projectcalico.org/v3 kind: BGPPeer metadata: name: client-to-rr-peering spec: nodeSelector: "!has(route-reflector)" peerSelector: has(route-reflector) ``` ### Advanced BGPPeer Configuration Create the referenced Secret and the `tor-policy` BGPFilter from the security section first. This example assumes a directly connected peer with matching GTSM and authentication settings. ```yaml apiVersion: projectcalico.org/v3 kind: BGPPeer metadata: name: advanced-peer spec: node: specific-node-name peerIP: 192.168.1.1 asNumber: 65100 password: secretKeyRef: name: bgp-secrets key: datacenter-password keepaliveTime: 30s maxRestartTime: 120s sourceAddress: UseNodeIP nextHopMode: Auto ttlSecurity: 1 filters: - tor-policy ``` | Field | Meaning in Calico 3.32.2 | | --- | --- | | `keepaliveTime` | Duration string; the lowercase `a` is significant. Verified against the released CRD and renderer. | | `maxRestartTime` | Graceful-restart time advertised to the neighbor; not a connection-retry interval. | | `sourceAddress` | `UseNodeIP` or `None`; a literal source IP is not accepted. | | `filters` | Names of existing `BGPFilter` resources, not embedded rule objects. | | `ttlSecurity` | GTSM path length in edges; `1` means a directly connected peer. | | `numAllowedLocalASNumbers` | Allowed occurrences of the local ASN in a received AS_PATH; relaxes loop prevention, not a multihop setting. Leave unset unless the routing design requires it. | The current `BGPPeer` API has no `holdTime`, `keepAliveTime` or `restartTime` field. `nextHopMode` is `Auto`, `Self` or `Keep`; the older `keepOriginalNextHop` field is deprecated, not removed. *** ## BGPConfiguration Resource The `BGPConfiguration` resource defines cluster-wide BGP settings. ### Basic BGPConfiguration ```yaml apiVersion: projectcalico.org/v3 kind: BGPConfiguration metadata: name: default spec: # Cluster AS number asNumber: 64512 # Set topology separately after validating its peerings. # Log level for BIRD logSeverityScreen: Info ``` ### Service IP Advertisement Calico can advertise existing Service IPs to an authorized routed network. Advertisement does not allocate the IP, create a cloud load balancer, or guarantee a reachable return path. The CIDRs below are examples: merge only the required ranges into the existing configuration and retain other settings. ```yaml apiVersion: projectcalico.org/v3 kind: BGPConfiguration metadata: name: default spec: asNumber: 64512 # Advertise Service ClusterIPs serviceClusterIPs: - cidr: 10.96.0.0/12 # Advertise Service ExternalIPs serviceExternalIPs: - cidr: 203.0.113.0/24 # Advertise Service LoadBalancerIPs serviceLoadBalancerIPs: - cidr: 198.51.100.0/24 ``` ### BGP Communities Configuration `prefixAdvertisements` adds communities to matching existing routes, including Pod routes in the current renderer. It does **not** originate the listed prefix or aggregate all Pod blocks into that prefix. Named communities take effect only when referenced; their names and arbitrary values do not implement a routing policy by themselves. ```yaml apiVersion: projectcalico.org/v3 kind: BGPConfiguration metadata: name: default spec: asNumber: 64512 # Community tagging for pod networks prefixAdvertisements: - cidr: 10.244.0.0/16 communities: - "64512:100" # Standard community - "64512:200" - cidr: 10.96.0.0/12 communities: - "64512:300" # Service IPs community # Named aliases, referenced by prefixAdvertisements in this configuration communities: - name: pod-networks value: "64512:100" - name: service-networks value: "64512:300" - name: no-export value: "65535:65281" # Well-known NO_EXPORT ``` ### Node-Specific AS Number For a Kubernetes datastore, annotate the existing node to preserve its addresses and other fields. Changing an ASN resets affected peerings; coordinate both endpoints and the routing topology. ```bash kubectl annotate node border-node-1 projectcalico.org/ASNumber=65001 ``` For an existing annotation, update it through the configuration owner after reviewing its current value. Other datastores use the Calico Node API; do not replace an existing Node with a partial example containing invented addresses. *** ## Service IP Advertisement ### Advertisement Types and Forwarding | Type | Address owner and prerequisite | | --- | --- | | ClusterIP | Kubernetes allocates it; advertising the Service CIDR exposes a route into the service network. | | ExternalIP | The operator must already own and route the assigned address. `spec.externalIPs` is deprecated since Kubernetes 1.36; existing support is not removal. | | LoadBalancer IP | A compatible controller allocates it. Calico can allocate owned VIPs itself, or interoperate with an explicitly chosen allocator. A cloud LB hostname is not an IP prefix. | With the default aggregation behavior, Cluster-mode Services use configured aggregate advertisements, while Local-mode Services use host routes (`/32` or `/128`) from nodes with ready local endpoints. Explicit host-prefix ranges and Calico 3.32’s `serviceLoadBalancerAggregation` setting can change the advertised routes; inspect the actual RIB/export rather than inferring it solely from the Service type. Validate endpoints, the Service dataplane, upstream ECMP and return paths. This is distinct from Pod IPAM block advertisement. ### Native Calico LoadBalancer IPAM Calico 3.32 includes a LoadBalancer controller in `calico-kube-controllers`. It requires an IPPool with `allowedUses: [LoadBalancer]`; the standard Pod pool does not supply those addresses automatically. Confirm that controller is enabled. This standalone bare-metal example also assumes an existing `calico-demo` namespace and ready `app=my-app` endpoints serving the stated port. Replace the documentation range with an owned, routable range. ```yaml apiVersion: projectcalico.org/v3 kind: IPPool metadata: name: service-lb-pool spec: cidr: 198.51.100.0/24 allowedUses: - LoadBalancer assignmentMode: Automatic --- apiVersion: projectcalico.org/v3 kind: BGPConfiguration metadata: name: default spec: serviceLoadBalancerIPs: - cidr: 198.51.100.0/24 --- apiVersion: v1 kind: Service metadata: name: my-lb-service namespace: calico-demo annotations: projectcalico.org/loadBalancerIPs: '["198.51.100.50"]' spec: type: LoadBalancer loadBalancerClass: calico externalTrafficPolicy: Local selector: app: my-app ports: - port: 443 targetPort: 8443 ``` The explicit `projectcalico.org/loadBalancerIPs` request must belong to an eligible pool and be available; it does not fall back to another address if allocation fails. Allocation and BGP advertisement are separate. Review the controller's `assignIPs` mode before changing it: `RequestedServicesOnly` can unassign existing unannotated Services. Preserve existing pool and controller ownership. MetalLB is an alternative allocator: its current requested-IP annotation is `metallb.io/loadBalancerIPs`. Choose allocation and BGP-speaker ownership deliberately rather than running competing allocators/speakers for the same VIP. Do not advertise AWS-managed load balancer addresses as a locally owned pool. ### Selective Service Advertisement There is no documented Calico Service opt-out annotation named `projectcalico.org/bgp-advertise`. Select advertised ranges in `BGPConfiguration`, and apply peer-specific BGPFilters where needed. The supported node label `node.kubernetes.io/exclude-from-external-load-balancers=true` excludes a node; it is not a per-Service opt-out. Rejecting one `/32` does not make an IP unreachable if a covering Service aggregate is still advertised. For a Service that must remain internal, ensure no advertised range covers it and enforce access policy independently; route filtering is not an authorization boundary. *** ## Physical Network Integration ### ToR Routing Policy and Vendor Adaptation Configure the router's ASN, node neighbors, address family, authentication, import/export policy and reachable next hops as one design. Decide whether nodes use a pre-existing underlay default route or receive a default from BGP. `network` originates an existing matching route; it is not a command to accept routes from a neighbor. Broad `redistribute connected` can leak unrelated networks. | Platform | Adaptation required | | --- | --- | | Cisco IOS XE / NX-OS | Use the exact platform/release syntax. IOS XE dynamic neighbors use a peer group and `bgp listen range`; do not combine IOS and NX-OS command hierarchies. Define every referenced route map and prefix list. | | Arista EOS | Use the deployed release’s peer-group, address-family, secret and import/export policy configuration. The former unverified EOS command block is not a runnable recipe. | | Junos | A plain prefix-list match is exact. Use an explicit route-filter match type when more-specific routes are intended. | For example, this **Junos policy fragment**, attached as import policy on the ToR's intended node-facing BGP group, accepts planned Pod `/26`–`/32` routes and LoadBalancer `/32` routes, then rejects the rest: ```text policy-options { policy-statement K8S-IMPORT { term approved { from { route-filter 10.244.0.0/16 prefix-length-range /26-/32; route-filter 198.51.100.0/24 prefix-length-range /32-/32; } then accept; } term reject-rest { then reject; } } } ``` The minimum Pod length assumes `/26` IPAM blocks; adapt it to the actual pool and route inventory. Borrowed addresses and some mobility paths can require `/32` routes, so `le 26` is not a generally safe Pod filter. This fragment neither creates neighbors nor advertises a default route. Vendor device configuration and failover have not been runtime tested here; complete and validate export policy, limits and next-hop behavior on the exact router release before deployment. ### Spine-Leaf Architecture Integration ![Nodes peer with local leaf switches, which connect to the spine layer.](https://raw.githubusercontent.com/Atom-oh/kubernetes-docs/main/en/.gitbook/assets/en-networking-calico-04-bgp-deep-dive-5.png) [🔍 View interactive diagram](https://www.atomai.click/kubernetes-docs/archmaps/en-networking-calico-04-bgp-deep-dive-5.html) > Grouped boxes summarize multiple sessions. The shared node ASN needs an explicit AS-loop/override design; dual spines alone do not provide leaf or node-uplink redundancy. Use the addresses and ASNs as an illustrative topology, not a complete deployable configuration. Calico peer fragments for a spine-leaf design follow. Confirm node labels, direct/recursive next-hop reachability, export policies and the return path first. Reusing ASN 64512 on nodes across racks can cause a route to be rejected when its AS_PATH contains the receiving node's ASN; design unique ASNs or a deliberately validated fabric AS-override/loop policy. Do not work around this by blindly raising `numAllowedLocalASNumbers`. Validate the replacement path before removing mesh sessions. ```yaml # Final topology alternative: establish fabric peerings before removing mesh. apiVersion: projectcalico.org/v3 kind: BGPConfiguration metadata: name: default spec: asNumber: 64512 --- # Peer nodes with their local leaf switch apiVersion: projectcalico.org/v3 kind: BGPPeer metadata: name: rack1-leaf-peer spec: nodeSelector: topology.kubernetes.io/zone == 'rack1' peerIP: 10.0.1.1 asNumber: 65001 --- apiVersion: projectcalico.org/v3 kind: BGPPeer metadata: name: rack2-leaf-peer spec: nodeSelector: topology.kubernetes.io/zone == 'rack2' peerIP: 10.0.2.1 asNumber: 65002 --- apiVersion: projectcalico.org/v3 kind: BGPPeer metadata: name: rack3-leaf-peer spec: nodeSelector: topology.kubernetes.io/zone == 'rack3' peerIP: 10.0.3.1 asNumber: 65003 ``` *** ## BGP Community Tagging Strategy ### Community Design Patterns The private values below are a local convention requiring router policy; they are not built-in priority controls. Standard communities contain two 16-bit values. Large communities contain three 32-bit values and can represent a four-byte ASN without squeezing it into a standard community. | Community | Meaning | Action | | ------------- | -------------- | -------------------------------- | | `64512:100` | Pod Networks | Accept, normal routing | | `64512:200` | Service IPs | Accept, may apply special policy | | `64512:300` | Infrastructure | Higher priority routing | | `65535:65281` | NO\_EXPORT | Do not advertise outside the AS confederation boundary (outside the AS when no confederation is used) | | `65535:65282` | NO\_ADVERTISE | Do not advertise to any peer | ### Community-Based Traffic Engineering ```yaml apiVersion: projectcalico.org/v3 kind: BGPConfiguration metadata: name: default spec: asNumber: 64512 communities: - name: production value: "64512:100" - name: staging value: "64512:200" - name: local-only value: "65535:65281" # NO_EXPORT prefixAdvertisements: # Tag existing production routes; actual propagation follows routing policy - cidr: 10.244.0.0/17 communities: - production # Add NO_EXPORT to existing staging routes - cidr: 10.244.128.0/17 communities: - staging - local-only # Service IPs - cidr: 10.96.0.0/12 communities: - production ``` *** ## BGP Security ### MD5 Authentication Calico supports the TCP MD5 signature option for BGP. It authenticates traffic from peers sharing the secret; it does not encrypt traffic or validate the legitimacy of routes sent by an authenticated peer. Provision `bgp-secrets` through your secret-management process in the namespace where `calico-node` runs (`calico-system` for the operator installation used here; manifest installations may use `kube-system`). The example requires the `datacenter-password` key. Other examples referencing `mesh-password`, rack-specific or leaf-specific keys require those keys too. Configure matching credentials on the corresponding routers and confirm the Calico service account can read the Secret. ```yaml apiVersion: projectcalico.org/v3 kind: BGPPeer metadata: name: secure-peer spec: peerIP: 192.168.1.1 asNumber: 65100 password: secretKeyRef: name: bgp-secrets key: datacenter-password ``` ### Prefix Filtering Rules are evaluated in order; the first match executes immediately. Unmatched routes default to **Accept**, so a whitelist needs an unconditional final Reject. `Equal 0.0.0.0/0` matches only the default route; `In 0.0.0.0/0` matches every IPv4 route and `NotIn 0.0.0.0/0` matches none. The following external-peer example accepts only a default route and the planned underlay `10.0.0.0/16` on import. On export it allows actual Pod `/26`–`/32` routes and LoadBalancer `/32` routes. Adapt the CIDRs and lengths to the actual route inventory; do not attach this external policy indiscriminately to RR/client sessions. ```yaml apiVersion: projectcalico.org/v3 kind: BGPFilter metadata: name: tor-policy spec: importV4: - action: Accept matchOperator: Equal cidr: 0.0.0.0/0 - action: Accept matchOperator: In cidr: 10.0.0.0/16 - action: Reject exportV4: - action: Accept matchOperator: In cidr: 10.244.0.0/16 prefixLength: min: 26 max: 32 operations: - addCommunity: value: "64512:100" - action: Accept matchOperator: In cidr: 198.51.100.0/24 prefixLength: min: 32 max: 32 - action: Reject --- apiVersion: projectcalico.org/v3 kind: BGPPeer metadata: name: filtered-peer spec: peerIP: 192.168.1.1 asNumber: 65100 filters: - tor-policy ``` `prefixLength` is an object with `min` and `max`, not a range string. Calico 3.32 also supports accepted-route operations such as `addCommunity`. An explicit export Accept returns before the built-in Calico export/aggregation/`prefixAdvertisements` processing. It may therefore export more-specific routes already in the RIB, and this example adds its Pod tag directly in the rule. Inspect `show route export` before applying it to the fabric; a BGPFilter does not create missing routes. ### GTSM (TTL Security) GTSM rejects packets arriving with a TTL below the expected path threshold; it reduces off-path spoofing exposure but does not authenticate the peer or stop an on-link attacker. Configure both endpoints consistently. ```yaml apiVersion: projectcalico.org/v3 kind: BGPPeer metadata: name: gtsm-enabled-peer spec: peerIP: 192.168.1.1 asNumber: 65100 ttlSecurity: 1 ``` For the pinned BIRD implementation, GTSM sends TTL 255 and sets minimum receive TTL to `256−hops`. Thus `ttlSecurity: 1` requires 255, not 254; two edges require at least 254. Verify the actual path before enabling it. This setting is unrelated to the count of local ASNs allowed in AS_PATH. *** ## Performance Tuning ### BGP Timer Configuration ```yaml apiVersion: projectcalico.org/v3 kind: BGPPeer metadata: name: tuned-peer spec: peerIP: 192.168.1.1 asNumber: 65100 keepaliveTime: 20s maxRestartTime: 120s ``` The pinned BIRD fork proposes a 240-second Hold Time by default and negotiates the smaller value with the neighbor. If no keepalive interval is configured, it uses one third of that negotiated Hold Time. An explicit `keepaliveTime` overrides the interval; it does **not** automatically change Hold Time to three times that value. Inspect the actual negotiated timers and choose an interval that fits them. `BGPPeer` does not expose `holdTime`. The former 60/180, 10/30 and 3/9 recommendations were not verified Calico defaults or failure-detection guarantees. BIRD's standalone BFD capability does not imply a supported Calico BFD CRD or configuration field. Test any separate BFD integration against the exact supported deployment rather than adding an invented field. ### Route Aggregation Calico normally aggregates local IPAM addresses into their allocated blocks; the current BIRD aggregation template also permits higher-priority more-specific routes. Borrowing and mobility may require host routes. `prefixAdvertisements` only tags existing matching routes and does not turn every `/26` into an originated `/16`. Larger IPAM blocks trade fewer block routes against allocation granularity and address utilization. Existing IPPool `blockSize` is immutable; use the pool migration procedure in [networking modes](https://www.atomai.click/kubernetes-docs/llms/en/networking/calico/03-networking-modes.md) if a new pool is required. Do not apply a new block size over an existing default pool or advertise a covering aggregate from a router that cannot reach all covered destinations. ### Graceful Restart Calico's BIRD template enables Graceful Restart. Its benefit requires negotiated capability and a still-working forwarding path; retained stale routes can otherwise blackhole traffic. It does not guarantee interruption-free updates. For explicit peers, `BGPPeer.maxRestartTime` sets the advertised restart time. The following setting applies to **automatic node mesh** sessions, not every explicit peer: ```yaml apiVersion: projectcalico.org/v3 kind: BGPConfiguration metadata: name: default spec: nodeMeshMaxRestartTime: 120s ``` This is a duration string, not an integer or an enable switch. Change it through the existing configuration owner and validate actual peer capability and recovery behavior. *** ## Debugging BGP ### Inspect BIRD from the Correct Node Choose an actual node and the installation namespace. These read-only commands run from the operator's shell against the IPv4 BIRD control socket. For IPv6 use `birdcl6` and `/var/run/calico/bird6.ctl`. A BGP-disabled installation need not have either daemon. ```bash CALICO_NAMESPACE=calico-system CALICO_NODE=worker-1 CALICO_POD="$(kubectl -n "$CALICO_NAMESPACE" get pods -l k8s-app=calico-node \ --field-selector "spec.nodeName=$CALICO_NODE" -o jsonpath='{.items[0].metadata.name}')" test -n "$CALICO_POD" kubectl -n "$CALICO_NAMESPACE" exec "$CALICO_POD" -c calico-node -- \ birdcl -s /var/run/calico/bird.ctl show protocols all kubectl -n "$CALICO_NAMESPACE" exec "$CALICO_POD" -c calico-node -- \ birdcl -s /var/run/calico/bird.ctl show route ``` ```bash CALICO_BGP_PROTOCOL=Global_192_168_1_1 kubectl -n "$CALICO_NAMESPACE" exec "$CALICO_POD" -c calico-node -- \ birdcl -s /var/run/calico/bird.ctl show protocols all "$CALICO_BGP_PROTOCOL" kubectl -n "$CALICO_NAMESPACE" exec "$CALICO_POD" -c calico-node -- \ birdcl -s /var/run/calico/bird.ctl show route export "$CALICO_BGP_PROTOCOL" kubectl -n "$CALICO_NAMESPACE" exec "$CALICO_POD" -c calico-node -- \ birdcl -s /var/run/calico/bird.ctl show route protocol "$CALICO_BGP_PROTOCOL" kubectl -n "$CALICO_NAMESPACE" exec "$CALICO_POD" -c calico-node -- \ birdcl -s /var/run/calico/bird.ctl 'show route where net ~ [10.244.0.0/16+]' ``` ```bash kubectl get bgpconfiguration.projectcalico.org default -o yaml kubectl get bgppeers.projectcalico.org -o wide kubectl get bgpfilters.projectcalico.org -o yaml kubectl -n "$CALICO_NAMESPACE" logs "$CALICO_POD" -c calico-node --tail=200 ``` Replace `CALICO_BGP_PROTOCOL` with a name returned by `show protocols`; actual names include `Mesh_…`, `Global_…` and `Node_…`, not a universal `bgp*` prefix. Quote route expressions so the local shell does not expand them. `show protocols all` includes non-BGP protocols too. Container logs can show startup and confd errors, but absence of matching stdout lines does not prove BIRD is healthy. Inspect the installation's BIRD log destination and session state. `calicoctl node status` is a node-local diagnostic requiring the node environment, not just a workstation kubeconfig. Likewise, `ip route` must be inspected on the intended node/network namespace. | Symptom | Checks | | --- | --- | | Session remains Active | Peer address/ASN, TCP listener and firewall, source address, MD5/GTSM agreement, transport reachability | | Established but no useful routes | Import/export filters, RR roles, endpoint/IPAM state, next-hop reachability and AS-loop rejection | | Flapping or resets | Transport loss, MTU, authentication, negotiated timers, controller changes | | Route exists but traffic fails | Actual kernel/FIB path, return route, Service forwarding, access policy and covering aggregates | Established BGP alone does not prove workload connectivity. *** ## Multi-Rack and Multi-Datacenter Design ### Multi-Rack with Route Reflectors ![Two route reflectors in one management rack each peer with compute nodes across racks.](https://raw.githubusercontent.com/Atom-oh/kubernetes-docs/main/en/.gitbook/assets/en-networking-calico-04-bgp-deep-dive-7.png) [🔍 View interactive diagram](https://www.atomai.click/kubernetes-docs/archmaps/en-networking-calico-04-bgp-deep-dive-7.html) > A surviving RR can preserve route distribution only if its transport and capacity remain available. Both RRs in one management rack share that rack’s failure risk; separate failure domains for rack-level resilience. ### Multi-Datacenter BGP Design ![Each datacenter has its own AS and route reflectors peering with WAN routers.](https://raw.githubusercontent.com/Atom-oh/kubernetes-docs/main/en/.gitbook/assets/en-networking-calico-04-bgp-deep-dive-8.png) [🔍 View interactive diagram](https://www.atomai.click/kubernetes-docs/archmaps/en-networking-calico-04-bgp-deep-dive-8.html) > The WAN group summarizes transit that must be separately configured; the visible links alone do not establish end-to-end reachability. DC1 origin tagging also requires the prefixAdvertisements reference shown in the text. DC1 configuration fragments follow, assuming its owned workload CIDR is `10.244.0.0/16` and its local RR topology is already working. A named community must also be referenced by `prefixAdvertisements` to tag matching routes. DC2 needs its own non-overlapping CIDRs, ASNs and peer definitions; the WAN needs explicit transit/return routing and policy. This fragment is not a complete two-DC deployment. ```yaml # DC1 Configuration apiVersion: projectcalico.org/v3 kind: BGPConfiguration metadata: name: default spec: asNumber: 64512 communities: - name: dc1-origin value: "64512:1" prefixAdvertisements: - cidr: 10.244.0.0/16 communities: - dc1-origin --- # Peer DC1 RRs with WAN routers apiVersion: projectcalico.org/v3 kind: BGPPeer metadata: name: dc1-to-wan spec: nodeSelector: has(route-reflector) peerIP: 10.255.0.1 # WAN Router asNumber: 65000 ``` *** ## Best Practices Summary ### Design Recommendations 1. Size full mesh and RR deployments using measured route count, churn and convergence targets. 2. Separate redundant RRs across failure domains and verify surviving capacity and transport. 3. Use rack-aware labels and a documented ASN, CIDR and next-hop plan. 4. Add a hierarchy only when its reflection/loop rules and per-tier redundancy are understood. 5. Treat multiple datacenters as a complete routing and security design, not merely two BGPPeer objects. ### Security Recommendations 1. Always enable MD5 authentication for external peers 2. Implement prefix filtering to prevent route injection 3. Use GTSM (TTL Security) where supported 4. Configure supported prefix limits on the external routers; do not invent a Calico BGPPeer limit field. 5. Monitor BGP sessions for anomalies ### Operational Recommendations 1. Label nodes consistently for BGP topology 2. Document AS number allocation scheme 3. Implement BGP monitoring and alerting 4. Test failover scenarios regularly 5. Inspect negotiated timers and test recovery; a shorter keepalive is not a guaranteed shorter Hold Time. *** ## References * [Calico BGP Documentation](https://docs.tigera.io/calico/latest/networking/configuring/bgp) * [BIRD Internet Routing Daemon](https://bird.network.cz/) * [RFC 4271 - BGP-4](https://www.rfc-editor.org/rfc/rfc4271) * [RFC 4456 - BGP Route Reflection](https://www.rfc-editor.org/rfc/rfc4456) * [RFC 5082 - GTSM](https://www.rfc-editor.org/rfc/rfc5082) * [Calico BGPPeer API](https://docs.tigera.io/calico/latest/reference/resources/bgppeer) * [Calico BGPConfiguration API](https://docs.tigera.io/calico/latest/reference/resources/bgpconfig) * [Calico BGPFilter API](https://docs.tigera.io/calico/latest/reference/resources/bgpfilter) * [Service IP advertisement](https://docs.tigera.io/calico/latest/networking/configuring/advertise-service-ips) * [Calico LoadBalancer IPAM](https://docs.tigera.io/calico/latest/networking/ipam/service-loadbalancer) * [Calico 3.32.2 BIRD configuration processing](https://github.com/projectcalico/calico/blob/v3.32.2/confd/pkg/backends/calico/bgp_processor.go) * [Calico 3.32.2 BIRD template](https://github.com/projectcalico/calico/blob/v3.32.2/confd/etc/calico/confd/templates/bird.cfg.template) * [Pinned BIRD best-path implementation](https://github.com/projectcalico/bird/blob/9111ec3c3ff3e769727a5940d3d829a0be8b5201/proto/bgp/attrs.c) * [Pinned BIRD timers and GTSM](https://github.com/projectcalico/bird/blob/9111ec3c3ff3e769727a5940d3d829a0be8b5201/proto/bgp/bgp.c) * [Cisco IOS XE dynamic neighbors](https://www.cisco.com/c/en/us/td/docs/routers/ios/config/17-x/ip-routing/b-ip-routing/m_irg-bgp-dynamic-neighbors.html) * [Junos route-filter match types](https://www.juniper.net/documentation/en_US/junos/topics/usage-guidelines/policy-configuring-route-lists-for-use-in-routing-policy-match-conditions.html) * [Kubernetes Service API and externalIPs deprecation](https://kubernetes.io/docs/concepts/services-networking/service/) * [Calico 3.32.2 Service route generation](https://github.com/projectcalico/calico/blob/v3.32.2/confd/pkg/backends/calico/routes.go) ---------------------------------------- Source: https://www.atomai.click/kubernetes-docs/en/networking/calico/05-network-policy ---------------------------------------- # Part 5: Network Policy > **Review baseline**: Calico 3.32.2; Kubernetes 1.34–1.36 is Calico 3.32's tested range. **Last Updated**: September 12, 2026. > > Examples assume the standard Calico API server (`projectcalico.org/v3`), Kubernetes datastore and a compatible policy-enforcing dataplane. Use a dedicated `calico-demo` namespace with known workload labels, ready endpoints and a working baseline before applying policies. The sections are independent patterns, not a single manifest bundle. Existing higher-priority policies, DNS implementation, Service NAT, host policy and application behavior affect results. No production cluster, admission server or packet-forwarding test was run for this review. ## Introduction Network policies control permitted connections between workloads and other endpoints. Calico adds ordered policy, explicit actions, global scope and host endpoint controls to the standard Kubernetes API. Feature availability depends on the product and enforcement path: DNS-domain policy is a commercial extension, while Open Source HTTP policy requires the documented Istio/Dikastes integration. This deep dive covers both Kubernetes standard policies and Calico's extended capabilities, providing patterns and examples for enterprise security requirements. *** ## Kubernetes Standard NetworkPolicy ### NetworkPolicy Fundamentals Kubernetes NetworkPolicy is a namespace-scoped resource that selects Pods. Ingress and egress isolation are independent; the allowed traffic for each isolated direction is the union of matching Kubernetes policies. If both ends are isolated, the source's egress and destination's ingress must allow a new connection. Replies to an allowed connection do not need a separate reverse-direction allow rule. Entries in a `from`/`to` list are **OR** alternatives. A `namespaceSelector` and `podSelector` in the same entry are **AND** conditions. A pod selector without a namespace selector refers to the policy's namespace. No applicable Kubernetes policy means no isolation by that API for the direction; it does not override host firewalls, Calico policies or other controls. Policy changes and existing connections are implementation dependent, so validate with new connections. The `ipBlock` address observed before/after Service or load-balancer NAT also depends on the implementation. ![Comparison showing that without a NetworkPolicy every pod can reach every other pod freely, while a NetworkPolicy narrows that mesh down to one explicitly allowed path and blocks the rest.](https://raw.githubusercontent.com/Atom-oh/kubernetes-docs/main/en/.gitbook/assets/en-networking-calico-05-network-policy-0.png) [🔍 View interactive diagram](https://www.atomai.click/kubernetes-docs/archmaps/en-networking-calico-05-network-policy-0.html) > Arrows represent new connections under the illustrated policies, not response packets of an allowed connection. The example assumes no other policy, firewall or path restriction; merely having any NetworkPolicy does not isolate every Pod/direction. ### Basic NetworkPolicy Structure ```yaml apiVersion: networking.k8s.io/v1 kind: NetworkPolicy metadata: name: example-policy namespace: calico-demo spec: # Which pods this policy applies to podSelector: matchLabels: app: web # Policy types: Ingress, Egress, or both policyTypes: - Ingress - Egress # Ingress rules (who can connect TO these pods) ingress: - from: - podSelector: matchLabels: app: frontend - namespaceSelector: matchLabels: purpose: monitoring - ipBlock: cidr: 10.0.0.0/8 except: - 10.0.1.0/24 ports: - protocol: TCP port: 8080 # Egress rules (where these pods can connect TO) egress: - to: - podSelector: matchLabels: app: database ports: - protocol: TCP port: 5432 ``` ### Kubernetes NetworkPolicy Limitations | Capability | Kubernetes NetworkPolicy | Calico extension / prerequisite | | --- | --- | --- | | Scope | Namespaced Pod policy | GlobalNetworkPolicy can select workloads across namespaces and HostEndpoints | | Ordering/actions | Additive allow rules; no user-defined policy order | Tier/order and Allow, Deny, Log, Pass | | Ports | TCP/UDP/SCTP, named ports, numeric ranges via `endPort` (stable since 1.25; plugin support required) | Calico port-range syntax and additional IP protocol/ICMP matches | | HTTP methods/paths | Not part of this API | Open Source `http` rules require configured Istio/Dikastes application-layer enforcement | | DNS domain names | Not part of this API | For example, Calico Enterprise domain-policy capability; `domains` is absent from the Open Source 3.32 CRD | | Host interfaces | Not a general node firewall | HostEndpoint policy with separate local/forwarded/failsafe semantics | The standard NetworkPolicy object's scope is distinct from newer Kubernetes cluster-policy APIs; do not assume that all Kubernetes network security APIs are namespace-only. Check the selected dataplane's protocol and logging support rather than equating schema acceptance with enforcement. *** ## Calico NetworkPolicy Extensions ### Extended Protocol Support The SCTP examples require a compatible classic dataplane: Calico 3.32 eBPF does not support SCTP policy or Services. Calico supports additional protocols beyond TCP and UDP: ```yaml apiVersion: projectcalico.org/v3 kind: NetworkPolicy metadata: name: extended-protocols namespace: calico-demo spec: selector: app == 'network-tools' ingress: # ICMP ping - action: Allow protocol: ICMP icmp: type: 8 # Echo Request code: 0 # ICMPv6 - action: Allow protocol: ICMPv6 icmp: type: 128 # Echo Request # SCTP - action: Allow protocol: SCTP destination: ports: - 3868 # Diameter # UDP with port range - action: Allow protocol: UDP destination: ports: - "5000:6000" # Port range ``` ### Port Ranges and Named Ports ```yaml apiVersion: projectcalico.org/v3 kind: NetworkPolicy metadata: name: port-examples namespace: calico-demo spec: selector: app == 'multi-port-app' ingress: # Port range - action: Allow protocol: TCP destination: ports: - "8080:8090" # Named ports (from pod spec) - action: Allow protocol: TCP destination: ports: - http # References containerPort name - metrics # References containerPort name # Mix of specific ports and ranges - action: Allow protocol: TCP destination: ports: - 22 - 80 - 443 - "3000:3100" ``` ### Enhanced Selector Syntax Calico uses expression selectors. The following rules illustrate alternatives: combining every broad Allow into one policy widens the permitted set. `app != 'untrusted'` also matches resources without the label; it is not evidence of trust. `selector: !has(x)` matches known in-scope resources lacking the label. `notSelector: has(x)` negates the packet match and can also match external addresses absent from that selector. A rule's `selector: all()` does not match every packet; omit endpoint selector conditions to match all packets. ```yaml apiVersion: projectcalico.org/v3 kind: NetworkPolicy metadata: name: selector-examples namespace: calico-demo spec: # Label equality selector: app == 'web' ingress: # Set membership - action: Allow source: selector: app in {'frontend', 'api-gateway', 'monitoring'} # Negation - action: Allow source: selector: app != 'untrusted' # Label existence - action: Allow source: selector: has(security-cleared) # Combining conditions (AND) - action: Allow source: selector: app == 'backend' && tier == 'internal' # OR inside one selector expression - action: Allow source: selector: (app == 'frontend') || (app == 'api') # Namespace selector - action: Allow source: namespaceSelector: environment == 'production' selector: app == 'authorized-client' ``` *** ## GlobalNetworkPolicy GlobalNetworkPolicy is non-namespaced and can select workload endpoints across namespaces or HostEndpoints. `selector: all()` alone is not “only all application Pods.” These examples explicitly limit the selected workloads to the demo namespace, preserving system and host traffic. ### Default Deny with Explicit Exceptions The empty policy selects both directions. Its omitted `order` follows explicitly ordered policies; 10,000 is not a special “lowest priority” value. Empty rules do not override an earlier terminal Allow. Use a separately controlled earlier tier for restrictions that application policies must not override. ```yaml apiVersion: projectcalico.org/v3 kind: GlobalNetworkPolicy metadata: name: default.demo-default-deny spec: tier: default namespaceSelector: kubernetes.io/metadata.name == 'calico-demo' types: [Ingress, Egress] ingress: [] egress: [] --- apiVersion: projectcalico.org/v3 kind: GlobalNetworkPolicy metadata: name: default.demo-essential-egress spec: tier: default order: 100 namespaceSelector: kubernetes.io/metadata.name == 'calico-demo' selector: needs-platform == 'true' types: [Egress] egress: - action: Allow protocol: UDP destination: namespaceSelector: kubernetes.io/metadata.name == 'kube-system' selector: k8s-app == 'kube-dns' ports: [53] - action: Allow protocol: TCP destination: namespaceSelector: kubernetes.io/metadata.name == 'kube-system' selector: k8s-app == 'kube-dns' ports: [53] - action: Allow destination: services: name: kubernetes namespace: default ``` This example grants platform egress only to workloads labeled `needs-platform=true`. The Service match follows the actual `kubernetes/default` endpoints and ports on the **Kubernetes datastore**; it is ignored with an etcd datastore. Do not mix that Service match with destination ports, CIDRs or selectors. Network access to the API is distinct from API authentication/RBAC. Confirm the actual DNS deployment. Namespace plus Pod selectors prevent an unrelated Pod labeled `kube-dns` from becoming a trusted resolver. Node-local DNS requires a different match for the actual path. Creating an empty policy on `kube-system` before enumerating system dependencies can break the cluster; demonstrate that pattern in a separate test namespace instead. ### An Earlier Egress Guardrail This independent example blocks the IPv4 metadata address for selected demo workloads, then delegates other traffic at the end of its tier. It does not claim complete SSRF protection, protection of privileged/host-networked processes, or coverage of every platform metadata endpoint. ```yaml apiVersion: projectcalico.org/v3 kind: Tier metadata: name: egress-guardrail spec: order: 50 defaultAction: Pass --- apiVersion: projectcalico.org/v3 kind: GlobalNetworkPolicy metadata: name: egress-guardrail.block-imds-v4 spec: tier: egress-guardrail order: 10 namespaceSelector: kubernetes.io/metadata.name == 'calico-demo' types: [Egress] egress: - action: Deny destination: nets: [169.254.169.254/32] ``` Do not place an unconditional Pass before later restrictions in the same tier. Preserve platform-required identity/DNS paths and review the actual workload-to-host enforcement behavior. *** ## NetworkSet and GlobalNetworkSet NetworkSets label reusable IP/CIDR groups; policy selectors reference their labels, not their object names. The address examples below are illustrative and are not a real country/threat feed. A label selector may also match endpoints carrying the same labels, so use controlled labels and the intended namespace/global scope. Use `namespaceSelector: global()` in a namespaced policy's entity match when selecting a GlobalNetworkSet. Namespaced NetworkSets remain in their selected namespace. If trusted and blocked ranges overlap, evaluate the deny first; an earlier Allow is terminal. ### NetworkSet (Namespace-scoped) ```yaml apiVersion: projectcalico.org/v3 kind: NetworkSet metadata: name: corporate-networks namespace: calico-demo labels: network-type: corporate spec: nets: - 10.0.0.0/8 - 172.16.0.0/12 - 192.168.0.0/16 --- # Reference in policy apiVersion: projectcalico.org/v3 kind: NetworkPolicy metadata: name: allow-corporate namespace: calico-demo spec: selector: app == 'internal-app' ingress: - action: Allow source: selector: network-type == 'corporate' # References NetworkSet by label ``` ### GlobalNetworkSet (Cluster-scoped) ```yaml apiVersion: projectcalico.org/v3 kind: GlobalNetworkSet metadata: name: external-trusted-ips labels: network-group: external-trusted spec: nets: - 203.0.113.0/24 # Partner network - 198.51.100.0/24 # CDN network - 192.0.2.50/32 # Specific trusted IP --- apiVersion: projectcalico.org/v3 kind: GlobalNetworkSet metadata: name: demo-blocked-networks labels: network-group: blocked spec: nets: # Illustrative test ranges, not geolocation or threat intelligence - 192.0.2.128/25 - 203.0.113.128/25 --- # Reference in GlobalNetworkPolicy apiVersion: projectcalico.org/v3 kind: GlobalNetworkPolicy metadata: name: external-access-control spec: namespaceSelector: kubernetes.io/metadata.name == 'calico-demo' selector: has(external-facing) order: 200 types: - Ingress ingress: # Deny overlaps before considering trusted ranges. - action: Deny source: namespaceSelector: global() selector: network-group == 'blocked' - action: Allow source: namespaceSelector: global() selector: network-group == 'external-trusted' ``` *** ## Tiered Policies Tiers are available in Calico Open Source 3.32. They group namespaced and global Calico policies; they are not a progression from Kubernetes NetworkPolicy to GlobalNetworkPolicy. ### Evaluation and Defaults For normal endpoint policy, evaluate tiers by increasing `order`, then policies within each tier by increasing `order`. An unset policy order follows explicitly ordered policies. Consider the selected endpoint **and traffic direction**. | Situation | Result | | --- | --- | | No policy in the tier selects that endpoint/direction | Skip the tier | | A rule Allows or Denies | Finish this endpoint/direction's policy decision | | A rule Logs | Continue to the next rule | | A rule Passes | Skip the remaining policies in this tier and try the next applicable tier | | Applicable tier has no terminal rule match | Apply its `defaultAction`, which defaults to Deny | | Last applicable tier Passes | Evaluate endpoint Profiles; no profile allow means deny | This is not “no rule matches, therefore always move to the next tier.” Also, an Allow at one endpoint does not bypass the other endpoint's policy. Pre-DNAT and untracked host policy have different fall-through behavior, covered below. The built-in `default` tier has fixed order **1,000,000**, not 1,000 or infinity. Kubernetes NetworkPolicy and Calico policies without an explicit tier belong there. Current `kube-admin` and `kube-baseline` tiers use 1,000 and 10,000,000 with Pass defaults for the corresponding Kubernetes cluster-policy integration; therefore `default` is not universally the last possible tier. ### Separate Security, Platform and Application Decisions These example tiers use end-of-tier Pass for security/platform so all their applicable rules are checked before delegation. Tier creation and reordering require centrally controlled privileges. ```yaml apiVersion: projectcalico.org/v3 kind: Tier metadata: name: security spec: order: 100 defaultAction: Pass --- apiVersion: projectcalico.org/v3 kind: Tier metadata: name: platform spec: order: 200 defaultAction: Pass --- apiVersion: projectcalico.org/v3 kind: Tier metadata: name: application spec: order: 500 defaultAction: Deny ``` The example below uses a documentation-only threat address, then a separate restricted-data rule. A Pass at the end of the first security policy would skip the second policy. Keeping delegation at the **end of the tier** avoids that bypass. The restricted-data labels illustrate segmentation, not complete PCI DSS compliance. ```yaml apiVersion: projectcalico.org/v3 kind: GlobalNetworkSet metadata: name: demo-threats labels: network-group: demo-threat spec: nets: - 192.0.2.100/32 --- apiVersion: projectcalico.org/v3 kind: GlobalNetworkPolicy metadata: name: security.block-threats spec: tier: security order: 10 namespaceSelector: kubernetes.io/metadata.name == 'calico-demo' types: [Ingress, Egress] ingress: - action: Deny source: namespaceSelector: global() selector: network-group == 'demo-threat' egress: - action: Deny destination: namespaceSelector: global() selector: network-group == 'demo-threat' --- apiVersion: projectcalico.org/v3 kind: GlobalNetworkPolicy metadata: name: security.restricted-data spec: tier: security order: 20 namespaceSelector: kubernetes.io/metadata.name == 'calico-demo' selector: data-scope == 'restricted' types: [Ingress] ingress: - action: Deny source: notSelector: data-scope == 'restricted' --- apiVersion: projectcalico.org/v3 kind: GlobalNetworkPolicy metadata: name: platform.dns spec: tier: platform order: 10 namespaceSelector: kubernetes.io/metadata.name == 'calico-demo' types: [Egress] egress: - action: Allow protocol: UDP destination: namespaceSelector: kubernetes.io/metadata.name == 'kube-system' selector: k8s-app == 'kube-dns' ports: [53] - action: Allow protocol: TCP destination: namespaceSelector: kubernetes.io/metadata.name == 'kube-system' selector: k8s-app == 'kube-dns' ports: [53] --- apiVersion: projectcalico.org/v3 kind: NetworkPolicy metadata: name: application.frontend namespace: calico-demo spec: tier: application order: 10 selector: app == 'frontend' types: [Ingress, Egress] ingress: - action: Allow protocol: TCP source: selector: app == 'gateway' destination: ports: [8080] egress: - action: Allow protocol: TCP destination: selector: app == 'backend' ports: [8080] ``` `namespaceSelector: global()` plus a separate label selector selects the GlobalNetworkSet. `global(label == 'value')` is not valid syntax. The platform DNS Allow is an intentional terminal exception for the selected workload's egress; application-tier rules cannot subsequently narrow that exception. The application policy governs only the selected frontend, not every workload in the namespace. The DNS example assumes conventional CoreDNS Pods with verified namespace/labels. NodeLocal DNSCache or EKS Auto Mode node-local DNS needs rules for the actual resolver path, not a Pod selector copied unchanged. Keep access to the required resolver and validate both UDP and TCP queries. ### Tier RBAC Integration Calico tier RBAC uses the pseudo-resources `tier.networkpolicies` and `tier.globalnetworkpolicies`, plus `get` on the target Tier. The Calico authorizer explicitly checks synthetic names such as `application.*`. This is not a general Kubernetes `resourceNames` wildcard on ordinary `networkpolicies`. The following complete binding example grants one service account namespaced policy editing in the application tier. It does not grant tier creation/reordering or global policy administration. ```yaml apiVersion: v1 kind: ServiceAccount metadata: name: policy-editor namespace: calico-demo --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: demo-get-application-tier rules: - apiGroups: ["projectcalico.org"] resources: ["tiers"] resourceNames: ["application"] verbs: ["get"] --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: demo-get-application-tier subjects: - kind: ServiceAccount name: policy-editor namespace: calico-demo roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: demo-get-application-tier --- apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: name: demo-edit-application-policies namespace: calico-demo rules: - apiGroups: ["projectcalico.org"] resources: ["tier.networkpolicies"] resourceNames: ["application.*"] verbs: ["get", "list", "watch", "create", "update", "patch", "delete"] --- apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: name: demo-edit-application-policies namespace: calico-demo subjects: - kind: ServiceAccount name: policy-editor namespace: calico-demo roleRef: apiGroup: rbac.authorization.k8s.io kind: Role name: demo-edit-application-policies ``` Global policy editing uses `tier.globalnetworkpolicies` in a deliberately scoped ClusterRole/ClusterRoleBinding. Keep `get` on the intended Tier separate from permissions to modify Tier order. The example assumes the standard Calico aggregated API server. Native v3 CRDs use an admission webhook for tier authorization of create/update/delete; that webhook cannot restrict GET/LIST/WATCH. Do not claim identical read isolation between the two modes. Plain `kubectl auth can-i` does not validate the combined Calico tier checks: test actual allowed and forbidden requests in an isolated environment with a principal that has no broader bindings. Kubernetes RBAC is additive, so an existing broad grant can defeat this intended restriction. *** ## FQDN-Based Egress Policy The Open Source 3.32 CRD has no `destination.domains` or Felix `dnsTrustedServers` field. Setting `policySyncPathPrefix` enables the policy-sync path used by application-layer integrations; it does not add DNS-domain policy to Open Source. Calico Enterprise 3.23 documents DNS-domain matches on **egress Allow** rules. The controller learns A/AAAA/CNAME answers from trusted DNS servers and permits matching destination IPs. This is IP-based enforcement, not HTTPS hostname authentication, so shared destination IPs and application identity still matter. Use workload/Service selectors for in-cluster services. The following commercial example assumes that feature is enabled, the trusted resolver is verified, and no earlier terminal Allow bypasses it. Each rule has one `destination` map; repeating that YAML key could silently discard its domain restriction. ```yaml # Calico Enterprise 3.23 example; NOT an Open Source 3.32 resource. apiVersion: projectcalico.org/v3 kind: NetworkPolicy metadata: name: allow-approved-domains namespace: calico-demo spec: selector: app == 'external-api-client' types: [Egress] egress: - action: Allow protocol: UDP destination: namespaceSelector: kubernetes.io/metadata.name == 'kube-system' selector: k8s-app == 'kube-dns' ports: [53] - action: Allow protocol: TCP destination: namespaceSelector: kubernetes.io/metadata.name == 'kube-system' selector: k8s-app == 'kube-dns' ports: [53] - action: Allow protocol: TCP destination: domains: - api.github.com - "*.example.com" ports: [443] ``` Replace the documentation domain with an approved real domain. `*.example.com` matches `api.example.com` and `deep.api.example.com`, but not the apex `example.com`. The wildcard must occupy a complete component and only one wildcard is supported. Inline DNS policy mode supports prefix wildcards; non-prefix wildcard patterns require an appropriate documented mode. A broad suffix such as `*.amazonaws.com` is not an AWS account or IAM boundary. Keep the actual resolver reachable over UDP and TCP and align its IPs with the trusted DNS configuration. Node-local resolvers require deployment-specific handling. The commercial guide excludes domain policy on the egress hook of egress-gateway Pods because its node-wide DNS cache can make matches absent or intermittent. On Open Source, use an explicit egress proxy/gateway with its own application authorization, or maintained IP/CIDR NetworkSets when suitable. Do not substitute a one-time DNS lookup for a continuously enforced domain policy. *** ## HTTP Method Filtering (Layer 7) Calico Open Source 3.32 supports HTTP policy through the documented **Istio + Dikastes** integration. Merely installing an arbitrary Envoy proxy or adding an `http` field to a normal CNI policy does not enable Layer 7 enforcement. Prerequisites include the Felix Policy Sync API, the Calico CSI socket mount, Dikastes injection and Envoy external authorization on the relevant traffic path. The current integration guide describes Kubernetes native-sidecar Istio injection and recommends Istio 1.28.1; that statement is not a blanket compatibility promise for every newer Istio. Check the maintained [Istio installation guide](https://www.atomai.click/kubernetes-docs/llms/en/service-mesh/istio/01-installation.md), both projects' support windows and the actual integration before selecting a production combination. No such integration deployment was run here. These are **ingress Allow** examples on an already integrated workload. HTTPS methods/paths require the enforcement proxy to see the HTTP request after TLS termination. Source Pod labels identify workloads, not authenticated end users; configure the integration's trusted workload identity/mTLS path and preserve the network access needed by DNS and the Istio control plane. Validate allowed and denied requests, including proxy/authorization-service failure behavior, before relying on the policy. ### HTTP Match Rules ```yaml apiVersion: projectcalico.org/v3 kind: GlobalNetworkPolicy metadata: name: l7-http-policy spec: namespaceSelector: kubernetes.io/metadata.name == 'calico-demo' selector: app == 'api-server' order: 300 types: - Ingress ingress: # Allow only GET and HEAD for read-only clients - action: Allow source: selector: role == 'reader' http: methods: - GET - HEAD paths: - prefix: /api/v1/ # Allow the listed methods for admin-labeled workloads - action: Allow source: selector: role == 'admin' http: methods: - GET - POST - PUT - DELETE - PATCH # Allow health checks - action: Allow http: methods: - GET paths: - exact: /health - exact: /ready ``` ### Path-Based Filtering ```yaml apiVersion: projectcalico.org/v3 kind: NetworkPolicy metadata: name: path-based-policy namespace: calico-demo spec: selector: app == 'web-app' ingress: # Public endpoints - action: Allow http: paths: - prefix: /public/ - exact: / # Admin endpoints - restricted - action: Allow source: selector: role == 'admin' http: paths: - prefix: /admin/ # API endpoints - authenticated only - action: Allow source: selector: has(api-access) http: paths: - prefix: /api/ ``` *** ## Host Endpoint Protection HostEndpoint represents an interface on a node that Calico manages. Creating one can change host connectivity immediately. `defaultEndpointToHostAction` controls workload-to-local-host behavior; it does not create HostEndpoints. `Installation.calicoNetwork.hostPorts` controls hostPort support, not automatic host protection. ### Manual Host Endpoint and Policy The following **field example is not a complete host firewall**. It assumes a self-managed test worker `demo-worker` at `10.0.1.10`, bastion `10.0.0.100`, control-plane source `10.0.1.5` and interface `eth0`. Replace them with verified identities/addresses and prepare all required management, DNS, DHCP, BGP, API, health and egress rules before creating the HostEndpoint. It does not model an EKS-managed control-plane node. ```yaml apiVersion: projectcalico.org/v3 kind: GlobalNetworkPolicy metadata: name: default.demo-worker-ingress spec: order: 100 selector: host-demo == 'true' && !has(projectcalico.org/namespace) types: [Ingress] ingress: - action: Allow protocol: TCP source: nets: [10.0.0.100/32] destination: ports: [22] - action: Allow protocol: TCP source: nets: [10.0.1.5/32] destination: ports: [10250] --- apiVersion: projectcalico.org/v3 kind: HostEndpoint metadata: name: demo-worker-eth0 labels: host-demo: "true" spec: node: demo-worker interfaceName: eth0 expectedIPs: [10.0.1.10] ``` The kubelet rule targets the worker's authenticated 10250 endpoint. Do not open the obsolete unauthenticated 10255 read-only port as a default requirement. The example defines ingress only; a manually created HostEndpoint without an egress policy/profile may deny host-originated traffic. Complete the actual baseline first. Calico's default failsafes include inbound TCP 22 and other connectivity ports. They bypass the restrictive intent of the SSH rule above, so this rule alone does **not** limit SSH to the bastion. Review the failsafe list and a tested recovery path before changing it; do not blindly empty the lists. ### Automatic Host Endpoints The node controller's `KubeControllersConfiguration.spec.controllers.node.hostEndpoint.autoCreate` controls automatic creation. Use a merge patch to preserve other controller settings: ```bash kubectl get kubecontrollersconfiguration.projectcalico.org default -o yaml # Apply only after reviewing existing host endpoints and global policies. kubectl patch kubecontrollersconfiguration.projectcalico.org default --type=merge \ -p '{"spec":{"controllers":{"node":{"hostEndpoint":{"autoCreate":"Enabled"}}}}}' ``` This can affect all eligible nodes. Automatic endpoints normally carry a default-allow profile; that profile does not override matching policy Deny. Custom templates and `createDefaultHostEndpoint` can narrow the generated endpoint set, but changing them on an existing deployment requires reviewing existing endpoints and policies first. ### Local and Forwarded Traffic Normal host policy defaults `applyOnForward` to false. With true it also applies to forwarded traffic, which must still pass the relevant workload policy. If no forward policy selects the endpoint/direction, forwarded traffic is allowed by default; selected forward policy with no allow denies it. Locally terminated host traffic has a different default-deny behavior (subject to profiles/failsafes). ## DoNotTrack and PreDNAT Policies These are Linux host-policy patterns. They apply to HostEndpoints, not a way to disable tracking on ordinary selected Pods. Confirm support on the chosen dataplane. `doNotTrack` and `preDNAT` cannot both be true, and either requires `applyOnForward: true`. ### DoNotTrack An untracked Allow skips connection tracking for matching traffic. It is not a universal performance improvement and may conflict with a Service/NAT path that needs conntrack. Requests and responses require explicit rules; this example assumes a DNS process on the test host serving the trusted client subnet directly. ```yaml apiVersion: projectcalico.org/v3 kind: GlobalNetworkPolicy metadata: name: default.demo-untracked-dns spec: selector: host-demo == 'true' && !has(projectcalico.org/namespace) order: 10 types: [Ingress, Egress] doNotTrack: true applyOnForward: true ingress: - action: Allow protocol: UDP source: nets: [10.0.0.0/24] destination: ports: [53] - action: Allow protocol: TCP source: nets: [10.0.0.0/24] destination: ports: [53] egress: - action: Allow protocol: UDP source: ports: [53] destination: nets: [10.0.0.0/24] - action: Allow protocol: TCP source: ports: [53] destination: nets: [10.0.0.0/24] ``` Unlike normal endpoint policy, a miss in the untracked stage does not impose an end-of-tier default drop; subsequent tracked policy can still apply. This is not an implicit deny-all firewall for the host. ### PreDNAT Pre-DNAT policy sees the original destination IP/port before DNAT. It is ingress-only and uses normal connection tracking for permitted return traffic. This example protects the specific TCP NodePort 30080 on the selected host path: ```yaml apiVersion: projectcalico.org/v3 kind: GlobalNetworkPolicy metadata: name: default.demo-nodeport spec: selector: host-demo == 'true' && !has(projectcalico.org/namespace) order: 20 types: [Ingress] preDNAT: true applyOnForward: true ingress: - action: Allow protocol: TCP source: nets: [10.0.0.0/24] destination: ports: [30080] - action: Deny protocol: TCP destination: ports: [30080] ``` There is no default-drop at the pre-DNAT stage. Unmatched traffic continues to subsequent host/workload policy. The explicit second rule rejects untrusted traffic to this NodePort; other ports are outside this example. A path that goes directly to a Pod without traversing this host NodePort is not covered by the rule. *** ## Policy Debugging Inspect actual labels, namespaces, Service endpoints, all applicable tiers and both directions. In the standard Calico API server, a list without a tier selector can default to the `default` tier; `-A` means all namespaces, not automatically all tiers. ```bash kubectl get networkpolicies.networking.k8s.io -n calico-demo -o yaml kubectl get tiers.projectcalico.org -o yaml for CALICO_TIER in $(kubectl get tiers.projectcalico.org -o jsonpath='{.items[*].metadata.name}'); do kubectl get networkpolicies.projectcalico.org -n calico-demo \ -l "projectcalico.org/tier=$CALICO_TIER" -o yaml kubectl get globalnetworkpolicies.projectcalico.org \ -l "projectcalico.org/tier=$CALICO_TIER" -o yaml done calicoctl get workloadendpoint -n calico-demo \ --selector="app == 'frontend'" -o yaml ``` ```bash CALICO_NAMESPACE=calico-system CALICO_NODE=demo-worker CALICO_POD="$(kubectl -n "$CALICO_NAMESPACE" get pods -l k8s-app=calico-node \ --field-selector "spec.nodeName=$CALICO_NODE" -o jsonpath='{.items[0].metadata.name}')" test -n "$CALICO_POD" kubectl -n "$CALICO_NAMESPACE" logs "$CALICO_POD" -c calico-node --tail=200 kubectl -n "$CALICO_NAMESPACE" exec "$CALICO_POD" -c calico-node -- \ calico-node -felix-ready ``` ```bash # Assumes named, ready test Pods and an nc binary in the client image. TARGET_POD=backend-test TARGET_IP="$(kubectl -n calico-demo get pod "$TARGET_POD" -o jsonpath='{.status.podIP}')" test -n "$TARGET_IP" kubectl -n calico-demo exec frontend-client -- nc -z -w 3 "$TARGET_IP" 8080 ``` The `calico-node -felix-ready` command checks readiness. It is not a policy trace or a measurement of per-packet evaluation time. The released Open Source `calicoctl` does not provide the former `policy-trace` command. Listing an endpoint or grepping policy text also does not calculate the complete effective policy. Test an allowed client, an untrusted client, a wrong port, a cross-namespace client and resolver access using **new** connections. Check the target's destination port, not the client's ephemeral source port. Test direct Pod and Service addresses separately to distinguish policy from endpoint/NAT/forwarding problems; kube-proxy or its replacement can therefore be relevant. In the iptables dataplane, inspect actual `cali-` chains on the correct node/network namespace; a placeholder hash is not a real chain name. iptables Log actions write to the host kernel log, whereas Felix stdout is primarily component/controller diagnostics. Neither a missing stdout message nor a zero counter proves an unused policy under every path. eBPF/nftables require their own backend diagnostics; `tc filter show` alone does not explain an effective policy verdict. ### Stage before Enforcement Open Source 3.32 provides `StagedNetworkPolicy`, `StagedGlobalNetworkPolicy` and `StagedKubernetesNetworkPolicy`. Staged resources do not enforce packet decisions. With the flow-log/Whisker pipeline configured, inspect `policies.pending` to preview observed effects. ```yaml apiVersion: projectcalico.org/v3 kind: StagedNetworkPolicy metadata: name: default.preview-backend-egress namespace: calico-demo spec: tier: default order: 100 selector: app == 'frontend' types: [Egress] egress: - action: Allow protocol: TCP destination: selector: app == 'backend' ports: [8080] ``` This preview deliberately contains only the backend connection. Check whether required DNS or other flows would be denied before creating an equivalent enforced policy. Absence of observed traffic is not proof that a dependency is unnecessary. `action: Log` alone is not a universal audit-only policy mode: evaluation continues and may still end in a deny. *** ## Common Policy Patterns Library ### Frontend → Backend → Database The demo assumes ready workloads with the shown labels and listeners. Every server port is a **destination** port; matching `source.ports: [8080]` would normally reject clients using ephemeral source ports. All numeric port rules specify TCP. The demo gateway is an application fixture, not an assumed label of a particular ingress controller. ```yaml apiVersion: projectcalico.org/v3 kind: NetworkPolicy metadata: name: default.frontend namespace: calico-demo spec: order: 100 selector: app == 'frontend' types: [Ingress, Egress] ingress: - action: Allow protocol: TCP source: selector: app == 'gateway' destination: ports: [8080] egress: - action: Allow protocol: TCP destination: selector: app == 'backend' ports: [8080] --- apiVersion: projectcalico.org/v3 kind: NetworkPolicy metadata: name: default.backend namespace: calico-demo spec: order: 100 selector: app == 'backend' types: [Ingress, Egress] ingress: - action: Allow protocol: TCP source: selector: app == 'frontend' destination: ports: [8080] egress: - action: Allow protocol: TCP destination: selector: app == 'database' ports: [5432] --- apiVersion: projectcalico.org/v3 kind: NetworkPolicy metadata: name: default.database namespace: calico-demo spec: order: 100 selector: app == 'database' types: [Ingress, Egress] ingress: - action: Allow protocol: TCP source: selector: app == 'backend' destination: ports: [5432] egress: [] ``` Allow DNS separately for the clients that need it, using the scoped essential-egress pattern and its prerequisite label. The database starts no new egress connections in this simplified pattern; stateful replies still work. Backups, replication and external dependencies need their own reviewed rules. ### Tenant Isolation Calico does not interpolate `$(namespace.tenant)`, `${namespace.labels.tenant}` or `${namespace.name}` inside selectors. Generate one policy per explicit tenant value, or use a namespaced policy for same-namespace isolation. ```yaml apiVersion: projectcalico.org/v3 kind: GlobalNetworkPolicy metadata: name: default.team-a-isolation spec: order: 500 namespaceSelector: tenant == 'team-a' types: [Ingress, Egress] ingress: - action: Allow source: namespaceSelector: tenant == 'team-a' egress: - action: Allow destination: namespaceSelector: tenant == 'team-a' - action: Allow protocol: UDP destination: namespaceSelector: kubernetes.io/metadata.name == 'kube-system' selector: k8s-app == 'kube-dns' ports: [53] - action: Allow protocol: TCP destination: namespaceSelector: kubernetes.io/metadata.name == 'kube-system' selector: k8s-app == 'kube-dns' ports: [53] ``` This allows all intra-team-a traffic, including across namespaces with that label. Namespace-label administration and policy-edit permissions must be controlled; earlier Allows can override the intended isolation. Test cross-tenant and unlabelled namespace cases. ### Same Namespace plus Shared Services In this namespaced policy, entity selectors without a namespace selector stay within `calico-demo`. This is an alternative to the restrictive microservice pattern, not an additional policy to layer on top of it. ```yaml apiVersion: projectcalico.org/v3 kind: NetworkPolicy metadata: name: default.namespace-and-shared namespace: calico-demo spec: order: 200 selector: all() types: [Ingress, Egress] ingress: - action: Allow source: selector: all() egress: - action: Allow destination: selector: all() - action: Allow protocol: TCP destination: namespaceSelector: kubernetes.io/metadata.name == 'logging' selector: app == 'log-receiver' ports: [24224] - action: Allow protocol: TCP destination: namespaceSelector: kubernetes.io/metadata.name == 'auth' selector: app == 'identity-provider' ports: [8080] ``` Add the required resolver rule separately and ensure destination-side policy permits the client. A logging port or identity-provider label is an explicit demo assumption; verify actual receiver protocol/port and application authentication. ### Open Source Egress Control Maintain approved addresses in a NetworkSet when the service has an address contract. The example IP is documentation-only; it is not a real API endpoint or a permanent DNS resolution. ```yaml apiVersion: projectcalico.org/v3 kind: NetworkSet metadata: name: approved-api-ips namespace: calico-demo labels: destination-group: approved-api spec: nets: [203.0.113.10/32] --- apiVersion: projectcalico.org/v3 kind: NetworkPolicy metadata: name: default.approved-api-egress namespace: calico-demo spec: order: 100 selector: app == 'external-api-client' types: [Egress] egress: - action: Allow protocol: TCP destination: selector: destination-group == 'approved-api' ports: [443] ``` Use the earlier DNS rule if the application resolves names. For dynamic external services, use an appropriately authorized proxy or the separately documented commercial domain policy. Allowing all RFC1918 space does not mean “only this cluster” and may grant access to unrelated private networks. ### Default Deny as Part of a Security Design Combine a scoped default-deny baseline with explicit required connections, controlled label/RBAC ownership and application authentication. This is network segmentation, not by itself a complete zero-trust or compliance implementation. Stage policies and test negative paths before enforcement. *** ## Policy Performance Impact Policy count alone is not a capacity benchmark. Cost depends on endpoint count, selector changes, rule structure, active flows, update rate and the chosen dataplane. The former “1,000 policies is very slow” and linear-cost assertions had no measured environment or raw results. Equality, set membership and label-existence selectors can all benefit from Calico's selector optimizations. Do not merge policies in a way that broadens access solely to reduce object count. Reuse maintained NetworkSets, bound logging volume and measure convergence under representative changes. Readiness checks, grepping “Policy sync” and counting lines from `iptables -L` do not measure rule-evaluation latency. Enable and scrape the documented Felix metrics endpoint, inspect metric TYPE/HELP and units, then correlate programming/update measurements with a controlled workload: ```bash # After enabling the documented Felix metrics endpoint through its config owner: kubectl -n "$CALICO_NAMESPACE" port-forward "pod/$CALICO_POD" 9091:9091 ``` ```bash # In another terminal while the localhost port-forward remains active: curl --fail --silent --show-error http://127.0.0.1:9091/metrics ``` Felix metrics are disabled by default; enable `prometheusMetricsEnabled` through the configuration owner first. The endpoint must be reachable on the selected node. This is metrics discovery, not a published performance result. Test positive and negative traffic paths while changing policies and preserve the exact Calico/Kubernetes/kernel versions, dataplane, topology, load and measurements. ## Operational Principles 1. Start in an isolated namespace, inventory required connections, then stage and enforce explicit rules. 2. Treat labels, namespace labels and policy-edit RBAC as part of the authorization boundary. 3. Review terminal Allow/Pass effects whenever tiers or their order change. 4. Keep host management/failsafe and Service/DNS requirements separate from application rules. 5. Combine network segmentation with workload/end-user authentication; do not claim that an IP rule alone prevents every SSRF or compliance failure. *** ## References * [Calico NetworkPolicy API](https://docs.tigera.io/calico/latest/reference/resources/networkpolicy) * [GlobalNetworkPolicy API](https://docs.tigera.io/calico/latest/reference/resources/globalnetworkpolicy) * [Tier evaluation](https://docs.tigera.io/calico/latest/reference/resources/tier) * [Tier RBAC](https://docs.tigera.io/calico/latest/network-policy/policy-tiers/rbac-tiered-policies) * [Kubernetes NetworkPolicy](https://kubernetes.io/docs/concepts/services-networking/network-policies/) * [Kubernetes RBAC](https://kubernetes.io/docs/reference/access-authn-authz/rbac/) * [Open Source Istio/Dikastes application policy](https://docs.tigera.io/calico/latest/network-policy/istio/app-layer-policy) * [Enterprise domain policy](https://docs.tigera.io/calico-enterprise/latest/network-policy/domain-based-policy) * [Staged policies](https://docs.tigera.io/calico/latest/network-policy/staged-network-policies) * [Host failsafes](https://docs.tigera.io/calico/latest/reference/host-endpoints/failsafe) * [Pre-DNAT](https://docs.tigera.io/calico/latest/reference/host-endpoints/pre-dnat) * [Forwarded host traffic](https://docs.tigera.io/calico/latest/reference/host-endpoints/forwarded) * [KubeControllersConfiguration](https://docs.tigera.io/calico/latest/reference/resources/kubecontrollersconfig) * [Policy logging](https://docs.tigera.io/calico/latest/network-policy/policy-rules/log-rules) * [Component metrics](https://docs.tigera.io/calico/latest/operations/monitor/monitor-component-metrics) * [Calico eBPF protocol support](https://docs.tigera.io/calico/latest/operations/ebpf/install) ---------------------------------------- Source: https://www.atomai.click/kubernetes-docs/en/networking/calico/06-ebpf-dataplane ---------------------------------------- # Part 6: eBPF Dataplane > **Review baseline**: Calico 3.32.2; Calico 3.32 tests Kubernetes 1.34–1.36. **Last Updated**: September 12, 2026. > > Examples assume an existing compatible Linux Calico cluster and the standard Calico API server. Choose the installation owner's workflow; these are alternative configuration fragments, not a sequence to apply to every cluster. This review did not load BPF programs, migrate a cluster or reproduce the reported benchmarks. ## Introduction Calico's eBPF dataplane uses BPF programs and maps for workload networking, policy and Kubernetes Service handling. It can reduce overhead on suitable paths, but performance depends on the workload and configuration. Calico also provides classic Linux dataplanes and Windows HNS; eBPF is not a universal upgrade for every platform. This deep dive explores eBPF fundamentals from a networking perspective, Calico's eBPF architecture, migration strategies, and performance optimization techniques. *** ## eBPF Fundamentals ### What is eBPF? eBPF (extended Berkeley Packet Filter) is a revolutionary technology that allows running sandboxed programs in the Linux kernel without modifying kernel source code or loading kernel modules. ![Generic BPF loading and hook model.](https://raw.githubusercontent.com/Atom-oh/kubernetes-docs/main/en/.gitbook/assets/en-networking-calico-06-ebpf-dataplane-1.png) [🔍 View interactive diagram](https://www.atomai.click/kubernetes-docs/archmaps/en-networking-calico-06-ebpf-dataplane-1.html) > The VM is an abstract instruction model. With JIT, the loaded program runs as native code at its hook; there is no additional guest VM after JIT. This is not an exact inventory of Calico hooks. ### Key eBPF Concepts for Networking | Concept | Description | Use in Calico | | ------------ | ---------------------------------------- | ----------------------------------- | | **Programs** | Bytecode executed at kernel hooks | Packet filtering, routing | | **Maps** | Key-value stores shared between programs | Route tables, policy rules | | **Hooks** | Attachment points in kernel | XDP, TC, socket | | **Helpers** | Kernel functions callable from eBPF | Packet manipulation, map operations | | **BTF** | Type information for maps/programs | Debug info, CO-RE | ### eBPF vs iptables Both iptables packet processing and eBPF programs run in the kernel. Walking an iptables rule does not normally cross into userspace. kube-proxy is a control-plane process that programs Service rules; packets do not pass through that process in iptables mode. Hash-map lookups, longest-prefix matches, compiled policy, tail calls and connection tracking have different costs. “Every eBPF policy is O(1)” and “memory is constant regardless of rules/flows” are not valid conclusions. iptables can also use indexed IP sets, and its NAT rule selection normally occurs for the first packet of a connection, with conntrack applying established translations later. ## Calico eBPF Architecture | Mechanism | Role and scope | | --- | --- | | TC packet hooks | Policy, routing, connection state and packet-level Service handling; interface ingress/egress is not automatically the same as workload ingress/egress | | Cgroup socket-address hooks | Connect-time Service destination translation; the released loader attaches connect hooks and, when enabled for UDP, sendmsg/recvmsg hooks | | XDP | Early packet handling where supported/configured; distinguish Calico's classic-dataplane XDP acceleration from the full eBPF dataplane's own attachment logic | | Program/IP-set/counter maps | Support compiled programs and state; policy is not a single universal tuple-to-action map | These are different execution contexts, not a mandatory XDP → TC → sockops → sk_msg → TC pipeline. Calico's connect-time balancing does not inspect HTTP methods through sk_msg. Application-layer policy uses the separate [Istio/Dikastes integration](https://www.atomai.click/kubernetes-docs/llms/en/networking/calico/05-network-policy.md). ## BPF Map Structures The following are **version-specific IPv4 layouts from Calico 3.32.2**, not a stable public ABI or a recipe for writing kernel maps: | Map family | Type | Key / value bytes | Purpose | | --- | --- | --- | --- | | Routes | LPM trie | 8 / 8 | Destination prefix, flags and next-hop-or-interface union | | NAT frontend | LPM trie | 16 / 20 | Service/source-prefix match, backend group/count, affinity and flags | | NAT backend | Hash | 8 / 8 | Backend group/ordinal → address and port | | Conntrack v4 format | LRU hash | 16 / 88 | Protocol, address pair, ports, state and NAT metadata | | Affinity | LRU hash | Version-specific | Cached client-to-backend selection | For the IPv4 route key, the first four bytes hold the prefix length in little-endian order, followed by the IPv4 address bytes. Its value contains flags and a four-byte next-hop/interface-index union, not a MAC address. Conntrack uses a 32-bit protocol field followed by addresses and ports; the former simplified five-tuple struct was not its actual layout. IPv6 uses different layouts. Policies are compiled into BPF instructions, assisted by maps; a rule-counter map is not the policy itself. Use the matching Calico debug tool to decode current maps and inspect their actual type, capacity and version before interpreting raw bytes. ## Direct Server Return (DSR) ![Service traffic through an ingress Kubernetes node, with a DSR return-path alternative.](https://raw.githubusercontent.com/Atom-oh/kubernetes-docs/main/en/.gitbook/assets/en-networking-calico-06-ebpf-dataplane-6.png) [🔍 View interactive diagram](https://www.atomai.click/kubernetes-docs/archmaps/en-networking-calico-06-ebpf-dataplane-6.html) > The frontend may be a NodePort or another Service address. Calico handles return-source translation; an external cloud load balancer is not shown and its return-path restrictions still apply. Here “load-balancer node” means the Kubernetes node performing Service forwarding. DSR can bypass that node on the return path; it does not automatically bypass an external cloud load balancer. Calico performs the return-source translation, rather than requiring the application Pod to bind a VIP. | `bpfExternalServiceMode` | Remote-backend path | | --- | --- | | `Tunnel` (default) | Request and reply use the ingress node/tunnel path | | `DSR` | Request is tunneled to the remote node; reply goes directly toward the client | There is no `Disabled` or `IPIP` value for this field. Calico uses VXLAN for this Service forwarding, so MTU/underlay requirements matter in both modes. DSR additionally requires the fabric to permit a node to send traffic using the original frontend/ingress-node source address. The Calico AWS guidance requires nodes in the same subnet and source/destination checks disabled; do not generalize that into an arbitrary cross-subnet deployment. The current Calico troubleshooting guide excludes AWS/GCP external-load-balancer return paths that require the original target. Do not enable DSR for such traffic solely because same-subnet/source-check conditions are met. On an already working, compatible eBPF path, the setting is: ```yaml apiVersion: projectcalico.org/v3 kind: FelixConfiguration metadata: name: default spec: bpfExternalServiceMode: DSR ``` Merge through the configuration owner and test return routing, source validation and active connections. Changing the mode can disrupt connections. DSR and connect-time balancing are separate optimizations. ## Connect-Time Load Balancing ![Connect-time Service translation compared with packet-level Service translation.](https://raw.githubusercontent.com/Atom-oh/kubernetes-docs/main/en/.gitbook/assets/en-networking-calico-06-ebpf-dataplane-7.png) [🔍 View interactive diagram](https://www.atomai.click/kubernetes-docs/archmaps/en-networking-calico-06-ebpf-dataplane-7.html) > kube-proxy programs kernel rules rather than carrying packets itself. Existing iptables connections use conntrack after initial rule selection. CTLB bypasses that Service DNAT path, not all packet processing. The diagram is conceptual: kube-proxy programs kernel state; subsequent packets use conntrack rather than reselecting a backend through the full Service-rule list. Connect-time balancing translates a supported socket's Service destination before packet processing. It does not remove all routing, policy, connection tracking or every other form of NAT. The current field is `bpfConnectTimeLoadBalancing: TCP` (default), `Enabled` or `Disabled`. The old boolean `bpfConnectTimeLoadBalancingEnabled` is deprecated but remains accepted; inspect/remove obsolete overrides through the owner rather than setting both forms blindly. ```yaml apiVersion: projectcalico.org/v3 kind: FelixConfiguration metadata: name: default spec: bpfConnectTimeLoadBalancing: TCP bpfHostNetworkedNATWithoutCTLB: Enabled ``` `Enabled` can include UDP socket handling; `TCP` limits CTLB to TCP. `bpfHostNetworkedNATWithoutCTLB` controls the complementary host-network NAT path, not whether ClusterIP is generally supported. A service mesh that needs original Service addresses can require CTLB disabled; follow the tested integration instead of assuming socket rewriting is always compatible. ## XDP Acceleration ![Generic XDP verdicts for an incoming packet.](https://raw.githubusercontent.com/Atom-oh/kubernetes-docs/main/en/.gitbook/assets/en-networking-calico-06-ebpf-dataplane-3.png) [🔍 View interactive diagram](https://www.atomai.click/kubernetes-docs/archmaps/en-networking-calico-06-ebpf-dataplane-3.html) > These are framework actions. Native, generic and offloaded execution have different prerequisites; the figure does not promise that Calico exposes every action or hardware feature. These are generic XDP actions, not a promise that Calico provides every offload/rate-limiting feature shown. Native driver XDP can act before skb allocation; Generic XDP runs later with an skb. Hardware offload support depends on the NIC, driver and program. No universal speed ranking follows from the mode name. The Felix `xdpEnabled` field is a **boolean** for suitable untracked ingress-deny acceleration in the classic iptables dataplane. `genericXDPEnabled` defaults to false; generic fallback is not automatically guaranteed. These knobs are distinct from the full eBPF dataplane's internal XDP programs. ```yaml # Separate classic iptables-dataplane acceleration example. apiVersion: projectcalico.org/v3 kind: FelixConfiguration metadata: name: default spec: xdpEnabled: true genericXDPEnabled: false ``` Inspect driver support and actual attachments on the intended interface. `xdpEnabled: Enabled`, `Offload` or `BestEffort` are not valid enum modes. *** ## eBPF Mode Requirements Use the current requirements for the selected release, not a historical minimum: | Requirement | Calico 3.32 eBPF scope | | --- | --- | | Base Linux kernel | 5.10 or newer; documented RHEL exception: RHEL 8.4 with kernel 4.18.0-305 or newer | | Architecture | x86-64 or little-endian arm64 | | Datastore | Kubernetes; etcd datastore is not supported for this mode | | Additional features | eBPF Log rules require kernel 5.16; documented QoS bandwidth controls require 6.6/TCX | | Underlay | Permit the configured VXLAN traffic between nodes, including NodePort forwarding even when Pod pools are unencapsulated | | Runtime | Required BPF/cgroup facilities, privileges and writable mounts; immutable OS variants require a suitable `CgroupV2Path` | BTF supplies type information for CO-RE and tooling; it is not the verifier itself or a guarantee of compatibility with every kernel. Calico's released loader selects CO-RE/non-CO-RE object variants where supported, so `/sys/kernel/btf/vmlinux` is not a complete readiness test. Verify the release requirements, node configuration and actual load diagnostics. Pinned objects in bpffs survive the creating process, not a host reboot as persistent disk data. ### Platform Boundaries The current Calico guide lists self-managed/kubeadm, kOps, OpenShift, EKS, MKE and qualified AKS/RKE paths. It explicitly excludes GKE, steady-state clusters mixing eBPF with the standard dataplane or Windows, and SCTP policy/services. IPv6 and IPv6-only operation are documented; “switch to dual-stack to fix missing IPv6 support” is not a general solution. AKS with Azure CNI cannot disable its managed kube-proxy; the guide treats Calico-networking AKS as a separate path still undergoing testing. An OS name, Ubuntu image or kernel version alone does not establish platform support. EKS node mode, CNI combination and OS variant must follow the specific Calico/EKS procedure; do not infer support for managed networking modes or environments unable to run the required privileged node components. Windows uses the Windows HNS dataplane, not Linux iptables. Do not plan a persistent per-node eBPF canary mixed with Windows/standard nodes. Validate in a separate representative cluster, then follow the documented coordinated transition. ### Read-only Inventory ```bash kubectl get nodes -o wide kubectl -n kube-system get daemonset kube-proxy -o yaml kubectl get installation.operator.tigera.io default -o yaml kubectl get felixconfiguration.projectcalico.org default -o yaml ``` Use the installation namespace and inspect the actual Calico/operator image versions. On each node, inspect `uname -r`, available BTF, bpffs and cgroup mounts in the host's mount namespace. A debug container's filesystem view is not automatically identical to the host. Do not upgrade an existing Helm release with a guessed release name or discard its values just to change dataplane mode. ## iptables to eBPF Migration ### Operator Automatic Bootstrap: Restricted Prerequisites This path applies to a self-managed kubeadm-based cluster installed with the Tigera Operator, with kube-proxy in `kube-system` **not managed by Helm, Argo CD or another reconciler**, and with the operator able to read the Kubernetes Service/endpoints. ```bash kubectl get installation.operator.tigera.io default -o yaml # Only when every automatic-bootstrap prerequisite above is met: kubectl patch installation.operator.tigera.io default --type=merge \ -p '{"spec":{"calicoNetwork":{"linuxDataplane":"BPF","bpfNetworkBootstrap":"Enabled","kubeProxyManagement":"Enabled"}}}' ``` The operator configures direct API access and manages kube-proxy during the transition. Rolling updates temporarily put nodes in different modes; the official guide explicitly notes possible NodePort disruption. Do not describe this as a guaranteed interruption-free migration. ### Manual Preparation and Ownership For other supported installations, first establish stable **direct** API-server access that does not depend on the Service implementation being replaced. Use the actual API load-balancer hostname/address and port. For EKS this is the cluster's API endpoint hostname, normally port 443; the example below is a placeholder for a self-managed API endpoint. ```yaml apiVersion: v1 kind: ConfigMap metadata: name: kubernetes-services-endpoint namespace: tigera-operator data: KUBERNETES_SERVICE_HOST: api.internal.example.com KUBERNETES_SERVICE_PORT: "6443" ``` Use `tigera-operator` for an operator installation; the standalone manifest workflow uses `kube-system`. The name is **`kubernetes-services-endpoint`** (plural). Confirm Calico has picked up the endpoint and can resolve/reach it before changing the Service dataplane. DNS bootstrap, security rules and reachability remain platform-specific prerequisites. If kube-proxy uses IPVS, the official migration requires switching it to iptables mode and a planned node restart first. Resolve this as a separate controlled change. Coordinate kube-proxy with its actual owner. Where it must remain running, such as the documented AKS Azure CNI case, merge these fields into the existing Felix configuration: ```yaml apiVersion: projectcalico.org/v3 kind: FelixConfiguration metadata: name: default spec: bpfKubeProxyIptablesCleanupEnabled: false bpfKubeProxyHealthzPort: 0 ``` The cleanup flag alone does not enable service handling. Running kube-proxy with cleanup enabled makes the components fight over iptables; leaving both health servers on 10256 also causes a conflict. Preserve unrelated Felix settings when merging. Enable the dataplane using **one** ownership path: ```bash # Operator installation: kubectl patch installation.operator.tigera.io default --type=merge \ -p '{"spec":{"calicoNetwork":{"linuxDataplane":"BPF"}}}' ``` ```bash # Alternative: standalone manifest installation, without operator ownership: kubectl patch felixconfiguration.projectcalico.org default --type=merge \ -p '{"spec":{"bpfEnabled":true}}' ``` For installations whose owner disables kube-proxy manually, follow the platform's documented sequence in the planned transition window. Save its desired configuration first. If a temporary nodeSelector is used, choose a previously unused key, verify it matches no nodes and later remove only the added key; never replace the entire existing selector map with null. Deleting a DaemonSet or scaling a nonexistent kube-proxy Deployment is not a general migration procedure. ### Validate Traffic, Not Only Loaded Programs Inspect rollout status and actual BPF programs/maps on the intended nodes. Test new Pod-to-Pod, DNS, ClusterIP, NodePort/external, policy-deny and required host-network connections across nodes. A program listing or a small iptables line count does not prove those paths work. The Kubernetes API normally uses HTTPS, not `http://kubernetes.default.svc`. Test API transport with the correct TLS trust and an appropriate identity; authentication failures and networking failures are different. Prefer a controlled application Service for connectivity checks rather than treating an unauthenticated API request as a success criterion. ### Rollback Use the same owner to reverse the mode change: ```bash # Operator installation: use its owner/GitOps source for the same change. kubectl patch installation.operator.tigera.io default --type=merge \ -p '{"spec":{"calicoNetwork":{"linuxDataplane":"Iptables"}}}' ``` ```bash # Alternative for standalone manifest installations: kubectl patch felixconfiguration.projectcalico.org default --type=merge \ -p '{"spec":{"bpfEnabled":false}}' ``` Automatic bootstrap lets the operator restore kube-proxy. If it was disabled manually, restore only the temporary change through its owner, retaining original selectors and other settings. Recheck Service rules and traffic. Disabling eBPF or changing external service mode can disrupt existing connections; do not promise that a node restart makes all application state clean. *** ## Reported Performance Records The earlier English and Korean guides contained **different, unverified records**. Their original numbers are preserved below; they are not measurements from this audit or a guarantee for Calico 3.32.2. Neither record provides raw output, a test date, exact Calico/Kubernetes/kernel versions, topology, NIC/CPU details, connection-state setup or a complete test method. ### Record A: Earlier English Guide | Reported latency | iptables | eBPF | Original rounded reduction | | --- | --- | --- | --- | | Same-node Pod | 45 μs | 25 μs | 44% | | Cross-node Pod | 120 μs | 80 μs | 33% | | ClusterIP | 150 μs | 60 μs | 60% | | NodePort | 180 μs | 70 μs | 61% | | Reported throughput | iptables | eBPF | Original rounded increase | | --- | --- | --- | --- | | TCP single stream | 15 Gbps | 23 Gbps | 53% | | TCP multi-stream | 35 Gbps | 48 Gbps | 37% | | UDP single stream | 8 Gbps | 18 Gbps | 125% | | 64-byte packets | 2M pps | 5M pps | 150% | | Reported rule count | iptables connections/s | eBPF connections/s | | --- | --- | --- | | 1,000 | 50,000 | 120,000 | | 5,000 | 35,000 | 115,000 | | 10,000 | 20,000 | 110,000 | The latency percentile is unspecified. Connections per second is not a direct CPU-utilization measurement, and these three points do not prove constant policy cost at arbitrary scale. ### Record B: Earlier Korean Guide | Reported metric | iptables | eBPF | Original rounded change | | --- | --- | --- | --- | | Throughput | 1.2M pps | 2.0M pps | +67% | | Latency | 120 μs | 75 μs | −38% | | CPU at 1,000 Services | 70% | 30% | −57% | | Connection setup | No absolute value | No absolute value | Claimed −50% | Record B is not the same experiment as Record A. Its memory/complexity claims were not measured data, and the connection-setup percentage has no underlying durations. Do not combine the records into one benchmark or use a fixed “20–40% improvement” as an expected result. ### A Reproducible Comparison Use matched client/server test images, the same nodes, traffic path, CPU/NIC allocation, MTU and load. Record dataplane, kernel and software versions, test duration, sample counts, warm-up, concurrency, conntrack state and logging settings. Test an application Service separately from direct Pod IPs. ```bash # Requires ready test Pods with netperf/netserver and an appropriate test policy. CLIENT_POD=netperf-client SERVER_POD=netperf-server SERVER_IP="$(kubectl -n calico-demo get pod "$SERVER_POD" -o jsonpath='{.status.podIP}')" test -n "$SERVER_IP" kubectl -n calico-demo exec "$CLIENT_POD" -- \ netperf -H "$SERVER_IP" -t TCP_RR -l 30 kubectl -n calico-demo exec "$CLIENT_POD" -- \ netperf -H "$SERVER_IP" -t TCP_STREAM -l 30 ``` `TCP_RR` normally reports **transactions per second**, not a latency percentile. A reciprocal can describe a mean transaction time only under the relevant test assumptions; it is not p99 network latency. netperf uses control and data connections, which must be permitted in the isolated test environment. Installing netperf on the operator's workstation does not install it in the client Pod. This procedure has not been run on a Calico cluster for this audit. *** ## eBPF Debugging The Calico node image embeds the debug tool as **`calico-node -bpf`**. A standalone `calico-bpf` source entry point also exists, but do not assume a separate binary is installed in the node image. The embedded tool uses `help`; its wrapper can consume `--help` before the BPF subcommand sees it. ```bash CALICO_NAMESPACE=calico-system CALICO_NODE=demo-worker CALICO_POD="$(kubectl -n "$CALICO_NAMESPACE" get pods -l k8s-app=calico-node \ --field-selector "spec.nodeName=$CALICO_NODE" -o jsonpath='{.items[0].metadata.name}')" test -n "$CALICO_POD" kubectl -n "$CALICO_NAMESPACE" exec "$CALICO_POD" -c calico-node -- \ calico-node -bpf help kubectl -n "$CALICO_NAMESPACE" exec "$CALICO_POD" -c calico-node -- \ calico-node -bpf routes dump kubectl -n "$CALICO_NAMESPACE" exec "$CALICO_POD" -c calico-node -- \ calico-node -bpf conntrack dump kubectl -n "$CALICO_NAMESPACE" exec "$CALICO_POD" -c calico-node -- \ calico-node -bpf nat dump kubectl -n "$CALICO_NAMESPACE" exec "$CALICO_POD" -c calico-node -- \ calico-node -bpf counters dump ``` ```bash # Choose an interface actually attached on this node. BPF_INTERFACE=eth0 kubectl -n "$CALICO_NAMESPACE" exec "$CALICO_POD" -c calico-node -- \ calico-node -bpf policy dump "$BPF_INTERFACE" all # IPv6, when enabled: put the debug-tool flag after its subcommand. kubectl -n "$CALICO_NAMESPACE" exec "$CALICO_POD" -c calico-node -- \ calico-node -bpf routes dump --ipv6 ``` ```bash kubectl -n "$CALICO_NAMESPACE" exec "$CALICO_POD" -c calico-node -- bpftool prog show kubectl -n "$CALICO_NAMESPACE" exec "$CALICO_POD" -c calico-node -- bpftool map show kubectl -n "$CALICO_NAMESPACE" exec "$CALICO_POD" -c calico-node -- bpftool net show kubectl -n "$CALICO_NAMESPACE" exec "$CALICO_POD" -c calico-node -- \ tc filter show dev "$BPF_INTERFACE" ingress kubectl -n "$CALICO_NAMESPACE" exec "$CALICO_POD" -c calico-node -- \ tc filter show dev "$BPF_INTERFACE" egress ``` `policy dump` requires both an interface and a hook (`ingress`, `egress`, `xdp` or `all`). Policy debug information must be available; `bpfPolicyDebugEnabled` defaults to true. Workload ingress is attached to the host-side veth's TC/TCX **egress** hook, while host ingress uses the host interface's ingress hook. Do not infer a workload direction from the interface-hook word alone. `nat dump` accepts no arguments or an IP/port/protocol triple. There is no `nat frontend list` command in the reviewed CLI. Inspect real program/map IDs before using bpftool's per-ID commands. Raw map lookup keys depend on the map's complete versioned layout; four IPv4 bytes alone are not a route-map key. Use bpftool's actual map `max_entries` and reported sizes; `/proc/sys/kernel/bpf_map_max_entries` is not a generic Linux map-capacity control. Runtime program counters such as `run_cnt` and `run_time_ns` require statistics to be enabled and are not end-to-end request latency. TCX attachments may require bpftool link/attachment inspection in addition to legacy `tc filter show`. ### Logs and Captures `bpfLogLevel` accepts **Off, Info or Debug**, not Warn/Warning. Those program logs go to the BPF trace pipe, while Felix's component logs go to container stdout. Use the appropriate trace tooling and node when examining program/policy logs; a successful Pod rollout is not proof of a permitted packet path. Calico can redirect directly to a workload peer, so host-side veth capture may miss traffic that bypasses that hook. Capture at the actual path and correlate policy state, routes, conntrack and Service backends. ## Kubernetes Service Replacement and Limits Calico eBPF implements Service forwarding; it does not provision an external cloud load balancer. Keep AWS/cloud controller responsibilities separate. The current implementation has IPv4/IPv6 NAT maps, local-traffic flags and affinity handling. Do not treat an old “IPv6/Local unsupported” table as the current feature matrix, nor assume that every kube-proxy option behaves identically without validation. The released WireGuard functional tests include BPF mode and IPv4/IPv6 configurations, so WireGuard is **not categorically incompatible** with eBPF. Verify the actual CNI, traffic class, kernel, MTU and encryption path; the generic encryption guide contains older limitations and installation examples that must not be blindly applied to a current OS. For host-networked workloads, inspect the CTLB/host-NAT setting and HostEndpoint policy separately. SCTP and steady-state mixed eBPF/standard/Windows clusters remain excluded by the current eBPF guide. Windows requires its supported HNS architecture. For AWS/GCP load-balancer paths, the current Calico troubleshooting guide explicitly warns that DSR does not work correctly when the external load balancer requires the return path through the original target. Use the documented supported mode and validate the complete path; same-subnet/source-check prerequisites alone do not establish cloud-LB compatibility. ## Configuration and Observability Prefer release defaults until measurements justify a change. The following fields illustrate current names and values on an already enabled eBPF installation; merge them through the configuration owner: ```yaml apiVersion: projectcalico.org/v3 kind: FelixConfiguration metadata: name: default spec: bpfLogLevel: "Off" bpfExternalServiceMode: Tunnel bpfConnectTimeLoadBalancing: TCP bpfHostNetworkedNATWithoutCTLB: Enabled ``` Do not overwrite `bpfDataIfacePattern` with an arbitrary `eth*` or narrow expression. It is a regular expression and must cover actual underlay/Service interfaces while excluding workload and special Calico devices. An interface name alone does not prove XDP offload support. `bpfKubeProxyEndpointSlicesEnabled` is not a current Felix field. The old CTLB boolean is deprecated, not removed. If tuning conntrack timeouts, the current keys include `tcpEstablished`, `tcpFinsSeen`, `tcpResetSeen`, `udpTimeout`, `genericTimeout` and `icmpTimeout`; `tcpClosing`, `udp` and `icmp` are not those keys. Measure map occupancy and memory rather than assigning one million entries to every deployment. The released endpoint manager registers these real gauges: | Metric | Meaning | | --- | --- | | `felix_bpf_dataplane_endpoints` | Managed BPF endpoints | | `felix_bpf_dirty_dataplane_endpoints` | Endpoints still dirty after a failure | | `felix_bpf_happy_dataplane_endpoints` | Successfully programmed endpoints | Enable the documented Felix metrics endpoint through its owner, inspect the actual scrape, and use its metric HELP/TYPE. The former `calico_bpf_*` list was not verified as exported metrics. Endpoint gauges are not a direct substitute for map occupancy, packet-denial counters or application latency. Choose the dataplane using current platform compatibility, required Service/policy/encryption features and measured workload behavior. Rehearse both transition and rollback; neither eBPF nor iptables is inherently the right choice for every cluster. *** ## References * [Calico 3.32 eBPF installation requirements](https://docs.tigera.io/calico/latest/operations/ebpf/install) * [Calico eBPF migration and rollback](https://docs.tigera.io/calico/latest/operations/ebpf/enabling-ebpf) * [Calico eBPF troubleshooting and CLI](https://docs.tigera.io/calico/latest/operations/ebpf/troubleshoot-ebpf) * [Felix configuration](https://docs.tigera.io/calico/latest/reference/resources/felixconfig) * [Operator installation API](https://docs.tigera.io/calico/latest/reference/installation/api) * [Kernel BTF](https://docs.kernel.org/bpf/btf.html) * [libbpf and CO-RE](https://docs.kernel.org/bpf/libbpf/libbpf_overview.html) * [Calico 3.32.2 route map layout](https://raw.githubusercontent.com/projectcalico/calico/v3.32.2/felix/bpf/routes/map.go) * [Calico 3.32.2 NAT maps](https://raw.githubusercontent.com/projectcalico/calico/v3.32.2/felix/bpf/nat/maps.go) * [Calico 3.32.2 conntrack v4 layout](https://raw.githubusercontent.com/projectcalico/calico/v3.32.2/felix/bpf/conntrack/v4/map.go) * [Calico 3.32.2 connect-time loader](https://raw.githubusercontent.com/projectcalico/calico/v3.32.2/felix/bpf/nat/connecttime.go) * [Calico 3.32.2 WireGuard functional tests](https://raw.githubusercontent.com/projectcalico/calico/v3.32.2/felix/fv/wireguard_test.go) * [bpftool program reference](https://raw.githubusercontent.com/libbpf/bpftool/main/docs/bpftool-prog.rst) * [bpftool map reference](https://raw.githubusercontent.com/libbpf/bpftool/main/docs/bpftool-map.rst) ---------------------------------------- Source: https://www.atomai.click/kubernetes-docs/en/networking/calico/07-advanced-topics ---------------------------------------- # Part 7: Advanced Calico Topics > **Supported Versions**: Calico 3.32.2 / Kubernetes 1.34–1.36 (tested range) > **Last Updated**: September 12, 2026 ## Overview This chapter covers advanced Calico topics for production environments, including IPAM deep dive, WireGuard encryption, Egress Gateway, multi-cluster federation, Windows container support, and large-scale cluster design patterns. ## IPAM Deep Dive This section describes **Calico IPAM**. Host-local and cloud-provider IPAM are different allocators; creating a Calico IPPool does not switch another CNI to Calico IPAM. ### Blocks, Affinity and Allocation Limits Calico allocates addresses from blocks associated with nodes. An IPv4 `/26` contains 64 addresses and an IPv6 `/122` also contains 64; that is not a guarantee of 64 usable Pod addresses in every platform. Windows reserves four addresses per Calico-owned block. ![The datastore hands out fixed-size /26 blocks from the IPPool 10.244.0.0/16 to each node, and each node allocates individual pod IPs out of its own affine blocks, receiving another block when one is exhausted.](https://raw.githubusercontent.com/Atom-oh/kubernetes-docs/main/en/.gitbook/assets/en-networking-calico-07-advanced-topics-0.png) [🔍 View interactive diagram](https://www.atomai.click/kubernetes-docs/archmaps/en-networking-calico-07-advanced-topics-0.html) > The diagram shows an allocation model, not a node-local cache that eliminates all datastore writes. A block's affinity is not necessarily released immediately when its last Pod disappears: allocations for tunnels/VMs and reconciliation/lifecycle state also matter. With normal automatic allocation, Calico can use an existing affine block, claim another eligible block, or borrow where permitted. `strictAffinity`, `autoAllocateBlocks`, global/per-request block limits, pool selection and platform constraints can make allocation fail even when another pool still has free addresses. ![On pod creation, Calico tries the node's own affine block first, then claims an unclaimed block from the pool, then borrows from another node's block, and only fails when no free IP exists anywhere.](https://raw.githubusercontent.com/Atom-oh/kubernetes-docs/main/en/.gitbook/assets/en-networking-calico-07-advanced-topics-1.png) [🔍 View interactive diagram](https://www.atomai.click/kubernetes-docs/archmaps/en-networking-calico-07-advanced-topics-1.html) > The simplified flow assumes eligible pools, automatic block allocation, permitted borrowing and no limiting cap. Windows does not support borrowing; do not use the figure as an unconditional guarantee that only total address exhaustion can fail. Use `IPAMConfiguration/default` to inspect global settings. The reference web page lists a default block cap of 20, but the released **3.32.2 implementation and public CRD initialize `maxBlocksPerHost` to 0** when no configuration exists. Zero means no global block cap; per-request/platform limits still apply, and existing clusters retain their configured value. A positive global cap must be paired with `strictAffinity: true` in the reviewed IPAM configuration path. ### Choose Block Size before Pool Creation The default is `/26` for IPv4 and `/122` for IPv6. Supported ranges are IPv4 `/20`–`/32` and IPv6 `/116`–`/128`. Choose by expected address demand, node count, routing aggregation and allocation overhead, not GPU bandwidth or a fixed “200 nodes means /28” rule. ```yaml # Fresh-pool example; do not apply over an existing pool or overlapping pools. apiVersion: projectcalico.org/v3 kind: IPPool metadata: name: demo-ipv4-pool spec: cidr: 10.244.0.0/16 blockSize: 26 ipipMode: Never vxlanMode: Always natOutgoing: true nodeSelector: all() ``` `blockSize` and pool CIDR cannot be changed in place. A new pool/migration must preserve the actual cluster's routing, Service/node CIDR boundaries and workload allocation plan; see [networking modes](https://www.atomai.click/kubernetes-docs/llms/en/networking/calico/03-networking-modes.md). Do not overlap this aggregate pool with the sub-pool example below. ### Host-Local IPAM Host-local uses node-local allocation state and the Kubernetes-provided per-node PodCIDR configuration. The operator selects it under **`spec.cni.ipam.type: HostLocal`**: ```yaml # Installation fragment: preserve other settings through the configuration owner. apiVersion: operator.tigera.io/v1 kind: Installation metadata: name: default spec: cni: type: Calico ipam: type: HostLocal ``` There is no `calicoNetwork.hostLocalIPAMEnabled` switch. The Kubernetes controller/networking setup must already provide valid distinct node PodCIDRs; manually patching existing nodes is not an IPAM migration procedure. Do not assume a universal immediate/delayed release or scale ranking between the two allocators. ### Multiple Pools and Explicit Requests Use disjoint pools with a documented purpose. The third example is manual-only so general workloads do not automatically consume the non-SNAT range: ```yaml # Alternative to demo-ipv4-pool; these sub-pools must not overlap another pool. apiVersion: projectcalico.org/v3 kind: IPPool metadata: name: production-pool spec: cidr: 10.244.0.0/18 blockSize: 26 vxlanMode: Always natOutgoing: true nodeSelector: node-type == 'production' --- apiVersion: projectcalico.org/v3 kind: IPPool metadata: name: development-pool spec: cidr: 10.244.64.0/18 blockSize: 28 vxlanMode: Always natOutgoing: true nodeSelector: node-type == 'development' --- apiVersion: projectcalico.org/v3 kind: IPPool metadata: name: routed-workloads-pool spec: cidr: 10.244.128.0/18 blockSize: 26 ipipMode: Never vxlanMode: Never natOutgoing: false assignmentMode: Manual allowedUses: [Workload] ``` `natOutgoing: false` needs a working external return route and may still be followed by upstream NAT. It does not create an egress gateway or stable per-namespace SNAT address. For LoadBalancer allocation, use the separate `allowedUses: [LoadBalancer]` workflow in the [BGP guide](https://www.atomai.click/kubernetes-docs/llms/en/networking/calico/04-bgp-deep-dive.md). The following Pod needs a prepared namespace, matching node labels and a reviewed workload image replacing the placeholder: ```yaml apiVersion: v1 kind: Pod metadata: name: production-app namespace: calico-demo annotations: cni.projectcalico.org/ipv4pools: '["production-pool"]' spec: nodeSelector: node-type: production containers: - name: app image: registry.example.com/team/app:approved ``` The annotation requests an IP pool; `nodeSelector` schedules the Pod. In the reviewed Calico IPAM code, an explicit pool request bypasses pool node/namespace selectors for compatibility, so those selectors are **not an authorization boundary**. Disabled or nonexistent pools still fail. Namespace annotations can supply defaults for Pods, but this does not turn pool selection into a network policy. ### IPv6 and Dual Stack The Kubernetes cluster, CNI/IPAM, node addresses and underlay must already support the chosen IP families. Adding pools or a Felix flag alone does not convert a cluster's IP-family configuration. ```yaml # A separate fresh dual-stack example. apiVersion: projectcalico.org/v3 kind: IPPool metadata: name: dual-ipv4-pool spec: cidr: 10.244.0.0/16 blockSize: 26 vxlanMode: Always natOutgoing: true --- apiVersion: projectcalico.org/v3 kind: IPPool metadata: name: dual-ipv6-pool spec: cidr: fd00:10:244::/48 blockSize: 122 ipipMode: Never vxlanMode: Always natOutgoing: false ``` IPv6 VXLAN is supported on the compatible Linux dataplane; IPv6 IP-in-IP is not. The ULA range above is not globally routable simply because it is IPv6. `natOutgoing: false` requires suitable return routing or another explicitly designed egress path. Node address autodetection belongs to operator configuration (or the corresponding installation environment), not invented Felix fields: ```yaml # Operator configuration fragment, not a replacement for the existing Installation. apiVersion: operator.tigera.io/v1 kind: Installation metadata: name: default spec: calicoNetwork: nodeAddressAutodetectionV4: kubernetes: NodeInternalIP nodeAddressAutodetectionV6: kubernetes: NodeInternalIP ``` The Felix `ipv6Support` field is a boolean, not `Enabled`. It controls Felix processing and is not a replacement for the full dual-stack prerequisites. ### Investigate Exhaustion before Releasing Addresses ```bash kubectl get ipamconfigurations.projectcalico.org default -o yaml calicoctl ipam show calicoctl ipam show --show-blocks calicoctl ipam check --show-problem-ips -o ipam-report.json ``` Review pool eligibility, reservations, affinity, per-host limits and actual workload/tunnel/VM ownership. Do not release an address solely because a sample command calls it orphaned. `calicoctl ipam release --block` is not a supported reviewed CLI option. The release tool supports `--from-report` and can intersect multiple reports; at least one must be fresh, and report-based cleanup carries allocation sequence information. Follow the versioned recovery procedure after verifying the reported allocations. Avoid `--force`, direct IPAMBlock/BlockAffinity deletion or arbitrary single-IP release as a general exhaustion fix. ## Inspect Node-Affine CIDR Blocks `BlockAffinity` is managed by Calico IPAM and exposes state, node, CIDR, deletion and affinity type. It is not the same thing as `Node.spec.podCIDR`, and it is not a complete snapshot of every host route when borrowed or migrating addresses exist. ```bash kubectl get blockaffinities.projectcalico.org \ -o custom-columns='NAME:.metadata.name,CIDR:.spec.cidr,NODE:.spec.node,STATE:.spec.state,DELETED:.spec.deleted,TYPE:.spec.type' kubectl get ippools.projectcalico.org \ -o custom-columns='NAME:.metadata.name,CIDR:.spec.cidr,BLOCK_SIZE:.spec.blockSize' # Review active host affinities; exclude deletion states and virtual affinities. kubectl get blockaffinities.projectcalico.org -o json | jq -r \ '.items[] | select(.spec.state == "confirmed" and .spec.deleted != true and ((.spec.type // "") == "" or .spec.type == "host")) | [.spec.cidr, .spec.node] | @tsv' ``` These are inventory outputs, not ready-to-execute `ip route add` commands. Routing also needs actual node next hops, current allocation state, pool export/encapsulation rules and any more-specific routes. Do not assume a placeholder node IP or all affinity records form a valid static routing plan. For **EKS Hybrid Nodes**, the [specialized CNI guide](https://docs.aws.amazon.com/eks/latest/userguide/hybrid-nodes-cni.html) documents AWS-maintained Cilium builds and moves Calico examples to the Hybrid Examples repository. AWS's [general alternate-CNI page](https://docs.aws.amazon.com/eks/latest/userguide/alternate-cni-plugins.html) still describes core Cilium/Calico support for Hybrid Nodes. These pages do not supply a consistent versioned Calico support matrix; moving examples alone does not establish that support ended. Confirm the exact distribution, capabilities and support owner for the planned deployment. This section covers Calico IPAM inventory, not a Hybrid installation recipe. ## WireGuard Encryption WireGuard protects supported traffic **between capable, configured nodes**. It is not application-to-application TLS, and same-node Pod traffic does not traverse that inter-node tunnel. Traffic involving a node without WireGuard support may remain unencrypted. Check the actual CNI, IP family and workload/host traffic path before treating encryption as a requirement that has been met. ![Traffic leaves Pod A in plaintext, is encrypted by Node 1's WireGuard interface (wireguard.cali), crosses the underlay from eth0 to eth0 as an encrypted UDP 51820 WireGuard tunnel, and is decrypted by Node 2's WireGuard interface back to plaintext before reaching Pod B.](https://raw.githubusercontent.com/Atom-oh/kubernetes-docs/main/en/.gitbook/assets/en-networking-calico-07-advanced-topics-2.png) [🔍 View interactive diagram](https://www.atomai.click/kubernetes-docs/archmaps/en-networking-calico-07-advanced-topics-2.html) > “Plaintext” in this diagram means not protected by the WireGuard tunnel on the local leg; the application may independently use TLS. The example shows the default IPv4 WireGuard port 51820. IPv6 has separate interface/port settings. ### Enable Only the Required IP Families ```yaml # Example for an already compatible dual-stack Linux deployment. apiVersion: projectcalico.org/v3 kind: FelixConfiguration metadata: name: default spec: wireguardEnabled: true wireguardEnabledV6: true ``` Use `wireguardEnabled` for IPv4 and `wireguardEnabledV6` for an enabled IPv6 path; do not enable IPv6 merely because the field exists. Preserve other Felix settings through the configuration owner and verify kernel support at both peers. There is no `WireguardCrossSubnet` operator IPPool encapsulation value. Leave MTU auto-detection in place unless the actual underlay/encapsulation path requires an override. A 1500-byte IPv4 underlay with WireGuard's 60-byte overhead suggests 1440; IPv6 overhead and platform-specific paths differ. The [MTU discussion](https://www.atomai.click/kubernetes-docs/llms/en/networking/calico/03-networking-modes.md) explains why alternative encapsulation paths must not be blindly added together. `wireguardHostEncryptionEnabled` concerns supported **inter-node host-originated/host-network** traffic, not encrypting a local host-to-Pod hop. Consult the platform's supported traffic matrix rather than applying it as a universal switch. ### Keys, Keepalives and Verification Calico manages node keys and publishes public-key information for peers. WireGuard's protocol derives fresh session keys during handshakes; that is distinct from an administrator's node-identity rotation policy. Persistent keepalives keep NAT/firewall state alive during idle periods. The familiar WireGuard example interval of 25 **seconds** is not a key-rotation interval. Neither `wireguardPersistentKeepAlive` nor `wireguardPersistentKeepalive` is a supported Felix field in the reviewed Open Source schema. ```bash # On the intended node with wireguard-tools available: WIREGUARD_INTERFACE=wireguard.cali wg show "$WIREGUARD_INTERFACE" public-key wg show "$WIREGUARD_INTERFACE" latest-handshakes wg show "$WIREGUARD_INTERFACE" endpoints wg show "$WIREGUARD_INTERFACE" transfer ``` ```bash # Kubernetes datastore: public identity information only. kubectl get nodes -o json | jq -r \ '.items[] | [.metadata.name, (.metadata.annotations["projectcalico.org/WireguardPublicKey"] // "-"), (.metadata.annotations["projectcalico.org/WireguardPublicKeyV6"] // "-")] | @tsv' ``` Choose the actual interface for the desired IP family. Public keys, handshakes and byte counters aid diagnosis but do not prove that every application flow uses encryption. Verify the intended traffic path and expected encrypted transport. `calicoctl node status` is not a WireGuard status table, and there is no need to print private keys for this check. ### Preserved Performance Records The earlier locales provided different, unverified figures. No test date, hardware, acceleration configuration, software versions or raw results were supplied. Preserve them as historical reported values, not current Calico/WireGuard performance guarantees. **Record A — earlier English guide:** | Metric | WireGuard | IPsec (AES-GCM) | | --- | --- | --- | | Throughput change from baseline | −5 to −10% | −15 to −25% | | Added latency | 0.1–0.3 ms | 0.5–1.0 ms | | CPU usage change | +10–15% | +30–50% | **Record B — earlier Korean guide:** | Metric | WireGuard | IPsec (AES-GCM) | | --- | --- | --- | | Throughput as percentage of baseline | 95–98% | 85–90% | | Latency change from baseline | +5–10% | +15–25% | | Qualitative CPU description | Medium | High | Record B used an unencrypted baseline of 100% and described unencrypted CPU usage as low. The percentage changes do not specify percentage points versus relative CPU change. These records must not be combined into one experiment. ### WireGuard and IPsec Trade-offs WireGuard uses a deliberately constrained cryptographic design; IPsec is a framework with multiple implementations, algorithms and key-management choices. CPU cost, packet overhead, hardware offload, roaming and configuration complexity depend on those choices and the measured path. Unversioned source-line counts are not a security metric, and the reviewed Open Source Felix schema has no `ipsecEnabled` field. For FIPS requirements, compare the selected product's current certification record, version and operating conditions. ## Egress Gateway Calico Enterprise's egress gateway is a **transit Pod** that performs SNAT for selected clients. It has its own product/platform requirements; the Open Source baseline at the top of this chapter is not an Enterprise compatibility matrix. The path is client egress policy → tunnel to gateway Pod → gateway SNAT → gateway egress policy → external network. NetworkPolicy Allow does not redirect packets or perform SNAT. `BGPConfiguration.serviceExternalIPs` advertises Service routes, rather than allocating workload egress identities. ### Current Commercial Configuration Shape The documented on-premises Calico-CNI path requires a supported Enterprise installation, prepared namespaces/Pod-security permissions, routed egress addresses and UDP 4790 connectivity. GKE and Windows are excluded. AWS and Azure have separate procedures; do not transplant this pool into a cloud-provider CNI setup. Through the existing default Felix configuration owner, enable `egressIPSupport` uniformly as `EnabledPerNamespace` or, where authorized, `EnabledPerNamespaceOrPerPod`. The gateway resource is **`operator.tigera.io/v1` EgressGateway**. The operator manages its image and configuration; do not invent a `calico/egress-gateway` Deployment. ```yaml # Calico Enterprise example, not an Open Source gateway installation. apiVersion: projectcalico.org/v3 kind: IPPool metadata: name: egress-demo-pool spec: cidr: 203.0.113.0/28 blockSize: 32 nodeSelector: "!all()" natOutgoing: false --- apiVersion: operator.tigera.io/v1 kind: EgressGateway metadata: name: approved-egress namespace: calico-egress spec: replicas: 2 ipPools: - cidr: 203.0.113.0/28 template: metadata: labels: egress-code: approved spec: nodeSelector: kubernetes.io/os: linux --- apiVersion: v1 kind: Namespace metadata: name: calico-demo annotations: egress.projectcalico.org/selector: egress-code == 'approved' egress.projectcalico.org/namespaceSelector: projectcalico.org/name == 'calico-egress' ``` Replace the documentation CIDR with addresses you control and configure encapsulation/routing for the actual network. `/32` blocks avoid reserving a larger block for each gateway. `!all()` prevents automatic general allocation; explicitly requesting the pool can still use it, so annotation permissions must be controlled. Two replicas require two available IPs and appropriate node/failure-domain placement; replicas alone do not guarantee availability. The namespace selector is necessary because gateway selection otherwise defaults to the client's namespace. With `natOutgoing: false`, the gateway Pod IP survives that Calico NAT stage, but upstream NAT can still change it. Enabling gateway-pool NAT can instead expose the gateway node IP. Verify the source observed by the external receiver and allow the intended address set. Gateway replacement or upgrade can break existing connections. ### Policy and Identity Boundaries Client egress policy sees the **external destination**. Allowing a client to contact the gateway Pod IP does not route or authorize its original external flow. At gateway egress, original client identity/source-port information has been translated. Destination CIDR/port policy remains useful, but domain-based policy at that hook is not supported. Advanced `EgressGatewayPolicy` routing has its own destination/gateway selection and `maxNextHops` field. It is not the former invented `maxGatewaysPerClient` field on a `projectcalico.org/v3 EgressGateway`. For Open Source, use an independently configured application proxy or underlay/cloud NAT solution where appropriate, and control allowed traffic separately. An Envoy Pod without bootstrap/listener/upstream configuration is not a functioning egress proxy. A stable source address supports an external allowlist; it does not by itself establish PCI DSS/HIPAA compliance or application authorization. ## Multi-Cluster Connectivity and Federation Separate routed reachability, endpoint identity and Service discovery. BGP exchanges routes but does not distribute Kubernetes policies or DNS records. Typha distributes state within its deployment; it is not a lead instance reporting to the invented shared Federation Controller in the earlier diagram. ### Open Source Routed Connectivity Prepare non-overlapping addresses, bidirectional routing, reachable next hops and policy in each cluster. Account for NAT: a remote Pod CIDR outside local Calico pools can be masqueraded by `natOutgoing`, changing the receiver's observed source. Plan appropriate exclusions and routing instead of assuming Pod identity survives. ```yaml # Receiving cluster only, after routing/source preservation is verified. apiVersion: projectcalico.org/v3 kind: GlobalNetworkPolicy metadata: name: default.remote-client-access spec: order: 100 namespaceSelector: kubernetes.io/metadata.name == 'calico-demo' selector: app == 'shared-service' types: [Ingress] ingress: - action: Allow protocol: TCP source: nets: [10.245.0.0/16] destination: ports: [8080] ``` This policy selects local receiving endpoints only. `GlobalNetworkPolicy` means cluster/datastore-wide scope, not automatic application to remote clusters. Verify sender egress, receiver ingress and actual source addresses. Policy distribution requires an explicitly managed workflow; it is not a BGP or Typha side effect. ### Enterprise Federation The current Enterprise guide separates: | Capability | What it does | | --- | --- | | Federated endpoint identity | Uses remote workload/host endpoint information as input to local policy calculation | | Federated Services Controller | Reads Service/endpoint information through remote Kubernetes APIs and maintains selected local federated Services | | Multi-cluster networking | Provides a supported overlay or works with separately configured routable Pod networks | Federated endpoint identity **does not replicate network policies**. Remote policies are not automatically enforced locally; each cluster's policies remain locally applied. Routable Pod IPs and source preservation are prerequisites for identity and useful remote Service endpoints. A commercial federated Service uses an annotation that selects **backing Services by labels**, not Pods: ```yaml # Commercial controller integration; backing Services already exist. apiVersion: v1 kind: Service metadata: name: catalog-federated namespace: calico-demo annotations: federation.tigera.io/serviceSelector: app == 'catalog' spec: type: ClusterIP ports: - name: http protocol: TCP port: 8080 ``` Backing Services must be in the same namespace name across the selected clusters and expose matching port **names and protocols**. The federated Service omits `spec.selector`; its `targetPort` is not the backing-port selector. Do not manually manage its endpoint records. Remote API credentials, controller installation, Kubernetes-version/EndpointSlice compatibility and network reachability are separate prerequisites. This example does not establish them or prove cross-cluster failover. Follow the product's current federation procedure and test the real paths; do not copy the guide's historical 2018 Endpoints output as a current deployment manifest. ## Windows Container Support Calico supports Windows through **HNS**, with substantial feature/platform constraints. Linux nodes are still needed for the control components and Typha. A mixed cluster is not a way to combine the Calico eBPF dataplane with Windows. ### Version and Platform Intersection For a Kubernetes 1.36 example within Calico 3.32's tested range, Windows Server 2022 is listed by both Kubernetes and Calico. Kubernetes 1.36 also lists Server 2025, while the Calico requirements page still includes older Server 1809 and Server 2022 entries. Do not assume either the old OS or every newly supported Kubernetes OS is validated by the selected Calico/provider combination. Match the host and container base-image OS/build and use compatible maintained runtime/kubelet/kube-proxy versions. Kubernetes Windows Pods use process isolation, not Hyper-V container isolation. Calico's current documented installation uses operator-managed HostProcess containers. The old 3.29 ZIP/manual-service example and old runtime/kubelet versions in legacy instructions are not a current installation recipe. | Area | Current Calico Windows constraints | | --- | --- | | Network | IPv4 VXLAN without CrossSubnet, or supported non-overlay BGP; not IPIP | | VXLAN | UDP 4789; no Windows CrossSubnet/custom VXLAN MTU support in this guide | | IPAM | No borrowing; four addresses reserved per Calico-owned block, so `/26` gives 60 Pod addresses; account for the Windows kube-proxy single-block constraint | | Routing | Windows can use supported BGP peering but cannot be a route reflector or advertise Service IPs | | Unsupported here | IPv6/dual stack, eBPF, WireGuard, host-endpoint policy, Istio application-layer policy | | Managed platforms | EKS Windows uses VPC CNI; AKS uses Azure CNI; GKE is not interchangeable with a self-managed GCE cluster | ### Operator Configuration Provision compatible Windows nodes first and verify Linux capacity for the controller/Typha HA profile. The Windows guide calls for three Linux workers for that profile. Prepare a stable direct API endpoint using `kubernetes-services-endpoint` in the operator namespace and use the actual Service CIDR, not an assumed kubeadm default. This is the configuration shape for the **self-managed Calico-CNI VXLAN alternative**. Do not replace an existing installation's pool list with the example or apply it as the EKS/Azure CNI profile: ```yaml # Self-managed Calico-CNI IPv4 VXLAN target configuration; preserve existing pools/settings. apiVersion: operator.tigera.io/v1 kind: Installation metadata: name: default spec: serviceCIDRs: - 10.96.0.0/12 cni: type: Calico calicoNetwork: linuxDataplane: Iptables windowsDataplane: HNS bgp: Disabled ipPools: - cidr: 10.244.0.0/16 blockSize: 26 encapsulation: VXLAN natOutgoing: Enabled ``` The valid field is `spec.calicoNetwork.windowsDataplane`, not root `spec.windowsDataplane` or `windowsIPAM`. VXLAN uses `VXLAN`, not `VXLANCrossSubnet`, with BGP disabled for this profile. A non-overlay BGP alternative uses different configuration; do not mix Linux IPIP pools with Windows peers. ```bash kubectl get ipamconfigurations.projectcalico.org default -o yaml # Required for the documented mixed Windows/Calico-IPAM installation: kubectl patch ipamconfigurations.projectcalico.org default --type=merge \ -p '{"spec":{"strictAffinity":true}}' ``` Strict affinity is required for the documented Calico-IPAM Windows setup. Plan block size before networking Pods: changing it later is not supported. Ensure kube-proxy is present on Windows with the appropriate version/owner. Migrating a legacy manual installation to HostProcess can remove old Calico services and replace files, so inventory and preserve its configuration first. ```bash kubectl get nodes -l kubernetes.io/os=windows -o wide kubectl get pods -n calico-system -l k8s-app=calico-node-windows -o wide kubectl logs -n calico-system -l k8s-app=calico-node-windows -c felix --tail=100 ``` This is configuration review, not a Windows provisioning or failover test. Test Pod/Service traffic and policy on both operating systems; Windows NAT changes may apply only to newly networked Pods and some HNS policy updates can reset connections. ### HNS and the Packet Path ![On a Windows node, traffic from the Windows containers converges on the Host Networking Service (HNS), which the Calico Node Windows Service programs with networking and policy, then passes through the Virtual Filtering Platform (VFP) for packet filtering before leaving via the physical NIC.](https://raw.githubusercontent.com/Atom-oh/kubernetes-docs/main/en/.gitbook/assets/en-networking-calico-07-advanced-topics-7.png) [🔍 View interactive diagram](https://www.atomai.click/kubernetes-docs/archmaps/en-networking-calico-07-advanced-topics-7.html) > HNS/HCS manage networks and endpoints; virtual-switch/VFP mechanisms enforce the data path. Packets do not pass through a Calico userspace service as a forwarding proxy. The “Windows Service” box is a logical agent: current operator installations run these components in HostProcess containers. ## Calico Product Editions Use the current edition and feature requirements rather than treating all observability/policy capabilities as Enterprise-only. | Capability | Open Source 3.32 baseline | Commercial distinction | | --- | --- | --- | | Networking and policy | Calico networking, global/namespaced policy, multiple supported dataplanes | Additional product/platform integrations | | Policy tiers/RBAC | Available, including Calico tier-aware authorization | Product management workflows and additional controls | | HTTP policy | Available through configured Istio/Dikastes integration | Check the product-specific enforcement path | | Flow visibility/UI | Goldmane/Whisker and staged-policy workflow are available | Additional analytics, reporting and management features | | DNS domain policy | `domains` is absent from the reviewed OSS CRD | Documented commercial DNS policy | | Egress/federation | Independent routing/proxy designs are possible; no invented OSS gateway/federation CR | Supported egress gateways, remote identity and federated Services | | Support | Community/project support | Terms depend on the purchased support offering | Calico Cloud is the managed SaaS product and Calico Enterprise is self-managed. The Cloud documentation also describes a Free Tier for single-cluster observability/policy management. Review current feature/retention/support terms; do not infer a universal 24/7 SLA, per-node price, identical dataplane feature set or internal SaaS dataflow from an unsourced comparison table. ## Large-Scale Cluster Design Use measured endpoint count, policy complexity, update churn, client connections, CPU/RSS and convergence targets. A node-count table alone cannot establish production capacity. ### Operator Typha Scaling The reviewed Tigera Operator **1.42.6** uses this calculation for its counted nodes: ```text N <= 2: 1 replica N <= 4: 2 replicas otherwise: max(3, floor(N / 200) + 2) 100 nodes -> 3 500 nodes -> 4 1,000 nodes -> 7 2,000 nodes -> 12 5,000 nodes -> 27 ``` This is the implementation's automatic replica target, not a benchmark proving “200 nodes per Typha.” Its node-count logic excludes explicitly unschedulable nodes and the relevant AKS virtual-node case; actual Linux placement/capacity must also accommodate the result. Typha fans datastore updates out to Felix; it does not aggregate datastore writes or act as a cross-cluster federation controller. Preserve the operator's service account, RBAC, TLS mounts, placement and lifecycle behavior. ### Supported Overrides The current `typhaDeployment` override does not expose `spec.replicas` or arbitrary container `env`. Do not replace the owned Deployment with the incomplete manual example merely to change the replica count. Use allowed override fields through the configuration owner: ```yaml # Override shape only: these illustrative requests are not a capacity recommendation. apiVersion: operator.tigera.io/v1 kind: Installation metadata: name: default spec: typhaDeployment: spec: template: spec: containers: - name: calico-typha resources: requests: cpu: 500m memory: 512Mi ``` Choose actual requests from observed usage and failure-domain capacity; the values above only demonstrate the field shape. Review limits, anti-affinity and topology constraints together, since an impossible placement rule can leave replicas Pending. ```bash kubectl get installation.operator.tigera.io default -o yaml kubectl -n calico-system get deployment calico-typha -o yaml # Requires a working resource-metrics API: kubectl -n calico-system top pods -l k8s-app=calico-typha ``` ### Route Reflectors ![Route Reflector hierarchy for 1000+ nodes: three Tier 1 Route Reflectors peer with each other in an iBGP full mesh, and each reflects routes down to its own rack RR and worker node group instead of a full node-to-node mesh.](https://raw.githubusercontent.com/Atom-oh/kubernetes-docs/main/en/.gitbook/assets/en-networking-calico-07-advanced-topics-6.png) [🔍 View interactive diagram](https://www.atomai.click/kubernetes-docs/archmaps/en-networking-calico-07-advanced-topics-6.html) > The drawing illustrates a hierarchy, not a complete resilient deployment: each rack has only one shown RR/uplink, and the labels are not a capacity guarantee. Cluster IDs and reflection relationships must match the intended hierarchy. Follow the maintained [BGP transition procedure](https://www.atomai.click/kubernetes-docs/llms/en/networking/calico/04-bgp-deep-dive.md): prepare suitable RR nodes, use field-preserving node annotations, establish explicit sessions/routes, verify real traffic and only then remove the old mesh. The figure alone does not supply the required redundancy, next hops or policy. ### Felix Tuning Has Specific Effects | Setting area | Meaning | | --- | --- | | Route/iptables refresh intervals | Re-check local dataplane state; not a generic Kubernetes API polling interval | | `iptablesBackend: NFT` | Selects the iptables-nft frontend; not the same as Calico's native `Nftables` dataplane | | Logging/flow logs | Observe the supported log pipeline and cost; do not add commercial-only file aggregation fields to OSS | | Health timeouts | Control failure/readiness detection; longer values do not make programming faster | | Marks, route-table ranges, failsafes | Affect shared host networking/control reachability; not generic memory/CPU tuning knobs | | eBPF/DSR | A separate dataplane/network-path change with platform prerequisites, not a capacity preset | `datastoreType`, `typhaAddr` and `typhaK8sServiceName` are not fields to add to the reviewed FelixConfiguration API. Old `...Secs`/`...Millis` field names in the earlier recipe were also not current API fields. Read the current resource and reference before changing its owner-managed settings. ### Datastore Choice Kubernetes datastore avoids operating a separate Calico etcd service and is required by the current eBPF dataplane. Direct etcd can be appropriate for supported non-Kubernetes or separately designed installations, but “over 5,000 nodes requires etcd” and “etcd is always faster” are not supported conclusions. A ConfigMap named `etcd-config` does not tune an etcd process unless the deployment consumes it. It also cannot tune the managed control-plane datastore of a cloud service. Direct etcd requires its own topology, TLS/authentication, backup, recovery and capacity plan. The etcd tuning guide relates heartbeat/election settings to network and disk latency. Do not transplant a quota/snapshot/timeout preset without measuring the actual cluster. Keep control-plane configuration separate from Calico dataplane refresh intervals. ## Validation before Scaling Changes 1. Establish current allocation, route, policy and client-connection baselines. 2. Change the intended setting through its owner and preserve unrelated fields. 3. Observe resource usage, reconciliation lag, readiness and real positive/negative traffic paths. 4. Test the planned component/node/failure-domain loss and a rollback. The previous CPU/memory/node-count ranges were unvalidated planning guesses, not measured capacity results. No large-cluster, Windows, gateway or datastore deployment was executed during this review. ## References - [Calico IPPool API](https://docs.tigera.io/calico/latest/reference/resources/ippool) - [IPAMConfiguration API](https://docs.tigera.io/calico/latest/reference/resources/ipamconfig) - [BlockAffinity API](https://docs.tigera.io/calico/latest/reference/resources/blockaffinity) - [Released IPAM defaults and allocation logic](https://raw.githubusercontent.com/projectcalico/calico/v3.32.2/libcalico-go/lib/ipam/ipam.go) - [Current AWS Hybrid Nodes CNI support](https://docs.aws.amazon.com/eks/latest/userguide/hybrid-nodes-cni.html) - [WireGuard protocol](https://www.wireguard.com/protocol/) - [WireGuard keepalive semantics](https://www.wireguard.com/quickstart/) - [Calico encryption](https://docs.tigera.io/calico/latest/network-policy/encrypt-cluster-pod-traffic) - [Enterprise Egress Gateway on premises](https://docs.tigera.io/calico-enterprise/latest/networking/egress/egress-gateway-on-prem) - [Enterprise Egress Gateway on AWS](https://docs.tigera.io/calico-enterprise/latest/networking/egress/egress-gateway-aws) - [Enterprise federation scope](https://docs.tigera.io/calico-enterprise/latest/multicluster/federation/overview) - [Federated Services Controller](https://docs.tigera.io/calico-enterprise/latest/multicluster/federation/services-controller) - [Calico Windows requirements](https://docs.tigera.io/calico/latest/getting-started/kubernetes/windows-calico/requirements) - [Calico Windows operator workflow](https://docs.tigera.io/calico/latest/getting-started/kubernetes/windows-calico/operator) - [Calico Windows limitations](https://docs.tigera.io/calico/latest/getting-started/kubernetes/windows-calico/limitations) - [Windows networking architecture](https://learn.microsoft.com/en-us/virtualization/windowscontainers/container-networking/architecture) - [Kubernetes 1.36 Windows documentation source](https://raw.githubusercontent.com/kubernetes/website/release-1.36/content/en/docs/concepts/windows/intro.md) - [Current Calico product overview](https://docs.tigera.io/calico-cloud/about) - [Operator 1.42.6 scaling function](https://raw.githubusercontent.com/tigera/operator/v1.42.6/pkg/common/autoscale.go) - [Operator 1.42.6 Typha autoscaler](https://raw.githubusercontent.com/tigera/operator/v1.42.6/pkg/controller/installation/typha_autoscaler.go) - [Operator API](https://docs.tigera.io/calico/latest/reference/installation/api) - [Felix API](https://docs.tigera.io/calico/latest/reference/resources/felixconfig) - [Component metrics](https://docs.tigera.io/calico/latest/operations/monitor/monitor-component-metrics) - [etcd tuning](https://etcd.io/docs/v3.6/tuning/) - [etcd configuration](https://etcd.io/docs/v3.6/op-guide/configuration/) ## Quiz [Advanced Topics Quiz](https://www.atomai.click/kubernetes-docs/en/quizzes/networking/calico/07-advanced-topics-quiz) ---------------------------------------- Source: https://www.atomai.click/kubernetes-docs/en/networking/calico/08-eks-integration ---------------------------------------- # Part 8: EKS Integration > **Reviewed baseline**: Calico 3.32.2 / Tigera Operator 1.42.6 / Kubernetes 1.34–1.36 tested by Calico. **Last Updated**: September 12, 2026 ## Overview This guide uses Calico to enforce policy on **ordinary Linux EC2 worker nodes with Amazon VPC CNI**. VPC CNI allocates Pod IPs and configures VPC networking; Calico programs policy in the node's dataplane. The installation below keeps the Iptables dataplane and kube-proxy. Calico networking and eBPF are separate deployment choices with additional prerequisites. As of this review, EKS lists 1.34–1.36 in standard support and 1.31–1.33 in extended support. Calico 3.32's published Kubernetes test range is 1.34–1.36. EKS availability, upstream Kubernetes releases, and Calico compatibility are separate checks; an upstream 1.37 release does not extend this matrix. Confirm the target Region's EKS and add-on versions before installation. ## VPC CNI + Calico Architecture ![VPC CNI manages Pod interfaces and VPC IP allocation, while Felix programs the node's policy dataplane; these are control and configuration relationships, not a packet path through the processes.](https://raw.githubusercontent.com/Atom-oh/kubernetes-docs/main/en/.gitbook/assets/en-networking-calico-08-eks-integration-0.png) [🔍 View interactive diagram](https://www.atomai.click/kubernetes-docs/archmaps/en-networking-calico-08-eks-integration-0.html) The figure summarizes component responsibilities. A packet does not traverse the pause container or Felix process as a forwarding proxy. Felix programs rules that the Linux kernel evaluates. Its `iptables / eBPF` label represents alternative dataplanes; this guide installs Iptables. Typha and kube-controllers run on customer worker capacity, not inside the AWS-managed EKS control plane. | Component | Responsibility in this configuration | | --- | --- | | `aws-node` / IPAMD and VPC CNI plugin | Manage ENIs/IP allocation and configure Pod connectivity | | Felix in `calico-node` | Program policy rules for local endpoints | | Typha | Distribute datastore updates to Felix; operator manages scaling | | kube-controllers | Reconcile Calico data with Kubernetes resources | | kube-proxy | Provide Kubernetes Service forwarding in this baseline | For cross-node traffic, the kernel evaluates the applicable policy and uses the VPC path. Same-node traffic can stay on the host. A dropped packet is discarded; a policy drop does not return the packet to the sender. Applicable source egress and destination ingress controls must both permit the connection. ## Choose a Policy Engine and Installation Method | Choice | What it installs | Lifecycle and scope | | --- | --- | --- | | Amazon VPC CNI network policy | AWS's policy implementation | Configure the compatible `vpc-cni` add-on; this does not install Calico | | Tigera Operator manifests | Operator and Calico custom resources | Pin the release, manage CRDs, then reconcile the Installation | | Tigera Operator Helm chart | The same operator, with Helm-managed configuration | Render and review values; use the existing release for upgrades | | Direct Calico manifests | Calico components without the operator | Preserve platform customization and own the upgrade procedure | EKS add-ons are not automatically updated when a new add-on version is released or the cluster minor version changes. AWS, Marketplace, and community add-ons also have different support owners. Do not assume an add-on named `calico` exists or that a Marketplace product is the same as this OSS installation; inspect the actual catalog, publisher, version, licensing and compute compatibility. **Use one network policy engine for the same endpoints.** The Calico EKS guide requires AWS VPC CNI network policy to be disabled. A migration needs a reviewed handover of policies, node state and availability; merely toggling a flag while both engines run is not a migration procedure. AWS warns that rules can remain after removing a policy agent and recommends replacing affected nodes when migrating from a third-party engine. See the [AWS policy considerations](https://docs.aws.amazon.com/eks/latest/userguide/cni-network-policy.html) and [disable procedure](https://docs.aws.amazon.com/eks/latest/userguide/network-policy-disable.html). ## Prepare the Existing VPC CNI Installation The following commands inspect an existing managed `vpc-cni` add-on. Use the actual Region and cluster name; a self-managed VPC CNI installation must instead be changed through its own manifest or Helm owner. ```bash EKS_CLUSTER=my-cluster EKS_REGION=us-east-1 EKS_VERSION=$(aws eks describe-cluster --name "$EKS_CLUSTER" \ --region "$EKS_REGION" --query cluster.version --output text) aws eks describe-addon-versions --addon-name vpc-cni \ --kubernetes-version "$EKS_VERSION" --region "$EKS_REGION" \ --query 'addons[0].addonVersions[].{version:addonVersion,compatibility:compatibilities,compute:computeTypes}' aws eks describe-addon --cluster-name "$EKS_CLUSTER" \ --addon-name vpc-cni --region "$EKS_REGION" > vpc-cni-current.json VPC_CNI_VERSION=$(jq -r '.addon.addonVersion' vpc-cni-current.json) aws eks describe-addon-configuration --addon-name vpc-cni \ --addon-version "$VPC_CNI_VERSION" --region "$EKS_REGION" \ --query configurationSchema --output text > vpc-cni-schema.json ``` Calico requires `ANNOTATE_POD_IP=true` so that VPC CNI promptly publishes `vpc.amazonaws.com/pod-ips`. The `aws-node` ServiceAccount needs permission to patch Pods. Current VPC CNI documentation says the EKS add-on updates this permission automatically; verify it rather than overwriting the existing ClusterRole: ```bash kubectl auth can-i patch pods --all-namespaces \ --as=system:serviceaccount:kube-system:aws-node ``` This authorization check requires permission to impersonate that ServiceAccount. If the permission is absent, a separate binding can add the required rule without replacing existing rules. For a nonstandard installation, use its actual ServiceAccount name. ```yaml apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: calico-vpc-pod-annotations rules: - apiGroups: [""] resources: ["pods"] verbs: ["patch"] --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: calico-vpc-pod-annotations roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: calico-vpc-pod-annotations subjects: - kind: ServiceAccount name: aws-node namespace: kube-system ``` For the reviewed **new or already policy-only configuration**, prepare the following candidate while preserving the current add-on settings: ```bash jq '(.addon.configurationValues // "") as $current | (if $current == "" then {} else ($current | fromjson) end) | .enableNetworkPolicy = "false" | .env.ANNOTATE_POD_IP = "true" | del(.env.NETWORK_POLICY_ENFORCING_MODE)' \ vpc-cni-current.json > vpc-cni-calico.json ``` This command expects JSON configuration values; if the existing values are YAML, parse and convert them without dropping fields before preparing the candidate. Stop on a parse error. Validate the candidate against the retrieved EKS build schema and review the diff. The `NETWORK_POLICY_ENFORCING_MODE` variable belongs to the AWS policy agent; leaving it configured when that agent is absent can break Pod creation. If AWS policy is currently active, complete the migration plan before using this candidate. Apply the approved configuration through the add-on owner: ```bash aws eks update-addon --cluster-name "$EKS_CLUSTER" \ --addon-name vpc-cni --region "$EKS_REGION" \ --configuration-values file://vpc-cni-calico.json \ --resolve-conflicts PRESERVE ``` Inspect the returned update status and the resulting DaemonSet. A preserved conflict can prevent the desired field from taking effect. Do not treat successful JSON parsing or an accepted update request as proof of policy enforcement. ## Install Calico with the Operator Choose **one** of the manifest or Helm paths below for a fresh installation. Do not install a second operator over an existing release. These examples assume ordinary EC2 Linux workers, reachable Kubernetes API/DNS, compatible VPC CNI and no competing policy engine. ### Operator Manifest Path Calico 3.32 separates the Calico CRDs from the operator manifest: ```bash kubectl create -f https://raw.githubusercontent.com/projectcalico/calico/v3.32.2/manifests/v1_crd_projectcalico_org.yaml kubectl create -f https://raw.githubusercontent.com/projectcalico/calico/v3.32.2/manifests/tigera-operator.yaml ``` Save the following as `calico-eks-installation.yaml`: ```yaml apiVersion: operator.tigera.io/v1 kind: Installation metadata: name: default spec: kubernetesProvider: EKS cni: type: AmazonVPC calicoNetwork: bgp: Disabled linuxDataplane: Iptables nodeUpdateStrategy: type: RollingUpdate rollingUpdate: maxUnavailable: 1 --- apiVersion: operator.tigera.io/v1 kind: APIServer metadata: name: default spec: {} --- apiVersion: operator.tigera.io/v1 kind: Goldmane metadata: name: default spec: {} --- apiVersion: operator.tigera.io/v1 kind: Whisker metadata: name: default spec: {} ``` ```bash kubectl apply -f calico-eks-installation.yaml kubectl get tigerastatus kubectl get pods -n calico-system ``` The API server, Goldmane flow aggregator and Whisker UI are available in OSS. Configure their access and resource capacity for your environment. Setting `cni.type: AmazonVPC` delegates IPAM/networking to VPC CNI; `bgp: Disabled` alone does not select the CNI. Let the operator manage Typha replicas; `typhaDeployment.spec.replicas` is not a supported Installation override. See [scaling details](https://www.atomai.click/kubernetes-docs/llms/en/networking/calico/07-advanced-topics.md). ### Helm Path Save this as `calico-eks-values.yaml`. `installation` maps to the Installation API. Top-level `nodeSelector` controls the operator Pod, not every Calico component. Unsupported values may be silently ignored by Helm, so inspect the rendered resources. ```yaml installation: enabled: true kubernetesProvider: EKS cni: type: AmazonVPC calicoNetwork: bgp: Disabled linuxDataplane: Iptables nodeUpdateStrategy: type: RollingUpdate rollingUpdate: maxUnavailable: 1 apiServer: enabled: true goldmane: enabled: true whisker: enabled: true manageCRDs: true ``` ```bash helm repo add projectcalico https://docs.tigera.io/calico/charts helm repo update projectcalico helm template calico projectcalico/tigera-operator \ --version v3.32.2 --namespace tigera-operator \ -f calico-eks-values.yaml > calico-rendered.yaml # After reviewing the rendered configuration for the prepared cluster: helm install calico projectcalico/tigera-operator \ --version v3.32.2 --namespace tigera-operator --create-namespace \ -f calico-eks-values.yaml ``` With `manageCRDs: true`, the operator manages required CRDs after starting. For an upgrade that uses new fields immediately, apply the matching CRDs first through their existing owner, following the [Calico upgrade procedure](https://docs.tigera.io/calico/latest/operations/upgrading/kubernetes-upgrade). Helm rollback alone does not guarantee that a CRD or stored-data migration is reversed. ## AWS Native Network Policy as an Alternative AWS VPC CNI standard NetworkPolicy support began in **VPC CNI 1.14**, not EKS 1.14. The current AWS guide requires VPC CNI **1.21+ for both standard and admin policies**, compatible EKS/platform versions, and Linux kernel 5.10+. Use the current compatibility guidance rather than old launch-version examples. AWS now documents `networking.k8s.aws/v1alpha1` **ClusterNetworkPolicy**, including Admin and Baseline tiers, alongside namespace-scoped `networking.k8s.io/v1` NetworkPolicy. It is a different API from Calico GlobalNetworkPolicy and Calico's configurable tiers. Do not describe native policy as permanently limited to namespace-scoped rules. | Capability | AWS native implementation | Calico OSS in this guide | | --- | --- | --- | | Kubernetes NetworkPolicy | Supported on eligible EC2 Linux nodes | Supported on managed endpoints | | Cluster policy | AWS ClusterNetworkPolicy, with its own rules and prerequisites | Calico GlobalNetworkPolicy and Tier | | Policy observability | Agent metrics/event logs; configure any CloudWatch delivery separately | Felix/Typha metrics, Goldmane and Whisker flow observability | | Application-layer policy | Do not infer it from L3/L4 network policy | Separate Dikastes/Istio integration; not enabled by this installation | | DNS/FQDN policy in Calico | Not a Calico API implementation | The documented domain-based policy feature requires a commercial Calico edition | For the native alternative, configure `enableNetworkPolicy` through the compatible VPC CNI add-on/Helm settings. Setting an invented `ENABLE_NETWORK_POLICY` environment variable on `aws-node` is insufficient. Native `standard` startup mode initially allows traffic while policy is programmed; `strict` starts with deny and requires needed paths, including DNS. These AWS settings do not configure Calico's startup behavior. Native enforcement has documented limits: EC2 Linux only, primary Pod interface only, IP-family constraints, and reliable operation with controller-owned Pods. Review port/protocol limits and Service-port requirements. Keep AWS-managed PolicyEndpoint resources controller-owned. Use the [maintained VPC CNI guide](https://www.atomai.click/kubernetes-docs/llms/en/networking/01-vpc-cni.md) for the full setup and migration details. ## Node Types and Networking Profiles | Profile | Applicability | | --- | --- | | Ordinary managed or self-managed EC2 Linux nodes + VPC CNI | Baseline policy-only installation above; management model alone does not decide Calico capability | | EC2 nodes + full Calico CNI | Separate Tigera-documented design; Pod addressing, control-plane reachability, CNI ownership and support boundaries change | | EKS Fargate | Neither Calico's node agent nor VPC CNI native network policy runs on these Pods; security groups for Pods are a separate supported control | | EKS Auto Mode | Built-in AWS networking/policy; alternate CNI and network policy plugins are unsupported | | EKS Hybrid Nodes | VPC CNI is incompatible; the specialized CNI guide lists AWS-maintained Cilium 1.17/1.18 builds; see the support qualification below | | Windows nodes | Separate Windows/VPC CNI and Calico HNS procedure; no Calico eBPF dataplane; see [Windows limitations](https://www.atomai.click/kubernetes-docs/llms/en/networking/calico/07-advanced-topics.md) | The specialized Hybrid CNI guide lists AWS-maintained Cilium builds and links Calico examples elsewhere, while the [general alternate-CNI page](https://docs.aws.amazon.com/eks/latest/userguide/alternate-cni-plugins.html) still describes core Cilium/Calico support for Hybrid Nodes. Do not infer Calico support termination from the example move or treat arbitrary upstream versions as AWS-supported. Confirm the specific distribution/capability support boundary. A Calico policy on an EC2 endpoint can restrict that endpoint's traffic involving a Fargate peer; this does not mean that policy is enforced inside Fargate. Mixed compute clusters need explicit scheduling and enforcement boundaries, rather than labeling every worker “full Calico support.” Full Calico CNI is not enabled by setting VPC CNI's `enableNetworkPolicy` to false. Tigera's fresh-cluster procedure removes the competing CNI before adding workers. Do not delete `aws-node` from an existing production cluster as a conversion shortcut. The documented overlay design also needs special consideration for API-server-to-Pod traffic, such as admission webhooks; trusted `hostNetwork` components are one documented workaround. Review Pod CIDRs, return paths, MTU, node IAM/source-destination checks where applicable, and the AWS/Tigera support boundary before adopting that profile. For Auto Mode, VPC CNI environment variables and ENIConfig settings do not configure the managed networking service. Use NodeClass. Auto Mode runs CoreDNS as a node system service; a pure Auto Mode cluster does not need the traditional CoreDNS Deployment, while mixed non-Auto nodes still require it. Initial DNS queries can be local while upstream forwarding still leaves the node. ## IAM, IRSA and Pod Identity The basic Calico policy-only installation uses Kubernetes RBAC; it does not require a broad EC2 discovery or CloudWatch IAM role on `calico-node`. VPC CNI needs its own documented AWS permissions. A separate log exporter or a commercial cloud integration may need additional permissions on **that component's** ServiceAccount. IRSA uses the cluster OIDC provider and an appropriately scoped trust policy to obtain AWS credentials for a ServiceAccount. EKS Pod Identity is another option where the component, SDK and compute type support it. Neither an IAM policy nor an invented `Installation.spec.nodeMetadata` value wires credentials into a workload. Follow the owning component's configuration and avoid competing with operator-managed ServiceAccounts. See [VPC CNI IAM configuration](https://docs.aws.amazon.com/eks/latest/userguide/cni-iam-role.html). ## Security Groups and Calico Policy ![Security groups, Calico policy and application authentication provide distinct layers of access control.](https://raw.githubusercontent.com/Atom-oh/kubernetes-docs/main/en/.gitbook/assets/en-networking-calico-08-eks-integration-3.png) [🔍 View interactive diagram](https://www.atomai.click/kubernetes-docs/archmaps/en-networking-calico-08-eks-integration-3.html) The figure is a conceptual layering diagram, not a fixed evaluation order. Calico evaluates policies by tier/order and rule semantics; NetworkPolicy does not always precede GlobalNetworkPolicy. CloudTrail records AWS API activity, not packet decisions. VPC Flow Logs and the configured policy logging tools provide different traffic evidence. Application mTLS/authorization must be deployed separately. Security groups attach to ENIs. **Security groups for Pods** can select workloads through a SecurityGroupPolicy, so “security groups only select instances” is incorrect. To combine SG-for-Pods with Calico policy, AWS requires VPC CNI 1.11+ and `POD_SECURITY_GROUP_ENFORCING_MODE=standard`; in strict mode those Pods' traffic is not subject to Calico enforcement. Check branch-ENI/instance support and recreate affected Pods after changing the mode. SG-for-Pods is not supported on Windows or Auto Mode. In standard mode with VPC CNI's usual external SNAT enabled (`AWS_VPC_K8S_CNI_EXTERNALSNAT=false`), traffic leaving the VPC uses the node's primary ENI IP and security groups. Do not assume Pod security group egress rules cover every path. See [AWS's exact conditions](https://docs.aws.amazon.com/eks/latest/userguide/security-groups-for-pods.html). ### A Namespace-Scoped Application Policy This example selects only `app=frontend` Pods in a prepared `calico-eks-demo` namespace. It allows an **in-cluster gateway Pod** in the same namespace to reach TCP 8080 and permits frontend egress to same-namespace backend Pods on TCP 8080. The example assumes ordinary CoreDNS Pods with `k8s-app=kube-dns`; NodeLocal DNS or another resolver needs different destinations. Other policies/tiers can change the result, so inspect the entire effective policy set. ```yaml apiVersion: projectcalico.org/v3 kind: NetworkPolicy metadata: name: frontend-policy namespace: calico-eks-demo spec: selector: app == 'frontend' types: [Ingress, Egress] ingress: - action: Allow protocol: TCP source: selector: app == 'gateway' destination: ports: [8080] egress: - action: Allow protocol: TCP destination: selector: app == 'backend' ports: [8080] - action: Allow protocol: UDP destination: namespaceSelector: projectcalico.org/name == 'kube-system' selector: k8s-app == 'kube-dns' ports: [53] - action: Allow protocol: TCP destination: namespaceSelector: projectcalico.org/name == 'kube-system' selector: k8s-app == 'kube-dns' ports: [53] ``` Keep the destination selector and port under the **same** `destination` mapping. Duplicate YAML keys can silently remove the selector and allow TCP 8080 to unintended endpoints. An ALB/NLB is not a Kubernetes Pod with `app=load-balancer`; account for target mode, health checks and observed source addresses separately using the [load balancer guide](https://www.atomai.click/kubernetes-docs/llms/en/networking/03-aws-lb-controller.md). A blank, cluster-wide GlobalNetworkPolicy selecting `all()` can interrupt DNS, the API, monitoring and application traffic. Build default-deny behavior within a selected test namespace first, with explicit dependency permissions; see [network policy](https://www.atomai.click/kubernetes-docs/llms/en/networking/calico/05-network-policy.md). ## Upgrade and Recovery 1. Inventory the EKS control plane, node OS/kubelet, VPC CNI, kube-proxy, Calico/operator/CRD and calicoctl versions. Export the owned configuration and policies. 2. Choose a Calico version compatible with **both sides of the planned transition**. There is no universal “always upgrade Calico first/last” rule. Follow the installation-specific procedure and CRD migration notes. 3. Review EKS upgrade insights, removed APIs and all add-on compatibility. Upgrade the EKS control plane one minor version at a time, then bring nodes and applicable add-ons to compatible versions. 4. Verify policy and Service behavior during the rolling transition, including negative cases and newly created Pods. Keep configuration ownership and rollback prerequisites explicit. ```bash aws eks describe-cluster --name "$EKS_CLUSTER" --region "$EKS_REGION" \ --query 'cluster.{version:version,platform:platformVersion,status:status}' kubectl get nodes -o wide kubectl get daemonset calico-node -n calico-system -o wide kubectl get tigerastatus helm get values calico -n tigera-operator -o yaml ``` Current EKS supports **conditional rollback to the previous minor version within seven days of an in-place upgrade**. The cluster must meet the documented eligibility and readiness requirements; the window is not a general downgrade capability. Ordinary managed/self-managed/hybrid nodes and incompatible add-ons need preparation before the control plane. Auto Mode handles its node rollback; Fargate needs its own workload treatment. Calico and EKS add-ons are not automatically reverted, and preserving etcd data does not make incompatible resources safe. Follow the [current rollback procedure](https://docs.aws.amazon.com/eks/latest/userguide/rollback-cluster.html), without using `--force` to hide unresolved compatibility issues. Outside the eligible window, plan migration to another supported cluster. Applying an older operator manifest or running `helm rollback` is not proof of safe Calico downgrade. Check the specific release's support and schema/data changes before choosing recovery steps. `calicoctl node status` reports node-local BGP state and is not a policy-only EKS acceptance test. ## Cost and Performance | Factor | What to evaluate | | --- | --- | | Worker resources | Measure Felix, Typha, controllers and flow aggregation under actual policy/endpoint churn; CPU requests are not a separate AWS tariff | | VPC IP capacity | Prefix delegation affects address allocation and density, not an automatic ENI attachment discount | | Logging/metrics | Retention, ingestion, queries and exporter delivery have separate costs; metrics are not flow logs | | Cross-AZ traffic | Evaluate actual source/destination paths and service pricing; locality also affects availability | | EKS lifecycle | Extended support can add a cluster charge; use the current support calendar | The earlier per-component dollar estimates lacked Region, instance pricing and allocation assumptions. They are not a usable cost model. Use observed resource demand and the relevant AWS prices rather than claiming fixed monthly savings from arbitrary resource limits. ### Prefix Delegation For ordinary VPC CNI nodes, configure `ENABLE_PREFIX_DELEGATION`, `WARM_PREFIX_TARGET`, `MINIMUM_IP_TARGET` and `WARM_IP_TARGET` in the add-on's **`env`** settings or its owning DaemonSet/Helm configuration. Lowercase `enable-prefix-delegation` entries in a ConfigMap do not configure IPAMD. Start with one documented allocation strategy. `WARM_IP_TARGET` and `MINIMUM_IP_TARGET` override `WARM_PREFIX_TARGET`; setting all of them does not stack their effects. IPv4 prefixes need contiguous `/28` subnet space and suitable instances. Verify subnet fragmentation, reservations, max-Pods/kubelet configuration and the rollout plan. See the [AWS prefix procedure](https://docs.aws.amazon.com/eks/latest/userguide/cni-increase-ip-addresses-procedure.html). ### Calico eBPF on EKS Calico documents EKS and compatible VPC CNI networking for its eBPF dataplane, but this changes Service handling and needs a separate migration. Follow [Part 6](https://www.atomai.click/kubernetes-docs/llms/en/networking/calico/06-ebpf-dataplane.md): kernel/platform checks, direct API-server FQDN access and bootstrap DNS, kube-proxy ownership, health ports and rollback state all matter. VPC CNI does **not** universally require kube-proxy to remain running. If kube-proxy must coexist, Calico requires both `bpfKubeProxyIptablesCleanupEnabled: false` and `bpfKubeProxyHealthzPort: 0` to avoid the documented conflicts. A generic DaemonSet selector patch may be reconciled by its owner and must not overwrite unrelated selectors. DSR is not a default EKS optimization: AWS subnet/source-address checks and external load balancer limitations require separate validation. This baseline keeps Iptables and kube-proxy. Generic sysctl presets, reducing memory to an arbitrary minimum, or shortening conntrack lifetimes do not establish better performance. Benchmark the actual workload and preserve return paths, established connections and failure recovery. ## eksctl Cluster Planning Example This is a **planning example for ordinary managed Linux nodes**, not a production-tested recipe or a conversion of Auto Mode. Confirm supported add-on builds through `describe-addon-versions` and pin the approved builds in the configuration before provisioning. Omitting an add-on version selects a compatible default; it is not an approval of every future release. Private API access requires a management path into the VPC. NAT, logs, worker capacity and address ranges require an environment-specific design. ```yaml apiVersion: eksctl.io/v1alpha5 kind: ClusterConfig metadata: name: calico-eks-demo region: us-east-1 version: "1.36" iam: withOIDC: true vpc: cidr: 10.0.0.0/16 clusterEndpoints: publicAccess: false privateAccess: true managedNodeGroups: - name: linux-workers instanceType: m5.large amiFamily: AmazonLinux2023 desiredCapacity: 3 minSize: 3 maxSize: 6 privateNetworking: true volumeType: gp3 volumeSize: 100 addons: - name: vpc-cni attachPolicyARNs: - arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy configurationValues: | enableNetworkPolicy: "false" env: ANNOTATE_POD_IP: "true" - name: coredns - name: kube-proxy cloudWatch: clusterLogging: enableTypes: [api, audit, authenticator, controllerManager, scheduler] ``` The IPv4 CNI policy belongs to the VPC CNI identity, not Calico. Revisit IAM for other IP families and identity methods. After the approved cluster exists, perform the annotation/RBAC checks and **one** Calico installation path above. ## Validate the Result ```bash kubectl get tigerastatus kubectl rollout status daemonset/calico-node -n calico-system --timeout=300s kubectl get pods -n calico-system -o wide kubectl get pods -n calico-eks-demo -o json \ | jq '.items[] | {name: .metadata.name, ip: .status.podIP, annotatedIPs: .metadata.annotations["vpc.amazonaws.com/pod-ips"]}' kubectl get networkpolicies.projectcalico.org -n calico-eks-demo ``` Use controller-managed test workloads and check allowed and denied connections on the same node, across nodes/AZs, after Pod recreation and during updates. Include DNS, API/identity endpoints needed by the application, Service traffic and load balancer health checks. A Running Pod or Ready node verifies neither the desired denial nor absence of a startup policy gap. Test IPv6 separately: Calico's current EKS guide excludes policy enforcement for IPv6 Pods with `ENABLE_V4_EGRESS=true`. This document's examples were checked against released schemas and rendered charts; no EKS cluster, IAM resources or production traffic were created to validate them. ## References - [Calico on EKS](https://docs.tigera.io/calico/latest/getting-started/kubernetes/managed-public-cloud/eks) - [Calico requirements](https://docs.tigera.io/calico/latest/getting-started/kubernetes/requirements) - [Calico Helm installation](https://docs.tigera.io/calico/latest/getting-started/kubernetes/helm) - [Calico upgrades](https://docs.tigera.io/calico/latest/operations/upgrading/kubernetes-upgrade) - [VPC CNI 1.23 configuration reference](https://github.com/aws/amazon-vpc-cni-k8s/blob/v1.23.0/README.md) - [EKS native network policy](https://docs.aws.amazon.com/eks/latest/userguide/cni-network-policy.html) - [EKS add-on updates](https://docs.aws.amazon.com/eks/latest/userguide/updating-an-add-on.html) - [EKS version lifecycle](https://docs.aws.amazon.com/eks/latest/userguide/kubernetes-versions.html) - [EKS Auto Mode networking](https://docs.aws.amazon.com/eks/latest/userguide/auto-networking.html) - [EKS Hybrid Nodes CNI](https://docs.aws.amazon.com/eks/latest/userguide/hybrid-nodes-cni.html) ## Next Steps and Quiz Continue with [operations](https://www.atomai.click/kubernetes-docs/llms/en/networking/calico/09-operations.md), review [advanced topics](https://www.atomai.click/kubernetes-docs/llms/en/networking/calico/07-advanced-topics.md) or the [glossary](https://www.atomai.click/kubernetes-docs/llms/en/networking/calico/glossary.md), and test your understanding with the [EKS Integration Quiz](https://www.atomai.click/kubernetes-docs/en/quizzes/networking/calico/08-eks-integration-quiz). ---------------------------------------- Source: https://www.atomai.click/kubernetes-docs/en/networking/calico/09-operations ---------------------------------------- # Part 9: Calico Operations Guide > **Reviewed baseline**: Calico 3.32.2 / Operator 1.42.6 / Kubernetes 1.34–1.36 tested by Calico. > **Last Updated**: September 12, 2026 ## Overview This chapter provides comprehensive operational guidance for Calico deployments, covering installation, monitoring, troubleshooting, upgrades, and best practices for production environments. Operations form a feedback loop: validate the installation, observe its behavior, diagnose changes, and retain recoverable configuration/state before upgrades. A configuration export and a tested datastore recovery serve different purposes. ## Installation Guide Use one installation owner and a profile that matches the platform. The following example is for a **fresh self-managed Linux cluster with full Calico CNI**, Iptables and VXLAN. Prepare a nonoverlapping Pod CIDR, compatible node OS/kernel, Kubernetes API connectivity and underlay UDP 4789 connectivity between eligible nodes. It is not the VPC CNI policy-only configuration; use [Part 8](https://www.atomai.click/kubernetes-docs/llms/en/networking/calico/08-eks-integration.md) for EKS. Review [networking modes](https://www.atomai.click/kubernetes-docs/llms/en/networking/calico/03-networking-modes.md) before choosing a different overlay/BGP design. ### Tigera Operator Manifests Calico 3.32 requires the separate Calico CRDs as well as the operator manifest: ```bash kubectl create -f https://raw.githubusercontent.com/projectcalico/calico/v3.32.2/manifests/v1_crd_projectcalico_org.yaml kubectl create -f https://raw.githubusercontent.com/projectcalico/calico/v3.32.2/manifests/tigera-operator.yaml kubectl wait --for=condition=Available deployment/tigera-operator \ -n tigera-operator --timeout=300s ``` Save the following as `installation.yaml`, replacing the example Pod CIDR to match the prepared cluster. Omit MTU to use the operator's detection; do not substitute a guessed MTU for measurement of the actual path. ```yaml apiVersion: operator.tigera.io/v1 kind: Installation metadata: name: default spec: variant: Calico cni: type: Calico calicoNetwork: bgp: Disabled linuxDataplane: Iptables ipPools: - cidr: 10.244.0.0/16 blockSize: 26 encapsulation: VXLAN natOutgoing: Enabled nodeSelector: all() nodeAddressAutodetectionV4: kubernetes: NodeInternalIP nodeUpdateStrategy: type: RollingUpdate rollingUpdate: maxUnavailable: 1 --- apiVersion: operator.tigera.io/v1 kind: APIServer metadata: name: default spec: {} --- apiVersion: operator.tigera.io/v1 kind: Goldmane metadata: name: default spec: {} --- apiVersion: operator.tigera.io/v1 kind: Whisker metadata: name: default spec: {} ``` ```bash kubectl apply -f installation.yaml ``` BGP is disabled in this VXLAN example; BGP diagnostics are relevant only if your chosen profile enables it. Calico API server and Goldmane/Whisker are OSS components. The current OSS flow-logs guide marks the observability feature as tech preview; assess that status before relying on it operationally. Leave component resource sizing and Typha scaling with the operator until measurements justify supported overrides. Arbitrary fixed memory limits, unsupported `typhaDeployment.spec.replicas`/`minReadySeconds` overrides, or a `KubeControllers` entry in the legacy `componentResources` list are not a production configuration. Use the versioned Installation API; see [architecture](https://www.atomai.click/kubernetes-docs/llms/en/networking/calico/02-architecture.md) and [scaling](https://www.atomai.click/kubernetes-docs/llms/en/networking/calico/07-advanced-topics.md). ### Helm Alternative The Helm chart installs the same operator. Save this as `calico-values.yaml`; use this path instead of installing a second operator with manifests. ```yaml installation: enabled: true variant: Calico cni: type: Calico calicoNetwork: bgp: Disabled linuxDataplane: Iptables ipPools: - cidr: 10.244.0.0/16 blockSize: 26 encapsulation: VXLAN natOutgoing: Enabled nodeSelector: all() nodeAddressAutodetectionV4: kubernetes: NodeInternalIP nodeUpdateStrategy: type: RollingUpdate rollingUpdate: maxUnavailable: 1 apiServer: enabled: true goldmane: enabled: true whisker: enabled: true manageCRDs: true ``` ```bash helm repo add projectcalico https://docs.tigera.io/calico/charts helm repo update projectcalico helm template calico projectcalico/tigera-operator \ --namespace tigera-operator --version v3.32.2 \ -f calico-values.yaml > calico-rendered.yaml # Apply only after reviewing the prepared cluster and rendered resources. helm install calico projectcalico/tigera-operator \ --namespace tigera-operator --create-namespace --version v3.32.2 \ -f calico-values.yaml ``` Top-level `podAnnotations` applies to the operator Pod. It does not enable Felix metrics or set up Prometheus scraping for every component. Configure metrics explicitly in the monitoring section below. `manageCRDs: true` lets the operator manage CRDs after startup; upgrade ordering for new fields is covered later. ### Direct Manifest Alternative For an installation already managed by direct manifests, use the matching release/profile and preserve its customization. Review the downloaded file before applying it: ```bash curl -fL https://raw.githubusercontent.com/projectcalico/calico/v3.32.2/manifests/calico.yaml \ -o calico.yaml ``` Edit the relevant configuration, including the Pod CIDR and enabled networking mode. A global `sed` replacement can change an example or commented value without configuring the actual IP pool. Do not mix direct-manifest resources in `kube-system` with an operator installation in `calico-system`. ### Validate Installation ```bash kubectl get tigerastatus kubectl get installation default -o yaml kubectl rollout status daemonset/calico-node -n calico-system --timeout=300s kubectl get pods -n calico-system -o wide kubectl get nodes -o wide ``` For direct manifests, inspect the actual namespace and resource names. Ready components do not prove policy enforcement or application reachability. Test required Service/DNS paths and both allowed and denied application connections with controller-managed workloads. Use the actual API server HTTPS endpoint and appropriate authentication when checking API access; an HTTP request to `kubernetes.default` is not an authenticated API health test. ## calicoctl Command Reference Install the matching **3.32.2** binary from the official release and verify its checksum as described in [the installation chapter](https://www.atomai.click/kubernetes-docs/llms/en/networking/calico/01-introduction.md). Release assets include Linux AMD64/ARM64, macOS AMD64/ARM64 and Windows AMD64; select the actual host architecture. Do not execute PowerShell download commands inside Bash or use an older client after an upgrade without investigating the compatibility warning. For Kubernetes datastore access, a typical Unix shell configuration is: ```bash export DATASTORE_TYPE=kubernetes export KUBECONFIG="$HOME/.kube/config" calicoctl version calicoctl get nodes -o wide calicoctl get networkpolicy -A calicoctl get globalnetworkpolicy calicoctl get tier calicoctl get networkset -A calicoctl get globalnetworkset calicoctl get workloadendpoint -A calicoctl get hostendpoint calicoctl get ippool -o yaml calicoctl get bgpconfiguration default -o yaml calicoctl get bgppeer -o wide calicoctl get felixconfiguration default -o yaml ``` The kubeconfig must select the intended cluster and have appropriate RBAC. You can supply a Calico API configuration file explicitly with `--config`; do not assume an arbitrary path under `~/.config` is auto-discovered. For direct etcdv3 datastore access, use its supported configuration and certificate validation. This is a different deployment profile, not permission to access an EKS-managed etcd service. ### Local Node Diagnostics `calicoctl node status` reports the **local node's BGP status**. It is not a remote, cluster-wide readiness check merely because a kubeconfig is set. Run it on the intended node with the documented access, or inspect that node's BIRD socket as shown below. `calicoctl node diags` collects a diagnostic archive on the selected node. Its supported `--log-dir` flag selects the **input log directory**; there is no `--output-dir` flag in 3.32.2. The implementation requires root and can invoke a privileged diagnostic container and signal Felix to dump state. Treat it as deliberate evidence collection, not a passive health probe. Protect the resulting archive and use the output path printed by the command. ### Calico IPAM Run these only when **Calico IPAM** allocates the addresses. With VPC CNI, host-local or another IPAM, troubleshoot the actual allocator. ```bash calicoctl ipam show calicoctl ipam show --show-blocks calicoctl ipam show --show-borrowed calicoctl ipam show --show-configuration calicoctl ipam show --ip=10.244.0.15 calicoctl ipam check --show-problem-ips -o ipam-report.json ``` `--show-blocks` reports block utilization. Use BlockAffinity records for the block-to-node association; it is not the same as the Kubernetes Node PodCIDR. `--ip` is a read-only allocation lookup. A report can identify candidates for investigation; a missing Pod alone is not proof that an allocation is safe to release. There is no `ipam release --block` or `--handle` flag in the reviewed CLI. Report-based release has allocation sequence checks and still requires the cleanup procedure in [advanced IPAM](https://www.atomai.click/kubernetes-docs/llms/en/networking/calico/07-advanced-topics.md). `ipam split NUMBER --cidr=...` is a real command, but splits an **IP pool**, not an allocation block; it requires a locked datastore and a power-of-two split count. It is a planned migration operation, not a routine fix for a stuck Pod. ### Resource Changes and Exports `get`, `create`, `apply`, `replace`, `patch` and `delete` operate on supported resource types. Prefer an explicit type/namespace, review the complete policy set and preserve unrelated fields when patching. For example, a change to logging belongs in the existing FelixConfiguration or its GitOps owner, not a replacement object containing only the new field. `calicoctl get all` is not an all-resource backup. Enumerate the required resource types and include Kubernetes policies and operator resources separately. `calicoctl get ... --export` exists, but the reviewed CLI ignores it when no resource name is supplied. It does not turn a list export into a complete portable disaster-recovery backup. See the backup section below. ## Prometheus Metrics Confirm names, types and labels from the **installed version's `/metrics` output**. Dataplane-specific series are not guaranteed to exist in every profile, and a missing series is not a zero. The names below were checked against Calico 3.32.2 source and the official metric references. ### Enable Component Metrics Felix metrics are disabled by default; its default port is **9091**. Typha metrics are also disabled by default; the Typha binary's default metrics port is **9091**, while this operator example explicitly selects **9093**. kube-controllers metrics are enabled by default on **9094**. For an existing operator installation, merge these settings through its configuration owner: ```bash kubectl patch felixconfiguration default --type=merge \ -p '{"spec":{"prometheusMetricsEnabled":true,"prometheusMetricsPort":9091}}' kubectl patch installation default --type=merge \ -p '{"spec":{"typhaMetricsPort":9093}}' kubectl get service calico-typha-metrics -n calico-system kubectl get service calico-kube-controllers-metrics -n calico-system ``` The operator creates the Typha metrics Service when `typhaMetricsPort` is configured. Do not replace it with a conflicting Service. A direct-manifest installation needs its own Typha environment configuration and Service. Restrict metrics access appropriately; host-network endpoints can require host/network security controls beyond workload NetworkPolicy. ### Metric Names and Meaning | Metric | Type / meaning | | --- | --- | | `felix_active_local_endpoints` | Gauge; local workload **and host** endpoints. Zero can be legitimate and is not a readiness test | | `felix_active_local_policies` | Gauge; policies active on this node. Summing it counts local policy instances, not unique cluster policies | | `felix_cluster_num_hosts`, `felix_cluster_num_policies` | Cluster-wide gauges observed by each Felix; do not sum identical copies across nodes | | `felix_int_dataplane_failures` | Counter; failed dataplane updates that will be retried; no `_total` suffix in the reviewed metric name | | `felix_int_dataplane_apply_time_seconds` | **Summary** of incremental dataplane update time; exports quantiles, `_sum` and `_count`, not histogram buckets | | `felix_iptables_restore_calls`, `felix_iptables_restore_errors` | Counters for iptables-restore calls/errors in the iptables dataplane | | `felix_log_errors`, `felix_logs_dropped` | Errors writing process logs / logs dropped by blocked output; not ERROR-level entry counts or denied packets | | `typha_connections_active` | Gauge; open connections, including handshakes | | `typha_connections_streaming{syncer="..."}` | Gauge; clients that completed the handshake and are streaming | | `typha_connections_accepted` | Counter; accepted connections | | `typha_connections_dropped` | Counter; connections dropped **for rebalancing**, not a general network-failure count | | `typha_cache_size{syncer="..."}` | Gauge; key/value entries in the cache | | `typha_updates_total{syncer="..."}` | Counter; updates **received from** the datastore syncer | | `ipam_allocations_in_use{ippool="...",node="..."}` | kube-controllers gauge; Calico IPAM addresses allocated to workloads/interfaces | | `ipam_ippool_size{ippool="..."}` | kube-controllers gauge; total addresses in the pool CIDR | | `ipam_allocations_gc_candidates` | Potential leaks under investigation, not permission to release addresses | BIRD's control socket is not a Prometheus exporter. Names such as `bird_protocol_up` or `calico_bgp_peer_status` require a separately selected exporter/collector with verified labels and semantics; this installation does not produce those series. Use the BGP diagnostics below or the [CalicoNodeStatus approach](https://www.atomai.click/kubernetes-docs/llms/en/networking/calico/04-bgp-deep-dive.md) and only add exporter alerts after checking actual output. ### ServiceMonitor Wiring This example assumes Prometheus Operator CRDs and the `monitoring` namespace already exist. Match the ServiceMonitor labels to your Prometheus resource's `serviceMonitorSelector`, and ensure its `serviceMonitorNamespaceSelector` includes this namespace. Match PrometheusRule labels to its `ruleSelector` as well. A valid custom resource that is not selected produces no scrape/rule. The following **separate Services** avoid changing operator-owned Services and give all three a named `http-metrics` port. If your installation already scrapes these endpoints, reuse that setup instead of adding duplicate scrapes. The ServiceMonitor's `jobLabel` produces the `calico-felix`, `calico-typha` and `calico-kube-controllers` jobs used below. ```yaml apiVersion: v1 kind: Service metadata: name: calico-audit-felix-metrics namespace: calico-system labels: audit.calico/component: calico-felix spec: clusterIP: None selector: k8s-app: calico-node ports: - name: http-metrics port: 9091 targetPort: 9091 protocol: TCP --- apiVersion: v1 kind: Service metadata: name: calico-audit-typha-metrics namespace: calico-system labels: audit.calico/component: calico-typha spec: clusterIP: None selector: k8s-app: calico-typha ports: - name: http-metrics port: 9093 targetPort: 9093 protocol: TCP --- apiVersion: v1 kind: Service metadata: name: calico-audit-kube-controllers-metrics namespace: calico-system labels: audit.calico/component: calico-kube-controllers spec: clusterIP: None selector: k8s-app: calico-kube-controllers ports: - name: http-metrics port: 9094 targetPort: 9094 protocol: TCP --- apiVersion: monitoring.coreos.com/v1 kind: ServiceMonitor metadata: name: calico-components namespace: monitoring labels: app.kubernetes.io/part-of: calico-monitoring spec: jobLabel: audit.calico/component selector: matchExpressions: - key: audit.calico/component operator: Exists namespaceSelector: matchNames: - calico-system endpoints: - port: http-metrics interval: 30s scrapeTimeout: 10s path: /metrics ``` Check endpoint discovery/RBAC, network access and the Prometheus Targets page. A ServiceMonitor `endpoints.port` selects the **Service port name**; it does not mean container port number. Confirm each discovered target, rather than scraping a load-balanced Service address and assuming every node is represented. ## Grafana Dashboard Use the configured Prometheus datasource and current time-series/stat panels. The following are panel queries, not a complete importable dashboard. Select one cluster's metrics; a central multi-cluster datasource needs the cluster label preserved in selectors and aggregations. | Panel | PromQL | | --- | --- | | Endpoints per node | `felix_active_local_endpoints{job="calico-felix"}` | | Active policies per node | `felix_active_local_policies{job="calico-felix"}` | | Observed cluster policy count | `max(felix_cluster_num_policies{job="calico-felix"})` | | Dataplane retries per second | `rate(felix_int_dataplane_failures{job="calico-felix"}[5m])` | | Typha streaming connections | `typha_connections_streaming{job="calico-typha"}` | | Local summary p99 | `felix_int_dataplane_apply_time_seconds{job="calico-felix",quantile="0.99"}` | A per-process summary quantile is not a cluster-wide p99 and cannot be combined by `histogram_quantile`. For mean incremental apply duration while updates are occurring: ```promql rate(felix_int_dataplane_apply_time_seconds_sum{job="calico-felix"}[5m]) / rate(felix_int_dataplane_apply_time_seconds_count{job="calico-felix"}[5m]) ``` With no observations, the mean is undefined (`0/0`), not proof of zero latency. Do not invent `_bucket` series for this Summary or retain unverified `felix_iptables_restore_time_seconds` queries. Use the available operation counters and dataplane timing metric. Calico IPAM address utilization can be inspected with: ```promql sum by (ippool) ( max by (ippool, node) (ipam_allocations_in_use{job="calico-kube-controllers",ippool!="no_ippool"}) ) / max by (ippool) (ipam_ippool_size{job="calico-kube-controllers",ippool!="no_ippool"}) ``` The `max` per pool/node avoids double-counting identical controller observations, then the sum totals allocations across nodes. This is **address utilization**, not blocks consumed or a guarantee of allocatable capacity for a particular node. Pool selectors, strict affinity, block caps, reserved/tunnel addresses and other constraints still matter. It does not describe VPC CNI allocation; empty/missing/zero-capacity metrics need separate investigation. ## Alert Rules These example rules assume the jobs above, a single selected cluster and kube-state-metrics for the DaemonSet metric. Enable missing-target rules only for components you expect to run. Adapt selectors if reusing existing monitoring, and tune thresholds/durations to your workload. ```yaml apiVersion: monitoring.coreos.com/v1 kind: PrometheusRule metadata: name: calico-alerts namespace: monitoring labels: app.kubernetes.io/part-of: calico-monitoring spec: groups: - name: calico.rules rules: - alert: CalicoDaemonSetUnavailable expr: kube_daemonset_status_number_unavailable{namespace="calico-system",daemonset="calico-node"} > 0 for: 5m labels: severity: critical annotations: summary: Calico DaemonSet has unavailable Pods description: Inspect the affected node, rollout and kube-state-metrics data. - alert: CalicoMetricsScrapeFailed expr: up{job=~"calico-(felix|typha|kube-controllers)"} == 0 for: 5m labels: severity: warning annotations: summary: Calico scrape failed for {{ $labels.job }} on {{ $labels.instance }} - alert: CalicoMetricsTargetsMissing expr: |- absent(up{job="calico-felix"}) or absent(up{job="calico-typha"}) or absent(up{job="calico-kube-controllers"}) for: 10m labels: severity: warning annotations: summary: No discovered metrics targets for {{ $labels.job }} - alert: CalicoDataplaneRetries expr: rate(felix_int_dataplane_failures{job="calico-felix"}[5m]) > 0 for: 5m labels: severity: warning annotations: summary: Dataplane updates are being retried on {{ $labels.instance }} - alert: CalicoDataplaneMeanSlow expr: |- (rate(felix_int_dataplane_apply_time_seconds_sum{job="calico-felix"}[5m]) / rate(felix_int_dataplane_apply_time_seconds_count{job="calico-felix"}[5m])) > 0.5 and (rate(felix_int_dataplane_apply_time_seconds_count{job="calico-felix"}[5m]) > 0) for: 10m labels: severity: warning annotations: summary: Mean dataplane update time exceeds 0.5s on {{ $labels.instance }} - alert: CalicoIPAMHighAddressUsage expr: |- (sum by (ippool) ( max by (ippool, node) (ipam_allocations_in_use{job="calico-kube-controllers",ippool!="no_ippool"}) ) / max by (ippool) (ipam_ippool_size{job="calico-kube-controllers",ippool!="no_ippool"})) > 0.8 and on (ippool) (max by (ippool) (ipam_ippool_size{job="calico-kube-controllers",ippool!="no_ippool"}) > 0) for: 10m labels: severity: warning annotations: summary: High address utilization in Calico IP pool {{ $labels.ippool }} description: Address utilization is {{ $value | humanizePercentage }}; inspect per-node eligibility and block constraints. ``` `up == 0` detects a discovered target whose scrape failed. It does not detect a target that disappeared entirely; the `absent` rules detect loss of **all** targets for an expected component. Detecting one missing node among healthy nodes needs comparison with the expected node/DaemonSet inventory. An alert missing from the UI is not proof of health if its metric or rule was never loaded. A Typha connection decrease or rebalance counter increase can be expected during scaling. Correlate persistent streaming/client lag and component availability before calling it an incident. Likewise, zero local endpoints does not mean Felix is unready. Use actual readiness/rollout status and separate synthetic allow/deny tests. ## Log Analysis and Troubleshooting ### Start with the Affected Workload and Node A Pending Pod may be unschedulable before any CNI is called. Inspect events and `spec.nodeName` first. For a CNI/IP allocation error, identify the allocator and inspect the affected node's kubelet/CNI logs; Felix's process log is not the source of every Pod IPAM error. ```bash CALICO_NAMESPACE=calico-system WORKLOAD_NAMESPACE=calico-demo WORKLOAD_POD=replace-with-actual-pod kubectl describe pod "$WORKLOAD_POD" -n "$WORKLOAD_NAMESPACE" CALICO_NODE=$(kubectl get pod "$WORKLOAD_POD" -n "$WORKLOAD_NAMESPACE" \ -o jsonpath='{.spec.nodeName}') test -n "$CALICO_NODE" || { echo "Pod is not scheduled to a node" >&2; exit 1; } kubectl get pods -n "$CALICO_NAMESPACE" -l k8s-app=calico-node \ --field-selector "spec.nodeName=$CALICO_NODE" -o wide # Select the actual agent Pod on this node, including during a rollout. CALICO_POD=replace-with-actual-calico-node-pod kubectl logs -n "$CALICO_NAMESPACE" "$CALICO_POD" -c calico-node \ --since=15m --tail=200 --timestamps ``` Use explicit time and tail limits. With selectors, `kubectl logs` can default to a short tail; a requested time window is not proof that all lines in that window were returned. For a restarted container, inspect its previous log where available. Preserve retrieval errors rather than converting them into “no errors.” Felix process logs describe programming and component activity. Turning `logSeverityScreen` to Debug does not create a per-packet policy decision log. Record the original field and its configuration owner before a temporary change, then restore the exact prior value or absence rather than assuming Info was the previous setting. File/syslog output also depends on its configured path and runtime. ### Address Allocation and Connectivity | Symptom | Check before changing state | | --- | --- | | No scheduled node | Scheduler events, capacity, affinity and taints; this is not yet an IPAM diagnosis | | CNI allocation failure | The actual allocator's logs, pool/address capacity, selector eligibility, block/affinity limits and API access | | Pod IP reachable but Service fails | Endpoints/EndpointSlices, Service ports, kube-proxy or BPF Service handling, DNS and policy | | Small packets work, larger ones fail | Underlay/overlay MTU, fragmentation/PMTUD and return path | | Intended policy does not block | Actual endpoint identity/labels, direction, namespace selectors, tier/order, prior allow rules, host-network/other-interface limitations and established connections | ```bash kubectl exec -n "$CALICO_NAMESPACE" "$CALICO_POD" -c calico-node -- ip route show kubectl exec -n "$CALICO_NAMESPACE" "$CALICO_POD" -c calico-node -- ip -d link show calicoctl get networkpolicy -n "$WORKLOAD_NAMESPACE" -o yaml calicoctl get globalnetworkpolicy -o yaml calicoctl get tier -o yaml calicoctl get workloadendpoint -n "$WORKLOAD_NAMESPACE" -o yaml kubectl get pod "$WORKLOAD_POD" -n "$WORKLOAD_NAMESPACE" --show-labels ``` Do not select the first `calico-node` Pod in the cluster and assume it is on the failing workload's node. ICMP success/failure alone does not validate TCP or HTTP policy. Use an approved diagnostic workload with known tools and the application's real protocol/port. For Calico IPAM, use the read-only commands above and the [IPAM cleanup procedure](https://www.atomai.click/kubernetes-docs/llms/en/networking/calico/07-advanced-topics.md). Pool CIDR and blockSize are immutable; adding a nonoverlapping eligible pool is a planned capacity change, not an in-place CIDR expansion. Do not release an address or restart an agent before proving the cause. For operator installations, MTU and address autodetection belong to `Installation.spec.calicoNetwork`, with tunnel-specific Felix fields where documented. `FelixConfiguration.spec.mtu` and `ipAutoDetectionMethod` are not the reviewed APIs. Follow [MTU/networking guidance](https://www.atomai.click/kubernetes-docs/llms/en/networking/calico/03-networking-modes.md), preserve other settings and validate new/existing Pods separately. ### BGP Diagnostics Run BGP checks only for a profile that uses BGP. On the selected `calico-node` Pod, use the actual BIRD socket: ```bash kubectl exec -n "$CALICO_NAMESPACE" "$CALICO_POD" -c calico-node -- \ birdcl -s /var/run/calico/bird.ctl show protocols all kubectl exec -n "$CALICO_NAMESPACE" "$CALICO_POD" -c calico-node -- \ birdcl -s /var/run/calico/bird.ctl show route calicoctl get bgpconfiguration -o yaml calicoctl get bgppeer -o yaml calicoctl get bgpfilter -o yaml ``` Use `bird6.ctl` for the corresponding IPv6 daemon when deployed. Verify local/peer ASNs, chosen source address, TCP 179 in both directions, authentication/TTL settings, route filters and the expected route advertisements. A successful TCP connection is not proof that the session established or the required prefixes were accepted. Check the packaged log configuration before assuming a log file exists; the released container's BIRD run/log scripts determine where it writes. See [BGP deep dive](https://www.atomai.click/kubernetes-docs/llms/en/networking/calico/04-bgp-deep-dive.md). ## Health Check Automation The following **operator-installation status check** runs from a management environment with Bash, jq and a compatible kubectl. It performs read-only API/log requests, checks the observed DaemonSet generation and replica availability, and fails if a request fails. It does not inspect BGP sockets, validate packet forwarding or prove every policy is correct. ```bash #!/usr/bin/env bash # calico-status-check.sh: operator component status and bounded log collection. set -euo pipefail CALICO_NAMESPACE=${CALICO_NAMESPACE:-calico-system} if ! calico_ds_json=$(kubectl get daemonset calico-node -n "$CALICO_NAMESPACE" \ --request-timeout=20s -o json); then echo "Unable to read calico-node DaemonSet status" >&2 exit 2 fi if ! jq -e ' .status.desiredNumberScheduled as $desired | ($desired > 0) and (.status.observedGeneration >= .metadata.generation) and (.status.updatedNumberScheduled == $desired) and (.status.numberReady == $desired) and (.status.numberAvailable == $desired) and ((.status.numberUnavailable // 0) == 0) ' <<<"$calico_ds_json" >/dev/null; then echo "Calico DaemonSet is not fully observed, updated and available" >&2 exit 1 fi if ! calico_status_json=$(kubectl get tigerastatus --request-timeout=20s -o json); then echo "Unable to read operator component status" >&2 exit 2 fi if ! jq -e ' (.items | length) > 0 and all(.items[]; any(.status.conditions[]?; .type == "Available" and .status == "True") and any(.status.conditions[]?; .type == "Progressing" and .status == "False") and any(.status.conditions[]?; .type == "Degraded" and .status == "False") ) ' <<<"$calico_status_json" >/dev/null; then echo "Operator components are unavailable, progressing, degraded or missing conditions" >&2 exit 1 fi if ! calico_logs=$(kubectl logs -n "$CALICO_NAMESPACE" -l k8s-app=calico-node \ -c calico-node --since=15m --tail=200 --timestamps --prefix \ --request-timeout=20s); then echo "Unable to retrieve selected Calico logs; do not report no errors" >&2 exit 2 fi printf '%s\n' "$calico_logs" echo "Component status checks passed; review these bounded logs and test application policy separately." ``` An empty/unscheduled DaemonSet, stale status or missing component conditions is not a successful check. Logs are limited to the selected window/tail and still need interpretation. A counter or ERROR word alone is not equivalent to a live outage. To schedule this in a CronJob, first package and validate those tools and the script in an approved image. Use a dedicated ServiceAccount with read access to the DaemonSet, Pods/Pod logs and TigeraStatus; do not reuse the privileged `calico-node` identity. Configure concurrency, deadlines and failure reporting. The `calico/ctl` image is not a general-purpose Bash/kubectl diagnostic environment, and a normal Job cannot inspect another node's BIRD socket without additional deliberate access. No working in-cluster CronJob is implied by this local script. ## Version Upgrade and Recovery ### Prepare the Transition ```bash calicoctl version kubectl version --output=yaml kubectl get deployment tigera-operator -n tigera-operator \ -o jsonpath='{.spec.template.spec.containers[*].image}' kubectl get tigerastatus kubectl get daemonset calico-node -n calico-system -o wide helm get values calico -n tigera-operator -o yaml ``` The Helm command applies only to a Helm-managed installation. Inventory the actual installed images, CRDs, datastore, node OS/kernel, dataplane and Kubernetes compatibility. Preserve owned manifests/values, policies and a tested recovery plan. `kubectl version --short` is not a current command option. Follow the [3.32 upgrade procedure](https://docs.tigera.io/calico/latest/operations/upgrading/kubernetes-upgrade) for the actual source version and installation method. Review the OwnerReference/UID migration notes when crossing the relevant releases. Pin the target version and update calicoctl as well. For Helm, either apply the matching Calico CRDs through their owner before the new operator, or use `manageCRDs: true` and wait until the operator has installed them before using new fields. Updating the operator is not a reason to blindly overwrite field ownership with `--force-conflicts`. After the reviewed change, monitor operator, calico-node and the other configured components, then test allow/deny paths during and after rollout. An operator reconciles its managed DaemonSet. Removing the agent from “canary” nodes with an affinity patch does not deploy a safe canary and can leave those nodes without enforcement. Test the version/configuration in a representative isolated environment and follow supported rollout controls. Do not improvise a second competing node DaemonSet. ### Recovery Limits `helm rollback`, applying an older operator, or applying a configuration export does not automatically undo CRD/data migration or restore packet-processing state. Check the source/target release's supported downgrade path and stored data before choosing recovery. An Installation resource remaining present is not proof of no data loss. For EKS control-plane recovery, use the current eligibility and seven-day rollback limits described in [Part 8](https://www.atomai.click/kubernetes-docs/llms/en/networking/calico/08-eks-integration.md). Calico/add-ons and application compatibility remain separate responsibilities. ## Backup and Disaster Recovery ### Separate Configuration Inventory from State Recovery | Material | Purpose and limitation | | --- | --- | | Git-managed manifests/Helm values and version records | Desired configuration and ownership; preserve matching CRD definitions and images | | Calico policies, tiers, sets, pools, BGP/filter and controller configuration | Configuration inventory; include namespaced, staged and global resources actually used | | Kubernetes NetworkPolicy, namespace/ServiceAccount labels and related RBAC | Policy identity/dependencies that a Calico-only export omits | | Host/node/endpoints and IPAM state | Runtime/topology-dependent data; do not replay old node addresses or allocations into another cluster blindly | | Datastore backup and application data | A consistent recovery mechanism and separately protected credentials/data; YAML lists are not an atomic datastore snapshot | There is no `kubectl export` command. `calicoctl get TYPE -o yaml` creates a resource export; the `--export` flag has the named-resource limitation described above. For self-managed Kubernetes/etcd, follow the [Kubernetes etcd backup/recovery procedure](https://kubernetes.io/docs/tasks/administer-cluster/configure-upgrade-etcd/) with matching versions and restore testing. Managed services require their supported recovery approach; this does not provide EKS etcd access. ### Protected Configuration Inventory Example This script exports a **declared subset** for a 3.32 operator installation using Calico IPAM. It requires Bash, calicoctl, kubectl and sha256sum. The target directory must not exist; partial exports retain `STATE=incomplete`. It does not collect Secrets, external IAM/network devices, all operator custom resources or complete IPAM allocation state. Extend the inventory deliberately for the installed features and secure the separate credential backup. ```bash #!/usr/bin/env bash # calico-config-inventory.sh: protected configuration inventory, not a datastore snapshot. set -euo pipefail umask 077 CALICO_EXPORT_DIR=${1:?Usage: calico-config-inventory.sh NEW_EXPORT_DIRECTORY} mkdir -m 700 -- "$CALICO_EXPORT_DIR" printf '%s\n' incomplete > "$CALICO_EXPORT_DIR/STATE" for calico_kind in node ippool ipreservation bgpconfiguration bgppeer bgpfilter \ globalnetworkpolicy stagedglobalnetworkpolicy globalnetworkset \ felixconfiguration kubecontrollersconfiguration ipamconfiguration \ tier hostendpoint profile; do calicoctl get "$calico_kind" -o yaml > "$CALICO_EXPORT_DIR/$calico_kind.yaml" done for calico_kind in networkpolicy stagednetworkpolicy stagedkubernetesnetworkpolicy \ networkset workloadendpoint; do calicoctl get "$calico_kind" -A -o yaml > "$CALICO_EXPORT_DIR/$calico_kind.yaml" done kubectl get installation default -o yaml > "$CALICO_EXPORT_DIR/installation.yaml" kubectl get networkpolicies.networking.k8s.io -A -o yaml \ > "$CALICO_EXPORT_DIR/kubernetes-networkpolicies.yaml" kubectl get namespaces -o yaml > "$CALICO_EXPORT_DIR/namespaces.yaml" kubectl get serviceaccounts -A -o yaml > "$CALICO_EXPORT_DIR/serviceaccounts.yaml" ( cd -- "$CALICO_EXPORT_DIR" sha256sum ./*.yaml > SHA256SUMS ) printf '%s\n' complete > "$CALICO_EXPORT_DIR/STATE" echo "Configuration inventory completed: $CALICO_EXPORT_DIR" ``` A `complete` marker means the declared queries and checksums completed, not that the snapshot is transactionally consistent or disaster recovery was tested. Treat exports as sensitive infrastructure data. Verify checksums, retain copies outside the failure domain and rehearse recovery with the actual datastore/versions. ### Restore Planning 1. Restore a compatible control plane/datastore and the required CRDs/operator through the chosen recovery method. A new cluster and a same-cluster recovery have different identity/IPAM requirements. 2. Review namespace/ServiceAccount identity, labels and RBAC, then restore owned declarative configuration in dependency order, including tiers and sets before dependent policies. 3. Review cluster-specific metadata, generated/controller-owned objects, old node addresses and allocations. Do not replay a raw dump as a portable desired-state manifest. 4. Verify IP allocation uniqueness, routes, encryption, Service/DNS behavior and both allowed and denied traffic before resuming normal change activity. `calicoctl datastore migrate export/import` is a real **etcd-to-Kubernetes migration** workflow with datastore locking and rollback boundaries. It is not a generic backup shortcut for an existing Kubernetes datastore. Locking affects new Pods, and the documented migration cannot be rolled back after the Kubernetes datastore is unlocked. See the [migration procedure](https://docs.tigera.io/calico/latest/operations/datastore-migration). ## Operational Best Practices ### Policy and Access Start default-deny validation in a selected test namespace with the required DNS, API, identity, monitoring and application dependencies. A blank global `all()` policy or an invented API-server/node label selector can cut off essential traffic. Pod and host endpoints have different policy paths; use [Part 5](https://www.atomai.click/kubernetes-docs/llms/en/networking/calico/05-network-policy.md) for scoped examples, tier semantics and host endpoint controls. Preserve an independently usable recovery path and test negative cases before widening scope. ### Flow Observability Current OSS operator/Helm installations can use Goldmane and Whisker. The [OSS flow logs guide](https://docs.tigera.io/calico/latest/observability/view-flow-logs) marks this feature as tech preview and describes aggregated flows rather than one record per packet/connection. The old file/DNS logger fields and invented `FlowLogsFileReporter` names are not a valid OSS configuration. ```bash kubectl get goldmane,whisker kubectl port-forward -n calico-system service/whisker 8081:8081 ``` The port-forward binds locally by default. Whisker/Goldmane contain sensitive workload/network data; configure authentication and access controls before exposing them elsewhere. For an upgrade from before these components existed, enable the relevant custom resources intentionally. Process debug logs, policy Log actions, aggregated flow logs and Prometheus metrics answer different questions. ### Performance and Resources Measure endpoint/policy churn, dataplane programming time, queueing, memory and actual application traffic. Resync/refresh intervals are not Kubernetes API polling intervals; increasing them is not a universal API-load optimization. Use the actual `iptablesPostWriteCheckInterval` duration field, not the removed `...Secs` spelling. Preserve the installation owner's supported resource overrides and operator scaling. Do not enable BPF, DSR or a guessed interface pattern as a generic tuning preset. [Part 6](https://www.atomai.click/kubernetes-docs/llms/en/networking/calico/06-ebpf-dataplane.md) covers kernel/platform requirements, Service handling, kube-proxy conflicts and rollback. Larger conntrack maps cost memory and do not remove all bottlenecks. Re-run the relevant workload and failure tests when a dataplane or resource change is justified. The checks accompanying this guide are offline schema, query and script-fixture validation. They do not establish production capacity, successful cluster upgrade or disaster recovery. ## References - [Calico requirements](https://docs.tigera.io/calico/latest/getting-started/kubernetes/requirements) - [Calico Installation API](https://docs.tigera.io/calico/latest/reference/installation/api) - [Monitor component metrics](https://docs.tigera.io/calico/latest/operations/monitor/monitor-component-metrics) - [Felix metrics](https://docs.tigera.io/calico/latest/reference/felix/prometheus) - [Typha metrics](https://docs.tigera.io/calico/latest/reference/typha/prometheus) - [kube-controllers metrics](https://docs.tigera.io/calico/latest/reference/kube-controllers/prometheus) - [Calico troubleshooting](https://docs.tigera.io/calico/latest/operations/troubleshoot/troubleshooting) - [Calico upgrade procedure](https://docs.tigera.io/calico/latest/operations/upgrading/kubernetes-upgrade) - [Prometheus Operator API](https://prometheus-operator.dev/docs/api-reference/api/) ## Next Steps and Quiz Review the [glossary](https://www.atomai.click/kubernetes-docs/llms/en/networking/calico/glossary.md), [advanced topics](https://www.atomai.click/kubernetes-docs/llms/en/networking/calico/07-advanced-topics.md), [EKS integration](https://www.atomai.click/kubernetes-docs/llms/en/networking/calico/08-eks-integration.md), and the [Operations Quiz](https://www.atomai.click/kubernetes-docs/en/quizzes/networking/calico/09-operations-quiz). ---------------------------------------- Source: https://www.atomai.click/kubernetes-docs/en/networking/calico/glossary ---------------------------------------- # Calico Glossary > **Reviewed baseline**: Calico 3.32.2; comparison terminology checked against Cilium 1.20.1. > **Last Updated**: September 12, 2026 This document provides definitions of key terms and concepts related to Calico networking and security. Understanding these terms is essential for effectively deploying and operating Calico in Kubernetes environments. ## Term Categories Terms are organized into the following categories: - **Networking Terms** - General networking concepts - **Calico Components** - Calico-specific components and services - **Policy Terms** - Network policy and security concepts - **Operations Terms** - Operational and management concepts --- ## Networking Terms ### A **AS (Autonomous System)** - A collection of IP networks and routers under the control of a single organization that presents a common routing policy to the Internet. In Calico, AS numbers are used for BGP peering configuration. **ASN (Autonomous System Number)** - An identifier used by BGP. RFC 6996 reserves private-use ranges **64512–65534** and **4200000000–4294967294**. Private ASNs are reusable within administrative domains and must not be treated as globally unique public assignments. ### B **BGP (Border Gateway Protocol)** - A routing protocol used both between autonomous systems (eBGP) and within one AS (iBGP). Calico can use BGP to distribute Pod/Service routes; it is not required by every Calico dataplane or overlay profile. **Block Affinity** - An association between an IPAM block and its owner, commonly a node. Without strict affinity, eligible allocations can borrow from another node’s block. Affinity does not guarantee that every local Pod address comes from a locally owned block; inspect the resource type/state and actual allocator. ### C **CIDR (Classless Inter-Domain Routing)** - A method for allocating IP addresses and IP routing. Example: 10.244.0.0/16 represents a range of 65,536 IP addresses. **CNI (Container Network Interface)** - A specification for configuring container network connectivity and associated plugins. Calico provides CNI implementations; behavior differs between Linux networking and supported Windows HNS configurations. **Conntrack (Connection Tracking)** - State used to track flows for stateful policy/NAT. The standard Linux dataplane uses kernel connection tracking, while Calico’s BPF dataplane also maintains BPF conntrack maps; they are not interchangeable tuning targets. ### D **DNAT (Destination NAT)** - Network address translation that modifies the destination IP address of packets. Used in Kubernetes for Service load balancing. **Direct Routing** - A networking mode where traffic between pods on different nodes is routed directly without encapsulation. Requires underlying network to support pod CIDR routing. **DSR (Direct Server Return)** - A Service load-balancing mode where the backend returns traffic without traversing the original forwarding node. Calico BPF supports it under specific network/source-address constraints; it is not universally compatible with cloud load balancers. ### E **eBPF (extended Berkeley Packet Filter)** - Linux kernel programmability used by Calico’s BPF dataplane for networking, policy and Service handling. Compatibility and performance depend on the kernel, platform and workload; it is not a guarantee of lower overhead for every deployment. **Encapsulation** - The process of wrapping network packets inside other packets. Calico supports IPIP and VXLAN encapsulation for overlay networking. ### F **FQDN (Fully Qualified Domain Name)** - A complete DNS name. The documented Calico domain-based egress policy feature requires a commercial edition; OSS NetworkSet CIDR entries do not become DNS rules. Do not confuse this with separately integrated HTTP application-layer policy. **Full Mesh** - A BGP topology in which each of N participating nodes peers with every other node: N(N−1)/2 sessions. Capacity depends on churn, routes and hardware; 100 nodes is not a universal protocol limit. ### I **IPAM (IP Address Management)** - The system responsible for allocating, tracking, and managing IP addresses. Calico includes a built-in IPAM system with block-based allocation. **IPIP (IP-in-IP)** - IP encapsulation. Calico’s IPv4 IP-in-IP mode adds a 20-byte outer IPv4 header and requires an underlay that permits IP protocol 4; it is not a UDP port. **IPset** - A Linux kernel feature for storing sets of IP addresses, networks, or ports. Calico uses ipsets to efficiently match traffic against multiple addresses. **iptables** - A userspace interface to Linux Netfilter packet-filtering/NAT rules. The iptables legacy and nft backends differ from Calico’s separate native Nftables dataplane; changing the iptables backend does not select that dataplane. ### M **MTU (Maximum Transmission Unit)** - The maximum packet size for a link/path. With a 1500-byte effective underlay, example Calico MTUs are 1480 for IPv4 IPIP, 1450/1430 for IPv4/IPv6 VXLAN and 1440/1420 for IPv4/IPv6 WireGuard. Use the actual minimum path MTU and chosen mode, including platform-specific restrictions; these are calculations, not measured performance. ### N **NAT (Network Address Translation)** - The process of modifying IP address information in packet headers. Calico uses NAT for pod egress and Service implementation. **nftables** - The Linux packet-filtering framework used by Calico’s native Nftables mode. This is a distinct Installation dataplane choice from Iptables with its NFT backend. ### O **Overlay Network** - A virtual network built on top of an existing physical network. Calico supports IPIP and VXLAN overlay modes for environments where direct routing isn't possible. ### R **Route Reflector** - A BGP speaker that reflects routes to clients. With N total nodes including r fully-meshed reflectors and each client peering with all r, the topology has r(N−r)+r(r−1)/2 sessions. N=100/r=2 gives 197 rather than 4950 full-mesh sessions; one reflector gives 99 sessions without reflector redundancy. **Routing Table** - A data structure that stores routes to network destinations. Calico programs routes for pod CIDRs into the Linux kernel routing table. ### S **SNAT (Source NAT)** - Network address translation that modifies the source IP address of packets. Used for pod egress traffic and masquerading. ### V **veth (Virtual Ethernet)** - A paired Linux virtual interface commonly created by the CNI plugin to connect a Pod network namespace to the host. It is not created by Felix for every Pod; hostNetwork Pods, Windows and other network attachment types differ. **VXLAN (Virtual Extensible LAN)** - An encapsulation protocol that extends Layer 2 networks over Layer 3 infrastructure. Provides better cloud compatibility than IPIP but with higher overhead. ### W **WireGuard** - An encrypted tunnel protocol used by Calico for supported traffic between configured, capable node peers. It does not automatically encrypt same-node, unsupported-peer or all external traffic, and it is distinct from application mTLS. **Workload Endpoint** - A namespaced Calico representation of a workload interface, with addresses, labels and profile references used in policy calculation. It is normally orchestrator/plugin-managed and is not a stored list of every effective policy decision. --- ## Calico Components ### B **BIRD (BIRD Internet Routing Daemon)** - The routing daemon in Calico profiles that use BGP. The Calico 3.32.2 release uses its patched BIRD 1.6.8 lineage; an arbitrary upstream BIRD 2 configuration is not equivalent. ### C **calicoctl** - The command-line tool for managing Calico resources. Used for viewing status, configuring policies, managing IPAM, and troubleshooting. **Calico API Server** - Calico API integration, distinct from the Kubernetes API server and available in OSS. The user-facing projectcalico.org/v3 API and backing CRDs/native-API mode depend on the configured installation; follow the maintained installation guide rather than treating all API groups as aliases. **CNI Plugin** - The binary that implements the CNI specification for Calico. Responsible for setting up pod networking (veth pairs, routes, IP assignment). **confd** - A configuration management tool that generates BIRD configuration files from the Calico datastore. Watches for changes and updates BIRD dynamically. ### D **Dikastes** - An application-layer policy decision component used with Istio/Envoy. Envoy handles the traffic and requests authorization from Dikastes; Dikastes is not itself the forwarding proxy. This integration is documented for Calico OSS, with version-specific prerequisites. ### F **Felix** - The per-node agent that programs the selected dataplane, policy and relevant routes. The CNI plugin sets up Pod interfaces/IPAM; Felix is not the per-packet userspace forwarding path. ### G **Goldmane / Whisker** - The OSS flow aggregation API and web UI respectively. Their deployment, access controls and preview status are separate from basic Felix metrics. ### K **kube-controllers** - Controllers for Kubernetes/Calico reconciliation, including node/IPAM work and datastore-dependent synchronization. The enabled controller set differs by installation/datastore; listing controller types does not mean all run in every Kubernetes-datastore deployment. ### T **Tigera Operator** - A Kubernetes operator that manages Calico installation and lifecycle. Provides declarative configuration through CRDs. **Typha** - A datastore-update fan-out and cache service that reduces per-Felix watch load. It can use multiple watches/syncer types and replicas; it is not one universal cluster-wide watch or an automatic policy-federation service. Operator scaling is described in the advanced chapter. --- ## Policy Terms ### A **Action** - A Calico rule result: Allow and Deny terminate policy evaluation for that path; Log continues; Pass delegates to the next applicable tier and eventually profiles. Log is not itself an allow decision. **applyOnForward** - A GlobalNetworkPolicy option for forwarded traffic through host endpoints. It does not create host endpoints and is required with preDNAT/doNotTrack policies. Workload policy and host-forwarding policy have different scopes. ### D **Default Deny** - A posture that denies traffic in a selected scope unless the effective policy set permits it. Introduce it with explicit dependencies and tested namespace scope, not an unqualified empty cluster-wide policy. **DoNotTrack** - The doNotTrack GlobalNetworkPolicy setting applies before connection tracking to host endpoint traffic and requires applyOnForward. Stateless return paths need explicit rules; it cannot be combined with preDNAT. ### E **Egress** - Outbound network traffic from a pod. Egress policies control what destinations a pod can communicate with. ### G **GlobalNetworkPolicy** - A cluster-scoped Calico policy that can select workloads across namespaces or host endpoints. Its selectors determine the actual scope; cluster-scoped does not mean it automatically affects every Pod. **GlobalNetworkSet** - A cluster-scoped labeled collection of IP addresses/CIDRs. Policy selectors can match it, including from namespaced Calico policies with the appropriate global namespace selection. It is not restricted to GlobalNetworkPolicy references. ### H **Host Endpoint** - A representation of a host interface used for host policy and, when configured, forwarded-traffic policy. Creating one can change traffic handling; review policies, profiles, failsafes and management access first. ### I **Ingress** - Inbound network traffic to a pod. Ingress policies control what sources can communicate with a pod. ### N **NetworkPolicy** - Two distinct APIs: Kubernetes networking.k8s.io/v1 NetworkPolicy and Calico projectcalico.org/v3 NetworkPolicy. Calico adds actions/order/tiers and selectors; HTTP rules require a separate supported application-layer integration, including OSS Dikastes, while domain-based rules have edition constraints. **NetworkSet** - A namespace-scoped set of IP addresses or CIDRs. Provides a way to group external endpoints for use in network policies. ### O **Order** - A numeric evaluation priority: lower tier order first, then policy order within that tier. Use explicit distinct priorities when ordering matters rather than relying on ties; an earlier terminal action can make later policies irrelevant. ### P **Pass** - Skip the remaining policies in the current tier and continue at the next applicable tier; after the last applicable tier, evaluate endpoint profiles. It is not a final allow. **Profile** - Shared labels inherited by endpoints. Profiles can contain legacy policy rules, but that use is deprecated in favor of NetworkPolicy/GlobalNetworkPolicy; do not treat profiles as Kubernetes RBAC. **Policy Selector** - A label-based expression that determines which endpoints a policy applies to. Uses Calico's selector syntax (e.g., `app == 'web'`). **PreDNAT** - The preDNAT GlobalNetworkPolicy setting evaluates ingress host endpoint traffic before destination NAT. It requires applyOnForward and cannot be combined with doNotTrack or an egress policy direction. ### S **Staged Policy** - A non-enforcing policy resource used to assess proposed changes. Staged policies are available in OSS with the documented resource/observability prerequisites; creating one does not by itself guarantee a complete decision log. **Selector** - An expression that matches resources based on labels. Calico uses selectors for both policy targets and source/destination matching. ### T **Tier** - An ordered group of policies. Lower numeric order has precedence. Allow/Deny is terminal, Pass delegates, and a selected tier with no matching rule uses its defaultAction, normally Deny. --- ## Operations Terms ### A **APIServer (Calico)** - The operator resource configuring Calico API integration. It is not the Kubernetes control plane and is not an Enterprise-only feature; API access still requires the correct configured API path and RBAC. ### B **Block** - An allocation unit in Calico IPAM. Default sizes are IPv4 /26 and IPv6 /122, each containing 64 addresses. Reservations and allocation constraints can reduce usable workload capacity, notably on Windows. **Block Affinity** - An association between an IPAM block and its owner, commonly a node. Without strict affinity, eligible allocations can borrow from another node’s block. Affinity does not guarantee that every local Pod address comes from a locally owned block; inspect the resource type/state and actual allocator. ### D **Dataplane** - The packet-processing implementation. Calico provides Linux Iptables, Nftables and BPF choices, and supported Windows HNS configurations; other integrations have their own requirements. **Datastore** - Storage/API access for Calico configuration and state, using the Kubernetes API datastore or supported direct etcdv3 deployments. Kubernetes datastore is recommended; features such as BPF have additional datastore constraints. ### F **FelixConfiguration** - The API resource for Felix settings, including cluster defaults and supported per-node overrides. It controls metrics/logging/dataplane options; it is not a substitute for the operator Installation API. **Flow Logs** - Aggregated connection-flow records, distinct from packet captures or process logs. Current OSS operator/Helm installations can use Goldmane and Whisker; the flow-log guide marks the feature tech preview. ### H **Health Check** - Component liveness/readiness reporting. Felix can expose health endpoints on its configured port (default 9099); component health does not prove correct application reachability or policy enforcement. ### I **IPPool** - A separate Calico resource defining an address range, encapsulation/NAT and allocation eligibility. Calico IPAM can allocate for supported Workload/Tunnel/LoadBalancer uses; the pool is not an alias of a Kubernetes Node PodCIDR. CIDR and blockSize are immutable. **Installation** - The Tigera Operator CRD that defines Calico deployment configuration. Specifies networking mode, resources, and component settings. ### M **Metrics** - Prometheus statistics with component-specific activation and ports. Felix defaults to 9091; Typha defaults to 9091 but is commonly explicitly configured to 9093; kube-controllers defaults to 9094. Read the actual settings and metric types. ### P **Pod CIDR** - A Pod address range, which may describe the cluster range or a Kubernetes Node assignment. A Calico IPPool is a separate object; its relationship to Node PodCIDRs depends on IPAM. With VPC CNI, Calico policy-only does not allocate Pod IPs. ### R **Rollout** - A controlled component update. Operator reconciliation and rolling update settings help manage availability but do not guarantee uninterrupted traffic or reversible schema/data changes. ### T **TigeraStatus** - A CRD that reports the status of Calico components. Shows deployment health and configuration state. --- ## Calico vs Kubernetes Terminology | Kubernetes Term | Calico Equivalent | Notes | |-----------------|-------------------|-------| | NetworkPolicy | Calico NetworkPolicy | Separate API groups/resources with different rule semantics | | - | GlobalNetworkPolicy | Cluster-wide policy (Calico-specific) | | - | Tier | Policy hierarchy (Calico-specific) | | Service CIDR | N/A | Calico respects K8s Service CIDR | | Pod CIDR | IPPool when using Calico IPAM | Not an alias or automatic match to Node PodCIDRs | | Node | Calico Node | Related node data; lifecycle and representation depend on the datastore | | Namespace | Namespace | Calico policies can select by namespace | | Labels | Labels | Same label syntax, used in selectors | | Pod network interface | WorkloadEndpoint | Not a Service Endpoint/EndpointSlice or a full list of applied policies | | - | HostEndpoint | Host interface policies (Calico-specific) | --- ## Calico vs Cilium Terminology These are functional comparisons, not interchangeable resources or feature guarantees. Check the mode, platform and installed version before migrating policies. | Concept | Calico | Cilium 1.20.1 comparison | | --- | --- | --- | | Node agent | Felix | Cilium Agent | | BGP | BIRD in BGP-enabled profiles | Built-in BGP Control Plane advertises reachability; it does not program the datapath or establish internal routing | | Datastore fan-out | Typha | No identical Typha component/API | | Pod IP allocation | IPPool + selected IPAM | Depends on Cilium IPAM mode; not one universal pool API | | Namespaced policy | Calico NetworkPolicy | CiliumNetworkPolicy; both differ from standard Kubernetes NetworkPolicy | | Cluster policy | GlobalNetworkPolicy | CiliumClusterwideNetworkPolicy; rule/ordering semantics differ | | Reusable external CIDRs | NetworkSet / GlobalNetworkSet | Cluster-scoped CiliumCIDRGroup, referenced by cidrGroupRef or cidrGroupSelector in CIDR rules; not CiliumIPSet | | Policy tiers | Calico Tier | No identical Calico Tier API; other policy APIs have their own ordering rules | | Workload interface | WorkloadEndpoint | CiliumEndpoint, with different lifecycle/status semantics | | Host protection | HostEndpoint policies, including configured forwarding rules | Linux host firewall with nodeSelector policies; not identical forwarding scope | | Dataplane | Linux Iptables/Nftables/BPF; Windows HNS | Linux eBPF requirements; do not describe Windows as a supported beta from this comparison | | Encryption | OSS WireGuard for supported peer paths | WireGuard or IPsec, with mode/platform-specific limits | | L7 policy | Separate Istio/Envoy/Dikastes integration documented for OSS | Envoy-based policy features with their own prerequisites | | Flow visibility | Goldmane/Whisker and component metrics | Hubble and component metrics | | Controllers | kube-controllers / Tigera Operator | Cilium Operator; responsibilities do not map one-to-one | | CLI | calicoctl | cilium and agent-side cilium-dbg have distinct roles | Calico's supported Windows feature set is narrower than Linux: for example, IPv4 HNS with the documented VXLAN/BGP limits, not WireGuard/eBPF/host endpoint parity. There is no evidence here for unconditional performance, maturity or community-size rankings. Compare a specific workload and operational requirement instead. --- ## Cross-References ### Architecture Deep Dive - **Felix**: See [Part 2: Architecture](https://www.atomai.click/kubernetes-docs/llms/en/networking/calico/02-architecture.md) - **BGP Configuration**: See [Part 4: BGP Deep Dive](https://www.atomai.click/kubernetes-docs/llms/en/networking/calico/04-bgp-deep-dive.md) - **Typha Scaling**: See [Part 7: Advanced Topics](https://www.atomai.click/kubernetes-docs/llms/en/networking/calico/07-advanced-topics.md) ### Network Policy - **Kubernetes NetworkPolicy**: See [Part 5: Network Policy](https://www.atomai.click/kubernetes-docs/llms/en/networking/calico/05-network-policy.md) - **GlobalNetworkPolicy**: See [Part 5: Network Policy](https://www.atomai.click/kubernetes-docs/llms/en/networking/calico/05-network-policy.md) - **Tier-Based Policies**: See [Part 5: Network Policy](https://www.atomai.click/kubernetes-docs/llms/en/networking/calico/05-network-policy.md) ### Operations - **Installation Methods**: See [Part 9: Operations](https://www.atomai.click/kubernetes-docs/llms/en/networking/calico/09-operations.md) - **calicoctl Commands**: See [Part 9: Operations](https://www.atomai.click/kubernetes-docs/llms/en/networking/calico/09-operations.md) - **Troubleshooting**: See [Part 9: Operations](https://www.atomai.click/kubernetes-docs/llms/en/networking/calico/09-operations.md) ### EKS Integration - **VPC CNI + Calico**: See [Part 8: EKS Integration](https://www.atomai.click/kubernetes-docs/llms/en/networking/calico/08-eks-integration.md) - **Installation Methods**: See [Part 8: EKS Integration](https://www.atomai.click/kubernetes-docs/llms/en/networking/calico/08-eks-integration.md) --- ## Primary References - [Calico resource reference](https://docs.tigera.io/calico/latest/reference/resources/) - [Calico tiers](https://docs.tigera.io/calico/latest/reference/resources/tier) - [Calico MTU](https://docs.tigera.io/calico/latest/networking/configuring/mtu) - [RFC 6996 private ASNs](https://www.rfc-editor.org/rfc/rfc6996.txt) - [Cilium CIDR group API](https://github.com/cilium/cilium/blob/v1.20.1/pkg/k8s/apis/cilium.io/v2/cidrgroups_types.go) - [Cilium BGP Control Plane](https://github.com/cilium/cilium/blob/v1.20.1/Documentation/network/bgp-control-plane/bgp-control-plane.rst) - [Cilium host firewall](https://github.com/cilium/cilium/blob/v1.20.1/Documentation/security/host-firewall.rst) ## Quiz To test what you learned in this chapter, try the [Glossary Quiz](https://www.atomai.click/kubernetes-docs/en/quizzes/networking/calico/glossary-quiz). ---------------------------------------- Source: https://www.atomai.click/kubernetes-docs/en/networking/02-vpc-lattice ---------------------------------------- # VPC Lattice Amazon VPC Lattice connects applications across VPCs and AWS accounts. This chapter explains the resource model, an EKS integration, routing, IAM authorization, monitoring, and troubleshooting. > **Last Updated**: September 11, 2026 against AWS Gateway API Controller **v2.1.3** and Gateway API **v1.5.0**. The examples describe configuration and validation steps; they have not been deployed to an AWS account as part of this review. ## Table of Contents - [Overview](#overview) - [Architecture](#architecture) - [EKS and VPC Lattice Integration](#eks-and-vpc-lattice-integration) - [Installation and Configuration](#installation-and-configuration) - [Service Management](#service-management) - [Routing and Traffic Management](#routing-and-traffic-management) - [Security and Authentication](#security-and-authentication) - [Monitoring and Logging](#monitoring-and-logging) - [Best Practices](#best-practices) - [Troubleshooting](#troubleshooting) - [References](#references) ## Overview ### What is VPC Lattice? VPC Lattice provides application networking without requiring a proxy beside every application. A **service network** groups services and resource configurations and connects them to authorized consumers. Services provide listeners, routing rules, target groups, and service DNS names. The current product also connects **resource configurations** through resource gateways, including resources such as RDS databases that use TCP. This resource access model is distinct from an HTTP service backed by a target group; service-network/service IAM auth policies do not authorize resource-configuration traffic. A **service network VPC endpoint**, powered by PrivateLink, can provide access from clients reached through peering, Transit Gateway, Direct Connect, or VPN. A direct VPC association alone does not extend access to clients behind a transit gateway or peering connection. Typical uses include cross-account application APIs, communication between EKS and other compute services, and shared data-resource access. Association, routing, security groups, authentication, and application authorization still require configuration. ### Comparison with Other Services | Service | Main responsibility | Important distinction | |---|---|---| | VPC Lattice | Private application and resource connectivity | HTTP/HTTPS/gRPC service routing and separate TLS/TCP resource capabilities; not an Internet API front door | | API Gateway | Managed API endpoints and API management | REST, HTTP, or WebSocket APIs have different features; GraphQL is not a separate API Gateway API type | | AWS App Mesh | Envoy-based service mesh | AWS will end support on **2026-09-30**; as of this review that date is upcoming. Plan migration instead of a new installation | | Transit Gateway | Network connectivity using IP routing | Connects networks; it does not replace per-service HTTP routing and authorization | | Istio / Linkerd / Cilium | Mesh capabilities implemented with their respective data planes | Features and operating costs differ. Sidecars are not mandatory in every mesh architecture | VPC Lattice eliminates the need to operate its managed data plane, but does not promise lower total cost or identical mesh functionality. Compare request/data/resource charges, controller operations, identity requirements, retries, routing features, and observability for the actual workload. See the [Istio–Lattice comparison](https://www.atomai.click/kubernetes-docs/llms/en/service-mesh/istio/comparison/02-istio-vs-lattice.md). ## Architecture ### Components and Traffic Flow | Component | Responsibility | |---|---| | Service network | Logical grouping and associations; optional IAM authorization boundary | | Service | Application endpoint with its own DNS name | | Listener and rules | Belong to a **service**; select actions and target groups | | Target group | Registered instance, IP, Lambda, or ALB targets, with target-type-specific behavior | | VPC association | Allows clients in an associated VPC to access the network, subject to security controls | | Service network VPC endpoint | PrivateLink-based access, including supported transit/on-premises paths | | Resource configuration / resource gateway | Separate resource access model, including TCP/database resources | ![Three VPCs in two AWS accounts associate with a service network, whose services use target groups for EC2, EKS, and Lambda workloads.](https://raw.githubusercontent.com/Atom-oh/kubernetes-docs/main/en/.gitbook/assets/en-networking-02-vpc-lattice-1.png) [View interactive diagram](https://www.atomai.click/kubernetes-docs/archmaps/en-networking-02-vpc-lattice-1.html) The figure shows logical associations, not a single router process. Access also depends on network reachability and the applicable policies. A request resolves the **service's** DNS name, reaches its listener, passes the applicable authorization checks, and is routed to a target according to the listener rules. A target group describes destinations; it is not another application hop. Use `get-service --query dnsEntry` or the controller's route annotation to discover the real domain. Do not construct one from the service name and service-network ID. An assigned name contains service-specific identifiers; recreating a service can change it. ### Security Model Network access, IAM authorization, and encryption are separate controls. `AWS_IAM` requires a supported signed request and appropriate policies. `NONE` disables IAM authentication at that particular layer; it does not bypass another layer's IAM policy, security groups, or application authorization. HTTPS protects client-to-Lattice traffic. Backend HTTP remains plaintext unless backend TLS is explicitly configured. ## EKS and VPC Lattice Integration The AWS Gateway API Controller reconciles Kubernetes resources into VPC Lattice resources: | Kubernetes resource | Lattice interpretation | |---|---| | GatewayClass | Selects `application-networking.k8s.aws/gateway-api-controller` | | Gateway | Refers to a service network by the **Gateway name**, without its namespace | | HTTPRoute / GRPCRoute | Creates a service with its own domain and listener/routing configuration | | Backend Service and its endpoints | Define target groups and registered pod endpoints | | TargetGroupPolicy | Configures the target group's protocol and health checks | | IAMAuthPolicy | Attaches an auth policy to a Gateway's network or a Route's service | | AccessLogPolicy | Configures a target resource's access-log destination | Two Gateways with the same name can refer to the same service network even when their Kubernetes namespaces differ. A Gateway alone does **not** create the network or one shared ingress IP. The network can be managed externally, with the controller's `defaultServiceNetwork` option for simple cases, or with the controller's ServiceNetwork CRD. Choose one owner for each cloud resource. The examples below use an externally managed network and VPC association. They leave `defaultServiceNetwork` unset and do not attach a VpcAssociationPolicy to that association. If adopting the CRD-based model, manage the network, VPC association, and authorization as separate resources; do not also manage the same resources with CloudFormation. ## Installation and Configuration ### Prerequisites The controller's v2.1 upgrade guide requires **Kubernetes 1.31 or later** and Gateway API **1.5 or later**. This example pins the version against which v2.1 was built, **1.5.0**. This minimum is not an EKS support matrix or proof of compatibility with every newer Gateway API release. Check the EKS version lifecycle and all controllers that share the Gateway API CRDs before changing them. In particular, a v2.0 controller can fail after the TLSRoute storage/API transition introduced with Gateway API 1.5. Use a supported EKS cluster, matching `kubectl`, Helm, AWS CLI v2, and an operator role permitted to configure the intended resources. The sample backend assumes Linux pods with IPs reachable by VPC Lattice. Confirm the cluster's CNI, subnet capacity, endpoint readiness, DNS, and network-policy configuration. ```bash export AWS_REGION=us-west-2 export CLUSTER_NAME=my-cluster export AWS_ACCOUNT_ID="$(aws sts get-caller-identity --query Account --output text)" export VPC_ID="$(aws eks describe-cluster --name "$CLUSTER_NAME" \ --query 'cluster.resourcesVpcConfig.vpcId' --output text)" export NETWORK_NAME=my-network export ASSOCIATION_SG_ID=sg-0123456789abcdef0 kubectl config current-context kubectl version ``` Replace the example security group ID. The VPC-association security group must allow **approved clients** on TCP 443. Backend pod/node security groups must allow the applicable Lattice managed prefix list on the actual backend/health port, TCP 8080 here. Inspect the groups attached to the real pod ENI or node ENI instead of assuming that every node uses the EKS cluster security group. Also permit the EKS control plane to reach the controller webhook on its required port. Do not open every port to the entire Internet. ### IAM Role Setup The **controller role** manages cloud resources. The **caller role** signs application requests and needs `vpc-lattice-svcs:Invoke`; they are different roles. Use EKS Pod Identity on supported nodes, or IRSA. The IRSA example below assumes the cluster's IAM OIDC provider already exists and creates a dedicated service account. For Pod Identity, use the current EKS add-on and an association for this same namespace/service account, with the appropriate trust policy; do not also rely on an IRSA annotation for the same example. The release's recommended controller policy includes broad `vpc-lattice:*` and logging/tagging permissions. Treat it as an upstream starting point, **not a least-privilege policy**. Review its resource scope and enabled features, retain the constrained service-linked-role conditions, and save the reviewed policy before creating it. Reuse an existing reviewed policy ARN instead of creating duplicate policies on later runs. ```bash curl --fail --location --output controller-policy-upstream.json \ https://raw.githubusercontent.com/aws/aws-application-networking-k8s/v2.1.3/files/controller-installation/recommended-inline-policy.json # Use the policy reviewed for this account and the enabled controller features. export REVIEWED_POLICY_FILE=controller-policy-reviewed.json test -s "$REVIEWED_POLICY_FILE" export CONTROLLER_POLICY_ARN="$(aws iam create-policy \ --policy-name VPCLatticeControllerPolicy \ --policy-document "file://$REVIEWED_POLICY_FILE" \ --query Policy.Arn --output text)" # Prerequisite: this cluster's IAM OIDC provider already exists. eksctl create iamserviceaccount \ --cluster "$CLUSTER_NAME" --region "$AWS_REGION" \ --namespace aws-application-networking-system \ --name gateway-api-controller \ --attach-policy-arn "$CONTROLLER_POLICY_ARN" \ --approve ``` An existing service account needs an intentional ownership/role migration; the example does not overwrite it automatically. ### Install the Released Controller ```bash curl --fail --location --output gateway-api-v1.5.0.yaml \ https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.5.0/standard-install.yaml # Inspect changes first if any Gateway API controller is already installed. kubectl apply --server-side -f gateway-api-v1.5.0.yaml helm pull oci://public.ecr.aws/aws-application-networking-k8s/aws-gateway-controller-chart \ --version v2.1.3 helm show crds ./aws-gateway-controller-chart-v2.1.3.tgz > lattice-crds.yaml kubectl apply --server-side -f lattice-crds.yaml helm install gateway-api-controller ./aws-gateway-controller-chart-v2.1.3.tgz \ --namespace aws-application-networking-system --create-namespace \ --set serviceAccount.create=false \ --set serviceAccount.name=gateway-api-controller \ --set-string awsRegion="$AWS_REGION" \ --set-string awsAccountId="$AWS_ACCOUNT_ID" \ --set-string clusterVpcId="$VPC_ID" \ --set-string clusterName="$CLUSTER_NAME" \ --wait --timeout 5m kubectl -n aws-application-networking-system get pods kubectl -n aws-application-networking-system logs \ -l control-plane=gateway-api-controller -c manager --tail=100 ``` For an existing Helm release, use a reviewed `helm upgrade` plan with its saved values. Helm does not automatically upgrade CRDs in `crds/`; review their changes separately. Do not remove shared Gateway API CRDs or admission policies to make an upgrade pass. For manifest-based delivery, render this **same chart** with `helm template --include-crds`, using the same values and service-account choice, then review and apply the resulting manifest. This preserves the released RBAC, EndpointSlice watches, leader-election permissions, and webhook configuration. Do not use the obsolete hand-written v1.0 deployment. The chart generates webhook certificates unless supplied explicitly or managed through its cert-manager option; keep the webhook Secret and CA bundle consistent during upgrades instead of independently regenerating one. ### Create the Service Network Choose **CLI or CloudFormation**, not both for the same network. The CLI example creates an `AWS_IAM` network. Until an applicable Allow policy is installed and propagated, requests are denied. Save the following as `api-auth-policy.json`, replacing the account and caller role. The network policy deliberately permits only this demo's `/api` endpoint and subpaths. A production network needs a reviewed policy covering its intended services and callers. ```json { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "AWS": "arn:aws:iam::123456789012:role/MyAppRole" }, "Action": "vpc-lattice-svcs:Invoke", "Resource": "*", "Condition": { "StringLike": { "vpc-lattice-svcs:RequestPath": [ "/api", "/api/*" ] } } } ] } ``` ```bash aws vpc-lattice create-service-network --name "$NETWORK_NAME" \ --auth-type AWS_IAM > service-network.json export SERVICE_NETWORK_ID="$(python3 -c \ 'import json; print(json.load(open("service-network.json"))["id"])')" export SERVICE_NETWORK_ARN="$(python3 -c \ 'import json; print(json.load(open("service-network.json"))["arn"])')" aws vpc-lattice create-service-network-vpc-association \ --service-network-identifier "$SERVICE_NETWORK_ID" \ --vpc-identifier "$VPC_ID" --security-group-ids "$ASSOCIATION_SG_ID" # Save the reviewed policy below as api-auth-policy.json, then compact it. python3 -c 'import json; print(json.dumps(json.load(open("api-auth-policy.json")),separators=(",",":")))' \ > api-auth-policy.compact.json aws vpc-lattice put-auth-policy --resource-identifier "$SERVICE_NETWORK_ID" \ --policy file://api-auth-policy.compact.json aws vpc-lattice get-service-network --service-network-identifier "$SERVICE_NETWORK_ID" aws vpc-lattice get-auth-policy --resource-identifier "$SERVICE_NETWORK_ID" aws vpc-lattice list-service-network-vpc-associations \ --service-network-identifier "$SERVICE_NETWORK_ID" ``` Verify that the association is `ACTIVE`, the network still has `authType: AWS_IAM`, and `get-auth-policy` returns the intended policy before exposing a route. Policy propagation can take a few minutes. The equivalent **network and association** CloudFormation template is: ```yaml AWSTemplateFormatVersion: '2010-09-09' Description: VPC Lattice service network and client VPC association Parameters: NetworkName: Type: String Default: my-network MinLength: 3 MaxLength: 63 AllowedPattern: '^[a-z0-9]+(-[a-z0-9]+)*$' Description: Must match the Kubernetes Gateway name VpcId: Type: AWS::EC2::VPC::Id Description: VPC containing the intended clients AssociationSecurityGroupIds: Type: List Description: Existing security groups allowing approved clients on listener ports Resources: ServiceNetwork: Type: AWS::VpcLattice::ServiceNetwork Properties: Name: {Ref: NetworkName} AuthType: AWS_IAM ClientAssociation: Type: AWS::VpcLattice::ServiceNetworkVpcAssociation Properties: ServiceNetworkIdentifier: {Ref: ServiceNetwork} VpcIdentifier: {Ref: VpcId} SecurityGroupIds: {Ref: AssociationSecurityGroupIds} Outputs: ServiceNetworkArn: Description: ARN used for authorization and sharing Value: {Fn::GetAtt: [ServiceNetwork, Arn]} ServiceNetworkId: Description: ID used with VPC Lattice API operations Value: {Fn::GetAtt: [ServiceNetwork, Id]} ``` This template does not attach an auth policy. Add an auth-policy resource in the same ownership model, or apply the reviewed network policy explicitly before testing requests. Obtain the network ID/ARN from stack outputs. Validate the template and inspect a change set before deployment; the example does not create the VPC or its security groups. ### Gateway and Application Save and apply this as `gateway.yaml`. The Gateway name must match `my-network` created above. ```yaml apiVersion: v1 kind: Namespace metadata: name: lattice-demo --- apiVersion: gateway.networking.k8s.io/v1 kind: GatewayClass metadata: name: amazon-vpc-lattice spec: controllerName: application-networking.k8s.aws/gateway-api-controller --- apiVersion: gateway.networking.k8s.io/v1 kind: Gateway metadata: name: my-network namespace: lattice-demo spec: gatewayClassName: amazon-vpc-lattice listeners: - name: https protocol: HTTPS port: 443 tls: mode: Terminate certificateRefs: - name: unused ``` `certificateRefs: [{name: unused}]` follows this controller's documented configuration: it satisfies the Gateway API TLS configuration but this controller does not read a Kubernetes TLS Secret there. With no custom hostname, Lattice supplies a certificate for its generated domain. This is **controller-specific**, not a portable certificate-management recipe. Save the following as `stable.yaml`. It configures NGINX to actually listen on 8080 and serve `/health`; declaring `containerPort` alone would not do either. ```yaml apiVersion: v1 kind: ConfigMap metadata: name: service-stable namespace: lattice-demo data: nginx.conf: | worker_processes 1; pid /tmp/nginx.pid; error_log stderr notice; events { worker_connections 1024; } http { access_log /dev/stdout; default_type application/json; client_body_temp_path /tmp/client_temp; proxy_temp_path /tmp/proxy_temp; fastcgi_temp_path /tmp/fastcgi_temp; uwsgi_temp_path /tmp/uwsgi_temp; scgi_temp_path /tmp/scgi_temp; server { listen 8080; location = /health { return 200 '{"status":"ok"}\n'; } location = /api { return 200 '{"version":"stable"}\n'; } location /api/ { return 200 '{"version":"stable"}\n'; } location / { return 404 '{"error":"not found"}\n'; } } } --- apiVersion: apps/v1 kind: Deployment metadata: name: service-stable namespace: lattice-demo spec: replicas: 2 selector: matchLabels: &id001 app: lattice-demo version: stable template: metadata: labels: *id001 spec: automountServiceAccountToken: false securityContext: runAsNonRoot: true runAsUser: 101 runAsGroup: 101 fsGroup: 101 seccompProfile: type: RuntimeDefault containers: - name: app image: nginx:1.30.4-alpine@sha256:dc5069ad14f19660b141b21236140b91656bf89bbc3e2417c70ae650cd66104c command: - nginx args: - -c - /etc/lattice/nginx.conf - -g - daemon off; ports: - name: http containerPort: 8080 readinessProbe: httpGet: path: /health port: http periodSeconds: 5 resources: requests: cpu: 50m memory: 32Mi limits: cpu: 250m memory: 64Mi securityContext: allowPrivilegeEscalation: false readOnlyRootFilesystem: true capabilities: drop: - ALL volumeMounts: - name: config mountPath: /etc/lattice readOnly: true - name: tmp mountPath: /tmp volumes: - name: config configMap: name: service-stable - name: tmp emptyDir: {} --- apiVersion: v1 kind: Service metadata: name: service-stable namespace: lattice-demo spec: selector: app: lattice-demo version: stable ports: - name: http port: 8080 targetPort: http ``` Create `canary.yaml` from the same three objects, changing every `service-stable` name to `service-canary`, both selector/template `version: stable` labels to `version: canary`, and the JSON response value `"stable"` to `"canary"`. Keep `app: lattice-demo`, the port, and the health endpoint unchanged. Apply both files in `lattice-demo`. The pinned image has Linux AMD64 and ARM64 variants. Resource requests and replica counts are demonstration settings, not measured production sizing. Save and apply the following `TargetGroupPolicy`; create an equivalent `canary-health` policy targeting `service-canary`. ```yaml apiVersion: application-networking.k8s.aws/v1alpha1 kind: TargetGroupPolicy metadata: name: stable-health namespace: lattice-demo spec: targetRef: group: '' kind: Service name: service-stable protocol: HTTP protocolVersion: HTTP1 healthCheck: enabled: true protocol: HTTP protocolVersion: HTTP1 port: 8080 path: /health intervalSeconds: 30 timeoutSeconds: 5 healthyThresholdCount: 2 unhealthyThresholdCount: 2 statusMatch: '200' ``` The CRD uses `intervalSeconds`, `timeoutSeconds`, and `statusMatch`. The AWS CLI uses different field names, shown later. Changing the protocol/version can replace a target group; deleting the policy reverts its settings, including the default HTTP/HTTP1 behavior. ## Service Management ### Create a Service Through HTTPRoute Save this as `api-route.yaml`. Also save the IAMAuthPolicy below as `api-iam.yaml`. Apply the application and health policies, then the route and auth policy. Keep the network-level `AWS_IAM` policy active while reconciliation creates and secures the route's service. ```yaml apiVersion: gateway.networking.k8s.io/v1 kind: HTTPRoute metadata: name: api namespace: lattice-demo spec: parentRefs: - name: my-network sectionName: https rules: - matches: - path: type: PathPrefix value: /api backendRefs: - name: service-stable port: 8080 weight: 90 - name: service-canary port: 8080 weight: 10 ``` ```yaml apiVersion: application-networking.k8s.aws/v1alpha1 kind: IAMAuthPolicy metadata: name: api-caller namespace: lattice-demo spec: targetRef: group: gateway.networking.k8s.io kind: HTTPRoute name: api policy: '{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"AWS":"arn:aws:iam::123456789012:role/MyAppRole"},"Action":"vpc-lattice-svcs:Invoke","Resource":"*","Condition":{"StringLike":{"vpc-lattice-svcs:RequestPath":["/api","/api/*"]}}}]}' ``` `spec.policy` is a JSON **string**. This CRD enables `AWS_IAM` on the target service; an auth-type annotation or ConfigMap containing a policy does not replace it. A policy targeting `Gateway` would instead manage the network's policy, so it must not compete with the externally managed network policy in this example. Inspect `Accepted` / `ResolvedRefs` and policy status, the relevant AWS resource state, and backend readiness. A successful `kubectl apply` is not proof that cloud reconciliation or log delivery succeeded. ```bash kubectl -n lattice-demo get gateway my-network -o yaml kubectl -n lattice-demo get httproute api -o yaml kubectl -n lattice-demo get iamauthpolicy api-caller -o yaml kubectl -n lattice-demo get endpointslices \ -l kubernetes.io/service-name=service-stable kubectl -n lattice-demo rollout status deployment/service-stable --timeout=120s kubectl -n lattice-demo rollout status deployment/service-canary --timeout=120s export SERVICE_DNS="$(kubectl -n lattice-demo get httproute api \ -o jsonpath='{.metadata.annotations.application-networking\.k8s\.aws/lattice-assigned-domain-name}')" test -n "$SERVICE_DNS" # A caller inside the associated VPC, with MyAppRole credentials, runs: lattice-client/bin/python lattice_get.py --region "$AWS_REGION" "https://${SERVICE_DNS}/api" ``` Set up the signed client in the next section before running the last command. Run it from an authorized network location with **caller-role** credentials. Your workstation needs an appropriate network path as well as AWS credentials. ### Signed HTTPS Client Save this as `lattice_get.py`. It uses the default AWS credential provider chain, freezes the credentials for each request, signs for **`vpc-lattice-svcs`**, and sets **`UNSIGNED-PAYLOAD`** as required by VPC Lattice. It validates TLS, does not follow redirects with a stale signature, and does not automatically retry requests. ```python import argparse import ssl import sys from urllib.error import HTTPError, URLError from urllib.parse import urlsplit from urllib.request import HTTPRedirectHandler, HTTPSHandler, Request, build_opener from botocore.auth import SigV4Auth from botocore.awsrequest import AWSRequest from botocore.exceptions import BotoCoreError from botocore.session import Session class NoRedirect(HTTPRedirectHandler): def redirect_request(self, req, fp, code, msg, headers, newurl): return None def signed_request(url: str, region: str, credentials) -> Request: parts = urlsplit(url) if (parts.scheme != "https" or not parts.hostname or parts.username or parts.password or parts.fragment): raise ValueError("Use an HTTPS URL without user info or a fragment") request = AWSRequest(method="GET", url=url, headers={ "x-amz-content-sha256": "UNSIGNED-PAYLOAD", }) request.context["payload_signing_enabled"] = False SigV4Auth(credentials, "vpc-lattice-svcs", region).add_auth(request) return Request(url, method="GET", headers=dict(request.headers.items())) def main() -> int: parser = argparse.ArgumentParser() parser.add_argument("--region", required=True) parser.add_argument("url") args = parser.parse_args() try: provider = Session().get_credentials() if provider is None: raise ValueError("No AWS credentials available") request = signed_request(args.url, args.region, provider.get_frozen_credentials()) opener = build_opener(NoRedirect(), HTTPSHandler(context=ssl.create_default_context())) with opener.open(request, timeout=10) as response: print(response.status) print(response.read(1048576).decode("utf-8", errors="replace")) return 0 except HTTPError as exc: print(f"HTTP {exc.code}; check the policy and access logs", file=sys.stderr) except (URLError, BotoCoreError, ValueError) as exc: print(f"Request failed: {type(exc).__name__}", file=sys.stderr) return 1 if __name__ == "__main__": sys.exit(main()) ``` ```bash python3.12 -m venv lattice-client lattice-client/bin/python -m pip install 'botocore==1.43.93' lattice-client/bin/python lattice_get.py --region "$AWS_REGION" "https://${SERVICE_DNS}/api" ``` This GET-only example was checked with Python 3.12 and botocore 1.43.93. Workloads should use their configured Pod Identity or IRSA credentials. Do not copy static credentials or signed headers into manifests, logs, or support tickets. SigV4A is also supported by VPC Lattice; this example uses regional SigV4. ### Direct AWS API Management The following is an **alternative** for independently managed resources. Use a reachable, stable backend IP serving HTTP on 8080 and `/health`; a temporary pod IP requires a controller to track replacements. Do not manually change an HTTPRoute-owned service and expect the controller to retain the change. ```bash # Separate API-managed example; do not use for controller-managed resources. export TARGET_IP=10.0.1.25 export TARGET_GROUP_ID="$(aws vpc-lattice create-target-group \ --name api-manual --type IP \ --config "{\"port\":8080,\"protocol\":\"HTTP\",\"protocolVersion\":\"HTTP1\",\"vpcIdentifier\":\"${VPC_ID}\"}" \ --query id --output text)" aws vpc-lattice register-targets --target-group-identifier "$TARGET_GROUP_ID" \ --targets "id=$TARGET_IP,port=8080" export SERVICE_ID="$(aws vpc-lattice create-service \ --name api-manual --auth-type AWS_IAM --query id --output text)" aws vpc-lattice put-auth-policy --resource-identifier "$SERVICE_ID" \ --policy file://api-auth-policy.compact.json export LISTENER_ID="$(aws vpc-lattice create-listener \ --service-identifier "$SERVICE_ID" --name https --protocol HTTPS --port 443 \ --default-action "{\"forward\":{\"targetGroups\":[{\"targetGroupIdentifier\":\"${TARGET_GROUP_ID}\",\"weight\":1}]}}" \ --query id --output text)" aws vpc-lattice create-service-network-service-association \ --service-identifier "$SERVICE_ID" --service-network-identifier "$SERVICE_NETWORK_ID" aws vpc-lattice list-targets --target-group-identifier "$TARGET_GROUP_ID" aws vpc-lattice get-service --service-identifier "$SERVICE_ID" --query dnsEntry ``` Wait for healthy targets and active associations before calling the discovered HTTPS domain. This example uses an AWS-managed certificate for the generated domain, not a custom domain. ### Updating and Deleting Services For Kubernetes-owned resources, change the Route, backend workload, or policy manifest and verify reconciliation. For API-owned resources, use the corresponding update API and check its resulting state. Capture resource IDs from responses rather than selecting the first service in the account. Before removal, identify all consumers, network associations, listeners/rules, target-group references, and ownership. Remove the specific route/service associations and service resources in dependency order, then unused target groups. A shared Gateway/network can affect other namespaces or accounts. Retain the controller until finalizers and cloud cleanup complete; do not use blanket deletes. **Deleting IAMAuthPolicy disables IAM authentication on its target (`NONE`) before detaching the policy.** It is not a way to deny access or safely roll back authorization. Keep a restrictive policy while removing a service, and verify the remaining network/service controls. ## Routing and Traffic Management ### Path and Header Matching The route above matches `/api` and its path subtree. To add an explicit header-based canary rule, replace the **same** HTTPRoute with: ```yaml apiVersion: gateway.networking.k8s.io/v1 kind: HTTPRoute metadata: name: api namespace: lattice-demo spec: parentRefs: - name: my-network sectionName: https rules: - matches: - path: type: PathPrefix value: /api headers: - name: x-version value: canary backendRefs: - name: service-canary port: 8080 weight: 1 - matches: - path: type: PathPrefix value: /api backendRefs: - name: service-stable port: 8080 weight: 90 - name: service-canary port: 8080 weight: 10 ``` The controller documents case-insensitive path matching, one method match per rule, up to five header matches, and no query-parameter matching. Do not assume that every Gateway API filter or match is implemented. A separate HTTPRoute creates another Lattice service/domain, rather than automatically adding a rule to the first service. ### Weighted Routing `backendRefs.weight: 90` and `10` are native Gateway API configuration; no weighted-routing annotation is needed. They express a relative distribution, not an exact result for ten requests. Verify both versions' endpoints, health, errors, and latency over an appropriate sample before increasing the canary weight. For independently managed AWS resources: ```bash # TG_STABLE and TG_CANARY are existing target groups managed by this API workflow. aws vpc-lattice create-rule --service-identifier "$SERVICE_ID" \ --listener-identifier "$LISTENER_ID" --name api-canary --priority 10 \ --match '{"httpMatch":{"pathMatch":{"match":{"prefix":"/api"},"caseSensitive":false}}}' \ --action "{\"forward\":{\"targetGroups\":[{\"targetGroupIdentifier\":\"${TG_STABLE}\",\"weight\":90},{\"targetGroupIdentifier\":\"${TG_CANARY}\",\"weight\":10}]}}" ``` The CLI prefix match is a lexical prefix; review boundary behavior separately from Kubernetes `PathPrefix` semantics. Routing matches are not an authorization boundary. Do not use a path-routing test as proof that an IAM policy covers all normalized or encoded path variants. ### Health Checks The Kubernetes example uses `TargetGroupPolicy`. The equivalent API update is: ```bash aws vpc-lattice update-target-group --target-group-identifier "$TARGET_GROUP_ID" \ --health-check '{"enabled":true,"protocol":"HTTP","protocolVersion":"HTTP1","port":8080,"path":"/health","healthCheckIntervalSeconds":30,"healthCheckTimeoutSeconds":5,"healthyThresholdCount":2,"unhealthyThresholdCount":2,"matcher":{"httpCode":"200"}}' ``` Health checks assess readiness according to thresholds; they do not guarantee availability or zero downtime. HTTP1 target groups enable them by default, while HTTP2 requires explicit consideration. gRPC targets use HTTP1/HTTP2 health checks, and Lambda/ALB target types have different health-check behavior. Check the current target-type documentation instead of applying the pod example to every target. ## Security and Authentication ### Auth Policies and Caller Permissions `put-auth-policy` / `get-auth-policy` manage invocation authorization. `put-resource-policy` is a different management/sharing API. Use the **`vpc-lattice-svcs:Invoke`** action for callers. When both network and service use `AWS_IAM`, the caller's identity policy and **both** applicable auth policies must permit access. An explicit Deny wins. `NONE` on one resource does not cancel another resource's IAM requirement. Direct traffic to a Kubernetes ClusterIP/Pod IP bypasses Lattice auth; protect those paths with appropriate network and application controls. `StringEquals` does not interpret `/api/*` as a wildcard. The example uses `StringLike` and includes `/api` as well as `/api/*`. IAM condition matching and application path normalization can differ from controller routing. For administrative functionality, prefer a dedicated service restricted to administrative roles and retain application authorization; do not add a broad general Allow and assume a path wildcard protects every alias. ### Cross-Account Access RAM sharing allows association with the shared entity; it does not itself grant application invocation. The network/service auth policies, caller permissions, association security groups, and network path must still allow the request. ```bash # Owner account: choose a verified account ID or the actual Organizations ARN. export CONSUMER_ACCOUNT_ID=111122223333 aws ram create-resource-share --name lattice-network-share \ --resource-arns "$SERVICE_NETWORK_ARN" --principals "$CONSUMER_ACCOUNT_ID" # Consumer account: inspect invitations only when the sharing mode requires one. aws ram get-resource-share-invitations # After verifying the owner, resources, and intended permissions: aws ram accept-resource-share-invitation \ --resource-share-invitation-arn "$VERIFIED_INVITATION_ARN" # Run with consumer credentials and that account's VPC/security group values. aws vpc-lattice create-service-network-vpc-association \ --service-network-identifier "$SERVICE_NETWORK_ARN" \ --vpc-identifier "$CONSUMER_VPC_ID" \ --security-group-ids "$CONSUMER_ASSOCIATION_SG_ID" ``` With Organizations sharing enabled, consumers inside the organization receive access without an invitation. Other supported sharing arrangements require invitation acceptance. To share with an organization or OU, use its **actual ARN from Organizations**, including the management-account identifier, rather than composing one from a member-account ID. Owners can share services, networks, and resource configurations, not individual IAM roles as RAM consumers. Stopping a share prevents new associations but **does not remove existing associations**. Review them explicitly when revoking access. ### TLS and Custom Domains The sample Gateway exposes only HTTPS. For a custom hostname, create the service with that hostname, obtain a matching ACM certificate, and configure DNS to the actual assigned domain. Only one custom domain is supported per service and it cannot be changed after service creation. For the controller, set the HTTPRoute's `spec.hostnames` and the Gateway listener's `tls.options["application-networking.k8s.aws/certificate-arn"]`, or use its documented ACM discovery. Do not put private keys in an annotation. ExternalDNS automation additionally needs its controller, permissions, and the DNSEndpoint CRD; setting a hostname alone is not proof that DNS records exist. ```bash # For an API-managed service created with the required custom domain name: aws vpc-lattice update-service --service-identifier "$SERVICE_ID" \ --certificate-arn "$ACM_CERTIFICATE_ARN" # Create an HTTPS listener separately if the service does not already have one. # create-listener uses --protocol HTTPS; there is no --tls mode=STRICT option. ``` Client-facing HTTPS and backend TLS are separate. A backend `TargetGroupPolicy` with `protocol: HTTPS` also needs a backend that actually speaks TLS and a compatible HTTPS health check. VPC Lattice **does not validate backend certificates**; this encrypts the connection without authenticating the backend's certificate identity. Use the separate TLSRoute/TLS passthrough model when that is the intended design, and review its feature limitations. ## Monitoring and Logging ### CloudWatch Metrics, Dashboard, and Alarm Service metrics use the **`AWS/VpcLattice`** namespace: | Metric | Meaning / statistic | |---|---| | `TotalRequestCount` | Request count; `Sum` | | `HTTPCode_4XX_Count` | 4xx responses; `Sum` | | `HTTPCode_5XX_Count` | 5xx responses; `Sum` | | `RequestTime` | Request duration in **milliseconds**; average or a suitable percentile | Service metrics use the `Service` dimension, optionally with `AvailabilityZone`; target-group metrics use `TargetGroup`. A name such as `ServiceName=my-service` does not identify these metrics. Discover the actual dimension values/set: ```bash aws cloudwatch list-metrics --namespace AWS/VpcLattice \ --metric-name HTTPCode_5XX_Count --dimensions Name=Service > metrics.json python3 - <<'PY' import json for metric in json.load(open("metrics.json"))["Metrics"]: print(json.dumps(metric["Dimensions"])) PY ``` After traffic has produced metrics, select the intended service's **service-wide** dimension array and save it as `service-dimensions.json`. Do not arbitrarily select the first result or mix an AZ metric with an aggregate. Verify the identifier against the service being observed. Build `dashboard.json` with: ```python import json import os dimensions = json.load(open("service-dimensions.json")) if {d["Name"] for d in dimensions} != {"Service"}: raise ValueError("Select the service-wide metric, without AvailabilityZone") pairs = [item for d in dimensions for item in (d["Name"], d["Value"])] dashboard = {"widgets": [{ "type": "metric", "width": 12, "height": 6, "properties": { "title": "VPC Lattice requests and errors", "region": os.environ["AWS_REGION"], "period": 60, "stat": "Sum", "metrics": [["AWS/VpcLattice", name, *pairs] for name in ("TotalRequestCount", "HTTPCode_4XX_Count", "HTTPCode_5XX_Count")], }, }]} with open("dashboard.json", "w") as output: json.dump(dashboard, output) ``` ```bash aws cloudwatch put-dashboard --dashboard-name VPCLattice \ --dashboard-body file://dashboard.json aws cloudwatch put-metric-alarm --alarm-name LatticeApi5xx \ --namespace AWS/VpcLattice --metric-name HTTPCode_5XX_Count \ --dimensions file://service-dimensions.json \ --statistic Sum --period 60 --evaluation-periods 3 --datapoints-to-alarm 2 \ --threshold 5 --comparison-operator GreaterThanThreshold \ --treat-missing-data missing ``` The alarm means **more than five 5xx responses per minute in two of three periods**, not a 5% error rate. Configure reviewed alarm actions separately if notifications are required. The missing-data choice is explicit: metrics are published after traffic begins, and NoData must not be silently treated as proof of health. Dashboard and alarm settings are examples, not workload-specific SLOs. ### Access Logging For CloudWatch Logs, use an existing destination or create a dedicated log group with a retention policy: ```bash export LOG_GROUP=/aws/vendedlogs/vpc-lattice/api aws logs create-log-group --log-group-name "$LOG_GROUP" aws logs put-retention-policy --log-group-name "$LOG_GROUP" --retention-in-days 30 export LOG_DESTINATION_ARN="arn:aws:logs:${AWS_REGION}:${AWS_ACCOUNT_ID}:log-group:${LOG_GROUP}:*" # API-managed service only; for an HTTPRoute use AccessLogPolicy below instead. aws vpc-lattice create-access-log-subscription \ --resource-identifier "$SERVICE_ID" --destination-arn "$LOG_DESTINATION_ARN" ``` The setup principal also needs the documented log-delivery permissions. AWS can create/update the log resource policy when the setup principal has the necessary permissions; otherwise preconfigure it. Verify the `delivery.logs.amazonaws.com` permissions and source-account/source-ARN conditions. For the Kubernetes-managed route, use this **instead of** a competing CLI-created subscription: ```yaml apiVersion: application-networking.k8s.aws/v1alpha1 kind: AccessLogPolicy metadata: name: api-logs namespace: lattice-demo spec: targetRef: group: gateway.networking.k8s.io kind: HTTPRoute name: api destinationArn: arn:aws:logs:us-west-2:123456789012:log-group:/aws/vendedlogs/vpc-lattice/api:* ``` Replace the ARN and confirm policy status plus actual delivered events. A policy can target a Gateway for network logs or a Route for service logs. There can be one destination of each supported destination type per target. For S3, use a reviewed destination bucket with Block Public Access, encryption, retention/lifecycle rules, and appropriate delivery permissions: ```bash # Existing reviewed destination bucket; no policy is overwritten by this snippet. aws vpc-lattice create-access-log-subscription \ --resource-identifier "$SERVICE_ID" --destination-arn "$LOG_BUCKET_ARN" ``` S3 delivery requires the documented `s3:GetBucketAcl` and `s3:PutObject` permissions for `delivery.logs.amazonaws.com`, the delivery prefix, `aws:SourceAccount`, and `aws:SourceArn` conditions. Existing policies must be merged, not overwritten. SSE-KMS requires a supported customer-managed key and its delivery key policy. `--destination-name` is not an access-log-subscription parameter. ### Log Analysis and Tracing HTTP service access logs contain fields such as `sourceIpPort`, `requestMethod`, `requestPath`, `responseCode`, `durationMS`, `callerPrincipal`, and `authDeniedReason`. Resource/TCP logs have a different schema. ```bash END_TIME="$(python3 -c 'import time; print(int(time.time()))')" START_TIME="$((END_TIME - 3600))" QUERY_ID="$(aws logs start-query --log-group-name "$LOG_GROUP" \ --start-time "$START_TIME" --end-time "$END_TIME" \ --query-string 'fields @timestamp, sourceIpPort, requestMethod, requestPath, responseCode, durationMS, callerPrincipal, authDeniedReason | filter responseCode >= 400 | sort @timestamp desc | limit 100' \ --query queryId --output text)" aws logs get-query-results --query-id "$QUERY_ID" # Repeat get-query-results until Complete; Failed/Cancelled/Timeout are errors. ``` VPC Lattice has no `update-service --tracing-config` option or controller annotation that automatically instruments applications for X-Ray. Instrument the applications with OpenTelemetry/ADOT or the appropriate tracing SDK, propagate trace context, and configure export/sampling. Correlate application traces with access logs and request IDs; a client-supplied request ID is not an authenticated identity. ## Best Practices - **Design and ownership:** Use clear network/service naming and environment boundaries. Account for same-named Gateways across namespaces, shared network consumers, quotas, and the ownership of each policy and association. - **Deployment:** Keep stable and canary backends independently selectable. Check endpoints, target health, and authorization before shifting weights. Record rollback criteria and preserve the last known configuration. - **Performance:** Use bounded timeouts and appropriate connection reuse. Make health endpoints lightweight and meaningful. Cache or batch only where application semantics permit it. Private Lattice services do not become CDN origins merely by enabling caching. - **Security:** Separate management and caller roles; keep credentials out of manifests. Test permitted and denied roles, root paths and subpaths, direct-backend access, and TLS behavior. Do not delete an IAM policy CRD to deny traffic. - **Observability:** Monitor request count, error count/rate, latency, target health, and missing telemetry separately. Retain access logs for the required period and instrument application traces explicitly. - **Cost:** Review current regional service/resource, request, data-processing, endpoint, and logging charges for the chosen model. Use tags, remove only confirmed unused resources, and size backend autoscaling separately from the managed Lattice data plane. ## Troubleshooting Use identifiers from the controller annotations/status and AWS inventory. Do not assume the direct-API sample's `$SERVICE_ID` is the Kubernetes route's service. ```bash aws vpc-lattice list-service-network-vpc-associations \ --service-network-identifier "$SERVICE_NETWORK_ID" aws vpc-lattice list-service-network-service-associations \ --service-network-identifier "$SERVICE_NETWORK_ID" aws vpc-lattice get-service --service-identifier "$SERVICE_ID" aws vpc-lattice get-auth-policy --resource-identifier "$SERVICE_NETWORK_ID" aws vpc-lattice get-auth-policy --resource-identifier "$SERVICE_ID" aws vpc-lattice list-listeners --service-identifier "$SERVICE_ID" aws vpc-lattice list-rules --service-identifier "$SERVICE_ID" \ --listener-identifier "$LISTENER_ID" aws vpc-lattice get-target-group --target-group-identifier "$TARGET_GROUP_ID" aws vpc-lattice list-targets --target-group-identifier "$TARGET_GROUP_ID" ``` | Symptom | Check | |---|---| | DNS/connectivity failure | Actual assigned DNS, client VPC association or endpoint path, association state, SGs, NACLs, pod reachability | | 403/auth failure | Caller role, credential expiry and signing region/service, `UNSIGNED-PAYLOAD`, both auth layers, propagation, denied-reason log fields | | Wrong route or version | Route conditions, listener/rule priority and matches, target group membership, weights, distinct Route domains | | Unhealthy targets | Actual listening port, `/health`, HTTP vs HTTPS, readiness, SGs, target type and health-check thresholds | | No logs/metrics | Destination permissions and delivery state, correct metric dimensions, initial traffic, retention, query status | | Controller reconciliation failure | `manager` logs, IAM role, EndpointSlices, CRD version compatibility, webhook and leader-election status | Use a bounded metric interval without relying on GNU-only `date -d`: ```bash export METRIC_END="$(python3 -c 'from datetime import datetime,timezone; print(datetime.now(timezone.utc).isoformat())')" export METRIC_START="$(python3 -c 'from datetime import datetime,timedelta,timezone; print((datetime.now(timezone.utc)-timedelta(hours=1)).isoformat())')" aws cloudwatch get-metric-statistics --namespace AWS/VpcLattice \ --metric-name HTTPCode_5XX_Count --dimensions file://service-dimensions.json \ --start-time "$METRIC_START" --end-time "$METRIC_END" \ --period 60 --statistics Sum ``` For an AWS service incident, consult AWS Health and relevant account events. Account-specific API access and support operations depend on the applicable plan and endpoints. A support case should include reviewed resource IDs, time range, failure symptoms, and redacted logs. Select current service/category/severity options for the account; do not paste a hard-coded `urgent` case-creation command. ## References - [VPC Lattice overview](https://docs.aws.amazon.com/vpc-lattice/latest/ug/what-is-vpc-lattice.html) - [Service network associations](https://docs.aws.amazon.com/vpc-lattice/latest/ug/service-network-associations.html) - [Controller v2.1.3 installation](https://github.com/aws/aws-application-networking-k8s/blob/v2.1.3/docs/guides/deploy.md) - [Controller v2.1 upgrade requirements](https://github.com/aws/aws-application-networking-k8s/blob/v2.1.3/docs/guides/upgrading-v2-0-x-to-v2-1-y.md) - [Controller API reference](https://github.com/aws/aws-application-networking-k8s/tree/v2.1.3/docs/api-types) - [Controller HTTPS and backend TLS](https://github.com/aws/aws-application-networking-k8s/blob/v2.1.3/docs/guides/https.md) - [VPC Lattice auth policies](https://docs.aws.amazon.com/vpc-lattice/latest/ug/auth-policies.html) - [Signing requests](https://docs.aws.amazon.com/vpc-lattice/latest/ug/sigv4-authenticated-requests.html) - [Sharing entities](https://docs.aws.amazon.com/vpc-lattice/latest/ug/sharing.html) - [CloudWatch metrics](https://docs.aws.amazon.com/vpc-lattice/latest/ug/monitoring-cloudwatch.html) - [Access logs](https://docs.aws.amazon.com/vpc-lattice/latest/ug/monitoring-access-logs.html) - [CloudWatch Logs delivery permissions](https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/AWS-logs-infrastructure-CWL.html) - [S3 delivery permissions](https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/AWS-logs-infrastructure-S3.html) ## Quiz Test your understanding with the [VPC Lattice quiz](https://www.atomai.click/kubernetes-docs/en/quizzes/networking/02-vpc-lattice-quiz). ---------------------------------------- Source: https://www.atomai.click/kubernetes-docs/en/networking/03-aws-lb-controller ---------------------------------------- # AWS Load Balancer Controller > **Review baseline**: AWS Load Balancer Controller / Helm chart v3.5.0 > **Last Updated**: September 11, 2026 ## Overview AWS Load Balancer Controller is a controller that manages AWS Elastic Load Balancers (ELB) for Kubernetes clusters. It automatically integrates Kubernetes Ingress and Service resources with AWS Application Load Balancer (ALB) and Network Load Balancer (NLB). ### Key Features - **Application Load Balancer (ALB)**: HTTP/HTTPS traffic, path-based routing, host-based routing - **Network Load Balancer (NLB)**: TCP/UDP traffic, high-performance L4 load balancing - **TargetGroupBinding**: Connect existing Target Groups to Kubernetes Services - **AWS WAF Integration**: Web Application Firewall enforcement - **AWS Shield**: DDoS protection ![Diagram showing Ingress and Service resources in an EKS cluster triggering the AWS Load Balancer Controller, which creates an Application Load Balancer and a Network Load Balancer each with its own Target Group, while TargetGroupBinding binds an existing Target Group directly, and both Target Groups register the same backend Pods.](https://raw.githubusercontent.com/Atom-oh/kubernetes-docs/main/en/.gitbook/assets/en-networking-03-aws-lb-controller-0.png) [🔍 View interactive diagram](https://www.atomai.click/kubernetes-docs/archmaps/en-networking-03-aws-lb-controller-0.html) ## Architecture ### How the Controller Works ![Sequence in which the AWS Load Balancer Controller reacts to a new Ingress or Service by creating the ALB or NLB, target group and listener rules through the ELBv2 API, updates the resource status, and keeps registering targets as Pods change.](https://raw.githubusercontent.com/Atom-oh/kubernetes-docs/main/en/.gitbook/assets/en-networking-03-aws-lb-controller-1.png) [🔍 View interactive diagram](https://www.atomai.click/kubernetes-docs/archmaps/en-networking-03-aws-lb-controller-1.html) ### Component Structure Install the complete released chart, including RBAC, CRDs, probes, and webhook certificates. The controller watches Kubernetes objects and calls AWS APIs; application traffic passes through the load balancer and its targets, not through the controller pod. With leader election, one replica reconciles while the others provide standby capacity and webhook availability. Replica count alone does not guarantee placement across nodes or Availability Zones. ## Prerequisites ### Ownership and compatibility This chapter configures the **self-managed open-source controller**. EKS Auto Mode supplies its own managed load balancing: NLB Services use `eks.amazonaws.com/nlb`, ALB IngressClass uses `eks.amazonaws.com/alb`, and its TargetGroupBinding API differs from `elbv2.k8s.aws/v1beta1`. Check the Auto Mode migration guide rather than changing a class or copying all annotations in place. Explicit classes prevent ambiguity when both models are present. Use a currently supported EKS Kubernetes release and verify every controller sharing cluster-wide CRDs. LBC **v3.5.0** was released on **2026-08-03**; the verified chart **3.5.0** packages that controller. Gateway API users need **v1.6.0** CRDs before upgrading, and LBC-specific Gateway CRDs now use `gateway.k8s.aws/v1`. This does not mean that an arbitrary latest Gateway API or Kubernetes release is compatible. The old generic “Kubernetes 1.22+” installation floor is not a current EKS support matrix. The controller webhook needs TCP 9443 reachability from the control plane. Set region/VPC values explicitly when IMDS is restricted or the controller runs on Fargate/Hybrid Nodes; choose a supported credential mechanism for that compute type. IP targets need VPC-routable pod addresses and supported endpoint/ENI discovery. Amazon VPC CNI is the common EKS choice, but it is not the only possible CNI configuration. Instance targets require a NodePort-capable Service and appropriate node networking. ### 1. Create IAM Policy Use the IAM policy shipped with **v3.5.0** and the correct AWS partition. Review its broad discovery and security-group permissions, resource/tag conditions, and the features enabled in this deployment. Save the reviewed policy before creating it. Do not treat the upstream policy as a least-privilege guarantee or copy an old v2.8 policy into a current installation. Controller AWS credentials can use **IRSA or EKS Pod Identity** on supported nodes; they are separate from Kubernetes API RBAC. ### 2. IRSA Setup ```bash export AWS_REGION=us-east-1 export CLUSTER_NAME=my-cluster export AWS_ACCOUNT_ID="$(aws sts get-caller-identity --query Account --output text)" export VPC_ID="$(aws eks describe-cluster --name "$CLUSTER_NAME" \ --query 'cluster.resourcesVpcConfig.vpcId' --output text)" kubectl config current-context aws eks describe-cluster --name "$CLUSTER_NAME" \ --query cluster.identity.oidc.issuer --output text # Only if this cluster's IAM OIDC provider does not already exist: eksctl utils associate-iam-oidc-provider --cluster "$CLUSTER_NAME" \ --region "$AWS_REGION" --approve curl --fail --location --output iam-policy-upstream.json \ https://raw.githubusercontent.com/kubernetes-sigs/aws-load-balancer-controller/v3.5.0/docs/install/iam_policy.json export REVIEWED_POLICY_FILE=iam-policy-reviewed.json test -s "$REVIEWED_POLICY_FILE" export CONTROLLER_POLICY_ARN="$(aws iam create-policy \ --policy-name AWSLoadBalancerControllerIAMPolicy \ --policy-document "file://$REVIEWED_POLICY_FILE" --query Policy.Arn --output text)" eksctl create iamserviceaccount --cluster "$CLUSTER_NAME" --region "$AWS_REGION" \ --namespace kube-system --name aws-load-balancer-controller \ --attach-policy-arn "$CONTROLLER_POLICY_ARN" --approve ``` Reuse existing reviewed policies/roles instead of recreating them. A reused IRSA role needs a trust statement for this cluster’s OIDC provider and the intended service account. For an existing service account, review ownership and annotations before changing it. Pod Identity uses its own agent/association and role trust configuration; do not copy static access keys into chart values. ## Installation ### Installation with Helm ```bash helm repo add eks https://aws.github.io/eks-charts helm repo update eks helm pull eks/aws-load-balancer-controller --version 3.5.0 # Review cluster-wide CRD changes and other controllers before applying. curl --fail --location --output gateway-api-v1.6.0.yaml \ https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.6.0/standard-install.yaml kubectl apply --server-side -f gateway-api-v1.6.0.yaml helm show crds ./aws-load-balancer-controller-3.5.0.tgz > lbc-crds.yaml kubectl apply --server-side -f lbc-crds.yaml # Save the values below as controller-values.yaml and replace its cluster/region/VPC. helm install aws-load-balancer-controller ./aws-load-balancer-controller-3.5.0.tgz \ -n kube-system -f controller-values.yaml --wait --timeout 5m ``` ```yaml # values.yaml example clusterName: my-cluster serviceAccount: create: false name: aws-load-balancer-controller region: us-east-1 vpcId: vpc-0123456789abcdef0 # Resource settings resources: requests: cpu: 100m memory: 128Mi limits: cpu: 200m memory: 256Mi # Replica count replicaCount: 2 # Pod Disruption Budget podDisruptionBudget: minAvailable: 1 # Anti-Affinity for HA affinity: podAntiAffinity: preferredDuringSchedulingIgnoredDuringExecution: - weight: 100 podAffinityTerm: labelSelector: matchExpressions: - key: app.kubernetes.io/name operator: In values: - aws-load-balancer-controller topologyKey: kubernetes.io/hostname # Webhook certificates enableCertManager: false # Log level logLevel: info # IngressClass settings ingressClass: alb createIngressClassResource: true # Additional settings enableShield: false enableWaf: false enableWafv2: true # Use explicit Service classes; do not claim unclassified LoadBalancer Services. enableServiceMutatorWebhook: false enableEndpointSlices: true keepTLSSecret: true clusterSecretsPermissions: allowAllSecrets: false ``` The resource values are examples, not measured production sizing. Existing releases need a reviewed `helm upgrade` with saved values; Helm does not automatically upgrade CRDs. With `enableServiceMutatorWebhook: false`, this chapter’s NLB Services explicitly select `service.k8s.aws/nlb`. The default webhook otherwise mutates newly created LoadBalancer Services, not an existing Service whose type is later changed. `keepTLSSecret: true` reuses the Helm-managed webhook Secret when available; coordinate the CA bundle and pod certificate during GitOps/rotation, or use a separately installed compatible cert-manager. Do not delete shared CRDs to force an upgrade. ### Verify Installation ```bash # Check Deployment status kubectl get deployment -n kube-system aws-load-balancer-controller # Check Pod status kubectl get pods -n kube-system -l app.kubernetes.io/name=aws-load-balancer-controller # Check logs kubectl logs -n kube-system -l app.kubernetes.io/name=aws-load-balancer-controller # Check IngressClass kubectl get ingressclass ``` ## Application Load Balancer (ALB) Treat each manifest below as an independent example. Replace account/resource IDs, domains, subnets, security groups and certificate ARNs with verified values in the correct region. Create the referenced namespaces, Services and ready backend workloads first; service port 80 and target port 8080 are different roles. A declared containerPort does not make an application listen or implement /health. The health endpoint, actual target port, HTTP/TLS protocol, security groups and NetworkPolicies must agree. The image in the overview illustrates **IP targets**; instance targets register nodes and use NodePorts. TargetGroupBinding is also reconciled by this controller, and the sequence diagram is illustrative rather than an atomic transaction. ### Basic Ingress Configuration ```yaml apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: my-ingress namespace: default annotations: # ALB scheme (internet-facing or internal) alb.ingress.kubernetes.io/scheme: internet-facing # Target Type (ip or instance) alb.ingress.kubernetes.io/target-type: ip # Listener ports alb.ingress.kubernetes.io/listen-ports: '[{"HTTP": 80}, {"HTTPS": 443}]' # SSL redirect alb.ingress.kubernetes.io/ssl-redirect: "443" # ACM certificate alb.ingress.kubernetes.io/certificate-arn: arn:aws:acm:us-east-1:ACCOUNT:certificate/CERT_ID # Subnet specification alb.ingress.kubernetes.io/subnets: subnet-xxx,subnet-yyy,subnet-zzz # Security groups alb.ingress.kubernetes.io/security-groups: sg-xxxxxxxxx alb.ingress.kubernetes.io/manage-backend-security-group-rules: "true" # Health check settings alb.ingress.kubernetes.io/healthcheck-path: /health alb.ingress.kubernetes.io/healthcheck-interval-seconds: "15" alb.ingress.kubernetes.io/healthcheck-timeout-seconds: "5" alb.ingress.kubernetes.io/healthy-threshold-count: "2" alb.ingress.kubernetes.io/unhealthy-threshold-count: "2" spec: ingressClassName: alb rules: - host: api.example.com http: paths: - path: / pathType: Prefix backend: service: name: api-service port: number: 80 ``` ### Advanced Ingress Configuration ```yaml apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: advanced-ingress annotations: alb.ingress.kubernetes.io/scheme: internet-facing alb.ingress.kubernetes.io/target-type: ip # Group multiple Ingresses into single ALB alb.ingress.kubernetes.io/group.name: my-app-group alb.ingress.kubernetes.io/group.order: "10" # Target group attributes alb.ingress.kubernetes.io/target-group-attributes: >- stickiness.enabled=true, stickiness.lb_cookie.duration_seconds=60, slow_start.duration_seconds=30, deregistration_delay.timeout_seconds=30 # IP address type alb.ingress.kubernetes.io/ip-address-type: dualstack # Load balancer attributes alb.ingress.kubernetes.io/load-balancer-attributes: >- idle_timeout.timeout_seconds=60, routing.http2.enabled=true, routing.http.drop_invalid_header_fields.enabled=true, access_logs.s3.enabled=true, access_logs.s3.bucket=my-alb-logs, access_logs.s3.prefix=my-app # Tags alb.ingress.kubernetes.io/tags: Environment=production,Team=platform # WAF v2 integration alb.ingress.kubernetes.io/wafv2-acl-arn: arn:aws:wafv2:us-east-1:ACCOUNT:regional/webacl/my-acl/xxx # Shield Advanced alb.ingress.kubernetes.io/shield-advanced-protection: "true" spec: ingressClassName: alb tls: - hosts: - api.example.com - www.example.com rules: - host: api.example.com http: paths: - path: /v1 pathType: Prefix backend: service: name: api-v1 port: number: 80 - path: /v2 pathType: Prefix backend: service: name: api-v2 port: number: 80 - host: www.example.com http: paths: - path: / pathType: Prefix backend: service: name: web-frontend port: number: 80 ``` ### Path-Based Routing ```yaml apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: path-based-routing annotations: alb.ingress.kubernetes.io/scheme: internet-facing alb.ingress.kubernetes.io/target-type: ip # Condition-based routing alb.ingress.kubernetes.io/conditions.api-v2: >- [{"field":"http-header","httpHeaderConfig":{"httpHeaderName":"X-Api-Version","values":["v2"]}}] spec: ingressClassName: alb rules: - host: api.example.com http: paths: # Exact path matching - path: /health pathType: Exact backend: service: name: health-service port: number: 80 # API version routing - path: /api pathType: Prefix backend: service: name: api-v2 port: number: 80 - path: /api pathType: Prefix backend: service: name: api-v1 port: number: 80 # Static files - path: /static pathType: Prefix backend: service: name: static-service port: number: 80 # Default path - path: / pathType: Prefix backend: service: name: default-service port: number: 80 ``` ### Authentication Configuration These examples require an existing HTTPS certificate and identity-provider application. Configure the callback `https://app.example.com/oauth2/idpresponse`, authorization-code flow, allowed scopes, and the required client secret. The ALB must reach the provider’s token/user-info endpoints over IPv4; an internal ALB may need an appropriate egress/NAT path. Authentication happens only on HTTPS listeners. `allow` for unauthenticated requests does not protect the backend. Restrict direct backend access and verify the ALB-signed user claims as required by the application. ```yaml apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: auth-ingress annotations: alb.ingress.kubernetes.io/scheme: internet-facing alb.ingress.kubernetes.io/target-type: ip alb.ingress.kubernetes.io/listen-ports: '[{"HTTPS": 443}]' alb.ingress.kubernetes.io/certificate-arn: arn:aws:acm:us-east-1:123456789012:certificate/12345678-1234-1234-1234-123456789012 # Cognito authentication alb.ingress.kubernetes.io/auth-type: cognito alb.ingress.kubernetes.io/auth-idp-cognito: >- {"userPoolARN":"arn:aws:cognito-idp:us-east-1:ACCOUNT:userpool/us-east-1_xxxxx", "userPoolClientID":"xxxxxxxxx", "userPoolDomain":"my-domain"} alb.ingress.kubernetes.io/auth-on-unauthenticated-request: authenticate alb.ingress.kubernetes.io/auth-scope: "openid profile email" alb.ingress.kubernetes.io/auth-session-cookie: "AWSELBAuthSessionCookie" alb.ingress.kubernetes.io/auth-session-timeout: "3600" spec: ingressClassName: alb rules: - host: app.example.com http: paths: - path: / pathType: Prefix backend: service: name: protected-app port: number: 80 ``` ```yaml # OIDC Authentication Example apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: oidc-ingress annotations: alb.ingress.kubernetes.io/scheme: internet-facing alb.ingress.kubernetes.io/target-type: ip alb.ingress.kubernetes.io/listen-ports: '[{"HTTPS": 443}]' alb.ingress.kubernetes.io/certificate-arn: arn:aws:acm:us-east-1:123456789012:certificate/12345678-1234-1234-1234-123456789012 # OIDC authentication alb.ingress.kubernetes.io/auth-type: oidc alb.ingress.kubernetes.io/auth-idp-oidc: >- {"issuer":"https://accounts.google.com", "authorizationEndpoint":"https://accounts.google.com/o/oauth2/v2/auth", "tokenEndpoint":"https://oauth2.googleapis.com/token", "userInfoEndpoint":"https://openidconnect.googleapis.com/v1/userinfo", "secretName":"oidc-secret"} alb.ingress.kubernetes.io/auth-on-unauthenticated-request: authenticate spec: ingressClassName: alb rules: - host: app.example.com http: paths: - path: / pathType: Prefix backend: service: name: protected-app port: number: 80 --- # OIDC Secret apiVersion: v1 kind: Secret metadata: name: oidc-secret type: Opaque stringData: clientID: your-client-id clientSecret: your-client-secret ``` The OIDC Secret must be in the Ingress namespace. The chart defaults to `clusterSecretsPermissions.allowAllSecrets: false`; grant this controller only the required Secret access. v3.5.0 watches a Secret using a `metadata.name` field selector, so the Role can constrain `resourceNames`. Create the real Secret through the approved secret-management workflow; do not commit a real client secret. ```yaml apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: name: lbc-oidc-secret namespace: default rules: - apiGroups: - '' resources: - secrets resourceNames: - oidc-secret verbs: - get - list - watch --- apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: name: lbc-oidc-secret namespace: default subjects: - kind: ServiceAccount name: aws-load-balancer-controller namespace: kube-system roleRef: apiGroup: rbac.authorization.k8s.io kind: Role name: lbc-oidc-secret ``` ## Network Load Balancer (NLB) ### Basic NLB Service Configuration ```yaml apiVersion: v1 kind: Service metadata: name: nlb-service annotations: # Specify NLB type service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: "ip" # Scheme service.beta.kubernetes.io/aws-load-balancer-scheme: "internet-facing" # Subnet specification service.beta.kubernetes.io/aws-load-balancer-subnets: subnet-xxx,subnet-yyy # Health check service.beta.kubernetes.io/aws-load-balancer-healthcheck-protocol: "HTTP" service.beta.kubernetes.io/aws-load-balancer-healthcheck-path: "/health" service.beta.kubernetes.io/aws-load-balancer-healthcheck-port: "8080" service.beta.kubernetes.io/aws-load-balancer-healthcheck-interval: "10" service.beta.kubernetes.io/aws-load-balancer-healthcheck-healthy-threshold: "2" service.beta.kubernetes.io/aws-load-balancer-healthcheck-unhealthy-threshold: "2" spec: type: LoadBalancer loadBalancerClass: service.k8s.aws/nlb selector: app: my-app ports: - name: tcp port: 80 targetPort: 8080 protocol: TCP ``` ### Weighted Target Groups The Service below gives its own implicit target group weight 90 and the existing `service-canary:8080` backend weight 10. Both Services must have the intended ready endpoints and compatible target settings; this annotation does not create the canary workload. The annotation suffix is the listener protocol and port, **`actions.TCP-80`**. Weights are relative values from **0 to 999** and apply to new connections. Ordinary weight changes preserve existing connections; **setting a target group’s weight to 0 closes its existing connections after a short period**, as well as stopping new ones. Do not describe this as a guaranteed zero-downtime drain. TLS listeners require compatible target-group protocols and do not support target-group stickiness. ```yaml apiVersion: v1 kind: Service metadata: name: nlb-weighted namespace: default annotations: service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: ip service.beta.kubernetes.io/aws-load-balancer-scheme: internal service.beta.kubernetes.io/actions.TCP-80: '{"type":"forward","forwardConfig":{"baseServiceWeight":90,"targetGroups":[{"serviceName":"service-canary","servicePort":8080,"weight":10}]}}' spec: type: LoadBalancer loadBalancerClass: service.k8s.aws/nlb selector: app: my-app version: stable ports: - name: tcp port: 80 targetPort: 8080 protocol: TCP ``` ### TLS Termination NLB ```yaml apiVersion: v1 kind: Service metadata: name: nlb-tls-service annotations: service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: "ip" service.beta.kubernetes.io/aws-load-balancer-scheme: "internet-facing" # TLS configuration service.beta.kubernetes.io/aws-load-balancer-ssl-cert: "arn:aws:acm:us-east-1:ACCOUNT:certificate/CERT_ID" service.beta.kubernetes.io/aws-load-balancer-ssl-ports: "443" service.beta.kubernetes.io/aws-load-balancer-ssl-negotiation-policy: "ELBSecurityPolicy-TLS13-1-2-2021-06" # Backend is HTTP service.beta.kubernetes.io/aws-load-balancer-backend-protocol: "tcp" spec: type: LoadBalancer loadBalancerClass: service.k8s.aws/nlb selector: app: my-app ports: - name: https port: 443 targetPort: 8080 protocol: TCP ``` ### Internal NLB ```yaml apiVersion: v1 kind: Service metadata: name: internal-nlb annotations: service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: "ip" # Internal scheme service.beta.kubernetes.io/aws-load-balancer-scheme: "internal" # Cross-zone load balancing service.beta.kubernetes.io/aws-load-balancer-attributes: "load_balancing.cross_zone.enabled=true" # Private subnets service.beta.kubernetes.io/aws-load-balancer-subnets: subnet-private-a,subnet-private-b # Security groups (optional) service.beta.kubernetes.io/aws-load-balancer-security-groups: sg-xxxxxxxxx service.beta.kubernetes.io/aws-load-balancer-manage-backend-security-group-rules: "true" spec: type: LoadBalancer loadBalancerClass: service.k8s.aws/nlb selector: app: internal-service ports: - port: 80 targetPort: 8080 ``` ### UDP Support NLB ```yaml apiVersion: v1 kind: Service metadata: name: udp-nlb annotations: service.beta.kubernetes.io/aws-load-balancer-enable-tcp-udp-listener: "true" service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: "ip" service.beta.kubernetes.io/aws-load-balancer-scheme: "internet-facing" spec: type: LoadBalancer loadBalancerClass: service.k8s.aws/nlb selector: app: dns-server ports: - name: dns-udp port: 53 targetPort: 53 protocol: UDP - name: dns-tcp port: 53 targetPort: 53 protocol: TCP ``` ### Proxy Protocol v2 Proxy Protocol v2 conveys the original client address as binary connection metadata; it does **not** preserve the IP packet’s source address. The example disables packet-level client-IP preservation to make the distinction explicit. The backend must parse Proxy Protocol before application data, including applicable health-check connections. An ordinary HTTP/TLS server cannot consume that prefix without configuration. `preserve_client_ip.enabled` controls NLB packet-source preservation where supported by the target type/protocol/network path. With instance/NodePort targets, `externalTrafficPolicy: Local` can avoid a subsequent kube-proxy SNAT hop; it is not a universal substitute for NLB preservation. IP-family translation and unsupported transit/hairpin paths require separate consideration. ```yaml apiVersion: v1 kind: Service metadata: name: proxy-protocol-nlb annotations: service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: "ip" service.beta.kubernetes.io/aws-load-balancer-scheme: "internet-facing" # Enable Proxy Protocol v2 # Target Group attributes service.beta.kubernetes.io/aws-load-balancer-target-group-attributes: >- proxy_protocol_v2.enabled=true, preserve_client_ip.enabled=false spec: type: LoadBalancer loadBalancerClass: service.k8s.aws/nlb selector: app: proxy-aware-app ports: - port: 80 targetPort: 8080 ``` ## IngressClass and IngressClassParams This optional class is named `alb-platform` to avoid overwriting the chart-owned `alb` class. Label the intended namespaces `alb-enabled=true` and set their Ingress `spec.ingressClassName: alb-platform`. Do not make it a cluster default unless that is the intended policy. IngressClassParams settings take precedence over corresponding annotations. ### IngressClass Definition ```yaml apiVersion: networking.k8s.io/v1 kind: IngressClass metadata: name: alb-platform spec: controller: ingress.k8s.aws/alb parameters: apiGroup: elbv2.k8s.aws kind: IngressClassParams name: alb-params ``` ### IngressClassParams Configuration ```yaml apiVersion: elbv2.k8s.aws/v1beta1 kind: IngressClassParams metadata: name: alb-params spec: # Default scheme scheme: internet-facing # IP address type ipAddressType: dualstack # Namespace selector (allow only specific namespaces) namespaceSelector: matchLabels: alb-enabled: "true" # Default tags tags: - key: Environment value: production - key: ManagedBy value: aws-load-balancer-controller # Load balancer attributes loadBalancerAttributes: - key: idle_timeout.timeout_seconds value: "60" - key: routing.http2.enabled value: "true" # Subnet selection # subnets: # ids: # - subnet-xxx # - subnet-yyy # tags: # kubernetes.io/role/elb: ["1"] # Group settings group: name: my-default-group ``` ## TargetGroupBinding The TargetGroupBinding CRD allows you to directly connect existing AWS Target Groups to Kubernetes Services. ### Basic TargetGroupBinding ```yaml apiVersion: elbv2.k8s.aws/v1beta1 kind: TargetGroupBinding metadata: name: my-tgb namespace: default spec: # Existing Target Group ARN targetGroupARN: arn:aws:elasticloadbalancing:us-east-1:ACCOUNT:targetgroup/my-tg/xxxxxxxxxxxx # Service to connect serviceRef: name: my-service port: 80 # Target Type (ip or instance) targetType: ip # Networking settings networking: ingress: - from: - securityGroup: groupID: sg-xxxxxxxxx ports: - port: 80 protocol: TCP ``` A TGB manages registrations, not the existing load balancer/listener lifecycle. Keep its service port, target-group protocol/IP family, backend target port, and security-group rules consistent. `nodeSelector` only filters **instance** targets; it does not choose IP-mode pods. Restrict TGB creation/update to trusted operators because the controller’s IAM permissions can allow references to other target groups in the account. When multiple clusters or TGBs share one target group, configure `spec.multiClusterTargetGroup: true` **from creation on every participating TGB**. The default `false` assumes full ownership and can deregister targets from other clusters. Do not casually toggle this flag after creation; the documented change can leak targets. Separate target groups per cluster are another ownership model. ### Advanced TargetGroupBinding ```yaml apiVersion: elbv2.k8s.aws/v1beta1 kind: TargetGroupBinding metadata: name: advanced-tgb namespace: production spec: targetGroupARN: arn:aws:elasticloadbalancing:us-east-1:ACCOUNT:targetgroup/prod-tg/xxxxxxxxxxxx serviceRef: name: production-service port: 8080 targetType: ip # IP address type ipAddressType: ipv4 # VPC ID (auto-detected, can be explicit) # vpcID: vpc-xxxxxxxxx # Networking settings networking: ingress: # Allow traffic from multiple security groups - from: - securityGroup: groupID: sg-alb-sg - securityGroup: groupID: sg-internal-sg ports: - port: 8080 protocol: TCP - port: 8443 protocol: TCP # Node selector applies to instance targets, not IP-mode pod selection # nodeSelector: # matchLabels: # node-type: compute ``` ### Multi-port TargetGroupBinding ```yaml # Separate TargetGroupBindings for multiple ports --- apiVersion: elbv2.k8s.aws/v1beta1 kind: TargetGroupBinding metadata: name: http-tgb spec: targetGroupARN: arn:aws:elasticloadbalancing:...:targetgroup/http-tg/xxx serviceRef: name: multi-port-service port: 80 targetType: ip --- apiVersion: elbv2.k8s.aws/v1beta1 kind: TargetGroupBinding metadata: name: https-tgb spec: targetGroupARN: arn:aws:elasticloadbalancing:...:targetgroup/https-tg/yyy serviceRef: name: multi-port-service port: 443 targetType: ip ``` ## WAF and Shield Integration Use an existing regional Web ACL in the ALB region and configure its intended rules. The installation values enable WAF v2 but disable Shield integration; to use the Shield Advanced example, first arrange the required subscription/permissions and enable the controller’s Shield integration. Its annotation alone does not activate a paid subscription or override a disabled controller feature. These ALB integrations do not imply WAF inspects arbitrary NLB TCP/UDP traffic. The S3 access-log example also requires an existing destination bucket and the documented ALB log-delivery bucket policy. ### AWS WAF v2 Integration ```yaml apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: waf-protected-ingress annotations: alb.ingress.kubernetes.io/scheme: internet-facing alb.ingress.kubernetes.io/target-type: ip # Connect WAF v2 WebACL alb.ingress.kubernetes.io/wafv2-acl-arn: arn:aws:wafv2:us-east-1:ACCOUNT:regional/webacl/my-webacl/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx spec: ingressClassName: alb rules: - host: api.example.com http: paths: - path: / pathType: Prefix backend: service: name: api-service port: number: 80 ``` ### AWS Shield Advanced ```yaml apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: shield-protected-ingress annotations: alb.ingress.kubernetes.io/scheme: internet-facing alb.ingress.kubernetes.io/target-type: ip # Enable Shield Advanced protection alb.ingress.kubernetes.io/shield-advanced-protection: "true" spec: ingressClassName: alb rules: - host: critical-app.example.com http: paths: - path: / pathType: Prefix backend: service: name: critical-service port: number: 80 ``` ## Notable Version Updates - **v2.16.0 — 2025-11-20:** ALB Target Optimizer and NLB weighted target groups. Target Optimizer requires its target-control agent and configuration; it is not enabled merely by installing LBC. - **v2.17.0 — 2025-12-19:** Global Accelerator support through the single `aga.k8s.aws/v1beta1` `GlobalAccelerator` CRD, with nested listeners, endpoint groups and endpoints; Gateway API GA release-candidate status. Global Accelerator needs its additional IAM permissions and feature configuration. - **v3.5.0 — 2026-08-03:** Gateway API v1.6.0 conformance and stable v1 TCPRoute/UDPRoute support. LBC Gateway configuration resources use `gateway.k8s.aws/v1`; the still-served v1beta1 version is deprecated. Current v3.5 supports QUIC/TCP_QUIC configuration and ALB JWT validation. These are distinct features with protocol-specific constraints. JWT validation is HTTPS-only and its JSON uses **`jwksEndpoint`**, not `jwksUri`. Add the following to the annotations of an HTTPS Ingress with a valid certificate, reachable trusted JWKS endpoint, and reviewed issuer/claims: ```yaml alb.ingress.kubernetes.io/jwt-validation: >- {"issuer":"https://accounts.example.com","jwksEndpoint":"https://accounts.example.com/.well-known/jwks.json"} ``` This is an annotation fragment, not a complete Kubernetes object. Validate the required audience/other claims for the application instead of assuming signature validation alone is sufficient authorization. See the [Gateway API guide](https://www.atomai.click/kubernetes-docs/llms/en/networking/04-gateway-api.md) for the separate Gateway configuration. ## Annotation Reference ### ALB Ingress Annotations | Annotation | Description | Default | |------------|-------------|---------| | `alb.ingress.kubernetes.io/scheme` | internet-facing or internal | internal | | `alb.ingress.kubernetes.io/target-type` | ip or instance | instance | | `alb.ingress.kubernetes.io/subnets` | Subnet IDs or names | Auto-detect | | `alb.ingress.kubernetes.io/security-groups` | Security group IDs | Auto-create | | `alb.ingress.kubernetes.io/listen-ports` | Listener ports JSON | HTTP 80, or HTTPS 443 when certificate-arn is specified | | `alb.ingress.kubernetes.io/certificate-arn` | ACM certificate ARN | - | | `alb.ingress.kubernetes.io/ssl-redirect` | SSL redirect port | - | | `alb.ingress.kubernetes.io/ssl-policy` | SSL policy | ELBSecurityPolicy-2016-08 | | `alb.ingress.kubernetes.io/healthcheck-path` | Health check path | / | | `alb.ingress.kubernetes.io/healthcheck-port` | Health check port | traffic-port | | `alb.ingress.kubernetes.io/healthcheck-protocol` | Health check protocol | HTTP | | `alb.ingress.kubernetes.io/healthcheck-interval-seconds` | Health check interval | 15 | | `alb.ingress.kubernetes.io/healthcheck-timeout-seconds` | Health check timeout | 5 | | `alb.ingress.kubernetes.io/healthy-threshold-count` | Healthy threshold | 2 | | `alb.ingress.kubernetes.io/unhealthy-threshold-count` | Unhealthy threshold | 2 | | `alb.ingress.kubernetes.io/group.name` | Ingress group name | - | | `alb.ingress.kubernetes.io/group.order` | Priority within group | 0 | | `alb.ingress.kubernetes.io/ip-address-type` | ipv4 or dualstack | ipv4 | | `alb.ingress.kubernetes.io/load-balancer-attributes` | LB attributes | - | | `alb.ingress.kubernetes.io/target-group-attributes` | TG attributes | - | | `alb.ingress.kubernetes.io/tags` | Resource tags | - | | `alb.ingress.kubernetes.io/wafv2-acl-arn` | WAF v2 WebACL ARN | - | | `alb.ingress.kubernetes.io/shield-advanced-protection` | Shield protection | false | | `alb.ingress.kubernetes.io/auth-type` | Auth type (none, cognito, oidc) | none | ### NLB Service Annotations | Annotation | Description | Default | |------------|-------------|---------| | `service.beta.kubernetes.io/aws-load-balancer-type` | external (NLB) or nlb | - | | `service.beta.kubernetes.io/aws-load-balancer-nlb-target-type` | ip or instance | instance | | `service.beta.kubernetes.io/aws-load-balancer-scheme` | internet-facing or internal | internal | | `service.beta.kubernetes.io/aws-load-balancer-subnets` | Subnet IDs | Auto-detect | | `service.beta.kubernetes.io/aws-load-balancer-ssl-cert` | ACM certificate ARN | - | | `service.beta.kubernetes.io/aws-load-balancer-ssl-ports` | SSL-enabled ports | - | | `service.beta.kubernetes.io/aws-load-balancer-ssl-negotiation-policy` | SSL policy | - | | `service.beta.kubernetes.io/aws-load-balancer-backend-protocol` | Backend protocol | - | | `service.beta.kubernetes.io/aws-load-balancer-proxy-protocol` | Proxy Protocol | - | | `service.beta.kubernetes.io/aws-load-balancer-cross-zone-load-balancing-enabled` | Deprecated; use aws-load-balancer-attributes | false | | `service.beta.kubernetes.io/aws-load-balancer-healthcheck-protocol` | Health check protocol | TCP | | `service.beta.kubernetes.io/aws-load-balancer-healthcheck-path` | Health check path | - | | `service.beta.kubernetes.io/aws-load-balancer-healthcheck-port` | Health check port | - | | `service.beta.kubernetes.io/aws-load-balancer-attributes` | LB attributes | - | | `service.beta.kubernetes.io/aws-load-balancer-target-group-attributes` | TG attributes | - | | `service.beta.kubernetes.io/aws-load-balancer-security-groups` | Security groups | Auto-create | ## EKS Best Practices ### 1. Subnet Tagging Role tags are a clear way to select intended public/private subnets. In self-managed LBC v2.12.1+, when there are no matching role-tagged subnets, the default `SubnetDiscoveryByReachability` behavior can instead classify them from route tables. Explicit subnet IDs or IngressClassParams tag filters are other paths. EKS Auto Mode still requires its documented subnet tags. Check cluster-tag filtering, available IPs, and one eligible subnet per selected AZ; an ordinary ALB requires at least two AZs. Tagging a subnet does not change its route table or make it public. ```bash # Public subnets (for internet-facing ALB/NLB) aws ec2 create-tags \ --resources subnet-xxx \ --tags Key=kubernetes.io/role/elb,Value=1 # Private subnets (for internal ALB/NLB) aws ec2 create-tags \ --resources subnet-yyy \ --tags Key=kubernetes.io/role/internal-elb,Value=1 # Cluster-specific tag (optional) aws ec2 create-tags \ --resources subnet-xxx subnet-yyy \ --tags Key=kubernetes.io/cluster/my-cluster,Value=shared ``` ### 2. Security Group Management ```yaml apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: secure-ingress annotations: alb.ingress.kubernetes.io/scheme: internet-facing alb.ingress.kubernetes.io/target-type: ip # Explicit security group specification alb.ingress.kubernetes.io/security-groups: sg-alb-external # Configure approved inbound sources on this explicit security group. # inbound-cidrs is ignored when security-groups is specified. # Additional security groups (for backend communication) alb.ingress.kubernetes.io/manage-backend-security-group-rules: "true" spec: ingressClassName: alb rules: - host: api.example.com http: paths: - path: / pathType: Prefix backend: service: name: api-service port: number: 80 ``` ### 3. Cost Optimization IngressGroup shares an ALB and its rule space. Use it only within a trust boundary: a user able to create an Ingress that joins the group can affect routing and priority. Enforce RBAC/admission and review merged/exclusive annotation settings. Group membership is not a namespace-isolation feature or an unconditional cost guarantee. ```yaml # Share ALB using Ingress groups apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: app1-ingress annotations: alb.ingress.kubernetes.io/group.name: shared-alb alb.ingress.kubernetes.io/group.order: "1" spec: ingressClassName: alb rules: - host: app1.example.com http: paths: - path: / pathType: Prefix backend: service: name: app1 port: number: 80 --- apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: app2-ingress annotations: alb.ingress.kubernetes.io/group.name: shared-alb alb.ingress.kubernetes.io/group.order: "2" spec: ingressClassName: alb rules: - host: app2.example.com http: paths: - path: / pathType: Prefix backend: service: name: app2 port: number: 80 ``` ### 4. High Availability Configuration ```yaml apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: ha-ingress annotations: alb.ingress.kubernetes.io/scheme: internet-facing alb.ingress.kubernetes.io/target-type: ip # Specify subnets in 3+ AZs alb.ingress.kubernetes.io/subnets: subnet-az-a,subnet-az-b,subnet-az-c # ALB cross-zone is enabled at the load-balancer level. # Review target-group overrides separately. # Health check optimization alb.ingress.kubernetes.io/healthcheck-interval-seconds: "10" alb.ingress.kubernetes.io/healthy-threshold-count: "2" alb.ingress.kubernetes.io/unhealthy-threshold-count: "2" # Draining timeout alb.ingress.kubernetes.io/target-group-attributes: deregistration_delay.timeout_seconds=30 spec: ingressClassName: alb rules: - host: api.example.com http: paths: - path: / pathType: Prefix backend: service: name: api-service port: number: 80 ``` ## Troubleshooting Set the named variables below from the actual namespace and resource inventory. Inspect the controller’s event/error reason before changing infrastructure. The optional exec health check assumes the application image contains curl; otherwise use an approved diagnostic container. Protect logs and credentials when gathering evidence. A 502 can have connection-reset, malformed-response or TLS causes; inspect ALB access-log error details rather than assuming every unhealthy target produces the same HTTP status. ### Common Issues #### 1. ALB Not Created ```bash # Check controller logs kubectl logs -n kube-system -l app.kubernetes.io/name=aws-load-balancer-controller # Check Ingress events kubectl describe ingress "$INGRESS_NAME" -n "$NAMESPACE" # Common causes: # - Insufficient IAM permissions # - Missing subnet tags # - IngressClass not specified ``` #### 2. Targets Unhealthy ```bash # Check Target Group status aws elbv2 describe-target-health \ --target-group-arn "$TARGET_GROUP_ARN" # Check Pod logs kubectl logs "$POD_NAME" -n "$NAMESPACE" --tail=100 # Test health check endpoint kubectl exec "$POD_NAME" -n "$NAMESPACE" -- curl --fail --max-time 5 http://localhost:8080/health # Check security groups aws ec2 describe-security-groups --group-ids "$SECURITY_GROUP_ID" ``` #### 3. 502 Bad Gateway ```bash # Root cause analysis: # 1. Pod not ready kubectl get pods -l app=my-app # 2. Target Group draining aws elbv2 describe-target-health --target-group-arn "$TARGET_GROUP_ARN" # 3. Health check failure # - Verify health check path # - Adjust health check timeout # 4. Security group rules # - Verify ALB -> Pod communication allowed ``` #### 4. SSL Certificate Issues ```bash # Check ACM certificate status aws acm describe-certificate --certificate-arn "$ACM_CERTIFICATE_ARN" # Verify certificate is ISSUED status # Check domain validation completed # Verify region (must be same region as ALB) ``` ### Debugging Commands ```bash # Controller detailed logs kubectl logs -n kube-system deployment/aws-load-balancer-controller -f # Ingress status check kubectl get ingress -o wide kubectl describe ingress "$INGRESS_NAME" -n "$NAMESPACE" # Service status check kubectl get svc -o wide kubectl describe svc "$SERVICE_NAME" -n "$NAMESPACE" # TargetGroupBinding status check kubectl get targetgroupbindings -A kubectl describe targetgroupbinding "$TGB_NAME" -n "$NAMESPACE" # AWS resource check aws elbv2 describe-load-balancers --query 'LoadBalancers[?contains(LoadBalancerName, `k8s`)]' aws elbv2 describe-target-groups --query 'TargetGroups[?contains(TargetGroupName, `k8s`)]' ``` --- ## References - [AWS Load Balancer Controller Documentation](https://kubernetes-sigs.github.io/aws-load-balancer-controller/) - [GitHub Repository](https://github.com/kubernetes-sigs/aws-load-balancer-controller) - [EKS User Guide](https://docs.aws.amazon.com/eks/latest/userguide/aws-load-balancer-controller.html) - [ALB Documentation](https://docs.aws.amazon.com/elasticloadbalancing/latest/application/) - [NLB Documentation](https://docs.aws.amazon.com/elasticloadbalancing/latest/network/) - [LBC v3.5.0 release](https://github.com/kubernetes-sigs/aws-load-balancer-controller/releases/tag/v3.5.0) - [LBC v3.5.0 Ingress annotations](https://github.com/kubernetes-sigs/aws-load-balancer-controller/blob/v3.5.0/docs/guide/ingress/annotations.md) - [LBC v3.5.0 Service annotations](https://github.com/kubernetes-sigs/aws-load-balancer-controller/blob/v3.5.0/docs/guide/service/annotations.md) - [TargetGroupBinding ownership](https://github.com/kubernetes-sigs/aws-load-balancer-controller/blob/v3.5.0/docs/guide/targetgroupbinding/targetgroupbinding.md) - [Subnet discovery](https://github.com/kubernetes-sigs/aws-load-balancer-controller/blob/v3.5.0/docs/deploy/subnet_discovery.md) - [NLB listener weights and connections](https://docs.aws.amazon.com/elasticloadbalancing/latest/network/load-balancer-listeners.html) - [ALB authentication prerequisites](https://docs.aws.amazon.com/elasticloadbalancing/latest/application/listener-authenticate-users.html) - [EKS Auto Mode NLB](https://docs.aws.amazon.com/eks/latest/userguide/auto-configure-nlb.html) ---------------------------------------- Source: https://www.atomai.click/kubernetes-docs/en/networking/04-gateway-api ---------------------------------------- # Kubernetes Gateway API > **API baseline**: Gateway API v1.6 Standard; select the exact bundle supported by your controller. > **Last Updated**: September 12, 2026 ## Overview Gateway API is the next-generation ingress API for Kubernetes, designed to overcome the limitations of the existing Ingress API and provide more expressive and extensible network routing capabilities. Developed by SIG-Network, it is supported by various implementations including Istio, Cilium, Envoy Gateway, and more. ### Limitations of Ingress API | Problem | Description | |---------|-------------| | **Limited Expressiveness** | Poor support for TCP/UDP/gRPC beyond HTTP routing | | **Combined Responsibilities** | RBAC/IngressClass can restrict access, but listener and route concerns are less explicitly separated | | **Annotation Abuse** | Implementation-specific features handled via annotations, reducing portability | | **Limited Extensibility** | Difficult to add new protocols or features | | **Cross-Namespace** | Complex routing across namespaces | ### Benefits of Gateway API ![Four Gateway API design goals: expressiveness, responsibility separation, portability and extensibility.](https://raw.githubusercontent.com/Atom-oh/kubernetes-docs/main/en/.gitbook/assets/en-networking-04-gateway-api-0.png) [🔍 View interactive diagram](https://www.atomai.click/kubernetes-docs/archmaps/en-networking-04-gateway-api-0.html) Expressiveness, separation of responsibilities, portability and extensibility are independent design goals. Actual feature support depends on the controller and conformance profile; Kubernetes RBAC and admission policies enforce who can change each resource. ## Resource Model Gateway API uses a layered resource model. ![GatewayClass, Gateway and Route relationships with backend Services in a typical implementation; the exact Gateway infrastructure is controller-specific.](https://raw.githubusercontent.com/Atom-oh/kubernetes-docs/main/en/.gitbook/assets/en-networking-04-gateway-api-1.png) [🔍 View interactive diagram](https://www.atomai.click/kubernetes-docs/archmaps/en-networking-04-gateway-api-1.html) The figure shows resource relationships and a common gateway deployment model. A Gateway is not universally one cloud load balancer: Istio can provision a proxy Deployment/Service, while VPC Lattice maps it to a service network. The owner of a **referenced namespace** grants cross-namespace backend/Secret access with ReferenceGrant. ### Role Separation | Role | Managed Resources | Responsibility | |------|------------------|----------------| | **Infrastructure Provider** | GatewayClass | Define basic infrastructure configuration | | **Cluster Operator** | Gateway | Gateway infrastructure and Route attachment policy | | **Referenced Namespace Owner** | ReferenceGrant | Authorize references to owned backends/Secrets | | **Application Developer** | HTTPRoute, GRPCRoute, etc. | Define application routing rules | ## GatewayClass GatewayClass defines the controller and configuration to use when creating Gateways. ```yaml apiVersion: gateway.networking.k8s.io/v1 kind: GatewayClass metadata: name: istio spec: controllerName: istio.io/gateway-controller description: Istio Gateway Controller for production workloads ``` A GatewayClass selects an already installed controller; creating the class does not install that controller. The definitions below are alternatives. Use a class whose `Accepted` condition is true, and replace the example class names with the accepted names in your cluster. `parametersRef` support and its group/kind depend on the implementation. For Istio 1.31, a per-Gateway ConfigMap goes under `Gateway.spec.infrastructure.parametersRef` in the Gateway's namespace. Class-wide defaults use a ConfigMap labeled `gateway.istio.io/defaults-for-class` in Istio's root namespace. The ALB→Istio example below demonstrates the per-Gateway form. ### GatewayClass by Implementation ```yaml # Istio apiVersion: gateway.networking.k8s.io/v1 kind: GatewayClass metadata: name: istio spec: controllerName: istio.io/gateway-controller --- # Cilium apiVersion: gateway.networking.k8s.io/v1 kind: GatewayClass metadata: name: cilium spec: controllerName: io.cilium/gateway-controller --- # AWS Gateway API Controller apiVersion: gateway.networking.k8s.io/v1 kind: GatewayClass metadata: name: amazon-vpc-lattice spec: controllerName: application-networking.k8s.aws/gateway-api-controller --- # Envoy Gateway apiVersion: gateway.networking.k8s.io/v1 kind: GatewayClass metadata: name: envoy-gateway spec: controllerName: gateway.envoyproxy.io/gatewayclass-controller --- # Contour apiVersion: gateway.networking.k8s.io/v1 kind: GatewayClass metadata: name: contour spec: controllerName: projectcontour.io/gateway-controller --- # NGINX Gateway Fabric apiVersion: gateway.networking.k8s.io/v1 kind: GatewayClass metadata: name: nginx spec: controllerName: gateway.nginx.org/nginx-gateway-controller ``` ## Gateway Gateway describes traffic-handling infrastructure and listeners. Its mapping to proxy workloads, managed load balancers, or a service network depends on the controller. ### Basic Gateway Configuration These are separate configuration scenarios, not a single set of Routes to apply together. Overlapping Routes on the same host/listener can change precedence. Install the selected controller and its compatible CRDs first, create `gateway-system`, and provide the named Services, ready endpoints, and TLS Secrets. Certificates must cover the configured DNS names. Configure the data-plane Service exposure, DNS and network controls for the platform; a GatewayClass or a requested IP address does not reserve an external address by itself. The HTTP/gRPC/TCP/TLS examples use Istio 1.31. The UDP example uses a separate Envoy Gateway instance because Istio 1.31 explicitly rejects UDP listeners. Envoy Gateway 1.9 requires Gateway API 1.6.1 and its published Kubernetes version combination. Review shared CRDs before changing their version/channel. The Namespace in the basic example has `gateway-access: "true"`. This is a **Namespace label**, and permission to change it should stay with the administrators controlling Gateway access. `allowedRoutes` does not authenticate application clients. ```yaml apiVersion: v1 kind: Namespace metadata: name: production labels: gateway-access: 'true' --- apiVersion: gateway.networking.k8s.io/v1 kind: Gateway metadata: name: production-gateway namespace: gateway-system spec: gatewayClassName: istio listeners: - name: http protocol: HTTP port: 80 allowedRoutes: namespaces: from: Selector selector: matchLabels: gateway-access: 'true' - name: https protocol: HTTPS port: 443 tls: mode: Terminate certificateRefs: - kind: Secret name: tls-cert namespace: gateway-system allowedRoutes: namespaces: from: Selector selector: matchLabels: gateway-access: 'true' ``` ### Advanced Gateway Configuration The following is a separate Gateway named `multi-protocol-gateway`. The gRPC, TLS and TCP Routes below attach to its matching listener names. The database and other TCP examples have distinct listeners, so both can be used without competing for one L4 listener. ```yaml apiVersion: gateway.networking.k8s.io/v1 kind: Gateway metadata: name: multi-protocol-gateway namespace: gateway-system spec: gatewayClassName: istio listeners: - name: http protocol: HTTP port: 80 allowedRoutes: namespaces: from: Selector selector: matchLabels: gateway-access: 'true' - name: https-wildcard protocol: HTTPS port: 443 hostname: '*.example.com' tls: mode: Terminate certificateRefs: - kind: Secret name: wildcard-cert allowedRoutes: namespaces: from: Selector selector: matchLabels: gateway-access: 'true' kinds: - kind: HTTPRoute - name: grpc protocol: HTTPS port: 443 hostname: grpc.example.com tls: mode: Terminate certificateRefs: - kind: Secret name: grpc-cert allowedRoutes: namespaces: from: Selector selector: matchLabels: gateway-access: 'true' kinds: - kind: GRPCRoute - name: tcp-passthrough protocol: TLS port: 8443 tls: mode: Passthrough allowedRoutes: namespaces: from: Selector selector: matchLabels: gateway-access: 'true' kinds: - kind: TLSRoute - name: tcp protocol: TCP port: 9000 allowedRoutes: namespaces: from: Selector selector: matchLabels: gateway-access: 'true' kinds: - kind: TCPRoute - name: database protocol: TCP port: 5432 allowedRoutes: namespaces: from: Selector selector: matchLabels: gateway-access: 'true' kinds: - kind: TCPRoute ``` ### TLS Modes Termination ends the downstream TLS connection at the gateway. The backend connection is configured separately and can use HTTP or TLS, for example through a supported BackendTLSPolicy. Passthrough uses a `TLS` listener with `mode: Passthrough`, and the backend terminates TLS. A `HTTPS` listener cannot be switched to passthrough merely by changing `mode`. | Mode | Description | Use Case | |------|-------------|----------| | **Terminate** | TLS termination at Gateway | Standard HTTPS | | **Passthrough** | Pass TLS to backend | End-to-end encryption | ```yaml # TLS Terminate example listeners: - name: https protocol: HTTPS port: 443 tls: mode: Terminate certificateRefs: - kind: Secret name: server-cert --- # TLS Passthrough example listeners: - name: tls-passthrough protocol: TLS port: 443 tls: mode: Passthrough ``` ## HTTPRoute HTTPRoute defines routing rules for HTTP/HTTPS traffic. ### Basic HTTPRoute ```yaml apiVersion: gateway.networking.k8s.io/v1 kind: HTTPRoute metadata: name: basic-route namespace: production spec: # Gateway to attach to parentRefs: - name: production-gateway namespace: gateway-system sectionName: https # Target specific listener # Host matching hostnames: - "api.example.com" - "www.example.com" # Routing rules rules: - matches: - path: type: PathPrefix value: /api/v1 backendRefs: - name: api-v1-service port: 80 - matches: - path: type: PathPrefix value: /api/v2 backendRefs: - name: api-v2-service port: 80 # Default path - backendRefs: - name: default-service port: 80 ``` ### Advanced Matching Rules Fields inside one `matches` item are ANDed; multiple items are ORed. PathPrefix matches path elements rather than an arbitrary string prefix. RegularExpression support and syntax are implementation-specific. The demo tenant header below is a routing selector that any client could supply, not authentication for the administrative application. ```yaml apiVersion: gateway.networking.k8s.io/v1 kind: HTTPRoute metadata: name: advanced-matching namespace: production spec: parentRefs: - name: production-gateway namespace: gateway-system sectionName: https hostnames: - api.example.com rules: - matches: - path: type: Exact value: /health backendRefs: - name: health-service port: 80 - matches: - path: type: RegularExpression value: /users/[0-9]+ backendRefs: - name: user-service port: 80 - matches: - headers: - name: X-Version value: v2 backendRefs: - name: api-v2-service port: 80 - matches: - queryParams: - name: debug value: 'true' backendRefs: - name: debug-service port: 80 - matches: - method: POST path: type: PathPrefix value: /api/data backendRefs: - name: write-service port: 80 - matches: - method: GET path: type: PathPrefix value: /api/data backendRefs: - name: read-service port: 80 - matches: - path: type: PathPrefix value: /admin headers: - name: X-Demo-Tenant type: Exact value: operations backendRefs: - name: admin-service port: 80 - matches: - path: type: PathPrefix value: /api - path: type: PathPrefix value: /v1 backendRefs: - name: api-service port: 80 ``` ### Filters Header modifiers set literal values. `X-Example-Source: gateway-demo` is a static marker, not a generated unique request ID; use the proxy/application's tracing facilities for IDs. The mirror example uses its own `/mirror` path so the earlier `/api` rule cannot shadow it. It copies GET requests to a shadow backend and ignores that backend's response. Isolate side effects and review the data/credentials copied to the shadow service. The public cache header is appropriate only for content that is actually safe to cache publicly. Filters allow modifying requests/responses. ```yaml apiVersion: gateway.networking.k8s.io/v1 kind: HTTPRoute metadata: name: filtered-route namespace: production spec: parentRefs: - name: production-gateway namespace: gateway-system sectionName: https rules: - matches: - path: type: PathPrefix value: /api filters: - type: RequestHeaderModifier requestHeaderModifier: add: - name: X-Example-Source value: gateway-demo set: - name: X-Api-Version value: v1 remove: - X-Internal-Header backendRefs: - name: api-service port: 80 - matches: - path: type: PathPrefix value: /public filters: - type: ResponseHeaderModifier responseHeaderModifier: add: - name: Cache-Control value: public, max-age=3600 set: - name: X-Content-Type-Options value: nosniff backendRefs: - name: public-service port: 80 - matches: - path: type: PathPrefix value: /old-api filters: - type: URLRewrite urlRewrite: path: type: ReplacePrefixMatch replacePrefixMatch: /new-api hostname: new-api.example.com backendRefs: - name: new-api-service port: 80 - matches: - path: type: PathPrefix value: /legacy filters: - type: RequestRedirect requestRedirect: scheme: https hostname: new.example.com port: 443 statusCode: 301 path: type: ReplacePrefixMatch replacePrefixMatch: /modern - matches: - method: GET path: type: PathPrefix value: /mirror filters: - type: RequestMirror requestMirror: backendRef: name: shadow-service port: 80 backendRefs: - name: main-service port: 80 ``` ### Traffic Splitting (Weights) ```yaml apiVersion: gateway.networking.k8s.io/v1 kind: HTTPRoute metadata: name: canary-route namespace: production spec: parentRefs: - name: production-gateway namespace: gateway-system sectionName: https hostnames: - app.example.com rules: - matches: - path: type: PathPrefix value: / backendRefs: - name: app-stable port: 80 weight: 90 - name: app-canary port: 80 weight: 10 ``` ### Timeouts and Retries The v1.6 Standard schema includes `timeouts`; it does not include `HTTPRoute.rules.retry`. Experimental schemas add retry fields and have separate admission/implementation requirements. The example below only sets budgets for GET requests: `backendRequest` must not exceed the nonzero total `request` budget. Omitting a retry field does not prove that a client, gateway, mesh proxy or SDK will never retry. Configure and verify each applicable layer, especially for non-idempotent writes. A retry count of zero is not a valid way to disable the experimental v1.6 `retry.attempts` field, whose minimum is one. Use the implementation's documented controls and application idempotency behavior. ```yaml apiVersion: gateway.networking.k8s.io/v1 kind: HTTPRoute metadata: name: resilient-route namespace: production spec: parentRefs: - name: production-gateway namespace: gateway-system sectionName: https rules: - matches: - path: type: PathPrefix value: /api method: GET timeouts: request: 30s backendRequest: 25s backendRefs: - name: api-service port: 80 ``` ## GRPCRoute Backends must serve the expected gRPC/HTTP2 transport and appropriate TLS configuration. A port number alone does not configure that behavior. Defines routing rules for gRPC traffic. ```yaml apiVersion: gateway.networking.k8s.io/v1 kind: GRPCRoute metadata: name: grpc-route namespace: production spec: parentRefs: - name: multi-protocol-gateway namespace: gateway-system sectionName: grpc hostnames: - grpc.example.com rules: - matches: - method: service: myapp.UserService backendRefs: - name: user-grpc-service port: 50051 - matches: - method: service: myapp.OrderService method: CreateOrder backendRefs: - name: order-grpc-service port: 50052 - matches: - headers: - name: x-environment value: staging backendRefs: - name: staging-grpc-service port: 50051 - backendRefs: - name: default-grpc-service port: 50051 ``` ## TCPRoute Defines TCP traffic routing. ```yaml apiVersion: gateway.networking.k8s.io/v1 kind: TCPRoute metadata: name: database-route namespace: production spec: parentRefs: - name: multi-protocol-gateway namespace: gateway-system sectionName: database rules: - backendRefs: - name: database-service port: 5432 --- apiVersion: gateway.networking.k8s.io/v1 kind: TCPRoute metadata: name: tcp-loadbalance namespace: production spec: parentRefs: - name: multi-protocol-gateway namespace: gateway-system sectionName: tcp rules: - backendRefs: - name: tcp-backend-1 port: 9000 weight: 50 - name: tcp-backend-2 port: 9000 weight: 50 ``` ## TLSRoute Defines TLS passthrough traffic routing. ```yaml apiVersion: gateway.networking.k8s.io/v1 kind: TLSRoute metadata: name: tls-passthrough-route namespace: production spec: parentRefs: - name: multi-protocol-gateway namespace: gateway-system sectionName: tcp-passthrough hostnames: - secure.example.com rules: - backendRefs: - name: secure-backend port: 8443 ``` ## UDPRoute This scenario requires an installed Envoy Gateway controller with an accepted `envoy-gateway` class, its compatible Gateway API bundle, and the `dns-service` UDP backend. It exposes UDP port 5300 and routes to backend port 53. Envoy's UDP proxy is non-transparent: the backend sees the gateway's source IP/port. Ensure the platform load-balancer/Service supports this UDP exposure. Defines UDP traffic routing. ```yaml apiVersion: gateway.networking.k8s.io/v1 kind: Gateway metadata: name: udp-gateway namespace: gateway-system spec: gatewayClassName: envoy-gateway listeners: - name: udp protocol: UDP port: 5300 allowedRoutes: namespaces: from: Selector selector: matchLabels: gateway-access: 'true' kinds: - kind: UDPRoute --- apiVersion: gateway.networking.k8s.io/v1 kind: UDPRoute metadata: name: dns-route namespace: production spec: parentRefs: - name: udp-gateway namespace: gateway-system sectionName: udp rules: - backendRefs: - name: dns-service port: 53 ``` ## ReferenceGrant ReferenceGrant is created in the namespace **containing the referenced Service or Secret**, by that namespace's owner. `from` selects source group/kind/namespace; `to.name` can constrain the target name. Grants are additive and authorize references, not application callers. Route→Gateway attachment across namespaces uses `parentRefs` plus the Gateway listener's `allowedRoutes` handshake, rather than a ReferenceGrant. Backend and certificate references use ReferenceGrant as shown below. The named `shared-api` Service and `shared-tls` Secret must exist; a grant alone does not create them. ReferenceGrant allows cross-namespace references. ```yaml apiVersion: gateway.networking.k8s.io/v1beta1 kind: ReferenceGrant metadata: name: allow-routes-to-backend namespace: backend-services spec: from: - group: gateway.networking.k8s.io kind: HTTPRoute namespace: production - group: gateway.networking.k8s.io kind: HTTPRoute namespace: staging to: - group: '' kind: Service name: shared-api --- apiVersion: gateway.networking.k8s.io/v1beta1 kind: ReferenceGrant metadata: name: allow-gateway-to-secrets namespace: cert-management spec: from: - group: gateway.networking.k8s.io kind: Gateway namespace: gateway-system to: - group: '' kind: Secret name: shared-tls ``` ## Implementation Comparison ### Major Implementations | Implementation | Controller | Features | |----------------|------------|----------| | **Istio** | istio.io/gateway-controller | Service Mesh integration, advanced traffic management | | **Cilium** | io.cilium/gateway-controller | Cilium networking with Envoy L7 processing | | **Envoy Gateway** | gateway.envoyproxy.io/gatewayclass-controller | Envoy-based, standards compliant | | **AWS Gateway API Controller** | application-networking.k8s.aws/gateway-api-controller | VPC Lattice integration | | **Contour** | projectcontour.io/gateway-controller | Envoy-based, simple configuration | | **NGINX Gateway Fabric** | gateway.nginx.org/nginx-gateway-controller | NGINX-based | | **Traefik** | traefik.io/gateway-controller | Dynamic configuration | ### Versioned Implementation Notes The API's release channel, a feature's Core/Extended/implementation-specific support level, and a controller's conformance profile are different concepts. A CRD accepting a field does not prove the controller implements it. Check published conformance results and resource conditions, including `Accepted`, `ResolvedRefs` and `Programmed` where applicable. | Implementation checked | Verified scope and important limits | |---|---| | Istio **1.31.0** | HTTP/gRPC and v1 TCP/TLS routes; UDP listeners are explicitly unsupported. Configurable Envoy data plane, not a guarantee that every Gateway API extension is supported | | Cilium **1.20.1** | Gateway API **1.6.1**, including TCPRoute/UDPRoute; combines Cilium networking with Envoy for L7 processing | | Envoy Gateway **1.9.1** | Gateway API **1.6.1**; published Kubernetes matrix is **1.33–1.36**. Supports UDP routing and TLS passthrough with their documented transport behavior | | AWS Load Balancer Controller **3.5.0** | Gateway API **1.6.0**; ALB handles HTTP/gRPC and NLB handles L4 routes. Only the oldest attached L4 Route is eligible per NLB listener; use one Route per listener | | AWS Gateway API Controller **2.1.3** | VPC Lattice integration; v2.1 requires Gateway API **1.5+**. HTTPRoute, GRPCRoute and TLSRoute are supported. TCP resource access is a separate Lattice resource-configuration model, not generic TCPRoute/UDPRoute support | | Contour **1.33.7** | Built with Gateway API **1.3.0** and release-tested on Kubernetes **1.32–1.34**. Documents HTTP/gRPC/TCP/TLS routes; use its matching channel/provisioning configuration rather than applying a newer bundle blindly | | NGINX Gateway Fabric **2.7.0** | Gateway API **1.6.1**, published Kubernetes minimum **1.32**; adds v1 TCPRoute/UDPRoute support. Separate product from retired community ingress-nginx | These notes replace a versionless yes/no feature grid. Consult each implementation's documentation for individual filters, TLS policies, extensions, supported versions and operating requirements. Contour's bundled compatibility page does not have a 1.33.7-specific row; the API dependency and Kubernetes range above are taken from that exact release's module file and release notes. ## AWS Load Balancer Controller Gateway API Support Gateway API reached GA in **LBC v3.0.0 on 2026-01-23**. Existing Ingress and Service APIs remain supported, so a Gateway migration can be planned independently of the controller upgrade. Current v3.5.0 requires the compatible Gateway API and LBC Gateway CRDs described in the [LBC installation guide](https://www.atomai.click/kubernetes-docs/llms/en/networking/03-aws-lb-controller.md). EKS Auto Mode has a separate managed implementation; self-managed LBC features do not automatically describe Auto Mode. The retired controller is the Kubernetes community **ingress-nginx** project, whose maintenance ended in March 2026. This does not mean the Kubernetes Ingress API or F5's other NGINX products were retired. The `keepTLSSecret=false` workaround in v3.0 release notes applied to users **staying on older versions** affected by the cert-manager ownership bug. Users upgrading to v3.0 received the fix without that extra action. Follow the current chart's certificate-management options rather than applying the historical mitigation to every upgrade. ### LBC v3.4.0 Migration Tools The **2026-06-03** release introduced the real `lbc-migrate` CLI and Migration Console. They target working **LBC Ingress** resources; they are not a generic converter for every Ingress implementation. - `lbc-migrate` reads files or, with `--from-cluster`, lists/gets cluster resources. It translates supported annotations and emits Gateway API resources. The default output has the LBC Gateway dry-run annotation. - The Migration Console compares controller-generated resource plans. It requires the appropriate plan annotations, feature configuration and read access. Treat plans as configuration data that may need access restrictions and redaction. - Applying reviewed live Gateway manifests creates **new ALBs alongside the existing ALBs**. Validate them and shift frontend traffic separately. Backend weights inside one HTTPRoute do not perform this frontend migration. For a binary built from the selected LBC release, file-based translation can start with: ```bash lbc-migrate -f ingress.yaml --output-dir ./gateway-output/ ``` The converter does not generate the existing Deployments/Services or revalidate all Ingress annotations. Review unsupported annotations, Service/IngressClassParams overrides, cross-namespace IngressGroup membership, rule precedence and TLS settings. External target groups already associated with the old ALB cannot simply be attached to the new ALB simultaneously; plan a compatible duplicate/cutover strategy. The tools provide a migration workflow, not a zero-downtime guarantee. See the [versioned migration guide](https://github.com/kubernetes-sigs/aws-load-balancer-controller/blob/v3.5.0/docs/guide/ingress2gateway/migrate_from_ingress.md) and [CLI reference](https://github.com/kubernetes-sigs/aws-load-balancer-controller/blob/v3.5.0/docs/guide/ingress2gateway/lbc_migrate_reference.md). ## Migrating from Ingress to Gateway API ### Step-by-Step Migration Guide #### Step 1: Analyze Existing Ingress The following is a **historical community ingress-nginx input** used to explain a manual configuration translation to Istio Gateway API. It is not a new ingress-nginx installation recommendation or input for the LBC-specific converter above. Preserve the actual request behavior, not just the names of settings. ```yaml apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: my-ingress annotations: kubernetes.io/ingress.class: nginx nginx.ingress.kubernetes.io/rewrite-target: / nginx.ingress.kubernetes.io/ssl-redirect: 'true' namespace: default spec: tls: - hosts: - api.example.com secretName: api-tls rules: - host: api.example.com http: paths: - path: /api/v1 pathType: Prefix backend: service: name: api-v1 port: number: 80 - path: /api/v2 pathType: Prefix backend: service: name: api-v2 port: number: 80 ``` #### Step 2: Create Gateway and GatewayClass The new Gateway is named `migration-gateway`. The existing `api-tls` Secret remains in `default`; a ReferenceGrant in that namespace explicitly allows this Gateway namespace to reference it. Provide a certificate valid for `api.example.com`. The `default` Namespace's built-in name label supplies the route-attachment selector. ```yaml apiVersion: gateway.networking.k8s.io/v1 kind: GatewayClass metadata: name: production spec: controllerName: istio.io/gateway-controller --- apiVersion: gateway.networking.k8s.io/v1beta1 kind: ReferenceGrant metadata: name: migration-tls namespace: default spec: from: - group: gateway.networking.k8s.io kind: Gateway namespace: gateway-system to: - group: '' kind: Secret name: api-tls --- apiVersion: gateway.networking.k8s.io/v1 kind: Gateway metadata: name: migration-gateway namespace: gateway-system spec: gatewayClassName: production listeners: - name: http protocol: HTTP port: 80 allowedRoutes: namespaces: from: Selector selector: matchLabels: kubernetes.io/metadata.name: default hostname: api.example.com - name: https protocol: HTTPS port: 443 tls: mode: Terminate certificateRefs: - kind: Secret name: api-tls namespace: default allowedRoutes: namespaces: from: Selector selector: matchLabels: kubernetes.io/metadata.name: default hostname: api.example.com ``` #### Step 3: Create HTTPRoute The redirect Route attaches only to the HTTP listener. The HTTPS Route forwards application requests. The old `rewrite-target: /` example replaces the entire matched request path with `/`, so the translated example uses **ReplaceFullPath**. ReplacePrefixMatch would preserve a suffix (`/api/v1/users` → `/users`) and change behavior. Test root paths, subpaths, query strings and redirects against the old application before cutover. The redirect below assumes ingress-nginx’s default **308**, preserving the request method/body; check any `http-redirect-code` override. Its rewrite annotation also enables case-insensitive regex locations for that host, while Gateway API PathPrefix is case-sensitive and matches path elements. Thus `/API/V1` or `/api/v10` can differ. The example demonstrates a stricter PathPrefix policy, not complete matching equivalence. If clients depend on the old behavior, design and test a supported regex match or another explicit compatibility rule before switching. ```yaml apiVersion: gateway.networking.k8s.io/v1 kind: HTTPRoute metadata: name: api-route namespace: default spec: parentRefs: - name: migration-gateway namespace: gateway-system sectionName: http hostnames: - api.example.com rules: - matches: - path: type: PathPrefix value: / filters: - type: RequestRedirect requestRedirect: scheme: https statusCode: 308 --- apiVersion: gateway.networking.k8s.io/v1 kind: HTTPRoute metadata: name: api-route-https namespace: default spec: parentRefs: - name: migration-gateway namespace: gateway-system sectionName: https hostnames: - api.example.com rules: - matches: - path: type: PathPrefix value: /api/v1 filters: - type: URLRewrite urlRewrite: path: type: ReplaceFullPath replaceFullPath: / backendRefs: - name: api-v1 port: 80 - matches: - path: type: PathPrefix value: /api/v2 filters: - type: URLRewrite urlRewrite: path: type: ReplaceFullPath replaceFullPath: / backendRefs: - name: api-v2 port: 80 ``` #### Step 4: Shift Frontend Traffic Validate the new Gateway's address, certificate, HTTP redirects, route matching, backend behavior and observability before moving clients. Shift traffic using the mechanism appropriate for the frontends, such as reviewed DNS/load-balancer routing, then monitor failures and latency. Account for DNS caches, persistent connections and sessions. Keep a tested way to send traffic back to the old frontend. HTTPRoute backend weights control traffic **inside the chosen Gateway**; the dedicated traffic-splitting section explains that operation. For the frontend migration, retain the old Ingress/controller until clients have moved and the required drain/rollback checks have completed. ### Migration Checklist - [ ] Analyze existing Ingress annotations - [ ] Select implementation and create GatewayClass - [ ] Create Gateway resource and configure listeners - [ ] Convert routing rules to HTTPRoute - [ ] Configure allowedRoutes for attachment and ReferenceGrant for backend/Secret references - [ ] Migrate TLS certificates - [ ] Verify and shift frontend traffic with a tested rollback path - [ ] Set up monitoring and logging - [ ] Remove old Ingress resources only after cutover and drain/rollback checks ## EKS Patterns ### AWS Gateway API Controller (VPC Lattice) Use the installed controller, `my-network` service network with its reviewed `AWS_IAM` policy, caller permissions, and `service-stable:8080` backend from the [VPC Lattice guide](https://www.atomai.click/kubernetes-docs/llms/en/networking/02-vpc-lattice.md). The Gateway name selects the network; it does not create it. This separate Route has its own Lattice service and domain. The IAMAuthPolicy below secures that service; retain the network-level policy during reconciliation. Retrieve the assigned Route domain and use the guide's signed HTTPS client. The `unused` certificate reference follows this controller's documented AWS-managed-certificate behavior, not generic Kubernetes Secret loading. ```yaml apiVersion: gateway.networking.k8s.io/v1 kind: GatewayClass metadata: name: amazon-vpc-lattice spec: controllerName: application-networking.k8s.aws/gateway-api-controller --- apiVersion: gateway.networking.k8s.io/v1 kind: Gateway metadata: name: my-network namespace: lattice-demo spec: gatewayClassName: amazon-vpc-lattice listeners: - name: https protocol: HTTPS port: 443 tls: mode: Terminate certificateRefs: - name: unused --- apiVersion: gateway.networking.k8s.io/v1 kind: HTTPRoute metadata: name: lattice-route namespace: lattice-demo spec: parentRefs: - name: my-network sectionName: https rules: - matches: - path: type: PathPrefix value: /api backendRefs: - name: service-stable port: 8080 --- apiVersion: application-networking.k8s.aws/v1alpha1 kind: IAMAuthPolicy metadata: name: lattice-route-auth namespace: lattice-demo spec: targetRef: group: gateway.networking.k8s.io kind: HTTPRoute name: lattice-route policy: '{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"AWS":"arn:aws:iam::123456789012:role/MyAppRole"},"Action":"vpc-lattice-svcs:Invoke","Resource":"*","Condition":{"StringLike":{"vpc-lattice-svcs:RequestPath":["/api","/api/*"]}}}]}' ``` ### Using with ALB Controller This topology uses an ALB in front of an Istio gateway when the application needs Istio's gateway behavior. The ConfigMap sets the generated Service to ClusterIP through Istio's documented infrastructure parameters. The ALB Ingress is in the **same namespace** as that Service and references its generated name, `internal-gateway-istio`. The application HTTPRoute is in the labeled `production` namespace and points to an existing `api-service:80` there. Replace the ACM ARN and configure the LBC/network prerequisites. TLS terminates at the ALB in this example; its connection to Istio is HTTP. Permit the actual gateway traffic and health ports through security controls. `/healthz/ready` on 15021 checks gateway readiness, not every application's health. Verify Route conditions and application responses separately. ```yaml apiVersion: v1 kind: ConfigMap metadata: name: internal-gateway-options namespace: istio-system data: service: | spec: type: ClusterIP --- apiVersion: gateway.networking.k8s.io/v1 kind: Gateway metadata: name: internal-gateway namespace: istio-system spec: gatewayClassName: istio listeners: - name: http protocol: HTTP port: 80 allowedRoutes: namespaces: from: Selector selector: matchLabels: gateway-access: 'true' infrastructure: parametersRef: group: '' kind: ConfigMap name: internal-gateway-options --- apiVersion: gateway.networking.k8s.io/v1 kind: HTTPRoute metadata: name: alb-internal-route namespace: production spec: parentRefs: - name: internal-gateway namespace: istio-system sectionName: http hostnames: - api.example.com rules: - backendRefs: - name: api-service port: 80 --- apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: alb-to-gateway annotations: alb.ingress.kubernetes.io/scheme: internet-facing alb.ingress.kubernetes.io/target-type: ip alb.ingress.kubernetes.io/listen-ports: '[{"HTTP":80},{"HTTPS":443}]' alb.ingress.kubernetes.io/ssl-redirect: '443' alb.ingress.kubernetes.io/certificate-arn: arn:aws:acm:us-east-1:123456789012:certificate/12345678-1234-1234-1234-123456789012 alb.ingress.kubernetes.io/healthcheck-port: '15021' alb.ingress.kubernetes.io/healthcheck-path: /healthz/ready namespace: istio-system spec: ingressClassName: alb rules: - host: api.example.com http: paths: - path: / pathType: Prefix backend: service: name: internal-gateway-istio port: number: 80 ``` ## API Channels and Maturity ### Channel and API Version Are Separate | Gateway API v1.6.0 bundle | Included resources/fields | |---|---| | Standard | GatewayClass, Gateway, HTTPRoute, GRPCRoute, TLSRoute, TCPRoute, UDPRoute, ReferenceGrant, BackendTLSPolicy and ListenerSet | | Experimental | The Standard content plus experimental fields such as HTTPRoute retry/session persistence, and XBackend, XBackendTrafficPolicy and XMesh | Current Standard examples use `v1` for the Route types. The v1.6.0 Standard bundle no longer **serves** the older TLSRoute/TCPRoute/UDPRoute alpha versions. The Experimental bundle still serves some deprecated versions, so a manifest working there does not prove it works with Standard. ReferenceGrant is a useful counterexample to “Standard always means v1 only”: the v1.6.0 bundle serves both `v1` and `v1beta1`, with `v1beta1` as its storage version. The ReferenceGrant examples here retain the served beta version. New experimental X resources use `gateway.networking.x-k8s.io`. Experimental fields on established resources can still be in `gateway.networking.k8s.io`; the entire Experimental channel did not move to another group. Its compatibility guarantees differ from Standard, and admission policies protect channel/field boundaries. Review the published upgrade procedure instead of deleting shared CRDs or admission policies to force a change. ### v1.6 Release Context Gateway API v1.6.0 was published on **2026-06-29 UTC / 2026-06-30 KST**. TCPRoute and UDPRoute graduated to Standard `v1`. GRPCRoute and TLSRoute are also in the current Standard bundle. Choose the bundle/channel supported by the selected implementation, rather than equating the newest catalog version with compatibility. ## Comparison with Ingress API | Aspect | Ingress | Gateway API | |---|---|---| | Resource model | Ingress and IngressClass; listener/routing concerns largely combined | GatewayClass, Gateway and separate Route types | | Authorization | Kubernetes RBAC/admission can restrict ownership | RBAC/admission plus explicit attachment and reference handshakes | | HTTP routing | Standard HTTP routing | Standard HTTPRoute fields, with support levels for individual features | | TCP/UDP/gRPC | Controller-specific extensions beyond the Ingress API | Dedicated API types; actual support depends on controller/version | | TLS passthrough / traffic split / rewrites | Controller-specific configuration | Relevant Route/filter fields and implementation support requirements | | Cross-namespace references | Implementation-specific behavior | ReferenceGrant for backends/Secrets; allowedRoutes for Gateway attachment | | Portability | Reduced by differing annotation semantics | Improved by standard fields and conformance, with extensions still varying | ## Best Practices ### 1. Follow Role Separation ```yaml # Infrastructure team: Manage GatewayClass # Platform team: Manage Gateway # App team: Manage HTTPRoute ``` ### 2. Least Privilege ReferenceGrant ```yaml # Explicitly allow only required namespaces apiVersion: gateway.networking.k8s.io/v1beta1 kind: ReferenceGrant metadata: name: minimal-access namespace: backend spec: from: - group: gateway.networking.k8s.io kind: HTTPRoute namespace: frontend # Specific namespace only to: - group: "" kind: Service name: specific-service # Specific service only ``` ### 3. Gateway Separation ```yaml # Separate Gateway by environment # production-gateway, staging-gateway # Separate Gateway by protocol # http-gateway, grpc-gateway ``` ### 4. Monitoring Configuration ```yaml # Prometheus metrics collection (varies by implementation) # - Request count, latency, error rate # - Backend status # - TLS certificate expiry ``` --- ## References - [Gateway API Official Documentation](https://gateway-api.sigs.k8s.io/) - [Gateway API GitHub](https://github.com/kubernetes-sigs/gateway-api) - [Istio Gateway API Support](https://istio.io/latest/docs/tasks/traffic-management/ingress/gateway-api/) - [Cilium Gateway API](https://github.com/cilium/cilium/blob/v1.20.1/Documentation/network/servicemesh/gateway-api/gateway-api.rst) - [AWS Gateway API Controller](https://github.com/aws/aws-application-networking-k8s/tree/v2.1.3/docs) - [Envoy Gateway](https://gateway.envoyproxy.io/) - [Gateway API 1.6 versioning](https://github.com/kubernetes-sigs/gateway-api/blob/v1.6.0/site/content/en/docs/concepts/versioning.md) - [ReferenceGrant and attachment exceptions](https://github.com/kubernetes-sigs/gateway-api/blob/v1.6.0/site/content/en/reference/api-types/referencegrant.md) - [Envoy Gateway compatibility](https://github.com/envoyproxy/gateway/blob/v1.9.1/site/content/en/news/releases/matrix.md) - [NGINX Gateway Fabric 2.7 release](https://github.com/nginx/nginx-gateway-fabric/blob/v2.7.0/CHANGELOG.md) - [Contour 1.33.7 release](https://github.com/projectcontour/contour/releases/tag/v1.33.7) - [Community ingress-nginx retirement](https://kubernetes.io/blog/2025/11/11/ingress-nginx-retirement/) - [Legacy ingress-nginx redirect and rewrite behavior](https://github.com/kubernetes/ingress-nginx/blob/main/docs/user-guide/nginx-configuration/annotations.md) - [Legacy ingress-nginx redirect-code configuration](https://github.com/kubernetes/ingress-nginx/blob/main/docs/user-guide/nginx-configuration/configmap.md) ---------------------------------------- Source: https://www.atomai.click/kubernetes-docs/en/networking/05-cross-org-vpc-connectivity ---------------------------------------- # Cross-Org VPC Connectivity > **Original report timestamp**: September 1, 2026 > > **Content review**: September 12, 2026 This chapter compares five patterns for connecting accounts in **different AWS Organizations**, such as an existing environment and a separately governed GPU environment. The tables retain the measurements reported in the earlier document. This review checks AWS behavior and the arithmetic; it does not claim a new live deployment or independently reproduced benchmark. ## Table of Contents 1. [Why Cross-Org Connectivity](#why-cross-org-connectivity) 2. [Comparing the Five Options](#comparing-the-five-options) 3. [Reported Verification Results](#reported-verification-results) 4. [Latency Measurements (M1–M7)](#latency-measurements-m1m7) 5. [Operational Findings](#operational-findings) 6. [Architecture Selection by Requirement](#architecture-selection-by-requirement) 7. [Limitations and Next Checks](#limitations-and-next-checks) ## Why Cross-Org Connectivity Contractual ownership, acquisitions, independent governance, or isolation requirements can place GPU workloads and existing services in different Organizations. Organization structure should follow those requirements, rather than an assumption that a second Organization automatically improves GPU discounts, quotas or compliance. EC2 resource quotas are generally set for an **account and Region**; a separate account can provide that separation without requiring another Organization. Billing aggregation, negotiated discounts and duplicated governance also need review. An Organization boundary does not replace application authorization, network segmentation or audit controls. For EKS, distinguish ordinary IP access to data pipelines/inference APIs from GPU collective communication. A CPU-instance request/response benchmark does not establish NCCL, throughput or RDMA performance. **EFA OS-bypass traffic cannot cross VPCs or Availability Zones**; normal IP traffic from its ENA interface remains routable. ## Comparing the Five Options The PrivateLink and Lattice columns describe the **tested NLB-backed endpoint-service and HTTP-service patterns**. PrivateLink also has resource and service-network endpoint types; Lattice also has TCP resource configurations. They are not universally “NLB required” or “L7 only” products. | Aspect | ① TGW RAM Sharing | ② VPC Peering | ③ PrivateLink endpoint service | ④ TGW Peering | ⑤ VPC Lattice HTTP service | |---|---|---|---|---|---| | Mechanism | Share a TGW with the external account | Direct VPC pair | Consumer interface endpoint → provider NLB/service | Connect each owner's TGW | Associate services and client VPCs with a service network | | Address overlap | Direct routing needs an unambiguous address plan | Overlapping CIDRs cannot be peered | Service access can handle overlapping VPC CIDRs | Direct routing needs an unambiguous address plan | Service access can handle overlapping VPC CIDRs | | Connection model | Bidirectional IP routing when permitted | Bidirectional IP routing when permitted | Consumer initiates; responses can return on the connection | Bidirectional IP routing when permitted | Clients initiate requests to published services; reverse access needs its own configuration | | Routing setup | VPC routes plus TGW tables/associations | Routes on both sides; no transitive VPC peering | Endpoint/service permissions and network controls, rather than general VPC transit | Explicit static routes toward the peer plus VPC routes | Service/network associations and policies, rather than general VPC transit | | Control | TGW owner manages its TGW tables; consumers retain their VPC controls | Each VPC owner | Provider controls service permissions/targets; consumer controls its endpoints | Each TGW owner, with coordinated routes | Network/service owners and client-network controls | | Original reported provisioning time | TGW ~3 min plus acceptance | Under 1 min | Endpoint ~3 min | ~7 min | ~5 min | The provisioning times are observations from the original report, not SLAs or end-to-end delivery estimates. The routing row describes the two-TGW topology in this chapter; it does not assert unrestricted transit through arbitrary chains of peers. NAT or address redesign are additional approaches to overlap and require their own design. ## Reported Verification Results The original report states that all five patterns were established and traffic was exchanged across two Organizations. AWS documentation supports cross-account deployment of these patterns; a common Organization is not inherently required. However, IAM/SCP/sharing restrictions can block setup, and routes, security groups, NACLs, DNS and service authorization determine whether traffic works. Account IDs and acceptance alone are insufficient. ![The original cross-organization topology shows TCP_RR p50 values for peering, TGW and PrivateLink paths, and an HTTP keep-alive p50 for the Lattice HTTP-service path.](https://raw.githubusercontent.com/Atom-oh/kubernetes-docs/main/en/.gitbook/assets/en-networking-05-cross-org-vpc-connectivity-0.png) [View interactive diagram](https://www.atomai.click/kubernetes-docs/archmaps/en-networking-05-cross-org-vpc-connectivity-0.html) The figure preserves the original observations. Its Lattice value is **HTTP KA**, while the other displayed values are **TCP_RR**; they are not one directly comparable metric. The “GPU” label identifies the proposed environment, not a GPU benchmark. ## Latency Measurements (M1–M7) **Reported setup:** `ap-northeast-2`, matching ZoneId `apne2-az1` across accounts, `c7g.large`, and one EC2 responder with nginx returning a fixed HTTP 200. The report describes three ENIs with per-path subnets/return routes, five round-robin interleaved rounds, 1,500 persistent TCP_RR samples per path, 100 ICMP samples per path, and 275 HTTP keep-alive samples per path. The nginx description identifies the HTTP responder; the page does not identify the TCP_RR implementation or message sizes. Raw samples, software/kernel versions, timer boundaries and Linux return-path policy configuration are not linked here. Persistent connections aim to reduce repeated setup effects, but their timer boundaries cannot be independently checked from these tables. **All latency values below are milliseconds; TTL is a separate packet field.** TCP_RR and ICMP are request/response round-trip measures. HTTP KA includes application processing. The two measurement campaigns below must be interpreted separately. | ID | Path | ICMP p50 | TCP_RR p50 | RR p99 | RR sd | HTTP KA p50 | TTL | |---|---|---|---|---|---|---|---| | M1 | Same VPC → EC2 (baseline) | 0.121 | **0.049** | 0.062 | 0.007 | 0.087 | 127 | | M2 | ② VPC Peering → EC2 | 0.125 | **0.048** | 0.057 | 0.011 | 0.080 | 127 | | M3 | ① Shared TGW (RAM) → EC2 | 0.535 | **0.619** | 0.695 | 0.141 | 0.686 | 126 | | M4 | ④ TGW Peering (two TGWs) → EC2 | 0.912 | **0.599** | 0.855 | 0.133 | 0.488 | 125 | | M5 | ③ PrivateLink → NLB → EC2 | not measured | **0.961** | 1.084 | 0.035 | 0.711 | — | | M6 | ⑤ VPC Lattice → EC2 target | not measured | not measured for this HTTP service | — | — | **1.635** | — | | M7 | ② Peering → NLB → EC2 (NLB hop isolation) | not measured | **0.841** | 0.909 | 0.119 | 0.883 | — | ### Differences Between Reported Medians These are **differences of path medians**, not isolated one-way hop costs or measurements of an individual ENI/proxy component. | Observed path comparison | Difference | Δ TCP_RR p50 | Δ ICMP p50 | Δ HTTP KA p50 | |---|---|---|---|---| | Peering vs same-VPC baseline | M2 − M1 | -0.001 | +0.004 | -0.007 | | Shared TGW path vs peering | M3 − M2 | +0.571 | +0.410 | +0.606 | | Two-TGW path vs peering | M4 − M2 | +0.551 | +0.787 | +0.408 | | Peering with NLB vs direct peering | M7 − M2 | +0.793 | — | +0.803 | | PrivateLink/NLB vs peering/NLB | M5 − M7 | +0.120 | — | -0.172 | | Lattice HTTP service vs direct peering HTTP | M6 − M2 | — | — | +1.555 | - M2 is close to the same-VPC baseline, but the tables do not establish statistical equivalence or zero overhead. - The two-TGW path's TCP_RR median is lower than the single shared-TGW path's median. The data therefore do not support a universal “0.4–0.6 ms per TGW hop” or a linear hop-cost formula. - M5−M7 is **+0.120 ms for TCP_RR but −0.172 ms for HTTP KA**. It cannot be labeled a pure PrivateLink ENI cost. - The Lattice comparison is **HTTP +1.555 ms**, not TCP_RR. It describes this HTTP-service test, not every Lattice mode. - TTL does not reveal the path's hop count without the initial TTL and relevant network behavior. ### Separate Service-Fronted Campaign The original report also placed NLBs on each L3 path. This is a useful comparison for that service-exposure pattern, not a requirement for every production Peering/TGW deployment. | Configuration | TCP_RR p50 | HTTP KA p50 | |---|---|---| | ② Peering → NLB → EC2 | **0.622** | 0.648 | | ③ PrivateLink → NLB → EC2 | **0.658** | 0.845 | | ① Shared TGW → NLB → EC2 | **1.273** | 1.257 | | ④ TGW Peering → NLB → EC2 | **1.425** | 1.279 | | ⑤ Lattice HTTP service (no separate NLB in this test) | — | **1.680** | In this campaign, PrivateLink/NLB minus Peering/NLB is **+0.036 ms TCP_RR** and **+0.197 ms HTTP KA**. The shared-TGW and peered-TGW TCP_RR medians are respectively **1.93× and 2.17×** the PrivateLink median; the HTTP ratios are **1.49× and 1.51×**. These are latency ratios, not throughput multipliers or proof that the paths are equivalent. Lattice's HTTP median exceeds the shared-TGW/NLB and peered-TGW/NLB HTTP medians by **+0.423 ms and +0.401 ms**. Do not combine this campaign with the M1–M7 campaign to derive a component cost: even the Peering/NLB medians differ between runs. The original report additionally describes a discarded burstable-instance/NLB→ALB/fresh-curl pilot with p95 around **7 ms**, and first-flow increments of **0.6–1.6 ms**. These remain attributed observations without linked raw samples, not AWS guarantees. Measure connection establishment and steady-state behavior separately for the actual application. ## Operational Findings 1. **RAM external sharing:** external principals must be allowed and the outside-Organization account must accept the share invitation. The `CreateResourceShare` API's `allowExternalPrincipals` default is **true**; explicitly setting `--allow-external-principals` documents intent, but omitting that literal CLI flag is not universally a failure cause. Verify the effective share configuration and permissions. 2. **Shared TGW VPC attachment acceptance:** with `AutoAcceptSharedAttachments` disabled (the default), the TGW owner must accept the shared attachment. Enabling it changes that workflow. RAM share acceptance and TGW attachment acceptance are different steps. Consumers cannot modify the owner's TGW route tables, but still control their own VPC routes and security settings. 3. **TGW peering acceptance:** the accepter TGW owner accepts the pending peering request **in the accepter Region**, even for same-account peering. Use that request's `TransitGatewayAttachmentId`; do not confuse it with a TGW ID or VPC-attachment ID. A `NotFound` response does not establish a rule that the two sides require different IDs. The original report's roughly two-minute visibility delay is an observation, not a fixed wait guarantee. 4. **Peering routes:** direct TGW-to-TGW peering uses explicitly configured static routes, not BGP route propagation across the peering attachment. Configure the relevant TGW and VPC route tables in both directions. Automation can manage these static routes. 5. **Route priority:** longest-prefix matching comes first. A static route wins over a propagated route **for the same destination prefix**; a less-specific static route does not override a more-specific propagated route. 6. **Lattice target security groups:** for the documented VPC-association service path, use the Region/IP-family managed prefix lists (`com.amazonaws.REGION.vpc-lattice` and `com.amazonaws.REGION.ipv6.vpc-lattice`) on the actual target and health-check ports. The original `169.254.171.0/24` example is not a universal list definition; managed lists can include link-local or non-routable public addresses. Endpoint/resource-gateway paths have their own controls. IAM service authentication must also be configured; it is not enabled merely by associating a VPC. 7. **Cleanup ownership:** the original report describes GuardDuty-managed networking dependencies, IAM policy attachments and remaining Lattice resources affecting teardown. Inspect the actual dependency IDs and owning service before acting. Do not disable managed security controls or delete unrelated resources simply to force a VPC/role deletion. ## Architecture Selection by Requirement | Requirement | Candidate pattern | Checks that matter | |---|---|---| | Each Organization must retain its own TGW routing authority | ④ TGW Peering | Static-route coordination, address plan, throughput, availability, inspection and transfer charges | | A small set of inference/service endpoints should be exposed | ③ PrivateLink endpoint service | Supported protocol/model, endpoint acceptance, application auth, DNS, cost and actual payload/concurrency | | Service access across overlapping CIDRs | ③ PrivateLink or ⑤ Lattice | Service/resource scope; evaluate NAT/address redesign if broader IP routing is required | | Another account can use a centrally controlled hub | ① TGW RAM Sharing | External share policy, acceptance settings and the owner's TGW control model | | A small number of direct VPC pairs | ② VPC Peering | Non-overlapping CIDRs, pairwise route maintenance, quotas and data-transfer charges | | Managed HTTP service identity/discovery/governance is required | ⑤ VPC Lattice | Explicit IAM auth policies, signed requests, service connectivity and workload measurements | A hybrid of TGW peering and PrivateLink may fit independent network governance plus limited API exposure. The published latency tables do not establish that it is optimal for most GPU environments. Choose based on the required connectivity and controls, then measure the actual workload. ## Limitations and Next Checks The original report excludes measured Network Firewall inspection paths, cross-Region latency, and throughput/concurrency. It reports functional overlap checks without publishing overlap latency results. GPU collectives, EFA/RDMA, representative payload sizes, uncertainty estimates and full reproduction artifacts are also not established by this page. Keep the reported numbers as historical context. Before deployment, validate the target accounts' policies and supported connection model, required bidirectional routes or service access, failure behavior and the application's latency/throughput budget. This review performed no AWS provisioning or live benchmark. ## References - [Scalable multi-VPC networking whitepaper](https://docs.aws.amazon.com/whitepapers/latest/building-scalable-secure-multi-vpc-network-infrastructure/welcome.html) - [Cross-account TGW sharing](https://docs.aws.amazon.com/prescriptive-guidance/latest/integrate-third-party-services/architecture-3-1.html) - [Single or multiple Organizations](https://aws.amazon.com/blogs/architecture/choosing-between-single-or-multiple-organizations-in-aws-organizations/) - [RAM CreateResourceShare API](https://docs.aws.amazon.com/ram/latest/APIReference/API_CreateResourceShare.html) - [TGW acceptance options](https://docs.aws.amazon.com/AWSEC2/latest/APIReference/API_TransitGatewayRequestOptions.html) - [TGW peering acceptance](https://docs.aws.amazon.com/vpc/latest/tgw/tgw-peering-accept-reject.html) - [TGW routing and evaluation order](https://docs.aws.amazon.com/vpc/latest/tgw/how-transit-gateways-work.html) - [PrivateLink endpoint types](https://docs.aws.amazon.com/vpc/latest/privatelink/what-is-privatelink.html) - [Private NAT and overlapping networks](https://docs.aws.amazon.com/vpc/latest/userguide/nat-gateway-scenarios.html) - [Lattice security groups](https://docs.aws.amazon.com/vpc-lattice/latest/ug/security-groups.html) - [EC2 account/Region quotas](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-resource-limits.html) - [EFA limitations](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/efa.html) - [VPC Lattice guide](https://www.atomai.click/kubernetes-docs/llms/en/networking/02-vpc-lattice.md) - [Cross-Org quiz](https://www.atomai.click/kubernetes-docs/en/quizzes/networking/05-cross-org-vpc-connectivity-quiz) ---------------------------------------- Source: https://www.atomai.click/kubernetes-docs/en/networking/06-pod-network-benchmark ---------------------------------------- # Pod Network Benchmark — Same Node, Same AZ, Cross-AZ, and DNS ndots > **Recorded test environment**: Kubernetes 1.36 (Amazon EKS), Amazon VPC CNI v1.21.1, kube-proxy iptables mode > **Measurement date**: September 2, 2026 · **Content review**: September 12, 2026 This page preserves the September 2, 2026 benchmark report from `fsi-demo-cluster` (Seoul): Pod-to-Pod RTT, HTTP/gRPC latency, iperf3 throughput and DNS query counts. In these runs, crossing an AZ increased latency while the two inter-node paths reached similar throughput. That does not establish an AZ-independent bandwidth guarantee. Application traffic in Measurements 1–2 used Pod IPs directly; Measurement 3 models its cost, while Measurement 4 used the existing `kube-dns` ClusterIP. The reported 10-query DNS walk depends on this resolver, search list and response sequence, rather than applying to every EKS Pod. Historical versions and measurements are retained; the audit did not rerun the EKS benchmark or verify a billing invoice. ![Client Pod on node A (ap-northeast-2a) reaching a server Pod on the same node, on node B in the same AZ and on node C in ap-northeast-2b — RTT 0.040 / 0.339 / 0.544 ms, single flow 29.97 / 4.96 / 4.96 Gbps.](https://raw.githubusercontent.com/Atom-oh/kubernetes-docs/main/en/.gitbook/assets/en-networking-06-pod-network-benchmark-0.png) [🔍 View interactive diagram](https://www.atomai.click/kubernetes-docs/archmaps/en-networking-06-pod-network-benchmark-0.html) The diagram records this test topology. Its +0.21 ms is a difference between these runs, and its $4.47 is the historical decimal-GB cost model explained below, not a verified charge or a universal per-AZ cost. ## TL;DR — What we measured 1. **Reported RTT**: same node **0.040 ms** → same AZ **0.339 ms** → cross-AZ **0.544 ms** (ping, average of 200 probes). The observed differences are +0.21 ms between the two inter-node paths and +0.50 ms relative to the same node. 2. **HTTP p50 / p99** (fortio, 100 qps, 4 connections, keepalive, 60 s): 0.259 / 0.350 ms → 0.461 / 0.667 ms → 0.704 / 0.812 ms — the same ladder seen from the application. 3. **Reported bandwidth**: one TCP flow reached **4.96 Gbps** on both inter-node paths; eight flows reached **9.94 Gbps**, near the m5.xlarge 10 Gbps burst peak. This is consistent with the ordinary 5 Gbps single-flow limit outside a cluster placement group; other instance features and paths can have different limits. 4. **Same-node Pod to Pod**: **29.97 Gbps** with one flow (client process CPU 99.8 %, consistent with CPU pressure) and **48.15 Gbps** with eight. In this VPC CNI configuration, traffic traverses the Pods’ veth pairs and host network stack without using the physical NIC. 5. **Cost model**: the 180 s cross-AZ run sent **223.4 decimal GB**. The original payload-based model estimates **$4.47** at $0.01/GB at each end; it is not an invoice. No step-down toward the 1.25 Gbps baseline was observed within 180 s. 6. **Reported DNS**: this glibc Pod resolving `sts.ap-northeast-2.amazonaws.com` under `ndots:5` sent **10 queries** (8 NXDOMAIN), warm median **3.78 ms**. A trailing dot gave **2 queries** / 0.80 ms; `ndots:1` gave 2 / 0.54 ms. These timings are samples, not guarantees. 7. **New connections**: disabling keepalive changed p50 from 0.259 → 0.664, 0.461 → 1.079 and 0.704 → **1.517 ms**. TCP establishment contributes to this increase, but the test did not isolate handshake, socket and application costs. ## Test environment | Item | Value | |---|---| | Cluster | Amazon EKS `fsi-demo-cluster`, ap-northeast-2 (Seoul), control plane `v1.36.2-eks-bca9cf6`, two AZs used (2a, 2b) | | Nodes | **3 × m5.xlarge** launched fresh by the Karpenter `system` NodePool for this test — a client node in 2a, a server node in 2a, a server node in 2b. 4 vCPU, Intel Xeon Platinum 8175M @ 2.50GHz | | Node OS | Amazon Linux 2023.12.20260817, kernel `6.18.41-94.142.amzn2023.x86_64`, containerd 2.2.5, kubelet v1.36.3-eks-cb19647 | | CNI | Amazon VPC CNI `v1.21.1-eksbuild.8` (+ network-policy-agent v1.3.4); `ENABLE_PREFIX_DELEGATION=false`, `ENABLE_POD_ENI=false`, `AWS_VPC_K8S_CNI_EXTERNALSNAT=false`, `NETWORK_POLICY_ENFORCING_MODE=standard`, `WARM_ENI_TARGET=1`, `WARM_IP_TARGET=3` | | kube-proxy | `v1.35.3-eksbuild.5`, `mode: "iptables"` | | CoreDNS | `v1.14.2-eksbuild.4`, 2 replicas — one per AZ (`10.0.2.106` / 2a, `10.0.3.14` / 2b); Service `kube-dns` ClusterIP `172.20.0.10`; Corefile `kubernetes cluster.local … { pods insecure }`, `forward . /etc/resolv.conf`, `cache 30`, `loadbalance`; **no NodeLocal DNSCache**, no `autopath` plugin | | Pod resolv.conf (default) | `search bench-net.svc.cluster.local svc.cluster.local cluster.local ap-northeast-2.compute.internal` / `nameserver 172.20.0.10` / `options ndots:5` | | Pod NIC | eth0 MTU **9001** (jumbo frames), TCP congestion control `cubic`, iperf3 `tcp_mss_default: 8949` | | EC2 network spec | m5.xlarge "Up to 10 Gigabit" — baseline **1.25 Gbps**, peak **10 Gbps**, 4 vCPU (for comparison: m5.large baseline 0.75 Gbps, peak 10 Gbps, 2 vCPU). Verified with `aws ec2 describe-instance-types`; ENA required | | Pricing | usagetype `APN2-DataTransfer-Regional-Bytes`, "Regional Data Transfer - in/out/between AZs or when using public IP or Elastic IP addresses", **$0.01/GB** (`aws pricing get-products --region us-east-1`, queried 2026-09) | | Tools | `nicolaka/netshoot:v0.14` — iperf **3.19**, fortio **1.69.5**, iputils ping 20250605, tcpdump 4.99.5; DNS client `python:3.12-slim` (Debian 13, **glibc 2.41**, Python 3.12.14) | | Test window | 2026-09-02 07:58–08:40 UTC (first Pod at 07:58:22Z, DNS Pods at 08:16:24Z) | AWS describes burst bandwidth as best effort even while network I/O credits remain; incoming and outgoing traffic have separate credit buckets. A new instance starts with maximum credits, but peak availability and burst duration vary. This 180 s run establishes only that no baseline step-down was observed during its window. The recorded m5.xlarge 1.25 Gbps baseline / 10 Gbps peak also appear in the official [M5 network specifications](https://docs.aws.amazon.com/ec2/latest/instancetypes/gp.html); current credit behavior is described in the [EC2 bandwidth guide](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-instance-network-bandwidth.html). Fixture placement during the run: | Pod | IP | Node | Zone | Role / requests | |---|---|---|---|---| | `cli` | 10.0.2.109 | ip-10-0-2-128 (nodeclaim `system-76r87`) | ap-northeast-2a | client; 2500m / 1Gi | | `srv-same` | 10.0.2.72 | ip-10-0-2-128 — same node as `cli` (required podAffinity) | ap-northeast-2a | server; 200m / 256Mi | | `srv-a` | 10.0.2.37 | ip-10-0-2-20 (nodeclaim `system-ksrbg`, podAntiAffinity to `cli`) | ap-northeast-2a | server; 2800m / 1Gi | | `srv-b` | 10.0.3.65 | ip-10-0-3-32 (nodeclaim `system-svdvk`) | ap-northeast-2b | server; 2500m / 1Gi | | `dns-default` | 10.0.2.5 | ip-10-0-2-20 (podAffinity to `srv-a`) | ap-northeast-2a | glibc resolver, default `ndots:5` | | `dns-ndots1` | 10.0.2.143 | ip-10-0-2-20 | ap-northeast-2a | glibc resolver, `dnsConfig.options ndots=1` | The server Pods run `sh -c "iperf3 -s -p 5201 & exec fortio server -http-port 8080 -grpc-port 8079 -tcp-port 8078"`, and every bench Pod carries `karpenter.sh/do-not-disrupt: "true"`. `srv-a` was first requested as m5.large / 1500m, but Karpenter reported `no instance type has enough resources` — DaemonSet overhead takes 821m of an m5.large's 1930m allocatable — so it was changed to m5.xlarge / 2800m. ### Fixture manifest The following is the recorded fixture, with its historical selectors, requests, images, commands and annotations preserved. It contains no application Service objects. Its selectors do not themselves guarantee new or isolated nodes. Before a new run, adapt a copy to an approved test NodePool, available AZs and resource budget; do not deploy it blindly to a shared `system` pool. Record image digests and tool versions: mutable image tags and netshoot’s build-time tool downloads do not guarantee the original binaries. The revised procedure below is separate from this historical artifact. ```yaml apiVersion: v1 kind: Namespace metadata: name: bench-net labels: bench: net --- # client — fresh m5.xlarge in ap-northeast-2a apiVersion: v1 kind: Pod metadata: name: cli namespace: bench-net labels: { app: cli, role: client } annotations: { karpenter.sh/do-not-disrupt: "true" } spec: nodeSelector: topology.kubernetes.io/zone: ap-northeast-2a node.kubernetes.io/instance-type: m5.xlarge karpenter.sh/nodepool: system terminationGracePeriodSeconds: 5 containers: - name: netshoot image: nicolaka/netshoot:v0.14 command: ["sleep", "infinity"] resources: requests: { cpu: "2500m", memory: "1Gi" } --- # same-node — co-located with cli through required podAffinity apiVersion: v1 kind: Pod metadata: name: srv-same namespace: bench-net labels: { app: srv-same, role: server, zone: a } annotations: { karpenter.sh/do-not-disrupt: "true" } spec: affinity: podAffinity: requiredDuringSchedulingIgnoredDuringExecution: - labelSelector: { matchLabels: { app: cli } } topologyKey: kubernetes.io/hostname terminationGracePeriodSeconds: 5 containers: - name: netshoot image: nicolaka/netshoot:v0.14 command: ["sh", "-c", "iperf3 -s -p 5201 & exec fortio server -http-port 8080 -grpc-port 8079 -tcp-port 8078"] ports: [{ containerPort: 8080 }, { containerPort: 5201 }] resources: requests: { cpu: "200m", memory: "256Mi" } --- # same-AZ — same AZ as cli, different node (podAntiAffinity). m5.large did not fit because of DaemonSet overhead, hence m5.xlarge apiVersion: v1 kind: Pod metadata: name: srv-a namespace: bench-net labels: { app: srv-a, role: server, zone: a } annotations: { karpenter.sh/do-not-disrupt: "true" } spec: nodeSelector: topology.kubernetes.io/zone: ap-northeast-2a node.kubernetes.io/instance-type: m5.xlarge karpenter.sh/nodepool: system affinity: podAntiAffinity: requiredDuringSchedulingIgnoredDuringExecution: - labelSelector: { matchLabels: { app: cli } } topologyKey: kubernetes.io/hostname terminationGracePeriodSeconds: 5 containers: - name: netshoot image: nicolaka/netshoot:v0.14 command: ["sh", "-c", "iperf3 -s -p 5201 & exec fortio server -http-port 8080 -grpc-port 8079 -tcp-port 8078"] ports: [{ containerPort: 8080 }, { containerPort: 5201 }] resources: requests: { cpu: "2800m", memory: "1Gi" } --- # cross-AZ — fresh m5.xlarge in ap-northeast-2b apiVersion: v1 kind: Pod metadata: name: srv-b namespace: bench-net labels: { app: srv-b, role: server, zone: b } annotations: { karpenter.sh/do-not-disrupt: "true" } spec: nodeSelector: topology.kubernetes.io/zone: ap-northeast-2b node.kubernetes.io/instance-type: m5.xlarge karpenter.sh/nodepool: system terminationGracePeriodSeconds: 5 containers: - name: netshoot image: nicolaka/netshoot:v0.14 command: ["sh", "-c", "iperf3 -s -p 5201 & exec fortio server -http-port 8080 -grpc-port 8079 -tcp-port 8078"] ports: [{ containerPort: 8080 }, { containerPort: 5201 }] resources: requests: { cpu: "2500m", memory: "1Gi" } ``` The two DNS Pods were reported on the same node as `srv-a`. The `app` image was reported as Debian 13 / glibc 2.41; musl and other resolvers were not measured. `sniffer` shares its Pod network namespace, so it can capture matching application DNS packets if the cluster permits packet capture. To produce both DNS objects from the example below, duplicate it: change the second name and `app` label to `dns-ndots1` and uncomment `dnsConfig` only in that second object. Keep both images identical and pin their recorded digests in a new run. ```yaml apiVersion: v1 kind: Pod metadata: name: dns-default # the second Pod is name: dns-ndots1 plus the dnsConfig block below namespace: bench-net labels: { app: dns-default, role: dns } annotations: { karpenter.sh/do-not-disrupt: "true" } spec: affinity: podAffinity: requiredDuringSchedulingIgnoredDuringExecution: - labelSelector: { matchLabels: { app: srv-a } } topologyKey: kubernetes.io/hostname # present only in dns-ndots1: # dnsConfig: # options: # - name: ndots # value: "1" terminationGracePeriodSeconds: 5 containers: - name: app image: python:3.12-slim command: ["sleep", "infinity"] resources: { requests: { cpu: "50m", memory: "64Mi" } } - name: sniffer image: nicolaka/netshoot:v0.14 command: ["sleep", "infinity"] resources: { requests: { cpu: "50m", memory: "64Mi" } } ``` ## Measurement 1 — RTT and HTTP latency: same node → same AZ → cross-AZ ICMP probes (`ping -c 200 -i 0.05 -q`) characterize the idle path, including endpoint kernel processing and scheduling. The same paths were then exercised with HTTP/1.1 and gRPC. A single fresh-connection `curl` connect / total time is listed for reference. | Path | RTT min / **avg** / max / mdev (ms) | Loss | curl, 1 cold request: connect / total | |---|---|---|---| | same node → 10.0.2.72 | 0.021 / **0.040** / 0.089 / 0.007 | 0/200 | 0.194 ms / 0.497 ms | | same AZ → 10.0.2.37 | 0.300 / **0.339** / 0.450 / 0.017 | 0/200 | 0.497 ms / 2.333 ms | | cross-AZ → 10.0.3.65 | 0.504 / **0.544** / 0.625 / 0.015 | 0/200 | 0.694 ms / 4.038 ms | Observed differences: same AZ − same node = +0.30 ms, cross-AZ − same AZ = **+0.21 ms**, cross-AZ − same node = +0.50 ms. The reported mdev values are at most 0.017 ms in this sample. curl’s `time_total` measures the transfer operation, not process startup; a single observation is insufficient to characterize its distribution. See the [curl timing definitions](https://curl.se/docs/manpage.html). ### HTTP/1.1 — 100 qps, 4 connections, keepalive, 60 s (6,000 requests), ms | Path | avg | **p50** | p90 | p99 | p99.9 | max | min | |---|---|---|---|---|---|---|---| | same node | 0.260 | **0.259** | 0.299 | 0.350 | 1.267 | 2.080 | 0.111 | | same AZ | 0.468 | **0.461** | 0.560 | 0.667 | 0.783 | 2.823 | 0.336 | | cross-AZ | 0.706 | **0.704** | 0.782 | 0.812 | 1.150 | 4.581 | 0.551 | ### gRPC ping — 100 qps, 4 connections, 30 s (3,000 requests), ms | Path | avg | **p50** | p90 | p99 | p99.9 | max | min | |---|---|---|---|---|---|---|---| | same node | 0.410 | **0.397** | 0.449 | 0.869 | 1.187 | 1.314 | 0.241 | | same AZ | 0.601 | **0.592** | 0.687 | 0.889 | 1.052 | 1.105 | 0.448 | | cross-AZ | 0.878 | **0.865** | 0.967 | 1.209 | 2.582 | 2.826 | 0.692 | The report describes an approximately 75-byte HTTP echo body with an empty request payload and zero errors in all runs. Check each protocol’s result counters separately: HTTP status 200, gRPC Ping results and gRPC health-check `SERVING` are different responses; `-grpc -ping` selects Ping rather than the default health-check workload. **How to read it.** HTTP p50 exceeds the ping mean by roughly 0.22 / 0.12 / 0.16 ms, but subtracting different statistics from different protocols does not isolate user-space overhead. The observed HTTP p50 steps are +0.202 ms and +0.243 ms; they are not constant per-node or per-AZ costs. gRPC p50 exceeds HTTP p50 by 0.138 / 0.131 / 0.161 ms in these implementations, without proving how much comes from HTTP/2, serialization, scheduling or client/server code. HTTP p99 is 0.350 → 0.667 → 0.812 ms; gRPC p99.9 is 1.187 → 1.052 → **2.582 ms**. These are reported distributions for one run per cell. > **Comparison with the mesh benchmark.** The [Istio sidecar vs ambient report](https://www.atomai.click/kubernetes-docs/llms/en/service-mesh/istio/comparison/03-sidecar-vs-ambient.md) records a **+1.29 ms** p50 difference (2.11 − 0.82 ms) for its whole sidecar scenario, not an isolated single proxy. Its Graviton hardware, 200 qps, 16 connections and Fortio 1.69.4 differ from this M5 / 100 qps / four-connection test. The numbers cannot be added or ranked as universal “mesh hop” versus “AZ hop” costs. ### The cost of a new connection — keepalive=false, 100 qps, 4 connections, 30 s (3,000 requests), ms What happens to latency when every request opens a fresh TCP connection (fortio `-keepalive=false`)? | Path | avg | **p50** | p90 | p99 | p99.9 | max | min | vs keepalive p50 | |---|---|---|---|---|---|---|---|---| | same node | 0.672 | **0.664** | 0.782 | 0.957 | 1.253 | 1.306 | 0.364 | **+0.405 ms** | | same AZ | 1.066 | **1.079** | 1.185 | 1.369 | 1.582 | 1.795 | 0.769 | **+0.618 ms** | | cross-AZ | 1.530 | **1.517** | 1.678 | 1.796 | 1.981 | 2.009 | 1.300 | **+0.813 ms** | The observed increases are **+0.405 / +0.618 / +0.813 ms**. A new TCP connection adds establishment work, but these measurements do not prove a fixed “one RTT + 0.3 ms” decomposition. Connection reuse is a useful optimization to measure with the real workload; it is not a prerequisite for correctness across AZs. Connection churn can also increase TIME_WAIT state on the active-closing endpoint, depending on how connections close; sockets and TIME_WAIT were not measured here. ### Maximum qps from a fixed connection pool — latency is throughput (closed loop, 16 connections, 20 s) With `-qps 0` (unlimited, closed loop), the maximum request rate that 16 connections can sustain turns the latency difference into a throughput difference. | Path | Requests | **Achieved qps** | avg ms | p50 | p90 | p99 | p99.9 | max | |---|---|---|---|---|---|---|---|---| | same node | 899,827 | **44,991** | 0.355 | 0.249 | 0.733 | 1.695 | 3.389 | 13.593 | | same AZ | 770,156 | **38,507** | 0.415 | 0.396 | 0.537 | 0.728 | 1.147 | 4.502 | | cross-AZ | 512,060 | **25,602** | 0.624 | 0.597 | 0.770 | 0.949 | 1.293 | 4.725 | For a steady closed loop with about 16 requests in flight and little client think time, Little’s law gives throughput ≈ concurrency / mean latency: 16 / 0.000355 = 45,070 (reported 44,991), 16 / 0.000415 = 38,554 (38,507), and 16 / 0.000624 = 25,641 (25,602). The reported cross-AZ rate is **33.5 % lower** than same-AZ in this test. This relationship does not identify the exclusive cause of latency or predict a universal AZ penalty. Shared CPU contention is a plausible explanation for the worse same-node tail, but no profiling here establishes that cause. ## Measurement 2 — Throughput: the 5 Gbps single-flow cap and the 10 Gbps instance cap iperf3 3.19, TCP, 20 s per run, `-J`, client `cli`. The CPU columns are iperf3's own per-process figures, where 100 % = one vCPU. | Path | Flows (-P) | Send Gbps | Recv Gbps | Retransmits | Bytes sent | Client CPU | Server CPU | Sender TCP mean RTT (stream 1) | max snd_cwnd | |---|---|---|---|---|---|---|---|---|---| | same node (cli→srv-same) | 1 | **29.97** | 29.97 | 13 | 74,921,541,632 | **99.8 %** | 80.9 % | 34 µs | 1,861,392 B | | same node | 8 | **48.15** | 48.08 | 14,567 | 120,375,083,008 | 179.0 % | 186.9 % | 201 µs / 767 µs (streams 1, 2) | 5,888,442 B | | same AZ (cli→srv-a, 2a→2a) | 1 | **4.96** | 4.96 | 4 | 12,411,731,968 | 19.5 % | 15.4 % | **5,641 µs** | 4,349,214 B | | same AZ | 8 | **9.94** | 9.93 | 5,874 | 24,846,139,392 | 36.3 % | 159.3 % | 2,720 µs / 1,626 µs | 1,163,370 B | | cross-AZ (cli→srv-b, 2a→2b) | 1 | **4.96** | 4.96 | 2 | 12,411,994,112 | 20.0 % | 22.5 % | **5,420 µs** | 4,304,469 B | | cross-AZ | 8 | **9.94** | 9.93 | 5,979 | 24,845,090,816 | 36.7 % | 138.2 % | 3,671 µs / 3,237 µs | 1,226,013 B | Four things to read here. 1. **Same-node traffic bypassed the physical NIC.** The single-flow result was 29.97 Gbps with 99.8 % client process CPU; eight flows reached 48.15 Gbps. Host routing, both Pods’ veth paths, kernel processing and CPU scheduling still matter. This is a network benchmark with evidence of CPU pressure, not a measurement of pure memory-copy speed. 2. **Both inter-node single-flow runs reached 4.96 Gbps.** This is consistent with the ordinary 5 Gbps limit outside a cluster placement group. AWS also documents up to 10 Gbps for flows within a cluster placement group, and up to 25 Gbps with eligible ENA Express paths in the same AZ. Low iperf3 process CPU alone does not rule out all host/network processing limits. 3. **Both eight-flow runs reached 9.94 Gbps.** This supports similar observed throughput for these paths during this window. Retransmits existed even at one flow (4 / 2 for inter-node paths) and increased at eight flows (5,874 / 5,979); retransmission counts alone do not identify ENA shaping or the location of loss. The ENA allowance counters were not collected. 4. **Loaded TCP RTT was higher than idle ICMP RTT.** The single-flow sender reported about **5.6 / 5.4 ms** and a roughly 4.3 MB congestion window, compared with idle ping means of 0.34 / 0.54 ms. Queueing is a candidate explanation, but protocol, sampling and load differ. Neither the queue location nor a guaranteed extra 5 ms for every multiplexed RPC was measured. The reported MSS 8949 is consistent with a 9001-byte MTU and the observed IPv4/TCP overhead; effective MSS also depends on headers and path MTU. The bytes-sent column below is application transfer volume, not independently verified billable usage. > For this ordinary EC2 path, parallel flows used more of the instance’s available burst bandwidth than one flow. Increasing parallelism also changes CPU load, congestion and cost. Check the actual instance/path limits before changing Kafka fetchers or transfer concurrency; the measurements do not support either “every connection is capped at 5 Gbps” or “one AZ always doubles bandwidth.” ### The 3-minute sustained run and burst credits The reported m5.xlarge baseline is 1.25 Gbps, with a best-effort peak up to 10 Gbps. The historical four-flow cross-AZ test ran for 180 s at 10 s intervals (`iperf3 -c 10.0.3.65 -p 5201 -t 180 -P 4 -i 10 -J`). That IP belongs to the recorded fixture; obtain current Pod IPs before any new test. | Item | Value | |---|---| | Gbps per 10 s interval (18 intervals) | 9.94, 9.93 ×12, 9.92, 9.93 ×4 — **min 9.92, max 9.94** | | Total sent | 223,376,179,200 B = **223.4 GB** in 180.0 s (9.93 Gbps) | | Retransmits | 44,842 (≈ 249/s; 2,273–2,669 per 10 s interval) | | CPU | client 30.7 % (system 30.1 %), server 54.2 % (system 52.2 %) | **No drop toward 1.25 Gbps was observed within 180 s.** This is not evidence of unlimited credits or guaranteed sustained peak bandwidth. AWS documents variable, best-effort bursts and throttling toward baseline when credits run out. Size long backups and rebalances using the applicable baseline and measured workload requirements, rather than extrapolating this short run. ## Measurement 3 — A model of cross-AZ data-transfer cost This section preserves the report’s cost arithmetic as an estimate. The benchmark supplied payload-byte counts and a public list price, not a Cost and Usage Report (CUR) or invoice. For direct EC2 private-IP transfers between AZs in the same Region, the [EC2 pricing page](https://aws.amazon.com/ec2/pricing/on-demand/) documents $0.01/GB at each end. The reported public Pricing API item was `APN2-DataTransfer-Regional-Bytes`, **$0.0100000000 USD/GB**. `get-products` returns catalog pricing, not an account-specific paid rate. One payload direction can incur both sending-end “out” and receiving-end “in” charges; it does not require an equally large reverse transfer. Other AWS service paths can have different charging rules. | Scenario | Payload volume in the historical decimal-GB model | Estimated cost (model GB × $0.01 × 2) | |---|---|---| | The 180 s run (measured payload; estimated cost) | 223.4 GB | 223.4 × $0.01 ≈ **$2.23 at each end, $4.47 total** | | Cross-AZ iperf3 transfers in Measurement 2 (12.41 + 24.85 + 223.38 GB) | 260.6 GB | ≈ $2.61 at each end, **≈ $5.21 total** (other traffic excluded) | | An average of 1 Gbps crossing AZs for 30 days (**assumption**) | 0.125 GB/s × 86,400 s × 30 days = 324,000 GB ≈ **324 TB** | 324,000 × $0.02 ≈ **$6,480 / month** | | An RF3 StatefulSet spread over 3 AZs with 100 MiB/s of leader ingest (**assumption**, replication traffic only) | two followers, each in another AZ → 2 × 100 MiB/s = 209,715,200 B/s × 2,592,000 s ≈ 543,600 GB ≈ **544 TB / month** | 543,600 × $0.02 ≈ **$10,870 / month** | All four cost rows use the original **decimal conversion, 1 GB = 10⁹ payload bytes**, as a modeling assumption. This audit did not establish that EC2’s billed usage quantity equals that conversion. The exact raw totals are 223,376,179,200 B for the sustained run and 260,633,264,128 B for all three cross-AZ iperf3 runs. Reconcile actual metered units, rates, both charged endpoints, protocol overhead/retransmissions and applicable credits or discounts with the [CUR data-transfer records](https://docs.aws.amazon.com/cur/latest/userguide/cur-data-transfers-charges.html). The last two rows additionally assume continuous traffic for 30 days; replication volume excludes producer/consumer traffic. **$4.47 and $5.21 are estimates, not amounts proven to have been spent.** **What an operator should do.** - **Prefer suitable local endpoints where supported.** Current Kubernetes documentation uses `Service.spec.trafficDistribution: PreferSameZone`; `PreferClose` is its deprecated alias. This is a preference with fallback, not a strict zone restriction. Check the API server, kube-proxy version and feature support; this recorded cluster used a 1.35 kube-proxy with a 1.36 control plane. Neither this preference nor an application Service path was measured, and it does not affect direct Pod-IP traffic. - **Balance locality with fault tolerance.** Zone-aware reads or clients can reduce avoidable transfer, but moving all RF3 replicas into one AZ sacrifices AZ failure protection. Retain the required replica/failover design; see [Zonal Cluster Operations](https://www.atomai.click/kubernetes-docs/llms/en/ops/15-zonal-operations-guide.md). - **Measure both billed endpoints.** Track source/destination AZs and metered volume for backups, rebalances and replays. Sum the relevant “in” and “out” usage records, which can belong to different accounts; do not infer a final bill from payload bytes alone. ## Measurement 4 — DNS: the query amplification of ndots:5 ![One glibc lookup under ndots:5 walks the four search suffixes with A+AAAA pairs (8 NXDOMAIN, 10 queries) before the absolute name answers, versus a trailing-dot lookup that ends in 2 queries.](https://raw.githubusercontent.com/Atom-oh/kubernetes-docs/main/en/.gitbook/assets/en-networking-06-pod-network-benchmark-1.png) [🔍 View interactive diagram](https://www.atomai.click/kubernetes-docs/archmaps/en-networking-06-pod-network-benchmark-1.html) This diagram illustrates the recorded glibc search sequence. Its 4.37 ms ends at the reported A response, not the full `getaddrinfo` call; the upstream arrows are explanatory, since the Pod capture did not observe that leg. “cache 30” is a TTL ceiling, not a guaranteed 30-second hit. The recorded Pod had four search domains and `ndots:5`; that configuration is not universal across EKS, DNS policies, operating systems or node settings. For this glibc `AF_UNSPEC` lookup, A and AAAA were requested for each candidate, and the four search candidates returned NXDOMAIN before the absolute STS name succeeded. A/AAAA concurrency and query counts can change with resolver options, address family, early success, retries and TCP fallback. The historical capture used `tcpdump -i eth0 -nn udp port 53`, which observes UDP DNS only. The report describes one first-process lookup followed by 20 timed repeats; a first-process call does not prove that CoreDNS or upstream caches were cold. See [Kubernetes Pod DNS configuration](https://kubernetes.io/docs/concepts/services-networking/dns-pod-service/). ### Queries sent for one resolution and warm latency (20 repeats), ms | Pod / ndots | Name (dots) | Queries sent | NXDOMAIN answers | warm min | **median** | p90 | max | |---|---|---|---|---|---|---|---| | default / 5 | `kubernetes.default` (1) | 4 | 2 | 0.87 | **1.71** | 1.97 | 2.61 | | default / 5 | `kubernetes.default.svc.cluster.local` (4) | **10** | 8 | 1.53 | **3.63** | 4.45 | 6.41 | | default / 5 | `kubernetes.default.svc.cluster.local.` (trailing dot) | 2 | 0 | 0.33 | **0.46** | 1.09 | 1.58 | | default / 5 | `sts.ap-northeast-2.amazonaws.com` (3) | **10** | 8 | 3.08 | **3.78** | 4.66 | 4.84 | | default / 5 | `sts.ap-northeast-2.amazonaws.com.` (trailing dot) | 2 | 0 | 0.42 | **0.80** | 1.25 | 2.17 | | default / 5 | `www.amazon.com` (2) | **10** | 8 | 2.51 | **3.46** | 3.74 | 5.86 | | ndots1 / 1 | `kubernetes.default` (1) | **6** | 4 | 1.16 | **2.04** | 2.80 | 4.54 | | ndots1 / 1 | `kubernetes.default.svc.cluster.local` (4) | 2 | 0 | 0.35 | **0.97** | 1.08 | 1.35 | | ndots1 / 1 | `kubernetes.default.svc.cluster.local.` | 2 | 0 | 0.34 | **0.40** | 0.97 | 1.17 | | ndots1 / 1 | `sts.ap-northeast-2.amazonaws.com` (3) | 2 | 0 | 0.45 | **0.54** | 1.22 | 1.42 | | ndots1 / 1 | `sts.ap-northeast-2.amazonaws.com.` | 2 | 0 | 0.47 | **0.75** | 1.20 | 1.30 | | ndots1 / 1 | `www.amazon.com` (2) | 2 | 0 | 0.63 | **0.90** | 1.27 | 2.74 | The reported first-process times were: default / `sts` 6.22 ms, default / `sts.` 2.87 ms, default / `www.amazon.com` 9.58 ms, default / `kubernetes.default.svc.cluster.local` 7.40 ms, ndots1 / `kubernetes.default` 10.52 ms, and ndots1 / `sts` 2.84 ms. These include resolver initialization work and are distinct from the wire timelines below. **How to read it.** In these samples, the external names and the cluster FQDN without a trailing dot produced **10 queries / 8 NXDOMAIN**. The trailing-dot STS median fell 3.78 → 0.80 ms, and the cluster FQDN median 3.63 → 0.46 ms. `kubernetes.default` succeeded on the second search candidate, so it needed only four queries: search expansion does not invariably consume the whole list. [CoreDNS cache](https://coredns.io/plugins/cache/) can cache negative answers, but `cache 30` sets a maximum TTL; response TTLs and minimums still apply, and each replica has its own cache. The Kubernetes plugin’s default TTL is 5 s unless configured otherwise. Sequential queries remain even on cache hits, but this capture does not prove that all warm lookups avoided upstream traffic. ### The walk itself — one cold resolution of `sts.ap-northeast-2.amazonaws.com` (ndots:5, tcpdump, ms from the first packet) | t (ms) | Candidate sent to 172.20.0.10 (A + AAAA in parallel) | Answer | |---|---|---| | 0.00 | `sts.ap-northeast-2.amazonaws.com.bench-net.svc.cluster.local.` | NXDomain (authoritative, CoreDNS kubernetes plugin) at 0.92 / 1.14 | | 1.21 | `sts.ap-northeast-2.amazonaws.com.svc.cluster.local.` | NXDomain at 2.01 / 2.26 | | 2.32 | `sts.ap-northeast-2.amazonaws.com.cluster.local.` | NXDomain at 3.15 / 3.41 | | 3.47 | `sts.ap-northeast-2.amazonaws.com.ap-northeast-2.compute.internal.` | NXDomain (forwarded to the VPC resolver — non-authoritative) at 3.68 / 3.93 | | 3.99 | `sts.ap-northeast-2.amazonaws.com.` | **A 10.0.3.84, A 10.0.2.129** at 4.37 (AAAA: no data) | The table records 10 queries and 8 NXDOMAIN. Its **4.37 ms** runs from the first query to the reported A answer; the AAAA completion time is not listed, so this is not the full 6.22 ms first-process call. Candidate RTTs are not uniformly 0.8–1.1 ms: the fourth pair took 0.21 / 0.46 ms and the final A took 0.38 ms. kube-proxy iptables chooses an endpoint for a new conntrack flow, rather than independently for every packet; A/AAAA requests can share that flow. With two equally selected endpoints, half of new flows would be an illustrative expectation, **not a measured fraction of cross-AZ DNS queries**. The Pod capture showing Service VIP `172.20.0.10` cannot identify the selected backend AZ. The original report attributes the two STS private addresses to interface endpoint ENIs, and records a 2.2 ms forwarded candidate / 5.6 ms wire walk for the cluster FQDN versus 0.4–0.5 ms with its trailing dot; these are separate observations. ### What `ndots:1` does — and its side effect - **External names in these samples**: 10 → **2 queries**, medians about 3.5–3.8 → **0.5–0.9 ms**. This gain depends on the names and resolver behavior. - **Short names can incur an extra failed attempt.** Here `kubernetes.default` has one dot, meeting `ndots:1`, so glibc first tried `kubernetes.default.`. CoreDNS forwarded it and received NXDOMAIN after a reported 1.6 ms, then tried the namespace suffix and finally succeeded with `svc.cluster.local` at `172.20.0.1`: six queries, four NXDOMAIN and a 2.04 ms median versus 1.71 ms. This can disclose internal-looking names upstream. Test every application’s service naming before changing `ndots`; full Service names reduce this ambiguity. - **A trailing dot makes the resolver name absolute**, avoiding search expansion in this resolver. It does not guarantee two wire queries or a fixed latency under retries, different address-family settings or caching. ### Amplification arithmetic (derived) Under the **same response pattern**, with no application DNS cache and one lookup per request, 1,000 resolutions/s × 10 queries gives 10,000 queries/s, versus 2,000 for the two-query form. Of those 10,000, 8,000 (80 %) would receive NXDOMAIN. This is a query-count model, not a measurement of CoreDNS CPU or the cross-AZ fraction. The observed median differences are 3.78 − 0.80 = 2.98 ms for STS and 3.63 − 0.46 = 3.17 ms for the cluster FQDN; they are not fixed per-request penalties. **Options to test with the actual application:** - Use absolute DNS names where the client supports them. Do not blindly append a dot to HTTPS or AWS SDK endpoint URLs: Host handling, SNI, certificate verification and request signing must still work. - Evaluate `dnsConfig: {options: [{name: ndots, value: "1"}]}` together with short-name behavior and application DNS caching. - Evaluate [NodeLocal DNSCache](https://kubernetes.io/docs/tasks/administer-cluster/nodelocaldns/) where appropriate. Cache hits stay local; misses can still go upstream. Current [EKS Auto Mode](https://docs.aws.amazon.com/eks/latest/userguide/auto-networking.html) already runs node-local CoreDNS as a system service; a pure Auto Mode cluster does not need a CoreDNS Deployment, while non-Auto nodes in a mixed cluster still do. - [CoreDNS autopath](https://coredns.io/plugins/autopath/) can do search completion server-side, but its Kubernetes integration requires `pods verified`, visibility of the originating Pod IP and the associated Pod watches/RBAC/memory. The recorded `pods insecure` configuration does not meet those requirements. This optimization was not tested here. ## How to reproduce — revised procedure Use an approved, isolated lab and an unused namespace dedicated to this test. Save an adapted copy of the first fixture as `bench-net.yaml` and both DNS objects as `bench-dns.yaml`; keep the namespace names consistent. Check NodePool capacity, AZs, scheduling and packet-capture permissions before applying. Do not weaken production admission or security controls to run the test. These bounded commands can still saturate nodes and incur charges. Run all commands from the **operator’s Bash shell**, without entering an interactive shell in `cli`. The audit checked syntax and selected local tool behavior; it did not deploy this fixture or run these network loads on EKS. **1. Deploy the adapted fixture and verify placement.** ```bash set -euo pipefail BENCH_NS=bench-net kubectl apply -f bench-net.yaml kubectl -n "$BENCH_NS" wait --for=condition=Ready \ pod/cli pod/srv-same pod/srv-a pod/srv-b --timeout=300s kubectl -n "$BENCH_NS" get pods -o wide kubectl get nodes -L topology.kubernetes.io/zone,node.kubernetes.io/instance-type,karpenter.sh/nodepool SAME_IP=$(kubectl -n "$BENCH_NS" get pod srv-same -o jsonpath='{.status.podIP}') AZ_IP=$(kubectl -n "$BENCH_NS" get pod srv-a -o jsonpath='{.status.podIP}') CROSS_IP=$(kubectl -n "$BENCH_NS" get pod srv-b -o jsonpath='{.status.podIP}') : "${SAME_IP:?missing srv-same IP}" "${AZ_IP:?missing srv-a IP}" "${CROSS_IP:?missing srv-b IP}" for bench_pod in srv-same srv-a srv-b; do kubectl -n "$BENCH_NS" logs "$bench_pod" --tail=30 kubectl -n "$BENCH_NS" exec "$bench_pod" -- ss -lnt done for bench_ip in "$SAME_IP" "$AZ_IP" "$CROSS_IP"; do kubectl -n "$BENCH_NS" exec cli -- \ curl --fail --silent --show-error --max-time 5 "http://$bench_ip:8080/" >/dev/null done ``` Before continuing, verify `cli` and `srv-same` share a node, `srv-a` uses a different node in the same AZ, and `srv-b` uses another AZ. Check server listeners on 5201/8080/8079 and startup errors: Pod Ready alone does not prove these processes are listening, because the historical fixture has no readiness probes. Record node IDs, IPs, image IDs and actual `iperf3 --version` / `fortio version`. Stop and refresh addresses if a Pod is recreated. **2. RTT and one reference HTTP request.** ```bash for bench_ip in "$SAME_IP" "$AZ_IP" "$CROSS_IP"; do kubectl -n "$BENCH_NS" exec cli -- ping -c 200 -i 0.05 -q "$bench_ip" done kubectl -n "$BENCH_NS" exec cli -- curl --fail --silent --show-error --max-time 5 \ -o /dev/null -w 'connect=%{time_connect} total=%{time_total}\n' "http://$CROSS_IP:8080/" ``` **3. Throughput, with bounded duration and operator-local output files.** ```bash for bench_ip in "$SAME_IP" "$AZ_IP" "$CROSS_IP"; do kubectl -n "$BENCH_NS" exec cli -- iperf3 -c "$bench_ip" -p 5201 -t 20 -P 1 -J > "t1-$bench_ip-P1.json" kubectl -n "$BENCH_NS" exec cli -- iperf3 -c "$bench_ip" -p 5201 -t 20 -P 8 -J > "t1-$bench_ip-P8.json" done kubectl -n "$BENCH_NS" exec cli -- \ iperf3 -c "$CROSS_IP" -p 5201 -t 180 -P 4 -i 10 -J > t1-cross-sustained180-P4.json ``` Check JSON errors as well as exit status. Read `end.sum_sent.bits_per_second`, `end.sum_sent.retransmits`, `end.cpu_utilization_percent.host_total` / `remote_total` and `end.streams[].sender.mean_rtt` / `max_snd_cwnd`. iperf3 3.19 process CPU uses 100 % for one CPU’s time over elapsed wall time; multiple threads can exceed 100 %. TCP RTT fields are in microseconds. **4. Request latency. Repeat for each verified server address.** ```bash for bench_ip in "$SAME_IP" "$AZ_IP" "$CROSS_IP"; do kubectl -n "$BENCH_NS" exec cli -- fortio load -quiet -r 0.00001 -json - \ -qps 100 -c 4 -t 60s "http://$bench_ip:8080/" > "http-$bench_ip.json" kubectl -n "$BENCH_NS" exec cli -- fortio load -quiet -r 0.00001 -json - \ -qps 100 -c 4 -t 30s -keepalive=false "http://$bench_ip:8080/" > "new-connection-$bench_ip.json" kubectl -n "$BENCH_NS" exec cli -- fortio load -quiet -r 0.00001 -json - \ -qps 0 -c 16 -t 20s "http://$bench_ip:8080/" > "closed-loop-$bench_ip.json" kubectl -n "$BENCH_NS" exec cli -- fortio load -quiet -r 0.00001 -json - \ -grpc -ping -qps 100 -c 4 -t 30s "$bench_ip:8079" > "grpc-$bench_ip.json" done ``` In Fortio 1.69.5, `-r` is the histogram’s lowest-bucket resolution in seconds: the default `0.001` is 1 ms, and `0.00001` is 10 µs; larger buckets can widen. Percentiles interpolate within bucket bounds, including the observed minimum/maximum at the edges. A single populated bucket therefore **does not imply p50 = 0.5 ms**. The report discarded its first coarse-resolution percentiles and reran with 10 µs resolution; retain that history without calling all interpolated percentiles fake. Several tail values and new-connection medians in the tables exceed 1 ms. Save the full histogram and error counters alongside averages. **5. DNS: separate the capture of one lookup from the timed repeats.** ```bash kubectl apply -f bench-dns.yaml kubectl -n "$BENCH_NS" wait --for=condition=Ready pod/dns-default pod/dns-ndots1 --timeout=300s for bench_pod in dns-default dns-ndots1; do kubectl -n "$BENCH_NS" exec "$bench_pod" -c app -- cat /etc/resolv.conf kubectl -n "$BENCH_NS" exec "$bench_pod" -c app -- ldd --version done ``` In terminal 1, start capture before the single lookup. This filter includes ordinary UDP and TCP DNS on port 53; it does not cover encrypted DNS or CoreDNS’s upstream leg. Stop capture with Ctrl-C after that lookup before doing warm repeats. ```bash kubectl -n bench-net exec -it dns-default -c sniffer -- \ tcpdump -l -i eth0 -nn '(udp or tcp) and port 53' ``` In terminal 2, create this local helper and forward it with **`kubectl exec -i`**. The first mode makes exactly one resolver call. Warm mode makes one unmeasured warm-up followed by 20 calls in the same process; this revised procedure makes the capture boundary explicit. ```bash cat > dns-probe.py <<'PY' import json import socket import statistics import sys import time name, mode = sys.argv[1:3] if mode not in ("first", "warm"): raise SystemExit("mode must be first or warm") def one(): started = time.perf_counter() socket.getaddrinfo(name, 80, socket.AF_UNSPEC, socket.SOCK_STREAM) return (time.perf_counter() - started) * 1000 first = one() if mode == "first": print(json.dumps({"name": name, "first_process_ms": first})) else: samples = [one() for _ in range(20)] ordered = sorted(samples) print(json.dumps({ "name": name, "warmup_ms": first, "samples_ms": samples, "min_ms": ordered[0], "median_ms": statistics.median(ordered), "p90_ms": ordered[17], "max_ms": ordered[-1], })) PY BENCH_NS=bench-net DNS_POD=dns-default DNS_NAME=sts.ap-northeast-2.amazonaws.com kubectl -n "$BENCH_NS" exec -i "$DNS_POD" -c app -- \ python3 - "$DNS_NAME" first < dns-probe.py ``` After stopping capture: ```bash kubectl -n "$BENCH_NS" exec -i "$DNS_POD" -c app -- \ python3 - "$DNS_NAME" warm < dns-probe.py ``` Repeat with the names in the table and `DNS_POD=dns-ndots1`, changing the capture target too. Count query/response pairs from the first-only window; do not count 21 lookups as one. Use matching digests, resolver versions and configuration, but expect timings and cache state to differ. The original image digests and complete packet/JSON artifacts are not supplied on this page, so exact reproduction is not guaranteed. **6. Clean up only this test’s resources.** If `bench-net` was created exclusively for this run, remove it with `kubectl delete namespace bench-net` after saving results. Verify the remaining nodes and cost separately. Karpenter consolidation depends on its policy, budgets and other workloads; deleting the namespace does not guarantee immediate node removal. `do-not-disrupt` is not protection against every forceful disruption method. ## Caveats - **The nodes were fresh, but not entirely alone.** Soon after Karpenter launched the three m5.xlarge nodes for this test, consolidation moved a few small Pods from other namespaces onto them (one onto the `cli` node, three onto the `srv-b` node — small internal services and controllers unrelated to the benchmark traffic). They were idle or low-traffic during the runs, and load was limited to bursts of at most 180 s. The `cli` node showed 3901m / 3920m (99 %) of CPU *requested*, which says nothing about actual utilisation. - **Single run (n = 1 per cell, one day).** There were no independent repetitions to estimate variance. Rankings, ratios and causal explanations are also limited by that sample size; none is an SLA. - **Application ClusterIP and traffic distribution were not measured.** The report says Service creation in the benchmark namespace failed with `failed calling webhook "mservice.elbv2.k8s.aws": … no endpoints available for service "aws-load-balancer-webhook-service"`. A failing `failurePolicy: Fail` webhook rejects matching requests; its rules, namespace/object selectors and match conditions determine scope. This historical incident is not a statement that today’s cluster cannot create any Service. The benchmark did not bypass the webhook. DNS still used the pre-existing `kube-dns` Service. See the [Troubleshooting Playbook](https://www.atomai.click/kubernetes-docs/llms/en/ops/16-troubleshooting-playbook.md). - **ENA allowance counters were not collected.** `ethtool -S` must target the host’s actual ENA interface with appropriate access; a Pod’s own `eth0` is generally a veth, and `hostNetwork` alone does not prove the right device or permissions. Relevant counters include `bw_in_allowance_exceeded`, `bw_out_allowance_exceeded`, `pps_allowance_exceeded`, `conntrack_allowance_exceeded` and `linklocal_allowance_exceeded`. Retransmits do not substitute for these measurements. - **Burst-credit exhaustion was merely not observed within 180 s.** On "Up to" instances, longer sustained transfers may be throttled toward the baseline (1.25 Gbps). Nothing beyond 180 s was tested. - **DNS cache state was not controlled.** First-process and repeated lookups differ, but `cache 30` does not ensure a hit for 30 s. Replica selection and upstream state can affect both sets of timings; the reported comparison is observational. - **Same-node CPU pressure is plausible.** The client’s 99.8 % process CPU supports that interpretation of the 29.97 Gbps result; it does not establish every bottleneck or make 29.97 / 48.15 Gbps portable to other instances. - **Other CNI modes and policy enforcement were not compared.** Prefix delegation and Security Groups for Pods were off; the namespace had no NetworkPolicies. These bare Pods also do not constitute a validation of VPC CNI NetworkPolicy enforcement for supported controller-owned workloads. ## Related reading - [Amazon VPC CNI](https://www.atomai.click/kubernetes-docs/llms/en/networking/01-vpc-cni.md) — the data plane under these measurements: Pods receiving VPC IPs directly, prefix delegation, ENI/IP warming - [Zonal Cluster Operations](https://www.atomai.click/kubernetes-docs/llms/en/ops/15-zonal-operations-guide.md) — zone-aligned placement and AZ failover design that reduce the bill in Measurement 3 - [Troubleshooting Playbook](https://www.atomai.click/kubernetes-docs/llms/en/ops/16-troubleshooting-playbook.md) — diagnosis of webhook failures; the incident here is historical - [Sidecar vs Ambient Mode Selection Guide](https://www.atomai.click/kubernetes-docs/llms/en/service-mesh/istio/comparison/03-sidecar-vs-ambient.md) — separate hardware/workload experiment; its +1.29 ms is a whole-scenario difference - [EBS gp2 vs gp3 Measured Benchmark](https://www.atomai.click/kubernetes-docs/llms/en/storage/01-ebs-gp2-gp3-benchmark.md) — the storage path of the same cluster, measured - [Kafka on EKS Measured Benchmark](https://www.atomai.click/kubernetes-docs/llms/en/data-on-eks/kafka/09-kafka-benchmark.md) — replication traffic, flow limits and availability tradeoffs - [Guidebook Roadmap — the measured-benchmark series](https://www.atomai.click/kubernetes-docs/llms/en/roadmap.md) - [Quiz: Pod Network Benchmark](https://www.atomai.click/kubernetes-docs/en/quizzes/networking/06-pod-network-benchmark-quiz) ### Primary references used in the review - [Fortio 1.69.5 histogram implementation](https://github.com/fortio/fortio/blob/v1.69.5/stats/stats.go) · [CLI flags](https://github.com/fortio/fortio/blob/v1.69.5/cli/fortio_main.go) - [glibc 2.41 search ordering](https://github.com/bminor/glibc/blob/glibc-2.41/resolv/res_query.c) · [A/AAAA transport](https://github.com/bminor/glibc/blob/glibc-2.41/resolv/res_send.c) - [CoreDNS Kubernetes / autopath requirements](https://coredns.io/plugins/kubernetes/) - [Kubernetes Service traffic distribution](https://kubernetes.io/docs/concepts/services-networking/service/) · [virtual IP handling](https://kubernetes.io/docs/reference/networking/virtual-ips/) - [EC2 ENA network metrics](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/monitoring-network-performance-ena.html) - [Karpenter disruption and cleanup conditions](https://karpenter.sh/docs/concepts/disruption/)