Kernel Features Behind Containers

Supported Versions: Linux 6.1 / 6.12 / 6.18 (Amazon Linux 2023), Kubernetes 1.25+ (cgroup v2) Last Updated: September 13, 2026

What This Document Covers

• Which kernel features combine to make a container — and the fact that there is no "container" kernel object
• What actually changed operationally in the cgroup v1 → v2 move (especially OOM diagnosis)
• Where netfilter and conntrack sit in Kubernetes networking

First: There Is No "Container" in the Kernel

This is the starting point for understanding containers. There is no struct container, and no single syscall that creates one.

A container is a convention built by combining several independent kernel features. When a runtime (containerd, runc) starts a process, it applies these together.

Purpose	Kernel feature
What can it see (isolation)	namespaces
How much can it use (limits)	cgroups
What can it do (privileges)	capabilities, seccomp, LSM (AppArmor/SELinux)
How is the filesystem composed	overlayfs (union mount)
How does traffic flow	veth, bridge/route, netfilter

Two practical conclusions follow from it being a combination.

First, isolation is not all-or-nothing. Some namespaces can be shared while others are isolated. A Kubernetes Pod is exactly that — containers in the same Pod share the network and IPC namespaces while mount and PID namespaces are usually separate. That is why containers in a Pod can reach each other over localhost (shared network) but cannot see each other's filesystems (separate mounts).

Second, isolation you forget becomes a silent hole. The kernel does not know "make a container," so if the runtime does not apply a seccomp profile, the workload simply runs without one. This is why container security is a matter of runtime and policy configuration.

Namespaces — What Can It See

A namespace separates the "name space" of a kernel resource, so the same name or number refers to different things in different namespaces.

Namespace	Isolates	In a Pod
mnt	Mount points	Per container
pid	Process IDs	Per container (shareProcessNamespace: true shares within the Pod)
net	Interfaces, routing tables, netfilter rules, sockets, ports	Shared per Pod
ipc	System V IPC, POSIX message queues	Shared per Pod
uts	hostname, domainname	Shared per Pod
user	UID/GID mapping	Not used by default (see below)
cgroup	cgroup root path	Per container
time	Boot time, monotonic clock (5.6+)	Not used

Why the net namespace is the Pod boundary

A Pod's identity is decided here. Kubernetes creates one net namespace per Pod (held by the pause container) and puts all of that Pod's containers into the same net namespace.

What follows:

• Containers in a Pod share the same IP and the same port space → two containers in one Pod cannot both bind 8080
• localhost communication works → the foundation of the sidecar pattern
• netfilter rules are also per net namespace → this is why a sidecar mesh's init container can install iptables rules inside the Pod's net namespace, and why those rules do not affect the whole node (see VPC Lattice Kernel Datapath)
• Routing tables are separate too → ip route inside a Pod is not the node's

user namespaces — why they were not the default for so long

A user namespace maps container UIDs/GIDs to a different host range. This reduces the privileges of many escaped operations but does not guarantee containment against a kernel vulnerability or other privilege-escalation path.

Yet it was not the default for a long time. The reason is file ownership. Files on a volume are recorded with host UIDs; if the container sees a different UID, permissions do not line up. Solving it requires translating UIDs at mount time (idmapped mounts, kernel 5.12+), and storage drivers and CSI must support it too.

Kubernetes user namespace support status

Per KEP-127, the maturity stages are:

Stage	Version
alpha	v1.25
beta	v1.35
stable (GA)	v1.36

The feature gate is UserNamespacesSupport, applying to kubelet and kube-apiserver. From 1.36 it is GA, so hostUsers: false works without enabling a feature gate.

Needs verification

Those maturity stages are upstream Kubernetes. Whether EKS offers that version, and whether your container runtime and CSI drivers support idmapped mounts, are separate questions. Confirm the EKS supported version and your runtime/storage combination before adopting.

cgroups — How Much Can It Use

cgroups (control groups) measure and limit resource use for a group of processes. This is where Kubernetes requests/limits ultimately land.

Structural differences between v1 and v2

Item	cgroup v1	cgroup v2
Hierarchy	A separate tree per controller (cpu, memory, blkio…)	A single unified tree
Process membership	Can be in different groups per controller	Belongs to exactly one group
Memory + IO cooperation	Hard (separate trees cannot coordinate)	Possible (same tree)
Pressure information	None	PSI (cpu.pressure, memory.pressure, io.pressure)
CPU limit representation	cpu.cfs_quota_us / cpu.cfs_period_us	cpu.max (one file: "quota period")
Memory limit representation	memory.limit_in_bytes	memory.max, plus memory.high (soft pressure)
AL2023 EKS AMI	—	Default

Where v1's "separate tree per controller" actually hurt was coordinating memory reclaim with IO. When memory runs low and page cache must be dropped, that reclaim itself causes disk IO — and in v1 the two controllers knew nothing about each other. v2's unified tree handles both in the same hierarchy.

The biggest operational change — OOM diagnosis

Here is the fact you must know about cgroup v2.

memory.current includes page cache.

So even when the memory the application actually holds (anon/RSS) is far below the limit, reading many files piles up page cache and pushes memory.current to the limit.

There is an important distinction here. Page cache is reclaimable. In the normal case the kernel hits the limit, drops page cache to make room, and no OOM occurs. The problem arises when reclaim cannot keep up with allocation — that is when the OOM killer acts.

Practical implications:

Misconception	Reality
"memory.current near the limit means OOM is imminent"	It may include reclaimable file cache; inspect anon/file/kernel usage and pressure instead of assuming its composition
"Just look at RSS"	OOM can happen with low RSS (when reclaim can't keep up)
"Raising the limit fixes it"	If the cause is slow reclaim, it will recur

Values to look at when diagnosing:

File/value	Meaning
memory.current	Current usage (includes page cache)
memory.stat → anon	Anonymous memory — what the application actually holds
memory.stat → file	Page cache
memory.events → oom / oom_kill	OOM occurrences / kills
memory.events → high / max	Times the soft/hard limit was hit
memory.pressure (PSI)	Fraction of time stalled by memory pressure

PSI is especially useful because it reports pressure (how long you waited as a result) rather than usage (how much you use). A rising some avg10 in memory.pressure means time is being spent on reclaim — something a usage graph alone will not show.

CPU limits and throttling — why low utilization can still be slow

A CPU limit is a bandwidth limit. cpu.max of 20000 100000 means "up to 20ms per 100ms period."

Something counterintuitive follows. If the application does its work in a short burst across several threads, it spends the whole quota early in the period and is forcibly stopped until the period ends. Average utilization looks like a low 20% while latency spikes.

It is worse with multiple threads. With 4 threads running concurrently, a 20ms quota is consumed in 5ms of wall time. The remaining 95ms is waiting.

Diagnosis: cpu.stat → nr_throttled (periods throttled) and throttled_usec (total throttled time). If nr_throttled is a meaningful fraction of nr_periods, the limit is the cause.

Directions for response (see Resource Optimization for request/limit design):

• Raise or remove the limit (trading off node stability)
• Align the application's thread count with the limit (JVM -XX:ActiveProcessorCount, Go GOMAXPROCS, etc.) — a mismatch between the CPU count the container perceives and its actual quota is often the root cause
• Check actual stall time with cpu.pressure PSI

Privileges — What Can It Do

Even with isolation (namespaces) and limits (cgroups) in place, reducing the actions a process can take is a separate layer.

Feature	What it does	In Kubernetes
capabilities	Grant/drop root privileges as fine-grained units (CAP_NET_ADMIN, CAP_SYS_ADMIN, …)	securityContext.capabilities.add/drop
seccomp	Restrict the allowed set of syscalls	securityContext.seccompProfile (RuntimeDefault recommended)
LSM (AppArmor/SELinux)	Control file and network access by policy	securityContext.appArmorProfile, etc.
no_new_privs	Block privilege escalation via setuid binaries	allowPrivilegeEscalation: false

The three layers answer different questions — capabilities ask "do you hold this privilege," seccomp asks "may you call this syscall," LSM asks "may you touch this object." One alone is insufficient; layering them is the norm.

CAP_NET_ADMIN deserves a mention. A sidecar mesh's init container needs it to install iptables rules, which is why adopting a mesh raises the security-review question "why does this Pod hold NET_ADMIN?"

netfilter and conntrack — The Reality of Kubernetes Networking

netfilter

netfilter is a framework providing hooks at defined points in the kernel network stack. iptables, nftables, and ipvs are all userspace tools using those hooks, or implementations on top of them.

Main hook points:

Hook	When
PREROUTING	Packet arrives, before the routing decision — the DNAT point
INPUT	Packets bound for local processes
FORWARD	Packets passing through
OUTPUT	Packets leaving locally
POSTROUTING	After routing, just before leaving — the SNAT/MASQUERADE point

Where these hooks are used in Kubernetes:

• Service ClusterIP → Pod IP translation: DNAT at PREROUTING/OUTPUT
• Source translation for Pod → external traffic: MASQUERADE at POSTROUTING
• NetworkPolicy: CNI inserts rules at FORWARD and elsewhere (Calico's iptables dataplane)
• Sidecar mesh traffic interception: REDIRECT at OUTPUT/PREROUTING inside the Pod net namespace

kube-proxy modes — iptables, IPVS, nftables

Service implementation comes in three flavors, and the landscape shifted in 2025–2026.

Mode	Rule evaluation	Status
iptables	Rule-chain lookup cost depends on rule layout; current kube-proxy optimizes updates	Default where not explicitly changed; verify the installed implementation
IPVS	In-kernel L4 load balancer, hash-based O(1)	Deprecated in Kubernetes 1.35 (December 2025); planned default disablement in 1.40 and removal in 1.43
nftables	O(1) lookup plus incremental rule updates	GA in Kubernetes 1.33 (alpha 1.29 → beta 1.31). Requires kernel 5.13+ on worker nodes

How to read this:

• In large clusters, the iptables-mode bottleneck is rule count and update cost. The more Services and Endpoints, the longer kube-proxy's sync takes — and during that window the rules are not current.
• If you run IPVS, you need a migration plan. Upstream plans default disablement in 1.40 and removal in 1.43; confirm the maintained KEP-5495 schedule when planning. The recommended replacement is nftables mode.
• AL2023 nodes run kernel 6.x, so they satisfy the nftables mode kernel requirement.
• Even with nftables GA, the default is still iptables — you must switch explicitly.

conntrack — the most frequent source of incidents

For netfilter to do NAT, it must remember connections. If you rewrote the address on the way out, you have to undo it on the way back. The kernel table holding that memory is nf_conntrack.

In kube-proxy netfilter modes, Service NAT relies on connection tracking. However, non-NAT traffic may also be tracked, and headless Services, external endpoints and eBPF implementations have different paths. Do not equate every Kubernetes Service with one mandatory DNAT path.

What happens on exhaustion is the crux of the problem. There is no loud error. New connections are silently dropped, and the application sees connection timeouts or refusals. From the application side there is no way to know why.

Observation point	Meaning
/proc/sys/net/netfilter/nf_conntrack_count	Current entries
/proc/sys/net/netfilter/nf_conntrack_max	Ceiling
conntrack -S → insert_failed	Insert failures; correlate with count/max, drops and kernel logs rather than treating this as proof of exhaustion alone
conntrack -S → drop	Dropped packets
dmesg → nf_conntrack: table full, dropping packet	Kernel warning

One EKS-specific caution. kube-proxy also manages conntrack values, and EKS ships a kube-proxy-config ConfigMap by default that takes precedence over command-line arguments. So you can raise the sysctl on the node and have kube-proxy set it back. The correct path is to adjust conntrack.maxPerCore/conntrack.min in the ConfigMap and restart the kube-proxy DaemonSet.

Raising nf_conntrack_max increases node memory use. Each entry costs memory, so you cannot raise it without bound — it must match node size. Concrete settings are in EKS Node Kernel Tuning.

Verify the active kube-proxy configuration

A historical Bottlerocket report shows node sysctl values being overwritten by kube-proxy. When --config is used, edit the active configuration (conntrack.maxPerCore and conntrack.min), not CLI flags that it overrides. Setting both to 0 intentionally delegates the ceiling to node sysctl; confirm behavior against the deployed add-on/version and verify the resulting node value. Preserve the node-memory budget.

The historical issue is not evidence that every current Bottlerocket release has the same behavior. Read the effective configuration and actual sysctl after rollout.

Reducing conntrack pressure

There are approaches that reduce the load itself.

• Headless Services avoid Service VIP DNAT, but do not inherently bypass conntrack.
• Cilium can replace kube-proxy/netfilter functions with eBPF maps; measure its own tracking/map pressure and any remaining netfilter path.
• Connection reuse reduces connection churn; verify both established capacity and timeout behavior.

overlayfs — How Image Layers Are Composed

Container images being layered, and those layers appearing as one filesystem, is union mount — specifically overlayfs.

Three parts:

Layer	Role
lowerdir	Read-only — image layers (several can stack)
upperdir	Writable — the container's changes
merged	The combined view the container sees

The operationally important property is copy-up. Modifying a file from lowerdir copies the entire file to upperdir first, then modifies it. So:

• Modifying a large file slightly still pays the full copy cost. Changing one byte of a 1GB file copies 1GB
• Heavy writes inside a container consume node disk (ephemeral storage)
• Write-heavy paths belong on volumes — emptyDir, PVC, and so on

Summary

• There is no "container" in the kernel. It is a combination of namespaces (isolation) + cgroups (limits) + capabilities/seccomp/LSM (privileges) + overlayfs (filesystem) + netfilter (network). That is why isolation is selective, and forgotten isolation becomes a silent hole.
• The net namespace is the Pod boundary. The shared IP/port space, localhost communication, and Pod-scoped netfilter rules all follow from it.
• In cgroup v2, memory.current includes page cache. OOM diagnosis needs memory.stat → anon, memory.events, and PSI (memory.pressure) together.
• A CPU limit is a bandwidth limit, so throttling spikes latency even at low utilization. cpu.stat → nr_throttled is the evidence.
• For kube-proxy, nftables is GA in 1.33 and IPVS is deprecated in 1.35; upstream plans default disablement of IPVS in 1.40 and removal in 1.43. The default is still iptables.
• Conntrack exhaustion can drop new connections. Diagnose with count/max, drop/insert counters and logs; check the effective kube-proxy configuration before changing limits.

Next: Kernel Networking Stack walks the full path a packet travels.

References

• Control Group v2 — Linux kernel documentation
• PSI - Pressure Stall Information
• namespaces(7) — Linux manual
• KEP-127: Support User Namespaces
• bottlerocket-os/bottlerocket#4221 — conntrack limit not applied
• NFTables mode for kube-proxy (Kubernetes Blog)
• KEP-5495: Deprecate IPVS mode in kube-proxy
• Running kube-proxy in nftables Mode — EKS Best Practices
• Increase nf_conntrack_max limit on EKS nodes
• Amazon EKS-Optimized Amazon Linux 2023 AMIs