Part 8: eBPF Datapath Internals

Review baseline: Cilium 1.20.2 (stable, 2026-09-16); Cilium CLI 0.20.1; Hubble CLI 1.20.2. Items marked [1.21 pre-release] come from v1.21.0-pre.3 and may change before GA. Last Updated: October 15, 2026

This chapter opens the Cilium datapath: which BPF programs run, where they attach, how they call each other, which maps they read, how the agent compiles and loads them, and how one packet is processed hop by hop. The generic eBPF primer (history, verifier, map types) stays in Part 2: eBPF. Overlay and routing behavior is in Part 9; IPAM in Part 10.

Stale-blog warning: cilium_tunnel_map no longer exists. Older posts and some upstream sample listings show a cilium_tunnel_map that mapped a Pod CIDR to a remote node IP. Cilium 1.17 still had it in BPF (bpf/lib/encap.h) and cilium-dbg bpf tunnel list dumped it. It was dropped from the datapath in 1.18 (the 1.18.14 BPF tree has no reference; the agent only unpins a leftover map), and the Go package was removed in 1.19. In Cilium 1.20.2 no BPF program references it. The remote node (tunnel endpoint) is stored in the value of cilium_ipcache_v2 (field tunnel_endpoint, with the flag flag_has_tunnel_ep). There is no cilium-dbg bpf tunnel list command in 1.18 or 1.20. If a guide tells you to bpftool map dump the tunnel map to debug overlay routing, dump the ipcache instead (cilium-dbg bpf ipcache list). One upstream development doc still lists the map in a sample output; that does not mean the map is live.

Learning Objectives

• Name each Cilium BPF object, its entry programs, and the device or cgroup hook each one attaches to.
• Explain why Cilium uses tail calls, and how policy for a destination endpoint is executed from the sender's program.
• Read the key/value layout of the policy, ipcache, conntrack, and service maps and size them with the right flags.
• Describe the agent-side flow: ELF template cache, .rodata.config constants, map pinning, attach, commit.
• Trace a Pod -> ClusterIP -> Pod connection and list the map lookups at every hop.
• Inspect a live node with cilium-dbg bpf ... and bpftool.

1. Program Set and Attach Points

Cilium compiles a small number of C sources under bpf/. Each produces one ELF object; each object has one or more entry programs (ELF section tc/entry, xdp/entry, cgroup/...) and many tail programs (section tc/tail). Program names shown by bpftool prog show carry the prefix cil_.

1.1 Entry programs in 1.20.2

ELF object (source)	Entry program	Attached to	Purpose
bpf_lxc.o	cil_from_container	ingress of the host-side lxc* veth (that is, traffic leaving the Pod); on netkit, the primary hook of the netkit peer	Per-endpoint egress path: identity, CT, egress policy, service LB (per-packet mode), forward
bpf_lxc.o	cil_to_container	egress of the host-side lxc* (traffic toward the Pod, delivered through the stack)	Ingress policy for stack-delivered traffic (endpoint routes, L7 proxy returns). Attached only when the endpoint needs an egress program
bpf_lxc.o	cil_lxc_policy, cil_lxc_policy_egress	not attached to a device; stored in the global prog arrays cilium_call_policy / cilium_egresscall_policy	Destination-ingress and source-egress policy, called by other programs
bpf_host.o	cil_from_netdev	ingress of every native/external device	Host-bound and forwarded traffic, NodePort, identity from ipcache, host firewall
bpf_host.o	cil_to_netdev	egress of native devices (attached only when masquerading, NodePort, WireGuard or IPsec need it)	BPF masquerade, NodePort SNAT, EDT/bandwidth, encryption marks
bpf_host.o	cil_from_host	egress of cilium_host	Host stack -> Pod traffic
bpf_host.o	cil_to_host	ingress of cilium_host and cilium_net	Pod -> host traffic, host firewall
bpf_host.o	cil_host_policy	prog-array slot (host endpoint policy)	Host firewall policy
bpf_overlay.o	cil_from_overlay, cil_to_overlay	ingress / egress of cilium_vxlan or cilium_geneve	Reads the tunnel ID as the source identity
bpf_wireguard.o	cil_from_wireguard, cil_to_wireguard	cilium_wg0 ingress (always) / egress (only when required)	Marks decrypted traffic
bpf_xdp.o	cil_xdp_entry	XDP on native devices	CIDR prefilter and NodePort/DSR acceleration
bpf_sock.o	cil_sock4_connect, cil_sock4_sendmsg, cil_sock4_recvmsg, cil_sock4_getpeername, cil_sock4_pre_bind, cil_sock4_post_bind, IPv6 equivalents, cil_sock_release	cgroup v2 sock_addr / sock hooks at the cgroup root	Socket-level load balancing
bpf_sock_term.o	cil_sock_{tcp,udp}_destroy_v{4,6}	BPF iterators (iter/tcp, iter/udp) calling the bpf_sock_destroy kfunc	Terminate sockets connected to deleted backends; the agent falls back to netlink sock_diag if the kfunc is unavailable
bpf_probes.o, bpf_alignchecker.o	n/a	run once with BPF_PROG_RUN at startup	Kernel capability probes; C/Go struct layout check

bpf_network.c (cil_from_network, IPsec decrypt on native-device ingress) exists in 1.18.14 and 1.19.8 and is removed in 1.20.

                       node A                                   node B
 pod C eth0 ----veth---- lxc_C           lxc_B ----veth---- pod B eth0
   |  cgroup/connect4          |             ^                     
   |  (bpf_sock.o)             | tcx ingress |  redirect_peer
   |                           v cil_from_container                 
   |                    +-------------------+   (tail calls, maps)
   |                    |   bpf_lxc.o       |-------+
   |                    +-------------------+       | encap / FIB redirect
   |                           |                     v
   |        cilium_host  <-----+          cilium_vxlan (cil_to_overlay)  ==underlay==>
   |        cil_from_host/cil_to_host        eth0 (cil_from_netdev / cil_to_netdev)
   |                                         eth0 XDP (cil_xdp_entry, optional)

1.2 tcx, legacy tc, netkit

For per-device programs the loader (pkg/datapath/loader/tc.go) chooses the mechanism in this order:

1. tcx (Helm bpf.enableTCX: true, the default): a bpf_link attached with link.AttachTCX at the tail anchor. The link is pinned under /sys/fs/bpf/cilium/endpoints/<id>/links/ (endpoints) or /sys/fs/bpf/cilium/devices/<dev>/links/ (devices). Updating a program updates the link atomically, so there is no window with a missing program during a reload. Requires kernel 6.6 or newer. After a successful tcx attach, stale legacy filters are removed.
2. Legacy tc: if the kernel returns "not supported" or enableTCX: false, the loader uses a clsact qdisc with cls_bpf in direct-action mode, and detaches any stale tcx link.
3. Hard failure only when neither works: the agent reports that it needs tc-bpf support or tcx (Linux 6.6 or newer).

netkit (maturity: beta): bpf.datapathMode: veth | netkit | netkit-l2 | auto. The default is veth. With netkit, the Pod-specific programs attach inside the netkit peer device; other devices still use tcx, and selecting netkit turns tcx on automatically. Source-enforced prerequisites (canUseNetkit): kernel probe passes (upstream docs say 6.8 or newer; the probe error text in the source mentions 6.7), BPF host routing (no bpf.hostLegacyRouting), BPF masquerade, kube-proxy replacement, and no bpf.tproxy (it needs bpf_sk_assign, valid only at tc ingress). auto is new in 1.20: it probes and falls back to veth with a warning. Once Pods exist as veth, switching to netkit or auto for them is refused; roll new node groups instead.

Mechanism	Kernel	Pinned object	Reload behavior	Maturity in Cilium
Legacy tc (clsact + cls_bpf)	5.10 baseline	filter on device	filter replace	stable, fallback
tcx	6.6+	bpf_link under /sys/fs/bpf/cilium/...	atomic link update	stable, default when available
netkit	6.8+ (docs)	netkit link on peer	atomic link update	beta

1.3 XDP and cgroup hooks

• XDP (cil_xdp_entry) is attached when loadBalancer.acceleration is native or best-effort. It runs a CIDR prefilter (cilium_cidr_v4_fix/dyn LPM maps) and the NodePort/DSR fast path. The loader retries up to four attach permutations (attach type and the XDP_HAS_FRAGS flag) when the driver returns EINVAL.
• cgroup hooks are attached once at the cgroup root (agent flag --cgroup-root, default /run/cilium/cgroupv2) with bpf_link, pinned under /sys/fs/bpf/cilium/socketlb/links/cgroup. They run in every Pod netns unless socketLB.hostNamespaceOnly=true.

2. Tail Calls and the Policy-Program Structure

2.1 Why tail calls

A tail call replaces the running program with another without returning. Cilium uses them for three reasons: the verifier limit on one program's complexity (1M instructions), the 512-byte stack, and the ability to keep large optional features out of the common path. Maximum chain length is 33 tail calls. Mixing bpf-to-bpf calls with tail calls requires kernel 5.10, one reason for the 5.10 baseline.

State crosses a tail call in two places: skb->cb[] (the CB_* slots, such as CB_SRC_LABEL) and per-CPU scratch maps cilium_tail_call_buffer4/6, which hold, for example, the conntrack tuple and result computed by the CT tail program.

2.2 Per-endpoint call arrays

Each endpoint has a private PROG_ARRAY named cilium_calls_<endpoint-id, 5 digits> (cilium_calls_00312). The host endpoint uses cilium_calls_hostns_<id>, and each native device uses cilium_calls_netdev_<ifindex>. Other loaders use cilium_calls_overlay_2 (the overlay device, named with the reserved world identity 2), cilium_calls_wireguard_<ifindex> and cilium_calls_xdp_<ifindex>. They are declared with CILIUM_CALL_SIZE entries and a pin flag that means "replace the pin on every load; never reuse an old array".

Version	CILIUM_CALL_SIZE
1.18.14	48
1.19.8	49
1.20.2	50
1.21.0-pre.3	50

Slots are static: the call site is inline assembly with a constant slot number. The loader reads BTF decl-tags (tail:cilium_calls/<idx>) on each tc/tail program, fills the array, and prunes tail programs that are unreachable once the constants are applied (section 5). Slot groups in 1.20:

Slots	Meaning
1-2, 4-5	drop/error notification, ICMPv6 neighbor solicitation, time-exceeded
6	ARP
3	unused (a numbering gap kept to avoid conflicts when backporting)
7, 10	IPv4 / IPv6 from-LXC (aliased for netdev, overlay, WireGuard entry)
8-9	IPv4-to-IPv6 and IPv6-to-IPv4 translation (CILIUM_CALL_IPV46_RFC6052, CILIUM_CALL_IPV64_RFC6052; NAT46/NAT64)
11-14	to-LXC policy only, to-endpoint
15-25, 36-39	NodePort NAT, rev-NAT, DSR, SNAT forward; from-host
26-27	from-LXC continuation (second half of the egress path, splits verifier complexity)
28-33	CT ingress / CT ingress policy-only / CT egress, IPv4 and IPv6
34-35, 40	SRv6, inter-cluster rev-SNAT
41-47	continuation from host/netdev, "no service" ICMP unreachable, multicast delivery
48-49	policy-denied ICMP response (new in 1.20, used when policy_deny_response is enabled)

2.3 Global policy program arrays

Policy is not in the per-endpoint call array. cil_lxc_policy and cil_lxc_policy_egress are inserted into two global prog arrays, indexed by endpoint ID and sized ENDPOINTS_MAP_SIZE:

Array	Index	Runs
cilium_call_policy	destination endpoint ID	destination ingress policy
cilium_egresscall_policy	source endpoint ID	source egress policy

The loader writes these slots before attaching the tc/tcx program (PolicyMap.Update(ep id, PolicyProg)). A sender that finishes its L3 rewrite for a local destination calls tail_call_policy(ep->lxc_id). This matters because with BPF host routing the packet is moved into the destination Pod's netns with bpf_redirect_peer, which skips the destination's tc egress hook. Calling the destination's policy program from the sender is how same-node ingress policy is still enforced. If the slot is empty (endpoint terminating), the packet is dropped with DROP_EP_NOT_READY.

cil_to_container still exists for stack-delivered traffic, so destination ingress policy can run either from the sender's tail call or from that egress hook.

3. Policy Enforcement Path

3.1 Policy map

Each endpoint has cilium_policy_v3_<id> (1.20; cilium_policy_v2_ in 1.18/1.19). Type LPM_TRIE, flags BPF_F_NO_PREALLOC | BPF_F_RDONLY_PROG_COND, size --bpf-policy-map-max (default 16384 per endpoint).

policy_key  { lpm prefixlen; u32 sec_label;   /* remote identity, 0 = wildcard */
              u8 egress:1;  u8 protocol;      /* 0 = any */
              be16 dport; }                   /* prefix match => port ranges */
policy_entry{ be16 proxy_port;                /* != 0 => redirect to L7 proxy */
              deny:1; lpm_prefix_length:5; auth_type:7; has_explicit_auth_type:1;
              u32 precedence; u32 cookie; }

The LPM prefix covers sec_label and egress exactly, protocol all-or-nothing, and dport as a bit prefix, which is how port ranges become a few entries.

3.2 Lookup algorithm

For a flow with remote identity R, protocol P, port D:

1. Specific lookup with sec_label = R. If the hit carries maximum precedence (a deny), stop.
2. Aggregate lookup: the identity's aggregate (for example a CIDR group) then sec_label = 0 (wildcard L3).
3. Choose the higher precedence (deny over proxy redirect over allow), then the longer LPM prefix. No hit means DROP_POLICY.

The match type is reported in events (L3+L4, L3 only, L4 only, protocol only, all). ICMP type is carried in dport when ICMP rules are enabled, and "fragmentation needed" is always allowed.

3.3 Verdict and side effects

• Deny: DROP_POLICY_DENY and a drop event. Allow with proxy_port != 0: redirect to the proxy (section 3.5). Allow with auth_type != 0: DROP_POLICY_AUTH_REQUIRED unless cilium_auth_map has a valid entry.
• Egress is enforced by the source endpoint (handle_ipv4_from_lxc), only for CT_NEW and CT_ESTABLISHED; replies (CT_REPLY, CT_RELATED) skip policy. Ingress is enforced by the destination (ipv4_policy() in its policy program) and creates the ingress CT entry after the allow.
• A policy-verdict event is emitted on deny and on a new allowed connection. Hairpin traffic (Pod to its own service IP) skips policy.
• Optional per-entry counters go to cilium_policystats (--bpf-policy-stats-map-max 65536).

3.4 Policy changes do not recompile

Policy updates are map operations on cilium_policy_v3_<id> (applyPolicyMapChangesLocked). The object is rebuilt and reloaded only if the endpoint or node configuration hash changes. A CiliumNetworkPolicy edit therefore never triggers clang.

3.5 L7 and FQDN redirect

When proxy_port != 0, ctx_redirect_to_proxy4() sends the flow to Envoy or the DNS proxy. Two implementations exist:

Method	How	Maturity
iptables TPROXY (default)	datapath sets mark = MARK_MAGIC_TO_PROXY | proxy_port << 16 and returns OK; CILIUM_PRE_mangle rules with xt_socket/TPROXY steer it to the listener	stable
bpf.tproxy: true	bpf_sk_assign assigns the listener socket at tc ingress	beta, default off, incompatible with netkit (agent refuses to start in 1.19.8, 1.20 and 1.21.0-pre.3; no such check found in the 1.18.14 source)

Proxy ports come from --proxy-portrange-min/max (10000-20000). The CT entry records proxy_redirect so replies return to the proxy. L7 events do not come from BPF; Envoy and the DNS proxy report access-log records to the agent.

Mutual authentication (auth_type) is deprecated in 1.20 (use ztunnel); [1.21 pre-release] removes --bpf-auth-map-max and cilium_auth_map, and the entry's auth bits become reserved.

4. Map Catalog

Global maps are pinned by name under /sys/fs/bpf/tc/globals/; links and plugin objects are under /sys/fs/bpf/cilium/. Helm: bpf.root (default /sys/fs/bpf), bpf.autoMount.enabled.

4.1 Core maps

Map	Type	Key -> Value (essentials)	Notes
cilium_lxc	HASH	endpoint IP (+family, cluster_id) -> {ifindex, lxc_id, flags, mac, node_mac, sec_id, parent_ifindex}	local endpoints and host IPs; 65535 entries (lxcmap.MaxEntries)
cilium_ipcache_v2	LPM_TRIE	{cluster_id, family, ip/prefix} -> {sec_identity, tunnel_endpoint, key, flag_skip_tunnel, flag_has_tunnel_ep, flag_ipv6_tunnel_ep, flag_remote_cluster}	identity and tunnel endpoint; 512000 entries (ipcachemap.MaxEntries)
cilium_policy_v3_<id>	LPM_TRIE	see 3.1	per endpoint
cilium_calls_<id>, _hostns_, _netdev_	PROG_ARRAY	slot -> program	private, replaced on each load
cilium_call_policy, cilium_egresscall_policy	PROG_ARRAY	endpoint ID -> policy program	global
cilium_ct4_global, cilium_ct6_global	LRU_HASH	5-tuple + direction flags -> ct_entry	TCP
cilium_ct_any4_global, cilium_ct_any6_global	LRU_HASH	same	non-TCP
cilium_snat_v4_external (+v6)	LRU_HASH	CT tuple -> NAT entry (to_saddr/to_sport or to_daddr/to_dport)	masquerade, NodePort SNAT
cilium_nodeport_neigh4/6	LRU_HASH	next-hop IP -> MAC
cilium_node_map_v2	HASH	{family, ip} -> {node id, spi}	IPsec/WireGuard/ClusterMesh node IDs (v1 removed in 1.19)
cilium_devices (new 1.20)	HASH	ifindex -> {mac, l3}	synced by the agent
cilium_runtime_config	ARRAY	index -> u64 (time offset, agent liveness)	cilium-dbg bpf config list
cilium_metrics	PERCPU_HASH	{reason, direction, line, file} -> {count, bytes}	feeds forward/drop counters
cilium_events	PERF_EVENT_ARRAY	CPU -> perf fd	monitor and Hubble source (section 8)
cilium_signals	PERF_EVENT_ARRAY	datapath -> agent signals	for example CT-full triggers GC

Conntrack addresses are stored with source and destination swapped relative to the packet; the ct_entry holds lifetime, rx/tx_closing, rev_nat_index, proxy_redirect, from_l7lb, from_tunnel, src_sec_id, and the seen TCP flags. A single map holds egress, ingress and service-scope entries, separated by tuple.flags. Cluster Mesh adds per-cluster inner maps (cilium_per_cluster_ct_*) for overlapping IPs.

Timeouts (agent flags, 1.20.2): TCP established 2h13m20s, SYN 1m, FIN 10s, non-TCP 1m, service TCP 2h13m20s, service grace 1m.

4.2 Load-balancing maps

Map	Type	Key -> Value	Notes
cilium_lb4_services_v2 (+6)	HASH	{address, dport, backend_slot, proto, scope} -> {backend_id or affinity/algorithm, count, rev_nat_index, flags, qcount}	slot 0 is the frontend; slots 1..count hold backend IDs; scope separates external and internal (externalTrafficPolicy)
cilium_lb4_backends_v3 (+6)	HASH	backend_id -> {address, port, proto, flags, cluster_id, zone}
cilium_lb4_reverse_nat (+6)	HASH	rev_nat_index -> {address, port}	per-packet LB reply restore
cilium_lb4_reverse_sk (+6)	LRU_HASH	{socket cookie, backend addr, port} -> {original service addr, port, rev_nat_index}	socket-LB reverse translation
cilium_lb4_affinity, cilium_lb_affinity_match	LRU_HASH / HASH	client (IP or netns cookie) + rev_nat -> backend	sessionAffinity
cilium_lb4_source_range	LPM_TRIE	{rev_nat_id, cidr} -> flag	loadBalancerSourceRanges
cilium_lb4_maglev	HASH_OF_MAPS	rev_nat_id -> lookup-table array	--bpf-lb-algorithm maglev, table size --bpf-lb-maglev-table-size
cilium_lb4_health	LRU_HASH	socket cookie -> peer backend	LB health datapath; kernel 5.12

The upper 8 bits of the service's affinity_timeout field carry the selection algorithm (random or maglev).

4.3 Feature maps (one line each)

cilium_ipmasq_v4/v6 (non-masquerade CIDRs), cilium_ipv4_frag_datagrams (8192 default), cilium_vtep_map, cilium_srv6_*, cilium_egress_gw_policy_v4_v2 (renamed from _v4 in 1.20), cilium_l2_responder_v4/v6, cilium_mcast_*, cilium_cidr_v4_fix/dyn (XDP prefilter), cilium_encrypt_state, cilium_auth_map, cilium_policystats, cilium_ratelimit, cilium_throttle, cilium_lb_act/cilium_lb_fct (active connection tracking, off by default), and per-CPU scratch maps (cilium_tail_call_buffer4/6, cilium_nodeport_nat_buffer, cilium_xdp_scratch, cilium_percpu_trace_id).

4.4 Sizing flags

Flag (Helm)	Default (1.20.2)	Affects
--bpf-map-dynamic-size-ratio (bpf.mapDynamicSizeRatio)	0.0025 (0.25% of RAM); dynamic sizing is on by default	CT TCP/any, NAT, socket reverse-NAT; the NodePort neighbor map follows the NAT size (the flag help says "CT, NAT and policy BPF maps", but the sizing code does not resize policy maps)
--bpf-ct-global-tcp-max (bpf.ctTcpMax)	524288	TCP CT
--bpf-ct-global-any-max (bpf.ctAnyMax)	262144	non-TCP CT
--bpf-nat-global-max (bpf.natMax)	524288	NAT; capped to 2/3 of the CT sum with a warning
--bpf-neigh-global-max (bpf.neighMax)	524288	NodePort neighbor
--bpf-policy-map-max (bpf.policyMapMax)	16384 per endpoint (maximum 65536)	policy
--bpf-fragments-map-max	8192	fragments
--bpf-node-map-max	16384	node map
--bpf-lb-map-max (bpf.lbMapMax)	65536	fallback for six per-LB-map flags (--bpf-lb-service-map-max, -service-backend-map-max, -rev-nat-map-max, -affinity-map-max, -source-range-map-max, -maglev-map-max), each defaulting to 0 = use lb-map-max; present in source (pkg/loadbalancer/config.go) but not listed in the generated cmdref, which shows only --bpf-lb-map-max
--bpf-sock-rev-map-max	dynamic, defaults to CT-any size	socket reverse map
bpf.distributedLRU.enabled	false	per-CPU LRU free lists; use with larger maps
bpf.preallocateMaps	false	trades memory for lower insert latency

Dynamic sizing computes entries = default_entries * (ratio * RAM) / total_default_bytes, clamped to a floor per map (CT TCP 128Ki, CT any 64Ki, NAT 128Ki, socket reverse 64Ki) and a ceiling of 16Mi (2^24). Setting a flag explicitly disables dynamic sizing for that map only. The agent logs option <name> set by dynamic sizing to <N> when it applies. Upstream's tuned performance profile uses ratio 0.08.

Resizing a map recreates it, and entries are lost until repopulated. Resizing the LB maps or CT maps on a live node disrupts established connections. Plan resizes as a rolling node operation.

Monitor with cilium_bpf_map_pressure{map_name} (emitted above --bpf-policy-map-pressure-metrics-threshold, default 0.1), cilium_bpf_map_capacity, and cilium_bpf_map_ops_total.

5. From Agent to Kernel: Compile, Load, Attach

5.1 Flow

agent start
  |  write node headers  (StateDir/globals: node_config.h, netdev_config.h, ...)
  |  hash node config -> if changed, wipe template cache (StateDir/templates)
  v
template config hash (compile-time #defines only)
  |  cache hit?  -> reuse StateDir/templates/<sha>/bpf_lxc.o
  |  miss        -> clang -O2 --target=bpf -mcpu=v3|v2|v1 ... -> new ELF template
  v
LoadCollection (per endpoint / device)
  |  copy spec -> patch maps -> rename maps (cilium_calls -> cilium_calls_<id>, cilium_policy -> cilium_policy_v3_<id>)
  |  write .rodata.config (endpoint ID, IPs, MAC, ifindex, feature switches)
  |  dead-code analysis -> remove unused tail calls and maps
  |  fill tail-call arrays from BTF decl tags
  |  load into kernel (verifier); reuse pinned global maps, replace incompatible ones
  v
attach: policy progs into cilium_call_policy/egresscall_policy
        -> tcx/tc/netkit entry programs -> commit() map pins -> endpoint routes

5.2 ELF templates and .rodata.config

The expensive compile step still runs in the agent (the container image ships clang/LLVM; running the agent natively needs LLVM 18.1 or newer). To avoid recompiling per Pod, Cilium hashes only the template configuration: #define ENABLE_* flags that change code shape. All endpoints with the same flags share one object under StateDir/templates/<sha>/. List them with cilium-dbg bpf sha list.

Per-endpoint values are not compiled in. They are globals in the .rodata.config section declared with DECLARE_CONFIG(type, name, desc) and read with CONFIG(name). The agent writes the values into a copy of the spec before load; the kernel freezes the read-only data, so the verifier sees the values as constants and prunes dead branches. This requires dead-code-elimination support (kernel 5.1 or newer). Go-side structs are generated by dpgen from BTF decl-tags (pkg/datapath/config/*_config.go). The effective constants are dumped per endpoint/device as JSON (for example bpf_lxc.json) in the state directory and are included in cilium sysdump.

Each release moves more #define switches into .rodata.config, which means fewer distinct templates and fewer recompiles. Counting DECLARE_CONFIG/NODE_CONFIG declarations with grep over the BPF tree gives 18 (1.18.14), 43 (1.19.8), 73 (1.20.2) and 104 ([1.21 pre-release], v1.21.0-pre.3), using grep -rE '^\s*(DECLARE_CONFIG|NODE_CONFIG)\(' bpf --include=*.h --include=*.c | wc -l in each tag's checkout. Treat these counts as a rough trend, not an API.

5.3 Regeneration

On each endpoint regeneration the agent computes the endpoint header hash. If it equals the last one, only maps are synchronized (RegenerateWithoutDatapath, which is the policy-change case). Otherwise headers are written and the datapath is reloaded. Relevant metrics: cilium_endpoint_regenerations_total, endpoint_regeneration_time_stats_seconds (stages such as bpfCompilation, bpfLoadProg, bpfWaitForELF), and cilium_endpoint_component_status (new in 1.20).

Order matters: commit() of map pins must happen after all entry programs are attached, otherwise a new program could tail-call into an old array and "miss" a tail call.

5.4 Verifier testing

Cilium compile-tests the maximum-feature builds against kernels 5.10, 6.1 and net-next (bpf/complexity-tests/). Verifier failures at load time are logged with the full verifier output.

6. Kernel and Feature Matrix

6.1 Hard startup requirements (1.20.2)

The agent refuses to start without: bpf() and JIT; tc-bpf or tcx; bpf_skb_change_tail; bpf_get_socket_cookie; bpf_get_current_cgroup_id; bpf_fib_lookup; dead-code elimination; writable skb->queue_mapping; large (1M instruction) programs; bpf_skb_adjust_room with BPF_ADJ_ROOM_MAC; bpf_jiffies64; BPF_MAP_LOOKUP_BATCH; bpf_get_netns_cookie; bpf_sk_assign; bpf_get_cgroup_classid; bpf_perf_event_output in cgroup sock_addr; bpf_csum_level; bpf_skb_change_head; bpf_redirect_neigh; bpf_redirect_peer. This is why the documented minimum is kernel 5.10 (RHEL 8.10 with 4.18 is an explicit exception because of Red Hat backports). The legacy per-helper header probing is now empty; the 5.10 baseline assumes all helpers.

Kernel config options listed upstream include BPF, BPF_SYSCALL, BPF_JIT, NET_CLS_BPF, NET_CLS_ACT, NET_SCH_INGRESS, CGROUPS, CGROUP_BPF, DEBUG_INFO_BTF, and feature extras such as VXLAN, GENEVE, NET_SCH_FQ (bandwidth manager) and NETKIT.

6.2 Optional accelerators

Feature	Minimum kernel	Maturity / note
tcx links	6.6	default when available; legacy tc otherwise
netkit (bpf.datapathMode)	6.8 (docs; source probe text says 6.7)	beta
IPv6 BIG TCP	5.19	needs supported NIC driver
IPv4 BIG TCP	6.3
BBR for Bandwidth Manager	5.18
LB health-check datapath	5.12
Multicast	5.10 on AMD64; 6.0 on AArch64
bpf_sock_destroy kfunc	not stated in source	falls back to netlink sock_diag automatically

The datapath does not use CO-RE relocation helpers (a search of the 1.20.2 bpf/ tree for preserve_access_index finds only a UAPI header comment): it builds against its own headers and the UAPI __sk_buff. BTF is used for map definitions, the .rodata.config data section and decl-tags, and CONFIG_DEBUG_INFO_BTF=y is required.

On EKS, check the real node kernel with uname -r and cilium-dbg status before enabling tcx- or netkit-dependent settings; this chapter does not assume which AMI ships which kernel.

7. Worked Trace: Pod -> Service -> Pod

Illustrative setup (numbers are examples; structures come from source): client Pod C 10.0.1.10, identity 4711, endpoint ID 312, on node A. Service web ClusterIP 10.100.0.20:80/TCP, rev_nat_index 42, two backends: B1 10.0.1.55 (node A, endpoint 87) and B2 10.0.2.77 (node B), both identity 9001. Settings: IPv4, veth, tcx, BPF host routing, kubeProxyReplacement: true (which forces socket-LB on), socketLB.hostNamespaceOnly=false.

Hop 0: connect() in the Pod, cgroup/connect4 (cil_sock4_connect)

1. Look up cilium_lb4_services_v2 with {10.100.0.20, 80, TCP, slot 0, scope 0}. No entry: leave the sockaddr untouched. Entry with count=2.
2. Pick a slot: random for TCP (socket cookie for UDP). Look up the service key with that slot to get backend_id, then cilium_lb4_backends_v3[backend_id] to get 10.0.2.77:80. If sessionAffinity is set, cilium_lb4_affinity is consulted first.
3. Write cilium_lb4_reverse_sk[{cookie, 10.0.2.77, 80}] = {10.100.0.20, 80, 42}.
4. Rewrite user_ip4/user_port in the syscall. No DNAT happens on the wire. tcpdump inside the Pod shows the backend IP, a frequent surprise. A trace_sock_notify event is emitted when socket-LB tracing is on.

Hop 1: SYN leaves the Pod, cil_from_container on the host-side veth (tcx ingress)

Step	Action	Maps touched
1	clear metadata, send_trace_notify(TRACE_FROM_LXC) (subject to aggregation)	cilium_events
2	tail call slot 7 (IPv4 from-LXC)	cilium_calls_00312
3	per-packet LB skipped (socket-LB already translated)	none
4	tail call slot 30 (CT egress): lookup CT_NEW, tuple stored in per-CPU buffer	cilium_ct4_global, cilium_tail_call_buffer4
5	tail call slot 26 (handle_ipv4_from_lxc)	cilium_calls_00312
6	resolve destination identity: LPM lookup of 10.0.2.77 gives identity 9001 and (overlay) the node B tunnel endpoint; a miss means world	cilium_ipcache_v2
7	egress policy: {egress=1, sec_label=9001, TCP, 80}, two-pass lookup; allow or drop; a new allowed flow emits a policy-verdict event	cilium_policy_v3_00312, cilium_events, cilium_metrics on drop
8	if proxy_port != 0: mark for proxy, flag CT proxy_redirect, hand to the stack (section 3.5)
9	ct_create4(CT_EGRESS): insert entry, SYN lifetime 1m until ACK, then 2h13m20s	cilium_ct4_global (and CT-any for related ICMP)
10	choose the forwarding path (below)

Forwarding choices at step 10:

• Destination on the same node (B1): cilium_lxc[dst] returns {ifindex, lxc_id=87, mac, node_mac}; the L2 header is rewritten; metadata (CB_SRC_LABEL=4711, delivery flags) is set; then tail_call_policy(87) runs B1's cil_lxc_policy out of cilium_call_policy (Hop 3).
• Remote, overlay: flag_has_tunnel_ep is set in the ipcache value, so bpf_skb_set_tunnel_key sets tunnel_id = source identity 4711 and remote_ipv4 = node B; the packet is redirected to cilium_vxlan/cilium_geneve, where cil_to_overlay runs before kernel encapsulation. Pod-to-pod traffic is not masqueraded.
• Remote, native routing with BPF host routing: bpf_fib_lookup then redirect to the NIC. With bpf.hostLegacyRouting=true the packet instead returns OK to the kernel stack and netfilter. There is no VNI; the receiver re-derives the source identity from ipcache by source IP (or from marks with IPsec/WireGuard).

Hop 2 (remote only): node B receives

• Overlay: cil_from_netdev passes UDP 8472 (VXLAN) / 6081 (Geneve) to the stack; the kernel decapsulates onto cilium_vxlan; cil_from_overlay reads bpf_skb_get_tunnel_key, maps tunnel_id to the source identity (an identity of host is rejected with DROP_INVALID_IDENTITY), stores it in CB_SRC_LABEL, emits TRACE_FROM_OVERLAY, then looks up cilium_lxc[10.0.2.77] and calls tail_call_policy(ep).
• Native routing: cil_from_netdev performs the same steps with the source identity from cilium_ipcache_v2[src ip].

Hop 3: destination ingress policy and delivery

1. cil_lxc_policy runs the CT-ingress tail call: cilium_ct4_global lookup, CT_NEW.
2. Ingress policy: cilium_policy_v3_00087 with {egress=0, sec_label=4711, TCP, 80}. Verdict event emitted.
3. ct_create4(CT_INGRESS) records src_sec_id=4711, from_tunnel, proxy_redirect.
4. Delivery: bpf_redirect_peer into the Pod netns (veth) or redirect (netkit), TRACE_TO_LXC event. The backend's stack answers with source 10.0.2.77; it never sees the ClusterIP.

Hop 4: reply and close

• B's cil_from_container finds the ingress CT entry as CT_REPLY, so policy is skipped; the packet is encapsulated back with VNI 9001 (or routed natively). On node A, tail_call_policy(312) runs C's policy program; CT_REPLY skips policy; redirect_peer delivers to C.
• Socket-LB reverse: the app believes it talks to 10.100.0.20. getpeername and recvmsg hooks look up cilium_lb4_reverse_sk[{cookie, 10.0.2.77, 80}] and return the service address.
• Without socket-LB (host-namespace-only mode, L7 LB, SCTP): Hop 1 performs DNAT in bpf_lxc and stores rev_nat_index in the CT entry; replies use cilium_lb4_reverse_nat[rev_nat_index] to restore the source.
• FIN/RST sets tx/rx_closing and drops the lifetime to the FIN timeout (10s); agent CT GC deletes expired entries. cilium_lb4_reverse_sk entries linger until LRU eviction or cil_sock_release.

Per-hop lookup checklist

Hop	Program	Lookups / writes
0	cil_sock4_connect	lb4_services_v2 x2, lb4_backends_v3, lb4_affinity (optional), write lb4_reverse_sk
1	cil_from_container + tails	calls_<id> x3, tail_call_buffer4, ct4_global (lookup, create), ipcache_v2, policy_v3_<src> (egress), cilium_lxc, call_policy, events, metrics
2	cil_from_overlay / cil_from_netdev	tunnel key or ipcache_v2, cilium_lxc, call_policy
3	cil_lxc_policy (destination)	ct4_global, policy_v3_<dst> (ingress), ct4_global create, events
4	reverse path, recvmsg	ct4_global (reply), lb4_reverse_sk or lb4_reverse_nat

1.20 behavior change that affects this trace

With kube-proxy replacement and socketLB.enabled=false or socketLB.hostNamespaceOnly=true, Pod-to-NodePort connections inside the cluster are now load-balanced at the client Pod's tc hook. Client egress and backend ingress NetworkPolicies must allow the traffic accordingly.

8. Hubble Event Pipeline

BPF program --ctx_event_output--> cilium_events (PERF_EVENT_ARRAY, per CPU)
      |   trace_notify / drop_notify / policy_verdict_notify / debug / capture / trace_sock_notify
      v
agent monitor (pkg/monitor/agent): perf reader, 64 pages per CPU, opened only if a consumer exists
      |-- lost samples (kernel buffer overrun) -> MonitorStatus.Lost
      |-- unix-socket listeners (cilium-dbg monitor), queue --monitor-queue-size
      v
Hubble consumer -> observer queue (--hubble-event-queue-size)
      |   full => LostEvent source=OBSERVER_EVENTS_QUEUE, hubble_lost_events_total
      v
Hubble parser (identity/endpoint/service enrichment) -> ring buffer (--hubble-event-buffer-capacity)
      v
gRPC: hubble observe / Relay / UI

Facts to remember:

• The event map is a perf event array in 1.20 and 1.21-pre.3, not a BPF ring buffer.
• Volume control in the datapath: bpf.monitorAggregation (Helm default medium; agent flag default None). At medium and above, intra-node receive traces are suppressed and established flows report only when bpf.monitorInterval (5s) elapses or a flag in bpf.monitorFlags is seen. Per-class toggles: bpf.events.drop.enabled, bpf.events.policyVerdict.enabled, bpf.events.trace.enabled (all true by default), and an experimental rate limit bpf.events.default.rateLimit / burstLimit (both set or both zero).
• Flows in hubble observe live in an in-memory ring (hubble.eventBufferCapacity, must be 2^n - 1, default 4095, maximum 65535). They are best-effort: perf-buffer overruns and observer-queue overflow both lose events, and they are counted under different source labels (perf_event_ring_buffer, observer_events_queue).
• L7 events do not originate in BPF; they arrive as agent events from the proxies.

9. Inspection Tooling

9.1 cilium-dbg

CILIUM_POD=$(kubectl -n kube-system get pod -l k8s-app=cilium -o jsonpath='{.items[0].metadata.name}')
kc() { kubectl -n kube-system exec "$CILIUM_POD" -c cilium-agent -- "$@"; }

kc cilium-dbg status --verbose         # KPR, datapath mode (configured vs operational), BPF map pressure
kc cilium-dbg bpf fs show              # bpffs mount
kc cilium-dbg map list -v              # pinned maps and sizes
kc cilium-dbg bpf endpoint list        # cilium_lxc
kc cilium-dbg bpf ipcache list         # identity + tunnel endpoint per IP/CIDR
kc cilium-dbg bpf ct list global       # conntrack
kc cilium-dbg bpf nat list             # SNAT maps
kc cilium-dbg bpf lb list --frontends  # also --backends, --revnat, --source-ranges
kc cilium-dbg bpf lb maglev list       # only with Maglev
kc cilium-dbg bpf policy get --all -n  # every cilium_policy_v3_*, numeric identities
kc cilium-dbg bpf metrics list         # cilium_metrics
kc cilium-dbg bpf config list          # cilium_runtime_config
kc cilium-dbg bpf sha list             # compiled ELF templates
kc cilium-dbg monitor -t drop -t policy-verdict --related-to <ENDPOINT_ID>

cilium-dbg map get <name> shows the agent's cached view, not necessarily the kernel's; cilium-dbg map events <name> shows the update log. There is no bpf tunnel and no bpf calls subcommand.

9.2 bpftool (run on the node)

sudo bpftool net show                                     # tcx/tc/xdp/netkit attachments per device
sudo bpftool prog show | grep -E 'cil_|tail_'             # entry and tail programs
sudo bpftool map show pinned /sys/fs/bpf/tc/globals/cilium_ct4_global
sudo bpftool cgroup tree /run/cilium/cgroupv2             # socket-LB hooks
sudo ls /sys/fs/bpf/cilium/endpoints/*/links /sys/fs/bpf/cilium/devices/*/links

Treat pinned maps as read-only. Do not write raw bytes into Cilium maps; the agent's cached state would diverge. The agent image may not include bpftool; this review did not verify it. Run it from the node or a debug Pod if absent.

[1.21 pre-release] --enable-bpf-stats (Helm bpf.stats.enabled, default false) enables kernel run-time statistics and adds a bpf/stats/report hive command, exposed as cilium-dbg bpf stats report (--sort avg|total|runs, --pod, --device, --json), that shows per-program run counts and average run time.

10. Datapath Plugins (beta, new in 1.20)

Datapath Plugins let an external component run its own BPF programs around Cilium's entry points without forking the datapath.

• Enable with Helm datapathPlugins.enabled=true (default false; agent --enable-datapath-plugins); state directory datapathPlugins.stateDir (default /var/run/cilium/plugins).
• Register a plugin with the CiliumDatapathPlugin CRD (cilium.io/v2alpha1): spec.attachmentPolicy: Always | BestEffort and spec.version.
• The agent talks to each plugin over gRPC on <stateDir>/<name>/plugin.sock using the PrepareCollection and InstrumentCollection calls. Cilium generates a dispatcher so plugin programs of type BPF_PROG_TYPE_EXT (freplace) run before or after Cilium's entry programs for tc, XDP and cgroup sock hooks.
• The plugin Pod needs SYS_ADMIN. Any create, update or delete of a plugin CR triggers a full datapath reinitialization. There is no cross-version compatibility guarantee. Absent in 1.18 and 1.19.

Use it for experiments and vendor integrations, not as a general extension point for production changes yet.

11. Labs

Prerequisites for all labs: a disposable Linux cluster running Cilium 1.20.2 with kube-proxy replacement, node shell access (SSM or equivalent) and kubectl. Nothing here exposes anything outside the cluster.

Lab 1: Map every Cilium program on a node

1. On a node, run sudo bpftool net show and sudo bpftool prog show | grep cil_.
2. List pinned links with sudo ls /sys/fs/bpf/cilium/endpoints/*/links /sys/fs/bpf/cilium/devices/*/links.
3. Run kc cilium-dbg bpf sha list.

Expected: cil_from_netdev on each native device, cil_from_container per lxc* device, cil_to_host/cil_from_host on cilium_host, and (overlay) cil_from_overlay/cil_to_overlay on cilium_vxlan or cilium_geneve. On kernels 6.6 or newer, per-endpoint link pins exist; on older kernels, tc filter show dev <lxc> ingress lists the legacy filter instead. Cleanup: none (read-only).

Lab 2: Policy changes do not recompile

1. Note kc cilium-dbg bpf sha list and the endpoint's map with kc cilium-dbg bpf policy get <ENDPOINT_ID>.
2. Apply a CiliumNetworkPolicy allowing a client label to TCP/80 on a test backend.
3. Re-run both commands.

Expected: a new ingress entry with the client's numeric identity appears in the backend endpoint's policy map; the template list is unchanged. Add an explicit deny rule and observe an entry with the deny flag. Cleanup: delete the policy objects. This should be rehearsed in a namespace with no other policies, because cluster-wide default-deny interacts.

Lab 3: Follow one connection through the maps

1. Create a Deployment with two replicas and a ClusterIP Service in a test namespace, and a client Pod.
2. From the client, curl the ClusterIP. In another terminal run cilium-dbg monitor -t trace -t policy-verdict --related-to <CLIENT_ENDPOINT_ID> on the client's node.
3. Run bpf lb list --frontends, --backends, and bpf ct list global | grep <backend ip>.
4. Compare tcpdump -ni eth0 port 80 in the client's netns with the Service IP.

Expected: the service frontend lists the backends; a CT entry exists for the chosen backend; with socket-LB on, the capture shows the backend IP, not the ClusterIP. Optional: with socketLB.hostNamespaceOnly=true the capture shows the ClusterIP (per-packet LB). That switch changes how traffic is load-balanced cluster-wide, so apply it only on a test cluster and revert by restoring the previous Helm value.

Lab 4: Hubble loss accounting

1. Set hubble.eventQueueSize to a deliberately small value on a test cluster (rollback: restore the previous value, which restarts agents).
2. Generate load with a traffic tool in a test namespace.
3. Query the agent metrics for hubble_lost_events_total by source.

Expected: observer_events_queue increases when the queue overflows, and the agent logs "hubble events queue is full". Perf-buffer overruns would instead appear as perf_event_ring_buffer.

Lab 5 (advanced): netkit on a fresh node group

Only on nodes with a kernel that supports netkit. Install or upgrade a new node group with bpf.datapathMode=auto, then check cilium-dbg status for configured versus operational mode, ip -d link show for the device type, and bpftool net show for netkit attachments. Setting bpf.tproxy=true together with netkit makes the agent refuse netkit in 1.19 and later (auto falls back to veth). Rollback: do not switch existing veth Pods in place; drain and replace nodes.

Version Delta

Item	1.18.14	1.19.8	1.20.2	1.21.0-pre.3
Minimum kernel (docs)	5.10 (4.18 on RHEL 8.10)	5.10	5.10	5.10
bpf_network.c	present	present	removed	removed
CILIUM_CALL_SIZE	48	49	50	50
Config variables (grep count)	18	43	73	104
Policy map name	cilium_policy_v2_	cilium_policy_v2_	cilium_policy_v3_	cilium_policy_v3_ (auth bits reserved)
Node map	v1 + v2	v2 only	v2	v2
Egress gateway map	..._v4	..._v4	..._v4_v2	..._v4_v2
cilium_devices map	no	no	yes	yes
--datapath-mode values	veth, netkit, netkit-l2	same	+ auto	+ auto
bpf.tproxy	beta (no netkit check found)	beta; refuses netkit	beta; refuses netkit	beta; refuses netkit
Datapath Plugins	no	no	yes (beta)	yes
Mutual auth map (--bpf-auth-map-max)	yes	yes	yes, mutual auth deprecated	removed
--enable-bpf-stats	no	no	no	yes
--node-port-mode, --node-port-algorithm	present	present	removed (use --bpf-lb-mode, --bpf-lb-algorithm)	removed
Policy-denied ICMP response	no	no	yes (tail slots 48-49)	yes

The 1.21 upgrade notes in the pre-release tree still carry the 1.20 text, so no authoritative 1.21 delta exists yet.

Troubleshooting

Symptom	Likely cause	Check
Drops with DROP_EP_NOT_READY (code -203)	policy slot empty while an endpoint is being created or deleted	cilium-dbg endpoint list; transient during churn, persistent means regeneration failing
"Missed tail call" drops	array not populated or commit ordering broken after a failed reload	agent log for load errors; cilium-dbg endpoint log <id>; restart the agent Pod
New connections fail, cilium_bpf_map_pressure near 1 for a CT map	CT/NAT map full	cilium-dbg map list -v; raise bpf.mapDynamicSizeRatio or the explicit max; keep NAT at most 2/3 of CT; remember resize drops entries
Services stop matching after growth	LB map limit reached	bpf.lbMapMax (affects six maps); resize is disruptive
Agent will not start, "Require support for tcx links" or helper errors	kernel below baseline	uname -r, cilium-dbg status; move to a supported AMI
Programs on legacy tc instead of tcx	kernel older than 6.6 or enableTCX: false	bpftool net show, tc filter show
netkit refused	missing prerequisite (host routing, BPF masquerade, KPR, tproxy, kernel)	agent log reason; cilium-dbg status configured vs operational mode
Verifier rejection at load	unusual feature combination or kernel limit	full verifier log in agent stderr; compare with the 5.10/6.1/net-next tested builds
hubble observe misses flows	perf-buffer or observer-queue loss, aggregation too high, ring too small	hubble_lost_events_total by source; bpf.monitorAggregation; hubble.eventBufferCapacity
Old guide says dump cilium_tunnel_map	map removed	use cilium-dbg bpf ipcache list
Pod -> NodePort breaks after upgrade to 1.20 with socket-LB off or host-namespace-only	LB now at client Pod tc hook	allow in client egress and backend ingress policies

Validation Limits and Sources

The labs were not executed against a live cluster in this review; commands and flags were checked against upstream source and documentation. IDs, IPs and identities in the trace are illustrative. Source facts were read from the Cilium v1.20.2 tree (for example bpf/bpf_lxc.c, bpf/lib/tailcall.h, bpf/lib/policy.h, bpf/lib/eps.h, pkg/datapath/loader/tc.go, pkg/bpf/collection.go, Documentation/cmdref/cilium-agent.md); entry-line numbers move between releases, so verify against your exact tag. Not verified: whether bpftool ships in the agent image; the exact kernel version that introduced the bpf_sock_destroy kfunc; the [1.21 pre-release] stats report was read from source (cilium-dbg/cmd/bpf_stats_report.go) but not run.

• Cilium v1.20.2 system requirements: https://github.com/cilium/cilium/blob/v1.20.2/Documentation/operations/system_requirements.rst
• Maps reference: https://github.com/cilium/cilium/blob/v1.20.2/Documentation/network/ebpf/maps.rst (note: it writes cilium_ct_{4,6}_global; the real names are cilium_ct4_global and cilium_ct6_global)
• BPF sources: https://github.com/cilium/cilium/tree/v1.20.2/bpf
• Datapath loader: https://github.com/cilium/cilium/tree/v1.20.2/pkg/datapath/loader
• Agent flags: https://github.com/cilium/cilium/blob/v1.20.2/Documentation/cmdref/cilium-agent.md
• Datapath Plugins: https://github.com/cilium/cilium/blob/v1.20.2/Documentation/contributing/development/plugins.rst
• Upgrade notes: https://github.com/cilium/cilium/blob/v1.20.2/Documentation/operations/upgrade-current.inc
• Performance tuning (netkit, BIG TCP): https://github.com/cilium/cilium/blob/v1.20.2/Documentation/operations/performance/tuning.rst
• [1.21 pre-release] BPF stats: https://github.com/cilium/cilium/tree/v1.21.0-pre.3/pkg/bpf/stats

Quiz

Test your understanding with the Part 8 quiz.