Packet Flow Comparison: Direct vs IPIP vs VXLAN

Packet Flow Comparison: Direct vs IPIP vs VXLAN A workflow diagram generated by Archify. 01 / Direct mode (no overlay) 02 / IPIP mode 03 / VXLAN mode Pod A · pod MTU 1500 · Direct mode (no overlay) Pod A pod MTU 1500 Physical Network · eth0 · native IP · Direct mode (no overlay) · 0 B overhead Physical Network eth0 · native IP 0 B overhead Pod B · Direct mode (no overlay) Pod B Pod A · pod MTU 1480 · IPIP mode Pod A pod MTU 1480 Physical Network · eth0 · IP protocol 4 · IPIP mode · +20 B Physical Network eth0 · IP protocol 4 +20 B Pod B · IPIP mode Pod B Pod A · pod MTU 1450 · VXLAN mode Pod A pod MTU 1450 Physical Network · eth0 · UDP 4789 · VXLAN mode · +50 B Physical Network eth0 · UDP 4789 +50 B Pod B · VXLAN mode Pod B no encap tunl0 decap tunl0 encap VTEP decap VTEP encap Legend Pod (workload endpoint) Physical network via node eth0

Direct

  • • Throughput 9.41 Gbps, p99 latency 45 µs
  • • 2.1% CPU per Gbps
  • • Needs L2 adjacency or BGP peering

IPIP

  • • Throughput 9.12 Gbps, p99 latency 52 µs
  • • 2.8% CPU per Gbps
  • • Firewalls may need IP protocol 4 allowed

VXLAN

  • • Throughput 8.89 Gbps, p99 latency 61 µs
  • • 3.4% CPU per Gbps
  • • UDP pass-through, better hardware offload