Advertising Remote Pod CIDRs with BGP

Advertising Remote Pod CIDRs with BGP An architecture diagram generated by Archify. Hybrid node 10.80.0.2 · Pod CIDR 10.85.1.0/24 · CNI · Remote Node CIDR 10.80.0.0/16 · Remote Pod CIDR 10.85.0.0/16 Hybrid node 10.80.0.2 Pod CIDR 10.85.1.0/24 · CNI Hybrid node 10.80.0.3 · Pod CIDR 10.85.2.0/24 · CNI · Remote Node CIDR 10.80.0.0/16 · Remote Pod CIDR 10.85.0.0/16 Hybrid node 10.80.0.3 Pod CIDR 10.85.2.0/24 · CNI On-prem router · Routing table · Architecture component On-prem router Routing table Gateway · On-prem ↔ AWS link · VPC 10.0.0.0/16 · Architecture component Gateway On-prem ↔ AWS link · VPC 10.0.0.0/16 BGP UPDATE 10.85.1.0/24 BGP UPDATE 10.85.2.0/24 10.0.0.0/16 static Remote Node CIDR 10.80.0.0/16 · Remote Pod CIDR 10.85.0.0/16 Legend Backend Cloud

How BGP works here

  • • Each node peers with the router over BGP and advertises its own pod CIDR
  • • Node 10.80.0.2 announces 10.85.1.0/24 via 10.80.0.2
  • • Node 10.80.0.3 announces 10.85.2.0/24 via 10.80.0.3

Router routing table

  • • 10.80.0.0/16 [local] via eth0 (CONNECTED)
  • • 10.0.0.0/16 [static] via Gateway (STATIC)
  • • 10.85.1.0/24 [dynamic] via 10.80.0.2 (BGP)
  • • 10.85.2.0/24 [dynamic] via 10.80.0.3 (BGP)

Trade-offs

  • • Routes update themselves when nodes are added or replaced
  • • You can drive it from the node side with Cilium's or Calico's BGP control plane
  • • It needs BGP configuration rights on the router, plus peer-state monitoring