InferencePool v1: configuration and requests

InferencePool v1: configuration and requests An architecture diagram generated by Archify. Client · HTTPS / authorization · Architecture component Client HTTPS / authorization Inference Gateway · verify implementation / configuration · Architecture component Inference Gateway verify implementation / configuration Selected model Pod · app=vllm-demo :8000 · Architecture component Selected model Pod app=vllm-demo :8000 HTTPRoute · backendRefs → InferencePool · Architecture component HTTPRoute backendRefs → InferencePool InferencePool v1 · model-pool / targetPorts:8000 · Architecture component InferencePool v1 model-pool / targetPorts:8000 EPP Service · model-epp :9002 · Architecture component EPP Service model-epp :9002 HTTPS inference proxy backendRefs endpointPickerRef selection request route configuration Legend Backend Cloud External

Configuration versus data path

  • • HTTPRoute and InferencePool are resources, not extra proxy hops.
  • • The gateway uses the EPP selection to proxy to a model Pod.
  • • Selectors match Pods in the same namespace.

Explicit prerequisites

  • • Example uses FailClose; configure EPP algorithm, authentication and TLS separately.