vLLM Monitoring and Logging

vLLM Monitoring and Logging An architecture diagram generated by Archify. Kubernetes Metrics · nodes / pods · Metrics Collection Kubernetes Metrics nodes / pods GPU Metrics · utilization / memory · Metrics Collection GPU Metrics utilization / memory vLLM Metrics · /metrics :8000 · Metrics Collection vLLM Metrics /metrics :8000 Prometheus · time-series store · Monitoring Stack · TSDB Prometheus time-series store TSDB Grafana · visualization · Monitoring Stack Grafana visualization Alertmanager · alert routing · Monitoring Stack Alertmanager alert routing Dashboards · GPU util · throughput · latency · Architecture component Dashboards GPU util · throughput · latency Alerts · latency · throughput · GPU error · OOM · Architecture component Alerts latency · throughput · GPU error · OOM Log collector · log collection · Logging Stack Log collector log collection CloudWatch Logs · /eks/vllm/logs · Logging Stack CloudWatch Logs /eks/vllm/logs Elasticsearch · log index · Logging Stack Elasticsearch log index Kibana · log search · Logging Stack Kibana log search scrape 15s container logs Metrics Collection Monitoring Stack Logging Stack Legend Frontend Backend Database Cloud External

Actual metrics

  • • vllm:e2e_request_latency_seconds / generation_tokens_total
  • • kv_cache_usage_perc:1means100%; not VRAM bytes.

Collection and protection

  • • Match Service label, named port and namespace.
  • • API keys do not protect every metrics/operational endpoint.