vLLM Architecture

vLLM Architecture An architecture diagram generated by Archify. Continuous Batching · queues / token budgets apply · Key Features Continuous Batching queues / token budgets apply Distributed Inference · Tensor-parallel sharding · Key Features Distributed Inference Tensor-parallel sharding PagedAttention · KV cache memory management · Key Features PagedAttention KV cache memory management Quantization · weight / KV formats differ · Key Features Quantization weight / KV formats differ High Throughput · workload-dependent benefit · Key Benefits High Throughput workload-dependent benefit Scalability · model / backend constraints · Key Benefits Scalability model / backend constraints Memory Efficiency · workload-dependent benefit · Key Benefits Memory Efficiency workload-dependent benefit Request Scheduler · Dynamic batching · Core Components Request Scheduler Dynamic batching Model Loader · Loads model weights · Core Components Model Loader Loads model weights Inference Engine · vLLM Engine · Core Components Inference Engine vLLM Engine KV Cache Manager · Block-level KV cache · Core Components KV Cache Manager Block-level KV cache API Server · OpenAI-compatible API · Core Components API Server OpenAI-compatible API batched requests weights KV blocks requests Key Features Key Benefits Core Components Legend Core Components Key Benefits Key Features

Execution boundaries

  • • Capabilities do not guarantee fixed speedups or queue-free responses.
  • • Model Runner V2 is an internal runner, not an API version.