vLLM System Requirements

vLLM System Requirements An architecture diagram generated by Archify. NVIDIA GPU · compute capability ≥7.5 · Hardware NVIDIA GPU compute capability ≥7.5 GPU Memory · Varies by model size · Hardware · Key requirement GPU Memory Varies by model size Key requirement CPU Cores · vCPU count · Hardware CPU Cores vCPU count 7B Model · FP16 weights ≈14GB + extras · Requirements by Model Size 7B Model FP16 weights ≈14GB + extras 13B Model · FP16 weights ≈26GB + extras · Requirements by Model Size 13B Model FP16 weights ≈26GB + extras 70B Model · FP16 weights ≈140GB + extras · Requirements by Model Size 70B Model FP16 weights ≈140GB + extras CUDA · explicit cu129 / cu130 variant · Software CUDA explicit cu129 / cu130 variant Python · package / image version bounds · Software Python package / image version bounds PyTorch · pin with the image · Software PyTorch pin with the image VRAM capacity Hardware Requirements by Model Size Software Legend Software Hardware Requirements by Model Size

Memory estimate

  • • parameter count × stored bytes + KV / activation / workspace
  • • Qwen2.5-7B GQA: FP16 KV ≈56KiB/token aggregate estimate

Device and artifact

  • • p4d uses40GB A100s; distinguish80GB p4de.
  • • V1007.0 is below the current NVIDIA7.5 minimum.