LLM Inference Framework Landscape

LLM Inference Framework Landscape An architecture diagram generated by Archify. LiteLLM · Multi-provider gateway · Dev / Gateway Tools LiteLLM Multi-provider gateway Ollama · local runtime choices · Dev / Gateway Tools Ollama local runtime choices SGLang · RadixAttention · structured output · Open Source Frameworks SGLang RadixAttention · structured output AIBrix · adapter / routing / scaling · Open Source Frameworks AIBrix adapter / routing / scaling vLLM · Open-source serving engine · Open Source Frameworks vLLM Open-source serving engine Ray Serve · Distributed inference at scale · Open Source Frameworks Ray Serve Distributed inference at scale NVIDIA NIM · Inference microservices · NVIDIA Stack NVIDIA NIM Inference microservices TensorRT-LLM · Inference engine · NVIDIA Stack TensorRT-LLM Inference engine NVIDIA Dynamo · Disaggregated serving · routing · NVIDIA Stack NVIDIA Dynamo Disaggregated serving · routing KubeRay Operator · Ray cluster management · Orchestration Layer KubeRay Operator Ray cluster management Karpenter · node provisioning / reclamation · Orchestration Layer Karpenter node provisioning / reclamation Neuron SDK · Inferentia2 · Inference on AWS silicon · AWS Native Neuron SDK · Inferentia2 Inference on AWS silicon request routing orchestration Dev / Gateway Tools NVIDIA Stack Open Source Frameworks Orchestration Layer AWS Native Legend LLM gateway Serving framework Provisioning / AWS silicon Local dev tool

Selection criteria

  • • No fixed speedup or savings is guaranteed by a feature.
  • • Neuron:distinguish Inf2 NxD from the new Trn2/3 beta.
  • • TGI is in maintenance mode; assess migration for existing systems.