vLLM vs SGLang

vLLM vs SGLang — Detailed Comparison of Two LLM Serving Engines · Status: 2026 snapshot — both projects ship releases every few weeks; always verify against the latest release notes. Real performance…

By pierreneter · 11 Nov 2026

#llm #inference #ai #comparison

On this board

  • vLLM vs SGLang — Detailed Comparison of Two LLM Serving Engines
  • Status: 2026 snapshot — both projects ship releases every few weeks; always verify against the latest release notes. Real performance depends on version, model, hardware and workload.
  • Criteria
  • vLLM
  • SGLang
  • vLLM — PagedAttention (illustration)
  • A0
  • A1
  • A2
  • 1. Origin
  • UC Berkeley Sky Computing Lab (SOSP 2023 paper). Now a PyTorch Foundation project with the largest community in LLM serving.
  • LMSYS (team behind Vicuna / Chatbot Arena; Berkeley, Stanford...), released Jan 2024 (NeurIPS 2024 paper). Contributions from xAI, AMD, NVIDIA, LinkedIn and others.
  • Request A (logical)
  • Block table: maps logical blocks → physical blocks (like an OS page table)
  • B0
  • B1
  • Request B (logical)
  • 2. Design philosophy
  • General-purpose serving engine: high throughput, easy to use, broadest model and hardware coverage. The de-facto industry default.
  • Co-design of a frontend language + fast runtime (SRT). Optimized for "LLM programs": chained calls, agents, multi-turn chat, structured output.
  • 3. Core innovation
  • PagedAttention: splits the KV cache into fixed-size blocks like virtual memory (page table) → < 4% memory waste, larger batches, block sharing for parallel sampling.
  • RadixAttention: stores the KV cache in a radix tree → automatic reuse of any shared prefix across requests, including branching structures.
  • 4. Prefix caching
  • Automatic Prefix Caching: per-block hashing (chained with previous blocks), on by default in V1, LRU eviction. Matches only at block boundaries.
  • Radix tree matches the longest prefix by token (or page), LRU eviction from leaves, cache-aware scheduling (longest-prefix-match first) → high hit rate.
  • free
  • Physical KV cache memory on GPU (fixed-size blocks, e.g. 16 tokens)
  • 5. Scheduler & batching
  • Continuous batching, chunked prefill (default in V1). V1 engine: separate scheduler / API server / workers in multiple processes, low CPU overhead, persistent batch.
  • Zero-overhead batch scheduler: CPU scheduling of the next batch overlaps with the current GPU step. Continuous batching, chunked prefill, cache-aware policy.
  • • Blocks need not be contiguous → almost no fragmentation (< 4% waste) • Allocated on demand as decoding produces new tokens • Copy-on-write: blocks shared across branches (beam search, n > 1) • Prefix caching: full blocks are hashed → other requests with the same prefix reuse them
  • 6. Structured output
  • Guided decoding: xgrammar (default), guidance, outlines. JSON schema, regex, choice, EBNF grammar. Heavily optimized in V1.
  • Compressed FSM + jump-forward decoding (skips deterministic token runs). Backends xgrammar / outlines / llguidance, mask computation overlapped with GPU → often faster for JSON.
  • 7. API & frontend
  • OpenAI-compatible server (chat, completions, embeddings, tool calling), Python LLM class for offline batch inference. No dedicated DSL.
  • SGLang DSL (gen, select, fork, join, regex...) for complex prompt programs + OpenAI-compatible server, native /generate API, offline Python Engine.
  • SGLang — RadixAttention (illustration)
  • 8. Parallelism
  • Tensor (TP), Pipeline (PP), Data (DP), Expert parallel (EP) for MoE, context parallel. Multi-node via Ray or multiprocessing.
  • TP, PP, DP, EP (DeepEP, EPLB), DP-Attention (DP for attention + TP/EP for MoE, very effective with DeepSeek MLA). Native multi-node.
  • System prompt (shared)
  • 9. Prefill/decode disaggregation
  • Disaggregated prefill/decode via KV Connectors (LMCache, NIXL, P2P NCCL); integrates with llm-d and NVIDIA Dynamo. Maturing quickly.
  • Native PD disaggregation (Mooncake, NIXL). Published DeepSeek-V3 deployment on 96 H100s with large-scale EP, close to DeepSeek's own reported throughput.
  • Few-shot examples
  • RAG document X
  • 10. Speculative decoding
  • EAGLE / EAGLE-3, Medusa, MTP (DeepSeek), n-gram (prompt lookup), small draft model.
  • Deeply optimized EAGLE-2 / EAGLE-3 (CUDA graphs for the draft), MTP for DeepSeek, n-gram, draft model. Typically strong decode speed-ups.
  • User 1 question
  • User 2 question
  • Turn 1 + answer
  • User 3 question
  • 11. Quantization
  • Broadest coverage: FP8, INT8, AWQ, GPTQ (Marlin kernels), GGUF, bitsandbytes, compressed-tensors, MXFP4, NVFP4, FP8 KV cache...
  • FP8 (incl. DeepSeek block-wise), INT8, AWQ, GPTQ, Marlin, GGUF, bitsandbytes, torchao, MXFP4, NVFP4, FP8 KV cache. Some kernels shared with vLLM.
  • Turn 2 (new)
  • 12. Kernels & optimization
  • FlashAttention 2/3, FlashInfer, Triton, FlashMLA; CUDA graphs + torch.compile (default in V1), custom all-reduce.
  • FlashInfer (original backend), FA3, Triton, FlashMLA, CUTLASS MLA; own sgl-kernel library; CUDA graphs + optional torch.compile.
  • • Each edge = a token segment plus its KV cache • A new request walks down the tree, matches the longest prefix → only the rest is prefilled • LRU eviction from leaves; nodes in use are locked (ref count) • Scheduler favors requests with long prefix matches → high hit rate (multi-turn, agents, few-…
  • 13. Hardware
  • Broadest: NVIDIA, AMD ROCm, Intel Gaudi & GPUs, Google TPU, AWS Neuron (Inferentia/Trainium), CPUs x86/ARM/PowerPC; plugins for Ascend NPU and more.
  • NVIDIA (best optimized, Hopper/Blackwell), AMD ROCm (MI300X/MI355), Intel Xeon CPUs, Ascend NPU, TPU (SGLang-Jax). Narrower than vLLM.
  • 14. Supported models
  • 200+ architectures: decoder LLMs, MoE, multimodal (image, video, audio), embedding, reward, cross-encoder, encoder-decoder. Day-0 support for most new models.
  • Popular families: Llama, Qwen, DeepSeek, Gemma, Mistral, GLM, Kimi, VLMs (Qwen-VL, LLaVA...), embedding, reward; frequent day-0 for DeepSeek/Qwen. Fewer than vLLM.
  • 15. Multi-LoRA
  • Multi-LoRA serving (Punica kernels), dynamic adapter load/unload via API, compatible with prefix caching.
  • Multi-LoRA batching (S-LoRA, Punica), Triton/CSGMV backends, dynamic adapter load/unload.
  • Quick summary
  • vLLM = "the Linux of LLM serving" General-purpose, broadest model & hardware support, largest ecosystem — the safe default for production.
  • SGLang = "the speed specialist" Deeply optimized for prefix sharing, structured output, agents and large MoE (DeepSeek) on NVIDIA/AMD; low latency.
  • 16. Performance

More boards

Start a board free