# vLLM full site content for AI assistants
---
---
This file concatenates core vLLM site descriptions and vLLM blog markdown content for AI assistant retrieval.
---
---
# vLLM
Source: https://vllm.ai/
vLLM is a high-throughput, memory-efficient inference and serving engine for large language models. The project focuses on production-grade serving performance, broad model support, and an open ecosystem for LLM inference.
---
# vLLM Blog
Source: https://vllm.ai/blog
The vLLM blog publishes technical deep dives, release announcements, model support guides, benchmarks, and community updates from the vLLM project.
---
# vLLM Events
Source: https://vllm.ai/events
The events page tracks vLLM office hours, community meetups, conferences, release events, and other project gatherings.
---
# vLLM Releases
Source: https://vllm.ai/releases
The releases page lists recent vLLM releases and links to release notes for users tracking new versions.
---
# Contact vLLM
Source: https://vllm.ai/contact
The contact page provides channels for community questions, website feedback, partnerships, collaborations, and other vLLM project communications.
---
# Following the Bottleneck: Optimizing MiniMax M3 on AMD Instinct MI355X
Source: https://vllm.ai/blog/2026-09-10-minimax-m3-mi355x
Published: 2026-09-10
Authors: AMD and Embedded LLM Teams
Tags: hardware, performance, moe, speculative-decoding, disaggregation
Summary: A performance model for LLM serving: inspect local shapes, remove repeated work, verify data movement and dispatch, then follow the queue.
The [MiniMax M3 day-0 post](/blog/2026-06-12-minimax-m3-vllm) described the first working vLLM implementation: MiniMax Sparse Attention (MSA), multimodal inputs, reasoning and tool outputs, MXFP8 weights, and EAGLE3 on AMD Instinct MI355X.
This follow-up is about what happened after the model ran. The useful result is not only a higher throughput number. It is a way to decide what to optimize next when the bottleneck keeps moving.
## Results in one minute
Public results from the [SemiAnalysis InferenceX benchmark](https://inferencex.semianalysis.com/inference) show what MiniMax-M3 now delivers on AMD Instinct MI355X:
- At concurrency 32, fixed-topology MXFP8 standard serving rose from **109.1 to 342.4 output tokens/s/GPU**, or **3.14×** the day-0 result. Median TTFT fell from 1.46 to 0.67 seconds, and mean TPOT fell from 69.1 to 22.1 milliseconds.
- At concurrency 128, the same TP4/EP1 four-GPU path rose from **297.8 to 623.7 output tokens/s/GPU**, or **2.09×**. Median TTFT fell from 3.53 to 1.54 seconds, and mean TPOT fell from 100.7 to 48.8 milliseconds.
- MXFP4 first rose from **212.1 to 716.8 output tokens/s/GPU** at concurrency 128 under the same TP4/EP1 four-GPU contract. A later TP2/EP1 result reached **943.5 output tokens/s/GPU**, **31.6%** above that TP4 checkpoint and **4.45×** the initial per-GPU result.
- We added EAGLE3 speculative decoding, reaching **682.4 output tokens/s/GPU** at concurrency 128 on TP4/EP1.
- We added P/D disaggregation and retuned the prefill/decode topology, reaching **6,370.5 total tokens/s/GPU** at concurrency 512 with **1.32 seconds median TTFT**.
_Figure 1. Standard decoding at 8K input and 1K output. Compare points within a row. MXFP8 stays on TP4/EP1; the last MXFP4 point moves from TP4 to TP2, so it shows higher deployment density, not a fixed-topology speedup._
Each checkpoint is cumulative and can bundle several changes. The sections below use isolated PR measurements to explain individual optimizations.
## One decode step, five questions
The optimization work followed a familiar workflow: estimate the dominant costs, measure them, batch repeated work, precompute invariants, and move up the stack when leaf profiles flatten.
MiniMax M3 has 60 decoder layers; 57 use sparse MoE and sparse attention. For a 1K-token response, a small per-layer cost can appear tens of thousands of times in one request. That makes five questions useful:
1. What local shape reached this rank?
2. What work repeats at every layer or token?
3. Which bytes move, and can metadata move instead?
4. Did the intended fast path run, with the intended math?
5. When kernels are no longer dominant, which queue grows?
_Figure 2. The top lane follows one sparse decoder layer in execution order; AR marks the tensor-parallel collectives after attention and MoE. The lower lane applies the same reasoning to P/D: validate the KV handoff, then add capacity where requests wait._
The rest of this post answers those questions with code and benchmark evidence.
## 1. What shape reached this rank?
Model diagrams show global dimensions, but kernels run on local M, N, and K after tensor parallelism, head replication, padding, and token routing. At TP8, MiniMax M3's 64 query heads shard to eight per rank, while its four KV and four index heads replicate to one per rank. The fused QKV projection therefore sees local N=1536—not global N divided by eight.
Prefill and decode also arrive with different M. Prefill processes many tokens at once; decode often has only a few rows. [vLLM #45725](https://github.com/vllm-project/vllm/pull/45725) split the launcher into large-M and small-M regimes, improving TP8 8K/1K output throughput by 7.8%–9.4%. [vLLM #46117](https://github.com/vllm-project/vllm/pull/46117) then selected tiles from the full local shape: narrower N tiles exposed more independent work in decode, while a larger K step reduced loop iterations. Prefill used wider tiles when M already supplied enough parallelism.
_Figure 3. TP sharding and head replication determine local N; the serving phase determines M. The launcher selects tiles for the shape each rank actually runs._
The same PR reordered grouped-MoE programs so neighboring programs could reuse activation rows and expert-weight tiles from GPU cache instead of fetching them again from HBM. Across its TP4 end-to-end tests, the combined changes produced 1.08×–1.46× gains. The benefit was largest at low concurrency, where the original launch had the least parallel work.
Local shape also decides which backend is legal. The AITER sparse-attention path used in the final MXFP8 recipe requires one KV head per TP rank. TP4 satisfies that condition. TP2 uses vLLM's Triton fallback. Changing TP therefore changes more than collective size; it can change the operator graph.
There was no universally fastest backend, either. [InferenceX #2003](https://github.com/SemiAnalysisAI/InferenceX/pull/2003) initially selected an emulated linear backend for the whole sweep. Later measurements in [InferenceX #2187](https://github.com/SemiAnalysisAI/InferenceX/pull/2187) showed that native MXFP8 linear was faster at low and middle concurrency, while emulation won only for long-input, high-concurrency runs. Sparse paged attention had a similar crossover. The final recipe enables both only for 8K input at concurrency 64 or higher.
That is the first reusable lesson: tune and dispatch on the shape distribution that actually runs. “Prefill,” “decode,” or “TP4” is only a label.
## 2. What work repeats?
The first easy-to-miss repeated cost was launch overhead. The day-0 recipe ran eagerly. [InferenceX #1754](https://github.com/SemiAnalysisAI/InferenceX/pull/1754) and [#1755](https://github.com/SemiAnalysisAI/InferenceX/pull/1755) enabled graph execution for standard and EAGLE3 serving. That change is part of the cumulative history, although the published results do not isolate its gain.
The larger structural win was the shared expert. Originally, every sparse-MoE layer ran it as a separate dense MLP: gate/up projection, activation, down projection, intermediate storage, and addition. The math was required. The separate path was not.
[vLLM #46545](https://github.com/vllm-project/vllm/pull/46545) appended the shared expert to the routed expert table and selected it for every token. The grouped GEMMs then handled routed and shared experts together. This removed launches and intermediate traffic without removing model FLOPs. Output throughput improved 30.2% at concurrency 1 and 5.6% at concurrency 128. The shrinking gain is useful evidence: it is what launch amortization looks like.
_Figure 4. Fusion appends the shared expert as a slot selected by every token. Routed and shared experts then use the same grouped GEMMs, preserving the model math while removing a separate MLP path and its intermediate traffic._
The AITER path applied the same idea in [vLLM #46474](https://github.com/vllm-project/vllm/pull/46474). [vLLM #46184](https://github.com/vllm-project/vllm/pull/46184), backed by [AITER #3811](https://github.com/ROCm/aiter/pull/3811), also moved MXFP8 weight and scale reshuffling to model load. AITER carries tuned MoE configurations from one to 32,768 tokens and for the local intermediate widths produced by TP4 and TP8. Layout conversion happens once; the serving loop consumes the prepared form.
Speculative decoding supplied a clean example of batching repeated work. The original MSA indexer launched one workgroup per speculative token. [vLLM #45743](https://github.com/vllm-project/vllm/pull/45743) launched one workgroup per request and processed all draft positions together, reusing key loads. It also removed a positive score scale because only top-k order matters and multiplying every score by the same positive constant cannot change that order.
The index kernel improved by as much as 48.9%, while end-to-end serving improved by about 3.3% in the PR tests. That gap is Amdahl's law doing its job: a large kernel win can be real without being the application's whole critical path.
## 3. Which bytes move?
Sparse attention reduces attention math, but it adds a control plane: score blocks, select top-k, map logical blocks to physical pages, and pass that metadata to the attention kernel.
[vLLM #47269](https://github.com/vllm-project/vllm/pull/47269) observed that adjacent sparse layers often selected nearly the same blocks. With index sharing enabled, one layer computes the top-k decision and later layers reuse it. Mean TPOT fell by about 10% at concurrency 1 and about 4% at high concurrency.
Skipping the selector was only half the job. The fused projection still produced index Q/K values, normalized them, applied RoPE, and wrote the index cache. [vLLM #47287](https://github.com/vllm-project/vllm/pull/47287) made the reuse decision visible to that fused kernel, so the unused producer branch is compiled away.
The same PR integrated AITER sparse paged attention across a layout mismatch. MiniMax M3 selects logical 128-token blocks; AITER consumes 16-token pages. Instead of copying KV data, vLLM turns each selected block ID into eight page IDs and builds a compact page table. The KV cache stays where it is. This is the serving equivalent of passing a view instead of copying a container.
_Figure 5. Each selected 128-token block resolves to a physical block, then expands into eight 16-token page entries. AITER reads them through a view of the existing KV allocation; only the table is rebuilt._
At TP4, concurrency 256, the PR improved output throughput by 6.93% for MXFP4 and 5.56% for MXFP8 in its isolated A/B.
There is an important benchmark boundary here. InferenceX's fixed 8K/1K review policy excluded cross-layer index reuse because it reduces architecture work. The fixed-shape recipe uses the page adapter but not top-k reuse. AgentX enables reuse under its workload rules. We do not credit the fixed-contract curve with work it did not run.
Quantization makes the “which bytes?” question even more important. It is helpful to separate three planes:
| Byte plane | MiniMax M3 example | What must be proved |
| --- | --- | --- |
| Weights and activations | MXFP8 or MXFP4 GEMM/MoE | Packing, scales, activation math, backend layout |
| Persistent state | FP8 KV and sparse index cache | Platform dtype, page layout, read/write geometry |
| Communication | Quantized all-reduce or KV transfer | Eligibility, selected codec, ownership, completion |
These planes have independent dispatch and correctness contracts. “The model is MXFP4” does not tell us the KV dtype or the collective path.
## 4. Did the fast path run—and was it correct?
This audit changed one claim in the post. We initially believed a roughly 1.5 MB decode collective used INT4 QuickReduce. The available evidence does not prove it.
[InferenceX #2104](https://github.com/SemiAnalysisAI/InferenceX/pull/2104) configured INT4 and a 256 KB **codec** threshold, but not QuickReduce's separate eligibility threshold. For BF16 at TP4, the [pinned built-in table](https://github.com/vllm-project/vllm/blob/69715823df89b11ee684b84066390cbb9092d5c1/vllm/distributed/device_communicators/quick_all_reduce.py#L49-L61) requires 16 MB for INT4. The 1.5 MB collective therefore does not reach codec selection; the 256 KB threshold is consulted only after QuickReduce is eligible.
_Figure 6. QuickReduce checks eligibility before choosing FP or INT4. Here the collective falls below the built-in eligibility gate, so configuration alone cannot establish execution._
The available logs prove that INT4 was configured, not that the QuickReduce kernel ran. We therefore treat #2104 as a cumulative image and recipe checkpoint and do not attribute its curve to INT4 all-reduce.
This distinction generalizes:
```text
configured != eligible != executed
```
Use a dispatch trace or profiler before assigning a gain to a backend.
Correctness needs the same discipline. Three examples caught different broken contracts:
- [vLLM #45794](https://github.com/vllm-project/vllm/pull/45794) mapped packed MXFP4 Q/K/V and gate/up checkpoint tensors into the correct fused-parameter slices and passed MiniMax M3's SwiGLU-OAI parameters into MoE.
- [vLLM #45720](https://github.com/vllm-project/vllm/pull/45720) fixed the FP8 KV view on FNUZ ROCm devices. On MI300X, the unpatched path scored 0.0099 strict match on GSM8K; the patched path scored 0.9575. This was a correctness fix, not a claimed MI355X speedup.
- [vLLM #47158](https://github.com/vllm-project/vllm/pull/47158) fixed the expert-parallel mask passed to AITER. The buggy path had cosine similarity 0.527; the corrected path reached 1.0 and restored GSM8K accuracy.
The last two examples do not explain the TP4/EP1 hero curve; they expose contracts that other configurations must satisfy. For performance work, “passed” should mean three things: the output is correct, the intended path executed, and the end-to-end metric improved under the same contract.
## EAGLE3 adds a second decode loop
EAGLE3 adds a draft model, multi-token verification, acceptance behavior, and a second set of attention metadata. It cannot be treated as a flag on the standard curve.
[vLLM #45546](https://github.com/vllm-project/vllm/pull/45546) connected the AMD model to the EAGLE3 interface. Then [vLLM #45564](https://github.com/vllm-project/vllm/pull/45564) fixed a subtle cache-key bug: the target and draft use different query-head counts, so they must not share an attention-group builder merely because their backend and KV type match.
That is a general cache rule: the key must contain every invariant that changes the cached object.
After the request-level index batching described above, [InferenceX #2107](https://github.com/SemiAnalysisAI/InferenceX/pull/2107) found that the target's attention-backend setting did not configure the draft. Pinning TRITON_ATTN inside the speculative config avoided the draft's slower fallback.
Finally, [vLLM #47984](https://github.com/vllm-project/vllm/pull/47984) extended AITER sparse paged attention from one-token decode to multi-token verification. It maps each flattened query row back to its request and local speculative position, reuses the existing page-table builder, and preserves the one-token fast path. Its TP4 tests improved output throughput by 8.32% for MXFP4 and 7.90% for MXFP8 without materially changing acceptance.
Together, this work produced the separate 682.4 output tok/s/GPU EAGLE3 result at concurrency 128.
## 5. Which queue grows?
Prefill/decode disaggregation moved the bottleneck above one process. Before tuning worker counts, the KV boundary had to be correct.
The initial MoRIIO path assumed that the first layer's KV layout represented every layer. MiniMax M3 has separated K/V tensors, interleaved K/V tensors, and a key-only index cache. The transfer completed and throughput looked healthy, but GSM8K fell to roughly 0.0008—effectively token salad.
The repair came in three steps:
- [vLLM #46039](https://github.com/vllm-project/vllm/pull/46039) derived transfer geometry and byte offsets per layer.
- [vLLM #46290](https://github.com/vllm-project/vllm/pull/46290) counted the writes actually scheduled for each request, sealed that count after forward, and released buffers only after those writes completed.
- [vLLM #46332](https://github.com/vllm-project/vllm/pull/46332) added heterogeneous-TP rank mapping and acknowledgment fan-in. With prefill TP4 and decode TP8, two decode ranks can consume one producer rank, so both must acknowledge before its blocks are reused.
Only then was worker allocation worth tuning.
_Figure 7. Two 8K/1K P/D operating points. The endpoints use different GPU counts and concurrency, so this is system evolution, not a controlled speedup._
The first public profile used one TP8 prefill worker and one TP8 decode worker. At concurrency 1024 it reached 2,084.6 total tok/s/GPU, but median TTFT was 223.20 seconds ([InferenceX #1762](https://github.com/SemiAnalysisAI/InferenceX/pull/1762)). The throughput number did not make that system usable; the prompt queue was the signal.
[InferenceX #2144](https://github.com/SemiAnalysisAI/InferenceX/pull/2144) moved every worker to TP4, synchronized them with the faster single-node recipe, and searched the prefill/decode ratio. For 8K/1K, two TP4 prefill workers fed one TP4 decode worker. At concurrency 512, the result reached 6,370.5 total tok/s/GPU and 1.32 seconds median TTFT.
Mean TPOT moved from 31.26 to 54.60 milliseconds. That is not a contradiction. Added prefill capacity cleared the admission queue, while the selected decode operating point produced each active sequence more slowly. P/D has at least two latency objectives. Publish both.
One high-concurrency run also exhausted the container's file-descriptor limit. Raising `nofile` fixed the TCP failures. Once a profile moves up the stack, an OS limit can be as real as a GEMM tile.
## AgentX shows the next bottleneck
Fixed 8K/1K is excellent for controlled comparisons. Agentic coding is not fixed-shape traffic: it has long, multi-turn traces, reusable prefixes, irregular outputs, and a KV-capacity knee.
[InferenceX #2487](https://github.com/SemiAnalysisAI/InferenceX/pull/2487) is the first MI355X MiniMax M3 AgentX point with MXFP4, EAGLE3-GQA, prefix caching, optional TP-sharded LMCache, and cross-layer index reuse. Throughput replay uses a committed synthetic acceptance length so compared systems do the same speculative work; eval uses real target verification.
In the [successful run](https://github.com/SemiAnalysisAI/InferenceX/actions/runs/31558297538), TP4 at concurrency 28 delivered **127.4 output tok/s/GPU**, **509.5 total output tok/s**, **0.582 mean QPS**, **645 ms p50 TTFT**, and **41.3 ms p50 TPOT**.
The service metrics make this more than another score:
- Theoretical prefix-cache hit rate: 96.7%
- Realized GPU cache hit rate: 92.1%
- GPU KV-cache use: 88.5%
- GPU KV capacity: 6,264,960 tokens
At this point, another GEMM is not automatically the best next project. The 4.6-point cache-realization gap and the near-capacity operating point direct attention toward prefix alignment, admission and eviction policy, scheduling, and offload. These are observations from one run, not yet an optimization claim. We will use this point as the baseline for the next round of agentic optimization.
## A checklist for the next model
When a new serving path works but is not yet fast:
1. Record local shape histograms after sharding. Include replicated heads and routed-token counts.
2. Estimate repetition. Multiply per-layer work by layers, output tokens, and active requests.
3. Separate the byte planes: compute tensors, persistent state, and communication.
4. For every fast path, record its eligibility condition and verify its execution.
5. Put a correctness gate beside every performance gate.
6. After each win, profile again. If leaf kernels flatten, inspect queues, ownership, cache capacity, and OS limits.
That is the main result of this work. MiniMax M3 became faster because the team kept changing the level of the question—from tiles, to repeated paths, to sparse metadata, to distributed state, and finally to workload queues.
## Reproduce the fixed-shape result
The public InferenceX runs record the container images, arguments, and artifacts. The final MXFP8 TP4 checkpoint used:
```text
vllm/vllm-openai-rocm:nightly-9e57de7197f234f9d9187715d96e07e007048c0f
```
```bash
export VLLM_ENGINE_READY_TIMEOUT_S=3600
export VLLM_USE_BREAKABLE_CUDAGRAPH=0
export VLLM_ROCM_USE_AITER=1
export VLLM_ROCM_USE_AITER_FUSION_SHARED_EXPERTS=1
vllm serve MiniMaxAI/MiniMax-M3-MXFP8 \
--tensor-parallel-size 4 \
--block-size 128 \
--no-enable-prefix-caching \
--language-model-only \
--moe-backend aiter \
--max-model-len 10240 \
--max-num-batched-tokens 32768 \
--kv-cache-dtype fp8 \
--attention-backend TRITON_ATTN \
--tool-call-parser minimax_m3 \
--reasoning-parser minimax_m3 \
--enable-auto-tool-choice
```
For the exact high-concurrency dispatch used by Figure 1, use the gated recipe in [InferenceX #2187](https://github.com/SemiAnalysisAI/InferenceX/pull/2187). For MXFP4 TP2, use [#2446](https://github.com/SemiAnalysisAI/InferenceX/pull/2446); for P/D, use [#2144](https://github.com/SemiAnalysisAI/InferenceX/pull/2144). Copying only the flags above does not reproduce a different image, topology, or workload.
The concurrency-128 benchmark command is:
```bash
vllm bench serve \
--backend vllm \
--model MiniMaxAI/MiniMax-M3-MXFP8 \
--dataset-name random \
--random-input-len 8192 \
--random-output-len 1024 \
--random-range-ratio 0.8 \
--num-prompts 1280 \
--max-concurrency 128 \
--request-rate inf \
--ignore-eos \
--num-warmups 256 \
--percentile-metrics ttft,tpot,itl,e2el \
--save-result
```
## Acknowledgements
We thank MiniMax for releasing MiniMax M3 and everyone who built, optimized, and validated this serving path: [Aakif Nawaz](https://github.com/akii96), [Ajith Sirra](https://github.com/ajith-sirra-amd), [Bryan Shan](https://github.com/Oseltamivir), [Bugen Zhao](https://github.com/BugenZhao), [Cameron Quilici](https://github.com/cquil11), [Chun Fang](https://github.com/chunfangamd), [Duyi Wang](https://github.com/Duyi-Wang), [Ethan Yang](https://github.com/amd-ethany), [Fangzhou Ai](https://github.com/Fangzhou-Ai), [Felix Marty](https://github.com/fxmarty-amd), [functionstackx](https://github.com/functionstackx), [Hongxia Yang](https://github.com/hongxiayang), [Isotr0py](https://github.com/Isotr0py), [Jun Kang Chow](https://github.com/junkang1991), [Pin Siang Tan](https://github.com/tanpinsiang), [Qiang Li](https://github.com/qli88), [Seung Rok Jung](https://github.com/seungrokj), [Sun Peng](https://github.com/sunway513), [Tian Di](https://github.com/TianDi101), [Tun Jian Tan](https://github.com/tjtanaa), [Uma Kannikanti](https://github.com/ukannika), [wangjiaxin99](https://github.com/wangjiaxin99), [Ye Hur Cheong](https://github.com/vllmellm), [youkaichao](https://github.com/youkaichao), [Yue Liu](https://github.com/yueliu14), and [Zheng Gong](https://github.com/ZhengGong-amd).
We also thank the broader vLLM, AMD, Embedded LLM, Inferact, and SemiAnalysis InferenceX reviewer communities.
---
# Tiered KV Cache Offloading in vLLM
Source: https://vllm.ai/blog/2026-09-10-tiered-kv-offloading
Published: 2026-09-10
Authors: Or Ozeri, Danny Harnik, Ronen Schaffer, Itay Etelis, Varun Sundar Rabindranath
Summary: A host-centric framework for scaling KV cache across host memory, filesystems, object stores, and remote peers — reducing recomputation and increasing serving capacity.
Long-context models and multi-turn conversations generate massive KV caches.
When accelerator memory (e.g., GPU HBM) fills up, previously computed KV data is evicted.
On the next request that needs it, vLLM must recompute it from scratch.
**Tiered KV cache offloading preserves evicted KV data** across host memory, storage, and remote peers.
Instead of recomputing, vLLM reloads the data from a lower tier — saving compute, reducing latency, and increasing the effective serving capacity of the cluster.
With secondary tiers, KV data also becomes **shareable across nodes** — enabling horizontal scaling of the cache, warm-starting new instances from shared storage, and transferring KV data between peers for disaggregated serving or load balancing.
The framework has been available in vLLM since v0.22 and a detailed usage guide can be found [here](https://docs.vllm.ai/en/latest/features/kv_offloading_usage/).
---
## The Host-Centric Design
The core design principle: **all KV data flows through host memory (CPU DRAM)**.
When offloading, KV data moves from accelerator to host first.
From the host, it propagates to secondary tiers — filesystem, object storage, or remote peers.
When reloading, the flow reverses: a secondary tier promotes data into host memory, then it is loaded to the accelerator.
All KV data flows through the host primary tier. Secondary tiers extend capacity beyond what host DRAM can hold. On offload, chunks cascade to all tiers. On reload, the first tier that holds the chunk serves it.
This design yields several key advantages:
### Fast accelerator release, just-in-time allocation
Copying from accelerator to host is a fast local PCIe transfer.
**Accelerator memory is freed as soon as this copy completes** — before any secondary tier transfer starts.
Storage writes, network sends, and remote RDMA all proceed from the host copy without touching accelerator memory again.
**On reload, accelerator memory is allocated only once the data is ready in the host** — not reserved in advance while waiting for tier transfers.
Together, these create a just-in-time allocation pattern: accelerator memory is held only while actively needed.
Accelerator memory is freed at t2 — as soon as the host copy completes. Secondary tier writes continue asynchronously from the host copy.
### Consolidated I/O
In a multi-accelerator setup (e.g., `tensor_parallel_size=8`), each device holds a shard of the KV cache.
The framework **consolidates all shards into a single shared host memory region**.
Multiple accelerator shards fan into one shared host region. Secondary tiers see fewer, larger I/O operations — improving storage and network throughput.
### Canonical memory layout
The host region uses a canonical memory layout: each page stores one block of one layer, with all KV heads from across TP ranks gathered into a single contiguous region. Locating any chunk is a simple offset calculation.
The fixed host-side layout ensures correct sharing even when the GPU memory layout differs across nodes — different accelerator types, attention backends (FlashAttention, FlashInfer, Triton), or parallelism configurations all map to the same canonical representation.
Because the layout is configuration-independent, **nodes with different setups share KV data directly** — no remapping or format conversion needed.
A TP=2 node and a TP=4 node produce identical host-side chunks for the same KV data.
### Simple secondary tiers
Routing all data through host memory makes secondary tiers **easy to build and operate**.
They are a single process per vLLM instance, they transfer data using standard CPU-based libraries (POSIX I/O, S3 SDKs, RDMA verbs), and they never touch accelerator memory or APIs.
No need to coordinate across multiple processes or understand accelerator-specific memory layouts.
---
## How Offloading and Reloading Work
The unit of operation is a **chunk** — a fixed-size piece of KV data covering a group of tokens.
By default, a chunk maps to a single accelerator block.
A configurable `blocks_per_chunk` parameter allows larger chunks, yielding larger I/Os to the host and secondary tiers.
### Offload path
New KV chunks move from accelerator to host via async DMA.
**Accelerator memory is freed immediately** — before any secondary tier transfer begins.
The tiering manager then cascades chunks to **all** configured secondary tiers simultaneously, reading from the host copy.
The host primary tier is a **proper LRU/ARC cache**, not a staging buffer.
Chunks remain in host memory and serve future hits directly.
Only when host capacity is exhausted are the least-recently-used chunks evicted — and even then, they survive in whichever secondary tier received them.
### Reload path
The scheduler checks the host cache first — if the chunk is there, it is an immediate hit.
On host miss, secondary tiers are queried in configured order; the first tier that holds the chunk serves it.
The tier promotes the chunk back into host memory asynchronously; during this time, the scheduler receives a `RETRY` and re-checks on the next cycle.
Different chunks within the same request can be served by different tiers — e.g., one chunk from the filesystem, another from a remote peer.
---
## Secondary Tiers
### Filesystem
Stores each KV chunk as a file on local or networked storage.
Uses content-addressed naming — identical token sequences map to the same key, so matching inputs share cached data automatically.
When multiple vLLM instances share the same storage mount point (e.g., network-attached storage, or multiple instances on the same node), they **share KV data automatically** with no additional configuration.
Highlights:
- **Non-blocking lookups**
- **Atomic writes**
- **Separate read/write thread pools**
```bash
vllm serve Qwen/Qwen3.6-35B-A3B \
--kv-transfer-config '{
"kv_connector_extra_config": {
"spec_name": "TieringOffloadingSpec",
"cpu_bytes_to_use": 107374182400,
"secondary_tiers": [{"type": "fs", "root_dir": "/mnt/kv-cache"}]
}
}'
```
### Object Storage
Stores KV chunks in S3-compatible object stores via NIXL.
Same content-addressed scheme as the filesystem tier.
Provides a cost-effective networked storage option — typically cheaper per GB than high-performance file storage, while still enabling shared access across instances.
```bash
--kv-transfer-config '{
"kv_connector_extra_config": {
"spec_name": "TieringOffloadingSpec",
"cpu_bytes_to_use": 107374182400,
"secondary_tiers": [{
"type": "obj",
"bucket": "my-kv-cache",
"endpoint_override": "http://minio:9000"
}]
}
}'
```
### Peer-to-Peer (P2P)
Enables **cross-instance KV cache sharing** over the network.
Uses ZMQ for coordination and RDMA (via NIXL) for bulk data transfer.
All transfers are **host-to-host** — no accelerator memory involved on either side.
The P2P tier does not decide which peer to pull KV data from — that is the orchestration layer's job (e.g., a router such as [llm-d](https://github.com/llm-d/llm-d)). The orchestrator drives cross-node transfers through the request's `kv_transfer_params`. An example and more details can be found in the [usage guide](https://docs.vllm.ai/en/latest/features/kv_offloading_usage/#orchestration-layer-protocol).
```bash
--kv-transfer-config '{
"kv_connector_extra_config": {
"spec_name": "TieringOffloadingSpec",
"cpu_bytes_to_use": 107374182400,
"secondary_tiers": [{"type": "p2p", "host": "10.0.0.1", "port": 5710}]
}
}'
```
Two key use-cases:
#### Prefill/Decode disaggregation
The prefill instance computes KV chunks and makes them available in its host tier.
The decode instance pulls them from the prefiller's host memory via RDMA.
A key advantage over GPU-based P/D approaches: **consolidated I/O turns many small per-GPU transfers into fewer, larger RDMA operations** — dramatically improving network throughput.
Additionally, with *chunked prefill*, each completed prefill-chunk becomes immediately available for transfer — computation and data movement overlap, reducing time-to-first-token.
#### Load balancing
Transfer KV chunks from an overloaded vLLM instance to one with available capacity.
Any node can pull chunks from any peer.
For more on P2P KV cache sharing with llm-d, see [this blog post](https://llm-d.ai/blog/p2p-kv-cache-sharing-llm-d).
---
## Hybrid Model Support
The framework integrates with vLLM's hybrid memory allocator.
Models that combine different layer types — full attention, sliding window, MLA, Mamba — are handled transparently.
The canonical layout normalizes all KV formats into a uniform byte-buffer representation.
**Each chunk has a fixed byte size on the host**, regardless of which layer types it contains.
Different layer types pack different numbers of tokens into the same chunk — for example, Mamba state layers cover many more tokens per chunk than full-attention layers, so they are offloaded less frequently.
This means:
- **Sliding window layers** reload only the tokens within their window, not the full history
- **State-space layers** (Mamba) offload and reload their state alongside attention KV
The framework supports state-of-the-art hybrid architectures including DeepSeek V4, GLM 5.3, Nemotron 3, and others.
---
## Observability
The framework exposes Prometheus metrics via vLLM's standard `/metrics` endpoint:
- **Host cache utilization** — current fill ratio of the primary tier
- **Transfer throughput** — bytes and time for accelerator ↔ host transfers
- **Per-tier latencies** — how long lookups and data transfers take for each tier
- **Per-tier hit rates** — which tiers are serving your workload
Secondary tiers can **define custom metrics** (counters, histograms, gauges) that are automatically registered and exposed — no framework changes needed.
---
## KV Events
As chunks move between tiers, the framework emits structured **KV events** reporting which chunks were stored or evicted, from which tier, and with what locality (local vs. remote).
Secondary tiers can emit their own events.
These events enable external orchestration systems to make intelligent routing decisions.
Projects such as llm-d and [Dynamo](https://github.com/ai-dynamo/dynamo) consume KV events to **route requests to the instance most likely to have a cache hit** — achieving significantly higher throughput and lower latency compared to cache-unaware scheduling.
Additionally, llm-d uses these events to orchestrate P2P KV transfers between peers.
---
## Adding a New Secondary Tier
The secondary tier interface is minimal — four core methods:
```python
class SecondaryTierManager(ABC):
def lookup(self, key, req_context) -> LookupResult:
"""Does this tier have a chunk? Returns HIT, MISS, or RETRY."""
def submit_store(self, job_metadata: JobMetadata) -> None:
"""Start async store from host to this tier."""
def submit_load(self, job_metadata: JobMetadata) -> None:
"""Start async load from this tier to host."""
def get_finished_jobs(self) -> Iterable[JobResult]:
"""Poll completed transfers."""
```
Each tier receives a **direct memoryview** into the shared host region at construction time.
When `submit_store()` is called, the tier reads KV data directly from this region.
When `submit_load()` is called, the tier writes into it.
**No intermediate copies or serialization needed** — the tier operates directly on the primary tier's memory.
Each secondary tier also manages its own eviction policy independently.
A complete in-memory reference implementation is available at [`vllm/v1/kv_offload/tiering/example/`](https://github.com/vllm-project/vllm/tree/main/vllm/v1/kv_offload/tiering/example).
Out-of-tree secondary tiers are supported — specify a `module_path` in the tier config and vLLM loads your custom `SecondaryTierManager` implementation without any code changes to vLLM itself.
---
## Performance — Scaling to More Users
The main benefit of KV cache offloading: avoiding costly repeated prefills by reloading KV data from a cheaper tier.
With few concurrent conversations, all caching methods achieve high throughput — accelerator memory holds everything.
As the conversation pool grows, capacity limits appear at each tier:
- **Up to ~64 conversations** — HBM holds the working set; all caching methods perform well.
- **64–128 conversations** — HBM fills up; throughput drops dramatically without offloading. CPU offloading maintains performance.
- **Beyond 128 conversations** — CPU cache also fills up. Storage offloading continues serving a high cache hit ratio, more than doubling throughput compared to the alternatives.
Storage has higher latency than CPU memory, so it does not reach peak throughput.
But at scale, the choice is between a storage-backed cache hit and a full recompute — storage wins decisively.
**Benchmark setup:**
- Model: Qwen/Qwen3.6-35B-A3B on 2× NVIDIA H100 (TP=2)
- Storage tier: filesystem backend on local NVMe
- Workload: multi-turn conversations, 12K-token initial prompts + 4K tokens per round, 8 rounds
- Max request concurrency: 64
- Measures prefiller throughput only (prefill-decode disaggregated)
Full performance results and reproduction scripts are available at [neuralmagic/fs-offload-experiments](https://github.com/neuralmagic/fs-offload-experiments).
---
## Acknowledgements
We would like to thank Liran Schour, Chang Guo, Srinivas Krovvidi, Rotem Shavitt, Effi Ofer, Omer Paz, Kfir Toledo, and Michal Malka for their contributions to the design and implementation of the tiered KV cache offloading framework, and all other community members who contributed code, reviews, and feedback.
---
# GLM 5.3 Optimizations, Part 1: Hybrid HiSparse Offloading in vLLM
Source: https://vllm.ai/blog/2026-09-08-glm53-part1-hybrid-sparse-offloading
Published: 2026-09-08
Authors: vLLM Team
Tags: glm, kv-cache, performance
Summary: vLLM integrates HiSparse as a pressure-driven memory tier that composes with the Hybrid Memory Allocator and KV offloading, letting GLM 5.3 requests keep decoding when their KV no longer fits in GPU memory, so concurrency stays high.
**TL;DR:** vLLM is on a mission to make inference faster and cheaper to serve. In this two-part series we cover new optimizations we've introduced for GLM 5.3 in pursuit of that goal: in Part 1 we demonstrate how Hybrid HiSparse assists with an aggregated deployment on a single 8× H200 node, which is tight on memory for a model of this size. Hybrid HiSparse enables running GLM 5.3 at full 1 million context length—previously impossible on this hardware—and achieves substantially higher concurrency across context lengths.
## Exploiting sparsity when we need to
Agentic workloads are characterized by many concurrent requests, each with a long context that keeps growing. Because the GPU block pool is fixed, the KV cache will eventually run out of room for concurrent requests to allocate new blocks.
So far there have been two main options for addressing this issue, each with its own tradeoffs:
* **Preemption** picks a request, drops its KV cache, and re-prefills it later. The request pays its full TTFT again on every eviction.
* **Offloading** moves blocks out to host memory, but dense attention requires every token to be resident on the GPU, so the number of concurrent requests remains bounded by GPU memory.
For sparse-MLA KV cache, the indexer selects top-K tokens and attends only to these. [HiSparse](https://arxiv.org/abs/2608.07009) exploits this behaviour by offloading all KV cache—except these selected tokens—to the CPU, which yields an effective upper bound for the GPU memory each request needs. The indexer KV stays GPU-resident and still grows with context length, but it is much smaller overall, and GLM 5.3's [IndexShare](https://arxiv.org/abs/2603.12201) means there is only one indexer layer per four sparse-MLA layers.
We introduce Hybrid HiSparse, which additionally keeps KV cache on the GPU as long as there is enough capacity. Only when KV cache is under pressure do we apply the above described HiSparse offloading mechanism. Hot buffer pages are indexed by tokens; thus, a page can hold tokens drawn from many different CPU blocks, which leads to reduction across a wide span of context. In this way, Hybrid HiSparse only pays the cost of CPU-GPU memory transfers when the system is under KV cache pressure, i.e., higher concurrency.
Preemption: B's slots are freed and its KV is gone. Conventional offload: B's KV survives on host and we don't need to re-prefill but B still can't run until all of it fits on GPU again, so A decodes alone. Hybrid sparse offload: each request releases its coldest pages in place, the same slots are re-leased as new tails and hot pages, and both keep decoding.
Only Hybrid HiSparse keeps both requests decoding. The hot pages are leased from the same block pool as the KV pages, and, more importantly, they live in the same KV-cache tensor, so they look like ordinary pages to the sparse MLA kernel. Uniquely to hybrid sparse, some tokens can exist in the hot buffers while some tokens can still exist in GPU-resident pages, reducing the amount of CPU reloading.
## How it works
The same six top-K tokens are ringed in every panel; only their residency changes. Solid arrows: misses copying one row into a hot page. Dashed arrows: hot hits reused without a copy.
Residency is tracked per page, so a request moves between three states as pressure rises and falls:
* **Full residency**: all sparse-MLA KV remains GPU-resident while completed prefix pages are proactively materialized in host memory.
* **Mixed residency**: the tail of the request stays on the GPU, older pages live only in CPU memory, and the rows the indexer wants from those pages sit in hot buffers. The block table holds real blocks and null placeholders side by side, and the tail is never evicted. One fused kernel resolves the top-K: resident tokens are read in place, hot tokens are read and their LRU entry refreshed, and a miss copies a single row from pinned host memory into an LRU slot. Nothing on the decode path waits on a CPU decision, so it stays CUDA-graph-capturable.
* **No residency**: a new request reusing a prefix that only exists in CPU memory starts with placeholders and a hot page. Rows arrive as the indexer selects them, so we pay for what the model attends to rather than the whole history.
All three states work because the hot buffers are not a separate allocation. A hot buffer page is an ordinary KV-cache block, leased from the same pool as the resident pages through vLLM's Hybrid Memory Allocator, taken when a request first needs one and returned when it does not. Both for rows sitting in resident page or the hot buffer the resolver hands HMA row IDs and HMA gathers them with one stride. A block freed by one request can become hot-buffer capacity for another.
HiSparse prepares for pressure before it arrives. When a cacheable prefix page is complete, HiSparse queues a copy to CPU memory while continuing to serve it from the GPU. If the GPU cache later fills up, that page can release its GPU slot without another copy. Even if pressure reaches a newer page first, its GPU slot becomes reusable as soon as the copy is queued, and the CPU copy becomes available for prefix reuse when the transfer completes.
The `hisparse-glm` branch keeps this path lightweight by copying all sparse-MLA layers together in one launch after the forward pass. The copy is ordered on the model's GPU stream, which keeps synchronization simple and safe.
## Composing with the rest of vLLM
Hybrid HiSparse is a residency policy over the shared HMA pool and a connector alongside vLLM's other KV machinery, so the rest of the stack keeps working as it did. Other cache groups still use normal prefix caching, transfer, and offloading, and the indexer KV in particular is untouched by HiSparse: the standard OffloadingConnector can offload it independently with ordinary block-granular storage. Imports from P/D disaggregation can land host-side when a prefix does not fit resident, and speculative decoding works through per-step replayable resolver plans that share the request's hot state.
Hot buffers default to 2x top-K rows per request, which ensures high hit rates while keeping the buffer size small. Since MLA KV is identical across TP ranks, the pinned host pool is allocated per DP replica and shared across its local TP ranks. TP rank 0 writes the shared copy, every rank can read it, and a CUDA event preserves stream ordering.
## The numbers
We benchmarked GLM 5.3 on 8× H200 using an OpenHands multi-turn agentic workload ([source](https://www.lmsys.org/blog/2026-07-13-glm52-optimization)): 13-turn conversations with a 74,160-token first turn, 753-token later turns, and fixed 220-token outputs. Both TP8 deployments used MTP3, FP8 KV cache, a 142K admission limit, `max_num_batched_tokens=32768`, `max_num_seqs=256`, and `gpu_memory_utilization=0.92`. The offloading baseline used a 512 GiB offload pool; Hybrid HiSparse split the same host budget into a 384 GiB HiSparse pool and 128 GiB of offloading.
Top: the interactivity-throughput sweep. Interactivity is 1000 divided by mean TPOT; logical total-token throughput includes prefix-cached prompt tokens and is divided by eight GPUs. Bottom: mean non-zero vllm:num_requests_running samples collected during each benchmark point. Hybrid HiSparse: e8ef1e07bd. Offloading baseline: 80cb71c9ff.
We are planning to make Hybrid HiSparse widely available in vLLM v0.30. In the meantime, the exact launch commands and benchmark client setup used for these results are in the [reproduction appendix](#appendix-reproducing-our-results) below.
## Offloading only where we need it
Hybrid HiSparse only offloads where we need it. KV starts on the GPU and stays there while there is room, then gives up residency page by page as the pool runs short. Hot buffers and resident pages share pool and tensor and thus a request under pressure keeps decoding at partial residency instead of waiting for a slot to free up or paying to prefill itself again.
## Estimate the benefit for your configuration
The calculator below estimates ordinary GPU-resident KV and hybrid sparse offloading capacity using the same available HBM. Adjust the workload, GPU, parallelism, hot buffer, and host pool to approximate a deployment. Adjusting the values gives a sense of the potential increase in concurrency.
The calculator shows the minimum HiSparse host pool required to keep CPU memory from limiting the concurrency that the GPU-side indexer and hot buffers can sustain. Native indexer offloading is modeled as a separate total CPU pool: it extends the prefix cache, but active indexer history still consumes HBM and therefore remains part of the running-request limit. The plot compares total HiSparse and ordinary GPU-resident concurrency across sequence lengths while assuming non-limiting HiSparse host capacity. Hot buffers add a fixed GPU cost per request, so ordinary residency can fit more requests at short contexts; at longer contexts, bounding sparse-MLA residency lets HiSparse sustain more concurrent requests. Increasing the hot buffer trades some of that capacity for greater hot-cache coverage.
> **Note:** These are planning estimates, not guaranteed serving limits: runtime workspaces, request-length skew, and scheduling behavior can lower the concurrency reached in practice.
> **Note:** MTP can further limit concurrency because its hot buffers must accommodate all verification tokens at once. At publication time, this means sizing each hot buffer to `(num_speculative_tokens + 2) × top-K`. This is subject to change as we work to shrink the buffers. Currently this is not taken into account by calculator below as we plan to relax this constraint.
