Efficient Decode Context Parallelism with vLLM for Long Context Workloads

August 7, 202612 min read

Seonghee Lee, Sungsoo Ha, Omri Almog (NVIDIA), Lucas Wilkinson (Red Hat AI)

1. Introduction

Long-context inference is becoming essential for agentic AI, where assistants may need to reason over large code repositories and long chat histories. Agent-trace benchmarks now run from 64K all the way to 1M tokens and their KV caches are correspondingly large. Under a baseline tensor-parallel (TP) setup, this KV cache is partitioned by attention head, which puts a hard floor on how much it can shrink.

Modern models use one of two attention schemes, and both hit this floor. Grouped-query attention (GQA) models store a small number of KV heads, and TP can only split the KV cache down to one head per GPU; once TP exceeds the number of KV heads, the cache starts duplicating across GPUs. Multi-head latent attention (MLA) models make this even worse: MLA compresses the Key/Value into a single low-rank latent vector shared across all query heads, so it effectively has only one KV head. Under normal TP there is nothing to split by head, meaning the latent KV cache is replicated in full across every TP rank. In both cases the duplicated KV cache eats into GPU memory, leaving very little room to serve additional requests. This caps the number of concurrent requests the system can handle, driving down throughput and pushing up cost per token.

Decode Context Parallelism addresses this by splitting KV cache across the GPUs so each GPU stores and reads only part of the KV cache. This frees up GPU memory, allowing each GPU to take on more requests and thus run at a larger batch size. On systems with high-bandwidth GPU-to-GPU interconnects, this helps preserve interactive responsiveness while serving many long-context agents at once.

vLLM has supported DCP for almost a year, but we are writing this blog now to highlight the feature, along with the recent improvements and advancements we have made to it, because the rise of long-context agentic use cases has made its benefits more relevant than ever.

2. Performance Results

To quantify the benefit of Decode Context Parallelism, we compared a baseline tensor-parallel deployment against DCP on an identical set of GPUs, holding the model, hardware, and workload fixed and varying only how the KV cache is sharded during decode.

2.1 Dataset

The dataset is a publicly available agentic long-context trace in Mooncake trace format, published here. See this section for more details on the dataset. It ships as JSONL where each line is a single request with input_length, output_length, and hash_ids fields, so it can be replayed directly with any Mooncake-compatible harness (e.g. aiperf --custom-dataset-type mooncake_trace). The hash_ids field encodes shared prefix blocks, making it well-suited for benchmarking KV-cache reuse and prefix-caching behavior.

It's an agentic multi-turn workload of long inputs paired with short generations, chosen to reflect realistic long-horizon agent behavior. Inputs are centered around a median of ~67k tokens and paired with short ~400-token outputs, but the input distribution is bimodal rather than uniformly huge: roughly half the requests sit at 64k+ (≈53%, with a heavy tail reaching ~1M tokens) and half are short-to-mid (≈47% under 64k, ~18% under 8k). About 8% of requests exceed 128k and ~3–4% exceed 256k.

2.2 Benefits of Decode Context Parallelism

We ran an experiment on a single 8×B200 node serving Kimi K2.6 in NVFP4 with vLLM, sweeping request concurrency from 16 to 512 (see table below). DCP sustains far higher concurrency and delivers markedly higher throughput per GPU across the entire throughput–interactivity Pareto frontier.

The difference comes down to where the KV cache lives. Baseline TP replicates the KV cache on every GPU, so peak memory fills quickly. It reaches 100% at a concurrency of 64 and hits a wall, and throughput plateaus near 1,863 tok/s/GPU because no additional requests can fit. On the other hand, DCP shards the KV cache along the sequence dimension, so each GPU stores only 1/N of every request's KV. This allows space on the GPU to support more incoming requests. As a result, even at high concurrencies DCP keeps scaling where TP hits a wall. DCP reaches 6,091 tok/s/GPU at c512 while still sitting at just 82% KV usage. The core value of DCP is that it sustains far higher concurrency, even on long-context runs, precisely the regime where replicated-KV TP runs out of memory first.

2.3 Comparison by Sequence Length

We also plotted performance against full sequence length (input + output). The figure shows a single throughput–interactivity Pareto frontier with requests grouped into five length bands (<32k, 32–64k, 64–128k, 128–200k, and 200k+) so we can see how performance shifts with context length. DCP keeps a high, stable frontier even in the 200k+ range, with the curves for short and long buckets nearly overlapping: throughput scales with concurrency while per-user speed stays usable at the long context lengths where the replicated-KV baseline runs out of memory and cannot scale.

3. Challenges of Serving Long Contexts

Under tensor parallelism, the KV cache is partitioned by the attention head. Each KV head owns its own separate K and V tensors, and the head is the smallest unit you can hand to a GPU. A standard TP has no mechanism to slice a single head's KV cache. So if you have K KV heads, you can give each GPU a distinct subset of those heads, but only down to the point where every GPU holds one head. Once TP goes beyond K, there aren't enough distinct heads to go around, so two or more GPUs end up holding a copy of the same head's KV cache instead of a unique slice.

4. What is DCP?

Unlike pure TP methods, DCP is able to split KV cache across GPUs by sequence (context) dimension. Each GPU is made responsible for the KV cache of a chunk of token positions from the same sequence. For a single 200K-token request, GPU 0 might hold the cache for tokens 0–50K, GPU 1 for tokens 50K–100K, GPU 2 for 100K–150K, and GPU 3 for 150K–200K. By sharding KV cache, the KV cache footprint per GPU keeps shrinking as you add GPUs, freeing the memory that lets you raise the batch size and serve higher concurrencies.

4.1 Decode Context Parallelism Process

Standard Decode Context Parallelism keeps the communication pattern simple, following the rhythm AllGather Q → Compute → AllGather + ReduceScatter.

AllGather Q: Each GPU has computed only a fragment of the query, but attention requires the full query vector to score against any key. An all-gather across the DCP group assembles a complete copy of the query on every GPU. This is cheap during decode because the query is a single token. As an opt-in alternative for MLA, vLLM #45964 can replicate the (small) query projection within each DCP group at load time so decode skips this query all-gather entirely (VLLM_DCP_Q_REPLICATE=1).
Compute: Each GPU runs attention between the gathered query and its local slice of the KV cache. In vLLM this is k_up for MLA or tensor_broadcast for GQA.
AllGather + ReduceScatter (cp_lse_ag_out_rs): The partial results are combined into the true output. AllGather shares each GPU's partial output and LSE; the LSE values reweight and merge the partials (the online-softmax trick), and ReduceScatter sums them while handing each GPU back only its own head-slice.

5. vLLM Usage

DCP is enabled with a single extra argument, decode_context_parallel_size, alongside your existing tensor-parallel setting.

5.1 Offline

from vllm import LLM, SamplingParams
 
prompts = [
    "The future of AI is",
]
sampling_params = SamplingParams(temperature=0.8, top_p=0.95)
 
llm = LLM(
    model="deepseek-ai/DeepSeek-V2-Lite",
    tensor_parallel_size=2,
    decode_context_parallel_size=2,
)
outputs = llm.generate(prompts, sampling_params)

5.2 Online

vllm serve deepseek-ai/DeepSeek-V2-Lite \
    --tensor-parallel-size 2 \
    --decode-context-parallel-size 2

5.3 MLA Backend

Models: DeepSeek-V2 / V3 / R1, Kimi K2.6 models using Multi-head Latent Attention.

Why it's different. MLA compresses the Key/Value into a single low-rank latent vector that is shared across all query heads — effectively one KV "head." Under pure tensor parallelism there's nothing to split by head, so that latent KV cache is replicated in full on every TP rank. TP does nothing to shrink it, which makes MLA the ideal candidate for DCP: the whole cache is redundant, so the whole cache can be sequence-split.

What they do. DCP splits the latent KV cache along the sequence dimension, so each rank stores only its chunk of the latent; at attention time each rank up-projects its latent slice (the k_up step) to reconstruct the Keys/Values it needs. Because the effective KV-head count is 1, the sequence can be split up to the full TP degree — hence the constraints:

tensor_parallel_size >= decode_context_parallel_size
tensor_parallel_size % decode_context_parallel_size == 0

vllm serve deepseek-ai/DeepSeek-R1 \
    --tensor-parallel-size 8 \
    --decode-context-parallel-size 8

5.4 GQA Backend

Example models: Qwen3-235B, and other Grouped-Query-Attention models (Llama-family, etc.).

Why it's different. GQA stores num_key_value_heads KV heads, and TP splits the KV cache by those heads first. That works cleanly only up to num_key_value_heads; once tensor_parallel_size exceeds it, the KV cache begins duplicating, with tp // num_key_value_heads identical copies across ranks.

What they do. DCP takes those would-be-duplicate copies and fills them with different sequence chunks instead, while the shared KV heads are broadcast across their query heads (the "tensor broadcast for GQA" step). So the sequence-split degree is capped by the duplication factor tp // num_key_value_heads:

(tensor_parallel_size // num_key_value_heads) >= decode_context_parallel_size
(tensor_parallel_size // num_key_value_heads) % decode_context_parallel_size == 0

# Qwen3-235B has num_key_value_heads = 4; tp=8 gives 8//4 = 2 redundant copies,
# so dcp can be up to 2.
vllm serve Qwen/Qwen3-235B-A22B \
    --tensor-parallel-size 8 \
    --decode-context-parallel-size 2

6. Future Work

Looking ahead, we plan to extend DCP along several main directions. We will add support for finer-grained parallelism sizes for both TP and DCP, giving users more precise control over their parallelism layout and reclaiming efficiency lost to over-provisioned sharding. We are also developing better DCP all-to-all (A2A) communication kernels for both multinode and single-node settings, reducing exposed communication and improving overlap with compute as context length and device count grow. We are working on better support for MTP and speculative decoding, so that DCP can deliver its efficiency gains without sacrificing the latency benefits of speculative methods, as well as hardening prefill/decode (P/D) disaggregation support to make DCP robust in disaggregated serving deployments. Finally, we aim to broaden DCP's reach by extending support to a wider variety of backends and integrating it with hybrid models and Dynamic Chunked Pipeline Parallelism, so a much wider range of workloads can benefit from context-parallel efficiency gains.

The community is also expanding DCP to additional models such as GLM-5.2 and Kimi K3, and there is a longer roadmap for Prefill Context Parallelism (PCP). We are working on DCP performance benchmarking for the Kimi K3 model and plan to share those results as that work matures. For deployment guidance and historical notes on DCP, see the vLLM Decode Context Parallel docs.

7. Conclusion

Decode Context Parallelism represents a fundamental rethinking of how GPUs are organized for long-context inference. Rather than forcing GPUs to duplicate KV cache or sit underutilized, DCP puts every GPU to work: sharding the sequence during attention, then immediately reconfiguring those same GPUs to amortize FFN weight loading across the full pool. The result is a system that scales gracefully with context length rather than degrading under it.

With native support in vLLM, Decode Context Parallelism is ready to power the next generation of long-context agentic applications, from document reasoning to multi-session agentic pipelines, at the throughput and latency that production demands. It joins a broader industry move toward Decode Context Parallelism, a direction NVIDIA has also pursued with Helix Parallelism in TensorRT-LLM. We are also working on DCP performance benchmarking for the Kimi K3 model and plan to share those results as that work matures.

About Us

Special thanks to the NVIDIA team Anahita Bhiwandiwalla, Xin Li, Pavani Majety, Nidhi Bhatia, Roman Ageev, Pen Chung Li, and Chris Hoge for their reviews, benchmarking support, and engineering input throughout this study. We also thank Moonshot AI for the initial Decode Context Parallel work upstreamed in vLLM #23734, and Lucas Wilkinson for substantial follow-up contributions that helped harden and extend DCP. We also thank the broader vLLM community, whose open-source engine and continued collaboration made this benchmarking effort possible. For more on DCP deployment and related history, see the vLLM Decode Context Parallel docs.

The DCP results in this post were measured on NVIDIA B200 GPUs with Kimi K2.6 in NVFP4, and the recipes can be reproduced with current vLLM releases that support --decode-context-parallel-size. We are also working on DCP performance benchmarking for the Kimi K3 model and plan to share those results as that work matures.