vLLM Reaches 25K Total TPS/GPU on Qwen3.5

August 6, 20269 min read

vLLM Team

Introduction

Qwen3.5 was released in early 2026 and remains one of the most widely used models among customers. Because of its novel hybrid attention architecture, serving it in disaggregated mode brings in additional challenges and performance optimization opportunities. Thanks to the continuous contributions from the vLLM community, the disaggregated serving path for Qwen3.5 is now mature, and we are excited to report on the major contributions, latest performance on GB200 NVL72 systems, and recipes and best practices for you to reproduce. In this blog post we show how you can get over 25K total TPS/GPU performance.

Challenges and Key Optimizations

Qwen3.5's hybrid architecture combines full-attention layers with Gated Delta Network (GDN) layers. This creates two distinct optimization challenges: accelerating GDN computation on Blackwell GPUs and transferring heterogeneous attention/GDN state correctly between prefill and decode workers.

SSM support for P/D serving was driven by the vLLM community through the NIXL disaggregation roadmap. For a deeper discussion of heterogeneous cache layouts, logical and physical block mapping, and tensor-parallel state transfer, see the detailed hybrid SSM disaggregation blog post. We highlight the following contributions as particularly important to Qwen3.5 performance.

1. Blackwell-Optimized GDN Prefill

FlashInfer: Add Blackwell GDN prefill kernel #3001

Compared with the previous FLA/Triton implementation, the new GDN kernel improves performance by approximately 1.02× to 5.78× across Qwen3.5 model sizes, tensor-parallel configurations, sequence lengths, and batch shapes.

The kernel was subsequently enabled on the prefill side in vLLM by vLLM PR #40717. On an 8×B200 system running Qwen3.5-397B-A17B-NVFP4, the vLLM integration delivered:

Up to 5.92× higher GDN kernel performance in the tested microbenchmarks.
1.13× higher end-to-end prefill throughput on a prefill-only workload (ISL/OSL = 8192/1).
A 12% reduction in mean TTFT (prefill-only workload 8K/1).

On supported Blackwell configurations, vLLM automatically selects the FlashInfer path when the GDN backend is set to auto. It can also be requested explicitly with:

--gdn-prefill-backend flashinfer

2. Hybrid Cache and GDN-State Transfer

Disaggregated serving for hybrid SSM-attention models builds on [Core][KVConnector] Support HMA+NixlConnector #35758 and a stack of connector changes described in the hybrid SSM disaggregation blog post. This PR is a necessary prerequisite: it maps HMA's logical blocks onto the correct physical memory regions so NIXL can transfer only the cache regions belonging to each layer type, reducing transferred descriptors from 4,284 to 1,650 and improving throughput by up to approximately 7% in a small-scale intra-node H100 setup. But Mamba-style state differs enough in layout, size, and transfer semantics that HMA support alone would not have been sufficient for correct or efficient P/D serving.

The main PR for hybrid SSM-FA disaggregation is [PD][Nixl] Add support for hybrid SSM-FA models #36687, which adds dual descriptor views and homogeneous-TP support so prefill and decode workers can transfer both full-attention KV cache and Mamba-style SSM state over NIXL. Related follow-ups in the same stack include:

See the hybrid SSM disaggregation blog post for how dual descriptor views, physical/logical block bridging, and conv-state transfer fit together.

For Qwen3.5 specifically, PD disagg with NIXL Connector: GDN support (Qwen3.5) #41869 extends this path to GDN layers.

3. Race-Free Async Scheduling

These two patches fix race conditions in KV block transfer that made async scheduling unusable — accuracy collapsed to zero with it enabled. Async scheduling turned out to be one of the key features behind crossing 25K tok/s/GPU, so both races had to be resolved.

Performance

1. Environment Setup

Measurements were conducted on a GB200 cluster connected via NVLink72. We used ISL/OSL = 8192/1024. The evaluated model was Qwen3.5-397B-A17B-NVFP4. Performance was measured on a fixed decode topology and a constant number of decode endpoints. In this setup, the decode side used one endpoint with DEP8 (Data Parallel + Expert Parallel across 8 GPUs). On the prefill side, we evaluated configurations ranging from 4 to 8 endpoints, each using a fixed DEP2 topology.

To reproduce the results, use the latest vLLM vllm/vllm-openai:nightly-d223c90 Docker image, Dynamo 1.2.0.dev20260526, and srt-slurm v1.0.32. All recipes used in this article are available in the srt-slurm-recipes repository.

Accuracy Results

First, we measured accuracy for all serving configurations to ensure that the performance results are valid. For this purpose, we used the standard GSM8K (Grade School Math 8K) benchmark. Running GSM8K with srt-slurm is straightforward. To enable it, add the following benchmarking block to your recipe file:

benchmark:
  type: "gsm8k"

Accuracy results for all five configurations are 88%, which matches the accuracy we observe for the aggregated Qwen3.5 run.

2. Comments on Recipe Settings Choice

For performance measurements we used a fixed input sequence length / output sequence length benchmark, on a random dataset with random_range_ratio=0.8. The recipes themselves, and the settings that matter most, are covered in Recipes & best practices below.

3. Performance Results

Pareto curves for the individual configurations are shown in Figure 1, and the final Pareto frontier obtained after combining all configurations is shown in Figure 2. Total TPS per GPU reaches 25,000 tokens per second. Concurrency was swept from 64 up to 5120. We did not measure low concurrencies in the range of 1 to 32, since our focus here was the left part of the Pareto curve — maximizing the total TPS per GPU metric. At the other end, we did not go beyond 5120 because that is where we started running out of KV cache capacity on the decode side, which we deliberately fixed at a single 8×GB200 endpoint throughout these measurements. Pushing concurrency higher is entirely possible, but it requires adding GPUs on the decode side.

Figure 1: Pareto curves for disaggregated Qwen3.5 serving with different numbers of prefill instances.

Figure 2: Final Pareto frontier for disaggregated serving of Qwen3.5 NVFP4 in vLLM.

Recipes & best practices

All recipes used in this article live in the srt-slurm-recipes repository, and each one is launched with a single command:

srtctl run --file <recipe>.yaml

The naming scheme is NxDEP2-1xDEP8, where N is the number of prefill endpoints running DEP2 against a single DEP8 decode endpoint; there are five base configurations, from 4×DEP2 to 8×DEP2. Each comes with three derived variants: the base file sweeps sa-bench over concurrencies 64…3072, the -acc variant runs GSM8K five times on the same topology, and -cc4096 / -cc5120 each capture a single high-concurrency point with the decode-side max-cudagraph-capture-size raised to 640 and 768 respectively.

Most settings in the recipes are standard and shared across all configurations, but several are worth calling out:

VLLM_SSM_CONV_STATE_LAYOUT=DS — mandatory for SSM models in disaggregated serving; conv-state transfer does not work without it. Our recipes also passed --no-disable-hybrid-kv-cache-manager; HMA has since been enabled by default in vLLM for several versions, so that flag is no longer required.
--async-scheduling — one of the key features behind reaching 25K tok/s per GPU. It requires a vLLM build that already contains the race-condition fixes discussed above.
--mamba-ssm-cache-dtype bfloat16 — significantly increases the effective KV cache capacity on the decode endpoint.
--language-model-only — Qwen3.5 is a multimodal model, and for a purely textual workload this flag not only disables multimodal inputs but also unlocks the fused QK-norm + RoPE + gate path in the attention layers.
--max-num-batched-tokens 16384 on the prefill side, i.e. 2× ISL. With fewer prefill endpoints ({4, 5, 6}×DEP2) prefill became the bottleneck and left the decode side idling below its peak throughput, so we let each prefill step batch two full prompts instead of one — worth about +8% of total TPS per GPU at high concurrencies.
--max-cudagraph-capture-size on decode — raised to cc/8 + 128 for the two highest concurrency points (640 at cc=4096, 768 at cc=5120), where 8 is the number of DP ranks on the decode endpoint. The vLLM default caps captured graphs at 512, which is enough up to cc=3072. We are not certain this is actually required for the Pareto numbers reported here, but we set it as a precaution.
Prefix caching is disabled: it buys nothing on a random dataset.
--stream-interval 100 — reduces frontend overhead at high concurrency. Note that it buffers streamed output in 100-token chunks, so it does affect measured per-token latency; keep that in mind if you are optimizing for ITL/TPOT rather than aggregate throughput.

Finally, a couple of practical things that saved us a lot of time.

--api-server-count 1 is very useful while you are investigating a particular configuration. On a data-parallel endpoint vLLM defaults the API server count to the data-parallel size, and with more than one API server it disables its default stats logging altogether in order not to report incomplete numbers. Forcing the count to 1 brings that logging back: every 10 seconds — the interval is configurable through VLLM_LOG_STATS_INTERVAL — the server prints prompt and generation throughput along with KV cache utilization. Without these metrics we would hardly have identified the bottlenecks of the individual configurations, or understood why a particular option helps on our workload.

It is also worth setting three environment variables: DYN_LOG=error, DYN_SDK_DISABLE_ANSI_LOGGING=1, and VLLM_LOGGING_COLOR=0. The first one drastically cuts down the amount of Dynamo logs, while the other two suppress some (not all!) ANSI escape sequences in the log output. Without them your log files are very likely to be unreadable for a human, mostly because Dynamo produces an enormous amount of logging by default.

What's next

Our measurements so far have concentrated mostly on the left part of the Pareto curve, squeezing out as much total TPS per GPU as possible. Next, we plan to sweep for the PD configurations that maximize Gen TPS per user instead. Reaching that regime will require shifting away from DEP topologies towards TEP (Tensor Parallel + Expert Parallel) or just TP, which as a rule deliver better per-user performance. Increasing the number of GPUs in use is another lever we expect to pay off here.

Acknowledgements

Artem Perevedentsev (NVIDIA), Vadim Gimpelson (NVIDIA), Jiangyun Zhu (Inferact), Nicolò Lucchesi (Mistral), Zhanqiu Hu (Red Hat), Nick Hill (Inferact), Linxuan Li (Alibaba), JingZe Cui (NVIDIA), Cyrus Chang (NVIDIA), Xin Li (NVIDIA)