Kimi K3 Is Here: Efficient Day-0 Support on vLLM

We're thrilled to announce efficient day-0 vLLM support for Kimi K3, one of the most powerful open-weight models ever released.

Last week, we previewed the production-scale integration work for Kimi K3; today, Moonshot AI's weights are public and the support is live.

Kimi K3 is a 2.8-trillion-parameter Mixture-of-Experts model (16 of 896 experts active per token) built on Kimi Delta Attention (KDA) and Attention Residuals (AttnRes), with a 1M-token context window and native vision. For us, the most exciting challenge Kimi K3 brings is making KDA, MXFP4 MoE, KV cache management, prefill/decode disaggregation, speculative decoding, and long-context deployment recipes work together in a runnable serving engine.

The preview post explained the kernel and cache architecture, in particular the challenge of prefix caching that works on recurrent state. This release post is the practical guide: how vLLM adapts to Kimi K3's architecture, the kernel work behind the numbers, and what is ready on day 0.

Quick start

# See the linked recipes for the exact Docker command.
vllm serve moonshotai/Kimi-K3 \
  --tensor-parallel-size 8 \
  --trust-remote-code \
  --load-format fastsafetensors \
  --enable-prefix-caching \
  --enable-auto-tool-choice \
  --tool-call-parser kimi_k3 \
  --reasoning-parser kimi_k3

The easiest way to run Kimi K3 is to use 8 NVIDIA B300 GPUs or 8 AMD MI355X GPUs with the above command.

Inferact has also trained and open-sourced a DSpark speculator for Kimi K3. Enable it by adding the following option to the serve command:

--speculative-config '{"model":"Inferact/Kimi-K3-DSpark","method":"dspark","num_speculative_tokens":7,"attention_backend":"FLASHINFER_MLA","draft_sample_method":"probabilistic","rejection_sample_method":"block"}'

For more details, including Docker images for various platforms and deployment strategies, refer to the detailed recipes. Because of complicated dependencies, only Docker images are usable now. The Docker images depend on several pre-release dependencies, including FlashInfer.

TL;DR

• A 2.8-trillion-parameter multimodal MoE: Kimi K3 activates 16 of 896 experts per token, supports a context window of up to 1M tokens, and is built on Kimi Delta Attention, Attention Residuals, LatentMoE, and native MXFP4 (4-bit) weights.
• Up to 370 tok/s per user: vLLM serves Kimi K3 at 118 tok/s without speculative decoding and 370 tok/s (a 3.14× improvement) with DSpark on 16 NVIDIA GB300 NVL72 GPUs, powered by extensive optimizations for Kimi K3's architecture.
• Broad production feature support: vLLM supports speculative decoding, prefill/decode disaggregation, agentic KV caching with Mooncake, tool calling, reasoning output, and structured output, with NVIDIA (Hopper and Blackwell) and AMD (MI355X) support at launch.
• Open-source DSpark support: vLLM supports the state-of-the-art block-diffusion speculative decoding algorithm for Kimi K3, trained with vLLM and TorchSpec and open-sourced by Inferact.
• Hybrid prefix caching: serving Kimi K3's recurrent and full-attention design required a redesign of hybrid prefix caching over recurrent KDA state. This change now benefits every hybrid linear model.

Kimi K3's architecture, and how vLLM serves it

Kimi K3 architecture innovations, from the original release blog post.

Kimi K3's architecture departs from a standard Transformer in a few ways, and each one changes what a serving engine has to do. The preview post covers the internals in depth; here we recap what's new and focus on how vLLM adapts to serve it.

Kimi Delta Attention: a hybrid recurrent + full-attention stack

What's new: Most of Kimi K3's layers are KDA, a linear-attention mechanism that keeps a fixed-size recurrent state instead of a growing KV cache, interleaved with periodic full-attention layers that preserve exact global recall. That is what makes a 1M-token context affordable.

How vLLM serves it: A single hybrid KV-cache manager holds two kinds of memory side by side under one scheduler: paged KV blocks for the full-attention layers, and compact recurrent-state blocks for the KDA layers. A dedicated KDA attention backend runs FlashKDA for prefill and a fused CUDA kernel (or the Flash-Linear-Attention/Triton path when running speculative decoding) for decode.

The hardest part is prefix caching across Kimi K3's hybrid cache: full-attention layers store per-token KV, while KDA layers update recurrent and convolution state at every token but cannot afford to retain a snapshot at every possible prefix boundary. vLLM decouples the large physical KDA state blocks from fine-grained prefix matching, registering state snapshots within those blocks and copying them before extension so long shared prompts can reuse both KDA state and paged KV. This hybrid-cache machinery is new in vLLM core and now benefits every hybrid model similar to Kimi K3.

Kimi K3 interleaves Kimi Delta Attention layers with periodic full-attention layers; vLLM's hybrid cache manages recurrent state and paged KV together.

Attention Residuals: learned mixing of residual contributions across depth

What's new: For each token, Block AttnRes replaces ordinary residual accumulation with depth-wise attention: every Transformer sublayer uses a learned pseudo-query to weight RMS-normalized residual states from preceding layer blocks, then receives the corresponding weighted combination as its input.

How vLLM serves it: vLLM uses optimized Triton and CUDA kernels to compute the depth-wise attention logits, softmax, and hidden-state aggregation in a single fused operation. Residual updates and output RMSNorm are folded into the same kernel where supported, reducing intermediate memory traffic and kernel-launch overhead in both prefill and decode.

Stable LatentMoE: quantile-balanced latent-space experts at 16-of-896 sparsity

What's new. LatentMoE, introduced by NVIDIA, projects dispatched token activations into a narrower latent dimension for routed-expert computation, then projects the combined expert output back to the model width—reducing expert-weight bandwidth and all-to-all traffic so more experts can be used at similar inference cost. Kimi K3's Stable LatentMoE scales this design to 896 experts with 16 active per token and uses Quantile Balancing to derive expert allocation from router-score quantiles instead of heuristic balancing updates.

How vLLM serves it: Experts are sharded with expert parallelism. vLLM offers two MoE backends tuned for different topologies: TRT-LLM-Gen for tensor-parallel (TP > 1) and MegaMoE for disaggregated/expert-parallel (DEP). It also supports optional Expert-Parallel Load Balancing (EPLB) to ensure each rank has a similar amount of compute. The weights execute natively in MXFP4 on the MoE path.

Chat template: a render program, not a Jinja template

What's new: Kimi K3's chat template must encode system, user, and assistant messages, multimodal content, tool definitions, and tool results using exact control tokens. Instead of the common approach of a Jinja chat template that renders the request as text before tokenization, Kimi K3 uses a Python program to build the prompt token sequence directly. Its output likewise contains distinct regions for reasoning, answer text, and tool calls that must be parsed into an API response.

How vLLM serves it: vLLM implements both the input renderer and streaming output parser in its Python and Rust frontends, preserving control-token boundaries while treating user-supplied and tool-supplied text as ordinary content. For tool calls and structured outputs, vLLM integrates Kimi K3's format with XGrammar so structured regions are constrained during decoding and returned as separate reasoning, content, and tool-call fields.

Built for production

Serving a 2.8T hybrid MoE well means being fast for each user, efficient for many concurrent sessions, and scalable for agents. vLLM ensures Kimi K3 is ready on all three.

Ultra-low latency: speculative decoding with DSpark

To reach ultra-low latency on a 2.8T-parameter model like Kimi K3 without accuracy loss, speculative decoding is the natural choice. That is why vLLM supports DSpark, a state-of-the-art speculative decoding algorithm, from day 0—and why we trained and released a DSpark speculator for Kimi K3. The draft model is trained with vLLM using TorchSpec to achieve full numerical parity between speculator inference and training.

DSpark uses a block-diffusion backbone to generate multiple speculative tokens in one parallel pass based on Kimi K3's rich intermediate states, so drafting cost stays flat as the block deepens. A low-rank Markov head supplies the intra-block dependency, and a confidence head predicts the likelihood for each draft to be accepted. We made the draft MLA-native, mirroring Kimi K3's own attention, so draft and target share a similar KV layout to be maximally compatible with advanced KV management and P/D-disaggregated setups.

Kimi K3 DSpark positional acceptance rates across various datasets.

With DSpark, we achieve a 3.14× speedup on a single-user request, from 118 tok/s to 370 tok/s, measured using SPEED Bench. We also benchmarked the acceptance rate and speedups on different tasks, with the results shown above. For coding and other low-entropy tasks, we achieve around 4.73 accepted tokens per step. For high-entropy tasks such as creative writing, we achieve around 2.61 accepted tokens per step.

Confidence-based scheduling with DSpark is an ongoing effort in vLLM. Once enabled, it uses the confidence head included in the DSpark model to predict how likely each drafted token is to be accepted, prioritizing strong proposals and pruning weak ones so verification is not spent on tokens that will not survive.

Both the draft model and the inference support are open source as of this release. See the deployment guide below to enable it.

A lightweight DSpark draft proposes candidate tokens that Kimi K3 verifies in a single parallel pass, accelerating single-stream decode.

Sequence parallelism for TEP prefill

Sequence parallelism shards token ownership across ranks; the attention residual is applied per shard, and one all-gather rebuilds the full batch before the next layer's QKV projection.

For the prefill phase, we combine attention tensor parallelism with MoE expert parallelism (TEP). Compared with pure TP, TEP reduces communication overhead and keeps whole experts on each rank, yielding more efficient expert GEMM shapes.

However, the naive TEP implementation performs two all-reduce operations per layer—one after the attention output projection and one after the MoE—so every rank materializes the full batch and redundantly applies the attention residual to all of it. To address this, we implement sequence parallelism: the all-reduce after o_proj is replaced with a reduce-scatter so that each rank owns a shard of the tokens, the attention residual is applied per shard, the MoE's all-to-all performs dispatch and combine, and a single all-gather restores the full batch before the next layer's QKV projection.

This design provides two key advantages:

• Reduced communication overhead: Reduce-scatter + all-to-all dispatch + all-to-all combine + all-gather are theoretically cheaper than two all-reduces. In practice, however, NCCL's reduce-scatter and all-gather are not optimized for prefill's message sizes and yield no speedup. We therefore implement custom reduce-scatter and all-gather kernels that are 1.7×–4.5× faster than NCCL, especially at small-to-medium message sizes.
• Sharded attention residual: The attention residual stays sharded across ranks throughout the layer, so each rank computes and maintains only its shard of the tokens rather than the entire batch. This matters especially for Kimi K3, where AttnRes turns the residual stream into persistent cross-layer state with its own compute and memory footprint.

Sequence parallelism is enabled by default when appropriate: when using TP with the MegaMoE kernel, or when combining TP + DP + EP. No extra flags are needed.

Large-scale serving: prefill/decode disaggregation

For high-throughput settings, vLLM serves Kimi K3 with expert and data parallelism across nodes and with prefill/decode (PD) disaggregation, which runs prefill-heavy and decode-heavy work on separate replicas so each is sized for its own bottleneck. One of our validated topologies routes TEP8 prefill to DEP16 decode, with NIXL as the KV transfer engine.

PD disaggregation is unforgiving for a hybrid model: the recurrent KDA state, the full-attention paged KV, and the block tables all have to arrive correctly. The NIXL connector treats the shared KV-cache page as two logical views: token-level MLA cache and request-level KDA state, including convolution and recurrent state. During the handshake, it exchanges the MLA/KDA metadata, then builds separate transfer descriptors for each transfer.

Under heterogeneous TP, vLLM's hybrid allocator uses different block sizes for prefill and decode. To support that case, vLLM's NIXL connector tracks the logical-to-physical block mapping and zeroes any untransferred tail regions, preventing stale data from previous requests from leaking through padding or layout gaps.

Reconciling partial block cache hits and KV cache offloading

As described in the Kimi K3 preview, vLLM supports fine-grained prefix hits that may end inside a physical cache block. This introduces a subtle challenge for KV offloading: vLLM may first find a local GPU hit with a partial tail, then discover a longer prefix in an external store such as Mooncake. With full-block hits, remote reuse extends cleanly beyond the local prefix. A partial tail, however, can overlap with the remote result.

The vLLM scheduler therefore compares the exact reusable token lengths from both tiers and selects the longer prefix. If the remote hit wins, it releases the block reserved for the shorter local tail and reconciles all cache groups to the new prefix length.

Importantly, we built this mechanism entirely through the existing KV Connector APIs, which already provide all the required semantics. This allows MooncakeStoreConnector, SimpleCPUOffloadConnector, and other connectors to support multi-tier partial-prefix reuse without model-specific integration paths.

The design is tracked in the RFC and implemented across PR #45939, PR #46384, and PR #49502.

Agentic serving: smarter cache retention policies

Kimi K3's linear-attention layers require only a constant-size KDA state, making them memory-efficient at long context lengths. A single layer's KDA state is roughly equivalent to the MLA cache for a few thousand tokens. Although large, this state does not grow with sequence length, unlike a conventional KV cache. That distinction becomes significant for agentic workloads spanning hundreds of thousands to one million tokens.

The same design complicates prefix caching. KDA state is updated in place during decoding, so vLLM must copy the state at a selected prefix boundary before the next forward pass overwrites it. Caching at every token position would be prohibitively expensive: each KDA checkpoint is much larger than one token's MLA cache and would quickly exhaust even a distributed cache pool.

To improve cache-space efficiency while preserving useful prefixes, vLLM supports two complementary retention policies.

Interval-based retention

Caching every KDA state is wasteful, but caching too sparsely forces the next request to recompute a large suffix. Interval-based retention balances these costs by treating selected positions as checkpoints—for example, one every 32K tokens.

Prompt boundaries are even better checkpoints. In agentic workloads, the next turn usually begins by replaying the previous turn's prompt, so the state at the end of that prompt is especially likely to be reused. vLLM detects and retains these boundaries automatically.

Users can control periodic checkpointing with VLLM_PREFIX_CACHE_RETENTION_INTERVAL. Setting it to 0 disables periodic checkpoints and retains only prompt-end states, which is a good fit for workloads dominated by multi-turn conversations. Larger intervals trade some recomputation for lower cache usage.

Interval-based retention was introduced for DeepSeek V4 and hybrid sliding-window-attention models in PR #43447, with day-0 support for Kimi K3 and hybrid linear-attention models added in PR #45845.

Interval-based cache retention. MLA caches KV for every block, while a KDA state is kept only at checkpoints: prompt ends (green) are always retained, and fixed-interval checkpoints (orange) are configurable.

Marconi-style selective retention

Prompt-end retention works well for conversational state, but valuable shared prefixes can appear elsewhere. A system prompt, repository snapshot, or tool specification may be reused across many requests without aligning with a prompt boundary.

Marconi-style retention (MLSys '26) handles these cases with a simple rule: cache on the second hit. The first observation provides evidence that the prefix exists; the second shows that it is actually shared. Only then does vLLM spend cache capacity on its KDA state.

This turns retention into a demand-driven decision. One-off prefixes do not crowd the cache, while recurring prefixes are promoted automatically—without requiring users to predict which parts of their workload will become hot.

Selective retention was introduced in PR #37898, with day-0 Kimi K3 support added in PR #47782.

Selective cache retention. Request 1 keeps a KDA state only at its own prompt end, past the shared prefix, so request 2 gets a KV hit but a KDA miss. That second sighting is evidence the prefix is shared, so a state is cached at the prefix boundary, and request 3 reuses it.

Together, the policies cover both predictable and emergent reuse: interval retention checkpoints structurally important boundaries, while Marconi-style retention learns which other prefixes are worth keeping.

Performance optimizations

Serving large models like Kimi K3 brings its own challenges because of its size. The entire model can barely fit in a single NVIDIA DGX B300 and requires a minimum of 16 NVIDIA B200/GB200 GPUs to serve on that hardware generation. Serving must trade off interactivity against total system throughput: tensor parallelism is good for interactivity but offers low overall throughput because effective KV cache size is limited, while large-scale expert parallelism can limit per-user output-token speed because of network-bandwidth bottlenecks. Here we highlight optimizations that improve performance in both cases so users can choose the recipe that suits their workload. Many of these optimizations are already covered in our preview blog.

Attention Residuals

Kimi K3 uses Block AttnRes, attending over up to eight cached block representations plus the current within-block residual. For each token, vLLM computes logits from RMS-normalized sources, applies softmax across these depth-wise candidates, and aggregates their representations. Its implementation resembles FlashAttention's online-softmax strategy but operates across model depth rather than sequence positions and has at most nine sources. vLLM performs this mixing in a single fused kernel, incorporating the residual update at the input and optionally applying RMSNorm to the output. A portable Triton implementation covers the general path, while a specialized CUDA kernel accelerates supported Blackwell configurations.

KDA decode

The fused KDA decode kernel folds the causal convolution, recurrent update, and RMSNorm into a single launch instead of a chain of separate kernels.

A KDA layer involves many operations: input projections, causal 1D convolutions, QK norm, gate computation, KDA recurrent update, and output gated RMSNorm. On supported configurations, vLLM fuses the post-projection decode path—from the causal convolutions through gated RMSNorm—into a single specialized CUDA kernel. The kernel updates the convolution and recurrent states in place and writes the normalized output directly, avoiding intermediate tensors, repeated state traffic, and per-operation launch overhead across Kimi K3's many KDA layers. Portable Triton fallback paths cover unsupported configurations.

KDA prefill

KDA prefill became one of our favorite examples of open-source development in practice. Moonshot AI first released FlashKDA, a high-performance CUTLASS implementation of KDA. We quickly integrated it into vLLM and worked through less glamorous production details: broader GPU coverage, metadata dtypes, tensor layouts, and reliable vendoring. Shikhar Mishra then optimized the kernels for H100 and published Flash-Flash-KDA, improving data movement while preserving numerical correctness. Within a day, we validated the improvements on GB300 NVL72, refined the recurrence pipeline and synchronization, and folded them into our FlashKDA integration. The result was not a one-way handoff, but a continuous loop in which an open kernel was extended by the serving community, improved by an independent contributor, and quickly carried into production.

KDA metadata builder

During Kimi K3 DSpark bring-up, KDA metadata preparation emerged as a significant source of overhead. Kimi K3 initially reused the generic GDN metadata builder, which prepared FLA metadata that K3 does not consume and used sequences of small eager PyTorch operations to assemble and stage GPU metadata. We introduced a dedicated Kimi K3 KDA metadata builder that prunes the unused paths and replaces those operation sequences with fused Triton kernels, reducing each sequence to a single launch. At batch size 1, this reduced metadata-preparation latency by 96%, from 870 μs to 34 μs, and improved end-to-end DSpark latency by 6%.

Low-latency BF16 GEMM

In low-batch-size, latency-sensitive settings, we replace generic BF16 GEMM—used in several linear projection layers—with our own skinnyGEMM implementation. Generic cuBLAS kernels do not achieve the best performance here because they are optimized for more general shapes. In the kernel, we bypass shared-memory data staging, load activations and weights directly into registers, and use CUDA Core FMA instructions to perform the math. This avoids the heavy TMA and Tensor Core setup phase used to achieve maximum throughput. Our microbenchmarks show kernel-level speedups ranging from 8% to 100% and an end-to-end latency reduction of about 10% in small-batch settings.

Low-latency MoE tail fusion

The LatentMoE tail optimization replaces two all-reduces, RMSNorm, latent up-projection, and an elementwise add with three kernels to reduce compute and better overlap communication and computation.

vLLM uses a novel strategy to reduce latent-MoE tail latency in ultra-low-latency serving. At the end of LatentMoE, the reduced activation from routed experts must be normalized with RMSNorm and up-projected before it is added to the shared-expert output. In the normal TP case, this requires two all-reduces on the routed and shared experts—or one all-reduce with concatenation—and replicates the up-projection.

To avoid redundant compute in the replicated linear projection, vLLM instead performs reduce-scatter on the shared experts and keeps all-reduce on the routed experts because their activations need to be normalized. The replicated routed-expert activation then performs matrix multiplication with the up-projection in a column-parallel fashion and is added elementwise to the already-sharded shared-expert output. Finally, the results are all-gathered onto each rank using broadcast. We observe about a 20% latency reduction in this step and about a 7%–8% end-to-end speedup.

Quality and Performance Benchmarks

Accuracy and correctness evaluation

vLLM takes accuracy as seriously as speed. We validated Kimi K3 end to end through a served OpenAI-compatible endpoint, with exact configurations in the recipes, and it passes the accuracy evaluations cleanly. At the maximum reasoning-effort level, Kimi K3 on vLLM scores 0.976 on GSM8K, 0.939 on GPQA-Diamond, 0.889 on OCRBench, and 0.818 on MMMU Pro Vision.

One caveat worth knowing for evaluation: Kimi K3 thinks a lot before it answers. A low score is more often a truncated answer than a wrong one, so increase the reasoning effort, set max_tokens generously, and check for cut-off generations before debugging anything else.

Serving performance

Kimi K3 decode throughput at batch size 1, measured on GB300 NVL72 GPUs in TP8 and TP16 configurations.

At launch, vLLM achieves 111 tok/s per user on TP8 and 118 tok/s per user on TP16 at batch size 1. DSpark speculative decoding boosts interactivity by roughly 3×, reaching 331 tok/s per user on TP8 and 370 tok/s per user on TP16.

We also present initial Pareto-frontier performance results for serving Kimi K3 on GB300 NVL72 across a range of scenarios, from high-throughput serving at 2K+ TPGS to low-latency serving at 100+ TPS/user.

Reproduce our benchmark

Here are the full recipes to reproduce the decode throughput numbers above for TP8 with DSpark:

export NCCL_DMABUF_ENABLE=0
export VLLM_ALLREDUCE_USE_FLASHINFER=1
export VLLM_USE_RUST_FRONTEND=1
export VLLM_ENGINE_READY_TIMEOUT_S=3600
export HEAD_ADDR=127.0.0.1  # Change if vllm-bench runs on another host.
 
vllm serve moonshotai/Kimi-K3 \
  --enable-prefix-caching \
  --tensor-parallel-size 8 \
  --nnodes 2 \
  --node-rank 0 \
  --moe-backend auto \
  --trust-remote-code \
  --load-format fastsafetensors \
  --max-num-seqs 512 \
  --gpu-memory-utilization 0.9 \
  --max-model-len auto \
  --max-cudagraph-capture-size 256 \
  --kv-cache-dtype fp8 \
  --attention-config '{"mla_prefill_backend":"FLASHINFER","use_prefill_query_quantization":true}' \
  --speculative-config '{"model":"Inferact/Kimi-K3-DSpark","method":"dspark","num_speculative_tokens":7,"attention_backend":"FLASHINFER_MLA","draft_sample_method":"probabilistic","rejection_sample_method":"block"}'
 
# Batch size = 1, 8K/1K random (no speculative decoding)
vllm-bench \
  --backend openai \
  --base-url "http://${HEAD_ADDR}:8000" \
  --model moonshotai/Kimi-K3 \
  --dataset-name random \
  --random-input-len 8192 \
  --random-output-len 1024 \
  --random-range-ratio 0.8 \
  --prompt-token-ids \
  --ignore-eos \
  --sweep-max-concurrency 1 \
  --sweep-num-prompts-factor 10 \
  --seed 42 \
  --percentile-metrics "ttft,tpot,itl,e2el" \
  --metric-percentiles "50,90,99" \
  --save-result
 
# Batch size = 1, SPEED Bench (speculative decoding)
vllm-bench \
  --backend openai \
  --base-url "http://${HEAD_ADDR}:8000" \
  --model moonshotai/Kimi-K3 \
  --dataset-name speed-bench \
  --speed-bench-config throughput_16k \
  --speed-bench-max-input-len 10240 \
  --speed-bench-category low_entropy \
  --output-len 1536 \
  --num-prompts 10 \
  --no-oversample \
  --max-concurrency 1 \
  --temperature 1.0 \
  --top-p 0.95 \
  --save-result \
  --save-detailed

Full recipes, including multi-node, expert-parallel, and vision configurations, are in the Kimi K3 recipes.

Important Deployment Tips

1. Prefix caching: --enable-prefix-caching turns prefix caching on. Prefix caching is typically enabled by default in vLLM, but it is currently disabled by default for Kimi K3 while the hybrid-cache design continues to evolve. Pass the flag explicitly.
2. Tool calling: Validate on your own traffic before depending on it. We've occasionally seen K3 emit a tool-call format its own parser does not expect, yielding an empty tool_calls result, while clean probes on the same setup parse perfectly. It is prompt- and run-dependent, not a blanket failure, but production agents should validate against your schema, retry or fall back when tool_calls comes back empty, and consider strict or structured tool calling, which constrains the output grammar during generation.
3. All-to-all backend: --all2all-backend determines how the MoE backend communicates during expert parallelism. Use flashinfer_nvlink_one_sided for NVIDIA NVLink and deepep_v2 for RDMA.
4. MoE backend: vLLM has several MoE backends for different scenarios. We recommend deep_gemm_mega_moe for any DEP environment and flashinfer_trtllm for TP > 1.
5. Rust frontend: Set VLLM_USE_RUST_FRONTEND=1 to enable the Rust frontend, which fully supports this model.
6. ViT parallelism: --mm-encoder-tp-mode=data is enabled by default. K3's vision encoder has head_size=12, which cannot be sharded evenly under TP=8. Because K3's vision encoder has fewer than 1B parameters while the backbone has about 2T, we enable ViT DP by default to avoid all-reduce overhead from the encoder.

Kimi K3 vLLM FAQ

How many GPUs do I need to serve Kimi K3?

At least one 8× B300 (or GB300 NVL72) node is required; 16× B200 is also supported. Most production deployments run multi-node with expert and data parallelism, connected over RDMA or NVLink.

How do I enable DSpark speculative decoding?

Add:

--speculative-config '{"model":"Inferact/Kimi-K3-DSpark","method":"dspark","num_speculative_tokens":7,"attention_backend":"FLASHINFER_MLA","draft_sample_method":"probabilistic","rejection_sample_method":"block"}'

It roughly triples single-stream decode on reasoning and coding workloads.

Which MoE and all-to-all backend should I use?

Use deep_gemm_mega_moe for disaggregated or expert-parallel (DEP) deployments and flashinfer_trtllm for TP > 1. Choose the all-to-all backend to match your interconnect: flashinfer_nvlink_one_sided for NVLink and deepep_v2 for RDMA.

Does Kimi K3 support prefix caching, and is it on by default?

It supports prefix caching over both full-attention KV and recurrent KDA state, but it is not enabled by default, so pass --enable-prefix-caching.

Does vLLM support Kimi K3 on AMD GPUs?

Yes. ROCm support ships at launch, with broader tuning on the roadmap.

How is this different from the Kimi K3 preview post?

The preview is the architecture and kernel deep dive, including how KDA prefix caching and the kernels are built. This post is the practical launch guide and includes the artifacts: how vLLM adapts to Kimi K3, recipes, flags, performance, and what Kimi K3 is ready for in production.

Roadmap and Future Work

• RL support for Kimi K3: vLLM rollout support has already been added. We will work closely with RL ecosystem projects to support end-to-end RL training for Kimi K3.
• Continuous performance improvement: continue improving performance after day 0.
• Decode Context Parallelism (DCP): our prototype shows good speedup, and we will soon upstream the support. Early experiments show 40% higher throughput than TP8 under selected workloads.
• Expert-Parallel Load Balancing (EPLB): improve EPLB performance.
• Confidence-based scheduling: use the confidence head in DSpark to prune the number of draft tokens to verify.
• Broader AMD ROCm tuning.

Quick links

• Model: moonshotai/Kimi-K3
• DSpark draft: Inferact/Kimi-K3-DSpark
• Recipes and Docker images: recipes.vllm.ai/moonshotai/Kimi-K3
• Kimi K3 technical blog: kimi.com/blog/kimi-k3
• vLLM design for K3: the preview post

Acknowledgements

Thank you to Moonshot AI for creating K3, sharing the architecture ahead of release, and co-designing the KDA-aware caching; to the Inferact team for the end-to-end vLLM integration and deployment validation; to NVIDIA for the fused KDA decode, KDA prefill, and Attention Residual kernels and the MXFP4 MoE collaboration; to AMD for ROCm bring-up; to our inference partners, including Alibaba Cloud, Baseten, DigitalOcean, and Modal; to Shikhar for Flash-Flash-KDA; and to the vLLM community. The cache infrastructure built for Kimi K3 now belongs to every hybrid model with a similar architecture. We can't wait to see what you serve.