Figure 2 · KV cache capacity on 8× GB300

How many long contexts fit on eight GPUs

Measured KV cache capacity of Kimi K3.1 Pro for each latent and index-key format, with tensor parallelism or decode context parallelism.

MLA latent

Index keys

Parallelism

Context length

–contexts fit

–tokens in cache

–vs TP8, FP8

–B / token / GPU

–KV memory / GPU

–KDA + other / GPU

KV memory per GPU–

MLA latents – index keys – KDA state and other –

Every format

Setup and method

Measured on 8× NVIDIA GB300 (two 4-GPU trays): vLLM with FlashInfer 0.7.3.dev20261031, dummy weights, max-model-len 1M, max-num-seqs 64, default GPU memory utilization. "Tokens in cache" is vLLM's reported KV cache size; "contexts fit" is that size divided by the context length, as vLLM's maximum-concurrency estimate. --max-num-seqs caps concurrency separately. Per-token bytes cover the 24 DSA layers: 576 B (FP8) or 352 B (NVFP4) of latent and 132 B (FP8) or 68 B (MXFP4) of index key per layer, divided by 8 under DCP8. KDA state is per request and sits in the same packed block rows.