Figure 2 · GPU memory

How many sequences fit on each GPU

MiniMax M3.1-Flash, real config. Changing dtype, parallelism or GPU refits the sequence count. MI350X has the same 288 GB as MI355X.

60layers

4KV heads

64query heads

128head dim

60index-K layers

250.1GB weights

KV cache dtype

Parallelism

GPU

Index K cache Count it (1 head × 128 per layer)

Context length Sequences

Memory per GPU

Each KV dtype, same workload: KV per GPU, max sequences

Notes and assumptions

MSA cuts what each query reads (2,048 keys per layer), not what the cache stores: every token’s K and V stay resident.

TP splits K/V by head; index K has one head, so every TP rank keeps a full copy. Index K uses the KV dtype. NVFP4 = 4-bit values + one FP8 scale per 16 (4.5 bits).

Weights = safetensors total_size, 250.1 GB (routed experts NVFP4 ≈ 232 GB, rest ≈ 18 GB). Routed experts split by TP/EP; under DEP the rest sits on every GPU. 95% of HBM usable. 1 GB = 10⁹ B.

weights K + V cache index K cache over capacity 95% usable selected dtype