HiSparse concurrency calculator

Estimate ordinary GPU KV capacity, HiSparse’s fully offloaded steady state, and native indexer offloading. Results are planning estimates, not guaranteed serving limits.

GPU

H200 · 141 GB HBM3e per GPU

FP8 KV cache

Workload and topology

Sequence length

Data parallel size

Tensor parallel size

Expert parallelism across DP×TP

With EP off, every DP replica holds the same TP-sharded experts, so DP no longer changes available KV HBM per GPU.

HiSparse

Minimum HiSparse host pool / DP rank

–

Avoids host capacity becoming the concurrency limit.

Host pool / DP rank

Indexer offload total

Hot-buffer rows

System RAM budget

Native indexer offloading extends the prefix cache in CPU memory. Active indexer history must still fit in HBM, so this pool does not directly raise the running-request limit.

Share the host cache across TP ranks

Sharing changes physical CPU RAM usage, not logical request capacity.

Advanced GLM-5.3 layout values

Available KV HBM / GPU override

Automatically estimated from GLM-5.3’s TP/EP-sharded weights at 92% GPU utilization. Override only to match vLLM’s reported startup value.

Max sequences / DP

Baseline GPU block bytes

Baseline block tokens

HiSparse GPU block bytes

Host block bytes

GPU block tokens

Host block tokens

Hot/resident groups

Retained pages / group

Other GPU blocks / request

Estimated non-HiSparse total

–

Estimated HiSparse total

–

Estimated capacity multiplier

–

Topology: – Model weights / GPU: – Physical host allocation: – Indexer offload coverage: –

Non-HiSparse

HiSparse

HiSparse memory composition

Concurrency / DP group –

Standard KV remains resident until pressure activates hot buffers and releases older pages.

GPU HBM per DP group –

Model weights – vLLM runtime + HBM headroom – Hot buffers – Indexer KV – Standard KV – Unused KV –

CPU host pools per DP group –

HiSparse latent KV – Native indexer offload – Unused –

Concurrency vs. sequence length

–

HiSparse Non-HiSparse

Shows total concurrency across the deployment and assumes the minimum non-limiting HiSparse host pool at each sequence length. Native indexer offloading does not change active-request HBM.

Per-DP-replica accounting

Resource	Non-HiSparse	HiSparse
GPU pool capacity
Unique GPU / request
↳ Indexer history	included
↳ Hot buffers	–
↳ Retained pages	–
Unique host / request	–
Native indexer offload / request	–
Native indexer offload coverage	–
GPU concurrency limit
Host concurrency limit	–
Final concurrency limit

Model and assumptions

HiSparse GPU usage is the unique indexer history plus per-request hot buffers, two retained resident pages per hot group, and any additional GPU blocks entered above. HiSparse host usage includes the full latent KV history. The native OffloadingConnector pool caches completed indexer blocks for prefix reuse, but does not remove active indexer history from HBM. GLM’s latent and indexer KV are replicated across TP ranks; therefore TP does not directly divide KV bytes per request. It can still leave more HBM for KV by reducing the per-GPU weight shard. With TP host sharing enabled, one host pool serves the whole TP group. This is a steady-state estimate; spill transitions, request-length skew, routing imbalance, and temporary workspaces can lower the reachable concurrency. The automatic HBM estimate shards routed-expert weights across DP×TP when EP is enabled; otherwise each DP replica duplicates them. Shared weights always shard across TP.