Scaling Kimi K3 on AMD Instinct MI355X: Matching NVIDIA HGX B300 Throughput per GPU on AgentX
·24 min read
Inside the Kimi K3 optimization effort: understanding the bottlenecks and the decisions that made inference faster
#speculative-decoding
21 posts
Serving Reflection Beam on vLLM from Day 0
·11 min read
vLLM supports Reflection's Beam-1.0 (501B total, 23B active) from day 0, with agent-driven optimization that cut decode latency 2.71× and DFlash2 speculative decoding with adaptive verification.
Day 0 support for Muse Spark Open on vLLM
·9 min read
Day-0 vLLM support for Muse Spark Open: 816B MoE, 1M context, multimodal inputs, DSpark speculative decoding, NVIDIA and AMD serving, initial TPU support, and LoRA RL.
Adaptive Verification Without a Confidence Head
·16 min read
An online-fitted acceptance estimator that lets every draft-model speculator use Adaptive Verification, by predicting the target's acceptance probability from the drafter's own logits and refitting continuously during serving.
Watermarking in vLLM
·13 min read
How vLLM implements distribution-preserving Gumbel-max text watermarking with efficient GPU kernels, statistical detection, speculative decoding, and repeated-context safeguards.
How we trained the fastest DSpark for Kimi-K3 using GB300 NVL72
·9 min read
How Speculators and Mooncake enabled multi-node DSpark training for Kimi K3.
Kimi K3 Performance Optimizations in vLLM: The Road to 2.8× Throughput
·7 min read
Kimi K3 serving optimizations across scheduling, KDA prefix caching, ReplaySSM state recovery, PD disaggregation and state offload, parallelism, MoE, and GPU kernels.
Following the Bottleneck: Optimizing MiniMax M3 on AMD Instinct MI355X
·16 min read
A performance model for LLM serving: inspect local shapes, remove repeated work, verify data movement and dispatch, then follow the queue.
Exploring Speculative Decoding in vLLM on AMD GPUs
·242 min read
A practical guide to speculative decoding in vLLM on AMD GPUs, covering draft-and-verify mechanics, MTP, EAGLE-3, DFlash, DSpark, configuration, tuning, and benchmark results.
Adaptive Verification in vLLM: DSpark confidence-scheduled verification
·8 min read
Sizing the DSpark draft-verification budget from per-request confidence instead of verifying every drafted token, so one configuration holds the throughput/latency frontier from batch size 1 to 256.
From Day 0 to Production SLAs: Serving GLM-5.2 on 24 NVIDIA B300 GPUs with vLLM
·18 min read
How we took GLM-5.2-NVFP4 from 40 ms to 17 ms mean TPOT on 24 B300 GPUs with vLLM: P/D disaggregation, MTP speculative decoding, Model Runner V2, and the SLA-first trade-offs behind the final configuration.
TML Inkling on vLLM: Day-0 Support with Optimized Performance
·8 min read
vLLM brings day-0 support to TML Inkling, a 1T-parameter multimodal model, with MTP, long-context serving, parallelism, and up to 380 tokens per second per user on NVIDIA GB200 GPUs.
Accelerating Laguna XS.2 Inference with vLLM, Speculators, and LLM Compressor
·3 min read
How Laguna XS.2 is served and optimized in vLLM using first-class model integration, a DFlash speculator trained with Speculators, and FP8, NVFP4, INT4, and INT8 checkpoints from LLM Compressor.
Speculators v0.5.0: DFlash Support and Online Training
·6 min read
What Speculators v0.5.0 adds for vLLM speculative decoding: DFlash block-diffusion draft models, unified online and offline training, native hidden-state extraction, and Gemma 4 latency results.
EAGLE 3.1: Advancing Speculative Decoding Through Collaboration Between the EAGLE Team, vLLM, and TorchSpec
·4 min read
How EAGLE 3.1 improves speculative decoding robustness in vLLM with FC normalization, post-norm hidden-state feedback, TorchSpec training support, and config-driven compatibility with EAGLE 3 checkpoints.
vLLM Tops the Artificial Analysis Leaderboard
·15 min read
How vLLM achieved leading Artificial Analysis results for DeepSeek V3.2, MiniMax-M2.5, and Qwen 3.5 397B using open-source kernel fusion, speculative decoding, Blackwell optimizations, and model-specific serving work.
Extracting hidden states from vLLM
·8 min read
How vLLM extracts verifier hidden states through dummy draft models and KV Connector APIs for speculative decoding, enabling offline and online Speculators training without patching vLLM internals.
P-EAGLE: Faster LLM inference with Parallel Speculative Decoding in vLLM
·12 min read
How P-EAGLE brings parallel speculative decoding to vLLM by generating multiple draft tokens in one forward pass, with pre-trained drafter heads, config support, and B200 speedups over EAGLE-3.
Diving into speculative decoding training support for vLLM with Speculators v0.3.0
·11 min read
How Speculators v0.3.0 supports end-to-end Eagle3 draft model training for vLLM, including hidden-state data generation, MoE and non-MoE verifiers, offline workflows, and seamless speculative decoding serving.
Inside vLLM: Anatomy of a High-Throughput LLM Inference System
·41 min read
How vLLM's inference engine works, covering PagedAttention, continuous batching, prefix caching, speculative decoding, multi-GPU serving, scheduling, and benchmarking for high-throughput LLM workloads.
How Speculative Decoding Boosts vLLM Performance by up to 2.8x
·10 min read
How speculative decoding works in vLLM, covering EAGLE, Medusa, n-gram proposals, draft and target runners, scheduler and memory-manager changes, and continuous batching for lower token latency.