vLLM Office Hours

#58 · Oct 1, 2026

Intro to llm-d Semantic Classifier

Christopher Nuland (Red Hat)

llm-d-sc is a lightweight classifier service that picks a request's model tier before KV-cache routing picks a pod: its llm-d and Praxis integration, how it compares to vLLM Semantic Router, and a NASA domain-routing proof of concept.

#57 · Sep 3, 2026

vLLM Omni Project Update & Demos

Alex Brooks (Red Hat), Ricardo Noriega (Red Hat), Nick Cao (Red Hat)

How vLLM-Omni extends vLLM with a stage-based pipeline for audio, image and video models, and why diffusion needs its own scheduler and cache. Ends with a Qwen3-Omni real-time voice assistant demo and side-by-side video generation.

#56 · Aug 20, 2026

GuideLLM v0.7: Benchmarking Reasoning and Agentic Traces

Samuel Monson (Red Hat), Jared O'Connell (Red Hat), Francisco Javier Arceo (Red Hat), Tun Jian Tan (Embedded LLM), Rahul Tuli (Red Hat)

What benchmarks actually measure and how GuideLLM v0.7 simulates realistic load, now with multi-turn conversations, tool calling, sub-agents and trace replay. Lightning talks: Codex and Claude Code CLIs on vLLM, and Speculators v0.6/v0.7.

#55 · Aug 6, 2026

Mooncake + vLLM/llm-d Deep Dive

Ke Yang (Approaching.AI), Christian Angelo Velarde (Approaching.AI), Greg Pereira (Red Hat), Timothy Vang (U.S. Air Force), David Bramante (CodexDeus)

Mooncake, the KV-cache-centric architecture behind Kimi, and how its connectors give vLLM prefill/decode disaggregation and cluster-wide KV cache reuse, with llm-d packaging it for Kubernetes. Plus TickyTalky, a winning Austin hackathon project.

#54 · Jul 23, 2026

RL at 1T Scale: prime-rl Performance Deep Dive

Matej Sirovatka (Prime Intellect)

How prime-rl v0.6.0 trains trillion-parameter MoE models like GLM-5.1 on agentic RL workloads: FP8 inference, Wide EP, prefill/decode disaggregation, and router replay to keep trainer and inference in sync.

#53 · Jul 9, 2026

llm-d Project Update and Wide EP for Agentic Workloads

Robert Shaw (Red Hat), Charles Hernandez (Red Hat), Brian Dellabetta (Red Hat)

Why agentic workloads need optimization beyond one vLLM node: prefix-cache-aware routing, KV cache tiering, P/D disaggregation and Wide EP, with a live walkthrough of serving GLM-5.2 across H200 nodes with real performance and cost numbers.

#52 · Jun 25, 2026

vLLM Semantic Router: Safer, Faster, Multi-Model Inference

Christopher Nuland (Red Hat), Cedric Clyburn (Red Hat)

Signal-driven routing across cost, latency, privacy and safety: tuning confidence scores and anchors, pairing routing with PII redaction and guardrails, and combining it with llm-d. Plus a first look at the free vLLM course on DeepLearning.AI.

#51 · Jun 11, 2026

vLLM v0.22, Speculators Update, Accelerating Sparse MLA

Fynn Schmitt-Ulms (Red Hat), Helen Zhao (Red Hat), Matt Bonanni (Red Hat)

Speculators v0.5 covers EAGLE-3, P-EAGLE, DFlash, MTP fine-tuning, a new evaluation suite and a live train-and-serve demo of a draft model. An engineering spotlight shows how masked MHA accelerates sparse MLA.

#50 · May 28, 2026

GenAI with vLLM on Intel CPUs

Krzysztof Wiśniewski (Intel)

Running GenAI without GPUs: which workloads fit Intel Xeon (SLMs, small-active-parameter MoE, batched long context), core pinning, Kubernetes deployment and scaling strategies for vLLM on CPUs, plus validated models and benchmarks.

#49 · May 14, 2026

Latest Trends in AI Agent Applications and vLLM

Mahdi Ghodsi (AMD), Andy Luo (AMD), Eda Zhou (AMD)

AMD's AI team on how developers are building agentic applications, what those workloads demand from inference infrastructure, and where vLLM fits in the agent ecosystem.

#48 · Apr 30, 2026

vLLM Project and Tool Calling Update

Rob Greenberg (Red Hat), Ben Browning (Red Hat)

Tool calling as a three-part system (model, chat template, tool-call parser) that must stay aligned, how to measure success, where it breaks in practice, and the vLLM tool-calling roadmap. Opens with what's new in the major v0.20.0 release.

#47 · Apr 16, 2026

LLM Compressor Update

Kyle Sayers (Red Hat), Charles Hernandez (Red Hat)

What's new in LLM Compressor: Distributed Data Parallel (DDP) support for multi-GPU quantization workflows and improved offloading, with practical guidance on using them to compress large models.

#46 · Apr 9, 2026

Intro to vLLM-Omni

Gao Han

vLLM-Omni extends vLLM to omni-modal comprehension and generation across text, image, video and audio, covering both autoregressive and diffusion architectures. How it works today, how to get started, and what's next.

#45 · Mar 12, 2026

Vienna vLLM Meetup Live Stream

Dan Alistarh (ISTA / Red Hat), Eldar Kurtić (Red Hat), Dan Blanaru (NVIDIA), Georg Narodoslawsky (Canva)

Livestream of the Vienna vLLM meetup: a vLLM project update, talks on transforming LLM quantization, a speculative decoding tutorial, communication in MoE architectures, and Canva's case study on hosting LLMs for 260 million monthly users.

#44 · Feb 26, 2026

vLLM v0.16.0 Release Update and Open Discussion

A walkthrough of the v0.16.0 release (core engine improvements, performance work, distributed serving and multimodal progress), followed by an extended open Q&A with the community.

#43 · Feb 12, 2026

vLLM Triton Backend Deep Dive

Burkhard Ringlein (IBM Research)

IBM's Triton-only attention backend for vLLM reaches state-of-the-art performance on both NVIDIA and AMD GPUs from one portable code base, about 800 lines of Triton in place of hand-written CUDA/HIP, using autotuning and careful kernel design.

#42 · Jan 29, 2026

Deep Dive Into the vLLM CPU Offloading Connector

Or Ozeri (IBM Research), Danny Harnik (IBM Research)

How the offloading connector extends vLLM's KV cache into CPU memory with asynchronous DMA copies at minimal overhead: use cases like multi-turn chat and RAG, a quickstart, monitoring, memory-layout changes for faster transfers, and the roadmap.

#41 · Jan 22, 2026

LLM Compressor Update & Case Study

Kyle Sayers (Red Hat), Brian Dellabetta (Red Hat), Zewen Shen (Cohere), Ali Edalati (Cohere)

LLM Compressor 0.9.0 adds attention and KV cache quantization, model-free FP8 PTQ, AutoRound, experimental MXFP4 and faster calibration. Cohere then shares how it quantizes MoE models to 4-bit for on-prem production deployments.

#40 · Jan 15, 2026

Intro to Speculators

Fynn Schmitt-Ulms (Red Hat), Helen Zhao (Red Hat), Rahul Tuli (Red Hat)

Speculators is a unified library for building, storing and experimenting with speculative decoding algorithms in vLLM. Covers why it exists, how it fits the vLLM ecosystem, and how to train and deploy draft models with it.

#39 · Jan 8, 2026

Intro to batch invariant in vLLM

Wentao Ye (Red Hat), Bram Wasti (Meta)

What batch-invariant execution is, why it matters for reproducibility and RL, and how vLLM implements it. Then a look at bitwise-consistent RL with TorchTitan: the theory, design implications and experiment results.

#38 · Dec 18, 2025

vLLM 2025 Retrospective & 2026 Roadmap

Simon Mo (Inferact)

Simon Mo looks back at vLLM's 2025 milestones and the key design decisions behind them, then lays out the priorities shaping the project's 2026 roadmap.

#37 · Nov 13, 2025

InferenceMAX & vLLM

Kimbo Chen (SemiAnalysis), Cam Quilici (SemiAnalysis)

How SemiAnalysis's InferenceMAX continuously benchmarks popular LLMs across accelerators and inference stacks, the throughput, latency and cost metrics it tracks, and a demo of its UI.

#36 · Nov 6, 2025

LIVE from Zürich vLLM Meetup

Eldar Kurtić (Red Hat), Dan Alistarh (ISTA / Red Hat), Patrick von Platen (Mistral AI), Julien Denize (Mistral AI), Mickaël Seznec (Mistral AI), Mathis Felardos (Mistral AI), Thomas Parnell (IBM), Tyler Michael Smith (Red Hat)

Livestream of the first official vLLM meetup in Europe, with talks on beginner-to-advanced quantization, Mistral's reasoning-model training and P/D disaggregated serving, hybrid models, and distributed inference with llm-d.

#35 · Oct 23, 2025

How to Build and Contribute to vLLM

Daniele Trifirò (Red Hat)

A walkthrough of building vLLM from source (environment setup, build flags, precompiled wheels, compiler caches) and of contributing effectively, from testing and benchmarking to PR guidelines.

#34 · Oct 9, 2025

AI-Powered vLLM Semantic Router

Huamin Chen (Red Hat)

How the vLLM Semantic Router classifies incoming requests to pick the right model, how it plugs into multi-model deployments, and where adaptive model routing is headed.

#33 · Sep 25, 2025

Hybrid Models as First-Class Citizens in vLLM

Thomas Parnell (IBM)

What hybrid models are, how Mamba, Mamba2 and linear-attention layers were enabled in the V1 engine, and how hybrid-model performance compares between vLLM V0 and V1.

#32 · Sep 11, 2025

Intelligent Inference Scheduling with vLLM and llm-d

Abdullah Gharaibeh (Google), Kaushik Mitra (Google), Nili Guy (IBM)

Why LLM serving needs LLM-aware scheduling, how the Inference Gateway (IGW) architecture in llm-d routes requests across vLLM replicas, and benchmark results for SLO-aware scheduling.

#31 · Aug 28, 2025

vLLM and LLM Compressor Update

Brian Dellabetta (Red Hat), Kyle Sayers (Red Hat)

What's new in LLM Compressor: transform-based modifiers that cut quantization loss without calibration data, non-uniform per-layer precision, and DeepSeek-V3-style block quantization.

#30 · Aug 14, 2025

Hybrid Memory Allocator Architecture in vLLM

Chen Zhang (UC Berkeley)

How vLLM's hybrid memory allocator manages KV cache for models that mix attention types (full, sliding-window, Mamba), how it is integrated into the engine, and future plans.

#29 · Jul 31, 2025

Scaling MoE with llm-d

Robert Shaw (Red Hat), Tyler Michael Smith (Red Hat), Clayton Coleman (Google)

How llm-d runs wide expert-parallel MoE deployments on vLLM, uses prefill/decode disaggregation for cluster-scale efficiency, and lessons from real multi-node MoE deployments.

#28 · Jun 26, 2025

GuideLLM: Evaluate your LLM Deployments for Real-World Inference

Jenny Yi (Red Hat), Mark Kurtz (Red Hat)

GuideLLM simulates real-world traffic against an LLM deployment to measure throughput and latency and check that it meets its SLOs, shown with a live demo.

#27 · Jun 12, 2025

Intro to llm-d for Distributed LLM Inference

Carlos Costa (IBM Research)

An introduction to llm-d, the Kubernetes-native distributed inference framework built on vLLM and co-designed with the Inference Gateway, with a live demo.

#26 · May 29, 2025

Intro to torch.compile and how it works with vLLM

Luka Govedič (Red Hat), Richard Zou (Meta)

Red Hat and Meta PyTorch engineers explain what torch.compile is, why and how vLLM uses it to optimize model execution, and the resulting performance benchmarks.

#25 · May 8, 2025

Structured Outputs in vLLM

Russell Bryant (Red Hat)

How structured outputs constrain LLM generation to a required format (JSON schema, regex, grammars), what vLLM supports today, how it works under the hood, and planned enhancements.

#24 · Apr 24, 2025

Performance Optimization of vLLM on Google TPUs

Chengji Yao (Google)

Recent work to make vLLM deployments on Google Cloud TPUs fast and efficient.

#23 · Apr 10, 2025

Deep Dive Into the LLM Compressor

Kyle Sayers (Red Hat), Rahul Tuli (Red Hat)

A hands-on tour of LLM Compressor, the library for quantizing and sparsifying models for fast vLLM inference, plus an update on Llama 4 day-0 support in vLLM.

#22 · Mar 27, 2025

Intro to vLLM V1

Michael Goin (Red Hat)

A tour of the vLLM V1 engine rearchitecture, covering the new scheduler, memory manager, model runner and API server.

#21 · Mar 6, 2025

vLLM Production Stack Deep Dive

Yihua Cheng (University of Chicago)

The components of the vLLM Production Stack and how to deploy it, with a close look at its prefix-aware router and at KV cache offloading for performance and scalability.

#20 · Feb 27, 2025

DeepSeek and vLLM

Michael Goin (Red Hat), Robert Shaw (Red Hat), Nick Hill (Red Hat), Tyler Michael Smith (Red Hat), Lucas Wilkinson (Red Hat)

Five vLLM committers cover DeepSeek's Open Source Week releases and how vLLM supports them: MLA, multi-token prediction, 256-expert MoE, multi-node serving of 671B models, and FP8 block quantization.

#19 · Feb 6, 2025

Multimodal LLMs With vLLM v1

Roger Wang (Roblox)

How vLLM V1 re-architects multimodal support, with an encoder cache and encoder-aware scheduler, prefix caching, decoupled input processing and feature caching, plus benchmarks and roadmap.

#18 · Jan 23, 2025

Distributed Inference with vLLM

Murali Andoorveedu (CentML)

Why large models need distributed inference, how tensor and pipeline parallelism work in vLLM, their latency and throughput trade-offs, profiling tools, and the challenges of disaggregated prefill and decode.

#17 · Jan 9, 2025

vLLM Project Update and Open Discussion

What's new in vLLM v0.6.6, including prefix caching for vision-language models and Apple Silicon support, a preview of the Q1 2025 roadmap, and an open community Q&A.

#16 · Dec 19, 2024

vLLM's 2024 Wrapped and 2025 Vision

Simon Mo (Inferact)

Year-end retrospective: office hours by the numbers, vLLM's 2024 milestones in adoption, models, and hardware, and the 2025 roadmap, plus what's new in v0.6.5 including performant structured outputs.

#15 · Dec 5, 2024

Exploring Machete, a Mixed-Input GEMM Kernel for Hopper GPUs

Lucas Wilkinson (Neural Magic)

How Machete, built on CUTLASS 3.5.1, speeds up weight-only quantized (w4a16/w8a16, GPTQ) GEMMs on H100 by upconverting weights on the fly, pre-shuffling weights, and overlapping compute with data movement.

#14 · Nov 14, 2024

Disaggregated Prefill and KV Cache Storage in vLLM

Kuntai Du (University of Chicago)

Why splitting prefill and decode onto separate instances avoids contention, a walk through the initial disaggregated prefill PR, and how sharing and composing KV caches across vLLM instances can speed up delivery.

#13 · Nov 7, 2024

SOTA Tool-Calling Implementation in vLLM

Kyle Mistele (Constellate AI)

The author of vLLM's OpenAI-compatible tool calling explains tools vs. functions, the challenges of model-specific chat templates and tool parsers, and how streaming partial JSON arguments enables more responsive agent UIs.

#12 · Oct 17, 2024

Deep Dive into Mistral on vLLM

Patrick von Platen (Mistral AI)

Mistral AI's architecture choices and how to deploy Mistral models efficiently on vLLM, plus v0.6.3 highlights: experimental fullgraph torch.compile, the feature compatibility matrix, and the Machete w4a16 kernel.

#11 · Oct 3, 2024

Speculative Decoding in vLLM

Lily Liu (UC Berkeley)

What speculative decoding is, the draft-model, n-gram (prompt lookup), Medusa and EAGLE approaches supported in vLLM, when they actually speed things up, and open research directions.

#10 · Sep 19, 2024

Advanced Techniques for Maximizing vLLM Performance

Robert Shaw (Neural Magic)

How vLLM v0.6.0 achieved up to 2.7x higher throughput and 5x lower latency: the fundamentals of LLM serving, vLLM's continuous-batching architecture, and the CPU overheads that were removed (e.g. multi-step scheduling).

#9 · Sep 5, 2024

Using NVIDIA CUTLASS for High-Performance Inference

Tyler Smith (Neural Magic)

How vLLM uses NVIDIA CUTLASS to build its INT8 and FP8 quantized GEMM kernels, plus a v0.6.0 update covering the system changes behind its large throughput and latency gains.

#8 · Aug 21, 2024

vLLM on AMD GPUs and Google TPUs

Woosuk Kwon (UC Berkeley)

vLLM co-creator Woosuk Kwon on how vLLM supports diverse hardware, with performance benchmarks and supported features for AMD GPUs and Google TPUs and what is next on the hardware roadmap.

#7 · Aug 8, 2024

Multimodal Models in vLLM

Roger Wang (Roblox)

How vLLM added support for transformer-based vision-language models, how to use them effectively, and the multimodal roadmap including multimodal Llama 3.1, plus v0.5.4 highlights.

#6 · Jul 25, 2024

Model Quantization for Efficient vLLM Inference

Eldar Kurtić (Neural Magic)

The why, when, and how of quantizing LLMs for vLLM inference, and the debut of the LLM Compressor library; also v0.5.2/v0.5.3 updates such as Llama 3.1 support, AWQ Marlin, and CPU offloading.

#5 · Jul 9, 2024

FP8 Quantization Deep Dive

Philipp Moritz (Anyscale), Cody Hao Yu (Anyscale)

Why FP8 matters, how vLLM implements FP8 weights, activations, and KV cache, how to produce FP8 checkpoints, and quality/performance results; plus v0.5.1 with initial pipeline parallelism.

#4 · Jun 20, 2024

vLLM v0.5.0 Update and One-Year Anniversary Q&A

Simon Mo (Inferact)

Marking vLLM's first birthday with the v0.5.0 release (FP8 weights and activations, speculative decoding, OpenAI Vision API) and Q&A on quantization kernels, GPU compatibility, embeddings, and GPTQ tuning.

#3 · Jun 5, 2024

vLLM v0.4.3 Update and Open Q&A

Simon Mo (Inferact)

An intro to vLLM and the v0.4.3 release, then Q&A on post-training quantization, fitting 70B models on GPUs, vLLM vs. Hugging Face TGI, prefix caching, and tensor parallelism.

May 15, 2024

Inaugural vLLM Office Hours: Open Q&A

Simon Mo (Inferact)

The first vLLM office hours, co-hosted by Neural Magic and UC Berkeley: an open hour of community questions on optimized LLM inference and the vLLM roadmap.