First vLLM Conference at Ray Summit

State of vLLM 2026 surveys where the project stands and the roadmap ahead across its engine, ecosystem, and open-source practices. The talk covers the Flat Model and Model Runner V2 migrations, disaggregated serving with multi-tier KV offloading, speculative decoding toward 1000+ TPS, and production-grade quantized KV cache compression. It also looks at the infrastructure, CI, and security work needed to support a rapidly growing contributor base, plus the expanding ecosystem around quantization, speculation, RL frameworks, and vLLM-Omni for real-time multimodal models.

This talk highlights how NVIDIA and the vLLM community collaborated to deliver record-setting DeepSeek and MiniMax performance in the SemiAnalysis InferenceX Benchmark. We will cover the full-stack optimization path, from kernel-level improvements powered by FlashInfer to distributed serving capabilities enabled by NVIDIA Dynamo, and show how these contributions help vLLM users unlock higher throughput, lower latency, and more efficient inference on NVIDIA GPUs.

This talk explores how PyTorch supports vLLM, touching on the Stable ABI, adoption of new PyTorch releases, and support for heterogeneous hardware and long-tail models.

Large Language Models (LLMs) demand significant computational power for inference. This talk explores the exciting work being done to leverage Google TPUs with the popular vLLM library, orchestrated by Ray. We'll dive into the benefits of using TPUs for LLM serving, including potential performance gains and cost-efficiency. We will discuss key enablers like the TorchTPU backend, which aims to provide a native PyTorch experience on TPUs, and share insights from our efforts to optimize and scale vLLM deployments. Attendees will learn about the architectural approaches, challenges overcome, and the future outlook for high-performance, TPU-accelerated LLM inference with vLLM and Ray.

The Transformers modeling backend for vLLM lets you serve Hugging Face Transformers model implementations at full vLLM speed with access to all of vLLM's features. This session takes a deep dive into how a Transformers model loads in vLLM and, as a consequence, why its performance and feature access are so strong.

llm-d brings production-grade orchestration and distributed inference optimizations to vLLM deployments on Kubernetes. This talk explores how intelligent, prefix-cache-aware routing, prefill/decode disaggregation, hierarchical KV cache offloading, and traffic- and hardware-aware scheduling improve throughput and reliability across multi-node serving clusters. It also covers how llm-d integrates vLLM with the Kubernetes Gateway API and operational tooling to create a modular serving stack that can be deployed, scaled, and tuned for real-world production traffic.

Training trillion-parameter MoE models on long-horizon agentic tasks demands that inference and optimization scale independently. Using GLM-5.1 as a case study, this talk presents Prime Intellect's asynchronous prime-rl stack, including runs at up to 131K sequence length with 256 rollouts and sub-five-minute steps on 28 H200 nodes. It explains how vLLM powers low-precision inference, prefill/decode disaggregation, prefix-aware routing, and distributed KV cache storage, while the trainer combines FSDP, context parallelism, and expert parallelism. Attendees will see how these pieces reduce idle time and make frontier-scale agentic post-training more efficient.

Agentic production traffic brings long multi-turn sessions, heavy prefix reuse, and bursty, heterogeneous requests that challenge conventional serving. This talk explains how vLLM addresses those patterns through independently scaled prefill/decode disaggregation, turn-aware serving, and session and correlation hints that improve prefix caching, prefetching, and eviction. It also covers distributed KV cache transfer and storage with Mooncake and multi-tier offloading, allowing cached context to move across turns and instances instead of being recomputed. Attendees will leave with practical guidance for architecting and tuning vLLM deployments for production agents.

vLLM has become the de facto standard for open-source LLM inference, and AMD has been deepening its investment in the project across hardware, software, and community. This talk covers the latest updates from AMD's collaboration with vLLM: new ROCm backend capabilities, performance improvements, upcoming hardware, recent upstream contributions, and how AMD is working with the vLLM maintainer community to keep AMD support on par with the broader ecosystem. Attendees will leave with a clear picture of where AMD and vLLM stand today and what is coming next.

Three focused eight-minute talks explore systems work for production agentic workloads. Embedded LM will cover Agentic API, Anyscale will discuss KV-cache-aware serving with Ray Serve LLM, and Novita's speaker and topic will be announced.

This talk explores Gemma's custom quantization efforts for producing highly compressed quantization-aware training (QAT) checkpoints while preserving model quality. It covers how quantization methods, checkpoint design, and serving integration work together to reduce memory requirements and inference costs, along with practical considerations for deploying these compressed models with vLLM.

This session delivers a technical deep dive into vLLM on Intel platforms. It introduces Intel's optimized attention, Mixture-of-Experts (MoE), and GEMM operators powered by Triton and SYCL/SYCL-TLA (CUTLASS)/oneDNN. We will showcase how Intel next-generation GPUs support native sub-8-bit microscaling formats (MXFP8/MXFP4). Additionally, we demonstrate how Intel eliminates inter-token latency (ITL) spikes by decoupling compute-bound prefill and memory-bound decode stages onto dedicated nodes. Finally, we highlight Intel's backend support for vLLM-Omni, enabling the seamless pipelining of heterogeneous stage graphs for complex, any-to-any multimodal models (text, video, and audio DiTs).

TorchSpec is a torch-native speculative decoding training framework built on top of Ray. We introduce a disaggregated way of training speculative decoding draft models, allowing each side to scale independently. Draft models for large models such as Kimi K2.5 trained with TorchSpec have accumulated more than 500K downloads on Hugging Face.

DigitalOcean's vLLM deployments reached leading Artificial Analysis results across frontier open-weight models, including 230 tokens per second per user on DeepSeek V3.2 and sub-second time to first token for 10,000-token Qwen 3.5 prompts. This talk breaks down the open-source optimization work behind those results, from low-batch kernel fusion and EAGLE3 speculative decoding to linear-attention fusions and dual-stream execution. Attendees will see how profiling model-specific bottlenecks and upstreaming the fixes into vLLM can deliver production inference performance that rivals proprietary stacks.