First vLLM Conference at Ray Summit

State of vLLM 2026 surveys where the project stands and the roadmap ahead across its engine, ecosystem, and open-source practices. The talk covers the Flat Model and Model Runner V2 migrations, disaggregated serving with multi-tier KV offloading, speculative decoding toward 1000+ TPS, and production-grade quantized KV cache compression. It also looks at the infrastructure, CI, and security work needed to support a rapidly growing contributor base, plus the expanding ecosystem around quantization, speculation, RL frameworks, and vLLM-Omni for real-time multimodal models.

This talk highlights how NVIDIA and the vLLM community collaborated to deliver record-setting DeepSeek and MiniMax performance in the SemiAnalysis InferenceX Benchmark. We will cover the full-stack optimization path, from kernel-level improvements powered by FlashInfer to distributed serving capabilities enabled by NVIDIA Dynamo, and show how these contributions help vLLM users unlock higher throughput, lower latency, and more efficient inference on NVIDIA GPUs.

This talk explores how PyTorch supports vLLM, touching on the Stable ABI, adoption of new PyTorch releases, and support for heterogeneous hardware and long-tail models.

Large Language Models (LLMs) demand significant computational power for inference. This talk explores the exciting work being done to leverage Google TPUs with the popular vLLM library, orchestrated by Ray. We'll dive into the benefits of using TPUs for LLM serving, including potential performance gains and cost-efficiency. We will discuss key enablers like the TorchTPU backend, which aims to provide a native PyTorch experience on TPUs, and share insights from our efforts to optimize and scale vLLM deployments. Attendees will learn about the architectural approaches, challenges overcome, and the future outlook for high-performance, TPU-accelerated LLM inference with vLLM and Ray.

The Transformers modeling backend for vLLM lets you serve Hugging Face Transformers model implementations at full vLLM speed with access to all of vLLM's features. This session takes a deep dive into how a Transformers model loads in vLLM and, as a consequence, why its performance and feature access are so strong.

llm-d brings production-grade orchestration and distributed inference optimizations to vLLM deployments on Kubernetes. This talk explores how intelligent, prefix-cache-aware routing, prefill/decode disaggregation, hierarchical KV cache offloading, and traffic- and hardware-aware scheduling improve throughput and reliability across multi-node serving clusters. It also covers how llm-d integrates vLLM with the Kubernetes Gateway API and operational tooling to create a modular serving stack that can be deployed, scaled, and tuned for real-world production traffic.

Training trillion-parameter MoE models on long-horizon agentic tasks demands that inference and optimization scale independently. Using GLM-5.1 as a case study, this talk presents Prime Intellect's asynchronous prime-rl stack, including runs at up to 131K sequence length with 256 rollouts and sub-five-minute steps on 28 H200 nodes. It explains how vLLM powers low-precision inference, prefill/decode disaggregation, prefix-aware routing, and distributed KV cache storage, while the trainer combines FSDP, context parallelism, and expert parallelism. Attendees will see how these pieces reduce idle time and make frontier-scale agentic post-training more efficient.

Agentic production traffic brings long multi-turn sessions, heavy prefix reuse, and bursty, heterogeneous requests that challenge conventional serving. This talk explains how vLLM addresses those patterns through independently scaled prefill/decode disaggregation, turn-aware serving, and session and correlation hints that improve prefix caching, prefetching, and eviction. It also covers distributed KV cache transfer and storage with Mooncake and multi-tier offloading, allowing cached context to move across turns and instances instead of being recomputed. Attendees will leave with practical guidance for architecting and tuning vLLM deployments for production agents.

This talk examines how vLLM and AMD AITER move ROCm support beyond simple porting through workload-aware kernel orchestration. It covers ROCM_AITER_FA's explicit routing of mixed prefill, extend, and decode traffic to specialized kernels, along with batch reordering, chunked long-context processing, hardware-optimized KV cache layouts, and AITER MLA backends for DeepSeek-class models. Benchmarks showing 1.2-4.4x throughput gains across AMD Instinct GPUs illustrate how software-hardware co-design turns ROCm into a high-performance production serving platform.

A single agent turn is rarely a single model call—it can span persisted context, tool execution, and multiple rounds of inference. This talk traces Agentic API from architectural proposal to v0.1.0: a Rust-powered stateful layer that rehydrates Responses conversations into self-contained vLLM requests, orchestrates tool loops, and supports HTTP, SSE, and WebSocket transports. We will show how this architecture enables Codex and other agent clients to run on open models today—and why agent state, token-prefix identity, KV-cache reuse, and state-aware routing matter to vLLM’s future.

As LLM applications evolve to agentic systems and RL rollouts, KV cache efficiency has become a primary determinant of serving performance. In this talk, we explore how Ray Serve LLM improves throughput and latency under heterogeneous workloads through KV-aware routing. We compare KV-aware routing against traditional approaches such as consistent hashing, highlighting how workload characteristics influence cache locality and overall system efficiency. Through evaluations on agentic and RL rollout workloads, we demonstrate how cache-aware serving can significantly improve performance, enabling Ray Serve LLM to deliver higher utilization and lower latency across diverse real-world applications.

This session delivers a technical deep dive into vLLM on Intel platforms. It introduces Intel's optimized attention, Mixture-of-Experts (MoE), and GEMM operators powered by Triton and SYCL/SYCL-TLA (CUTLASS)/oneDNN. We will showcase how Intel next-generation GPUs support native sub-8-bit microscaling formats (MXFP8/MXFP4). Additionally, we demonstrate how Intel eliminates inter-token latency (ITL) spikes by decoupling compute-bound prefill and memory-bound decode stages onto dedicated nodes. Finally, we highlight Intel's backend support for vLLM-Omni, enabling the seamless pipelining of heterogeneous stage graphs for complex, any-to-any multimodal models (text, video, and audio DiTs).

TorchSpec is a torch-native speculative decoding training framework built on top of Ray. We introduce a disaggregated way of training speculative decoding draft models, allowing each side to scale independently. Draft models for large models such as Kimi K2.5 trained with TorchSpec have accumulated more than 500K downloads on Hugging Face.

DigitalOcean's vLLM deployments reached leading Artificial Analysis results across frontier open-weight models, including 230 tokens per second per user on DeepSeek V3.2 and sub-second time to first token for 10,000-token Qwen 3.5 prompts. This talk breaks down the open-source optimization work behind those results, from low-batch kernel fusion and EAGLE3 speculative decoding to linear-attention fusions and dual-stream execution. Attendees will see how profiling model-specific bottlenecks and upstreaming the fixes into vLLM can deliver production inference performance that rivals proprietary stacks.