Elastic Expert Parallelism in vLLM
·11 min read
Expert parallelism (EP) is a key technique for serving Mixture-of-Experts (MoE) models at high throughput. WideEP deployments (where EP spans many workers) maximize KV cache capacity, enabling...
#large-scale-serving
12 posts
Serving Agentic Workloads at Scale with vLLM x Mooncake
·10 min read
TL;DR: Agentic workloads generate massive shared prefixes that are often recomputed across turns. By integrating Mooncake's distributed KV cache store into vLLM, we achieve 3.8x higher throughput,...
Driving vLLM WideEP and Large-Scale Serving Toward Maturity on Blackwell (Part I)
·10 min read
Building on our previous work achieving 2.2k tok/s/H200 decode throughput with wide-EP, the vLLM team has continued performance optimization efforts targeting NVIDIA's GB200 platform. This blog...
vLLM Large Scale Serving: DeepSeek @ 2.2k tok/s/H200 with Wide-EP
·8 min read
In v0.11.0, the last code from vLLM V0 engine was removed, marking the complete migration to the improved V1 engine architecture. This achievement would not have been possible without vLLM’s...
Encoder Disaggregation for Scalable Multimodal Model Serving
·9 min read
Modern Large Multimodal Models (LMMs) introduce a unique serving-time bottleneck: before any text generation can begin, all images must be processed by a visual encoder (e.g., ViT). This encoder...
vLLM Router: A High-Performance and Prefill/Decode Aware Load Balancer for Large-scale Serving
·5 min read
Efficiently managing request distribution across a fleet of model replicas is a critical requirement for large-scale, production vLLM deployments. Standard load balancers often fall short as they...
Streamlined multi-node serving with Ray symmetric-run
·4 min read
Ray now has a new command: ray symmetric-run. This command makes it possible to launch the same entrypoint command on every node in a Ray cluster, simplifying the workflow to spawn vLLM servers...
Inside vLLM: Anatomy of a High-Throughput LLM Inference System
·41 min read
In this post, I'll gradually introduce all of the core system components and advanced features that make up a modern high-throughput LLM inference system. In particular I'll be doing a breakdown...
Accelerating RLHF with vLLM, Best Practice from OpenRLHF
·5 min read
As demand grows for training reasoning-capable large language models (LLMs), Reinforcement Learning from Human Feedback (RLHF) has emerged as a cornerstone technique. However, conventional RLHF...
Introducing AIBrix: A Scalable, Cost-Effective Control Plane for vLLM
·4 min read
Today, we are excited to announce vllm-project/aibrix: a battery-included vLLM Kubernetes serving stack developed by Bytedance. Started in early 2024, AIBrix has been successfully deployed to...
Distributed Inference with vLLM
·5 min read
Serving large models often leads to memory bottlenecks, such as the dreaded CUDA out of memory error. To tackle this, there are two main solutions:
High Performance and Easy Deployment of vLLM in K8S with vLLM production-stack
·4 min read
- vLLM boasts the largest open-source community, but what does it take to transform vLLM from the best single-node LLM engine to a premier LLM serving system? - Today, we release “vLLM...