Micro-Agent: Beat Frontier Models with Collaboration inside Model API
·15 min read
How vLLM Semantic Router turns vllm-sr/auto into a bounded micro-agent runtime for Confidence, Ratings, ReMoM, Fusion, Workflows, and benchmark-shaped collaboration.
#ecosystem
25 posts
Beyond One Model: Fusion in vLLM Semantic Router
·10 min read
How vLLM Semantic Router Fusion runs a panel of models, uses a judge to analyze agreement and gaps, and synthesizes one answer while preserving routing policy, traces, and OpenAI-compatible serving.
DiffusionGemma: The First Diffusion LLM (dLLM) Natively Supported in vLLM
·6 min read
How vLLM supports DiffusionGemma, the first native diffusion language model in vLLM, using Model Runner V2 state hooks, iterative denoising, bidirectional attention, and reused speculative decoding paths.
Announcing vime: A Simple, Stable, and Efficient RL Framework for LLMs
·6 min read
vime connects slime's training stack with vLLM rollouts to provide a simple, stable, and efficient RL post-training pipeline.
vLLM Semantic Router v0.3 Themis: From Signals to Stateful Production Routing
·22 min read
What vLLM Semantic Router v0.3 Themis adds for production routing: canonical config, inspectable signal-decision-policy flows, safer operations, CLI/dashboard/Kubernetes alignment, and replayable routing behavior.
Fast & Efficient LLM Inference with vLLM: A New Course with DeepLearning.AI
·5 min read
What the DeepLearning.AI vLLM course teaches: optimizing, deploying, and benchmarking LLM inference with LLM Compressor quantization, GuideLLM, KV cache sizing, serving, and memory tradeoffs.
Session-Aware Agentic Routing: Continuity-Aware Model Selection for Long-Horizon LLM Agents
·14 min read
How Session-Aware Agentic Routing in vLLM Semantic Router preserves long-horizon agent continuity with session memory, safe model-switch boundaries, prefix-cache-aware switch pricing, and replayable traces.
Speculators v0.5.0: DFlash Support and Online Training
·6 min read
What Speculators v0.5.0 adds for vLLM speculative decoding: DFlash block-diffusion draft models, unified online and offline training, native hidden-state extraction, and Gemma 4 latency results.
From Text to Multimodal Routing: Hardening Vision Signals in vLLM Semantic Router
·12 min read
How vLLM Semantic Router hardens multimodal routing by turning visual evidence into trustworthy signals, debugging a Rust/Candle vision-encoder parity issue, and validating image signal correctness for production policy.
Announcing VeRL-Omni: Easy, Fast, and Stable RL Training for Diffusion and Omni-Modality Models
·7 min read
How VeRL-Omni extends verl with vLLM-Omni for reinforcement learning post-training of diffusion and multimodal generative models, including efficient rollouts, reward inference, trainers, hardware support, and recipes.
vLLM Semantic Router v0.2 Athena: ClawOS, Model Refresh, and the System Brain
·23 min read
What vLLM Semantic Router v0.2 Athena adds: refreshed multilingual and multimodal routing models, ONNX and ROCm acceleration, safety and memory signals, long-context handling, and ClawOS orchestration.
Building Mixture-of-Models on AMD GPUs with vLLM-SR
·10 min read
How vLLM Semantic Router builds a Mixture-of-Models system on AMD MI300X and MI355X GPUs, routing across specialized models with signals, decisions, safety checks, semantic caching, and live MoM deployment.
vLLM Semantic Router v0.1 Iris: The First Major Release
·9 min read
What vLLM Semantic Router v0.1 Iris introduces: signal-decision plugin architecture, model selection, safety filtering, semantic caching, hallucination detection, LoRA-based routing models, and production-ready MoM routing.
Introducing vLLM Playground: A Modern Web Interface for Managing and Interacting with vLLM Servers
·6 min read
How vLLM Playground provides a web UI for starting, configuring, testing, and monitoring vLLM servers across local macOS, Linux GPU or CPU, Kubernetes, and OpenShift environments.
vLLM-Omni Diffusion Cache Acceleration
·4 min read
How vLLM-Omni speeds up diffusion model inference with Cache-DiT and TeaCache, reusing intermediate computations across timesteps to deliver 1.5x to 2x image generation speedups with minimal quality loss.
AMD × vLLM Semantic Router: Building the System Intelligence Together
·11 min read
How AMD and vLLM Semantic Router build GPU-accelerated Mixture-of-Models routing with signals, semantic caching, response storage, PII, jailbreak, and hallucination guardrails.
Token-Level Truth: Real-Time Hallucination Detection for Production LLMs
·13 min read
How HaluGate adds token-level hallucination detection to vLLM Semantic Router by verifying assistant claims against tool outputs and grounding context in real time without LLM-as-judge overhead.
Diving into speculative decoding training support for vLLM with Speculators v0.3.0
·11 min read
How Speculators v0.3.0 supports end-to-end Eagle3 draft model training for vLLM, including hidden-state data generation, MoE and non-MoE verifiers, offline workflows, and seamless speculative decoding serving.
Advancing Low‑Bit Quantization for LLMs: AutoRound x LLM Compressor
·5 min read
How Intel AutoRound integrates with LLM Compressor to produce low-bit quantized checkpoints for vLLM, using tuning-based PTQ, W4A16 and related formats, compressed-tensors compatibility, and lightweight calibration.
Announcing vLLM-Omni: Easy, Fast, and Cheap Omni-Modality Model Serving
·4 min read
What vLLM-Omni adds to the vLLM ecosystem: omni-modality serving for text, image, video, and audio, diffusion and non-autoregressive generation support, disaggregated stages, OpenAI-compatible APIs, and pipelined execution.
Signal-Decision Driven Architecture: Reshaping Semantic Routing at Scale
·14 min read
How vLLM Semantic Router replaces fixed domain classification with signal-decision architecture, combining multi-dimensional signals, AND/OR decision logic, model selection, and plugin orchestration for production routing.
From Monolithic to Modular: Scaling Semantic Routing with Extensible LoRA
·8 min read
How vLLM Semantic Router refactors its Rust classification layer with modular model support, Qwen3-Embedding, EmbeddingGemma, LoRA-based multi-task classification, and concurrent routing execution.
vLLM Semantic Router: Next Phase in LLM inference
·4 min read
How vLLM Semantic Router routes requests by intent, covering semantic classification, smart reasoning-path selection, Rust and Candle execution, and Kubernetes Envoy integration for efficient inference.
Introducing AIBrix: A Scalable, Cost-Effective Control Plane for vLLM
·4 min read
What AIBrix adds as a Kubernetes control plane for vLLM: LoRA management, LLM gateway routing, autoscaling, unified runtime, distributed inference, distributed KV cache, heterogeneous serving, and GPU failure detection.
High Performance and Easy Deployment of vLLM in K8S with vLLM production-stack
·4 min read
What vLLM production-stack adds for Kubernetes serving: prefix-aware routing, LMCache-backed KV cache sharing, autoscaling, observability, fault tolerance, and cluster deployment with higher throughput and lower latency.