Beyond a Single Model: Building Mixture-of-Models Systems with vLLM Semantic Router

Most AI applications are built around a single model endpoint. But as models, devices, and deployment constraints diversify, no single model is the best fit for every request or environment. The practical question is how multiple specialized models can be coordinated, evaluated, and served through one interface. We call this systems approach Mixture-of-Models.

In less than a year since its public launch, vLLM Semantic Router has reached 5,000 stars, 150+ contributors, and more than 300,000 cumulative downloads across our Hugging Face model family. Across three major releases—Iris, Athena, and Themis—the system boundary moved from choosing a model, to governing multi-model inference, to preserving state and coordination across sessions. Those releases built the foundation for the MoM architecture envisioned from day 0.

This post describes the next step for vLLM Semantic Router: moving from routing among models to building dependable model systems from them. Under one versioned contract, independent models, policies, preferences, and execution paths become a system that can be trained, evaluated, exported, imported, deployed, and invoked through one interface. Our goal is to make vLLM Semantic Router a training, evaluation, and inference engine for Mixture-of-Models.

How vLLM-SR Got Here

The first vLLM Semantic Router post asked a practical question: why give simple and difficult requests the same reasoning budget? A lightweight classifier used fixed domain labels to choose between fast and reasoning paths, helping vLLM spend inference compute more selectively.

Production traffic quickly exposed the limit of that design. Domain alone could not represent privacy, safety, context, language, modality, tools, preferences, latency, and authorization. A static label also could not account for an endpoint that was cheap but overloaded, capable but remote, or unsafe to switch into midway through an agent session.

We rebuilt the classifier layer around modular model support, shared LoRA computation, Rust/Candle inference, and Go integration. We then replaced fixed classification with a Signal–Decision architecture that separated observed evidence from policy and execution. This became the spine of the next three releases.

Iris made routing composable. Domain, keyword, embedding, factuality, feedback, and preference signals fed explicit decisions, while safety, PII protection, caching, hallucination detection, and tool selection became route-scoped behavior. Iris also introduced the MoM model family and described vLLM-SR as “System Level Intelligence for Mixture-of-Models.”

Athena added first-class model selection, memory and RAG, a multilingual and multimodal model stack, ROCm acceleration, and an operating dashboard. The project was becoming the control system around multi-model inference, not just a classifier in front of vLLM.

Themis turned that broader system into an operable contract:

Signals become projections. Projections feed decisions. Decisions choose algorithms. Algorithms select models.

Themis added session-aware agentic routing, replayable traces, stronger protocol support, an operator console, and runtime paths across AMD ROCm, NVIDIA CUDA, Intel OpenVINO, and CPU environments. It also made a route explainable: operators can see the evidence, policy, algorithm, and physical model behind each decision.

From Signal–Decision to Workload–Router–Pool

The releases built the runtime. Two project papers explained the architecture behind it.

The white paper, Signal Driven Decision Routing for Mixture-of-Modality Models, formalized the separation between neural evidence and symbolic policy. Fast heuristics and learned classifiers turn prompts, context, identity, safety, and modality into a structured signal vector; a Boolean engine then composes those signals into auditable policy. A typed neural-symbolic DSL parses and validates that policy before compiling it into deployable configuration. When the paper was published, the system covered thirteen signal types and thirteen model-selection algorithms, with per-decision plugins for caching, RAG, memory, safety, provider handling, and response validation.

The vision paper, The Workload–Router–Pool Architecture for LLM Inference Optimization, widened the frame. It argues that three variables have to be designed together:

• Workload: chat or agent, single-turn or multi-turn, warm or cold, prefill-heavy or decode-heavy
• Router: static semantic policy, online feedback or bandit adaptation, RL-based selection, and quality-aware cascades
• Pool: homogeneous or heterogeneous accelerators, prefill/decode topology, model placement, and KV-cache management

Those variables cannot be optimized independently. Workload shape changes which routing policy works; routing policy changes the required pool size and topology; pool state changes which route is efficient. Safety and privacy cut across all three dimensions, while cost, quality, latency, and energy define the optimization frontier. The paper maps the project's research into a 3 × 3 WRP matrix and identifies twenty-one open directions where those dimensions still need to meet.

Together, the papers made routing programmable and tied it to workload and hardware—the two foundations MoM brings under one model contract.

Meanwhile, the runtime was already moving beyond single-model selection. Fusion, ReMoM, Confidence, Ratings, and bounded Workflows let one request invoke a controlled collaboration among models. As the Micro-Agent work showed, a client can call one model name while the serving layer selects a recipe, fans out to workers, verifies or synthesizes their results, and returns one ordinary response.

Routing remains fundamental. It is how a Mixture-of-Models allocates work, applies policy, and coordinates its parts. But routing is the mechanism. The model system is the product.

Why the Model Boundary Has to Move

Today's AI stack is fragmented along four axes:

• Models are fragmented. Closed frontier models, open general models, domain experts, compact local models, verifiers, and multimodal models will coexist. None wins simultaneously on quality, cost, latency, trust, privacy, and domain fit.

• Compute is fragmented. GPUs, CPUs, specialized accelerators, edge devices, cloud capacity, and private clusters differ in memory, kernels, availability, price, and energy use. Model choice and placement are becoming the same decision.

• Location is fragmented. Inference spans cloud, data center, and edge. Privacy or residency may rule out a stronger remote model, while a local workload may still need an on-demand cloud expert.

• Preference is fragmented. There is no universal “best.” Products and users make different tradeoffs among accuracy, latency, price, privacy, safety, style, and multimodality. Those choices should shape execution directly.

Today, each application has to reconcile these fragments on its own.

Mixture-of-Models moves that responsibility behind one model boundary.

At that boundary, intelligent allocation becomes part of the model. The engine determines which models are eligible, where execution can run, whether models should collaborate, and how to satisfy hard constraints.

Energy makes allocation inseparable from efficiency. Hardware and inference engines improve the supply side by producing more tokens per watt per dollar. The allocation layer controls demand: which work deserves those tokens, and which model or collaboration can provide them within the required quality, latency, and energy budget.

The application selects one versioned model identity and receives one attributable response. Its physical realization can still span open and closed models, cloud and edge, and different accelerator generations. The fragmentation remains, but it becomes internal to the model system instead of leaking into every application.

What We Mean by Mixture-of-Models

A Mixture-of-Models is a versioned composite model whose engine realizes each request through a preference-conditioned, resource-bounded path across independent models and operators. It is presented to the user through one model interface and returns one attributable result.

A multi-upstream gateway can forward traffic without owning system quality. An MoM owns an objective, an evaluation contract, a reproducible composition, and the runtime that executes it.

MoM also differs from Mixture-of-Experts. MoE routes tokens among internal experts during one forward pass; MoM coordinates independent models that may differ in architecture, owner, license, modality, protocol, context window, and hardware. An MoE checkpoint can itself be one MoM component.

A portable MoM therefore needs more than weights and configuration: it needs a component manifest, capability metadata, routing and collaboration recipes, policies, preferences, evaluation suites, runtime constraints, provenance, and version history.

Open checkpoints can travel with the artifact; closed models remain authenticated external references with explicit capability and policy contracts. Exporting an MoM does not make a proprietary checkpoint portable. It makes the model system reproducible.

Turn Preferences into Models

Preferences become concrete when they are published as model identities. One MoM family can offer several operating points:

Each name is a versioned model contract, not a router preset. The application chooses the behavior it needs; vLLM-SR selects and coordinates the models that deliver it while preserving hard privacy, residency, authorization, and safety constraints.

To an application, the full system remains an ordinary model call:

{
  "model": "vllm-sr/mom-v1-ultra",
  "messages": [
    {"role": "user", "content": "Review this design and identify its weakest assumption."}
  ]
}

That identity may select one model, escalate through a cascade, compare parallel answers, require grounding, or run a bounded workflow—without changing the external interface, version, or response contract.

Four planes separate ownership:

A deployment must map logical requirements onto the models and machines available in its environment. The proposal uses four objects:

1. The bundle fixes the interface, graph, policies, behavior variant, bounds, and immutable semantic assets.
2. The binding maps logical components to eligible deployments without changing the model's decision semantics.
3. The resolution lock freezes the constituent revisions, runtimes, images, accelerators, and provider observations.
4. The run record attributes every decision, call, constraint check, cost, and outcome to the bundle, binding, and lock that produced it.

This separation keeps portability honest. The same mom-v1-ultra can bind to ROCm, CUDA, a private CPU or NPU node, or a hybrid deployment without promising identical outputs from opaque providers. Instead, it preserves control semantics, exposes substitutions, and gives serving and evaluation the same resolved system.

vLLM-SR as the MoM Engine

Training, evaluation, and inference must share one contract; otherwise research, benchmarks, and production drift into different systems.

Training allocation, not only weights

MoM training covers router-owned embeddings, signal encoders, preference and safety models, and selectors. It also learns allocation and collaboration: which path fits a workload and budget, when a cascade should stop, how a panel should judge or synthesize, and when an agent session should switch models. Because constituents may be independent or closed, progress does not require gradients through all of them; policies, thresholds, pools, prompts, contracts, and topology can be optimized from traces and outcomes.

The target is a frontier across quality, latency, cost, safety, privacy, reliability, locality, and energy. Replay and outcomes feed production experience back into offline training without letting the hot path silently rewrite policy.

Evaluating the MoM as one model

Evaluation must score the model identity end to end; backend benchmarks are inputs, not the result. A versioned scorecard should measure routing regret, collaboration gain, recovery, session continuity, tail latency, cost, safety, privacy, and energy. It should stress provider failures, device loss, model disagreement, workload drift, and preference changes. Each declared operating point also needs its own test: flash on its latency–quality frontier, light against its quality floor, and ultra within its budget.

The scientific test is stricter than asking whether more calls improve a benchmark. Under matched active compute, can a conditional system exploit complementary strengths and failure modes better than the best fixed model? Without that control, MoM can hide brute-force scaling behind a clever graph. Evaluations must report calls, tokens, cost, latency, and energy alongside quality—and publish when composition does not help.

Executing intelligence at inference time

At inference time, the engine decides whether one model is enough. It may choose a local specialist, preserve a warm session, escalate through a confidence cascade, require retrieval or verification, run a Fusion panel, or execute a bounded workflow. The runtime owns the budget, topology, fallback, trace, and response contract; the application makes a normal model call.

One Model That Can Move

Our target is a complete MoM that can be built, exported, imported, versioned, evaluated, deployed, and invoked as a unified model. A logical specification compiles into an immutable bundle, binds to an environment, resolves the concrete deployment, and retains the same identity for serving and evaluation.

The artifact should run across developer machines, private clusters, cloud fleets, and edge environments while its physical realization changes. A specialist may resolve to an admissible local checkpoint or managed endpoint; an accelerator runtime may be replaced. If privacy makes a remote expert unavailable, the engine follows a declared fallback or abstention path. A binding cannot silently rewrite the graph, relax a guard, or turn a panel into a cascade—those changes require a new model version.

“Run on any hardware” is an architectural requirement, not a claim that every component is portable today. The project already supports paths across ROCm, CUDA, OpenVINO, and CPU. Next, hardware capability and placement become part of the MoM contract, allowing the engine to map the model system onto what is available.

The standard for the user experience is simple:

One model identity. Many models. Any hardware.

If the application needs to know which provider owns every submodel, which device runs it, or which fallback graph to execute, the abstraction has leaked.

What Changes Now

The next stage focuses on four connected areas:

1. Define a portable MoM specification. Package components, objectives, policy, preferences, evaluation, constraints, and execution semantics as one versioned artifact.
2. Close the training–evaluation–inference loop. Improve models and recipes from evaluation and replay, then ship them through reviewable, rollback-safe releases.
3. Build a heterogeneous runtime. Map one MoM across cloud, data center, and edge using hardware, locality, energy, and data boundaries as inputs.
4. Keep the model interface boring. Make an MoM as easy to import, deploy, and invoke as a single model.

This is a research program for how independent models should specialize, compete, verify, and collaborate; how to measure the resulting system; and how one model contract can survive across devices and environments. Our mission is:

Advancing the science of intelligence across models, devices, and environments.

We will study when composition produces capabilities beyond a single checkpoint, treat placement and energy as part of intelligence, and carry the same model contract from edge to cloud and from research to production.

Build It With Us

Building Mixture-of-Models requires more than routing. The work spans model training, evaluation, serving systems, hardware, and production operations.

Iris, Athena, and Themis improved because contributors brought real workloads, added backends, trained models, published benchmarks, found failure cases, and argued for better interfaces. MoM needs the same range of work: learned allocation, preference optimization, model cooperation, energy-aware inference, portable artifacts, open evaluation, and heterogeneous runtimes.

If you work on these problems, we want to learn from your workloads and measurements. Build an operating point, add a runtime, test a collaboration recipe, or publish a case where composition fails. MoM will be stronger if its assumptions are tested in the open.

Acknowledgments

vLLM-SR has grown through work across engineering, research, and the wider ecosystem. We thank Xunzhuo Liu, Huamin Chen, Bowei He, Yankai Chen, Fuyuan Lyu, and Steve Liu for helping shape its technical and research direction. We also thank Andy Luo and Haichen Zhang for their work on ROCm enablement, router-model training, and open MoM experimentation.

The work has also been carried by FAUST, David Shrader, Yang Wu, Ramakrishnan Sathyavageeswaran, Kuntai Wu, Aayush Saini, siloteemu, Chen Wang, Yue Zhu, Senan Zedan, Yossi Ovadia, Samzong Lu, Liav Weiss, Asaad Balum, Yehudit, Noa Limoy, Marina Koushnir, Jared Wen, Abdallah Samara, Hen Schwartz, Srinivas A, Yang Zhu, Jintao Zhang, yuluo-yx, cryo, Bishen Yu, Zhijie Wang, Hao Wu, and Qiping Pan. Their code, reviews, testing, documentation, and stewardship carried the project from one release to the next.

At this milestone, the project stands at 1,734 commits and 150+ contributors. We thank collaborators at MBZUAI, McGill University, Mila, and Rice University, and the broader vLLM, AMD, Intel, Meta, Red Hat, Microsoft, Google, IBM, NVIDIA, Hugging Face, NASA, Nutanix, DaoCloud, and open-source communities. This milestone belongs to everyone who helped turn an early router into a real system.

Join us on GitHub, explore the documentation, try the MoM model family, and meet the community in the #semantic-router channel on vLLM Slack.

vLLM Semantic Router began by helping infrastructure choose the right model for each request.

Now we are extending that foundation beyond a single model: toward systems that can coordinate, evaluate, and operate multiple models across devices and environments.

We invite the community to help build and test that approach in the open.