Announcing vllm-metal: Concurrent Serving on Apple Silicon

Local inference on a Mac is straightforward until several requests overlap. Then time to first token (TTFT), memory growth, and admission control become serving problems rather than model-execution problems. vllm-metal brings vLLM's scheduler, paged KV cache, and OpenAI-compatible server to Apple Silicon, with MLX and Metal handling execution.

Our first official release, v0.28.0, aligned vllm-metal's version numbering with upstream vLLM. It introduced batched multi-token prediction (MTP), GGUF and hybrid-model support, and faster prefill on M5. You can install v0.29.0 with Homebrew.

How vllm-metal fits into vLLM

vllm-metal plugs into upstream vLLM. vLLM provides the V1 scheduler, paged KV block management, chunked prefill, sampling, and the OpenAI-compatible frontend with streaming and tool-call parsing. mlx_lm provides the model implementations; MLX executes them.

At the model level, vllm-metal reuses mlx_lm's weight loading, RMSNorm, linear, MoE, and MLP layers unchanged. Those layers process each token independently, so they run on a packed token axis without knowing request boundaries. Attention does need those boundaries, so vllm-metal replaces stock attention with a paged varlen Metal kernel. Most of the plugin's model-specific code therefore sits in one layer.

Start an OpenAI-compatible server

On Apple Silicon with macOS 15 or later, install the stable release with Homebrew:

brew tap vllm-project/vllm-metal https://github.com/vllm-project/vllm-metal
brew install vllm-project/vllm-metal/vllm-metal

Homebrew manages Python and the dependencies. Run vllm directly to launch a model:

# --gpu-memory-utilization sets the serving memory budget; see below.
vllm serve Qwen/Qwen3.5-0.8B --gpu-memory-utilization 0.5
 
# 64 GB Macs: the 27B hybrid
# vllm serve mlx-community/Qwen3.8-27B-4bit --gpu-memory-utilization 0.7
 
# Speculative decoding: Gemma 4 with its MTP assistant
# vllm serve google/gemma-4-E4B-it --gpu-memory-utilization 0.5 \
#   --max-model-len 16384 --no-async-scheduling \
#   --speculative-config '{"method":"mtp","model":"mlx-community/gemma-4-E4B-it-assistant-bf16","num_speculative_tokens":1}'

See the model matrix for more models and the installation guide for other installation methods.

The server speaks the OpenAI API:

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model": "Qwen/Qwen3.5-0.8B",
       "messages": [{"role": "user", "content": "Say hi"}]}'

Anything that takes an OpenAI-compatible base URL can point at http://localhost:8000/v1, coding agents included; the vLLM docs cover Claude Code and Codex setup.

Set a predictable memory budget

vllm-metal's memory guard lets you set an inference budget with --gpu-memory-utilization, leaving headroom for macOS and your apps. Like upstream vLLM, it runs a warmup pass at startup to account for model weights, activations, and temporary buffers before allocating the remaining budget to a fixed KV cache. It also caps MLX's reusable buffer cache to prevent memory from accumulating during serving. Requests that do not fit in the KV pool wait until pages become available.

Packed queries and paged KV

In mlx_lm's padded batches, attention queries have shape [B, H, T_max, D]: every request gets the longest query length in the batch. MLX's scaled_dot_product_attention has no varlen interface.

vllm-metal preserves vLLM V1's unified model step for chunked prefill and decode. It packs every scheduled query token into [total_q, H, D], with cu_seqlens marking request boundaries, and runs the mixed step in one model forward.

KV is separate: mlx_lm keeps a contiguous [B, H, T, D] cache, while vllm-metal stores KV in fixed-size pages addressed by per-request block tables. Admitted requests can grow without reshaping a padded cache.

Prefill and decode use different batching strategies:

Prefill attention describes how prompt queries enter the attention kernel: separately, padded to a common length, or concatenated. Decode batching processes multiple requests in one model step; “+ prefill” includes prompt tokens in that batch. KV describes logical storage: contiguous buffers, individual token cells, or token blocks.

On Qwen3.6-35B-A3B in 4-bit, we compared batches of eight requests with roughly 6,000 prompt tokens in total and 20 output tokens per request. Batch A has similar prompt lengths; batch B has one longer prompt. The table reports batch wall time in seconds, with prompt lengths rounded.

Packing also keeps ragged work in one batch. In a speculative step, one request may contribute a single decode token, another its last token plus a request-specific number of drafts, and another a prefill chunk. A [B, H, T_max, D] query tensor must either pad those rows to a common width or split them across forwards. vllm-metal instead concatenates the windows as [total_q, H, D] and verifies them in one target-model forward.

The Metal kernel ports vLLM's unified Triton kernel, described in The Anatomy of a Triton Attention Kernel, to Apple GPUs, down to the binary search each threadgroup runs over cu_seqlens to find which request owns its query token.

Concurrent serving under agent load

Multiple coding agents or sessions can send model requests at the same time. We measured this with SiliconBench's agent split: 100 multi-turn prompts with roughly 4.6K input tokens each, on an M5 Pro with 64 GB of memory. All three model comparisons use 4-bit weights.

The SiliconBench paper provides a broader evaluation of nine Apple Silicon serving engines, covering speed, memory use, and output fidelity.

Each concurrency level starts with a fresh server. oMLX is shown with its default SSD cache and with a RAM-only cache; both start empty. The appendix gives the serving configurations, and the figure captions report completion counts.

Qwen3.8-27B

vllm-metal has the lowest TTFT and end-to-end latency at concurrency 2 and 4. oMLX with SSD offload leads at concurrency 1.

Gemma 4 E4B

For Gemma 4 E4B, we extend the sweep to concurrency 16 and include vllm-metal with its MTP drafter.

vllm-metal keeps TTFT low throughout this sweep. llama.cpp uses its default four server slots.

Qwen3.6-35B-A3B

Qwen3.6-35B-A3B has 35B total parameters with 3B active per token. It combines mixture-of-experts layers with standard attention and gated-delta-net (GDN) linear attention.

At concurrency 4, vllm-metal and oMLX with a RAM cache are close in throughput and end-to-end latency, with a larger gap in TTFT. mlx_lm's line covers only the minority of requests it completed.

Batched MTP under concurrent load

MTP uses an assistant model to draft tokens, which the target model verifies within the continuous batch. The table compares one draft token per step with generation without MTP on Gemma 4 E4B:

MTP is opt-in through --speculative-config. The Metal path currently supports Gemma 4 with plain greedy sampling (temperature=0) and synchronous scheduling (--no-async-scheduling).

Other features in v0.28.0

Faster prefill on M5

On M5 Macs, vllm-metal automatically uses the NAX attention kernel, which uses the GPU's tensor hardware, for compatible prefill batches. Earlier Macs keep using the existing path. This comparison uses Qwen3-0.6B:

Reusing conversation history on hybrid models

Multi-turn agents resend most of their growing conversation on every turn. Prefix caching lets the next turn reuse blocks computed for earlier turns instead of prefilling the full history again.

For Qwen3.5-style hybrid models, vllm-metal supports vLLM's align mode. It saves GDN recurrent state at the same block boundaries as attention KV, allowing both to resume from a cached prefix (PR #634). This path remains experimental and cannot yet be combined with speculative decoding.

Models and serving features

v0.28.0 also included:

• LoRA adapters, structured outputs, and three speculative-decoding methods: Gemma 4 MTP, separate draft models, and prompt-lookup n-grams.
• GGUF checkpoints, including Hugging Face config sources for local GGUF weights.
• Hybrid-attention models from the Qwen3.5, Qwen3.6, Qwen3.8, and Qwen3-Next families.
• Pipeline parallelism across multiple Macs over the MLX ring backend.
• Experimental vision-language models, text embeddings and reranking, and speech-to-text.

The supported-model matrix and feature guides are in the vllm-metal documentation.

The same stack from M1 Pro to M5 Pro

We ran the same Gemma 4 E4B 4-bit workload at concurrency 1 and 8 on four Macs.

brew tap vllm-project/vllm-metal https://github.com/vllm-project/vllm-metal
brew install vllm-project/vllm-metal/vllm-metal

vllm serve mlx-community/gemma-4-e4b-it-4bit \
  --gpu-memory-utilization 0.5 \
  --max-model-len 2048 \
  --no-enable-prefix-caching \
  --host 127.0.0.1 --port 8000

curl -fsSL https://raw.githubusercontent.com/vllm-project/vllm/main/benchmarks/sonnet.txt \
  -o sonnet.txt
 
BENCH_MACHINE=m1pro-32gb
BENCH_CONCURRENCY=1
 
vllm bench serve \
  --backend vllm \
  --model mlx-community/gemma-4-e4b-it-4bit \
  --base-url http://127.0.0.1:8000 \
  --dataset-name sonnet --dataset-path sonnet.txt \
  --sonnet-input-len 1024 --sonnet-output-len 128 \
  --num-prompts 100 --num-warmups 3 \
  --request-rate 10 --max-concurrency "$BENCH_CONCURRENCY" \
  --temperature 0 --ignore-eos --seed 0 \
  --save-result --result-dir benchmark-results \
  --result-filename "${BENCH_MACHINE}-c${BENCH_CONCURRENCY}.json"

Appendix: benchmark reproduction

The cross-engine benchmark scripts and results are in SiliconBench. Runs use vllm-metal 0.28.0.dev20260901062632 with vLLM 0.28.0, llama.cpp 0eadefeb, and oMLX dc312e6e.

Runs use greedy sampling in a closed loop at fixed concurrency. Results cover completed requests; empty responses count as failures. Each engine uses its own 4-bit conversion. Memory settings are vllm-metal's default --gpu-memory-utilization 0.92, oMLX's balanced memory guard, and no explicit cap for llama.cpp.

# llama.cpp
llama-server -m <model>.gguf --host 0.0.0.0 --port 8001 \
  -ngl 99 --parallel 4 -c 65536
 
# vllm-metal
vllm serve <model> --host 0.0.0.0 --port 8004 \
  --enable-prefix-caching --max-model-len 16384
 
# vllm-metal + MTP (Gemma only)
vllm serve <model> --host 0.0.0.0 --port 8004 \
  --enable-prefix-caching --max-model-len 16384 --no-async-scheduling \
  --speculative-config '{"method":"mtp","model":"mlx-community/gemma-4-E4B-it-assistant-bf16","num_speculative_tokens":1}'
 
# oMLX, SSD offload
omlx serve --model-dir <dir> --host 0.0.0.0 --port 8005 \
  --paged-ssd-cache-dir <fresh-empty-dir> --paged-ssd-cache-max-size 100GB \
  --hot-cache-max-size 0
 
# oMLX, RAM-only cache
OMLX_HOT_CACHE_ONLY=true omlx serve --model-dir <dir> --host 0.0.0.0 --port 8005 \
  --paged-ssd-cache-dir <fresh-empty-dir> --paged-ssd-cache-max-size 100GB \
  --hot-cache-max-size 8GB

Acknowledgments

vllm-metal builds on MLX and mlx_lm from Apple's MLX team, mlx-vlm for the vision-language paths, and vLLM's engine and hardware-plugin interface. Thanks to the upstream vLLM maintainers for review and support along the way, and to everyone who filed issues and shared benchmarks against the v0.2 and v0.3 releases.