Day 0 Support for Qwen3.8-2.4T-A95B on vLLM

We are announcing Day-0 vLLM support for Qwen3.8-2.4T-A95B. This is the first model from the Qwen family to bring a Qwen-Max-class model to open-weight release.

Qwen3.8-2.4T-A95B is built on the Qwen 3.5 architecture and runs on vLLM out of the box. In addition to the official FP8 and BF16 checkpoints, Inferact has released MXFP4 and NVFP4-quantized weights that match full-precision quality while significantly reducing memory and bandwidth overhead.

Qwen3.8-2.4T-A95B is a 2.4-trillion-parameter sparse MoE model featuring 512 experts. Within its 92-layer hybrid backbone, full attention is applied at every 4th layer while the remaining 69 layers run linear attention. As one of the largest open-weight models released to date, running inference requires at least two NVIDIA B300 / AMD MI355X nodes (or a single node for the FP4 quantized version).

TL;DR

• Day-0 support: Qwen3.8-2.4T-A95B reuses the Qwen 3.5 architecture and runs on vLLM from day one with no architecture changes required.
• Flexible precision: FP8, BF16, NVFP4, and MXFP4 checkpoints are available.
• Multi-vendor optimization: Validated across hardware partners including NVIDIA and AMD.

Quick start

For NVFP4:

# See recipes for the exact docker run command
vllm serve Inferact/Qwen3.8-2.4T-A95B-NVFP4 \
  --linear-backend flashinfer_cutedsl \
  --tensor-parallel-size 8 \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_coder \
  --reasoning-parser qwen3 \
  --speculative-config '{"method":"mtp","num_speculative_tokens":3}'

For MXFP4:

vllm serve Inferact/Qwen3.8-2.4T-A95B-MXFP4 \
  --tensor-parallel-size 8 \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_coder \
  --reasoning-parser qwen3 \
  --speculative-config '{"method":"mtp","num_speculative_tokens":3}'

See the vLLM recipes for the full serving guide and recommended flags.

FP4 quantization: quality at lower cost

To minimize inference costs and maximize GPU memory efficiency, the Inferact team quantized selected layers — including the routed experts — to FP4 weights using Round-to-Nearest (RTN) quantization with activation calibration to enable 4-bit activations.

We ran initial verifications to confirm that quantization accuracy remains intact. Note that increasing the reasoning budget is required to reproduce these evaluation results.

Optimizations

To enable efficient inference for this 2.4T parameter model, we collaborated closely with NVIDIA and AMD to develop optimized kernels based on existing Qwen 3.5 support.

On NVIDIA platforms, NVIDIA and Inferact co-developed ultra-fast kernels for Linear Attention (Gated Delta Rule), Attention (GQA), Dense GEMMs, and MoE routing. New fused kernels were added to reduce communication overhead. Significant effort was also dedicated to identifying the best decomposition of work to maximize performance, including combining Data Parallelism and Tensor Parallelism for Attention and Expert Parallelism for the MoE.

On AMD Instinct GPUs, vLLM accelerates Qwen3.8 with AITER-fused Gated DeltaNet decode, attention, and MoE kernels, reducing kernel-launch and data-movement overhead. For Shared Expert MoE, the shared-expert path leverages highly optimized hipBLASLt GEMM kernels, while routed experts use AITER FusedMoE. AMD Quark quantization support enables efficient MXFP4 deployment, substantially reducing model memory requirements while maintaining strong accuracy.

Deployment tips

The Qwen 3.8 model card recommends the following generation parameters for optimal performance:

temperature=1.0, top_p=0.95, top_k=20, min_p=0.0, presence_penalty=0.0, repetition_penalty=1.0

Here is a Python client snippet to query the model once the vLLM server is running. Because Qwen 3.8 is a reasoning model, ensure you allocate a sufficient token budget for agentic workflows by setting a high max_tokens value.

from openai import OpenAI
 
client = OpenAI(api_key="EMPTY", base_url="http://localhost:8000/v1", timeout=3600)
 
resp = client.chat.completions.create(
    model="Qwen/Qwen3.8-2.4T-A95B",
    messages=[{"role": "user", "content": "Give me three primes above 100."}],
    temperature=1.0, top_p=0.95, max_tokens=128_000,
)
print(resp.choices[0].message.content)

Acknowledgements

We thank the Qwen team for releasing the model weights as well as their ongoing collaboration, and our hardware partners, NVIDIA and AMD, for their joint engineering contributions. We also thank the Inferact team for delivering quantized checkpoints and end-to-end vLLM integration, as well as the broader vLLM community. Thanks for our inference partners, including DigitalOcean, Together AI, who helped with early testing.