VeRL-Omni v0.2.0: Faster Diffusion RL and Stable Omni Training

Following our May announcement of VeRL-Omni, v0.2.0 establishes a stronger foundation for production-grade omni-modal reinforcement learning. This release improves the training stack across rollout performance, model integration, reward support, hardware coverage, and documentation, with two changes carrying the most impact:

• Faster diffusion RL, centered on higher-throughput Qwen-Image FlowGRPO rollout via vLLM-Omni and verl V1 trainer support.
• Stable omni training, built around the omni V1 trainer, reusable model adapters, FSDP2, and vLLM-Omni rollout.

1. Faster Diffusion RL

Diffusion RL is expensive, but not in the same way as autoregressive language-model RL. A single rollout carries many denoising steps, large latent tensors, prompt embeddings, optional classifier-free guidance, reward-model scoring, old-log-prob recomputation, and policy-weight synchronization. For Qwen-Image FlowGRPO, there is no single villain in the profile. Step time is shaped by rollout generation, old-log-prob computation, reward scoring, actor update, and LoRA weight sync together.

Key Features

The faster diffusion RL work has two main features.

• Request-level batching leads the rollout side. For supported diffusion adapters, it becomes the default vLLM-Omni rollout path. Instead of sending diffusion generations through a serial loop, vLLM-Omni packs compatible requests into larger transformer forwards and exposes explicit concurrency knobs for scheduling rollout work.

• The trainer path matters just as much. Diffusion now has a V1 trainer path, bringing diffusion RL closer to the modern trainer architecture used elsewhere in VeRL-Omni and laying the groundwork for decoupled rollout and training execution.

Faster rollout only matters if the generated trajectories and log-probs still describe the same policy. This release fixes several correctness-sensitive areas: request-batched diffusion log-probs, async rollout semantics, rank-local LoRA weight-update routes, and the hooks used by optional rollout-correction recipes.

New Support

The rollout batching guide explains both diffusion batching modes, how to enable them, and when to choose each mode. For the vLLM-Omni runtime design behind diffusion batching, see the diffusion continuous batching docs. Current faster diffusion RL support is organized around these recipes:

A full diffusion post-training support table in VeRL-Omni is available at README.md.

Recipe and Benchmark

The Qwen-Image LoRA OCR recipe is a good place to see the change. In the v0.1 line, rollout was the core bottleneck: each request effectively ran as serial B≈1 DiT forwards, with 10 denoising steps and True-CFG doubling each step into two forwards. GPU utilization hovered around 80%, not because the model was small, but because the engine could not keep enough diffusion work packed together.

In v0.2.0, vLLM-Omni's request-level packing changes that shape. Multiple complete requests are packed into one transformer forward, GPU utilization rises to about 100%, and isolated generation time drops from 226s to 108s, a 52% reduction. The same story shows up in per-image generation latency, which falls with the packed vLLM-Omni rollout path. Reference runs: Qwen-Image OCR LoRA v0.1 and Qwen-Image OCR LoRA v0.2.

In the charts below, the blue curve is v0.1 and the green curve is v0.2.

GPU utilization rises after request-level packing. Blue: v0.1; green: v0.2.

Generation time drops from the v0.1 path to the v0.2 path. Blue: v0.1; green: v0.2.

Step time follows the same trend. Blue: v0.1; green: v0.2.

The production-style Qwen-Image FlowGRPO LoRA recipe enables vLLM-Omni request-level batching by default. The main entry point is run_qwen_image_ocr_lora.sh for the baseline OCR reward setup. The acceleration comes from switching off step-wise execution and letting the vLLM-Omni engine schedule multiple rollout requests up to max_num_seqs:

actor_rollout_ref.rollout.step_execution=false
++actor_rollout_ref.rollout.engine_kwargs.vllm_omni.max_num_seqs=32

For Qwen-Image LoRA with True-CFG at 512 px, a practical tuning range is max_num_seqs=8 to 32; larger values can run into HBM pressure. SD3.5 has a lighter request-level memory shape and can use max_num_seqs=256.

The recipe-level step-time numbers line up with that story: the baseline Qwen-Image FlowGRPO LoRA run is about 420s per step on 4 × H800, while the async reward variant reaches about 360s per step on 5 GPUs.

2. Stable Omni Training

The other half of the release is stable omni training. Omni models are not just bigger language models; they are small systems with processors, modality-specific towers, trainable stages, and rollout-time behavior that has to stay aligned with the actor. v0.2.0 moves the project from model-specific integrations toward a reusable omni training stack, so multimodal autoregressive training fits more naturally into VeRL-Omni's trainer, adapter, rollout, and recipe structure.

Key Features

Here, the release pulls on two levers.

One lever is the verl V1 trainer architecture. Omni recipes get clearer worker orchestration, standard configuration overrides, and better alignment with vLLM-Omni rollout.

The other is the reusable omni model adapter layer. Instead of wiring each architecture as a one-off path, the trainer can rely on a shared interface for model setup, processor setup, trainable-stage selection, FSDP preparation, and rollout alignment.

The repository-level call flow is roughly:

Omni PPO trainer and OmniModelBase adapter call flow.

The module boundary is intentionally narrow. main_omni.py only decides that an online omni job should enter the verl PPO V1 path. The PPO trainer then owns the generic RL loop: rollout scheduling, advantage computation, and policy updates. When the actor model is built, the FSDP omni engine loads the Hugging Face model and asks OmniModelBase to resolve the adapter for the configured architecture and stage. That adapter is where model-specific work lives. For Qwen3-Omni thinker training, Qwen3OmniThinkerAdapter strips inactive modules, redirects forward to the thinker component, and prepares the processor and rollout alignment hooks before control returns to the PPO loop.

New Support

The current Qwen3-Omni adapter supports thinker-only training by redirecting training to the target component, stripping unused modules such as Talker and codec-related components, and working with FSDP/FSDP2 wrapping. Current stable omni training support is organized around these recipes:

A full omni post-training support table in VeRL-Omni is available at README.md.

Recipe and Benchmark

The best single recipe to highlight is MMK12. It exercises the new stable Qwen3-Omni path with real multimodal input: image plus text prompt, text answer, GSPO optimization, FSDP actor training, and vLLM-Omni rollout.

MMK12 anchor recipe. run_qwen3_omni_thinker_gspo_lora_mmk12_v1.sh trains Qwen3-Omni on K12 visual math reasoning (image -> text) with GSPO, LoRA rank 32, and colocated actor-rollout workers on 4 × H800 80GB. The rollout shape is 128 prompts × 16 responses, or 2048 samples per rollout. After training, the run reaches 0.833 validation reward, 0.998 actor-rollout Pearson correlation, and about 59 GB GPU memory usage. See some training results in the reference run: MMK12 (wandb).

MMK12 training rewards mean scores.

MMK12 validation rewards mean scores.

The MMK12 data pipeline converts raw MMK12 parquet shards into the verl RL parquet layout. Each row carries the image bytes inline and uses a prompt format that asks the model to produce a structured answer. The reward combines math_verify accuracy with a progressive format reward on the <answer>...\boxed{}...</answer> template.

To run the recipe:

python examples/gspo_trainer/data_process/mmk12.py \
    --local_dataset_path /path/to/mmk12/ \
    --local_save_dir ~/data/mmk12
 
TRAIN_FILE=$HOME/data/mmk12/train.parquet \
VAL_FILE=$HOME/data/mmk12/test.parquet \
bash examples/gspo_trainer/qwen3_omni/run_qwen3_omni_thinker_gspo_lora_mmk12_v1.sh

This anchors the v0.2.0 stability story: Qwen3-Omni training is no longer just a model-specific launch path. It is a V1 trainer recipe with a reusable omni adapter, multimodal data handling, actor-rollout consistency metrics, and a documented image-to-text benchmark.

Model and Algorithm Extensions

The release also expands the broader VeRL-Omni model and algorithm surface:

Outside the model-algorithm matrix, v0.2.0 also adds Ascend NPU Dockerfiles and install guidance.

Future Plan

• Optimize omni-modal models via fully async training.
• Extend new models and algorithms, such as MiniMax-H3, MiniCPM-o models, and OPD/M-OPD trainers.
• Make video diffusion model training more efficient via batching, TQ, and the V1 trainer.
• Harden diffusion and omni-modal rollout code for async training.
• Support agentic RL with multi-stage and multi-turn generation.

Join the Community

• Code: github.com/verl-project/verl-omni
• Docs: verl-omni.readthedocs.io
• Contribution Guideline: see CONTRIBUTING.md