Field noteNov 16, 20265 min read1 source

vLLM reported zero-mismatch training alignment for DeepSeek-V4.1-Flash

In one minute

Inferact, NVIDIA researchers and collaborators report a method for stabilizing asynchronous reinforcement-learning training by aligning rollout and learner kernels, then correcting weight lag with unclipped importance sampling. Their experiments include stable training at policy lag 64 and 2.40× the training-step throughput of synchronous training on a 16-GPU setup, while several flagship-model runs remain validation rather than long training results.

What changed: vLLM reports zero-mismatch alignment while training DeepSeek-V4.1-Flash with asynchronous reinforcement learning, extending its prior work on serving the model.

What happened

In a November 6, 2026, vLLM project blog post, authors from Inferact, NVIDIA Research, NVIDIA DevTech, AMD and the REINFORCE Zero team describe asynchronous RL training, in which rollout generation and learner updates run at the same time. This improves throughput but means that a sample can be generated with older policy weights than those used to train on it. The post separates that weight lag from kernel mismatch: the latter occurs when rollout and learner compute different log probabilities for the same weights and tokens. The authors argue that importance sampling can correct weight lag but does not reliably control fluctuations caused by kernel mismatch.

Their proposed REINFORCE Zero recipe has three parts. First, align rollout and learner kernels so they return identical log probabilities for the same weights and tokens. Second, correct the remaining weight lag with token-level importance sampling, using the ratio of the learner’s probability to the rollout’s recorded probability without clipping, filtering or rejection gates. Third, use plain REINFORCE, with each response’s advantage calculated as its reward minus its group’s mean reward. The post says the method uses no value model.

The blog reports that, on a 16-GPU setup and over 12,000 updates, REINFORCE Zero reached 2.40× the training-step throughput of synchronous training with aligned kernels: 68.55 hours versus 164.77 hours, with comparable quality. In a separate Qwen3-4B agentic tool-use run with Python tools and a 16K context, it trained through 5,000 updates at policy lag 64. Mean reward rose from 0.287 over the first 100 updates to about 0.78. The authors also report that standard-kernel importance sampling without alignment saw online AIME24 pass@1 fall from 34.27% to 13.96%; a sequence gate stabilized training but rejected 24.48% of sequences by the final 1,000 updates.

The post describes zero-mismatch validation on DeepSeek-V4.1-Flash and Kimi K3.1. For DeepSeek-V4.1-Flash, the team reports bit-for-bit agreement at every step in SWE-bench Pro agentic RL at 128K context, using 16 nodes with four GB200 GPUs each. A DAPO math rollout configuration with 128 trajectories per node and responses up to 8K tokens stayed within 10% of the fast baseline. For Kimi K3.1, a 100-step DAPO run with LoRA on four nodes with four GB300 GPUs each kept diff 0, while AIME 24/25/26 mean@8 rose from 0.451 to 0.664. The post says long asynchronous runs on these large models are still next steps.

What the archive adds

New Runtime covered serving DeepSeek-V4.1-Flash at scale on a single GB300 NVL72 rack in an earlier Field Note dated October 27, 2026. It also covered NVIDIA’s Blackwell-specific NVFP4 checkpoint in a November 1 note. This is a new event in the same model and infrastructure thread: training-kernel alignment, not serving performance or quantization.

Trend signal

This supports harness architecture outlives model choice: the reported stability depends on the training and rollout system aligning computation, rather than on model identity alone. The post also underscores that bringing a new model into the workflow requires checking its operators and numerical behavior, not assuming that an alignment recipe transfers unchanged.

Why now

The authors say DeepSeek-V4.1-Flash and Kimi K3.1 have now reached zero-mismatch validation, including Kimi support on its release day. They describe long asynchronous runs on those flagship models and further Kimi performance optimization as work still to come.

Why it matters

For teams training agents with asynchronous RL, throughput gains are useful only if stale samples remain safe to learn from. The post’s central engineering lesson is to treat numerical agreement between rollout and learner as a prerequisite for importance sampling, not as a cosmetic benchmark. It gives teams a way to reason about two different sources of divergence and a measurable trade-off: aligned kernels can cost speed against the fastest kernels, while avoiding rejection gates and maintaining stable learning may preserve more usable samples. The reported results are research-team claims, but the open recipes and code offer a concrete starting point for reproducing the method in a controlled training pipeline.

What happens next

  • Long asynchronous training runs on DeepSeek-V4.1-Flash and Kimi K3.1 will be reported. (6 months, 70%). Watch for: published multi-thousand-update runs with learning metrics and kernel-alignment checks.

Limits and uncertainty

The results are reported by the method’s authors and are not independently verified here. The blog says long asynchronous runs on DeepSeek-V4.1-Flash and Kimi K3.1 are still ahead; the reported flagship results establish zero mismatch in specified runs, not long-run training quality. The cost of alignment also varies: the post reports a 6.8% slower mean step than standard-kernel importance sampling with a sequence gate on one OSWorld setup, while optimization remains ongoing for Kimi K3.1. The broader cost and reproducibility across workloads, hardware configurations and implementations remain uncertain.

Skill to build now

If you operate an asynchronous RL trainer, split rollout-to-learner divergence into weight lag and same-weight kernel mismatch before tuning importance sampling. Start with a small reproducibility run that compares per-token log probabilities for identical weights and tokens, then test unclipped importance sampling only after that comparison is stable. The post’s open repository, Inferact/zero-mismatch-rl, includes model-specific recipes and fixes described by the authors.

Reader takeaway

Asynchronous RL can learn from older policy samples, but importance sampling alone is not a cure for numerical disagreement between rollout and learner. The authors’ practical claim is that aligning the kernels first makes the remaining weight lag manageable, with higher throughput in their experiments. Treat the flagship-model checks as validation steps, not proof of long-run training performance.

Evidence / sources

  1. 1github.com/vllm-project/vllm-project.github.io/blob/main/_posts/2026-11-06-zero-mismatch-async-rl.md
All Field Notes

Next reads

Read next

Open the graph
  1. 01topicInferenceExplore the inference topic hub.
  2. 02topicAi InfrastructureExplore the ai-infrastructure topic hub.
  3. 03archiveField NotesOpen the latest editorial analysis.
  4. 04sourcesSourcesSee the original sources this archive cites.
Who read this page?Machine requests, hidden until opened

Loading the privacy-safe route aggregate...

Open the JSON contract