---
type: "post"
id: "trained-with-consensus-filtering-on-monte-carlo-step-127897"
slug: "trained-with-consensus-filtering-on-monte-carlo-step-127897"
title: "Trained with consensus filtering on Monte Carlo step labels, detailed in arXiv:2501.07301."
description: "Alibaba’s Qwen team studies how to train process reward models (PRMs) that judge intermediate steps in mathematical reasoning."
retrieval_nugget: "Alibaba’s Qwen team studies how to train process reward models (PRMs) that judge intermediate steps in mathematical reasoning."
published_at: "2026-11-16"
updated_at: "2026-11-16"
record_date: "2026-11-16"
date_kind: "published_at"
topics: ["models"]
entities: ["Alibaba Group","Qwen Team","Qwen2.5-Math-PRM-7B","Qwen2.5-Math-PRM-72B","ProcessBench","PRM800K"]
related_patterns: ["scientific-agents-need-artifact-evidence"]
novelty_label: "new_fact"
what_changed: "The paper reports a consensus-filtering method for training mathematical-reasoning process reward models, combining Monte Carlo estimates with LLM judging to improve performance and data efficiency."
prior_refs: ["https://newruntime.com/posts/trained-via-supervised-cold-start-plus-adagrpo-reinforcement-5f960f/"]
source_url: "https://arxiv.org/html/2501.07301v1"
source_urls: ["https://arxiv.org/html/2501.07301v1"]
schema_version: "newruntime-agent-readable-v0.2"
stable_id: "post:trained-with-consensus-filtering-on-monte-carlo-step-127897"
status: "published"
visuals: []
editor: {"role":"editor","name":"Andrey Reshetnikov","profile":"https://newruntime.com/owner-profile.md","profile_json":"https://newruntime.com/owner-profile.json","same_as":["https://www.linkedin.com/in/reshetnikov1/","https://t.me/lapetuse"]}
routes: {"html":"https://newruntime.com/posts/trained-with-consensus-filtering-on-monte-carlo-step-127897/","markdown":"https://newruntime.com/posts/trained-with-consensus-filtering-on-monte-carlo-step-127897.md","json":"https://newruntime.com/posts/trained-with-consensus-filtering-on-monte-carlo-step-127897.json"}
markdown_url: "https://newruntime.com/posts/trained-with-consensus-filtering-on-monte-carlo-step-127897.md"
---

# Trained with consensus filtering on Monte Carlo step labels, detailed in arXiv:2501.07301.

## Retrieval answer

Alibaba’s Qwen team studies how to train process reward models (PRMs) that judge intermediate steps in mathematical reasoning.

## Summary
Alibaba’s Qwen team studies how to train process reward models (PRMs) that judge intermediate steps in mathematical reasoning. The paper proposes retaining training examples only when Monte Carlo estimates and an LLM judge agree, and argues that answer-selection scores alone can misrepresent a PRM’s ability to find flawed reasoning.

For builders of verifiers and reasoning systems, the central lesson is that a score for reaching the right answer can conceal failure to recognize the first wrong step.

## What happened
In “The Lessons of Developing Process Reward Models in Mathematical Reasoning,” the Qwen Team describes experiments on data construction and evaluation for PRMs. These models assign scores to steps in a solution, with the aim of identifying where reasoning goes wrong. The authors contrast this with Monte Carlo (MC) estimation, which estimates the chance that a continuation from a given step will eventually produce the correct final answer. They argue that this measures future potential rather than whether the current step is correct: an incorrect step can lead to a correct answer, and a correct step can be followed by a bad continuation.

The team’s preliminary experiments used roughly 500,000 queries with gold answers. For each query, it generated six to eight responses using Qwen2-Math-Instruct and Qwen2.5-Math-Instruct models at 7B and 72B sizes. Responses were split into steps using a double-newline delimiter. Eight continuations from each step were then used to estimate labels. For hard labels, a step was deemed correct if at least one continuation reached the correct final answer; soft labels used the proportion of successful continuations. The PRMs were initialized from Qwen2.5-Math-7B or 72B-Instruct, with the language-model head replaced by a two-layer scalar-value head.

The paper reports that PRMs trained on these MC-derived labels lagged a model trained on human-annotated PRM800K in locating erroneous steps on ProcessBench. ProcessBench asks a model to identify the first erroneous step in a solution or determine that every step is correct. In the reported Best-of-8 setup, the MC-trained models also did not beat majority voting across the listed mathematical benchmarks. Candidate answers were scored by multiplying the model’s step scores across each response, then choosing the highest-scored candidate.

The proposed consensus filter combines MC estimates with an LLM-as-a-judge. Training examples are retained only when both methods agree on the locations of error steps. The paper compares approaches using MC-derived labels, Qwen2.5-72B-Instruct as a judge on the same query-response data, and the human-annotated PRM800K dataset. The authors report that human annotations performed best on ProcessBench with the least data, followed by LLM judging, while MC estimation performed worst despite having the largest dataset in that comparison. Their results also show a different picture on Best-of-8, where their MC-trained model achieved the best average accuracy among the compared data-construction methods.

That contrast is central to the paper’s warning about optimization against answer selection alone. A policy model can produce an answer with the correct final result while containing flawed reasoning, and a PRM can tolerate that flawed process. The authors say this can inflate Best-of-N performance while moving the model away from its intended role as a step verifier. They also report that, for existing PRMs, minimum step scores were often concentrated at final-answer steps, which they interpret as a drift toward outcome assessment.

The authors advocate combining response-level Best-of-N evaluation with step-level evaluation such as ProcessBench. They release Qwen2.5-Math-PRM-7B and Qwen2.5-Math-PRM-72B through Hugging Face. The paper is dated 13 January 2025 and reports experiments in mathematical reasoning; it does not establish that the same findings apply to other domains.

## What the archive adds
New Runtime’s earlier coverage on [mixed-precision, mixed-accelerator samplers](https://newruntime.com/posts/toward-rl-on-mixed-precision-mixed-accelerator-samplers-e45555/) noted that heterogeneous sampling can be useful when training explicitly corrects for the policy mismatch it introduces. This is a new research result on supervision and evaluation rather than a restatement: it focuses on noisy step labels and the distinction between predicting future success and verifying present reasoning.

## Trend signal
This supports [scientific agents need artifact evidence](https://newruntime.com/patterns/scientific-agents-need-artifact-evidence/), cautiously. The paper makes evaluation design part of the claimed result, pairing answer-level performance with step-wise error identification rather than relying on a single aggregate number. Its evidence remains benchmark-based and specific to mathematical reasoning, so it does not by itself establish a general evaluation standard for agents or production verification systems.

## Why it matters
For teams training models to check reasoning, a correct final answer is not enough to show that the checker understands the process. If a verifier rewards a plausible outcome while overlooking an invalid intermediate step, it can look successful in selection tests and still fail at the task it was built for. The Qwen paper makes that mismatch concrete by comparing Best-of-8 results with ProcessBench step-localization results.

The consensus filter offers a practical research direction: use outcome-based estimates as one signal, but retain examples only when an independent judge agrees about the error location. That changes the training target from “could this partial solution still end well?” toward “is this step justified now?” The distinction matters wherever a verifier will be used to select, critique, or supervise multi-step outputs.

Teams should also separate two operational questions. One is whether a scorer improves the final answer selected from several candidates. The other is whether it detects the first failure in a chain of work. The paper shows that the same training approach can look favorable on the former and weak on the latter. Evaluation should therefore match the failure mode the verifier is meant to prevent, rather than treating one answer-level score as proof of reliable process checking.

## Limits and uncertainty
The findings concern mathematical reasoning and the benchmarks and model configurations evaluated in the paper. They do not establish how well consensus filtering transfers to coding, science, tool use, or other domains, or whether the same trade-offs hold at different scales. The reported gains are experimental results from the authors, not an independent replication.

The paper’s comparison also shows that rankings vary with the evaluation: Best-of-8 and step-localization do not answer the same question. Agreement between an MC estimate and an LLM judge may reduce one kind of label noise, but agreement is not proof that either signal is correct. The practical cost, latency, and model dependence of applying the filtering process to larger training runs are not independently measured here.

## Skill to build now
If a team already trains or evaluates a step verifier, create a small held-out set where each example identifies the first invalid step, not only the final answer. Score the verifier separately on first-error localization and on candidate-answer selection. If Monte Carlo continuation labels are used, sample a subset for independent judging and measure where the two labels disagree before treating the continuation rate as a ground-truth step label.

## Reader takeaway
A reward model that helps select correct answers is not necessarily a reliable checker of reasoning steps. For process supervision, evaluate whether the model can locate errors, treat Monte Carlo labels as a noisy signal unless they agree with a more direct judgment of the step, and keep answer selection distinct from verification.

Editor: [Andrey Reshetnikov](https://newruntime.com/owner-profile.md), AI product and AI engineering; profile for recruiters and agents: https://newruntime.com/owner-profile.json

## Continue in the graph

Next records for an agent, linked to their Markdown twins. Replace `.md` with `.json` for structured fields, including `next_reads`.

- topic hub: [Models](https://newruntime.com/topics/models.md): Explore the models topic hub.
- collection: [Field Notes](https://newruntime.com/posts.json): Open the latest editorial analysis.
- collection: [Sources](https://newruntime.com/sources/): See the original sources this archive cites.
- whole graph: [discovery graph](https://newruntime.com/discovery-graph.json) (typed nodes and edges) and [site index](https://newruntime.com/site-index.md)
