---
type: "post"
id: "toward-rl-on-mixed-precision-mixed-accelerator-samplers-e45555"
slug: "toward-rl-on-mixed-precision-mixed-accelerator-samplers-e45555"
title: "Toward RL on mixed-precision, mixed-accelerator samplers"
description: "**Thesis:** Heterogeneous RL samplers can be useful when training explicitly corrects for the policy mismatch they introduce."
retrieval_nugget: "**Thesis:** Heterogeneous RL samplers can be useful when training explicitly corrects for the policy mismatch they introduce."
published_at: "2026-10-13"
updated_at: "2026-10-13"
record_date: "2026-10-13"
date_kind: "published_at"
topics: []
source_urls: ["https://modal.com/blog/toward-rl-on-mixed-precision-mixed-accelerator-samplers"]
schema_version: "newruntime-agent-readable-v0.2"
stable_id: "post:toward-rl-on-mixed-precision-mixed-accelerator-samplers-e45555"
status: "published"
visuals: []
editorial_provenance: {"schema_version":"newruntime-editorial-copy-v1","content_status":"source_grounded_final","final_copy_sha256":"sha256:7da481c900c062f878713bbd5f8aedd1a8bb74ad61d84f5f10d9ef2137c12693","reviewed_at":"2026-10-13T20:07:20.881Z","source_evidence_count":1,"verified_claim_count":3,"site_analysis_schema_version":"newruntime-site-analysis-v1","site_object_kind":"field_note","observed_fact_count":3,"implication_count":1,"watch_condition_count":1,"related_record_count":0}
analysis: {"schema_version":"newruntime-site-analysis-v1","object_kind":"field_note","thesis":"Heterogeneous RL samplers can be useful when training explicitly corrects for the policy mismatch they introduce.","observed_facts":[{"text":"Modal trained Qwen3.6-35B-A3B on agentic coding tasks with fully asynchronous RL, using samplers across six accelerator types and a BF16 B200 trainer.","source_urls":["https://modal.com/blog/toward-rl-on-mixed-precision-mixed-accelerator-samplers"]},{"text":"The source reports that only SC + MIS, with or without top-p, trained stably to the end; uncorrected runs collapsed.","source_urls":["https://modal.com/blog/toward-rl-on-mixed-precision-mixed-accelerator-samplers"]},{"text":"Reported pass@4 reached roughly 65–75% on HARD-51 and about 94% on the full SWE-bench Pro V2 set.","source_urls":["https://modal.com/blog/toward-rl-on-mixed-precision-mixed-accelerator-samplers"]}],"mechanism":"Masked importance sampling reweights token contributions and masks extreme ratios, while score centering removes expected gradient drift under the sampling policy; together they address the sampler-trainer policy mismatch.","why_now":"Modal says sampling consumes 80–85% of GPUs in large RL runs, making serving-side efficiency consequential to total training economics.","implications":["RL infrastructure teams can evaluate sampler hardware and precision independently, while treating off-policy correction as part of the training design."],"evidence_boundary":"This is a vendor-reported experiment on one model, coding-task setup, and benchmark protocol; it does not establish broad generalization or independent replication.","watch_conditions":["Whether independent teams reproduce stable convergence and cost savings on other models and workloads using mixed-accelerator samplers."],"related_records":[],"new_branch_reason":"No prior New Runtime record was supplied; this creates a distinct record about off-policy correction enabling mixed-accelerator samplers in asynchronous RL."}
editor: {"role":"editor","name":"Andrey Reshetnikov","profile":"https://newruntime.com/owner-profile.md","profile_json":"https://newruntime.com/owner-profile.json","same_as":["https://www.linkedin.com/in/reshetnikov1/","https://t.me/lapetuse"]}
routes: {"html":"https://newruntime.com/posts/toward-rl-on-mixed-precision-mixed-accelerator-samplers-e45555/","markdown":"https://newruntime.com/posts/toward-rl-on-mixed-precision-mixed-accelerator-samplers-e45555.md","json":"https://newruntime.com/posts/toward-rl-on-mixed-precision-mixed-accelerator-samplers-e45555.json"}
---

# Toward RL on mixed-precision, mixed-accelerator samplers

## Retrieval answer

**Thesis:** Heterogeneous RL samplers can be useful when training explicitly corrects for the policy mismatch they introduce.

**Thesis:** Heterogeneous RL samplers can be useful when training explicitly corrects for the policy mismatch they introduce.

**Mechanism:** Masked importance sampling reweights token contributions and masks extreme ratios, while score centering removes expected gradient drift under the sampling policy; together they address the sampler-trainer policy mismatch. Modal's experiment tested this with asynchronous RL: samplers ran across six accelerator types and precisions, while the trainer updated the policy in BF16 on B200 GPUs. The source reports that only score centering plus masked importance sampling, with or without top-p, trained stably to the end; uncorrected runs collapsed.

**Why now:** Modal says sampling consumes 80–85% of GPUs in large RL runs, making serving-side efficiency consequential to total training economics. That imbalance gives teams a reason to consider sampler hardware separately from trainer hardware, while the experiment indicates that the resulting policy mismatch needs explicit treatment. **First implication:** RL infrastructure teams can evaluate sampler hardware and precision independently, while treating off-policy correction as part of the training design.

**Evidence boundary:** This is a vendor-reported experiment on one model, coding-task setup, and benchmark protocol; it does not establish broad generalization or independent replication. Modal reports pass@4 reaching roughly 65–75% on HARD-51 and about 94% on the full SWE-bench Pro V2 set. **First watch condition:** Whether independent teams reproduce stable convergence and cost savings on other models and workloads using mixed-accelerator samplers.

The reported collapse diagnostics—falling reward, sampler-trainer KL increasing at least threefold, and repeated format errors—offer signals to monitor in this experiment. They are observations from Modal's runs, not universal failure criteria.

Editor: [Andrey Reshetnikov](https://newruntime.com/owner-profile.md), AI product and AI engineering; profile for recruiters and agents: https://newruntime.com/owner-profile.json

## Continue in the graph

Next records for an agent, linked to their Markdown twins. Replace `.md` with `.json` for structured fields, including `next_reads`.

- collection: [Field Notes](https://newruntime.com/posts.json): Open the latest editorial analysis.
- collection: [Source Ledger](https://newruntime.com/source-ledger.json): Inspect the public source evidence graph.
- whole graph: [discovery graph](https://newruntime.com/discovery-graph.json) (typed nodes and edges) and [site index](https://newruntime.com/site-index.md)