Mechanism: Masked importance sampling reweights token contributions and masks extreme ratios, while score centering removes expected gradient drift under the sampling policy; together they address the sampler-trainer policy mismatch. Modal's experiment tested this with asynchronous RL: samplers ran across six accelerator types and precisions, while the trainer updated the policy in BF16 on B200 GPUs. The source reports that only score centering plus masked importance sampling, with or without top-p, trained stably to the end; uncorrected runs collapsed.
Why now: Modal says sampling consumes 80–85% of GPUs in large RL runs, making serving-side efficiency consequential to total training economics. That imbalance gives teams a reason to consider sampler hardware separately from trainer hardware, while the experiment indicates that the resulting policy mismatch needs explicit treatment. First implication: RL infrastructure teams can evaluate sampler hardware and precision independently, while treating off-policy correction as part of the training design.
Evidence boundary: This is a vendor-reported experiment on one model, coding-task setup, and benchmark protocol; it does not establish broad generalization or independent replication. Modal reports pass@4 reaching roughly 65–75% on HARD-51 and about 94% on the full SWE-bench Pro V2 set. First watch condition: Whether independent teams reproduce stable convergence and cost savings on other models and workloads using mixed-accelerator samplers.
The reported collapse diagnostics—falling reward, sampler-trainer KL increasing at least threefold, and repeated format errors—offer signals to monitor in this experiment. They are observations from Modal's runs, not universal failure criteria.