Orbax can checkpoint as fast as storage finishes; MaxText ships that as an opt-in, not a measured goodput default

What we checked

• What exactly changed in Orbax/MaxText continuous checkpointing, and are there quantified reliability or goodput gains? partly confirmed. Orbax’s ContinuousCheckpointingPolicy saves as often as possible unless a save is already in progress, and MaxText maps enable_continuous_checkpointing onto that policy, overriding checkpoint_period. Google’s 31 March 2026 blog reports smaller P50 intervals versus saving every 100 steps on Llama 3.1 70B CPT across two v5p-128 slices, plus an MTBF-based resource-preservation chart, but the HTML does not expose extractable numeric goodput or reliability percentages. [5][1][3][2]
• Is there a source other than the failing Google Developers page, such as repo docs, release notes, or code changes? confirmed. The Developers URL is live, and the same mechanism is documented in Orbax 0.11.1 release notes (28 January 2025), Orbax v1 save-decision policy docs, MaxText base.yml, and checkpoint_context.py. A later MaxText elastic-training post also names the flag as an alternative to a short checkpoint_period. [5][2][1][4][3][8]
• Have independent third parties reproduced the March 2026 continuous-checkpointing goodput or reliability numbers? refuted. No independent paper, filing, or non-Google evaluation of this specific feature was found. The 2026 Orbax paper shares lead authors with the blog and benchmarks isolated save/load versus PyTorch DCP, not end-to-end continuous-checkpointing goodput. Related academic work such as CheckFreq is cited as prior art for dynamic intervals, not as a replication of these results. [11][12][5]
• Is continuous checkpointing MaxText’s default save policy, and is it documented in MaxText checkpointing guides? refuted. base.yml still defaults enable_continuous_checkpointing to false, with async checkpointing on and checkpoint_period 10,000. MaxText’s GCS checkpointing guide and core checkpoints reference document checkpoint_period, async saving, emergency, and autocheckpoint, but they do not document the continuous flag. An Orbax v1 RFC said the policy might become the default for internal users; that is not the MaxText public default. [4][9][10][14]

What we found

• Continuous checkpointing is a save-decision policy, not a new on-disk format. Orbax documents ContinuousCheckpointingPolicy as evaluating true as often as possible, with two constraints: it will not start a new save while DecisionContext.is_saving_in_progress is true, and it may enforce an optional minimum_interval_secs cooldown. The decision is computed on the primary host and broadcast. Orbax 0.11.1’s changelog, dated 28 January 2025, explicitly added a policy that checkpoints as often as possible as long as a save is not already in progress. [1][2]
• MaxText exposes the policy as enable_continuous_checkpointing, default false. When that flag is true, build_save_decision_policy returns ContinuousCheckpointingPolicy and does not also attach PreemptionCheckpointingPolicy or FixedIntervalPolicy. The blog states that continuous checkpointing overrides checkpoint_period. Enabling the flag also sets the Orbax async timeout to 60 minutes. Async checkpointing itself remains the recommended path: Orbax’s async guide reports blocking save-time reductions of about 40 percent on a 300M model, 85 percent on 8B, and 97 percent on 340B; those figures are for async I/O hiding, not for the continuous policy. [3][4][5][6]
• The quantified reliability claims in the March 2026 blog are directional, not numeric in the published HTML. The authors compared Llama 3.1 70B continuous pre-training on two slices of a v5p-128 cluster, 64 chips per slice: continuous checkpointing versus checkpointing every 100 steps. They report markedly smaller P50 checkpoint intervals, a higher average step time from more frequent device-to-host copies, and an MTBF-based resource-preservation chart. They also say a multi-slice benchmark showed no significant slowdown, because the heavy I/O stays asynchronous between storage and typically slice 0. The chart values themselves are images; this investigation did not recover numeric goodput percentages from the page. [5]
• Google Cloud’s 10 April 2024 goodput post still frames runtime goodput around time since last checkpoint, auto-checkpointing on SIGTERM, container preload, and a persistent compilation cache. Orbax’s current PreemptionCheckpointingPolicy docs note that if continuous checkpointing is frequent enough, replaying steps can be cheaper than blocking for a preemption save. Those are complementary knobs. MaxText’s July 2026 elastic-training post separately tells operators they can set enable_continuous_checkpointing true so Orbax starts the next save as soon as the previous one finishes, making a fixed period less important for in-process recovery. [7][1][8]
• Public MaxText guides lag the flag. The GCS checkpointing guide and the core checkpoints reference still describe enable_checkpointing, async_checkpointing, and checkpoint_period, plus emergency and autocheckpoint paths, without listing enable_continuous_checkpointing. The blog’s recommended MaxText snippet also sets max_num_checkpoints_to_keep to 10 to cap storage; the default in base.yml is None. Builders who only read MaxText user docs would not discover the feature. [9][10][4][5]
• The 2026 Orbax paper, led by the same Gaffney and Li, is useful context but not a confirmation of the blog’s goodput charts. It reports isolated save and load times versus PyTorch DCP on Llama 3.1 8B, 70B, and 405B, including background-save speedups up to about 3.4 times on 405B, plus OCDBT, subchunking, replica-parallel save, and load-and-broadcast results. Those measurements deliberately avoid a full training loop. The paper cites CheckFreq and related work on dynamic intervals as motivation, not as an independent replication of continuous checkpointing on MaxText. [11][12][13]

What it means

Treat continuous checkpointing as a real, opt-in Orbax policy that MaxText already wires, not as an unverified news item. If you train on JAX with Orbax, enable it only together with async saving, a retention cap, and co-located storage. Expect more frequent device-to-host copies and higher storage churn in exchange for less lost work when jobs die. Do not treat the March 2026 charts as a portable goodput SLA: the published HTML gives no extractable percentages, the cluster was two v5p-128 slices, and the resource-saving argument is an MTBF model, not a multi-week production A/B.

Also inspect how your framework composes policies. MaxText’s builder returns only ContinuousCheckpointingPolicy when the flag is on, so you do not automatically also save on preemption unless you add that yourself in Orbax. Keep measuring runtime goodput (useful steps over available-resource time) rather than step time alone. Pair the policy with emergency or multi-tier local checkpoints if restore latency, not lost steps, is the bottleneck. Until MaxText’s user guides catch up, configure from base.yml and checkpoint_context.py, not from the GCS checkpointing doc.

Still open

• What numeric P50 interval, step-time delta, and resource-preservation values sit in the unpublished blog charts?
• How does continuous checkpointing behave at multi-pod scale, on GPUs, and on cross-region buckets, beyond the two-slice v5p-128 write-up?
• Should MaxText OR ContinuousCheckpointingPolicy with PreemptionCheckpointingPolicy, or is exclusive continuous-only the intended contract?
• When, if ever, will the public MaxText default match the Orbax v1 RFC’s suggested ContinuousCheckpointingPolicy default?