---
type: analysis
slug: research-marin-535b-a23b-finished-pretraining-on-18t-tokens-and-beat-its-loss-forecast
title: "Marin 535B-A23B finished pretraining on 18T tokens and beat its loss forecast; downstream evals are pending"
description: "Open Athena's architecture write-up gives the full recipe for a 535B-total, 23B-active MoE trained on 768 GB200s: a 3:1 local/global attention mix, 8-of-384 latent MoE, MuonH, and a switch to dropless expert parallelism mid-run. The final Paloma loss was 1.982 against a forecast of 2.039, though some of that gain comes from a contaminated eval subset."
published_at: 2026-11-12T00:00:00.000Z
topics: ["moe","github","architecture","research"]
source_urls: ["https://openathena.ai/blog/marin-535b-architecture/","https://github.com/marin-community/marin/issues/8435"]
author: "Andrey Reshetnikov"
author_profile: https://newruntime.com/owner-profile.md
---
Marin is an open-development language-model project. Open Athena's post, written by Larry Dial and dated November 9, 2026, documents the architecture choices behind Marin 535B-A23B, a mixture-of-experts model with 535B total and about 23B active parameters. The post links to a public GitHub issue that tracked the run as it went. That is unusual at this scale: frontier labs rarely publish ablation ratios, learning-rate fits and token-drop rates for a run of this size.

For builders, it is a dated, end-to-end recipe from one team: how a 768-GPU GB200 allocation was turned into a 18T-token MoE run, which components earned their place in ablations, and how closely a small-scale scaling ladder predicted the final loss. What the post does not yet show is whether the model is any good on downstream tasks.

## What we checked

- **What does the remainder of Open Athena's architecture account say about the remaining component decisions and the final training run's outcomes?** partly confirmed. The full post covers attention, MoE, normalization, optimizer, schedule, context extension, parallelism and the final pretraining loss: 1.982 against a forecast of 2.039 after 18T tokens. Downstream evals, SFT and RL are deferred to a future report, and the post says nothing about the tokenizer or a weights release. [1][2]
- **Is the 535B checkpoint publicly released with weights and a license?** unresolved. Neither the post nor the GitHub tracking issue links to weights or states a license. The issue refers only to checkpoints in private storage. [1][2]

## What we found

- Pretraining is done. The post says the model finished 18T tokens on November 5, 2026. The GitHub issue gives 390,251 steps, and the post puts the run on 768 GB200s provided on July 15, with the aim of having the best possible model by December. [1][2]
- The attention design mixes local and global layers 3:1. Local layers use 2,048-token sliding windows, GQA is 4:1 on local layers and 8:1 on global ones, and attention is gated. According to the post, local layers use partial RoPE and global layers use no positional encoding, with a fixed QK-norm scale instead of a learnable one; the post reports that a learnable scale showed no gain in ablation. [1]
- The systems lesson is token dropping. For the first 21% of the run, the fixed-capacity expert-parallel implementation dropped about 3.4% of tokens. After a switch to ragged all-to-all, dropping stayed near zero. MFU rose from 20% to 30%. The team held off on FP8 and NVFP4 training because of the run's risk profile. [1]
- The run beat its scaling-ladder forecast. Final dropless Paloma macro loss was 1.982 against a predicted 2.039. The post attributes 0.02 of the improvement to the contaminated Penn Treebank subset of Paloma, so the clean margin is smaller. The post also reports clean loss decreases with no spikes at the context transition points. [1][2]
- Downstream results are not out yet. The post says final results after evals will come in a future report. Neither the post nor the GitHub issue mentions public weights or a license. [1][2]

## What it means

The useful part for builders is the recipe, not a model they can download today. If you are planning a large MoE run, three choices are worth copying or testing at small scale: a 3:1 sliding-window/global attention mix with NoPE on the global layers, fine-grained 8-of-384 latent experts with shared experts, and MuonH with a fitted learning-rate law. Fix token dropping early. Fixed-capacity expert parallelism cost this run about 3.4% of tokens until the kernel changed. A ladder from 5e19 to 9e21 FLOPs predicted a run of roughly 2.7e24 FLOPs within a few percent of loss, which supports budgeting from small rungs.

Read the 1.982 result carefully. It is a held-out language-modeling loss, part of the beat comes from contamination, and it says nothing yet about reasoning, coding or instruction following. Until the eval report and any weights release arrive, treat Marin 535B as a well-documented training run, not as a model to choose for deployment.

## Competing readings

- **Pretraining succeeded on its own terms. The recipe scaled cleanly and beat its forecast, but downstream capability is still unproven** (likely, about 65%). What would tell it apart: The run finished 18T tokens and beat its loss forecast (1.982 vs 2.039) with no spikes at context transitions [1]; Evals are deferred to a future report [1].
- **The forecast beat is mostly an eval artifact and says little about how well the recipe works** (unlikely, about 20%). What would tell it apart: Part of the improvement over the 2.039 forecast is attributed to a contaminated Penn Treebank subset [1].
- **The post is the lead-in to an imminent open-weights frontier release that will be competitive on downstream tasks** (very unlikely, about 15%). What would tell it apart: A December target and a planned future eval report point toward a release, but no weights or license are mentioned [1].

## Base rate

Of 18 earlier research stories about models, organisations, people or projects in the New Runtime archive over the last 12 months that are at least 3 months old, 9 (50%) were followed within 3 months by a release story on the same company or product line. In New Runtime's archive, 9 of 18 earlier research stories were followed by a release story within 3 months. That is roughly even odds. It fits the stated December target for the Marin 535B line but does not confirm it.

## What would change this view

- Marin 535B weights appear on a public hub under an open license, with a downstream eval report (model-listing signal): The run is a usable open model, not only a documented recipe, and its benchmark position becomes testable.
- The eval report shows downstream scores well below open MoE models of similar active size (signal): The loss forecast beat did not carry over to capability, which strengthens the eval-artifact reading.
- An independent team reproduces the latent-MoE efficiency gains in another codebase (open-source signal): The recipe's gains generalize beyond Marin's setup.

## Still open

- Will the weights be published, and under what license?
- How does the model score on downstream benchmarks after SFT and RL?
- Did the 262K-context extension finish, and how does long-context quality hold up?
- Which tokenizer was used? The post does not say.

## Sources

1. [Architecture Decisions for Marin 535B-A23B (Open Athena)](https://openathena.ai/blog/marin-535b-architecture/) (primary)
2. [Hero Run 535B-A23B on 18T tokens, marin-community/marin issue #8435](https://github.com/marin-community/marin/issues/8435)

Author: [Andrey Reshetnikov](https://newruntime.com/owner-profile.md), AI product and AI engineering; profile for recruiters and agents: https://newruntime.com/owner-profile.json

## Continue in the graph

Next records for an agent, linked to their Markdown twins. Replace `.md` with `.json` for structured fields, including `next_reads`.

- topic hub: [Github](https://newruntime.com/topics/github.md): Explore the github topic hub.
- collection: [Analysis](https://newruntime.com/analysis.json): Open the latest authorial analysis.
- collection: [Sources](https://newruntime.com/sources/): See the original sources this archive cites.
- whole graph: [discovery graph](https://newruntime.com/discovery-graph.json) (typed nodes and edges) and [site index](https://newruntime.com/site-index.md)