{"type":"analysis","id":"analysis:research-marin-535b-a23b-finished-pretraining-on-18t-tokens-and-beat-its-loss-forecast","slug":"research-marin-535b-a23b-finished-pretraining-on-18t-tokens-and-beat-its-loss-forecast","title":"Marin 535B-A23B finished pretraining on 18T tokens and beat its loss forecast; downstream evals are pending","description":"Open Athena's architecture write-up gives the full recipe for a 535B-total, 23B-active MoE trained on 768 GB200s: a 3:1 local/global attention mix, 8-of-384 latent MoE, MuonH, and a switch to dropless expert parallelism mid-run. The final Paloma loss was 1.982 against a forecast of 2.039, though some of that gain comes from a contaminated eval subset.","published_at":"2026-11-12T00:00:00.000Z","updated_at":"2026-11-12T00:00:00.000Z","record_date":"2026-11-12","topics":["moe","github","architecture","research"],"source_urls":["https://openathena.ai/blog/marin-535b-architecture/","https://github.com/marin-community/marin/issues/8435"],"language":"en","routes":{"html":"https://newruntime.com/analysis/research-marin-535b-a23b-finished-pretraining-on-18t-tokens-and-beat-its-loss-forecast/","markdown":"https://newruntime.com/analysis/research-marin-535b-a23b-finished-pretraining-on-18t-tokens-and-beat-its-loss-forecast.md","json":"https://newruntime.com/analysis/research-marin-535b-a23b-finished-pretraining-on-18t-tokens-and-beat-its-loss-forecast.json"},"author":{"role":"author","name":"Andrey Reshetnikov","profile":"https://newruntime.com/owner-profile.md","profile_json":"https://newruntime.com/owner-profile.json","same_as":["https://www.linkedin.com/in/reshetnikov1/","https://t.me/lapetuse"]},"editorial_provenance":{"desk":"authorial","target_ref":"deep_research:671fc71d-fec1-4998-beef-74530eb33941","authorial_article_id":"671fc71d-fec1-4998-beef-74530eb33941"},"markdown_url":"https://newruntime.com/analysis/research-marin-535b-a23b-finished-pretraining-on-18t-tokens-and-beat-its-loss-forecast.md","retrieval_nugget":"Open Athena's architecture write-up gives the full recipe for a 535B-total, 23B-active MoE trained on 768 GB200s: a 3:1 local/global attention mix, 8-of-384 latent MoE, MuonH, and a switch to dropless expert parallelism mid-run.","next_reads":[{"type":"topic","path":"/topics/github/","url":"https://newruntime.com/topics/github/","title":"Github","reason":"Explore the github topic hub.","media_type":"text/html","markdown_url":"https://newruntime.com/topics/github.md","json_url":"https://newruntime.com/topics/github.json"},{"type":"collection","path":"/analysis/","url":"https://newruntime.com/analysis/","title":"Analysis","reason":"Open the latest authorial analysis.","media_type":"text/html","json_url":"https://newruntime.com/analysis.json"},{"type":"collection","path":"/sources/","url":"https://newruntime.com/sources/","title":"Sources","reason":"See the original sources this archive cites.","media_type":"text/html"}]}