In a preliminary test on eight NVIDIA B300 GPUs, a 47-billion-parameter MoE processed 52,000 tokens per second per GPU, versus 19,400 with Ai2’s earlier implementation, or about 2.7 times the throughput. A separate benchmark increased the expert pool from 8 to 128 while selecting four experts per token: total capacity rose from 4.6 billion to 47 billion parameters while throughput fell by less than 5%. Ai2 also reports a 1.2-trillion-parameter configuration benchmarked across 512 GPUs. Those tests used random routing to measure system performance, not the quality of a trained model; a 2.38-trillion-parameter configuration was a short-capacity test, not a full training run. The stack is intended for Ai2’s next Olmo generation and is being released for outside use.
Raw SignalOct 1, 20261 min read1 source
Introducing Olmo-core 3: Open, scalable training infrastructure for large MoEs
In one minute
Ai2 has released Olmo-core 3, an open training framework redesigned for large mixture-of-experts (MoE) models.