Details
- Ai2 released Olmo-core 3 on October 1, 2026, an open training framework for scaling mixture-of-experts language models into the trillion-parameter range.
- The system supports the next generation of Olmo and is designed for researchers and developers to adapt, inspect, and use across different hardware configurations.
- Its redesigned stack replaces the earlier FSDP-based approach with distributed data parallelism, keeping experts resident on GPUs while routing relevant data to them. Expert, pipeline, and distributed-optimizer parallelism further divide model computation and training state across clusters.
- In an eight-NVIDIA-B300 benchmark, a 47-billion-parameter MoE processed 52,000 tokens per second per GPU, compared with 19,400 previously—about 2.7× higher throughput. Increasing the expert pool from 8 to 128 raised total capacity from 4.6B to 47B parameters while reducing throughput by less than 5% at roughly 3.2B active parameters per token.
- Ai2 benchmarked a 1.2-trillion-parameter configuration across 512 B300 GPUs, with 58.36B active parameters per token and peak throughput of 858 TFLOP/s per GPU. The test used random routing to measure systems performance, not model quality; a separate capacity test reached 2.38T parameters using DeepEP v2.
Impact
Olmo-core 3 lowers infrastructure barriers for open MoE research by combining distributed scaling with routing, grouped computation, and MXFP8 optimizations; MXFP8 improved measured throughput by about 21% while reducing active memory from 103 GiB to 95 GiB. Over the next 12–24 months, its open implementation could shift more academic and independent-lab work toward sparse, large-capacity models, although sustained training quality and hardware costs remain separate from the reported systems benchmarks.
