Details
- Google for Developers highlights a new systems-engineering playbook for optimizing the Qwen 3.5-397B Mixture-of-Experts model on Google Cloud TPUs (Ironwood v7x).
- The work delivers approximately 3.1× higher decode performance and 4.7× higher prefill performance compared with the April 2026 implementation at high concurrency.
- Engineers use a hybrid Attention Data Parallelism plus MoE Expert Parallelism sharding scheme to better map Qwen’s 512 experts and limited KV heads onto the TPU v7x device mesh.
- Routing metadata collectives are consolidated and optimized, including packing routing indices and weights to cut communication latency and halve collective launch overhead.
- Custom JAX/Pallas kernel fusions, such as Ragged Page Attention and co-designed SparseCore/TensorCore GEMMs, push prefill throughput to 3,707 tokens per second per chip, nearing Ironwood’s roofline efficiency.
- The optimizations are structured as a reusable, model-agnostic playbook intended for broader frontier MoE deployment on Google Cloud TPUs.
- Google presents these results as a reference for scaling large open-weights models like Alibaba’s Qwen on its latest TPU generation, emphasizing systematic engineering over ad hoc tuning.
Impact
By publishing a detailed optimization strategy for Qwen 3.5-397B on Ironwood TPUs, Google positions its cloud hardware as a performant target for large open-weights MoE models and signals a push toward standardized, reusable inference playbooks. This narrows the gap with GPU-centric serving stacks and encourages frontier-model users to consider TPUs for high-throughput workloads.
