AI Inference Is a Data Movement Problem: Microsoft's Saurabh Dighe on Maia 300

Microsoft's Maia 300 targets 7x token throughput and 2.5x perf/watt over Maia 200 by treating inference as a data-movement problem, scaling to multi-gigawatt deployments by 2027.

• 7x throughput, 2.5x efficiency: Maia 300 delivers 7x more tokens per rack and 2.5x better performance-per-watt versus Maia 200, scaling to multi-gigawatt deployments starting 2027.
• HBM4 jump: Eight HBM4 stacks (vs six HBM3E) via chiplet design on TSMC N2P yield 18 TB/s bandwidth per chip (2.5x Maia 200) and 5x at rack scale, with flops up only ~35-37% and power up 750W to 950W.

Themes, sentiment and reasons are AI-written paraphrases, checked so they never quote an episode. They may contain errors. Not investment advice. Episodes remain the property of their creators.