Microsoft’s Maia 300 Bets on Inference-First Design to Power the Agentic AI Era

Key points

• Microsoft unveiled Maia 300, which begins deployment in 2027 and is designed for inference-first workloads such as long-context reasoning and multi-step agent tasks, with a claimed 40% more tokens per megawatt than other leading GPU-based systems ².
• Each accelerator pairs eight HBM4 stacks with 288 GB of capacity and 18 TB/s of bandwidth, which is 2.5 times the memory bandwidth and 33% more capacity than Maia 200 ¹.
• A 96-accelerator rack can scale into a superpod of 12,288 accelerators over a single Ethernet-based scale-up fabric, and Microsoft claims up to seven times the rack-scale token throughput of a Maia 200 rack ^{1 2}.
• Compute chiplets are built on TSMC N2P, with enhanced MXFP4 and sub-3-bit storage formats cutting the bytes moved per token ¹.

An Inference-First Philosophy

Microsoft frames Maia 300 as the fullest expression of a principle that has guided the Azure Maia line from the start: optimize the whole system for LLM inference so each watt and each dollar yields more useful output. Silicon, software, rack infrastructure, networking and datacenters are co-designed as one platform ¹.

The motivation is a shift in workload shape. Agentic tasks involve multi-step execution, long-context decoding and tool use, which change the arithmetic intensity of inference and put a premium on locality, memory capacity and bandwidth, and fitting more capability inside a fixed power and cooling envelope ¹. Microsoft says it sees these demands directly through Microsoft Copilot, GitHub Copilot and Microsoft Foundry ¹.

The headline claim is efficiency: roughly 40% more tokens per megawatt than other leading GPU-based systems, aimed at long-context reasoning, responsive conversational AI and sustained agent work ².

Inside the Silicon

Maia 300 builds on the Software Defined Locally Accessed Dataflow Architecture, or SDLA, introduced with Maia 200. It separates control-path programming from data-path execution so software can explicitly manage data placement, movement, conversion and synchronization ¹.

The new chip uses a multi-chiplet design. Compute sits on TSMC N2P, where the process gives the most performance-per-watt benefit, while memory-controller and I/O functions go on dies optimized for bandwidth, connectivity and cost. Microsoft says this raises compute FLOPs by 36%, HBM bandwidth by 2.5 times and scale-up bandwidth by 14% ¹.

Memory shifts from a uniform to a locality-aware NUMA organization across chiplets, so data sits near the compute that uses it and travels shorter distances. Eight HBM4 stacks deliver 288 GB and 18 TB/s per accelerator, roughly double the HBM bandwidth per watt of Maia 200, with software-managed on-die SRAM acting as a scratchpad to ease pressure on HBM ^{1 2}. Enhanced MXFP4 and sub-3-bit storage formats shrink footprint and data movement, allowing larger models, longer contexts and more concurrent sessions in the same power envelope ¹. TechRadar reports a 950W thermal design power for each chip ².

One Fabric From Rack to Superpod

Microsoft treats the network as a native extension of the dataflow design. A custom AI transport based on Ethernet connects accelerators in a two-tier topology that reaches up to 12,288 accelerators in a superpod ¹.

Inside a rack, 96 accelerators connect through a mix of direct links and switched paths, all electrical to save power and cost. Second-tier optical switches join up to 128 racks. Accelerators can initiate communication with any peer in the superpod without involving the host or a separate scale-out NIC, and one accelerator can drive up to 750 GB/s to a peer when only some are going cross-rack ¹.

There are three latency domains: a directly linked pair, accelerators within a pod one switch hop away, and cross-pod accelerators three hops away. That lets software keep latency-sensitive tensor parallelism local while expert and pipeline parallelism span racks. The same fabric supports disaggregated inference, moving KV state from prefill to decode workers without leaving the domain, and lets schedulers borrow hot-spare capacity from another rack when hardware drops out ¹.

Rack Scale and Product Impact

A Maia 300 rack holds 96 accelerators in a 167kW envelope, with 27.6 TB of HBM and 1.7 petabytes per second of aggregate HBM bandwidth. Microsoft says it delivers up to seven times the rack-scale token throughput of a Maia 200 rack ².

Deployment begins in 2027, as Microsoft prepares to scale its in-house AI infrastructure toward multi-gigawatt levels ². Microsoft expects the chip to support agentic coding in GitHub, model serving in Foundry, Microsoft AI team workloads, and features in Copilot, Cowork and Teams ².

Maia 200 is already running production workloads, including Microsoft 365 Copilot and Microsoft AI models. Microsoft claims up to 1.7 times more tokens per watt than leading GPU infrastructure for built-in AI features in Teams, Copilot and Cowork, with GitHub Copilot and Foundry expected to follow by the end of this year ². These are vendor claims, and independent benchmarks will matter once the hardware ships.

Companies mentioned: Microsoft (MSFT $540.77 ▲1.1%) • TSMC (TSM $446.75 ▼1.4%)

Primary sources

1. Deep dive into Azure Maia 300 architecture (azure.microsoft.com) – Microsoft's own technical deep dive into Maia 300 describes an inference-first system co-designed across silicon, software, racks, networking and datacenters. It covers the SDLA dataflow architecture, the multi-chiplet design on TSMC N2P with a 36% compute FLOPs gain, 2.5 times HBM bandwidth and 14% more scale-up bandwidth, plus eight HBM4 stacks giving 288 GB and 18 TB/s per accelerator. It details narrow-precision formats (enhanced MXFP4, sub-3-bit storage), NUMA memory organization and vector engine kernel optimizations. The networking section explains the Ethernet-based scale-up fabric, the 96-accelerator rack, the 12,288-accelerator superpod, three latency domains, up to 750 GB/s per peer, flexible job placement, disaggregated prefill and decode, and cross-rack hot spares. The text provided cuts off during the rack-scale discussion. Published on azure.microsoft.com.
2. Microsoft unveils Maia 300, its most powerful AI chip yet – and it might make using Teams, Copilot and Cowork much better at last (techradar.com) – TechRadar's news report summarizes Microsoft's Maia 300 announcement. It cites deployment starting in 2027, a claimed 40% more tokens per megawatt than other leading GPU-based systems, and up to seven times the rack-scale token throughput of a Maia 200 rack. It adds specifics such as a 950W TDP, eight HBM4 stacks at 18TB/s, and a 96-accelerator rack at 167kW with 27.6TB of HBM and 1.7 petabytes per second of bandwidth. It also notes the intended benefits for GitHub, Foundry, Copilot, Cowork and Teams, and that Maia 200 already runs Microsoft 365 Copilot with up to 1.7 times more tokens per watt. Written by Mike Moore, Deputy Editor at TechRadar Pro.

Powered by News Ranker