Communication offloading is achieved via 16 dedicated message engines with RISC-V cores, NIC interfaces, and near-memory compute blocks that perform reductions at 128 bytes per cycle, delivering over 2.8 TB/s of reduction throughput and enabling line-rate AllReduce and ReduceScatter without touching the compute grid.

MTIA 300 is Meta’s first training chip in the MTIA family, optimized for training recommendation and ranking models.

The model integrates with PyTorch via c10d and torchcomms, routing collectives through a compiled graph that respects topology-aware algorithms to minimize cross-rack traffic.

MTIA 300’s 216 GB of HBM3E supports larger local batch sizes, its 1:1 CPU-to-accelerator ratio enables offloading optimizer operations, and high network bandwidth enables higher-precision data types while maintaining precision.

Looking ahead, the architectural principles of integrated networking and offloaded communication are positioned to support broader workloads beyond training, including future inference with smaller, more frequent, latency-sensitive messages; Meta views this as foundational for next-generation AI silicon.

MTIA 300 introduces express doorbells to reduce per-transaction latency by eliminating an extra memory read, saving approximately 800 nanoseconds per operation.

In production, HCCL delivers up to 940 GB/s within a single rack; a 150-billion-parameter model trained on 40 accelerators achieves 3.9x faster total communication time than a comparable GPU cluster.

HCCL co-designs the communication library with MTIA 300 to compile collectives into subgraphs dispatched to MEs, enabling autonomous device-side execution without CPU involvement after transfer of instructions to high-bandwidth memory.

Scalability is achieved by using the same 12 Ethernet-based NICs for both scale-up (within a rack) and scale-out (across racks) communication, and hardware reconfiguration can adapt to changing needs without new hardware.

The chip integrates network interfaces directly on the package, with two network chiplets containing twelve 800 Gbps RDMA NICs, delivering 1.2 TB/s total I/O bandwidth without PCIe mediation, eliminating host-device bottlenecks.

The offload architecture isolates communication from compute; GEMMs running concurrently with collectives incur less than 0.5% compute throughput degradation, versus over 20% on traditional GPUs.