Meta has put networking front and center in its next-generation custom silicon, placing network interface cards (NICs) inside the chip package itself to reduce interconnectivity bottlenecks.
Its Meta Training and Inference Accelerator (MTIA) 300 packs two network chiplets – each packing six custom 800 Gb/s remote direct memory access (RDMA) NICs – to provide 1.2 TB/s of total I/O bandwidth without the need to touch a peripheral component interconnect express (PCIe) bus.
Normally, NICs would sit off-chip, with the central processing unit (CPU) used to broker traffic between accelerators and the network. Meta engineers have essentially removed that extra hop to eliminate that bottleneck.
Meta claims the on-board NICs make the MTIA 300 specially suited for training recommendation and ranking models, which are the engine behind what shows up in feeds of Meta apps like Facebook and Instagram
Unlike large language models (LLMs), which require vast floating-point throughput, recommendation models require constant communication between accelerators. Their embedding tables, which contain over 99% of the model’s parameters, require memory capacity and high memory bandwidth (HBM).
Meta's latest earnings showed $59.4 billion in advertising revenue, highlighting that having custom hardware is a competitive necessity.
The hyperscaler has been designing its own custom MTIA silicon to power its social media empire since 2023. Back in March, it unveiled the next four generations of MTIA, with the 400 version a compute powerhouse offering six petaflops of eight-bit floating point (FP8) compute performance.
The network-enhanced MTIA 300s boast impressive performance details. On a 150 billion parameter recommendation model running across 40 accelerators, the chip’s total communication time is 3.9-times faster than equivalent graphic processing unit (GPU) clusters. Meta uses the same 12 Ethernet-based NICs for both scale-up and scale-out, which allows for flexible partitioning of the NICs to adjust the changing workload needs.
The chip houses huge on-chip memory, 216 gigabytes of high-bandwidth memory 3E (HBM3E), which supports larger local batch sizes and reduces communication overheads. It's paired with Meta's HCCL custom communication library that can execute network operations without host CPU involvement by compiling data patterns into autonomous work-queue subgraphs, while high-network bandwidth lets Meta use higher-precision datatypes to maintain precision.
“Rather than driving communication from the host during execution, HCCL compiles each collective into a complete set of subgraphs – arrays of work-queue entries with explicit dependencies – dispatched to message engines for fully autonomous execution. Once work reaches the device, the host is uninvolved," Meta noted in a blog post.
Meta engineers contend that while the package is designed for training recommendation models, the integrated networking and offloaded collective execution concepts could apply to broader workloads, including AI.
“As AI inference evolves toward reasoning, agentic, and long-context use cases, the communication demands a shift: messages become smaller, more frequent, and latency-sensitive, with tighter per-collective budgets,” the blog post reads.