# How Small Can We Slice an AI Model?

> Source: <https://operator0x.com/field-notes/weights-still-need-a-home/>
> Published: 2026-11-02 00:15:04+00:00

AI Beyond the Data Center

# How Small Can We Slice an AI Model?

The question that could change who gets to run one.

Every few months, someone points out that the world is drowning in idle compute. Your laptop idles at an office desk. Your phone spends most of its battery waiting in your pocket. Meanwhile, tech monopolies pour hundreds of billions of dollars into sprawling, [water-guzzling data centers](https://bies.lbl.gov/publications/2024-lbnl-data-center-energy-usage-report) packed with high-end clusters to train and run modern language models.

The intuitive alternative is obvious: split the model up and run it across everyone’s devices.

Yet whenever people attempt decentralized AI, they run into the same brick wall. Projects like [Petals](https://arxiv.org/abs/2312.08361) and [BloomBee](https://arxiv.org/abs/2604.21072) proved you could run inference over the public internet, but they mostly approach it through pipeline parallelism—handing entire layers to individual machines.

The trouble with that strategy is the entry fee. If Layer 14 requires 8 GB or 16 GB of memory, every device with only 2 GB or 4 GB of spare RAM gets locked out. The network ends up looking like a rickety chain of modest servers rather than a truly universal computing mesh.

The real breakthrough isn’t distributing layer by layer. It is slicing within the sub-layers themselves.

## The Two Engines of a Layer

A standard Transformer block is made of two distinct computational engines: the feed-forward / MLP block and the attention mechanism.

For a visual way to picture what changes as information passes through those blocks, I explored that in [The words are the shadow](https://operator0x.com/field-notes/model-execution-graph/). Here, the question is how to split that computation across devices.

Historically, both engines demanded continuous, dense matrix multiplication across all active parameters. But recent architectural innovations—driven by frontier open models like [DeepSeek-V4-Flash](https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash)—have fundamentally fractured that monolithic structure.

### 1. The MLP Layer: Radical Sparsity and Granular Experts

Modern [mixture-of-experts (MoE) models](https://arxiv.org/abs/2401.06066) no longer use one monolithic MLP block per layer. They break the feed-forward step into many isolated, specialized sub-networks. In a 284-billion-parameter model, only a fraction—around 13 billion parameters—fires for any given token.

Because experts operate independently, you do not need a machine to host an entire MLP layer. You don’t even need a machine to host an entire expert:

- **Expert-level sharding:** A device can host a single isolated expert, or even a[subset of weights within that expert’s projection matrices](https://arxiv.org/abs/1909.08053) .
- **Targeted activation:** Instead of sending massive hidden states everywhere, a smart routing layer routes the token only to the specific nodes holding the experts chosen for that exact step.
- **Marginal entry costs:** In this proposed design, a machine with 500 MB to 1 GB of free RAM could hold a sliver of an expert matrix and only wake up when its slice is summoned. That is a target memory budget, before runtime and communication overhead.

### 2. The Attention Mechanism: Compacting the KV Cache

The second bottleneck has always been the Key-Value (KV) cache. In traditional Multi-Head Attention, as context expands, the memory needed to store attention states explodes, making long-context inference nearly impossible on distributed consumer nodes.

Recent developments have altered this equation:

- **Latent compression:** Techniques like[Multi-Head Latent Attention (MLA)](https://arxiv.org/html/2405.04434v5) compress key and value projections into tiny, low-rank latent vectors. In a distributed design, nodes could exchange compressed latent representations where the partition permits it.
- **Adaptive attention blocks:** Rather than every token attending to every previous position across the full context window,[sparse attention](https://arxiv.org/abs/2502.11089) and[computation skipping](https://arxiv.org/abs/2404.02258) reduce context connections or computation. These are separate techniques that a swarm design could combine.
- **Traffic suppression:** The goal is to combine state compression with[cache traffic reduction](https://arxiv.org/abs/1911.02150) , so attention work can be sharded across edge devices without saturating consumer upload speeds.

## Breaking the Data-Center Bottleneck

Conventional wisdom holds that running tensor parallelism outside of a server rack is impossible because consumer Wi-Fi latency is orders of magnitude slower than high-speed server interconnects.

That is true if you try to replicate a dense supercomputer over home broadband. It is not necessarily true if the architecture itself is built around sparsity and localization:

| Existing pipeline structure compared with proposed sub-layer swarm design targets. |  |  | 
|---|---|---|
| Dimension | Conventional Pipeline Parallelism | Proposed Sub-Layer Edge Swarm | 
|---|---|---|
| Granularity | Whole layers assigned to whole machines | Micro-shards of individual experts and attention work | 
| Device memory | Depends on model, quantization, and block placement | 500 MB to 2 GB of everyday consumer RAM as a design target | 
| Token activation | Tokens cross sequential layer stages, which may already contain MoE | Sparse execution: chosen experts and selected attention paths run | 
| Network payload | Hidden-state activations between stages; KV caches can stay with their hosting layers | Compressed representations where supported, plus communication between tensor shards | 

The comparison describes a proposed architecture, not a measured deployment. Smaller shards reduce the weight memory needed at each device; the resulting communication still has to be engineered and benchmarked.

When nodes are organized by network topology, the router dispatches tokens based on both semantic specialty and physical proximity. Adjacent expert shards can live on nodes sharing low-latency local networks, eliminating unnecessary transatlantic round-trips.

### 1. Who is Building the Decentralized Swarm? (The Protocols)

On the decentralized edge side, several major projects are building the infrastructure to host, shard, and route models dynamically across distributed devices:

- **Petals (BigScience / Open-Source):** This is the direct implementation of the "BitTorrent for AI" model. Built by open-source researchers, Petals lets users shard massive models across standard internet connections. Instead of one server running a 70B parameter model, Node A runs layers 1–8, Node B runs 9–16, and activations are routed peer-to-peer.
- **Aethir & Akash Network:** These are decentralized compute networks designed to aggregate consumer GPUs, edge nodes, and underutilized data centers into a unified global cloud. They provide the raw decentralized substrate where micro-experts can be hosted.
- **Exollama / Exo:** Projects emerging in the local LLM community focused on**P2P cluster inference** . Exo allows everyday users to link laptops, phones, and desktop GPUs over local Wi-Fi or peer networks to run unified split-model inference without relying on cloud APIs.

### 2. Who is Building the Architecture? (The Frontier Labs)

While decentralized protocols build the *network*, traditional frontier labs are building the *sparse geometric mathematics* (MoE, compressed cache, dynamic depth) to run internally on their hardware:

- **DeepSeek:** They have proven to be the most aggressive innovators in**KV cache compression** . Their introduction of**Multi-head Latent Attention (MLA)** —which compresses historical tokens into low-rank latent vectors—is the exact mathematical mechanism required to run high-context models on limited edge memory.
- **OpenAI (Noam Shazeer & Architecture Research):** Shazeer (who invented MoE and co-invented the Transformer) leads next-generation architecture research. His focus centers on breaking away from rigid Transformer stacks toward dynamic-depth, continuous execution, and non-standard architectures like State-Space Models (SSMs).
- **Google DeepMind:** They pioneered**Mixture-of-Depths (MoD)** , proving mechanically that you can route tokens down variable vertical paths while preserving fixed GPU batching shapes using top-`k` capacity routing.

### 3. Why This Could Be Massive (The Market Shift)

This architecture isn't just an interesting engineering trick; it fundamentally restructures the economics of AI:

1. **Destruction of the Cloud API Monopoly:** Today, AI intelligence is tied to cloud subscriptions controlled by a handful of centralized data center owners. An efficient, decentralized edge-swarm protocol bypasses compute monopolies completely, democratizing frontier-level intelligence.
2. **Unlocking Edge NPUs:** Millions of modern smartphones, laptops, and local devices ship with integrated Neural Processing Units (NPUs) that sit mostly idle. A peer-to-peer routing network transforms that global, dormant consumer hardware into the largest supercomputer on Earth.
3. **Zero-Trust Privacy:** If processing takes place across local micro-experts and compressed latent shadows, sensitive data never has to be reconstructed into plain text on a centralized corporate server.

Whoever successfully connects these two halves—**the sparse dynamic geometry** (MoE + compressed KV + dynamic paths) with **the decentralized protocol layer** (P2P routing)—will own the post-cloud infrastructure of intelligence.

## The Democratization of Compute

For the last decade, AI progress has favored centralization: bigger clusters, higher thermal envelopes, and deeper corporate pockets.

Building a swarm on sub-layer sharding inverts that incentive structure. Millions of office laptops, phones, smart TVs, and retired hardware—previously dismissed as e-waste in the LLM era—could become functional cells in an emergent supercomputer.

The path forward won’t come from waiting for consumer home internet to mimic data center fiber. It will come from software that treats sparsity, adaptive attention, and sub-layer fragmentation as native design principles. When models can be sliced thin enough to fit anywhere, intelligence stops belonging exclusively to the companies that own the data centers.

An architectural argument based on the linked research. The proposed swarm, tiny-device memory budgets, and end-to-end network efficiency have not been demonstrated here.