cd /news/ai-infrastructure/show-hn-shard-distributed-ai-inferen… · home › topics › ai-infrastructure › article
[ARTICLE · art-171431] src=github.com ↗ pub= topic=ai-infrastructure verified=true sentiment=↑ positive

Show HN: Shard – Distributed AI Inference Across Multiple Computers

An open-source project called SHARD (shard-engine) released a distributed local LLM inference engine that pools GPU VRAM and system RAM across LAN machines to run 14B, 32B, and 70B+ open-weight models that no single card can hold. SHARD, written in Python with FastAPI and PyTorch on a GGUF/llama.cpp backend plus native C++20/MASM AVX2 SIMD kernels, uses UDP discovery on port 8001 and FastAPI SSE endpoints on port 8000, and exposes OpenAI-compatible (/v1/chat/completions, /v1/completions, /v1/models) and Anthropic-compatible (/v1/messages) endpoints so coding agents such as Cursor, Claude Code, Aider, Continue, Zed, and OpenCode can connect. The project also ships a `shard fit` command that checks whether a Hugging Face model will fit a single GPU, a hybrid CPU+GPU setup, or a pooled LAN cluster before weights are downloaded.

key takeaway

SHARD splits transformer layers across multiple consumer GPUs and host RAM over gigabit LAN and serves OpenAI- and Anthropic-compatible endpoints on port 8000, letting existing coding agents point at pooled local hardware instead of a cloud API.

read16 min views4 publishedNov 17, 2026
Show HN: Shard – Distributed AI Inference Across Multiple Computers
Image: Michielbdejong (auto-discovered)

English  |  Русская версия  |  Қазақша нұсқасы

One Network. Many Servers. Shared AI Infrastructure.

<sub>Distributed local LLM inference engine pooling GPU VRAM and system RAM across LAN machines — powered by PyTorch, FastAPI, llama.cpp RPC, and native C++20 / MASM AVX2 SIMD kernels.</sub>

SHARD Distributed Inference Cluster: Central master coordinator pooling VRAM across connected LAN worker nodes over UDP discovery (:8001) and FastAPI SSE endpoints (:8000).

Overview · How It Works · Features · Quick Start · CLI Reference · Architecture · Dashboard · Community

SHARD (shard-engine) is an open-source distributed local LLM inference engine written in Python (FastAPI + PyTorch) with a GGUF / llama.cpp backend and native C++20 / MASM AVX2 SIMD acceleration.

Running capable 14B, 32B, or 70B+ open-weight language models locally typically hits a hard hardware wall: single-card VRAM limits. Most developers, labs, and small teams own multiple machines—a desktop gaming GPU, a workstation laptop, an auxiliary mini-PC—whose memory sits fragmented across the local network while no single device can fit the full model weights and KV cache.

SHARD unites idle GPUs and system memory across your Local Area Network (LAN) into a single logical inference cluster:

  • Pool Hardware Without Cloud Lock-In: Split transformer layers across multiple consumer GPUs and host RAM over standard gigabit LAN (lan.py ,gguf_backend.py ) with zero external cloud dependencies or recurring API fees.

  • Know Before You Download (shard fit): Inspect whether any Hugging Face model will fit your single GPU, hybrid CPU+GPU setup, or pooled LAN cluster before down gigabytes of weights (fit.py ).

  • Drop-In Agent & IDE Backend: Expose both OpenAI (/v1/chat/completions ,/v1/completions ,/v1/models ) and Anthropic (/v1/messages ) compatible endpoints on port8000 so coding agents and SDKs connect immediately.

  • Local AI Developers & Researchers running 7B–70B GGUF or PyTorch SafeTensors models across home lab or office workstations.

  • Software Engineering Teams hosting private, zero-telemetry coding assistants forCursor ,Claude Code ,Aider ,Continue ,Zed , andOpenCode .

  • AI Agent Builders orchestrating multi-agent pipelines viaOpenAI Python SDK ,LangChain , orLlamaIndex against self-hosted hardware.

SHARD turns distributed local AI infrastructure into a four-step workflow: CREATE → SHARE → CONNECT → USE.

| Start a local SHARD node with shard start orpython main.py . The engine probes your CPU, system RAM, and NVIDIA CUDA / Apple MPS GPU viahardware.py , loads your chosen GGUF or Hugging Face model, and brings up the FastAPI server and Web UI onhttp://localhost:8000 . | Open a LAN compute pool with shard lan create --open (or generate a pairing code viashard lan create ). SHARD broadcasts UDP discovery beacons on port8001 and exposes/api/lan/* endpoints so peer machines on your subnet can pool their VRAM and RAM. | | On any secondary computer on the network, run shard lan join <master-ip> --share . The worker completes an HTTP handshake (POST /api/lan/pair ), launches its local RPC tensor worker, sends periodic health heartbeats, and contributes its GPU/RAM to the cluster. | Connect coding IDEs, autonomous AI agents, or custom Python apps to http://<master-ip>:8000/v1/chat/completions or/v1/messages . Master coordinates layer execution across connected nodes and streams tokens back via Server-Sent Events (SSE). |

Every feature listed below is verified directly against the SHARD source tree (v1.8.1).

Category Capability Source Module Description
LAN Cluster & Sharding UDP Auto-Discovery & LAN Pooling lan.py , docs/LAN_SHARDING_DESIGN.md Broadcasts UDP beacons ( :8001 ), pairs nodes via code or--open mode, monitors node heartbeats (/api/lan/heartbeat ), and pools VRAM/RAM across machines.
Inference Backends Dual PyTorch & GGUF (llama.cpp) Engine engine.py , gguf_backend.py Runs Hugging Face transformers /safetensors models on CUDA/CPU/MPS and quantized GGUF weights via managedllama-server &ggml-rpc-server subprocesses with configurable GPU layer offload (-ngl ).
Hardware Planning Pre-Flight Model Fit Advisor (shard fit) fit.py , hardware.py Inspects model architecture parameters ( config.json or parameter heuristics) against host VRAM and RAM before down, returningGPU ,HYBRID , orIMPOSSIBLE verdicts, max context length, estimated tok/s, and quantization advice.
Native Acceleration C++20 & MASM AVX2 SIMD Kernels native/shard_core.cpp , native/shard_asm.asm , shard_bridge.py Custom C++20 and x86-64 AVX2 assembly routines for RMSNorm ,LayerNorm , Rotary Position Embeddings (RoPE ),Softmax ,GELU ,SiLU , andMatMul loaded viactypes .
API & Streaming OpenAI & Anthropic Compatible Server server.py Serves /v1/chat/completions ,/v1/completions ,/v1/models ,/v1/messages (Anthropic protocol),/api/generate , and/api/generate/stream with real-time Server-Sent Events (SSE) and optional API key auth (shard key ).
Web Control Panel Zero-Dependency Dashboard & Playground dashboard.html Built-in browser UI at http://localhost:8000 with live GPU/VRAM/RAM telemetry, Hugging Face model search & download, quantization selector, streaming chat playground, LAN node manager, and one-click IDE integration snippets.
Containerization Docker & NVIDIA GPU Compose Stack Dockerfile , docker-compose.yml Production-ready container setup with persistent named volumes ( shard_models_volume ,shard_data_volume ) and NVIDIA GPU reservation.
Collaborative Training Optional SOUP Fine-Tuning Ecosystem soup_adapter.py ,feedback.py ,training.py ,docs/COLLABORATIVE_TRAINING.md Privacy-first consent feedback, secret/PII redaction, immutable SFT/DPO dataset snapshots, dynamic SOUP training orchestration with cancellation, Baseline vs Candidate eval, model versioning, deployment, and 1-click rollback.
Unified Hosting & Privacy Model-First Hosting & Private Sessions server.py ,lan.py Enforces loaded model verification before server creation/publishing (HTTP 409 MODEL_NOT_READY ), guarantees zero-leakage participant-isolated private chats, and enforces compute worker VRAM/CPU quotas.
Global Network & Discovery Automatic Network & 1-Click Clusters net_strategy.py ,net_discovery.py ,cluster.py Automatic network initialization on shard start , public Server Catalog with search and VRAM filters, STUN/NAT traversal, blind encrypted WebSocket relay, pre-join hardware probing, explicit consent enforcement, and distributed tensor pipelines.
  • Multi-Host llama.cpp RPC Tensor Distribution (gguf_backend.py, lan.py): Distributed layer execution over TCP/RPC across heterogeneous LAN GPUs works best on low-latency wired Gigabit/2.5GbE Ethernet subnets; Wi-Fi links may experience higher inter-token latency.

  • Native Win32 Desktop GUI (native/shard_gui.cpp): Native C++ Windows desktop wrapper alongside the cross-platform web dashboard (dashboard.html ).

  • Automated Dynamic Re-Sharding on Node Dropout: Seamless mid-generation layer migration if a worker node disconnects unexpectedly (docs/LAN_SHARDING_DESIGN.md ).

  • Linux / macOS ARM64 NEON Intrinsics: Expanding the native SIMD kernel suite (native/ ) beyond x86-64 AVX2/FMA to native Apple Silicon / ARM64 assembly paths.

  • OS: Windows 10/11 (64-bit), Linux (Ubuntu 20.04+ / Debian 12+), or macOS.

  • Python: Python3.8+ (3.10+ recommended).

  • Hardware: NVIDIA GPU with CUDA drivers (recommended for GPU acceleration), Apple Silicon (MPS), or x86-64 CPU with AVX2 support.

  • Network (Optional for LAN Sharding): Two or more machines on the same LAN with TCP port8000 (HTTP/API) and UDP/TCP port8001 (LAN discovery & RPC) open.

Install SHARD directly from PyPI in any environment:

pip install shard-engine

pip install "shard-engine[full]"

To install the latest development snapshot directly from GitHub:

pip install "git+https://github.com/avelone123123/shard.git"

pip install "shard-engine[full] @ git+https://github.com/avelone123123/shard.git"

Verify your environment right after installation:

shard doctor

Windows PATH Note: If Windows reports that shard.exe is not on your PATH, you can run SHARD directly through Python: python -m shard_engine <command> (for example, python -m shard_engine help or python -m shard_engine start). GPU & Runtime Note: GGUF inference auto-downloads platform binaries for llama-server / ggml-rpc-server on first run and needs no extra ML packages. For CUDA PyTorch inference, install a matching CUDA build of torch before pip install "shard-engine[full]".

git clone https://github.com/avelone123123/shard.git
cd shard
pip install -e ".[dev]"
git clone https://github.com/avelone123123/shard.git
cd shard
docker-compose up --build

This builds shard-engine, mounts persistent volumes (/data/models and /data/shard), exposes port 8000:8000, and enables NVIDIA GPU passthrough.

To compile shard_core.dll (Windows MSVC cl.exe + ml64.exe) or shard_core.so (Linux GCC/Clang):

python native/build_native.py
shard update --check
pip install --upgrade shard-engine
pip uninstall shard-engine

Uninstalling the Python package keeps your downloaded models, settings, and API keys intact in your user data directory.

shard doctor

shard start --daemon
shard status

shard gui

shard restart
shard stop
shard fit Qwen/Qwen2.5-Coder-7B-Instruct
shard fit meta-llama/Llama-3.1-8B-Instruct
shard run Qwen/Qwen2.5-Coder-7B-Instruct

shard list

On the Master Machine (shard lan create prints the live LAN IPv4 addresses detected on this host):

shard lan create --open --share --model Qwen/Qwen2.5-Coder-7B-Instruct

shard lan create --share

On each Worker Machine on the same subnet, replace <SERVER_IP> with the IP printed by shard lan create on the Master Machine:

shard lan join <SERVER_IP> --share

shard lan join <SERVER_IP> <PAIR_CODE> --share

shard lan status

shard lan off
Command Arguments & Flags Description
shard start [--host HOST] [--port PORT] [-d|--daemon] [--no-browser] Launches the SHARD FastAPI server and Uvicorn runtime (default 0.0.0.0:8000 ).
shard stop (none) Safely stops the background SHARD server managed by PID tracking.
shard restart [--host HOST] [--port PORT] [--foreground] [--no-browser] Stops and restarts the SHARD server cleanly.
shard gui (none) Launches the server (or native Win32 desktop UI if present) and opens the Web Dashboard.
shard status (none) Displays detected CPU, RAM, GPU VRAM, active model, and detected LAN IPv4 addresses.
shard doctor (none) Audits Python, packaged assets, dependencies, CUDA/GPU, LAN IPs, ports, and user directories.
shard version [-V|--version] Prints installed SHARD version, Python executable path, OS info, and active directories.
shard update [--check] Checks PyPI for newer shard-engine releases or upgrades non-destructively viapip .
shard fit <model_id> [--ctx N] [--no-fetch] Calculates VRAM/RAM requirements, quantization fit, and estimated throughput before down.
shard run [model_id] Downloads (if needed), loads the model, and opens interactive terminal chat REPL.
shard list [--category CAT] Lists cached and curated catalog models ( all ,ideal ,coding ,reasoning ,compact ).
shard lan [status|create|join|model|off] [extra] [--share] [--open] [--model ID] [--port N] Creates, joins, configures, inspects, or stops distributed LAN compute pooling across machines.
shard network [status|doctor|radmin [status|install|repair|gui|decline]] Inspects multi-tier autonomous network, runs network doctor, and manages automated Radmin VPN fallback.
shard training [status|doctor|privacy|feedback|dataset|start|logs|cancel|resume|models|eval|prepare|approve|deploy|rollback] Manages consent feedback, dataset snapshots, SOUP fine-tuning jobs, evaluation, model deployment, and rollback ( Guide ).
shard key [--gen] Views the current local API key or generates a new token ( --gen ).
shard models-dir [path | reset] Views, updates, or resets the directory where model weights are stored.
shard test [--strict-native] Verifies tensor math kernels and AVX2/pure-Python fallback throughput.
shard help [-h|--help] Prints full CLI command usage and examples.

SHARD stores user data outside site-packages following OS conventions (and reads existing ~/.shard data cleanly on upgrade):

  • Windows:%LOCALAPPDATA%\SHARD (config,keys_db.json ,logs/ ,run/ ,models/ )
  • Linux:~/.local/share/shard ,~/.config/shard ,~/.cache/shard ,~/.local/state/shard (respectsXDG_* variables)
  • macOS:~/Library/Application Support/SHARD ,~/Library/Caches/SHARD ,~/Library/Logs/SHARD
Variable / File Default Purpose
SHARD_HOST 0.0.0.0 Network interface IP address bound by server.py .
SHARD_PORT 8000 HTTP port for the API server and Web Dashboard.
SHARD_DATA_DIR OS data directory ( /data/shard in Docker) Overrides root directory for settings.json ,keys_db.json ,lan.json , logs, and PID metadata.
SHARD_MODELS_DIR <data_dir>/models (/data/models in Docker) Storage directory for downloaded Hugging Face SafeTensors and GGUF weights.
SHARD_NO_BROWSER /SHARD_DOCKER (unset) Suppresses auto-opening a browser tab when starting SHARD.
SHARD_NATIVE_LIB (auto-discovered) Path override to shard_core.dll /.so /.dylib (setnone to force Python fallback).
SHARD_SOUP_PYTHON /SHARD_SOUP_CMD (auto-discovered) Optional path to a Python 3.11–3.13 environment or binary running soup-cli when training on Python 3.14+ hosts.
HF_HOME /TRANSFORMERS_CACHE ~/.cache/huggingface Standard Hugging Face cache override respected by engine.py andhf_hub.py .
engine_optimizations.json Packaged inside shard_engine Runtime optimization profile loaded via importlib.resources .

SHARD Ecosystem Topology: Coding IDEs, agent frameworks, master coordinator, and distributed LAN worker nodes operating as a unified inference fabric.

flowchart TB
    subgraph Clients["AI Agents, IDEs & SDK Clients"]
        IDE["Cursor / Claude Code / Aider / Continue / Zed"]
        SDK["OpenAI SDK / Anthropic SDK / LangChain / LlamaIndex"]
        Browser["Web Dashboard UI (dashboard.html)"]
    end

    subgraph Master["SHARD Master Node (FastAPI :8000)"]
        Server["server.py\n(/v1/chat/completions, /v1/messages, /api/*)"]
        Fit["fit.py & hardware.py\n(Pre-Flight VRAM/RAM Advisor)"]
        Engine["engine.py\n(PyTorch Transformers & Token Streamer)"]
        GGUF["gguf_backend.py\n(llama-server & Layer Offload -ngl)"]
        Bridge["shard_bridge.py\n(ctypes C++20 / MASM AVX2 SIMD Core)"]
        LANCoord["lan.py\n(UDP Discovery :8001 & Cluster Coordinator)"]
    end

    subgraph Workers["Distributed LAN Worker Nodes (:8001)"]
        W1["Worker Node 1\nggml-rpc-server / GPU VRAM Shard"]
        W2["Worker Node 2\nggml-rpc-server / GPU VRAM Shard"]
        W3["Worker Node N\nggml-rpc-server / System RAM + GPU"]
    end

    IDE -->|HTTP / SSE :8000| Server
    SDK -->|HTTP / SSE :8000| Server
    Browser -->|REST / SSE :8000| Server

    Server --> Fit
    Server --> Engine
    Server --> LANCoord
    Engine --> GGUF
    Engine --> Bridge

    LANCoord <-->|UDP Broadcast & Heartbeat :8001| W1
    LANCoord <-->|UDP Broadcast & Heartbeat :8001| W2
    LANCoord <-->|UDP Broadcast & Heartbeat :8001| W3

    GGUF <-->|TCP Tensor RPC Stream| W1
    GGUF <-->|TCP Tensor RPC Stream| W2
    GGUF <-->|TCP Tensor RPC Stream| W3
shard/
├── assets/
│   ├── branding/              # Official SHARD SVG & PNG logos, icons, and favicon
│   ├── animations/            # Isometric pixel-art / voxel animated GIFs
│   └── screenshots/           # Dashboard and CLI fit advisor visuals
├── docs/
│   └── LAN_SHARDING_DESIGN.md # Distributed LAN sharding protocol & architecture spec
├── native/
│   ├── build_native.py        # Cross-platform compiler for C++20 & MASM AVX2 kernels
│   ├── shard_core.cpp         # C++20 RMSNorm, LayerNorm, RoPE, Softmax, GELU, SiLU, MatMul
│   ├── shard_asm.asm          # x86-64 MASM AVX2 SIMD assembly kernels
│   └── shard_gui.cpp          # Native Win32 desktop GUI wrapper
├── shard_engine/                # Installable Python package (paths, process, CLI, API, inference)
│   ├── paths.py                 # Cross-platform XDG / AppData / macOS paths & packaged assets
│   ├── process.py               # Detached daemon launcher & verified PID lifecycle management
│   ├── cli.py                   # `shard` command-line interface entry point (`shard_engine.cli:main`)
│   ├── main.py                  # Application launcher & Uvicorn bootstrapper
│   ├── server.py                # FastAPI server (OpenAI, Anthropic, Model & LAN routes)
│   ├── engine.py                # Core LLM inference orchestrator & SSE generator
│   ├── gguf_backend.py          # Managed llama.cpp / GGUF server & RPC integration
│   ├── lan.py                   # Dynamic subnet IP discovery, UDP beacons & LAN pooling
│   ├── fit.py                   # Pre-flight model memory & throughput calculator
│   ├── hardware.py              # GPU VRAM (nvidia-smi), CPU & RAM probing
│   ├── hf_hub.py                # Hugging Face Hub search, metadata & down
│   ├── shard_bridge.py          # Python ctypes bridge to native shard_core.dll / .so
│   ├── settings.py              # Persistent configuration manager
│   └── dashboard.html           # Packaged web control panel & agent config generator
├── pyproject.toml               # Single-source packaging configuration (version from __init__.py)
├── tests_shard.py               # Unit, API, fit advisor, LAN sharding & kernel verification suite
└── tests_cli_pkg.py             # Packaging, OS path, CLI, PID safety & 2-node LAN cluster suite

Navigating to http://localhost:8000 opens the built-in SHARD Control Panel:

SHARD Web Dashboard: Real-time per-node VRAM/RAM gauges, Pre-Flight Model Fit Advisor, one-click IDE/agent config snippets, and streaming playground.

Pre-Flight Fit Verification: Evaluate model quantization, context ceiling, and pooled LAN cluster fit before down model weights.

Point any OpenAI- or Anthropic-compatible client to your SHARD instance:

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:8000/v1",
    api_key="shard-local",  # Or generate a token via `shard key --gen`
)

stream = client.chat.completions.create(
    model="Qwen/Qwen2.5-Coder-7B-Instruct",
    messages=[{"role": "user", "content": "Explain tensor parallelism across LAN nodes."}],
    stream=True,
)

for chunk in stream:
    if chunk.choices[0].delta.content:
        print(chunk.choices[0].delta.content, end="", flush=True)

SHARD supports collaborative model-parallel execution across independent machines (LAN, Radmin VPN, Tailscale, WireGuard, or NAT-traversed encrypted tunnels). A coordinator can schedule a large model across multiple consented worker nodes and expose one unified OpenAI/Anthropic endpoint (/v1/chat/completions, /v1/messages).

Coordinator (Create & Publish Global Server):

shard start --daemon
shard cluster create Qwen/Qwen2.5-Coder-7B-Instruct --name MyGlobalCluster --strategy auto --publish --slots 4
shard cluster status

Discover Global Servers & 1-Click Join:

shard cluster catalog --q deepseek --min-vram 16

shard cluster join shn_coordinator_node_id --client

shard cluster join shn_coordinator_node_id --consent --vram-pct 70 --max-ram-gb 8 --max-threads 4

For workers behind restrictive CGNAT/NAT, SHARD automatically traverses NAT using STUN or establishes an end-to-end ChaCha20-Poly1305 encrypted blind WebSocket relay tunnel with zero configuration. 3. Deploy & Monitor:

shard cluster plan
shard cluster deploy
shard cluster status

SHARD — One Network. Many Servers. Shared AI Infrastructure.

<sub>Built for the open-source local AI community · English · Русский · Қазақша</sub>

── more in #ai-infrastructure 4 stories · sorted by recency
── more on @shard 3 stories trending now
sponsored brought to you by zahid.host 4,200+ EU-deployed projects
reading about agents? ship yours in a single git push.

Run your AI side-project on zahid.host

EU-based hosting, git-push deploys, automatic HTTPS, no cold starts. Free tier with a custom domain — perfect for shipping the agent you just read about.

$git push zahid main
→ Live at https://your-agent.zahid.host ✓
Get free account → Pricing
from €0/mo · no card required
LIVE [news/show-hn-shard-distri…] indexed:0 read:16min 2026-11-17 · —