{"slug": "show-hn-shard-distributed-ai-inference-across-multiple-computers", "title": "Show HN: Shard – Distributed AI Inference Across Multiple Computers", "summary": "An open-source project called SHARD (shard-engine) released a distributed local LLM inference engine that pools GPU VRAM and system RAM across LAN machines to run 14B, 32B, and 70B+ open-weight models that no single card can hold. SHARD, written in Python with FastAPI and PyTorch on a GGUF/llama.cpp backend plus native C++20/MASM AVX2 SIMD kernels, uses UDP discovery on port 8001 and FastAPI SSE endpoints on port 8000, and exposes OpenAI-compatible (/v1/chat/completions, /v1/completions, /v1/models) and Anthropic-compatible (/v1/messages) endpoints so coding agents such as Cursor, Claude Code, Aider, Continue, Zed, and OpenCode can connect. The project also ships a `shard fit` command that checks whether a Hugging Face model will fit a single GPU, a hybrid CPU+GPU setup, or a pooled LAN cluster before weights are downloaded.", "body_md": "**English**  | \n  [Русская версия](https://github.com/avelone123123/shard/blob/master/README.ru.md)  | \n  [Қазақша нұсқасы](https://github.com/avelone123123/shard/blob/master/README.kz.md)\n\n**One Network. Many Servers. Shared AI Infrastructure.**\n\n<sub>Distributed local LLM inference engine pooling GPU VRAM and system RAM across LAN machines — powered by PyTorch, FastAPI, `llama.cpp` RPC, and native C++20 / MASM AVX2 SIMD kernels.</sub>\n\n**SHARD Distributed Inference Cluster:** Central master coordinator pooling VRAM across connected LAN worker nodes over UDP discovery (`:8001`) and FastAPI SSE endpoints (`:8000`).\n\n[Overview](#what-is-shard) ·\n  [How It Works](#how-shard-works) ·\n  [Features](#features) ·\n  [Quick Start](#installation--quick-start) ·\n  [CLI Reference](#cli-reference--configuration) ·\n  [Architecture](#architecture) ·\n  [Dashboard](#web-dashboard--demonstrations) ·\n  [Community](#project-status--community)\n\n**[SHARD](https://github.com/avelone123123/shard)** (`shard-engine`) is an open-source distributed local LLM inference engine written in Python ([FastAPI](https://fastapi.tiangolo.com/) + [PyTorch](https://pytorch.org/)) with a [GGUF / `llama.cpp` backend](https://github.com/ggml-org/llama.cpp) and native C++20 / MASM AVX2 SIMD acceleration.\n\nRunning capable 14B, 32B, or 70B+ open-weight language models locally typically hits a hard hardware wall: **single-card VRAM limits**. Most developers, labs, and small teams own multiple machines—a desktop gaming GPU, a workstation laptop, an auxiliary mini-PC—whose memory sits fragmented across the local network while no single device can fit the full model weights and KV cache.\n\nSHARD unites idle GPUs and system memory across your Local Area Network (LAN) into a single logical inference cluster:\n\n- **Pool Hardware Without Cloud Lock-In:** Split transformer layers across multiple consumer GPUs and host RAM over standard gigabit LAN (`lan.py` ,`gguf_backend.py` ) with zero external cloud dependencies or recurring API fees.\n- **Know Before You Download (`shard fit`):** Inspect whether any Hugging Face model will fit your single GPU, hybrid CPU+GPU setup, or pooled LAN cluster before downloading gigabytes of weights (`fit.py` ).\n- **Drop-In Agent & IDE Backend:** Expose both OpenAI (`/v1/chat/completions` ,`/v1/completions` ,`/v1/models` ) and Anthropic (`/v1/messages` ) compatible endpoints on port`8000` so coding agents and SDKs connect immediately.\n\n- **Local AI Developers & Researchers** running 7B–70B GGUF or PyTorch SafeTensors models across home lab or office workstations.\n- **Software Engineering Teams** hosting private, zero-telemetry coding assistants for[Cursor](https://www.cursor.com/) ,[Claude Code](https://docs.anthropic.com/en/docs/agents-and-tools/claude-code/overview) ,[Aider](https://aider.chat/) ,[Continue](https://www.continue.dev/) ,[Zed](https://zed.dev/) , and[OpenCode](https://github.com/opencode-ai/opencode) .\n- **AI Agent Builders** orchestrating multi-agent pipelines via[OpenAI Python SDK](https://github.com/openai/openai-python) ,[LangChain](https://www.langchain.com/) , or[LlamaIndex](https://www.llamaindex.ai/) against self-hosted hardware.\n\nSHARD turns distributed local AI infrastructure into a four-step workflow: **CREATE → SHARE → CONNECT → USE**.\n\n|             Start a local SHARD node with `shard start` or`python main.py` . The engine probes your CPU, system RAM, and NVIDIA CUDA / Apple MPS GPU via`hardware.py` , loads your chosen GGUF or Hugging Face model, and brings up the FastAPI server and Web UI on`http://localhost:8000` . |             Open a LAN compute pool with `shard lan create --open` (or generate a pairing code via`shard lan create` ). SHARD broadcasts UDP discovery beacons on port`8001` and exposes`/api/lan/*` endpoints so peer machines on your subnet can pool their VRAM and RAM. | \n|             On any secondary computer on the network, run `shard lan join <master-ip> --share` . The worker completes an HTTP handshake (`POST /api/lan/pair` ), launches its local RPC tensor worker, sends periodic health heartbeats, and contributes its GPU/RAM to the cluster. |             Connect coding IDEs, autonomous AI agents, or custom Python apps to `http://<master-ip>:8000/v1/chat/completions` or`/v1/messages` . Master coordinates layer execution across connected nodes and streams tokens back via Server-Sent Events (SSE). | \n\nEvery feature listed below is verified directly against the [SHARD source tree](https://github.com/avelone123123/shard) (`v1.8.1`).\n\n| Category | Capability | Source Module | Description | \n|---|---|---|---|\n| **LAN Cluster & Sharding** | **UDP Auto-Discovery & LAN Pooling** | [`lan.py`](https://github.com/avelone123123/shard/blob/master/lan.py) ,[` docs/LAN_SHARDING_DESIGN.md`](https://github.com/avelone123123/shard/blob/master/docs/LAN_SHARDING_DESIGN.md) | Broadcasts UDP beacons ( `:8001` ), pairs nodes via code or`--open` mode, monitors node heartbeats (`/api/lan/heartbeat` ), and pools VRAM/RAM across machines. | \n| **Inference Backends** | **Dual PyTorch & GGUF (`llama.cpp`) Engine** | [` engine.py`](https://github.com/avelone123123/shard/blob/master/engine.py) ,[` gguf_backend.py`](https://github.com/avelone123123/shard/blob/master/gguf_backend.py) | Runs Hugging Face `transformers` /`safetensors` models on CUDA/CPU/MPS and quantized GGUF weights via managed`llama-server` &`ggml-rpc-server` subprocesses with configurable GPU layer offload (`-ngl` ). | \n| **Hardware Planning** | **Pre-Flight Model Fit Advisor (`shard fit`)** | [` fit.py`](https://github.com/avelone123123/shard/blob/master/fit.py) ,[` hardware.py`](https://github.com/avelone123123/shard/blob/master/hardware.py) | Inspects model architecture parameters ( `config.json` or parameter heuristics) against host VRAM and RAM before downloading, returning`GPU` ,`HYBRID` , or`IMPOSSIBLE` verdicts, max context length, estimated tok/s, and quantization advice. | \n| **Native Acceleration** | **C++20 & MASM AVX2 SIMD Kernels** | [`native/shard_core.cpp`](https://github.com/avelone123123/shard/blob/master/native/shard_core.cpp) ,[` native/shard_asm.asm`](https://github.com/avelone123123/shard/blob/master/native/shard_asm.asm) ,[` shard_bridge.py`](https://github.com/avelone123123/shard/blob/master/shard_bridge.py) | Custom C++20 and x86-64 AVX2 assembly routines for `RMSNorm` ,`LayerNorm` , Rotary Position Embeddings (`RoPE` ),`Softmax` ,`GELU` ,`SiLU` , and`MatMul` loaded via`ctypes` . | \n| **API & Streaming** | **OpenAI & Anthropic Compatible Server** | [`server.py`](https://github.com/avelone123123/shard/blob/master/server.py) | Serves `/v1/chat/completions` ,`/v1/completions` ,`/v1/models` ,`/v1/messages` (Anthropic protocol),`/api/generate` , and`/api/generate/stream` with real-time Server-Sent Events (SSE) and optional API key auth (`shard key` ). | \n| **Web Control Panel** | **Zero-Dependency Dashboard & Playground** | [`dashboard.html`](https://github.com/avelone123123/shard/blob/master/dashboard.html) | Built-in browser UI at `http://localhost:8000` with live GPU/VRAM/RAM telemetry, Hugging Face model search & download, quantization selector, streaming chat playground, LAN node manager, and one-click IDE integration snippets. | \n| **Containerization** | **Docker & NVIDIA GPU Compose Stack** | [`Dockerfile`](https://github.com/avelone123123/shard/blob/master/Dockerfile) ,[` docker-compose.yml`](https://github.com/avelone123123/shard/blob/master/docker-compose.yml) | Production-ready container setup with persistent named volumes ( `shard_models_volume` ,`shard_data_volume` ) and NVIDIA GPU reservation. | \n| **Collaborative Training** | **Optional SOUP Fine-Tuning Ecosystem** | [`soup_adapter.py`](https://github.com/avelone123123/shard/blob/master/shard_engine/soup_adapter.py) ,[`feedback.py`](https://github.com/avelone123123/shard/blob/master/shard_engine/feedback.py) ,[`training.py`](https://github.com/avelone123123/shard/blob/master/shard_engine/training.py) ,[`docs/COLLABORATIVE_TRAINING.md`](https://github.com/avelone123123/shard/blob/master/docs/COLLABORATIVE_TRAINING.md) | Privacy-first consent feedback, secret/PII redaction, immutable SFT/DPO dataset snapshots, dynamic SOUP training orchestration with cancellation, Baseline vs Candidate eval, model versioning, deployment, and 1-click rollback. | \n| **Unified Hosting & Privacy** | **Model-First Hosting & Private Sessions** | [`server.py`](https://github.com/avelone123123/shard/blob/master/shard_engine/server.py) ,[`lan.py`](https://github.com/avelone123123/shard/blob/master/shard_engine/lan.py) | Enforces loaded model verification before server creation/publishing (HTTP 409 `MODEL_NOT_READY` ), guarantees zero-leakage participant-isolated private chats, and enforces compute worker VRAM/CPU quotas. | \n| **Global Network & Discovery** | **Automatic Network & 1-Click Clusters** | [`net_strategy.py`](https://github.com/avelone123123/shard/blob/master/shard_engine/net_strategy.py) ,[`net_discovery.py`](https://github.com/avelone123123/shard/blob/master/shard_engine/net_discovery.py) ,[`cluster.py`](https://github.com/avelone123123/shard/blob/master/shard_engine/distributed/cluster.py) | Automatic network initialization on `shard start` , public Server Catalog with search and VRAM filters, STUN/NAT traversal, blind encrypted WebSocket relay, pre-join hardware probing, explicit consent enforcement, and distributed tensor pipelines. | \n\n- **Multi-Host `llama.cpp` RPC Tensor Distribution (`gguf_backend.py`, `lan.py`):** Distributed layer execution over TCP/RPC across heterogeneous LAN GPUs works best on low-latency wired Gigabit/2.5GbE Ethernet subnets; Wi-Fi links may experience higher inter-token latency.\n- **Native Win32 Desktop GUI (`native/shard_gui.cpp`):** Native C++ Windows desktop wrapper alongside the cross-platform web dashboard (`dashboard.html` ).\n\n- **Automated Dynamic Re-Sharding on Node Dropout:** Seamless mid-generation layer migration if a worker node disconnects unexpectedly (`docs/LAN_SHARDING_DESIGN.md` ).\n- **Linux / macOS ARM64 NEON Intrinsics:** Expanding the native SIMD kernel suite (`native/` ) beyond x86-64 AVX2/FMA to native Apple Silicon / ARM64 assembly paths.\n\n- **OS:** Windows 10/11 (64-bit), Linux (Ubuntu 20.04+ / Debian 12+), or macOS.\n- **Python:** Python`3.8+` (`3.10+` recommended).\n- **Hardware:** NVIDIA GPU with CUDA drivers (recommended for GPU acceleration), Apple Silicon (MPS), or x86-64 CPU with AVX2 support.\n- **Network (Optional for LAN Sharding):** Two or more machines on the same LAN with TCP port`8000` (HTTP/API) and UDP/TCP port`8001` (LAN discovery & RPC) open.\n\nInstall SHARD directly from PyPI in any environment:\n\n```\n# Core install: CLI, FastAPI server, web dashboard, GGUF (llama.cpp) + LAN clustering\npip install shard-engine\n\n# Full install: adds PyTorch/Transformers safetensors & Hugging Face extras\npip install \"shard-engine[full]\"\n```\n\nTo install the latest development snapshot directly from GitHub:\n\n```\n# Core install from GitHub\npip install \"git+https://github.com/avelone123123/shard.git\"\n\n# Full install from GitHub\npip install \"shard-engine[full] @ git+https://github.com/avelone123123/shard.git\"\n```\n\nVerify your environment right after installation:\n\n```\nshard doctor\n```\n\n**Windows PATH Note:** If Windows reports that `shard.exe` is not on your PATH, you can run SHARD directly through Python: `python -m shard_engine <command>` (for example, `python -m shard_engine help` or `python -m shard_engine start`).\n**GPU & Runtime Note:** GGUF inference auto-downloads platform binaries for `llama-server` / `ggml-rpc-server` on first run and needs no extra ML packages. For CUDA PyTorch inference, install a matching CUDA build of `torch` before `pip install \"shard-engine[full]\"`.\n\n```\ngit clone https://github.com/avelone123123/shard.git\ncd shard\npip install -e \".[dev]\"\ngit clone https://github.com/avelone123123/shard.git\ncd shard\ndocker-compose up --build\n```\n\nThis builds `shard-engine`, mounts persistent volumes (`/data/models` and `/data/shard`), exposes port `8000:8000`, and enables NVIDIA GPU passthrough.\n\nTo compile `shard_core.dll` (Windows MSVC `cl.exe` + `ml64.exe`) or `shard_core.so` (Linux GCC/Clang):\n\n```\npython native/build_native.py\nshard update --check\npip install --upgrade shard-engine\npip uninstall shard-engine\n```\n\nUninstalling the Python package keeps your downloaded models, settings, and API keys intact in your user data directory.\n\n```\n# Check Python runtime, dependencies, GPU VRAM, ports, and writable paths\nshard doctor\n\n# Start FastAPI server in background (--daemon / -d) and inspect logs/status\nshard start --daemon\nshard status\n\n# Or open the interactive Web Dashboard at http://localhost:8000 immediately\nshard gui\n\n# Stop or restart background server cleanly\nshard restart\nshard stop\n# Check if a model fits your local machine or LAN cluster before downloading\nshard fit Qwen/Qwen2.5-Coder-7B-Instruct\nshard fit meta-llama/Llama-3.1-8B-Instruct\n# Download and start an interactive terminal chat (default: Qwen/Qwen2.5-Coder-1.5B-Instruct)\nshard run Qwen/Qwen2.5-Coder-7B-Instruct\n\n# List cached and catalog models\nshard list\n```\n\nOn the **Master Machine** (`shard lan create` prints the live LAN IPv4 addresses detected on this host):\n\n```\n# Create an open LAN pool (auto-accepts subnet workers and sets target model)\nshard lan create --open --share --model Qwen/Qwen2.5-Coder-7B-Instruct\n\n# Or create a code-protected LAN pool\nshard lan create --share\n```\n\nOn each **Worker Machine** on the same subnet, replace `<SERVER_IP>` with the IP printed by `shard lan create` on the Master Machine:\n\n```\n# Join an open LAN pool and contribute local GPU/RAM\nshard lan join <SERVER_IP> --share\n\n# Or join a code-protected pool\nshard lan join <SERVER_IP> <PAIR_CODE> --share\n\n# Inspect cluster nodes, host IPv4 addresses, and pooled VRAM/RAM\nshard lan status\n\n# Leave or disable LAN pooling\nshard lan off\n```\n\n| Command | Arguments & Flags | Description | \n|---|---|---|\n| `shard start` | `[--host HOST] [--port PORT] [-d\\|--daemon] [--no-browser]` | Launches the SHARD FastAPI server and Uvicorn runtime (default `0.0.0.0:8000` ). | \n| `shard stop` | *(none)* | Safely stops the background SHARD server managed by PID tracking. | \n| `shard restart` | `[--host HOST] [--port PORT] [--foreground] [--no-browser]` | Stops and restarts the SHARD server cleanly. | \n| `shard gui` | *(none)* | Launches the server (or native Win32 desktop UI if present) and opens the Web Dashboard. | \n| `shard status` | *(none)* | Displays detected CPU, RAM, GPU VRAM, active model, and detected LAN IPv4 addresses. | \n| `shard doctor` | *(none)* | Audits Python, packaged assets, dependencies, CUDA/GPU, LAN IPs, ports, and user directories. | \n| `shard version` | `[-V\\|--version]` | Prints installed SHARD version, Python executable path, OS info, and active directories. | \n| `shard update` | `[--check]` | Checks PyPI for newer `shard-engine` releases or upgrades non-destructively via`pip` . | \n| `shard fit` | `<model_id> [--ctx N] [--no-fetch]` | Calculates VRAM/RAM requirements, quantization fit, and estimated throughput before downloading. | \n| `shard run` | `[model_id]` | Downloads (if needed), loads the model, and opens interactive terminal chat REPL. | \n| `shard list` | `[--category CAT]` | Lists cached and curated catalog models ( `all` ,`ideal` ,`coding` ,`reasoning` ,`compact` ). | \n| `shard lan` | `[status\\|create\\|join\\|model\\|off] [extra] [--share] [--open] [--model ID] [--port N]` | Creates, joins, configures, inspects, or stops distributed LAN compute pooling across machines. | \n| `shard network` | `[status\\|doctor\\|radmin [status\\|install\\|repair\\|gui\\|decline]]` | Inspects multi-tier autonomous network, runs network doctor, and manages automated Radmin VPN fallback. | \n| `shard training` | `[status\\|doctor\\|privacy\\|feedback\\|dataset\\|start\\|logs\\|cancel\\|resume\\|models\\|eval\\|prepare\\|approve\\|deploy\\|rollback]` | Manages consent feedback, dataset snapshots, SOUP fine-tuning jobs, evaluation, model deployment, and rollback ( [Guide](https://github.com/avelone123123/shard/blob/master/docs/COLLABORATIVE_TRAINING.md) ). | \n| `shard key` | `[--gen]` | Views the current local API key or generates a new token ( `--gen` ). | \n| `shard models-dir` | `[path \\| reset]` | Views, updates, or resets the directory where model weights are stored. | \n| `shard test` | `[--strict-native]` | Verifies tensor math kernels and AVX2/pure-Python fallback throughput. | \n| `shard help` | `[-h\\|--help]` | Prints full CLI command usage and examples. | \n\nSHARD stores user data outside `site-packages` following OS conventions (and reads existing `~/.shard` data cleanly on upgrade):\n\n- **Windows:**`%LOCALAPPDATA%\\SHARD` (config,`keys_db.json` ,`logs/` ,`run/` ,`models/` )\n- **Linux:**`~/.local/share/shard` ,`~/.config/shard` ,`~/.cache/shard` ,`~/.local/state/shard` (respects`XDG_*` variables)\n- **macOS:**`~/Library/Application Support/SHARD` ,`~/Library/Caches/SHARD` ,`~/Library/Logs/SHARD`\n\n| Variable / File | Default | Purpose | \n|---|---|---|\n| `SHARD_HOST` | `0.0.0.0` | Network interface IP address bound by `server.py` . | \n| `SHARD_PORT` | `8000` | HTTP port for the API server and Web Dashboard. | \n| `SHARD_DATA_DIR` | OS data directory ( `/data/shard` in Docker) | Overrides root directory for `settings.json` ,`keys_db.json` ,`lan.json` , logs, and PID metadata. | \n| `SHARD_MODELS_DIR` | `<data_dir>/models` (`/data/models` in Docker) | Storage directory for downloaded Hugging Face SafeTensors and GGUF weights. | \n| `SHARD_NO_BROWSER` /`SHARD_DOCKER` | *(unset)* | Suppresses auto-opening a browser tab when starting SHARD. | \n| `SHARD_NATIVE_LIB` | *(auto-discovered)* | Path override to `shard_core.dll` /`.so` /`.dylib` (set`none` to force Python fallback). | \n| `SHARD_SOUP_PYTHON` /`SHARD_SOUP_CMD` | *(auto-discovered)* | Optional path to a Python 3.11–3.13 environment or binary running [`soup-cli`](https://github.com/MakazhanAlpamys/Soup) when training on Python 3.14+ hosts. | \n| `HF_HOME` /`TRANSFORMERS_CACHE` | `~/.cache/huggingface` | Standard Hugging Face cache override respected by `engine.py` and`hf_hub.py` . | \n| `engine_optimizations.json` | Packaged inside `shard_engine` | Runtime optimization profile loaded via `importlib.resources` . | \n\n**SHARD Ecosystem Topology:** Coding IDEs, agent frameworks, master coordinator, and distributed LAN worker nodes operating as a unified inference fabric.\n\n```\nflowchart TB\n    subgraph Clients[\"AI Agents, IDEs & SDK Clients\"]\n        IDE[\"Cursor / Claude Code / Aider / Continue / Zed\"]\n        SDK[\"OpenAI SDK / Anthropic SDK / LangChain / LlamaIndex\"]\n        Browser[\"Web Dashboard UI (dashboard.html)\"]\n    end\n\n    subgraph Master[\"SHARD Master Node (FastAPI :8000)\"]\n        Server[\"server.py\\n(/v1/chat/completions, /v1/messages, /api/*)\"]\n        Fit[\"fit.py & hardware.py\\n(Pre-Flight VRAM/RAM Advisor)\"]\n        Engine[\"engine.py\\n(PyTorch Transformers & Token Streamer)\"]\n        GGUF[\"gguf_backend.py\\n(llama-server & Layer Offload -ngl)\"]\n        Bridge[\"shard_bridge.py\\n(ctypes C++20 / MASM AVX2 SIMD Core)\"]\n        LANCoord[\"lan.py\\n(UDP Discovery :8001 & Cluster Coordinator)\"]\n    end\n\n    subgraph Workers[\"Distributed LAN Worker Nodes (:8001)\"]\n        W1[\"Worker Node 1\\nggml-rpc-server / GPU VRAM Shard\"]\n        W2[\"Worker Node 2\\nggml-rpc-server / GPU VRAM Shard\"]\n        W3[\"Worker Node N\\nggml-rpc-server / System RAM + GPU\"]\n    end\n\n    IDE -->|HTTP / SSE :8000| Server\n    SDK -->|HTTP / SSE :8000| Server\n    Browser -->|REST / SSE :8000| Server\n\n    Server --> Fit\n    Server --> Engine\n    Server --> LANCoord\n    Engine --> GGUF\n    Engine --> Bridge\n\n    LANCoord <-->|UDP Broadcast & Heartbeat :8001| W1\n    LANCoord <-->|UDP Broadcast & Heartbeat :8001| W2\n    LANCoord <-->|UDP Broadcast & Heartbeat :8001| W3\n\n    GGUF <-->|TCP Tensor RPC Stream| W1\n    GGUF <-->|TCP Tensor RPC Stream| W2\n    GGUF <-->|TCP Tensor RPC Stream| W3\nshard/\n├── assets/\n│   ├── branding/              # Official SHARD SVG & PNG logos, icons, and favicon\n│   ├── animations/            # Isometric pixel-art / voxel animated GIFs\n│   └── screenshots/           # Dashboard and CLI fit advisor visuals\n├── docs/\n│   └── LAN_SHARDING_DESIGN.md # Distributed LAN sharding protocol & architecture spec\n├── native/\n│   ├── build_native.py        # Cross-platform compiler for C++20 & MASM AVX2 kernels\n│   ├── shard_core.cpp         # C++20 RMSNorm, LayerNorm, RoPE, Softmax, GELU, SiLU, MatMul\n│   ├── shard_asm.asm          # x86-64 MASM AVX2 SIMD assembly kernels\n│   └── shard_gui.cpp          # Native Win32 desktop GUI wrapper\n├── shard_engine/                # Installable Python package (paths, process, CLI, API, inference)\n│   ├── paths.py                 # Cross-platform XDG / AppData / macOS paths & packaged assets\n│   ├── process.py               # Detached daemon launcher & verified PID lifecycle management\n│   ├── cli.py                   # `shard` command-line interface entry point (`shard_engine.cli:main`)\n│   ├── main.py                  # Application launcher & Uvicorn bootstrapper\n│   ├── server.py                # FastAPI server (OpenAI, Anthropic, Model & LAN routes)\n│   ├── engine.py                # Core LLM inference orchestrator & SSE generator\n│   ├── gguf_backend.py          # Managed llama.cpp / GGUF server & RPC integration\n│   ├── lan.py                   # Dynamic subnet IP discovery, UDP beacons & LAN pooling\n│   ├── fit.py                   # Pre-flight model memory & throughput calculator\n│   ├── hardware.py              # GPU VRAM (nvidia-smi), CPU & RAM probing\n│   ├── hf_hub.py                # Hugging Face Hub search, metadata & downloader\n│   ├── shard_bridge.py          # Python ctypes bridge to native shard_core.dll / .so\n│   ├── settings.py              # Persistent configuration manager\n│   └── dashboard.html           # Packaged web control panel & agent config generator\n├── pyproject.toml               # Single-source packaging configuration (version from __init__.py)\n├── tests_shard.py               # Unit, API, fit advisor, LAN sharding & kernel verification suite\n└── tests_cli_pkg.py             # Packaging, OS path, CLI, PID safety & 2-node LAN cluster suite\n```\n\nNavigating to `http://localhost:8000` opens the built-in SHARD Control Panel:\n\n**SHARD Web Dashboard:** Real-time per-node VRAM/RAM gauges, Pre-Flight Model Fit Advisor, one-click IDE/agent config snippets, and streaming playground.\n\n**Pre-Flight Fit Verification:** Evaluate model quantization, context ceiling, and pooled LAN cluster fit before downloading model weights.\n\nPoint any OpenAI- or Anthropic-compatible client to your SHARD instance:\n\n``` python\nfrom openai import OpenAI\n\nclient = OpenAI(\n    base_url=\"http://localhost:8000/v1\",\n    api_key=\"shard-local\",  # Or generate a token via `shard key --gen`\n)\n\nstream = client.chat.completions.create(\n    model=\"Qwen/Qwen2.5-Coder-7B-Instruct\",\n    messages=[{\"role\": \"user\", \"content\": \"Explain tensor parallelism across LAN nodes.\"}],\n    stream=True,\n)\n\nfor chunk in stream:\n    if chunk.choices[0].delta.content:\n        print(chunk.choices[0].delta.content, end=\"\", flush=True)\n```\n\nSHARD supports collaborative model-parallel execution across independent machines (LAN, Radmin VPN, Tailscale, WireGuard, or NAT-traversed encrypted tunnels). A coordinator can schedule a large model across multiple consented worker nodes and expose one unified OpenAI/Anthropic endpoint (`/v1/chat/completions`, `/v1/messages`).\n\n1. \n**Coordinator (Create & Publish Global Server):**\n\n```\nshard start --daemon\nshard cluster create Qwen/Qwen2.5-Coder-7B-Instruct --name MyGlobalCluster --strategy auto --publish --slots 4\nshard cluster status\n```\n\n2. \n**Discover Global Servers & 1-Click Join:**\n\n```\n# Browse public catalog\nshard cluster catalog --q deepseek --min-vram 16\n\n# Join as an AI Consumer (Client):\nshard cluster join shn_coordinator_node_id --client\n\n# Join as a Compute Worker (Explicit Consent & Quota):\nshard cluster join shn_coordinator_node_id --consent --vram-pct 70 --max-ram-gb 8 --max-threads 4\n```\n\n For workers behind restrictive CGNAT/NAT, SHARD automatically traverses NAT using STUN or establishes an end-to-end ChaCha20-Poly1305 encrypted blind WebSocket relay tunnel with zero configuration.\n3. \n**Deploy & Monitor:**\n\n```\nshard cluster plan\nshard cluster deploy\nshard cluster status\n```\n\n- **GGUF Distributed RPC (`gguf_rpc_layer_split`)** : Splits model layers and KV cache across` ggml-rpc-server` workers and local offload devices via`llama-server --rpc` , with continuous batching (`--parallel` ,`--cont-batching` ).\n- **PyTorch Pipeline Parallelism (`torch_pipeline_parallel`)** : Directly loads only assigned safetensors layer shards per worker (`llama` ,`qwen2` ,`qwen3` ,`mistral` ,`gemma` ,`gemma2` ,`phi3` ,`gpt2` ), maintains per-worker distributed KV caches, and streams binary tensor activations over authenticated`SHPT` TCP sockets.\n- **Autonomous Global Network & Infrastructure Specs** :\n  - Autonomous Architecture: [`AUTONOMOUS_NETWORK_ARCHITECTURE.md`](https://github.com/avelone123123/shard/blob/master/AUTONOMOUS_NETWORK_ARCHITECTURE.md)\n  - Radmin VPN Integration & Fallback: [`RADMIN_VPN_INTEGRATION_REPORT.md`](https://github.com/avelone123123/shard/blob/master/RADMIN_VPN_INTEGRATION_REPORT.md)\n  - Production Deployment & Docker: [`SHARD_PRODUCTION_INFRASTRUCTURE_REPORT.md`](https://github.com/avelone123123/shard/blob/master/SHARD_PRODUCTION_INFRASTRUCTURE_REPORT.md)\n  - Autonomous Installation & OS Workflows: [`AUTONOMOUS_INSTALLATION_REPORT.md`](https://github.com/avelone123123/shard/blob/master/AUTONOMOUS_INSTALLATION_REPORT.md)\n  - Real-World Test Verification Report: [`REAL_WORLD_NETWORK_TEST_REPORT.md`](https://github.com/avelone123123/shard/blob/master/REAL_WORLD_NETWORK_TEST_REPORT.md)\n  - Catalog & Discovery: [`GLOBAL_SERVER_DISCOVERY_REPORT.md`](https://github.com/avelone123123/shard/blob/master/GLOBAL_SERVER_DISCOVERY_REPORT.md)\n  - 1-Click Worker Onboarding: [`AUTOMATIC_WORKER_JOIN_REPORT.md`](https://github.com/avelone123123/shard/blob/master/AUTOMATIC_WORKER_JOIN_REPORT.md)\n- Autonomous Architecture: \n- **Technical Specs** :[`DISTRIBUTED_AI_ARCHITECTURE.docx`](https://github.com/avelone123123/shard/blob/master/docs/reports/DISTRIBUTED_AI_ARCHITECTURE.docx) |[`DISTRIBUTED_AI_IMPLEMENTATION_REPORT.docx`](https://github.com/avelone123123/shard/blob/master/docs/reports/DISTRIBUTED_AI_IMPLEMENTATION_REPORT.docx) |[`DISTRIBUTED_AI_TEST_REPORT.docx`](https://github.com/avelone123123/shard/blob/master/docs/reports/DISTRIBUTED_AI_TEST_REPORT.docx)\n\n- **Runtime & API:** Python 3.8+ / 3.10+,[FastAPI](https://fastapi.tiangolo.com/) ,[Uvicorn](https://www.uvicorn.org/) ,[Pydantic v2](https://docs.pydantic.dev/)\n- **ML & Quantization:**[PyTorch](https://pytorch.org/) , Hugging Face[`transformers`](https://huggingface.co/docs/transformers) ,[` accelerate`](https://huggingface.co/docs/accelerate) ,[` safetensors`](https://huggingface.co/docs/safetensors) ,[` huggingface_hub`](https://huggingface.co/docs/huggingface_hub) ,[` llama.cpp` GGUF](https://github.com/ggml-org/llama.cpp)\n- **Native Acceleration:** C++20 (`native/shard_core.cpp` ) + x86-64 MASM AVX2 Assembly (`native/shard_asm.asm` )\n- **System & Networking:**`psutil` , UDP broadcast discovery (`:8001` ), HTTP/SSE streaming (`:8000` )\n\n- **Architecture & LAN Design Spec:**[`docs/LAN_SHARDING_DESIGN.md`](https://github.com/avelone123123/shard/blob/master/docs/LAN_SHARDING_DESIGN.md)\n- **Contribution Guidelines:**[`CONTRIBUTING.md`](https://github.com/avelone123123/shard/blob/master/CONTRIBUTING.md) — covers zero-copy memory rules, native C++20/AVX2 build steps, and running`shard test` .\n- **Code of Conduct:**[`CODE_OF_CONDUCT.md`](https://github.com/avelone123123/shard/blob/master/CODE_OF_CONDUCT.md)\n- **Release History:**[`CHANGELOG.md`](https://github.com/avelone123123/shard/blob/master/CHANGELOG.md) (current release:`v1.8.1` )\n- **Bug Reports & Feature Requests:**[GitHub Issues](https://github.com/avelone123123/shard/issues)\n- **Community Q&A & Ideas:**[GitHub Discussions](https://github.com/avelone123123/shard/discussions)\n- **Security Policy:** Please report security vulnerabilities privately via[GitHub Security Advisories](https://github.com/avelone123123/shard/security) or by contacting the maintainer before public disclosure.\n- **License:** Distributed under the[MIT License](https://github.com/avelone123123/shard/blob/master/LICENSE) .\n\n**SHARD — One Network. Many Servers. Shared AI Infrastructure.**\n\n<sub>Built for the open-source local AI community · [English](https://github.com/avelone123123/shard/blob/master/README.md) · [Русский](https://github.com/avelone123123/shard/blob/master/README.ru.md) · [Қазақша](https://github.com/avelone123123/shard/blob/master/README.kz.md)</sub>", "url": "https://wpnews.pro/news/show-hn-shard-distributed-ai-inference-across-multiple-computers", "canonical_source": "https://github.com/avelone123123/shard", "published_at": "2026-11-17 23:56:20+00:00", "updated_at": "2026-11-18 00:13:06.732538+00:00", "lang": "en", "topics": ["ai-infrastructure", "large-language-models", "ai-tools", "developer-tools", "ai-agents"], "entities": ["SHARD", "shard-engine", "FastAPI", "PyTorch", "llama.cpp", "Hugging Face", "Cursor", "Claude Code"], "also_reported_by": [], "takeaway": "SHARD splits transformer layers across multiple consumer GPUs and host RAM over gigabit LAN and serves OpenAI- and Anthropic-compatible endpoints on port 8000, letting existing coding agents point at pooled local hardware instead of a cloud API.", "recommendation": null, "alternates": {"html": "https://wpnews.pro/news/show-hn-shard-distributed-ai-inference-across-multiple-computers", "markdown": "https://wpnews.pro/news/show-hn-shard-distributed-ai-inference-across-multiple-computers.md", "text": "https://wpnews.pro/news/show-hn-shard-distributed-ai-inference-across-multiple-computers.txt", "jsonld": "https://wpnews.pro/news/show-hn-shard-distributed-ai-inference-across-multiple-computers.jsonld"}}