{"slug": "show-hn-xet-server-host-your-own-models-anywhere", "title": "Show HN: Xet Server – Host your own models anywhere", "summary": "A developer released Xet Server, a Go server that is wire-compatible with Hugging Face's Xet storage protocol, letting users point the real `hf` CLI (`huggingface_hub` plus `hf_xet`) at their own infrastructure for `hf upload` and `hf download` with byte-identical round-trips. The project includes `xet-proxyd`, a caching pull-through proxy for huggingface.co that keeps any repo or file it has served once available after the Hub becomes unreachable, and its cache directory can be handed to a plain `xetd` as a disconnected replacement. The implementation was verified end-to-end against a live `hf_xet` client with real GGUF model files up to 27.6 GB, using a from-scratch BLAKE3 Merkle hashing port, real xorb/shard binary formats, and an LZ4 decoder and ByteGrouping4 codec.", "body_md": "A Go server that is **wire-compatible** with Hugging Face's real [Xet\nstorage protocol](https://huggingface.co/docs/hub/en/xet/index) - the\ncontent-defined-chunking, dedup-first storage layer that replaces Git LFS\nfor large model files on the Hub. Point the real `hf` CLI\n(`huggingface_hub` + `hf_xet`) at this server and it works: `hf upload`,\n`hf download`, byte-identical round-trips, real BLAKE3 hashing, real\nxorb/shard binary formats, real LZ4/byte-grouping compression.\n\nThis started as a simplified, non-wire-compatible chunking/dedup demo\n(kept in `internal/chunk`, `internal/manifest`, `internal/api`, `cmd/xet`\nfor quick manual testing) and grew into a from-scratch, spec-driven\nreimplementation of the actual protocol - verified end-to-end against a\nlive `hf_xet` client, with several real wire-format quirks discovered only\nby capturing and replaying genuine client traffic. See\n[docs/PROTOCOL.md](https://github.com/guilt/xet-server/blob/main/docs/PROTOCOL.md) for that story.\n\nAlso included: **`xet-proxyd`**, a caching pull-through proxy for the real\nhuggingface.co - not just a performance cache, but an offline-resilience\nlayer. Point it at the real Hub, use it like a normal `HF_ENDPOINT`, and\nevery repo/file it has successfully served once stays servable via `hf download`/` hf upload` even after huggingface.co becomes unreachable -\nincluding handing its cache directory to a plain `xetd` as a permanent,\ndisconnected replacement. See [Caching pull-through proxy](#caching-pull-through-proxy-xet-proxyd)\nbelow.\n\n- **Wire-compatible CAS HTTP API** (`internal/casserver` ): xorb\nupload/fetch, shard upload, file reconstruction with Range-based paging,\nmatching xet-core's own`openapi/cas.openapi.yaml` - verified\nbyte-identical against a real`hf_xet` client, for both compressible and\nincompressible content.\n- **Hub API shim** (`internal/hubserver` ): enough of huggingface.co's Hub\nREST API (repo create, preupload,`xet-{read,write}-token` , commit,\nresolve/HEAD) that the real`hf upload` /`hf download` shell commands work\nagainst this server via`HF_ENDPOINT` - including real, independent\nrevisions/branches per repo, not just an implicit`main` .\n- **`xet-proxyd`: a caching pull-through proxy for the real huggingface.co** (`cmd/xet-proxyd` ,`internal/proxycas` ,`internal/proxyhub` ,`internal/hfclient` ) - an offline-resilience layer, not just a\nperformance cache: any repo metadata or xorb bytes it has successfully\nrelayed once stay servable even after the real huggingface.co becomes\nunreachable, and its cache directory is a real`xetd` data directory a\nplain`xetd` can take over directly, no proxy process required. See[Caching pull-through proxy](#caching-pull-through-proxy-xet-proxyd) .\n- **Real BLAKE3-keyed Merkle hashing** (`internal/merklehash` ): a\nbyte-for-byte port of xet-core's`DataHash` , verified against xet-core's\nown published reference vectors - not an approximation.\n- **Real xorb and shard binary formats** (`internal/xorbformat` ,`internal/shardformat` ), each verified against real bytes captured from\na live`hf_xet` upload, including the footer-less upload behavior real\nclients actually use (see[docs/PROTOCOL.md](https://github.com/guilt/xet-server/blob/main/docs/PROTOCOL.md) ).\n- **From-scratch LZ4 decoder and ByteGrouping4 codec** (`internal/lz4` ,`internal/bg4` ): written directly from the public LZ4 spec / verified\nagainst a third-party reference implementation, so chunk hashes can be\nindependently re-verified regardless of compression scheme.\n- **Pluggable, streaming storage** (`internal/storage` ): a`Store` interface with filesystem (`fsstore` ) and S3-compatible (`s3store` )\nbackends, built on`io.Reader` /`io.ReadCloser` rather than`[]byte` -\nneither backend ever buffers a full object in memory, so a multi-GB\nupload/download costs a fixed amount of memory. The S3 backend uses a\nfrom-scratch AWS SigV4 signer (`internal/sigv4` ), no AWS SDK dependency.\nVerified end-to-end with real GGUF model files up to 27.6 GB.\n- **Zero required external dependencies to build the demo path** ; the\nprotocol path adds exactly one pure-Go module\n(`github.com/zeebo/blake3` ), pinned in`go.sum` .\n- **Optional storage auto-pruning and request rate limiting** (`internal/eviction` ,`internal/ratelimit` ): bounded, observable\ndefense-in-depth for a server left running against untrusted traffic -\nneither adds a dependency, and both are off unless explicitly enabled.`xetd` rate-limits uploads only (the expensive local operation there);`xet-proxyd` rate-limits every route on both ports, since even a read\ncan trigger a real outbound call to the real huggingface.co.\n- **Optional dedup-hit content verification** (`internal/storage` 's`VerifyingStore` ): byte-compares an incoming upload against the stored\nblob on a dedup hit instead of trusting the content hash alone,\nbailing at the first mismatch - a hash-collision/corruption safety net,\noff by default since it roughly doubles I/O on a dedup hit.\n- **A real global chunk-dedup index and V2 (multi-range) reconstruction** (`internal/casserver` ):`GET /v1/chunks/{prefix}/{hash}` returns the\nactual shard bytes referencing a queried chunk (the real wire contract,\nconfirmed against xet-core's own client source), and`GET /v2/reconstructions/{file_id}` returns the multi-range-optimized\nresponse shape instead of always falling back to V1.\n- **Restart-surviving metadata persistence** :`casserver` /`hubserver` (and, identically,`xet-proxyd` 's embedded copies of each) periodically\n(and on graceful shutdown) checkpoint their in-memory\nreconstruction/repo indices to disk as an atomic JSON snapshot - see`-snapshot-interval` below.\n- **Real-client regression fixtures** : several packages carry`testdata/` captured directly from a live`hf_xet` session, replayed in\nunit tests - the strongest guard against silently regressing wire\ncompatibility.\n\nSee [docs/ARCHITECTURE.md](https://github.com/guilt/xet-server/blob/main/docs/ARCHITECTURE.md) for the full system\ndiagram (upload/download sequence diagrams, package responsibility table,\nand the key design decisions). Short version:\n\n- `cmd/xetd` runs the CAS server, and - with`-hub-addr` - the Hub API shim\nas a second HTTP listener, matching how huggingface.co's real Hub and\nCAS are actually separate services.\n- `cmd/xet-proxyd` runs a caching pull-through proxy in front of the real\nhuggingface.co, on the same two-port shape as`xetd` - a drop-in\nalternative for the same ports, backed by the real Hub instead of being\nthe only copy of the data. See[Caching pull-through proxy](#caching-pull-through-proxy-xet-proxyd) .\n- `cmd/xet` is a small CLI for the Xet Data API only (not the\nwire-compatible protocol - use the real`hf` CLI for that).\n\nWith `-hub-addr` set, xetd is also a Git LFS server: set a repo's `lfs.url`\nto `http://<hub>/{owner}/{name}.git/info/lfs` and the stock `git-lfs`\nclient plus `git-xet` push LFS objects into the Xet CAS with chunk-level\ndedup, pull them back over the basic transfer, and use `git lfs lock`.\nWith `-sync-folder` the data directory can be a Dropbox/Syncthing folder\nshared by one xetd per machine, no server anywhere. See\n[docs/GIT_LFS.md](https://github.com/guilt/xet-server/blob/main/docs/GIT_LFS.md).\n\n- Go 1.27 or later\n- GNU Make\n- Bash (for integration tests)\n- (Optional, for the real `hf` CLI round-trip test)`pipenv` , to install`huggingface_hub` +`hf_xet`\n- (Optional) `merman-cli` or`mmdc` (mermaid-cli), to render Mermaid\ndiagrams as SVG in`make docs` /`make docs-serve` - falls back to a\nsource + mermaid.live link if neither is available\n\nIf `pipenv` can't reach `pypi.org` directly (a corporate proxy, an\nair-gapped environment), copy `.env.example` to `.env` and set\n`PIPENV_PYPI_MIRROR` to a reachable index mirror; `.env` is read by\n`make install` and is gitignored.\n\n``` php\nmake pre-check   # verify Go/bash are present\nmake build       # -> bin/xetd, bin/xet-proxyd, bin/xet\n```\n\nRequires Go 1.27 or later. Installs into `$(go env GOPATH)/bin`\n(typically `~/go/bin`; add that directory to your `PATH` if it isn't\nalready):\n\n```\ngo install github.com/guilt/xet-server/cmd/xetd@latest\ngo install github.com/guilt/xet-server/cmd/xet-proxyd@latest\ngo install github.com/guilt/xet-server/cmd/xet@latest\n```\n\nEvery tagged release publishes per-platform zips\n(`xet-server-<version>-<platform>.zip`) from the [GitHub Releases\npage](https://github.com/guilt/xet-server/releases) - each zip bundles all\nthree binaries (`xetd`, `xet-proxyd`, `xet`) plus this README, the\nCHANGELOG, the LICENSE, CONTRIBUTING, the `docs/` directory (both the\nMarkdown sources and the browsable HTML generated by `make docs`,\nMermaid diagrams included), and a `VERSION` file. A `checksums.txt`\n(SHA-256 of every zip) is published alongside them.\n\n```\n# CAS server only, on :8420\n./bin/xetd -addr :8420 -data ./xet-data\n\n# CAS server + Hub API shim (needed for the real hf CLI), on :8420 / :8421\n./bin/xetd -addr :8420 -hub-addr :8421 -data ./xet-data\n\n# With storage auto-pruning (evict least-recently-used xorbs over 10GB,\n# checked every 5 minutes) and per-source-IP upload rate limiting (5\n# requests/sec sustained, burst of 20) - both optional, both off by default:\n./bin/xetd -addr :8420 -data ./xet-data \\\n  -max-storage-bytes 10737418240 -eviction-interval 5m \\\n  -rate-limit-rps 5 -rate-limit-burst 20\n\n# With dedup-hit content verification (see \"Storage backends\" below) and\n# a faster persistence checkpoint interval than the 1-minute default:\n./bin/xetd -addr :8420 -data ./xet-data -verify-dedup -snapshot-interval 15s\n```\n\nOpening either port's root path (`http://localhost:8420/` or, if\n`-hub-addr` is set, `http://localhost:8421/`) in a browser shows a landing\npage listing every endpoint that port serves - useful for orienting\nyourself without needing to read this README first.\n\nBy default `xetd` enforces no authentication at all - any client can read\nand write, matching this project's behavior prior to v0.8.0. Passing\n`-auth-token <secret>` (or, preferably, setting `$XETD_AUTH_TOKEN`) requires\nevery request to carry `Authorization: Bearer <secret>`, split into\nread/write scope per endpoint (uploads/commits need write; downloads/\nreconstructions/resolve need read) the same way real Xet/HF Hub tokens\nwork. `-auth-token None` (the default) or leaving it unset disables\nenforcement entirely.\n\nMinted tokens (the short-lived credentials the Hub shim issues to `hf_xet`\nand git-xet, and puts on git-lfs download links) are also **bound to one\nrepository**: the repo type and ID are inside the HMAC-signed payload, so a\ntoken for `alice/one` gets `403` on `bob/two`'s read, write, and token-refresh\nroutes. A repo-scoped write token may create a new repo only inside its own\nnamespace; creating under another owner requires the shared secret. The\nshared secret itself stays unrestricted (it carries the `admin` scope).\n\n```\nexport XETD_AUTH_TOKEN=\"a-long-random-secret\"\n./bin/xetd -addr :8420 -hub-addr :8421 -data ./xet-data\n\n./bin/xet push -server http://localhost:8420 -auth-token \"a-long-random-secret\" ./model.safetensors\n# or, equivalently, export XET_AUTH_TOKEN (or HF_TOKEN, the same variable\n# the real `hf` CLI reads) instead of passing -auth-token:\nexport XET_AUTH_TOKEN=\"a-long-random-secret\"\n./bin/xet push -server http://localhost:8420 ./model.safetensors\n```\n\n**Prefer the environment variable over the flag** ($XETD_AUTH_TOKEN /\n$XET_AUTH_TOKEN) whenever the machine is shared or the command runs where\nits argument list might be logged: a `-auth-token` flag value is visible to\nany other local user via `ps -ef` / `/proc/<pid>/cmdline` and gets written\nto shell history, while an environment variable set through a secrets\nmanager, a `.env` file kept out of history, or `read -s` is not. An\nexplicit `-auth-token` flag always takes precedence over the environment\nvariable if both are set, so scripts that need to override a\nmachine-wide/session-wide token can still do so per invocation.\n\n`-auth-token`/the env vars above configure this project's built-in\n`auth.StaticTokenAuth` (server) and `auth.BearerCredentialHelper` (client)\n\n- a single shared secret. Both sides are defined as small interfaces\n(`auth.Authenticator` /`auth.Principal` server-side,`auth.CredentialHelper` client-side) in`internal/auth` , so implementing\nyour own (e.g. per-user tokens, JWT validation, mTLS) is a matter of\nsatisfying those interfaces and calling`SetAuthenticator` /setting`client.Client.Cred` - no changes to`casserver` ,`hubserver` , or`internal/api` required.\n\n```\nexport HF_ENDPOINT=\"http://localhost:8421\"   # the Hub shim's address\nexport HF_TOKEN=\"anything\"                    # ignored unless xetd was started with -auth-token/$XETD_AUTH_TOKEN - see Authentication above\n\nhf upload myuser/my-model ./model.safetensors model.safetensors\nhf download myuser/my-model model.safetensors --local-dir ./downloaded\ncmp ./model.safetensors ./downloaded/model.safetensors   # byte-identical\n```\n\nYou can also drive `hf_xet`'s low-level `XetSession` Python API directly\nagainst the CAS server's own address (`:8420` above) with no Hub API\ninvolved at all - useful for isolating whether an issue is in the CAS\nprotocol or the Hub shim.\n\nWant to run this as a real, standing mirror rather than a one-off local\ntest? See **[docs/MIRRORING.md](https://github.com/guilt/xet-server/blob/main/docs/MIRRORING.md)** for a full\nstep-by-step guide (securing it, pointing multiple machines at it,\ntroubleshooting).\n\nA simple, non-wire-compatible gear-hash-CDC + JSON-manifest API is also\navailable for quick manual testing, mounted on the same `xetd` process at\n`/v1/upload`, `/v1/files`, `/v1/stats` - sharing the `/v1` namespace with\nthe CAS protocol on a disjoint set of literal paths (`internal/api`'s\nexported `UploadPath`/` FilesPrefix`/` StatsPath` constants are the single\nsource of truth for these, referenced by both `cmd/xetd`'s route table\nand `internal/client`, so the path never drifts between server and\nclient). `-server` (both flags below are optional) falls back to\n`$XET_SERVER`, then `http://localhost:8420`:\n\n```\nexport XET_SERVER=\"http://localhost:8420\"   # optional; -server overrides it per-invocation\n./bin/xet push /path/to/model.safetensors\n# -> file_id:  2d53223aa33715f0eff757537ed9cf8f\n#    chunks:   28 total, 28 new\n#    stored:   2000000 bytes (0.0% deduplicated)\n\n./bin/xet push /path/to/model-v2.safetensors   # a near-duplicate checkpoint\n# -> chunks:   30 total, 4 new\n#    stored:   436017 bytes (81.0% deduplicated)\n\n./bin/xet pull -out ./restored.safetensors 2d53223aa33715f0eff757537ed9cf8f\n./bin/xet stats\n```\n\n**No per-file ownership/isolation.** `file_id` is the SHA-256 of the\nuploaded content - this API has no repo or user concept at all, so `-auth-token`\nhere (if enabled) only gates read/write access to the API as a whole, not\nper-file: any caller with a valid read-scoped token (or none, if auth is\ndisabled) can fetch any `file_id` it knows, the same trust model as CAS's\nown `/v1/xorbs/{hash}`. File IDs are not secrets and this store is not\nmulti-tenant - don't run it multi-tenant without adding that isolation\nyourself first.\n\n`xet-proxyd` sits in front of the **real** huggingface.co and transparently\ncaches everything it relays - not primarily for speed, but for\n**offline resilience**: once it has successfully served a repo's metadata\nor a file's xorb bytes, that data keeps being servable via `hf download`/\n`hf upload` even if huggingface.co goes down, gets rate-limited, or\ndisappears entirely. It's built by embedding the exact same\n`internal/casserver.Server`/` internal/hubserver.Server` engines `xetd`\nuses, rather than reimplementing caching logic separately - so its cache\ndirectory is a real `xetd` data directory a plain `xetd` can take over\ndirectly, with no proxy process running at all (see\n[Offline handoff](#offline-handoff-proving-the-point) below).\n\n```\n# Two ports, same shape as xetd: -addr (CAS-facing), -hub-addr (Hub-facing).\n# Point HF_ENDPOINT at -hub-addr and the real hf CLI works through it\n# end-to-end, same as pointing it at a local xetd.\n./bin/xet-proxyd -addr :8420 -hub-addr :8421 -data ./xet-proxy-data\n\nexport HF_ENDPOINT=\"http://localhost:8421\"\nexport HF_TOKEN=\"anything\"   # forwarded upstream unchanged - see below\nhf download someuser/some-model --local-dir ./downloaded\n```\n\nOr via `make run-proxy` (same as above, with `./xet-proxy-data`).\n\n**Verified end-to-end against the real huggingface.co** (not just this\nproject's own local stand-ins): a real `hf download` of a Xet file through\nthe proxy returns a byte-identical copy (the proxy caches the file's xorb\nbytes via the presigned CDN URLs the real CAS's reconstruction returns),\nand a real `hf upload` of a small (Git LFS mode) and a large (Xet mode)\nfile both succeed through the proxy to the live Hub. Use the `:port` form\nfor `-addr`/`-hub-addr` (`:8420`, not `127.0.0.1:8420`): the default CAS\nURL this proxy hands out to clients is built as `http://localhost<addr>`,\nwhich is correct for `:8420` but would mangle an explicit host. For a\ncustom externally-reachable address, pass `-cas-url http://<host>:<port>`\nexplicitly.\n\nSame as `xetd`, opening either port's root path in a browser shows a\nlanding page (\"Xet Proxy Server\" / \"Xet Proxy Server - Hub API shim\")\nlisting that port's endpoints; the CAS-facing port also serves the same\noffline Swagger UI at `/api-docs/` (identical spec - the wire protocol is\nthe same one `xetd` implements). Note the CAS-facing port can't answer\nanything about a repo it hasn't seen yet until the Hub-facing port relays\nat least one real Hub call - see the \"bootstrap\" note below.\n\n**Every request that can't be served from cache is relayed live to the\nreal huggingface.co** (`-upstream-hub-url`, defaulting to\n`https://huggingface.co`, falling back to `$HF_URL`) - the caller's own\n`Authorization` header is forwarded upstream completely unchanged\n(`internal/hfclient`'s pure-passthrough design: this proxy never holds or\nuses a credential of its own). `-auth-token`/`$XET_PROXYD_AUTH_TOKEN`\ngates access to **this proxy itself** - a separate concern from the\nupstream credential entirely.\n\n**Expired Xet access tokens are healed transparently.** Xet's per-repo\naccess tokens are short-lived, and on a long, high-concurrency download\n(e.g. `hf download` of a 900+-file dataset) a token can expire between\nthe client's refresh and the moment a file's reconstruction is requested\nfrom the real CAS - which answers `401 Unauthorized` for a repo the client\nwas just downloading from successfully. `xet-proxyd` detects that 401 on\nthe CAS-facing port, mints a fresh replacement token from the real Hub\nusing the **same credential the client already presented** (never a secret\nthis proxy holds itself), and retries the request once - so the failure is\ninvisible to the client instead of aborting the download. The replacement\nis scoped to the same repo/ref/read-vs-write kind as the original, so this\ncan only ever restore access the client already had. Relatedly, the\nstale-fallback for cached xet tokens refuses to serve a token whose `exp`\nhas already passed: a transient upstream `xet-{read,write}-token` failure\ncan never hand the client a token that is dead on arrival, it surfaces the\nupstream error instead (a clean, retryable failure).\n\n**Any upstream failure - network error, timeout, 5xx - falls back to\nwhatever is already cached, no matter how old**, rather than erroring;\nthis fallback is the whole reason the proxy exists. `-cache-ttl` (default\n`-1`, meaning \"always try upstream first\") controls how long cached Hub\nmetadata (repo-info, tree, xet-token, resolve) is served without even\nattempting a live refresh; it never affects the CAS byte cache, which is\ncontent-addressed and can't go stale by definition. `-metadata-call-timeout`\n(default `30s`) bounds each individual metadata call so a slow-but-not-dead\nupstream doesn't hold up the fallback path longer than necessary - it\nnever applies to tree listing, which pages internally within one call and\ncan legitimately take longer for a very large repo.\n\n`-no-cache` disables all of this - a pure-relay escape hatch (every\nrequest goes straight to upstream, nothing is read or written locally)\nfor when you specifically want a stateless relay rather than the default\noffline-resilient caching mode.\n\n**Bootstrap note: the CAS-facing port needs the Hub-facing port to have\nrelayed at least one real request first.** Real Xet CAS has no single\nwell-known base URL - each repo's Hub `xet-{read,write}-token` response\nhands back a *per-repo* CAS URL, and the CAS-facing proxy only learns it\nas a side effect of the Hub-facing proxy relaying one of those calls. A\nrequest straight to `-addr` (a raw `curl`, or `hf_xet`'s low-level API\npointed directly at the CAS port) before ANY traffic has gone through\n`-hub-addr` gets `503 upstream CAS URL not yet known`. Always drive\ntraffic through `HF_ENDPOINT` pointed at `-hub-addr` first - a real `hf download`/` hf upload` (or even just one xet-token round-trip) - and the\nCAS port works for the rest of that process's life, including everything\nalready cached from a previous run.\n\nRate limiting (`-rate-limit-rps`/`-rate-limit-burst`, off by default)\ngates **every route on both ports**, not just uploads like `xetd`'s own\n`-rate-limit-rps` - a cache-missing read costs a real outbound call to\nhuggingface.co just as much as a write does, so both need the same\nprotection. One shared limiter covers both ports, so a client can't double\nits effective budget by splitting requests across them.\n\n`-snapshot-interval` (default `1m`) persists both the CAS-facing and\nHub-facing caches to `-data` - using the **exact same filenames** `xetd`\nitself reads and writes (`casserver-snapshot.json`/\n`hubserver-snapshot.json`), which is what makes the handoff below work at\nall.\n\n```\n# 1. Cache something while online:\n./bin/xet-proxyd -addr :8420 -hub-addr :8421 -data ./xet-proxy-data &\nHF_ENDPOINT=http://localhost:8421 hf download someuser/some-model --local-dir ./downloaded\n\n# 2. Stop the proxy (a final snapshot is taken automatically):\nkill %1\n\n# 3. huggingface.co is gone / unreachable / you're fully offline now.\n#    Point a PLAIN xetd at the SAME data directory - no -upstream-hub-url,\n#    no network, nothing but what step 1 already cached:\n./bin/xetd -addr :8420 -hub-addr :8421 -data ./xet-proxy-data\n\n# 4. The exact same download still works, byte-identical, forever:\nHF_ENDPOINT=http://localhost:8421 hf download someuser/some-model --local-dir ./downloaded-again\ncmp -r ./downloaded ./downloaded-again\n```\n\nThis isn't a hypothetical - `integration-tests/xet_proxyd_offline_handoff.sh`\nruns exactly this sequence (against a local stand-in Hub, so it needs no\nnetwork access) as part of `make integration-test`: upload+download\nthrough the proxy via the real `hf` CLI, shut the proxy down, tear down\nits upstream entirely, start a fresh plain `xetd` on the handed-off data\ndirectory, and download again with nothing else running. This is `xet-proxyd`'s\nwhole reason to exist: a practical path off huggingface.co, not just a\nfaster mirror of it.\n\n**Partial xorb slices are cached too.** A file's reconstruction\n`fetch_info` gives the byte ranges *that file* needs, which for a\ndeduplicated repo is frequently only part of a xorb originally written\nfor some other file (e.g. chunks 15-21 of 1042). Those bytes cannot\nreproduce the original xorb's hash, so the proxy does not store them\nunder that hash. Instead it fetches the cited chunk group, repackages\nthose chunks as a new xorb under **its own** content hash, and rewrites\nthe file's terms to reference it. The cached file is complete,\nhash-verifiable, and works offline after the handoff above - including\nheavily deduplicated datasets.\n\nA file still falls back to a live relay if `fetch_info` leaves a gap in\na term's coverage (an incomplete/malformed upstream response): the\ndownload succeeds from the CDN, but that file is not cached. The proxy\nlogs that at `DEBUG` (`proxycas: file not cacheable...`). Non-Xet blobs\nrelayed live are likewise not cached; for a library that includes those,\nsee the download-then-upload approach in\n[MIRRORING.md section 10](https://github.com/guilt/xet-server/blob/main/docs/MIRRORING.md).\n\nThis was verified against the real Hub with a `bigcode/the-stack-v2`\nfile whose reconstruction cites only a 222-chunk slice of a shared\nxorb: online download through the proxy, then a plain `xetd` on the\nsame `-data` directory with no upstream, produced a byte-identical\nSHA-256.\n\nEvery endpoint below is also documented as an OpenAPI 3.0 spec\n(`docs/openapi.yaml` - hand-authored and verified against the actual\nhandler source, not generated) and served through a fully offline Swagger\nUI at `/api-docs/` on the CAS server's address:\n\n```\n./bin/xetd -addr :8420 -data ./xet-data\nopen http://localhost:8420/api-docs/   # or just visit it in a browser\n```\n\n\"Fully offline\" means exactly that: the Swagger UI static assets\n(`third_party/swagger-ui-dist`, vendored from the\n[swagger-ui](https://github.com/swagger-api/swagger-ui) project) and the\nspec itself are both compiled directly into the `xetd` binary via Go's\n`embed` package (see `internal/apidocs`) - no CDN dependency, and it works\nidentically on an air-gapped machine.\n\n- `POST /v1/xorbs/{prefix}/{hash}` - upload a serialized xorb (chunk\nheaders + payloads, no footer - see PROTOCOL.md)\n- `GET /v1/xorbs/{prefix}/{hash}` - fetch raw (possibly compressed) xorb\nbytes, honors`Range`\n- `POST /v1/shards` - upload a serialized shard (file/xorb info sections,\nno footer)\n- `GET /v1/reconstructions/{file_id}` - file -> xorb/chunk-range map,\nhonors`Range` , returns`416` at EOF\n- `GET /v1/chunks/{prefix}/{hash}` - global chunk-dedup lookup: returns\nthe raw bytes of whichever uploaded shard referenced this chunk hash\n(the real wire contract - a client parses the shard itself),`404` if\nno uploaded shard has ever referenced it\n- `GET /v2/reconstructions/{file_id}` - multi-range-optimized\nreconstruction: same underlying terms/byte-ranges as V1, grouped by\nxorb (one signed URL covering multiple ranges) instead of one entry per\nterm\n- `POST /v1/telemetry` - no-op ack\n- `GET /v1/storage-stats` - eviction policy stats (operator-facing; not\npart of the real Xet CAS API - see[Storage backends](#storage-backends) )\n\n- `POST /api/repos/create`\n- `GET /api/{repo_type}s/{repo_id}/revision/{revision}` - repo info at a\nrevision (huggingface_hub's`snapshot_download` resolves this before a\nwhole-repo`hf download` )\n- `GET /api/{repo_type}s/{repo_id}/tree/{revision}` - list every file\ncommitted to a revision\n- `POST /api/{repo_type}s/{repo_id}/branch/{branch}` - create a branch\n- `POST /api/{repo_type}s/{repo_id}/preupload/{revision}`\n- `GET /api/{repo_type}s/{repo_id}/xet-{read,write}-token/{revision}`\n- `POST /api/{repo_type}s/{repo_id}/commit/{revision}`\n- `HEAD` /`GET /{repo_id}/resolve/{revision}/{filename}`\n\n- `POST /v1/upload?name=<optional>` - chunks + dedups the uploaded file\n- `GET /v1/files/{id}` /`GET /v1/files/{id}/manifest` /`GET /v1/stats`\n\n`internal/storage.Store` is the abstraction both `casserver` and the Xet\nData API store chunk/xorb bytes through:\n\n- **`fsstore`** - content-addressed filesystem directory (the default)\n- **`s3store`** - any S3-compatible endpoint (AWS S3, MinIO), signed with\nthe from-scratch`internal/sigv4` signer. Set`XET_S3STORE_LIVE_TEST=1` plus`XET_TEST_S3_*` env vars to run its tests against a real MinIO\ninstance.\n\nAll off/on-defaults below; opt in or tune via `xetd` flags:\n\n- `-max-storage-bytes N -eviction-interval 5m` - once total xorb storage\nexceeds`N` bytes, a background sweep (`internal/eviction` ) deletes\nleast-recently-accessed xorbs (upload or fetch both count as access)\nuntil back under budget, skipping any xorb with a fetch currently in\nprogress.`GET /v1/storage-stats` reports the configured budget and\ncumulative evictions/bytes freed. A client that later needs an evicted\nxorb must re-upload it - real Xet clients already treat CAS storage as\nnon-permanent and handle this by re-deriving from the source file. Off\nby default.\n- `-rate-limit-rps N -rate-limit-burst N` - caps xorb/shard uploads per\nsource IP via a hand-rolled token bucket (`internal/ratelimit` , no new\ndependency). Exceeding the limit returns`429` with`Retry-After` .\nFetch/reconstruction/HEAD traffic is never rate-limited. Off by default.\n- `-verify-dedup` - on every dedup hit (a`Put` for a content hash that\nalready exists), byte-compare the incoming upload against the stored\nblob instead of trusting the content hash alone, bailing at the first\nmismatched byte rather than reading either side in full\n(`internal/storage.VerifyingStore` ). A mismatch (a hash collision or\nundetected storage corruption) refuses the write - the original stored\nblob is never overwritten - and logs at`Error` , distinct from routine\nrequest-failure logging. Roughly 10x slower than the default trust-the-hash\npath on a dedup hit (a full extra read), which is why it's opt-in, not\nthe default. Off by default.\n- `-snapshot-interval 1m` - how often`casserver` /`hubserver` 's in-memory\nreconstruction/repo indices are checkpointed to`-data` as JSON (an\natomic stage-then-rename write, the same pattern the storage backends\nuse), so they survive a restart. A graceful shutdown (`Ctrl-C` /`SIGTERM` )\nalways takes one final snapshot first. This is a periodic checkpoint,\nnot a write-ahead log - anything written between two checkpoints is\nlost on an*un* graceful termination (a crash,`kill -9` , power loss);\nsee[docs/PROTOCOL.md](https://github.com/guilt/xet-server/blob/main/docs/PROTOCOL.md) 's persistence section for the\nfull tradeoff writeup. Set to`0` to disable periodic snapshotting\n(a final snapshot is still taken on graceful shutdown). Defaults to 1\nminute.\n\nThe two binaries play different roles, and each has its own tool for a store that must not grow without bound:\n\n- \n**`xet-proxyd` is a cache** in front of huggingface.co. Every xorb it\nholds can be fetched again, so deleting the least-recently-used ones\nis always safe; that is what`-max-storage-bytes` eviction is for.\n- \n**`xetd` is the only copy** of what was pushed to it. Eviction there\ndeletes xorbs that files still reference (leave`-max-storage-bytes` off), and the server cannot tell on its own which files are still\nwanted: a file pushed through git-lfs is live as long as some git ref\nreaches its pointer, and only git knows that. So the operator says\nwhat to keep and`xetd gc` deletes the rest:\n\n```\n# on the git host, for every repo the server backs\ngit -C /srv/git/team/game.git lfs ls-files --all --long > /tmp/keep.txt\n# against the running server, with the shared secret\nxetd gc -server http://xet:8420 -auth-token \"$XETD_AUTH_TOKEN\" -keep /tmp/keep.txt -dry-run\nxetd gc -server http://xet:8420 -auth-token \"$XETD_AUTH_TOKEN\" -keep /tmp/keep.txt\n```\n\n Files committed through the Hub shim ( `hf upload` ) are kept without\nbeing listed; the shim's registry is the record that they are wanted.\nEverything else is dropped from the index, the affected shards are\nrewritten, unreferenced xorbs are deleted from the store, and a\nsnapshot is taken. The report says what happened (`-json` for the raw\nform). Only the shared secret may run it (`POST /v1/gc` needs the`admin` scope, which no minted token carries); with no auth\nconfigured anyone can, like everything else.**The grace period is the part to understand before lowering it.** A\nxet client keeps the shards of its own uploads for three weeks and\ndeduplicates against them without asking the server, and it caches\nevery dedup answer the server gave it for the same three weeks. A\nshard it uploads later may therefore reference a xorb the server has\nnot heard about since; delete that xorb and the new file cannot be\ndownloaded, and nothing says so at push time.`xetd gc` therefore\nnever deletes a xorb uploaded, fetched or advertised in a dedup\nanswer within the grace (`-gc-grace` on the server,`-grace` per run;\ndefault 22 days), and it keeps every file whose shard arrived within\nthe grace even if the keep list does not name it yet (a push whose\ngit ref has not moved). Growth is bounded by live content plus three\nweeks of churn.`-grace 0` is for a store whose clients you control\n(the integration tests, a single machine whose cache you cleared).Not available with `-sync-folder` : other replicas may still be\nadvertising or indexing what one replica deletes.\n\n```\nmake test               # unit tests, all packages\nmake test-race          # unit tests under the race detector\nmake fuzz-seeds         # replay every committed fuzz corpus (fast, deterministic)\nmake test-fuzz          # fuzz every target, 30s each (override: FUZZTIME=5m)\nmake integration-test   # bash integration suite against a live server\n```\n\n`make test-fuzz` depends on `fuzz-seeds`, so every previously-found crash\n(committed under `testdata/fuzz/`) is replayed before any time is spent\nhunting for new inputs - a regression shows up immediately rather than\nafter `FUZZTIME` per target. `-race` and `-fuzz` are separate targets on\npurpose: the race detector costs roughly an order of magnitude in\nexecution speed, so combining them within a fixed time budget explores far\nfewer inputs. The race detector also requires a 64-bit toolchain (a 32-bit\nGo/MinGW setup fails at startup with `exit status 0xc0000139`), which is\nwhy CI runs it on 64-bit Linux.\n\n`integrationTests.sh` starts one `xetd` instance (CAS + Hub shim) and runs\nevery script in `integration-tests/`, with a per-test timeout so a hang\ndoesn't block the suite. `integration-tests/hf_cli_roundtrip.sh` drives the\n**real, unmodified `hf` CLI** through a full upload+download round-trip -\nthe strongest compatibility check available - and skips cleanly (not a\nfailure) if `pipenv`/its environment aren't set up. `integration-tests/xet_proxyd_offline_handoff.sh`\nproves `xet-proxyd`'s whole reason to exist the same way: real `hf`\nupload+download through a running proxy, then the proxy AND its upstream\nare both torn down entirely and a plain `xetd` takes over the proxy's data\ndirectory directly - the same download must still succeed, byte-identical,\nwith nothing but that directory:\n\n```\nmake install             # pipenv --python 3.14 && pipenv install\nmake integration-test    # now includes hf_cli_roundtrip.sh, xet_proxyd_offline_handoff.sh, multi_client_*.sh\n```\n\n`integration-tests/multi_client_*.sh` run that same real `hf` CLI as\nseveral **machines** - each with its own `HF_HOME`, `HF_XET_CACHE` and\ntoken - against one `xetd`. A xet client dedups against its own shard\ncache before it asks the server, so a single cache never parses the\nserver's global-dedup answer; only these tests do. They assert the\ncross-client path from the server's DEBUG log and the client's log (a\ndedup query answered 200 and parsed as `found`, reconstructions served to\na machine that never uploaded), so they cannot pass by uploading\neverything twice.\n\nSee [CONTRIBUTING.md](https://github.com/guilt/xet-server/blob/main/CONTRIBUTING.md#2-running-tests) for the full test\nlayer breakdown, including why several packages carry real-client\n`testdata/` fixtures, native Go fuzz tests for every wire-format parser,\nadversarial/chaos tests against a live server, and dedup/hashing/\ncompression benchmarks.\n\n- **[docs/FAQ.md](https://github.com/guilt/xet-server/blob/main/docs/FAQ.md)** - why Xet's chunking/dedup model exists\nat all instead of plain S3/HTTP hosting, why this project exists, why`xet-proxyd` exists, and other questions worth answering once instead\nof repeatedly\n- **[docs/MIRRORING.md](https://github.com/guilt/xet-server/blob/main/docs/MIRRORING.md)** - practical guide to\nself-hosting your own model mirror:`hf download` /`hf upload` against\nyour own server instead of huggingface.co, step by step, including`xet-proxyd` 's transparent-caching alternative\n- **[docs/ARCHITECTURE.md](https://github.com/guilt/xet-server/blob/main/docs/ARCHITECTURE.md)** - system diagram,\nupload/download sequence diagrams, package responsibility table, design\ndecisions\n- **[docs/PROTOCOL.md](https://github.com/guilt/xet-server/blob/main/docs/PROTOCOL.md)** - wire-compatibility deep dive:\nevery place the real client's behavior diverges from the documented spec,\nhow each was discovered, and why the fix is correct\n- **[CONTRIBUTING.md](https://github.com/guilt/xet-server/blob/main/CONTRIBUTING.md)** - dev setup, test layers, doc-comment\nconventions, branching/PR conventions\n- **[CHANGELOG.md](https://github.com/guilt/xet-server/blob/main/CHANGELOG.md)** - release history\n\nPackage-level godoc comments are the source of truth for implementation details not covered above:\n\n```\ngo doc ./internal/merklehash\ngo doc ./internal/casserver\n```\n\n`make docs` regenerates the committed `docs/godoc/*.md` package reference\n(via `go doc -all`) and renders every Markdown doc in this repo to\nbrowsable HTML in `docs/build/`; `make docs-serve` does the same and then\nserves it locally. Both are implemented by `scripts/build_docs.go` (its\nown Go module, so the main `xet-server` module keeps zero external\ndependencies).\n\nEvery divergence tracked in prior versions of this doc has been closed:\nreal revisions/branches, V2 reconstruction, a global chunk-dedup index,\nrestart-surviving persistence (all as of v0.7.0), pluggable\nauthentication/authorization (as of v0.8.0 - see\n[Authentication](#authentication)), and - as of v1.0.0 - `xet-proxyd`\nworking against the **real** huggingface.co end to end (the v0.9.0 proxy\nwas verified only against this project's own local stand-ins; the\nreal-Hub gaps found - resolve redirects, presigned-URL xorb caching,\nContent-Type headers, raw write-body relay, Git LFS passthrough - are all\nfixed and documented in [CHANGELOG.md](https://github.com/guilt/xet-server/blob/main/CHANGELOG.md)).\n\n`xet-proxyd` has one known, deliberate gap: on a cold\nstart with `-hub-addr` disabled (or before the Hub-facing port has relayed\nany traffic), the CAS-facing port has no way to learn the real upstream\nCAS base URL and returns `503` for anything it doesn't already have\ncached - see the bootstrap note in\n[Caching pull-through proxy](#caching-pull-through-proxy-xet-proxyd).\nThis is inherent to how real Xet CAS's base URL is discovered (per-repo,\nvia the Hub's own token response, not a fixed well-known address), not a\ngap this project could close without deviating from the real protocol.\n\nRun `xetd` with `DEBUG=1` to log every HTTP request it receives (method,\npath, status, duration) plus commit/shard/resolve lookup details at debug\nlevel via the standard library's `log/slog`:\n\n```\nDEBUG=1 ./bin/xetd -addr :8420 -hub-addr :8421 -data ./xet-data\n```\n\nThis is the fastest way to see whether a request from `hf upload`/` hf download` (or anything else) actually reached the server, and what it did\nonce it got there - see [docs/PROTOCOL.md](https://github.com/guilt/xet-server/blob/main/docs/PROTOCOL.md)'s \"How these\nwere found\" section for how this was used to track down real bugs.\n`DEBUG=1 ./bin/xet-proxyd ...` does the same for the proxy - useful for\ntelling whether a slow/failed request actually reached the real\nhuggingface.co or was served from cache. `xet` (the CLI client) has no\n`DEBUG` flag: it's a one-shot command that already prints its result\ndirectly, not a long-running server with request traffic to log.\n\nIf you're in a sandboxed/corporate network that proxies all outbound\ntraffic (including `localhost`), `hf_xet`'s Rust HTTP client may not\nconsistently honor `NO_PROXY`/` no_proxy` for localhost, and the upload call\nhangs. This is an environment limitation, not a xetd bug - the identical\nupload/download flow works when driven directly against the CAS server via\n`hf_xet`'s low-level Python API. Try setting\n`NO_PROXY=localhost,127.0.0.1` / `no_proxy=localhost,127.0.0.1`, or run\noutside the proxied environment. `integration-tests/hf_cli_roundtrip.sh`\nenforces a timeout so this fails visibly instead of hanging the test suite.\n\nThe runner picks ports `18420`/` 18421` by default to avoid colliding with a\n`make run`-style instance on `8420`/` 8421`. Override with\n`XETD_PORT=<port> XETD_HUB_PORT=<port> make integration-test` if those are\nalso taken.\n\nThe runner uses `$TMPDIR` (or `/tmp` if unset) for all scratch state. Make\nsure your environment allows writes there.\n\nThe Xet Data API's chunker targets an average chunk size of 64 KiB; edits smaller than that still land inside one chunk boundary, and byte-level insertions can shift downstream boundaries until the rolling hash resynchronizes. Expected content-defined-chunking behavior, not a bug.\n\nMIT License - see [LICENSE.md](https://github.com/guilt/xet-server/blob/main/LICENSE.md) for details.\n\nSee [CONTRIBUTING.md](https://github.com/guilt/xet-server/blob/main/CONTRIBUTING.md) for the full guide. Short version:\n\n1. Fork the repository, branch from `main`\n2. `make all` before opening a PR (format, vet, build, unit tests,\nintegration tests)\n3. PR titles follow [Conventional Commits](https://www.conventionalcommits.org/en/v1.0.0/)\n4. If you touch protocol/wire-format behavior, read\n[docs/PROTOCOL.md](https://github.com/guilt/xet-server/blob/main/docs/PROTOCOL.md) first\n\nThis project only exists because of work other people published openly. Thank you to:\n\n- [Hugging Face](https://huggingface.co) and the**Xet** team for\ndesigning and documenting[the Xet protocol](https://huggingface.co/docs/hub/en/xet/index) this project reimplements, and for[xet-core](https://github.com/huggingface/xet-core) (the real Rust\nimplementation this project is wire-compatible with) - the ultimate\nsource of truth every byte here was checked against.\n- [jedisct1](https://github.com/jedisct1) for[zig-xet](https://github.com/jedisct1/zig-xet) , an independent\nthird-party reference implementation used to verify this project's own`ByteGrouping4` codec against a second, unrelated source rather than\ntrusting a single implementation.\n- [zeebo](https://github.com/zeebo) for[`blake3`](https://github.com/zeebo/blake3) , the one external Go\ndependency this project takes on for its main module - a correct,\nwell-tested BLAKE3 implementation is exactly the kind of primitive\nworth depending on rather than reimplementing (see[docs/FAQ.md](https://github.com/guilt/xet-server/blob/main/docs/FAQ.md) for why).\n- The [swagger-ui](https://github.com/swagger-api/swagger-ui) project,\nwhose static assets (vendored under`third_party/swagger-ui-dist` ,\nApache-2.0 - see its own`VENDORED.md` ) power this project's fully\noffline`/api-docs/` UI.\n- [yuin/goldmark](https://github.com/yuin/goldmark) ,[alecthomas/chroma](https://github.com/alecthomas/chroma) , and[abhg/goldmark-mermaid](https://github.com/abhg/goldmark-mermaid) ,\nwhich render this repo's own Markdown docs (`scripts/build_docs.go` ,\nits own module so these never touch the main`xet-server` module's\ndependency graph).\n- The [Git LFS](https://git-lfs.com/) project, for defining the large-file\nGit workflow that Xet's chunking/dedup model improves on - and for\nbeing the widely-understood baseline this README's[FAQ](https://github.com/guilt/xet-server/blob/main/docs/FAQ.md#why-not-just-put-model-files-in-s3-or-any-plain-http-file-host-instead-of-building-all-this) compares against.\n- Everyone who publishes public specs (LZ4, AWS Signature Version 4, OpenAPI 3.0) clearly enough that a from-scratch, independently-verified implementation is actually possible without access to any reference server's source code.", "url": "https://wpnews.pro/news/show-hn-xet-server-host-your-own-models-anywhere", "canonical_source": "https://github.com/guilt/xet-server", "published_at": "2026-10-24 05:13:18+00:00", "updated_at": "2026-10-24 05:41:28.814653+00:00", "lang": "en", "topics": ["ai-infrastructure", "developer-tools", "ai-tools", "large-language-models"], "entities": ["Xet Server", "Hugging Face", "hf_xet", "huggingface_hub", "xet-proxyd", "xetd", "BLAKE3", "Go"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/show-hn-xet-server-host-your-own-models-anywhere", "markdown": "https://wpnews.pro/news/show-hn-xet-server-host-your-own-models-anywhere.md", "text": "https://wpnews.pro/news/show-hn-xet-server-host-your-own-models-anywhere.txt", "jsonld": "https://wpnews.pro/news/show-hn-xet-server-host-your-own-models-anywhere.jsonld"}}