{"slug": "show-hn-scalable-source-control-in-rust-inspired-by-anthropic-s-scion", "title": "Show HN: Scalable source control in Rust, inspired by Anthropic's Scion", "summary": "Adrian Chifor released scn, an open-source source-control system written in Rust and inspired by Anthropic's Scion engineering work, offering Git-style CLI commands plus an HTTP API that lets AI agents commit and push without a local checkout. The system splits work across stateless controllers, SQLite WAL shards, Valkey for shard assignments and cached blobs, and local disk or S3 blob storage that chunks files over 32 MiB into 8 MiB SHA-256-addressed, Zstandard-compressed pieces; Chifor says the design can potentially scale to multi-terabyte monorepos and thousands of concurrent users and agents. Prebuilt binaries ship for Linux amd64, Linux arm64 and macOS arm64, and a Docker Compose deployment starts two controllers behind an Nginx gateway, two shards, persistent Valkey and RustFS with a default 256mb cache limit.", "body_md": "Scalable source control system in Rust, inspired by [Scion](https://www.anthropic.com/engineering/redesigning-source-control). Git-style CLI commands and an HTTP API for agents, with server-side commits, sparse checkouts, and chunked storage for gigabyte-scale binaries and datasets.\n\nIts stateless controllers, SQLite shards, and S3 storage can potentially scale to multi-terabyte monorepos and thousands of concurrent users and agents.\n\n- **Controllers** run commits, history queries, diff, blame, merge, and rebase. Multiple controllers share one repository without local durable state. Each keeps a bounded, disposable cache of reconstructed trees.\n- **Valkey** holds permanent shard assignments,`main` , and tags, plus compressed blobs with a configurable TTL. AOF and RDB persist the index;`volatile-lru` evicts only expiring cache entries. Cache writes skip blobs larger than 1/32 of Valkey’s memory limit.\n- **Shards** store immutable file deltas, commit headers, and verified blob lengths in independent SQLite WAL databases. New paths go to the shard with the fewest assignments. Existing paths never move.\n- **Blob storage** is local disk per shard or a shared S3 bucket. Files over 32 MiB use 8 MiB chunks and a manifest. Chunks are addressed by SHA-256, deduplicated, and Zstandard compressed. New chunks can be distributed across shards.\n\n`commit` creates an immutable changeset on the server. `push` separately advances `main` with compare-and-swap. Agents can do both entirely through the API, without a checkout. See [AGENTS.md](https://github.com/adrianchifor/scn/blob/main/AGENTS.md) for the client and API guide.\n\nUse the release binary for your platform. No Rust toolchain is required.\n\n| Platform | Download | \n|---|---|\n| Linux amd64 | [scn-linux-amd64](https://github.com/adrianchifor/scn/releases/latest/download/scn-linux-amd64) | \n| Linux arm64 | [scn-linux-arm64](https://github.com/adrianchifor/scn/releases/latest/download/scn-linux-arm64) | \n| macOS arm64 | [scn-macos-arm64](https://github.com/adrianchifor/scn/releases/latest/download/scn-macos-arm64) | \n\nInstall your chosen binary, for example on Apple Silicon:\n\n```\nmkdir -p ~/.local/bin\ncurl -fL https://github.com/adrianchifor/scn/releases/latest/download/scn-macos-arm64 -o ~/.local/bin/scn\nchmod +x ~/.local/bin/scn\nexport PATH=\"$HOME/.local/bin:$PATH\"\nscn --version\n```\n\nLinux binaries are statically linked. Checksums are in [SHA256SUMS](https://github.com/adrianchifor/scn/releases/latest/download/SHA256SUMS).\n\nRequires Docker with Compose. From this repository:\n\n```\ndocker compose -f deploy/compose.yaml up --build -d --wait\ndocker compose -f deploy/compose.yaml ps\ncurl --fail http://127.0.0.1:7878/health\n```\n\n[deploy/compose.yaml](https://github.com/adrianchifor/scn/blob/main/deploy/compose.yaml) starts two controllers behind an Nginx gateway, two SQLite shards, persistent Valkey, and [RustFS](https://docs.rustfs.com/en/installation/container/docker), an S3-compatible store. An initialization job creates the `scn` bucket before shards start.\n\n| Endpoint | Purpose | \n|---|---|\n| `http://127.0.0.1:7878` | CLI and controller API, balanced across both controllers | \n| `http://127.0.0.1:9000` | Local S3 API | \n| `http://127.0.0.1:9001` | S3 web console | \n\nLocal S3 credentials default to `SCNLOCAL` / `scn-local-secret-change-me`. Override them with `SCN_S3_ACCESS_KEY` and `SCN_S3_SECRET_KEY` before starting the stack. `SCN_CACHE_MEMORY` sets Valkey's memory limit, default `256mb`. Only the gateway and S3 ports are published, bound to loopback.\n\n`SCN_MAX_REQUESTS` and `SCN_TRANSFER_MEMORY` tune each controller and shard's request capacity and transfer budget in MiB, defaulting to `128` and `256`.\n\nEach shard has its own volume. Valkey and S3 also have persistent volumes. Inspect or stop the stack with:\n\n```\ndocker compose -f deploy/compose.yaml logs -f controller-a shard-a\ndocker compose -f deploy/compose.yaml down\n```\n\n`down` keeps data. `docker compose -f deploy/compose.yaml down -v` deletes the repository, index, and blobs.\n\nAfter installing scn, clone through the gateway or your existing controller:\n\n```\nscn clone http://127.0.0.1:7878 ./work\ncd work\nscn remote -v\nscn remote get-url origin\n```\n\nThe clone saves the endpoint in `.scn/state.json`. Later commands use it automatically. To select another controller for the same repository:\n\n```\nscn remote set-url origin https://scn.example.com\nscn fetch\n```\n\nChanging the endpoint preserves staged changes and working files, and clears the cached `origin/main` value. There is one remote, `origin`, and one shared branch, `main`.\n\n```\nprintf 'hello\\n' > hello.txt\nscn status\nscn add hello.txt\nscn diff --cached\nscn commit -m \"Add greeting\"\nscn push origin main\nscn log -n 5\nscn show HEAD hello.txt\n```\n\n`add` snapshots contents. Later edits stay unstaged. `commit -am \"Update tracked files\"` stages modifications and deletions of tracked files. `diff` shows unstaged tracked changes; `diff --cached` shows staged changes. `.gitignore` and `.scnignore` both work. Set `SCN_AUTHOR` or pass `commit --author` for your identity.\n\nSynchronize with other clients:\n\n```\nscn fetch\nscn log origin/main -n 5\nscn pull --ff-only\n```\n\n`fetch` updates `origin/main` without changing files. `pull` fetches and merges. If histories diverge, use `scn pull --rebase` to replay your commits, then `scn push`. Push works with uncommitted edits; pull and checkout require a clean working copy. File conflicts leave working files unchanged.\n\nFor sparse working copies:\n\n```\nscn clone http://127.0.0.1:7878 ./agent-work --sparse src --sparse Cargo.toml\ncd agent-work\nscn checkout main --sparse docs\nscn checkout main --full\n```\n\nCommits preserve omitted files. Revisions accept `HEAD`, `main`, `origin/main`, tag names, or a full scn ID. `checkout` has a `switch` alias; `push` has a `land` alias. There is no Git wire protocol or offline commit support.\n\n- Use normal `add` ,`commit` , and checkout commands for binaries and datasets. Files over 32 MiB automatically use deduplicated 8 MiB Zstandard chunks with bounded memory.\n- API requests are limited to 32 MiB each. Upload chunks separately, then call `POST /v1/files` ; see[AGENTS.md](https://github.com/adrianchifor/scn/blob/main/AGENTS.md) .\n- Chunked files use binary diff and have no line blame. Git import preserves LFS pointers without fetching external objects.\n\n```\nscn tag v1.0.0\nscn tag -a v1.1.0 HEAD -m \"Release 1.1.0\"\nscn tag\nscn tag -f v1.0.0 HEAD\nscn tag -d v1.0.0\n```\n\nTags default to `HEAD`. Use `-a -m` for annotations, `-f` to replace, and `-d` to delete. Changes are shared immediately without pushing. Use tag names as revisions, or `refs/tags/NAME` to disambiguate. API routes are in [AGENTS.md](https://github.com/adrianchifor/scn/blob/main/AGENTS.md).\n\nInstall scn as above, Git, and `valkey-server`. Run each server command in its own terminal, in this order:\n\n```\nscn cache --dir /tmp/scn-cache --listen 127.0.0.1:6379 --maxmemory 1gb\nscn shard --store /tmp/scn-shard-a --listen 127.0.0.1:7879\nscn shard --store /tmp/scn-shard-b --listen 127.0.0.1:7880\nscn shard controller --listen 127.0.0.1:7878 --cache redis://127.0.0.1:6379/ --namespace demo --shard http://127.0.0.1:7879 --shard http://127.0.0.1:7880\n```\n\nThese commands use local blob storage. A second controller can discover registered shards from Valkey:\n\n```\nscn shard controller --listen 127.0.0.1:7881 --cache redis://127.0.0.1:6379/ --namespace demo --blob-ttl 600\n```\n\nClients can use either controller, or a load balancer in front of them. Register another running shard by passing its URL with `--shard` when starting any controller. All controllers then allocate new paths to it without rebalancing. Shard URLs must remain stable; shard removal is not supported.\n\nFor a single process without Valkey, use `scn serve --store /tmp/scn-store`.\n\nSmall requests and transfers have separate pools of 128 slots. Transfers also share a 256 MiB payload budget: uploads reserve their announced size, while downloads and unknown-length bodies reserve 33 MiB. Requests wait up to one second in bounded queues before HTTP 503.\n\nTune each server with `--max-requests 256 --transfer-memory 1024`. The budget covers payloads; allow memory for compression, trees, and other buffers. Adding controllers and shards adds independent capacity.\n\nThe bucket must exist. Supply AWS credentials through environment variables, an AWS profile, or an instance role. Start each shard with its own SQLite directory and a shared bucket/prefix:\n\n```\nscn shard --store /data/shard-a --listen 0.0.0.0:7879 --s3-bucket my-bucket --s3-region eu-west-1 --s3-prefix my-repo/blobs\n```\n\nFor S3-compatible services, also pass `--s3-endpoint http://s3-host:9000`. The same storage flags work with `serve`. Changing backends does not copy blobs.\n\nUse a separate Valkey namespace, shard directories, and bucket prefix per repository. Back up Valkey with the shard databases and blob store.\n\n`scn cache` defaults to `appendfsync always`; `--appendfsync everysec` trades durability for throughput. Authentication and automatic failover are not built in.\n\nServers log requests, timings, retries, storage operations, commits, and ref updates to stderr. Health probes and storage details use debug level. Logs include request IDs that follow controller-to-shard calls; API responses return them in `x-request-id`.\n\n```\nRUST_LOG=warn,scn=debug scn --log-format json serve --store /tmp/scn-store\nSCN_LOG_FORMAT=json RUST_LOG=warn,scn=debug docker compose -f deploy/compose.yaml up -d\nscn --no-progress pull\n```\n\nText logs are the default; `SCN_LOG_FORMAT=json` also selects JSON. Logging uses a bounded background queue, dropping log lines if overloaded. Request bodies, query strings, and authorization headers are omitted.\n\nCLI spinners show staging, transfers, and synchronization in terminals. They clear on completion or failure and are hidden for redirected stderr or JSON logs. Use `--no-progress` to disable them explicitly. Command results and file contents stay on stdout.\n\nWith an empty `main`:\n\n```\nscn import-git /path/to/git-repo http://127.0.0.1:7878 --rev main\n```\n\n- Imports the selected history plus tagged histories, preserving lightweight and annotated tags, metadata, and original Git IDs. The selected revision becomes `main` after success; other branches are not copied.\n- Bare repositories work. Uncommitted files are untouched. Retries accept identical tags and reject conflicting ones; failed imports may leave drafts and tags.\n- Paths and metadata must be UTF-8. Symlinks, submodules, shallow histories, octopus merges, and tags targeting trees or blobs are unsupported. Tag signatures are not verified.\n\n```\ncargo build --release --locked\ncargo test\ncargo clippy --all-targets -- -D warnings\ncargo fmt --check\nSCN_VALKEY_SERVER=valkey-server cargo test --release --test cluster -- --ignored\ndocker compose -f deploy/compose.yaml config --quiet\nSCN_PERF_COMMITS=500 cargo test --release --test perf -- --ignored --nocapture\ncargo test --release --test large_files -- --ignored\n```\n\nTests cover cluster persistence, eviction, concurrent writes, S3 requests, and chunked files. Cluster tests include four concurrent 1 GiB transfers; the separate large-file test streams 5 GiB of repeated data.\n\nThe local performance test uses `SCN_PERF_COMMITS` for the total commit count (default 500). It measures sequential commits, warm tree reads, direct-parent checks, and 1 MiB blob reads. Cluster throughput also depends on network latency and Valkey persistence settings.", "url": "https://wpnews.pro/news/show-hn-scalable-source-control-in-rust-inspired-by-anthropic-s-scion", "canonical_source": "https://github.com/adrianchifor/scn", "published_at": "2026-10-17 15:21:59+00:00", "updated_at": "2026-10-17 15:32:50.626323+00:00", "lang": "en", "topics": ["developer-tools", "ai-agents", "ai-tools", "ai-infrastructure"], "entities": ["Adrian Chifor", "scn", "Anthropic", "Scion", "Rust", "Valkey", "SQLite", "RustFS"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/show-hn-scalable-source-control-in-rust-inspired-by-anthropic-s-scion", "markdown": "https://wpnews.pro/news/show-hn-scalable-source-control-in-rust-inspired-by-anthropic-s-scion.md", "text": "https://wpnews.pro/news/show-hn-scalable-source-control-in-rust-inspired-by-anthropic-s-scion.txt", "jsonld": "https://wpnews.pro/news/show-hn-scalable-source-control-in-rust-inspired-by-anthropic-s-scion.jsonld"}}