{"slug": "crabbox-sh-pond-runtime-pools-for-ai-agents-and-ci", "title": "Crabbox.sh Pond – Runtime Pools for AI Agents and CI", "summary": "Crabbox has introduced Pond, a lightweight runtime grouping system that allows developers to organize related cloud leases under a shared label for AI agents and CI workflows. The system supports three transport planes—Tailscale peer-to-peer mesh, URL bridge for HTTP(S) endpoints, and SSH-mesh for operator-side tunnels—enabling mixed-provider connectivity between pond members. Pond is currently in preview for v0.x, with the reserved `pond=` label key expected to remain stable through the v1.0 release.", "body_md": "# Pond\n\nA pond is a lightweight way to group related leases, discover how to reach each one, and release them together. It is not a central cluster object: a pond is an emergent set of active leases that share the reserved `pond=<name>`\n\nprovider label, plus local claim sidecars for providers that do not own cloud labels. A pond exists for as long as at least one active lease carries the label.\n\nReachability between pond members depends on the transport plane each member's provider supports. Tailscale gives true peer-to-peer `<slug>.cbx`\n\nnames; the URL bridge gives provider-native HTTP(S) endpoints; the SSH-mesh gives operator-side `ssh -L`\n\nforwards. A pond can mix providers and planes.\n\nA `--pond`\n\nof one is the default — single-box flows are unchanged.\n\n> **Preview.** Pond is preview for v0.x. The reserved `pond=`\n\nlabel key is > intended to stay, but metadata shape and command flags may evolve before v1.0.\n\n[#](#quick-start)Quick start\n\n```\n# Lease a few members into one pond (slug = stable role name).\ncrabbox warmup --pond pr-42 --slug api --provider hetzner --tailscale\ncrabbox warmup --pond pr-42 --slug db  --provider hetzner --tailscale\n\n# Discover peers, run work against a member, then tear the pond down.\ncrabbox pond peers   --pond pr-42\ncrabbox run --id api -- \"DB_HOST=db.cbx go test ./...\"\ncrabbox pond release pr-42\n```\n\nUse `--slug`\n\nas the stable role name; it is what shows up in discovery and in `<slug>.cbx`\n\nnames. Whether that slug is directly dialable from another member depends on the transport plane (see below).\n\n[#](#naming-a-pond)Naming a pond\n\n`--pond <name>`\n\naccepts any string and normalizes it: lowercased, characters outside `[a-z0-9-]`\n\ncollapsed to `-`\n\n, runs collapsed, leading/trailing dashes trimmed. The normalized name must contain at least one letter or digit and be at most 41 characters. The same name is reused everywhere the pond appears (the label value, the Tailscale ACL tag, peer hostnames), so it stays in a regular DNS-like identifier space.\n\n[#](#the-three-transport-planes)The three transport planes\n\nEach provider self-declares which planes its leases can serve via its `Spec().Features`\n\n(`FeatureTailscale`\n\n, `FeatureURLBridge`\n\n, `FeatureSSH`\n\n). A single provider can advertise more than one — for example a direct Hetzner box advertises both Tailscale and SSH, so Tailscale is the preferred peer mesh while `pond connect`\n\ncan still build operator-side SSH forwards. URL-only sandboxes (such as Islo or E2B) do not join the peer mesh; they surface HTTP(S) endpoints instead.\n\n| Plane | Feature flag | Providers that advertise it (today) | What you get |\n|---|---|---|---|\n| Tailscale | `FeatureTailscale` | Hetzner, Azure, GCP | true peer-to-peer mesh, `<slug>.cbx` DNS |\n| Bridge | `FeatureURLBridge` | Islo, E2B, Modal, Cloudflare, Tensorlake (adapters report `unsupported` until they ship a per-sandbox HTTPS ingress) | provider-native HTTP(S) endpoints for discovery and sharing |\n| SSH-mesh | `FeatureSSH` | any provider advertising SSH: Hetzner, Azure, GCP, AWS, Proxmox, static `ssh` , RunPod, exe-dev, Daytona, Sprites, Namespace, Semaphore, local-container, Parallels | operator-side `ssh -L` tunnels via `pond connect` |\n\nmacOS and Windows peer reachability are not covered by any plane yet.\n\nCapabilities are derived from each provider's `FeatureSet`\n\n, not from a static table — a provider opts into a plane by declaring the feature. See [providers.md](providers.html) for the full capability matrix and [tailscale.md](tailscale.html) for the Tailscale transport.\n\n[#](#commands)Commands\n\n```\ncrabbox warmup --pond NAME --slug ROLE --provider PROVIDER [--tailscale] [--expose PORT]...\ncrabbox run    --id LEASE_OR_SLUG -- COMMAND\ncrabbox list   --pond NAME\ncrabbox doctor --pond NAME\n\ncrabbox pond peers      --pond NAME [--provider P] [--share-port PORT] [--share-ttl D] [--json]\ncrabbox pond connect    NAME [--provider P] [--export] [--json]\ncrabbox pond disconnect NAME\ncrabbox pond release    NAME\n```\n\n[#](#pond-peers)`pond peers`\n\nLists every member of the pond, regardless of provider. `--pond`\n\nis required. With no `--provider`\n\n, the resolver fans out across every provider represented in the pond and concatenates the result; pass `--provider P`\n\nto restrict to one.\n\nEach row carries a primary `transport`\n\nhint plus the full `transports`\n\nlist of every plane that member's provider supports:\n\n``` php\n// crabbox pond peers --pond pr-42 --json  ->  { \"members\": [ ... ] }\n{\n  \"slug\": \"api\",\n  \"leaseID\": \"cbx_0a1b2c3d4e5f\",\n  \"provider\": \"hetzner\",\n  \"pond\": \"pr-42\",\n  \"transport\":  \"tailnet\",          // primary / recommended plane\n  \"transports\": [\"tailnet\", \"ssh\"], // every plane this provider supports\n  \"endpoint\":   \"100.64.1.3\"\n}\n```\n\nThe endpoint shape depends on the primary plane: a tailnet IPv4/FQDN for Tailscale members, `ssh://host:port`\n\nfor SSH-lease members, and a per-port HTTPS URL for URL-bridge members. Members whose endpoint is not yet recorded surface with `transport: \"pending\"`\n\nand an honest note; providers with no networking adapter (e.g. Blacksmith) surface with `transport: \"none\"`\n\n.\n\nThe bridge plane is HTTP-only by design. For URL-bridge providers you can publish a per-peer public URL for a port:\n\n```\ncrabbox pond peers --pond pr-42 --provider islo --share-port 8080 --share-ttl 12h --json\n```\n\n`--share-port`\n\nis idempotent (existing shares are reused), bounded to `1..65535`\n\n, and `--share-ttl`\n\ndefaults to 24h. Providers without a per-sandbox HTTPS ingress report `unsupported`\n\nrather than pretending to bridge.\n\n[#](#pond-connect-pond-disconnect)`pond connect`\n\n/ `pond disconnect`\n\n`pond connect <name>`\n\nopens operator-side `ssh -L`\n\nforwards to every pond member that declared `--expose`\n\nports, across all SSH-mesh-capable providers in the pond (not just one). `--provider P`\n\nnarrows it to a single provider.\n\nFor each forwarded port it renders, under `~/.crabbox/pond/<name>/`\n\n:\n\n`hosts`\n\n—`<slug>.cbx`\n\naliases pointing at`127.0.0.1`\n\n, with the assigned`env`\n\n— shell exports of the form`CRABBOX_POND_<PEER>_<PORT>=127.0.0.1:<local>`\n\n,\n\nloopback port in a comment (a hosts file cannot encode `host:port`\n\n).\n\nwhich is the supported way to reach a forwarded peer port.\n\nLocal forward ports are auto-allocated from `51820..52819`\n\n(probed against the kernel so they do not collide with services you already run).\n\nBy default `pond connect`\n\nruns in the foreground and holds the tunnels open until you press Ctrl-C. `--export`\n\ndaemonizes the tunnels so they survive the CLI exit and prints the exports for `eval`\n\n:\n\n```\neval \"$(crabbox pond connect pr-42 --export)\"\ncurl \"$CRABBOX_POND_API_8080\"          # reach api's exposed :8080\ncrabbox pond disconnect pr-42          # stop the daemonized forwards\n```\n\n`--export`\n\n(and `pond disconnect`\n\n) are macOS/Linux only: cleanup validates the local daemon process command before stopping it, and that validator is not yet implemented for Windows operators. Use foreground `pond connect`\n\non Windows.\n\n`--json`\n\nprints the forward table and exits without opening any tunnels.\n\n[#](#pond-release)`pond release`\n\nStops every lease in the named pond and removes their local claims, iterating across all providers represented in the pond — you do not pass `--provider`\n\n. Individual failures are logged as warnings and do not block the rest; the first error is returned so scripts can tell whether the release was clean.\n\n[#](#pool-vs-pond)`pool`\n\nvs `pond`\n\n`pool`\n\nis inventory — \"what runners exist?\" — exposed through `crabbox list`\n\nand the broker's `/v1/pool`\n\n. `pond`\n\nis grouping — \"which leases belong to this environment?\" — created by tagging leases with `--pond <name>`\n\nand operated through `crabbox pond ...`\n\n. `crabbox list --pond <name>`\n\nfilters inventory down to one pond.\n\n[#](#exposing-ports-for-the-ssh-mesh)Exposing ports for the SSH-mesh\n\n`--expose <port>`\n\n(repeatable, comma lists accepted) declares a TCP port a lease wants reachable over the SSH-mesh plane. Ports are validated (`1..65535`\n\n), deduplicated, and stored in the lease's provider label so `pond connect`\n\ncan discover them without a separate store. Up to 10 distinct ports per lease.\n\n```\ncrabbox warmup --pond pr-42 --slug api --provider hetzner --expose 8080 --expose 5432\n```\n\n[#](#example-use-cases)Example use cases\n\n**Per-PR isolated E2E environment.** Every PR gets its own staging; the pond dies with the PR:\n\n```\ncrabbox warmup --pond \"pr-$PR\" --slug api --provider hetzner --tailscale\ncrabbox warmup --pond \"pr-$PR\" --slug web --provider hetzner --tailscale\n# ... run E2E against api.cbx / web.cbx ...\ncrabbox pond release \"pr-$PR\"\n```\n\n**Mixed-vendor integration test.** CPU on Hetzner, GPU on a delegated provider, DB on Hetzner. Use Tailscale names for tailnet members and the URL bridge for HTTP(S) peers:\n\n```\ncrabbox warmup --pond \"it-$SHA\" --slug api --provider hetzner --tailscale\ncrabbox warmup --pond \"it-$SHA\" --slug ml  --provider modal   --class a10g\ncrabbox warmup --pond \"it-$SHA\" --slug db  --provider hetzner --tailscale\ncrabbox run --id api -- \"DB_HOST=db.cbx go test ./...\"\n```\n\n**Per-PR build farm.** A large delegated pond of latency-tolerant helpers that dies with the PR. This is a work queue, not a low-latency fabric:\n\n```\ncrabbox warmup --pond \"build-$PR\" --slug coord --provider hetzner\nfor i in $(seq 1 30); do\n  crabbox warmup --pond \"build-$PR\" --slug \"helper-$i\" --provider islo &\ndone\nwait\n```\n\n[#](#when-not-to-use-it)When not to use it\n\n**Tightly-coupled HPC (MPI / NCCL) across providers**— public-internet** macOS / Windows peer reachability**— not yet covered by any plane.** Untrusted multi-tenant isolation**— connectivity inside a pond is\n\nlatency is too high. Keep tightly-coupled jobs on one provider and region.\n\ndefault-allow. Do not put mutually untrusted workloads in one pond.\n\n[#](#tailscale-names-and-the-cbx-hosts-file)Tailscale names and the `.cbx`\n\nhosts file\n\nOn every Tailscale-capable Linux peer, bootstrap installs a systemd timer that rewrites `/etc/hosts.cbx`\n\nand a managed `/etc/hosts`\n\nblock every 30 seconds from the box-local `tailscale status --json`\n\noutput. Each peer renders as `<tailnet-ipv4> <slug>.cbx`\n\n, so members resolve each other as `<slug>.cbx`\n\nwithin roughly one refresh interval — and the broker never sees a Tailscale credential. Discovery is gated by the per-pond ACL tag (below): only peers carrying that tag are written.\n\n[#](#tailscale-acl-bootstrap)Tailscale ACL bootstrap\n\nEvery Tailscale pond peer is advertised under the ACL tag `tag:cbx-pond-<owner>-<pond>`\n\n, so a single concrete policy row gates the whole pond. `<owner>`\n\nis derived from your local git email's local-part (sanitized, and hashed when it would overflow the tag length budget).\n\nAuto-generated pond tags are **direct-provider only**: they are added to the provider's Tailscale config when you use a direct (non-brokered) Tailscale-capable provider. Brokered coordinators keep enforcing their own configured `CRABBOX_TAILSCALE_TAGS`\n\nallowlist and do not receive dynamic per-pond tags.\n\nInstalling the tag's `tagOwners`\n\nand self-peering grant on your tailnet policy is **opt-in**. To let lease creation auto-install the rows, set both:\n\n`TS_API_KEY`\n\n— a tailnet API key (optionally`TS_TAILNET`\n\nto target a specific`CRABBOX_POND_ACL_BOOTSTRAP=1`\n\n— explicit consent to edit the tailnet policy.\n\ntailnet; defaults to `-`\n\n, your default tailnet).\n\n`TS_API_KEY`\n\nalone is never treated as consent.\n\nWhen enabled, Crabbox reads the policy with an ETag, merges in the missing `tagOwners`\n\nentry and a self-peering rule (grants-shape if the policy already uses grants, otherwise a legacy `acls`\n\nrow), and writes it back with `If-Match`\n\nso concurrent edits fail fast. If the control plane does not expose a Tailscale-compatible policy API (e.g. Headscale), lease creation is not blocked.\n\nWithout the opt-in, `crabbox doctor --pond <name>`\n\nverifies the required policy row (using `TS_API_KEY`\n\nread-only) and points you at this document if it is missing. The broker never sees Tailscale credentials.\n\nPoint the CLI at a self-hosted control plane with `CRABBOX_TS_API_URL`\n\n(it wins over `TS_API_URL`\n\n); the default is `https://api.tailscale.com`\n\n.\n\n[#](#manual-policy-snippet)Manual policy snippet\n\nIf you prefer to manage the policy yourself, add the tag owner and a self-peering rule for your pond's tag (replace `cbx-pond-alice-pr-42`\n\nwith the tag `doctor --pond`\n\nreports). Grants shape:\n\n```\n{\n  \"tagOwners\": {\n    \"tag:cbx-pond-alice-pr-42\": [\"autogroup:admin\"]\n  },\n  \"grants\": [\n    { \"src\": [\"tag:cbx-pond-alice-pr-42\"], \"dst\": [\"tag:cbx-pond-alice-pr-42\"], \"ip\": [\"*\"] }\n  ]\n}\n```\n\nLegacy `acls`\n\nshape:\n\n```\n{\n  \"tagOwners\": {\n    \"tag:cbx-pond-alice-pr-42\": [\"autogroup:admin\"]\n  },\n  \"acls\": [\n    { \"action\": \"accept\", \"src\": [\"tag:cbx-pond-alice-pr-42\"], \"dst\": [\"tag:cbx-pond-alice-pr-42:*\"] }\n  ]\n}\n```\n\n[#](#see-also)See also\n\n[providers.md](providers.html)— provider capability matrix.[tailscale.md](tailscale.html)— the Tailscale transport and tags.[doctor.md](doctor.html)— readiness checks, including the pond ACL check.[identifiers.md](identifiers.html)— lease IDs and slugs.", "url": "https://wpnews.pro/news/crabbox-sh-pond-runtime-pools-for-ai-agents-and-ci", "canonical_source": "https://crabbox.sh/features/pond.html", "published_at": "2026-05-29 10:26:02+00:00", "updated_at": "2026-05-29 10:46:52.362422+00:00", "lang": "en", "topics": ["ai-infrastructure", "ai-agents", "ai-tools"], "entities": ["Crabbox", "Tailscale", "Hetzner"], "alternates": {"html": "https://wpnews.pro/news/crabbox-sh-pond-runtime-pools-for-ai-agents-and-ci", "markdown": "https://wpnews.pro/news/crabbox-sh-pond-runtime-pools-for-ai-agents-and-ci.md", "text": "https://wpnews.pro/news/crabbox-sh-pond-runtime-pools-for-ai-agents-and-ci.txt", "jsonld": "https://wpnews.pro/news/crabbox-sh-pond-runtime-pools-for-ai-agents-and-ci.jsonld"}}