# Crabbox.sh Pond – Runtime Pools for AI Agents and CI > Source: > Published: 2026-05-29 10:26:02+00:00 # Pond A pond is a lightweight way to group related leases, discover how to reach each one, and release them together. It is not a central cluster object: a pond is an emergent set of active leases that share the reserved `pond=` provider label, plus local claim sidecars for providers that do not own cloud labels. A pond exists for as long as at least one active lease carries the label. Reachability between pond members depends on the transport plane each member's provider supports. Tailscale gives true peer-to-peer `.cbx` names; the URL bridge gives provider-native HTTP(S) endpoints; the SSH-mesh gives operator-side `ssh -L` forwards. A pond can mix providers and planes. A `--pond` of one is the default — single-box flows are unchanged. > **Preview.** Pond is preview for v0.x. The reserved `pond=` label key is > intended to stay, but metadata shape and command flags may evolve before v1.0. [#](#quick-start)Quick start ``` # Lease a few members into one pond (slug = stable role name). crabbox warmup --pond pr-42 --slug api --provider hetzner --tailscale crabbox warmup --pond pr-42 --slug db --provider hetzner --tailscale # Discover peers, run work against a member, then tear the pond down. crabbox pond peers --pond pr-42 crabbox run --id api -- "DB_HOST=db.cbx go test ./..." crabbox pond release pr-42 ``` Use `--slug` as the stable role name; it is what shows up in discovery and in `.cbx` names. Whether that slug is directly dialable from another member depends on the transport plane (see below). [#](#naming-a-pond)Naming a pond `--pond ` accepts any string and normalizes it: lowercased, characters outside `[a-z0-9-]` collapsed to `-` , runs collapsed, leading/trailing dashes trimmed. The normalized name must contain at least one letter or digit and be at most 41 characters. The same name is reused everywhere the pond appears (the label value, the Tailscale ACL tag, peer hostnames), so it stays in a regular DNS-like identifier space. [#](#the-three-transport-planes)The three transport planes Each provider self-declares which planes its leases can serve via its `Spec().Features` (`FeatureTailscale` , `FeatureURLBridge` , `FeatureSSH` ). A single provider can advertise more than one — for example a direct Hetzner box advertises both Tailscale and SSH, so Tailscale is the preferred peer mesh while `pond connect` can still build operator-side SSH forwards. URL-only sandboxes (such as Islo or E2B) do not join the peer mesh; they surface HTTP(S) endpoints instead. | Plane | Feature flag | Providers that advertise it (today) | What you get | |---|---|---|---| | Tailscale | `FeatureTailscale` | Hetzner, Azure, GCP | true peer-to-peer mesh, `.cbx` DNS | | Bridge | `FeatureURLBridge` | Islo, E2B, Modal, Cloudflare, Tensorlake (adapters report `unsupported` until they ship a per-sandbox HTTPS ingress) | provider-native HTTP(S) endpoints for discovery and sharing | | SSH-mesh | `FeatureSSH` | any provider advertising SSH: Hetzner, Azure, GCP, AWS, Proxmox, static `ssh` , RunPod, exe-dev, Daytona, Sprites, Namespace, Semaphore, local-container, Parallels | operator-side `ssh -L` tunnels via `pond connect` | macOS and Windows peer reachability are not covered by any plane yet. Capabilities are derived from each provider's `FeatureSet` , not from a static table — a provider opts into a plane by declaring the feature. See [providers.md](providers.html) for the full capability matrix and [tailscale.md](tailscale.html) for the Tailscale transport. [#](#commands)Commands ``` crabbox warmup --pond NAME --slug ROLE --provider PROVIDER [--tailscale] [--expose PORT]... crabbox run --id LEASE_OR_SLUG -- COMMAND crabbox list --pond NAME crabbox doctor --pond NAME crabbox pond peers --pond NAME [--provider P] [--share-port PORT] [--share-ttl D] [--json] crabbox pond connect NAME [--provider P] [--export] [--json] crabbox pond disconnect NAME crabbox pond release NAME ``` [#](#pond-peers)`pond peers` Lists every member of the pond, regardless of provider. `--pond` is required. With no `--provider` , the resolver fans out across every provider represented in the pond and concatenates the result; pass `--provider P` to restrict to one. Each row carries a primary `transport` hint plus the full `transports` list of every plane that member's provider supports: ``` php // crabbox pond peers --pond pr-42 --json -> { "members": [ ... ] } { "slug": "api", "leaseID": "cbx_0a1b2c3d4e5f", "provider": "hetzner", "pond": "pr-42", "transport": "tailnet", // primary / recommended plane "transports": ["tailnet", "ssh"], // every plane this provider supports "endpoint": "100.64.1.3" } ``` The endpoint shape depends on the primary plane: a tailnet IPv4/FQDN for Tailscale members, `ssh://host:port` for SSH-lease members, and a per-port HTTPS URL for URL-bridge members. Members whose endpoint is not yet recorded surface with `transport: "pending"` and an honest note; providers with no networking adapter (e.g. Blacksmith) surface with `transport: "none"` . The bridge plane is HTTP-only by design. For URL-bridge providers you can publish a per-peer public URL for a port: ``` crabbox pond peers --pond pr-42 --provider islo --share-port 8080 --share-ttl 12h --json ``` `--share-port` is idempotent (existing shares are reused), bounded to `1..65535` , and `--share-ttl` defaults to 24h. Providers without a per-sandbox HTTPS ingress report `unsupported` rather than pretending to bridge. [#](#pond-connect-pond-disconnect)`pond connect` / `pond disconnect` `pond connect ` opens operator-side `ssh -L` forwards to every pond member that declared `--expose` ports, across all SSH-mesh-capable providers in the pond (not just one). `--provider P` narrows it to a single provider. For each forwarded port it renders, under `~/.crabbox/pond//` : `hosts` —`.cbx` aliases pointing at`127.0.0.1` , with the assigned`env` — shell exports of the form`CRABBOX_POND__=127.0.0.1:` , loopback port in a comment (a hosts file cannot encode `host:port` ). which is the supported way to reach a forwarded peer port. Local forward ports are auto-allocated from `51820..52819` (probed against the kernel so they do not collide with services you already run). By default `pond connect` runs in the foreground and holds the tunnels open until you press Ctrl-C. `--export` daemonizes the tunnels so they survive the CLI exit and prints the exports for `eval` : ``` eval "$(crabbox pond connect pr-42 --export)" curl "$CRABBOX_POND_API_8080" # reach api's exposed :8080 crabbox pond disconnect pr-42 # stop the daemonized forwards ``` `--export` (and `pond disconnect` ) are macOS/Linux only: cleanup validates the local daemon process command before stopping it, and that validator is not yet implemented for Windows operators. Use foreground `pond connect` on Windows. `--json` prints the forward table and exits without opening any tunnels. [#](#pond-release)`pond release` Stops every lease in the named pond and removes their local claims, iterating across all providers represented in the pond — you do not pass `--provider` . Individual failures are logged as warnings and do not block the rest; the first error is returned so scripts can tell whether the release was clean. [#](#pool-vs-pond)`pool` vs `pond` `pool` is inventory — "what runners exist?" — exposed through `crabbox list` and the broker's `/v1/pool` . `pond` is grouping — "which leases belong to this environment?" — created by tagging leases with `--pond ` and operated through `crabbox pond ...` . `crabbox list --pond ` filters inventory down to one pond. [#](#exposing-ports-for-the-ssh-mesh)Exposing ports for the SSH-mesh `--expose ` (repeatable, comma lists accepted) declares a TCP port a lease wants reachable over the SSH-mesh plane. Ports are validated (`1..65535` ), deduplicated, and stored in the lease's provider label so `pond connect` can discover them without a separate store. Up to 10 distinct ports per lease. ``` crabbox warmup --pond pr-42 --slug api --provider hetzner --expose 8080 --expose 5432 ``` [#](#example-use-cases)Example use cases **Per-PR isolated E2E environment.** Every PR gets its own staging; the pond dies with the PR: ``` crabbox warmup --pond "pr-$PR" --slug api --provider hetzner --tailscale crabbox warmup --pond "pr-$PR" --slug web --provider hetzner --tailscale # ... run E2E against api.cbx / web.cbx ... crabbox pond release "pr-$PR" ``` **Mixed-vendor integration test.** CPU on Hetzner, GPU on a delegated provider, DB on Hetzner. Use Tailscale names for tailnet members and the URL bridge for HTTP(S) peers: ``` crabbox warmup --pond "it-$SHA" --slug api --provider hetzner --tailscale crabbox warmup --pond "it-$SHA" --slug ml --provider modal --class a10g crabbox warmup --pond "it-$SHA" --slug db --provider hetzner --tailscale crabbox run --id api -- "DB_HOST=db.cbx go test ./..." ``` **Per-PR build farm.** A large delegated pond of latency-tolerant helpers that dies with the PR. This is a work queue, not a low-latency fabric: ``` crabbox warmup --pond "build-$PR" --slug coord --provider hetzner for i in $(seq 1 30); do crabbox warmup --pond "build-$PR" --slug "helper-$i" --provider islo & done wait ``` [#](#when-not-to-use-it)When not to use it **Tightly-coupled HPC (MPI / NCCL) across providers**— public-internet** macOS / Windows peer reachability**— not yet covered by any plane.** Untrusted multi-tenant isolation**— connectivity inside a pond is latency is too high. Keep tightly-coupled jobs on one provider and region. default-allow. Do not put mutually untrusted workloads in one pond. [#](#tailscale-names-and-the-cbx-hosts-file)Tailscale names and the `.cbx` hosts file On every Tailscale-capable Linux peer, bootstrap installs a systemd timer that rewrites `/etc/hosts.cbx` and a managed `/etc/hosts` block every 30 seconds from the box-local `tailscale status --json` output. Each peer renders as ` .cbx` , so members resolve each other as `.cbx` within roughly one refresh interval — and the broker never sees a Tailscale credential. Discovery is gated by the per-pond ACL tag (below): only peers carrying that tag are written. [#](#tailscale-acl-bootstrap)Tailscale ACL bootstrap Every Tailscale pond peer is advertised under the ACL tag `tag:cbx-pond--` , so a single concrete policy row gates the whole pond. `` is derived from your local git email's local-part (sanitized, and hashed when it would overflow the tag length budget). Auto-generated pond tags are **direct-provider only**: they are added to the provider's Tailscale config when you use a direct (non-brokered) Tailscale-capable provider. Brokered coordinators keep enforcing their own configured `CRABBOX_TAILSCALE_TAGS` allowlist and do not receive dynamic per-pond tags. Installing the tag's `tagOwners` and self-peering grant on your tailnet policy is **opt-in**. To let lease creation auto-install the rows, set both: `TS_API_KEY` — a tailnet API key (optionally`TS_TAILNET` to target a specific`CRABBOX_POND_ACL_BOOTSTRAP=1` — explicit consent to edit the tailnet policy. tailnet; defaults to `-` , your default tailnet). `TS_API_KEY` alone is never treated as consent. When enabled, Crabbox reads the policy with an ETag, merges in the missing `tagOwners` entry and a self-peering rule (grants-shape if the policy already uses grants, otherwise a legacy `acls` row), and writes it back with `If-Match` so concurrent edits fail fast. If the control plane does not expose a Tailscale-compatible policy API (e.g. Headscale), lease creation is not blocked. Without the opt-in, `crabbox doctor --pond ` verifies the required policy row (using `TS_API_KEY` read-only) and points you at this document if it is missing. The broker never sees Tailscale credentials. Point the CLI at a self-hosted control plane with `CRABBOX_TS_API_URL` (it wins over `TS_API_URL` ); the default is `https://api.tailscale.com` . [#](#manual-policy-snippet)Manual policy snippet If you prefer to manage the policy yourself, add the tag owner and a self-peering rule for your pond's tag (replace `cbx-pond-alice-pr-42` with the tag `doctor --pond` reports). Grants shape: ``` { "tagOwners": { "tag:cbx-pond-alice-pr-42": ["autogroup:admin"] }, "grants": [ { "src": ["tag:cbx-pond-alice-pr-42"], "dst": ["tag:cbx-pond-alice-pr-42"], "ip": ["*"] } ] } ``` Legacy `acls` shape: ``` { "tagOwners": { "tag:cbx-pond-alice-pr-42": ["autogroup:admin"] }, "acls": [ { "action": "accept", "src": ["tag:cbx-pond-alice-pr-42"], "dst": ["tag:cbx-pond-alice-pr-42:*"] } ] } ``` [#](#see-also)See also [providers.md](providers.html)— provider capability matrix.[tailscale.md](tailscale.html)— the Tailscale transport and tags.[doctor.md](doctor.html)— readiness checks, including the pond ACL check.[identifiers.md](identifiers.html)— lease IDs and slugs.