Crabbox.sh Pond – Runtime Pools for AI Agents and CI

Crabbox has introduced Pond, a lightweight runtime grouping system that allows developers to organize related cloud leases under a shared label for AI agents and CI workflows. The system supports three transport planes—Tailscale peer-to-peer mesh, URL bridge for HTTP(S) endpoints, and SSH-mesh for operator-side tunnels—enabling mixed-provider connectivity between pond members. Pond is currently in preview for v0.x, with the reserved `pond=` label key expected to remain stable through the v1.0 release.

Pond A pond is a lightweight way to group related leases, discover how to reach each one, and release them together. It is not a central cluster object: a pond is an emergent set of active leases that share the reserved pond=<name provider label, plus local claim sidecars for providers that do not own cloud labels. A pond exists for as long as at least one active lease carries the label. Reachability between pond members depends on the transport plane each member's provider supports. Tailscale gives true peer-to-peer <slug .cbx names; the URL bridge gives provider-native HTTP S endpoints; the SSH-mesh gives operator-side ssh -L forwards. A pond can mix providers and planes. A --pond of one is the default — single-box flows are unchanged. Preview. Pond is preview for v0.x. The reserved pond= label key is intended to stay, but metadata shape and command flags may evolve before v1.0. quick-start Quick start Lease a few members into one pond slug = stable role name . crabbox warmup --pond pr-42 --slug api --provider hetzner --tailscale crabbox warmup --pond pr-42 --slug db --provider hetzner --tailscale Discover peers, run work against a member, then tear the pond down. crabbox pond peers --pond pr-42 crabbox run --id api -- "DB HOST=db.cbx go test ./..." crabbox pond release pr-42 Use --slug as the stable role name; it is what shows up in discovery and in <slug .cbx names. Whether that slug is directly dialable from another member depends on the transport plane see below . naming-a-pond Naming a pond --pond <name accepts any string and normalizes it: lowercased, characters outside a-z0-9- collapsed to - , runs collapsed, leading/trailing dashes trimmed. The normalized name must contain at least one letter or digit and be at most 41 characters. The same name is reused everywhere the pond appears the label value, the Tailscale ACL tag, peer hostnames , so it stays in a regular DNS-like identifier space. the-three-transport-planes The three transport planes Each provider self-declares which planes its leases can serve via its Spec .Features FeatureTailscale , FeatureURLBridge , FeatureSSH . A single provider can advertise more than one — for example a direct Hetzner box advertises both Tailscale and SSH, so Tailscale is the preferred peer mesh while pond connect can still build operator-side SSH forwards. URL-only sandboxes such as Islo or E2B do not join the peer mesh; they surface HTTP S endpoints instead. | Plane | Feature flag | Providers that advertise it today | What you get | |---|---|---|---| | Tailscale | FeatureTailscale | Hetzner, Azure, GCP | true peer-to-peer mesh, <slug .cbx DNS | | Bridge | FeatureURLBridge | Islo, E2B, Modal, Cloudflare, Tensorlake adapters report unsupported until they ship a per-sandbox HTTPS ingress | provider-native HTTP S endpoints for discovery and sharing | | SSH-mesh | FeatureSSH | any provider advertising SSH: Hetzner, Azure, GCP, AWS, Proxmox, static ssh , RunPod, exe-dev, Daytona, Sprites, Namespace, Semaphore, local-container, Parallels | operator-side ssh -L tunnels via pond connect | macOS and Windows peer reachability are not covered by any plane yet. Capabilities are derived from each provider's FeatureSet , not from a static table — a provider opts into a plane by declaring the feature. See providers.md providers.html for the full capability matrix and tailscale.md tailscale.html for the Tailscale transport. commands Commands crabbox warmup --pond NAME --slug ROLE --provider PROVIDER --tailscale --expose PORT ... crabbox run --id LEASE OR SLUG -- COMMAND crabbox list --pond NAME crabbox doctor --pond NAME crabbox pond peers --pond NAME --provider P --share-port PORT --share-ttl D --json crabbox pond connect NAME --provider P --export --json crabbox pond disconnect NAME crabbox pond release NAME pond-peers pond peers Lists every member of the pond, regardless of provider. --pond is required. With no --provider , the resolver fans out across every provider represented in the pond and concatenates the result; pass --provider P to restrict to one. Each row carries a primary transport hint plus the full transports list of every plane that member's provider supports: php // crabbox pond peers --pond pr-42 --json - { "members": ... } { "slug": "api", "leaseID": "cbx 0a1b2c3d4e5f", "provider": "hetzner", "pond": "pr-42", "transport": "tailnet", // primary / recommended plane "transports": "tailnet", "ssh" , // every plane this provider supports "endpoint": "100.64.1.3" } The endpoint shape depends on the primary plane: a tailnet IPv4/FQDN for Tailscale members, ssh://host:port for SSH-lease members, and a per-port HTTPS URL for URL-bridge members. Members whose endpoint is not yet recorded surface with transport: "pending" and an honest note; providers with no networking adapter e.g. Blacksmith surface with transport: "none" . The bridge plane is HTTP-only by design. For URL-bridge providers you can publish a per-peer public URL for a port: crabbox pond peers --pond pr-42 --provider islo --share-port 8080 --share-ttl 12h --json --share-port is idempotent existing shares are reused , bounded to 1..65535 , and --share-ttl defaults to 24h. Providers without a per-sandbox HTTPS ingress report unsupported rather than pretending to bridge. pond-connect-pond-disconnect pond connect / pond disconnect pond connect <name opens operator-side ssh -L forwards to every pond member that declared --expose ports, across all SSH-mesh-capable providers in the pond not just one . --provider P narrows it to a single provider. For each forwarded port it renders, under ~/.crabbox/pond/<name / : hosts — <slug .cbx aliases pointing at 127.0.0.1 , with the assigned env — shell exports of the form CRABBOX POND <PEER <PORT =127.0.0.1:<local , loopback port in a comment a hosts file cannot encode host:port . which is the supported way to reach a forwarded peer port. Local forward ports are auto-allocated from 51820..52819 probed against the kernel so they do not collide with services you already run . By default pond connect runs in the foreground and holds the tunnels open until you press Ctrl-C. --export daemonizes the tunnels so they survive the CLI exit and prints the exports for eval : eval "$ crabbox pond connect pr-42 --export " curl "$CRABBOX POND API 8080" reach api's exposed :8080 crabbox pond disconnect pr-42 stop the daemonized forwards --export and pond disconnect are macOS/Linux only: cleanup validates the local daemon process command before stopping it, and that validator is not yet implemented for Windows operators. Use foreground pond connect on Windows. --json prints the forward table and exits without opening any tunnels. pond-release pond release Stops every lease in the named pond and removes their local claims, iterating across all providers represented in the pond — you do not pass --provider . Individual failures are logged as warnings and do not block the rest; the first error is returned so scripts can tell whether the release was clean. pool-vs-pond pool vs pond pool is inventory — "what runners exist?" — exposed through crabbox list and the broker's /v1/pool . pond is grouping — "which leases belong to this environment?" — created by tagging leases with --pond <name and operated through crabbox pond ... . crabbox list --pond <name filters inventory down to one pond. exposing-ports-for-the-ssh-mesh Exposing ports for the SSH-mesh --expose <port repeatable, comma lists accepted declares a TCP port a lease wants reachable over the SSH-mesh plane. Ports are validated 1..65535 , deduplicated, and stored in the lease's provider label so pond connect can discover them without a separate store. Up to 10 distinct ports per lease. crabbox warmup --pond pr-42 --slug api --provider hetzner --expose 8080 --expose 5432 example-use-cases Example use cases Per-PR isolated E2E environment. Every PR gets its own staging; the pond dies with the PR: crabbox warmup --pond "pr-$PR" --slug api --provider hetzner --tailscale crabbox warmup --pond "pr-$PR" --slug web --provider hetzner --tailscale ... run E2E against api.cbx / web.cbx ... crabbox pond release "pr-$PR" Mixed-vendor integration test. CPU on Hetzner, GPU on a delegated provider, DB on Hetzner. Use Tailscale names for tailnet members and the URL bridge for HTTP S peers: crabbox warmup --pond "it-$SHA" --slug api --provider hetzner --tailscale crabbox warmup --pond "it-$SHA" --slug ml --provider modal --class a10g crabbox warmup --pond "it-$SHA" --slug db --provider hetzner --tailscale crabbox run --id api -- "DB HOST=db.cbx go test ./..." Per-PR build farm. A large delegated pond of latency-tolerant helpers that dies with the PR. This is a work queue, not a low-latency fabric: crabbox warmup --pond "build-$PR" --slug coord --provider hetzner for i in $ seq 1 30 ; do crabbox warmup --pond "build-$PR" --slug "helper-$i" --provider islo & done wait when-not-to-use-it When not to use it Tightly-coupled HPC MPI / NCCL across providers — public-internet macOS / Windows peer reachability — not yet covered by any plane. Untrusted multi-tenant isolation — connectivity inside a pond is latency is too high. Keep tightly-coupled jobs on one provider and region. default-allow. Do not put mutually untrusted workloads in one pond. tailscale-names-and-the-cbx-hosts-file Tailscale names and the .cbx hosts file On every Tailscale-capable Linux peer, bootstrap installs a systemd timer that rewrites /etc/hosts.cbx and a managed /etc/hosts block every 30 seconds from the box-local tailscale status --json output. Each peer renders as <tailnet-ipv4 <slug .cbx , so members resolve each other as <slug .cbx within roughly one refresh interval — and the broker never sees a Tailscale credential. Discovery is gated by the per-pond ACL tag below : only peers carrying that tag are written. tailscale-acl-bootstrap Tailscale ACL bootstrap Every Tailscale pond peer is advertised under the ACL tag tag:cbx-pond-<owner -<pond , so a single concrete policy row gates the whole pond. <owner is derived from your local git email's local-part sanitized, and hashed when it would overflow the tag length budget . Auto-generated pond tags are direct-provider only : they are added to the provider's Tailscale config when you use a direct non-brokered Tailscale-capable provider. Brokered coordinators keep enforcing their own configured CRABBOX TAILSCALE TAGS allowlist and do not receive dynamic per-pond tags. Installing the tag's tagOwners and self-peering grant on your tailnet policy is opt-in . To let lease creation auto-install the rows, set both: TS API KEY — a tailnet API key optionally TS TAILNET to target a specific CRABBOX POND ACL BOOTSTRAP=1 — explicit consent to edit the tailnet policy. tailnet; defaults to - , your default tailnet . TS API KEY alone is never treated as consent. When enabled, Crabbox reads the policy with an ETag, merges in the missing tagOwners entry and a self-peering rule grants-shape if the policy already uses grants, otherwise a legacy acls row , and writes it back with If-Match so concurrent edits fail fast. If the control plane does not expose a Tailscale-compatible policy API e.g. Headscale , lease creation is not blocked. Without the opt-in, crabbox doctor --pond <name verifies the required policy row using TS API KEY read-only and points you at this document if it is missing. The broker never sees Tailscale credentials. Point the CLI at a self-hosted control plane with CRABBOX TS API URL it wins over TS API URL ; the default is https://api.tailscale.com . manual-policy-snippet Manual policy snippet If you prefer to manage the policy yourself, add the tag owner and a self-peering rule for your pond's tag replace cbx-pond-alice-pr-42 with the tag doctor --pond reports . Grants shape: { "tagOwners": { "tag:cbx-pond-alice-pr-42": "autogroup:admin" }, "grants": { "src": "tag:cbx-pond-alice-pr-42" , "dst": "tag:cbx-pond-alice-pr-42" , "ip": " " } } Legacy acls shape: { "tagOwners": { "tag:cbx-pond-alice-pr-42": "autogroup:admin" }, "acls": { "action": "accept", "src": "tag:cbx-pond-alice-pr-42" , "dst": "tag:cbx-pond-alice-pr-42: " } } see-also See also providers.md providers.html — provider capability matrix. tailscale.md tailscale.html — the Tailscale transport and tags. doctor.md doctor.html — readiness checks, including the pond ACL check. identifiers.md identifiers.html — lease IDs and slugs.