{"slug": "sandboxes-for-ai-agents", "title": "Sandboxes for AI Agents", "summary": "A developer outlined seven designs for sandboxing AI coding agents, ranging from single-service mocks to ephemeral per-task VMs, arguing that agents need isolated practice environments to validate their own code before opening pull requests. The writeup frames every design as a trade-off among realistic, safe, and fast-and-cheap, and cites Stripe's Minions running on pre-warmed devboxes that start in about 10 seconds and Ramp running each session in a Modal sandbox. The author recommends judging setups by their 'blast radius' — the worst thing an agent can reach while doing exactly what it was asked.", "body_md": "## **Without a sandbox, your agent is guessing**\n\nAsk your agent to create a new feature - it will confidently put up PRs ready for review. But how do you know any of it works? You have to test it out. Send back the failures, iterate, continue back and forth until you have something usable.\n\nIsn’t there a way to get agents to test and validate their own code? Is there a way for agents to prove to you their changes work. This is where your sandbox comes in.\n\n## **What a sandbox is**\n\nA sandbox is a practice copy of your app. Think flight simulator: close enough to the real thing that practice counts, fake enough that a crash hurts nobody.\n\nanatomy of a sandbox · wall, access, stack, data, receipts\n\nThe agent comes in through a dev-only login, works against the full stack and its own data, and leaves “receipts” on the PR - proof that a change was validated. Customer data and real credentials stay outside the wall.\n\n### **The running stack**\n\nYour services, databases and queues, plus fakes or vendor sandboxes for third-party APIs like payments and email. Sounds easy. It rarely is. Every service drags in a few more, and a mature app can need dozens running before the home page loads.\n\n### **Isolation**\n\nIsolation makes everything else safe. Keep customer data out. Keep shared data out too, like the staging database three other teams rely on. Lock down outbound traffic, and never put real third-party credentials inside.\n\nA handy way to judge any setup is its **blast radius**: the worst thing an agent can reach while doing exactly what you asked. Once I asked an agent to validate a change to the “delete org” flow. It did, by deleting the org. In a staging or production environment, this could be a problem - but in a sandbox it’s fine.\n\n## **Seven sandbox designs**\n\n### **Realistic, safe, fast: pick two**\n\nEvery sandbox trades off **realistic** (it behaves like production), **safe** (mistakes stay inside) and **fast and cheap** (you can start lots of them in seconds). Nobody gets all three.\n\nsandbox trade-offs · 7 designs on 3 goals\n\nThese are example designs I’ve seen in practice - I provided names.\n\n### **0. Island**\n\nOne service on its own, with every dependency faked: mocks, emulators like LocalStack, recorded HTTP responses. It’s the fastest and cheapest check you have, and nothing can escape it. It’s also only as true as your fakes, and it misses everything that happens between services, which is exactly where the agent in the intro got caught. Use it as a first check. I barely count it as a sandbox.\n\nIsland · one service, all fakes\n\n### **1. Local Dev**\n\nThe sandbox most people already have: the stack running on a developer’s laptop, with native processes or Docker Compose, maybe behind a CLI. Feedback is instant and it costs nothing.\n\nLocal Dev · the stack on a laptop\n\nIt also runs one agent at a time and drifts from machine to machine. The agent runs with the developer’s credentials, so the blast radius is the laptop plus anything those credentials can reach.\n\n### **2. Devbox**\n\nA devbox is local dev moved to the cloud. A persistent machine built from a prebuilt image, either one per engineer or a few set aside for agents. It’s bigger than a laptop, identical for everyone, and keeps running when you close the lid. You pay for it while it sits idle, it drifts unless you rebuild it, and agents sharing a box share its mess.\n\nDevbox · persistent, per engineer or per agent\n\n### **3. Ephemeral**\n\nA fresh VM or container for every task, started from a pre-warmed snapshot and thrown away afterwards. Production and the open internet are out of reach.\n\nEphemeral · one box per task\n\nThis is where teams running agents at scale have landed. [Stripe’s Minions](https://stripe.dev/blog/minions-stripes-one-shot-end-to-end-coding-agents) run on pre-warmed devboxes that start in about 10 seconds. [Ramp](https://modal.com/blog/how-ramp-built-a-full-context-background-coding-agent-on-modal) runs each session in a Modal sandbox booted from a snapshot rebuilt every 30 minutes. [GitHub Copilot’s coding agent](https://docs.github.com/copilot/concepts/coding-agent/about-copilot-coding-agent) works on an Actions runner behind a firewall allowlist.\n\nSnapshots need rebuilding, every task costs something, and you get a dev stack rather than production’s real infrastructure.\n\n### **Pets vs. cattle**\n\nA pet has a name, and when it gets sick you nurse it back. Cattle get numbers and get replaced. **Devboxes** are pets. **Ephemeral sandboxes** are cattle.\n\nAn agent shouldn’t burn an hour debugging its own environment. If a sandbox gets into a weird state, throw it out and start a fresh one.\n\npets vs. cattle · Devbox vs. Ephemeral\n\n### **4. Preview Environment**\n\nA production twin. You deploy production’s own config with sandbox settings, one isolated copy per environment, and agents reach in from outside. It’s the most realistic option, and there’s no second stack definition to keep in sync.\n\nPreview Environment · production’s config, copied\n\nIt’s also slow to start, expensive and complicated, and some third-party services can’t be copied at all. [Lyft](https://www.infoq.com/news/2022/03/scaling-testing-acceptance-lyft/) ran full per-engineer environments until they stopped scaling, then moved to request routing. A small warm pool for integration and infrastructure work is usually enough.\n\n### **5. Hybrid**\n\nRun only the services you’re changing and borrow the rest from a shared environment. Tagged requests reach your version; everything else hits the shared copies. [Uber’s SLATE](https://www.uber.com/blog/simplifying-developer-testing-through-slate) and Lyft’s staging overrides both work this way. It starts in seconds, and the dependencies are real.\n\nHybrid · your service in, the rest borrowed\n\nThe catch is that data and queues are shared, so there is always a risk your “sandbox” may break everyone else’s. This is trading cost for safety.\n\n### **6. Shadow**\n\nThe new version runs beside production on a copy of real traffic. Its responses are compared with production’s and then thrown away, so users never see them. Twitter open-sourced this as [Diffy](https://www.infoq.com/news/2015/09/twitter-diffy).\n\nShadow · mirrored traffic, compared and discarded\n\nYou get real inputs and catch regressions nobody wrote a test for. You also get customer data flowing through it, which clashes with the test-accounts rule, and every write and side effect has to be blocked or sent to test storage. This can act like a validation check.\n\n### **Seven designs at a glance**\n\n## **Why sandboxes are hard**\n\nThe idea is simple. Building it is where teams get stuck.\n\n### **Too big for one machine**\n\nWhen the stack won’t fit on one machine, you can start trimming. Smaller data seeds helps. So do lighter stand-ins for the heaviest services. Bonus points if you have well defined service boundaries. If you don’t, you may have to start big.\n\nIt’s rarely just one service needed. A service will depend on other services, which depend on other databases, and on and on. Even a minimal viable sandbox may be hard to achieve.\n\n### **Many repos**\n\nIf you have a single monorepo, cloning may be large but more manageable. \n\nBut often a companies codebase spans across multiple repos. You have some options:\n\n- **Clone them all.** One sandbox checks out every repo on the task’s branch, so the agent can change any of them. It is the most flexible and the slowest to start.[Ona](https://ona.com/docs/ona/configuration/multi-repository) and[GitHub Codespaces](https://github.blog/changelog/2022-04-20-codespaces-multi-repository-and-monorepo-support) both support it.\n- **Build one, pull the rest.** Build the repo you are changing from source and run the others from the images they last published from main. Fast and cheap, but only one repo changes at a time.\n- **Link sandboxes by branch.** Each repo’s PR gets its own sandbox, a shared branch name groups them, and one routing key sends requests through all of them. Optionally consider the PR number from a common repo\n\nmulti-repo sandboxes · 3 patterns\n\n### **Cost**\n\nA full copy of the stack for every open PR, idle most of the day, adds up to a second production bill. Agents make it worse. They open more PRs and leave them open longer.\n\nSo turn sandboxes off when nobody is using them: sleep after a quiet period, wake on the next push, tear down when the PR closes.\n\nPack services together too. You don’t need a database server per service when one server can hold all the databases.\n\n### **How current does it need to be?**\n\nYour own change is always current. What matters is how far behind main everything else is.\n\n## **Making a sandbox useful**\n\nWhichever design you pick, the same three things decide whether an agent can actually check its work in it.\n\n### **The data**\n\nWithout data, your app isn’t testable. Real bugs show up in specific states, like a huge account or an expired trial or one missing field, so each change needs its own scenario.\n\nI’d combine two sources.\n\n**A nightly copy from a test account.** Most companies already have QA, demo and internal orgs used for testing or development. Dump those from a read replica, following foreign keys so related rows come along. Then publish a dated snapshot that each sandbox restores copy-on-write.\n\ndaily data dump · test accounts to sandboxes\n\nCustomer data should never make it into the snapshot.\n\n**Seeded scenarios**. Describe the required data in plain English, turn it into a reusable template, and create the data through the product’s own APIs so it’s valid.\n\nseeding from a prompt · prompt, plan, API calls\n\n**Data processing.** Sometimes rows aren’t enough. If your product runs on events, streams or batch jobs, the sandbox needs a small version of that pipeline to process that data as well. That may mean running a data pipeline on your seeded data to produce the write output values.\n\n### **Access**\n\nAn agent stuck at the login screen can’t check anything. Give sandboxes a dev-only login that skips SSO and captchas, and let the agent sign in as any seeded role: admin, viewer, a new user.\n\nPasswords shouldn’t matter much in a sandbox - it’s a playground. Mock out your calls to password services.\n\n### **Receipts**\n\nEvery PR should come with **receipts** - evidence the changes work. Screenshots or a recording of the pages the agent visited, the API calls and their responses, server logs, the rows it read and wrote. A reviewer can skim that in a minute, or click through to a preview environment - which beats pulling the branch and running it.\n\nRead the sessions themselves. Each place an agent couldn’t run something, or hacked around it, goes on a checklist ranked by how often it comes up. The platform team works down from the top, and every fix helps every session after it.\n\nsession review · gaps become a ranked checklist\n\n## **Where to start**\n\nHumans likely already have ways to test and validate most changes - you just need to find a way to allow agents to run similar processes and view the results.\n\nYou don’t have to build all of it at once. Pick the design that fits your stack; if it fits on one machine, that’s probably Ephemeral. Wall it off from real data and production services. Load it with test-account data and give the agent a way to log in. Have the harness record what the agent actually did.\n\nThen read the sessions. The first few will show you exactly where agents get stuck, and the most common blocker is your next piece of work.", "url": "https://wpnews.pro/news/sandboxes-for-ai-agents", "canonical_source": "https://shmck.substack.com/p/sandboxes-for-ai-agents", "published_at": "2026-11-15 22:02:01+00:00", "updated_at": "2026-11-15 22:30:47.552633+00:00", "lang": "en", "topics": ["ai-agents", "ai-tools", "developer-tools", "mlops", "ai-infrastructure"], "entities": ["Stripe", "Minions", "Ramp", "Modal", "LocalStack"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/sandboxes-for-ai-agents", "markdown": "https://wpnews.pro/news/sandboxes-for-ai-agents.md", "text": "https://wpnews.pro/news/sandboxes-for-ai-agents.txt", "jsonld": "https://wpnews.pro/news/sandboxes-for-ai-agents.jsonld"}}