{"slug": "building-an-autonomous-a-b-testing-agent-on-docker-cloud-sandboxes", "title": "Building an Autonomous A/B Testing Agent on Docker Cloud Sandboxes", "summary": "A developer built an autonomous A/B testing agent that uses Docker Cloud Sandboxes to spin up isolated microVMs where AI agents generate, test, and compare alternative versions of a demo e-commerce shop. The project, published on GitHub as sbx-ab-agent, uses the @docker/sandboxes TypeScript SDK and Codex-based agents to run shopper simulations, pick a winning variant, and return the winning code.", "body_md": "Have you ever wondered if an AI agent could run an A/B test all by itself?\n\nIn this post, we will build one together. We will go slowly, one file at a time, so you can follow along even if you have never used Docker Cloud Sandboxes before.\n\nFull code: [github.com/ajeetraina/sbx-ab-agent](https://github.com/ajeetraina/sbx-ab-agent)\n\nThink of a small online shop. Some people say it is hard to find things in it. We want to fix that, but we are not sure which fix is best.\n\nSo we do this:\n\nHere is the same idea as a picture:\n\n```\n                ┌──────────── Docker Cloud Sandboxes ────────────┐\nnpm start ──►   │ A  original shop  → shoppers test → analyst    │\n                │ B  new sandbox → builder adds fix B → re-test  │\n                │ C  new sandbox → builder adds fix C → re-test  │\n                └────────────────────────────────────────────────┘\n                                   │\n                                   ▼\n                    winner + report + winning code\n```\n\n\"A\" is the original shop. \"B\" and \"C\" are the new versions. That is all A/B testing really means: try a few versions and see which one works better.\n\nOur builder agent will write and run code that nobody has checked. You do not want that running on your laptop.\n\nA Docker Cloud Sandbox is a small, throwaway computer in the cloud (a microVM). It gives us four things we need:\n\nWe control all of this from a small TypeScript program using the `@docker/sandboxes` SDK.\n\n```\ngit clone https://github.com/ajeetraina/sbx-ab-agent\ncd sbx-ab-agent\nnpm install\n```\n\nThe project is small. Here is a map:\n\n```\napp/                 the demo shop\n  server.mjs         a tiny web server\n  index.html         the page\n  app.js             shows the products\n  styles.css         the look\n  products.json      the product list\nsrc/\n  sbx.ts             helpers that talk to Docker Cloud Sandboxes\n  prompts.ts         the instructions for each AI agent\n  ab.ts              the main program that runs every step\n  dashboard.ts       a live web page to watch the run\n  cleanup.ts         deletes the sandboxes when you are done\n```\n\nA good reading order is: `app/` first, then `prompts.ts`, then `sbx.ts`, then `ab.ts`. That is the order we will follow.\n\n`app/`)\nThe shop shows a long list of products. That is it. There is no search, no filter and no sort. This is on purpose. It is the problem we want our agents to find and fix.\n\nHere is the code that draws the products, from `app/app.js`:\n\n``` js\nasync function main() {\n  const products = await (await fetch('/products.json')).json();\n  render(products);\n}\n```\n\nIt loads every product and puts them all on the page. Nothing more.\n\nNow look at the server in `app/server.mjs`. The key line is this one:\n\n``` js\nconst body = await readFile(file);\n```\n\nThe server reads the file from disk every time someone asks for it. It never keeps an old copy. This matters later: when the builder agent changes `app.js`, the live site changes right away. No restart needed.\n\nTry the shop on your own machine:\n\n```\nnpm run shop\n# open http://localhost:3000\n```\n\n`src/prompts.ts`)\nWe have three roles. They all use the same AI (Codex). The only difference is the instructions (the prompt) we give each one.\n\nWe have three shoppers, each with a simple goal:\n\n| Shopper | Who they are | What they want | \n|---|---|---|\n| `budget-student` | a student with about $20 | a gift under $20 for a developer friend | \n| `hoodie-hunter` | a busy engineer | all hoodies in size L, then the cheapest one | \n| `sticker-collector` | a sticker fan | see every sticker and compare prices | \n\nIn code, a shopper is just a small object:\n\n```\n{\n  id: 'hoodie-hunter',\n  who: 'a busy engineer who already knows they want a hoodie in size L',\n  task: 'find all hoodies available in size L and pick the cheapest one',\n}\n```\n\nThe shopper is told: \"You are a real person, not a developer. Do not edit any files.\" It looks at the shop using `curl` and imagines the clicks and scrolling a real person would need.\n\nMost important: the shopper must answer in JSON, not in a paragraph. That way our program can read the answer easily.\n\n```\n{\n  \"persona\": \"hoodie-hunter\",\n  \"completed\": true,\n  \"steps\": 14,\n  \"score\": 4,\n  \"friction\": [\"had to scroll the whole list; no size filter\"],\n  \"quote\": \"I found it, but I had to look at everything.\"\n}\n```\n\n`score` goes from 1 to 10. A 10 means the task was very easy.\n\nThe analyst reads all the shopper reports and suggests a few fixes. The rules we give it:\n\n`index.html`, `app.js` and `styles.css`.\nIt also answers in JSON:\n\n```\n{\n  \"summary\": \"Shoppers scroll the whole list. There is no way to narrow it down.\",\n  \"variants\": [\n    { \"id\": \"B\", \"title\": \"Search box and category filters\", \"hypothesis\": \"...\", \"spec\": \"...\" },\n    { \"id\": \"C\", \"title\": \"Sort by price\", \"hypothesis\": \"...\", \"spec\": \"...\" }\n  ]\n}\n```\n\nThe builder gets one fix (for example \"Sort by price\") and writes the code. It has strict rules too:\n\n`curl`.\nThese rules keep every version fair. Each version is the same shop plus one small change.\n\n`src/sbx.ts`)\nNow we start talking to Docker Cloud Sandboxes. Everything in this file is a small helper we will call later.\n\n``` js\nconst auth = oauth({\n  store: fileOAuthCredentialStore({ path: '.docker-sandboxes/oauth.json' }),\n  onVerification: ({ verificationUriComplete, verificationUri, userCode }) => {\n    console.log(`Sign in to Docker: ${verificationUriComplete ?? verificationUri}`);\n    console.log(`Code: ${userCode}`);\n  },\n});\nawait auth.getAccessToken();\nreturn new Sandboxes({ auth });\n```\n\nIn plain words:\n\n`new Sandboxes({ auth })` gives us a client. We use this client for everything else.\nIf you prefer no browser step, set `DOCKER_USERNAME` and `DOCKER_PAT` (a personal access token). The code then uses `pat()` instead.\n\n``` js\nconst secret = await client.secrets.create({\n  displayName: 'sbx-ab-agent-openai',\n  serviceType: 'openai',\n  token: { value: key },\n});\n```\n\nThis saves your OpenAI key in Docker, not inside any sandbox. Later we attach the secret to each sandbox. When Codex calls OpenAI, the sandbox adds the key to the request on the way out. Codex never sees the key itself.\n\nOne small thing to remember: the SDK does not attach your OpenAI secret for you. You must pass it in when you create a sandbox. We will do that in the next step.\n\n`launchShop`)\nThis one function does five small jobs. Let us take them one by one.\n\n``` js\nconst sandbox = await client.kits.launchAndWait('codex', {\n  displayName,\n  resources: 'small',\n  storage: { secrets: [secret] },\n});\n```\n\n`'codex'` is a kit: a ready-made sandbox that already has Codex installed. `launchAndWait` creates it and waits until it is ready. Notice we pass our secret here.\n\n``` js\nawait sandbox.files.mkdir(APP_DIR, { mode: 0o755, parents: true });\nfor (const f of await readdir(localAppDir)) {\n  await sandbox.files.write(`${APP_DIR}/${f}`, await readFile(join(localAppDir, f), 'utf8'));\n}\n```\n\nMake a folder in the sandbox, then copy each file from your `app/` folder into it.\n\n```\nawait sandbox.processes.start({\n  args: ['node', 'server.mjs'],\n  workingDir: APP_DIR,\n  env: { PORT: '3000' },\n  label: 'shop-server',\n});\n```\n\n`processes.start` runs a command in the background. The server keeps running while we do other things.\n\nThe `waitForHttp` helper uses `sandbox.run(...)` to try the server once a second, up to 30 times. When it gets a good answer, we move on.\n\n``` js\nconst port = await sandbox.ports.create({ number: 3000, protocol: 'tcp' });\nreturn { sandbox, url: port.url };\n```\n\nThis opens port 3000 to the internet and gives back a real HTTPS URL. You can open it in your browser.\n\nThat is it. One function, and we have a live shop running in the cloud.\n\n`runAgent`)\nEvery AI helper (shopper, analyst, builder) runs through this one function. It runs Codex inside the sandbox with no chat window, just a prompt in and an answer out.\n\n``` js\nconst script =\n  `printf '%s' \"$1\" | codex exec -C ${APP_DIR} --skip-git-repo-check --ephemeral ` +\n  `--dangerously-bypass-approvals-and-sandbox -o ${outFile} -`;\n\nconst r = await sandbox.processes.run(\n  { args: ['bash', '-lc', script, 'bash', prompt], workingDir: APP_DIR },\n  { timeoutMs },\n);\n```\n\nThis looks scary, so let us break it down:\n\n`codex exec` runs Codex once, without asking us questions.`printf ... | ... -` sends the prompt into Codex. The `-` at the end means \"read the prompt from input\". If you pass the prompt another way, Codex waits and then fails.`--dangerously-bypass-approvals-and-sandbox` turns off Codex's own safety box. That is fine here, because we are already inside a throwaway sandbox.`-o ${outFile}` saves only Codex's final answer to a file.\nAfter it finishes, we read the answer back:\n\n```\nreturn await sandbox.files.read(outFile, { encoding: 'utf8' });\n```\n\nAI answers are not always clean. Sometimes the JSON comes wrapped in a code block, or with a sentence before it. The `extractJson` helper finds the first { and the last } and reads what is between them. Simple, and it works well.\n\n`src/ab.ts`)\nNow the main program. It just calls the helpers above in order. Read `main()` from top to bottom and you will see these steps:\n\n``` js\nconst base = await launchShop(client, 'ab-A-baseline', secret, 'app');\njs\nconst baseSessions = await simulate(base.sandbox, 'A', PERSONAS);\n```\n\n`simulate` runs all three shoppers at the same time with `Promise.all`. Each one returns its JSON report.\n\n``` js\nconst plan = extractJson(\n  await runAgent(base.sandbox, analystPrompt(GOAL, baseSessions, N_VARIANTS)),\n);\njs\nconst settled = await Promise.allSettled(\n  variants.map((v) => runVariant(client, secret, v)),\n);\n```\n\nFor each fix, `runVariant` does three things:\n\nWe use `Promise.allSettled` instead of `Promise.all`. The difference: if one fix fails, the others still finish. One bad version does not ruin the whole run.\n\n``` js\nconst ranked = [...arms].sort((a, b) =>\n  b.avgScore - a.avgScore || b.completion - a.completion || a.avgSteps - b.avgSteps);\n```\n\nIn plain words: highest average score wins. If two are tied, the one where more shoppers finished wins. If still tied, fewer steps wins.\n\n``` js\nconst body = await winner.sandbox.files.read(`${APP_DIR}/${f}`, { encoding: 'utf8' });\nawait writeFile(`results/winner/${f}`, body);\n```\n\nThe winning files are copied from the sandbox to `results/winner/` on your laptop. The program also writes `results/report.md` and `results/report.json` with all the scores and shopper quotes.\n\nSet up your key:\n\n```\ncp .env.example .env\n# open .env and add: OPENAI_API_KEY=sk-...\nset -a; source .env; set +a\n```\n\nThen start the run:\n\n```\nnpm start\n```\n\nThe first time, you will see a Docker sign-in link. Open it, sign in, and the run continues.\n\nA normal run (2 fixes, 3 shoppers) uses 3 small sandboxes and about 12 Codex calls. It takes around 5 to 10 minutes.\n\nSome handy options:\n\n```\nnpm start -- --variants 3 --personas 2\nnpm start -- --goal \"Users can't tell which items are on sale\"\nnpm start -- --cleanup          # delete the sandboxes at the end\nnpm start -- --no-dashboard     # skip the live web page\n```\n\nWhile it runs, open [http://localhost:4000](http://localhost:4000). This is the live dashboard (`src/dashboard.ts`). You will see each step light up, and a box for A, B and C showing scores as they come in.\n\nThe terminal also prints a public link for each version. Open A, B and C in three tabs. You can watch B and C change while the builder is still working.\n\nWhen it finishes, you get a table like this:\n\n| Arm | Variant | Ease | Done | Steps | Lift | \n|---|---|---|---|---|---|\n| A | Baseline | 4.0 | 67% | 14.0 | - | \n| B (winner) | Search and filters | 8.3 | 100% | 4.0 | +108% | \n| C | Sort by price | 6.0 | 100% | 8.0 | +50% | \n\n(Your numbers will be different. Every run is a little different.)\n\n\"Lift\" means how much better the score is compared to A.\n\nTo see exactly what the AI changed, compare the files:\n\n```\ndiff -u app/app.js results/winner/app.js\n```\n\nThat diff is basically your pull request.\n\nA quick note on ports, since there are a few:\n\n| URL | Where | What | \n|---|---|---|\n| `http://localhost:4000` | your laptop | the live dashboard | \n| `https://3000-....sbx.sandboxes-cloud.docker.com` | the cloud | each version of the shop | \n| `http://localhost:3000` | your laptop | only when you run npm run shop | \n\nBy default, the sandboxes stay running so you can click around each version. When you are done:\n\n```\nnpm run cleanup\n```\n\nThis deletes every sandbox whose name starts with ab-. It uses `client.allSandboxes()` to list them and `delete({ force: true })` to remove each one.\n\n| What we wanted | SDK call | \n|---|---|\n| Sign in | `oauth()` or`pat()` | \n| Save the OpenAI key | `client.secrets.create(...)` | \n| Create a sandbox | `client.kits.launchAndWait('codex', ...)` | \n| Copy files in and out | `sandbox.files.mkdir` /`files.write` /`files.read` | \n| Start a server in the background | `sandbox.processes.start(...)` | \n| Run a quick command | `sandbox.run([...])` | \n| Run a command and wait for it | `sandbox.processes.run(...)` | \n| Get a public URL | `sandbox.ports.create({ number: 3000 })` | \n| Clean up | `client.allSandboxes()` ,`sandbox.delete(...)` | \n\nThat is the whole SDK surface for this project. About ten calls.\n\n`OPENAI_API_KEY`. Check your `.env` and run set -a; source `Promise.allSettled`.` npm run cleanup`.`@docker/sandboxes` 0.53.0, so some names may change later.\nYou now know how to:\n\nThe shop is just an example. The same pattern works for many things: testing prompts, tuning settings, trying different copy. Anywhere you can pretend to be a user and give a score, you can use this loop.\n\nThe big idea is simple. When a safe, throwaway computer is just one function call away, letting an agent \"try it and see\" stops being risky.\n\nGive it a try, and let me know what you build with it!", "url": "https://wpnews.pro/news/building-an-autonomous-a-b-testing-agent-on-docker-cloud-sandboxes", "canonical_source": "https://dev.to/ajeetraina/building-an-autonomous-ab-testing-agent-on-docker-cloud-sandboxes-5926", "published_at": "2026-10-12 06:04:38+00:00", "updated_at": "2026-10-12 06:10:17.364298+00:00", "lang": "en", "topics": ["ai-agents", "ai-tools", "developer-tools", "ai-infrastructure", "mlops"], "entities": ["Docker", "Docker Cloud Sandboxes", "Codex", "GitHub", "@docker/sandboxes"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/building-an-autonomous-a-b-testing-agent-on-docker-cloud-sandboxes", "markdown": "https://wpnews.pro/news/building-an-autonomous-a-b-testing-agent-on-docker-cloud-sandboxes.md", "text": "https://wpnews.pro/news/building-an-autonomous-a-b-testing-agent-on-docker-cloud-sandboxes.txt", "jsonld": "https://wpnews.pro/news/building-an-autonomous-a-b-testing-agent-on-docker-cloud-sandboxes.jsonld"}}