# Building an Autonomous A/B Testing Agent on Docker Cloud Sandboxes > Source: > Published: 2026-10-12 06:04:38+00:00 Have you ever wondered if an AI agent could run an A/B test all by itself? In this post, we will build one together. We will go slowly, one file at a time, so you can follow along even if you have never used Docker Cloud Sandboxes before. Full code: [github.com/ajeetraina/sbx-ab-agent](https://github.com/ajeetraina/sbx-ab-agent) Think of a small online shop. Some people say it is hard to find things in it. We want to fix that, but we are not sure which fix is best. So we do this: Here is the same idea as a picture: ``` ┌──────────── Docker Cloud Sandboxes ────────────┐ npm start ──► │ A original shop → shoppers test → analyst │ │ B new sandbox → builder adds fix B → re-test │ │ C new sandbox → builder adds fix C → re-test │ └────────────────────────────────────────────────┘ │ ▼ winner + report + winning code ``` "A" is the original shop. "B" and "C" are the new versions. That is all A/B testing really means: try a few versions and see which one works better. Our builder agent will write and run code that nobody has checked. You do not want that running on your laptop. A Docker Cloud Sandbox is a small, throwaway computer in the cloud (a microVM). It gives us four things we need: We control all of this from a small TypeScript program using the `@docker/sandboxes` SDK. ``` git clone https://github.com/ajeetraina/sbx-ab-agent cd sbx-ab-agent npm install ``` The project is small. Here is a map: ``` app/ the demo shop server.mjs a tiny web server index.html the page app.js shows the products styles.css the look products.json the product list src/ sbx.ts helpers that talk to Docker Cloud Sandboxes prompts.ts the instructions for each AI agent ab.ts the main program that runs every step dashboard.ts a live web page to watch the run cleanup.ts deletes the sandboxes when you are done ``` A good reading order is: `app/` first, then `prompts.ts`, then `sbx.ts`, then `ab.ts`. That is the order we will follow. `app/`) The shop shows a long list of products. That is it. There is no search, no filter and no sort. This is on purpose. It is the problem we want our agents to find and fix. Here is the code that draws the products, from `app/app.js`: ``` js async function main() { const products = await (await fetch('/products.json')).json(); render(products); } ``` It loads every product and puts them all on the page. Nothing more. Now look at the server in `app/server.mjs`. The key line is this one: ``` js const body = await readFile(file); ``` The server reads the file from disk every time someone asks for it. It never keeps an old copy. This matters later: when the builder agent changes `app.js`, the live site changes right away. No restart needed. Try the shop on your own machine: ``` npm run shop # open http://localhost:3000 ``` `src/prompts.ts`) We have three roles. They all use the same AI (Codex). The only difference is the instructions (the prompt) we give each one. We have three shoppers, each with a simple goal: | Shopper | Who they are | What they want | |---|---|---| | `budget-student` | a student with about $20 | a gift under $20 for a developer friend | | `hoodie-hunter` | a busy engineer | all hoodies in size L, then the cheapest one | | `sticker-collector` | a sticker fan | see every sticker and compare prices | In code, a shopper is just a small object: ``` { id: 'hoodie-hunter', who: 'a busy engineer who already knows they want a hoodie in size L', task: 'find all hoodies available in size L and pick the cheapest one', } ``` The shopper is told: "You are a real person, not a developer. Do not edit any files." It looks at the shop using `curl` and imagines the clicks and scrolling a real person would need. Most important: the shopper must answer in JSON, not in a paragraph. That way our program can read the answer easily. ``` { "persona": "hoodie-hunter", "completed": true, "steps": 14, "score": 4, "friction": ["had to scroll the whole list; no size filter"], "quote": "I found it, but I had to look at everything." } ``` `score` goes from 1 to 10. A 10 means the task was very easy. The analyst reads all the shopper reports and suggests a few fixes. The rules we give it: `index.html`, `app.js` and `styles.css`. It also answers in JSON: ``` { "summary": "Shoppers scroll the whole list. There is no way to narrow it down.", "variants": [ { "id": "B", "title": "Search box and category filters", "hypothesis": "...", "spec": "..." }, { "id": "C", "title": "Sort by price", "hypothesis": "...", "spec": "..." } ] } ``` The builder gets one fix (for example "Sort by price") and writes the code. It has strict rules too: `curl`. These rules keep every version fair. Each version is the same shop plus one small change. `src/sbx.ts`) Now we start talking to Docker Cloud Sandboxes. Everything in this file is a small helper we will call later. ``` js const auth = oauth({ store: fileOAuthCredentialStore({ path: '.docker-sandboxes/oauth.json' }), onVerification: ({ verificationUriComplete, verificationUri, userCode }) => { console.log(`Sign in to Docker: ${verificationUriComplete ?? verificationUri}`); console.log(`Code: ${userCode}`); }, }); await auth.getAccessToken(); return new Sandboxes({ auth }); ``` In plain words: `new Sandboxes({ auth })` gives us a client. We use this client for everything else. If you prefer no browser step, set `DOCKER_USERNAME` and `DOCKER_PAT` (a personal access token). The code then uses `pat()` instead. ``` js const secret = await client.secrets.create({ displayName: 'sbx-ab-agent-openai', serviceType: 'openai', token: { value: key }, }); ``` This saves your OpenAI key in Docker, not inside any sandbox. Later we attach the secret to each sandbox. When Codex calls OpenAI, the sandbox adds the key to the request on the way out. Codex never sees the key itself. One small thing to remember: the SDK does not attach your OpenAI secret for you. You must pass it in when you create a sandbox. We will do that in the next step. `launchShop`) This one function does five small jobs. Let us take them one by one. ``` js const sandbox = await client.kits.launchAndWait('codex', { displayName, resources: 'small', storage: { secrets: [secret] }, }); ``` `'codex'` is a kit: a ready-made sandbox that already has Codex installed. `launchAndWait` creates it and waits until it is ready. Notice we pass our secret here. ``` js await sandbox.files.mkdir(APP_DIR, { mode: 0o755, parents: true }); for (const f of await readdir(localAppDir)) { await sandbox.files.write(`${APP_DIR}/${f}`, await readFile(join(localAppDir, f), 'utf8')); } ``` Make a folder in the sandbox, then copy each file from your `app/` folder into it. ``` await sandbox.processes.start({ args: ['node', 'server.mjs'], workingDir: APP_DIR, env: { PORT: '3000' }, label: 'shop-server', }); ``` `processes.start` runs a command in the background. The server keeps running while we do other things. The `waitForHttp` helper uses `sandbox.run(...)` to try the server once a second, up to 30 times. When it gets a good answer, we move on. ``` js const port = await sandbox.ports.create({ number: 3000, protocol: 'tcp' }); return { sandbox, url: port.url }; ``` This opens port 3000 to the internet and gives back a real HTTPS URL. You can open it in your browser. That is it. One function, and we have a live shop running in the cloud. `runAgent`) Every AI helper (shopper, analyst, builder) runs through this one function. It runs Codex inside the sandbox with no chat window, just a prompt in and an answer out. ``` js const script = `printf '%s' "$1" | codex exec -C ${APP_DIR} --skip-git-repo-check --ephemeral ` + `--dangerously-bypass-approvals-and-sandbox -o ${outFile} -`; const r = await sandbox.processes.run( { args: ['bash', '-lc', script, 'bash', prompt], workingDir: APP_DIR }, { timeoutMs }, ); ``` This looks scary, so let us break it down: `codex exec` runs Codex once, without asking us questions.`printf ... | ... -` sends the prompt into Codex. The `-` at the end means "read the prompt from input". If you pass the prompt another way, Codex waits and then fails.`--dangerously-bypass-approvals-and-sandbox` turns off Codex's own safety box. That is fine here, because we are already inside a throwaway sandbox.`-o ${outFile}` saves only Codex's final answer to a file. After it finishes, we read the answer back: ``` return await sandbox.files.read(outFile, { encoding: 'utf8' }); ``` AI answers are not always clean. Sometimes the JSON comes wrapped in a code block, or with a sentence before it. The `extractJson` helper finds the first { and the last } and reads what is between them. Simple, and it works well. `src/ab.ts`) Now the main program. It just calls the helpers above in order. Read `main()` from top to bottom and you will see these steps: ``` js const base = await launchShop(client, 'ab-A-baseline', secret, 'app'); js const baseSessions = await simulate(base.sandbox, 'A', PERSONAS); ``` `simulate` runs all three shoppers at the same time with `Promise.all`. Each one returns its JSON report. ``` js const plan = extractJson( await runAgent(base.sandbox, analystPrompt(GOAL, baseSessions, N_VARIANTS)), ); js const settled = await Promise.allSettled( variants.map((v) => runVariant(client, secret, v)), ); ``` For each fix, `runVariant` does three things: We use `Promise.allSettled` instead of `Promise.all`. The difference: if one fix fails, the others still finish. One bad version does not ruin the whole run. ``` js const ranked = [...arms].sort((a, b) => b.avgScore - a.avgScore || b.completion - a.completion || a.avgSteps - b.avgSteps); ``` In plain words: highest average score wins. If two are tied, the one where more shoppers finished wins. If still tied, fewer steps wins. ``` js const body = await winner.sandbox.files.read(`${APP_DIR}/${f}`, { encoding: 'utf8' }); await writeFile(`results/winner/${f}`, body); ``` The winning files are copied from the sandbox to `results/winner/` on your laptop. The program also writes `results/report.md` and `results/report.json` with all the scores and shopper quotes. Set up your key: ``` cp .env.example .env # open .env and add: OPENAI_API_KEY=sk-... set -a; source .env; set +a ``` Then start the run: ``` npm start ``` The first time, you will see a Docker sign-in link. Open it, sign in, and the run continues. A normal run (2 fixes, 3 shoppers) uses 3 small sandboxes and about 12 Codex calls. It takes around 5 to 10 minutes. Some handy options: ``` npm start -- --variants 3 --personas 2 npm start -- --goal "Users can't tell which items are on sale" npm start -- --cleanup # delete the sandboxes at the end npm start -- --no-dashboard # skip the live web page ``` While it runs, open [http://localhost:4000](http://localhost:4000). This is the live dashboard (`src/dashboard.ts`). You will see each step light up, and a box for A, B and C showing scores as they come in. The terminal also prints a public link for each version. Open A, B and C in three tabs. You can watch B and C change while the builder is still working. When it finishes, you get a table like this: | Arm | Variant | Ease | Done | Steps | Lift | |---|---|---|---|---|---| | A | Baseline | 4.0 | 67% | 14.0 | - | | B (winner) | Search and filters | 8.3 | 100% | 4.0 | +108% | | C | Sort by price | 6.0 | 100% | 8.0 | +50% | (Your numbers will be different. Every run is a little different.) "Lift" means how much better the score is compared to A. To see exactly what the AI changed, compare the files: ``` diff -u app/app.js results/winner/app.js ``` That diff is basically your pull request. A quick note on ports, since there are a few: | URL | Where | What | |---|---|---| | `http://localhost:4000` | your laptop | the live dashboard | | `https://3000-....sbx.sandboxes-cloud.docker.com` | the cloud | each version of the shop | | `http://localhost:3000` | your laptop | only when you run npm run shop | By default, the sandboxes stay running so you can click around each version. When you are done: ``` npm run cleanup ``` This deletes every sandbox whose name starts with ab-. It uses `client.allSandboxes()` to list them and `delete({ force: true })` to remove each one. | What we wanted | SDK call | |---|---| | Sign in | `oauth()` or`pat()` | | Save the OpenAI key | `client.secrets.create(...)` | | Create a sandbox | `client.kits.launchAndWait('codex', ...)` | | Copy files in and out | `sandbox.files.mkdir` /`files.write` /`files.read` | | Start a server in the background | `sandbox.processes.start(...)` | | Run a quick command | `sandbox.run([...])` | | Run a command and wait for it | `sandbox.processes.run(...)` | | Get a public URL | `sandbox.ports.create({ number: 3000 })` | | Clean up | `client.allSandboxes()` ,`sandbox.delete(...)` | That is the whole SDK surface for this project. About ten calls. `OPENAI_API_KEY`. Check your `.env` and run set -a; source `Promise.allSettled`.` npm run cleanup`.`@docker/sandboxes` 0.53.0, so some names may change later. You now know how to: The shop is just an example. The same pattern works for many things: testing prompts, tuning settings, trying different copy. Anywhere you can pretend to be a user and give a score, you can use this loop. The big idea is simple. When a safe, throwaway computer is just one function call away, letting an agent "try it and see" stops being risky. Give it a try, and let me know what you build with it!