Building an Autonomous A/B Testing Agent on Docker Cloud Sandboxes

A developer built an autonomous A/B testing agent that uses Docker Cloud Sandboxes to spin up isolated microVMs where AI agents generate, test, and compare alternative versions of a demo e-commerce shop. The project, published on GitHub as sbx-ab-agent, uses the @docker/sandboxes TypeScript SDK and Codex-based agents to run shopper simulations, pick a winning variant, and return the winning code.

Have you ever wondered if an AI agent could run an A/B test all by itself? In this post, we will build one together. We will go slowly, one file at a time, so you can follow along even if you have never used Docker Cloud Sandboxes before. Full code: github.com/ajeetraina/sbx-ab-agent https://github.com/ajeetraina/sbx-ab-agent Think of a small online shop. Some people say it is hard to find things in it. We want to fix that, but we are not sure which fix is best. So we do this: Here is the same idea as a picture: ┌──────────── Docker Cloud Sandboxes ────────────┐ npm start ──► │ A original shop → shoppers test → analyst │ │ B new sandbox → builder adds fix B → re-test │ │ C new sandbox → builder adds fix C → re-test │ └────────────────────────────────────────────────┘ │ ▼ winner + report + winning code "A" is the original shop. "B" and "C" are the new versions. That is all A/B testing really means: try a few versions and see which one works better. Our builder agent will write and run code that nobody has checked. You do not want that running on your laptop. A Docker Cloud Sandbox is a small, throwaway computer in the cloud a microVM . It gives us four things we need: We control all of this from a small TypeScript program using the @docker/sandboxes SDK. git clone https://github.com/ajeetraina/sbx-ab-agent cd sbx-ab-agent npm install The project is small. Here is a map: app/ the demo shop server.mjs a tiny web server index.html the page app.js shows the products styles.css the look products.json the product list src/ sbx.ts helpers that talk to Docker Cloud Sandboxes prompts.ts the instructions for each AI agent ab.ts the main program that runs every step dashboard.ts a live web page to watch the run cleanup.ts deletes the sandboxes when you are done A good reading order is: app/ first, then prompts.ts , then sbx.ts , then ab.ts . That is the order we will follow. app/ The shop shows a long list of products. That is it. There is no search, no filter and no sort. This is on purpose. It is the problem we want our agents to find and fix. Here is the code that draws the products, from app/app.js : js async function main { const products = await await fetch '/products.json' .json ; render products ; } It loads every product and puts them all on the page. Nothing more. Now look at the server in app/server.mjs . The key line is this one: js const body = await readFile file ; The server reads the file from disk every time someone asks for it. It never keeps an old copy. This matters later: when the builder agent changes app.js , the live site changes right away. No restart needed. Try the shop on your own machine: npm run shop open http://localhost:3000 src/prompts.ts We have three roles. They all use the same AI Codex . The only difference is the instructions the prompt we give each one. We have three shoppers, each with a simple goal: | Shopper | Who they are | What they want | |---|---|---| | budget-student | a student with about $20 | a gift under $20 for a developer friend | | hoodie-hunter | a busy engineer | all hoodies in size L, then the cheapest one | | sticker-collector | a sticker fan | see every sticker and compare prices | In code, a shopper is just a small object: { id: 'hoodie-hunter', who: 'a busy engineer who already knows they want a hoodie in size L', task: 'find all hoodies available in size L and pick the cheapest one', } The shopper is told: "You are a real person, not a developer. Do not edit any files." It looks at the shop using curl and imagines the clicks and scrolling a real person would need. Most important: the shopper must answer in JSON, not in a paragraph. That way our program can read the answer easily. { "persona": "hoodie-hunter", "completed": true, "steps": 14, "score": 4, "friction": "had to scroll the whole list; no size filter" , "quote": "I found it, but I had to look at everything." } score goes from 1 to 10. A 10 means the task was very easy. The analyst reads all the shopper reports and suggests a few fixes. The rules we give it: index.html , app.js and styles.css . It also answers in JSON: { "summary": "Shoppers scroll the whole list. There is no way to narrow it down.", "variants": { "id": "B", "title": "Search box and category filters", "hypothesis": "...", "spec": "..." }, { "id": "C", "title": "Sort by price", "hypothesis": "...", "spec": "..." } } The builder gets one fix for example "Sort by price" and writes the code. It has strict rules too: curl . These rules keep every version fair. Each version is the same shop plus one small change. src/sbx.ts Now we start talking to Docker Cloud Sandboxes. Everything in this file is a small helper we will call later. js const auth = oauth { store: fileOAuthCredentialStore { path: '.docker-sandboxes/oauth.json' } , onVerification: { verificationUriComplete, verificationUri, userCode } = { console.log Sign in to Docker: ${verificationUriComplete ?? verificationUri} ; console.log Code: ${userCode} ; }, } ; await auth.getAccessToken ; return new Sandboxes { auth } ; In plain words: new Sandboxes { auth } gives us a client. We use this client for everything else. If you prefer no browser step, set DOCKER USERNAME and DOCKER PAT a personal access token . The code then uses pat instead. js const secret = await client.secrets.create { displayName: 'sbx-ab-agent-openai', serviceType: 'openai', token: { value: key }, } ; This saves your OpenAI key in Docker, not inside any sandbox. Later we attach the secret to each sandbox. When Codex calls OpenAI, the sandbox adds the key to the request on the way out. Codex never sees the key itself. One small thing to remember: the SDK does not attach your OpenAI secret for you. You must pass it in when you create a sandbox. We will do that in the next step. launchShop This one function does five small jobs. Let us take them one by one. js const sandbox = await client.kits.launchAndWait 'codex', { displayName, resources: 'small', storage: { secrets: secret }, } ; 'codex' is a kit: a ready-made sandbox that already has Codex installed. launchAndWait creates it and waits until it is ready. Notice we pass our secret here. js await sandbox.files.mkdir APP DIR, { mode: 0o755, parents: true } ; for const f of await readdir localAppDir { await sandbox.files.write ${APP DIR}/${f} , await readFile join localAppDir, f , 'utf8' ; } Make a folder in the sandbox, then copy each file from your app/ folder into it. await sandbox.processes.start { args: 'node', 'server.mjs' , workingDir: APP DIR, env: { PORT: '3000' }, label: 'shop-server', } ; processes.start runs a command in the background. The server keeps running while we do other things. The waitForHttp helper uses sandbox.run ... to try the server once a second, up to 30 times. When it gets a good answer, we move on. js const port = await sandbox.ports.create { number: 3000, protocol: 'tcp' } ; return { sandbox, url: port.url }; This opens port 3000 to the internet and gives back a real HTTPS URL. You can open it in your browser. That is it. One function, and we have a live shop running in the cloud. runAgent Every AI helper shopper, analyst, builder runs through this one function. It runs Codex inside the sandbox with no chat window, just a prompt in and an answer out. js const script = printf '%s' "$1" | codex exec -C ${APP DIR} --skip-git-repo-check --ephemeral + --dangerously-bypass-approvals-and-sandbox -o ${outFile} - ; const r = await sandbox.processes.run { args: 'bash', '-lc', script, 'bash', prompt , workingDir: APP DIR }, { timeoutMs }, ; This looks scary, so let us break it down: codex exec runs Codex once, without asking us questions. printf ... | ... - sends the prompt into Codex. The - at the end means "read the prompt from input". If you pass the prompt another way, Codex waits and then fails. --dangerously-bypass-approvals-and-sandbox turns off Codex's own safety box. That is fine here, because we are already inside a throwaway sandbox. -o ${outFile} saves only Codex's final answer to a file. After it finishes, we read the answer back: return await sandbox.files.read outFile, { encoding: 'utf8' } ; AI answers are not always clean. Sometimes the JSON comes wrapped in a code block, or with a sentence before it. The extractJson helper finds the first { and the last } and reads what is between them. Simple, and it works well. src/ab.ts Now the main program. It just calls the helpers above in order. Read main from top to bottom and you will see these steps: js const base = await launchShop client, 'ab-A-baseline', secret, 'app' ; js const baseSessions = await simulate base.sandbox, 'A', PERSONAS ; simulate runs all three shoppers at the same time with Promise.all . Each one returns its JSON report. js const plan = extractJson await runAgent base.sandbox, analystPrompt GOAL, baseSessions, N VARIANTS , ; js const settled = await Promise.allSettled variants.map v = runVariant client, secret, v , ; For each fix, runVariant does three things: We use Promise.allSettled instead of Promise.all . The difference: if one fix fails, the others still finish. One bad version does not ruin the whole run. js const ranked = ...arms .sort a, b = b.avgScore - a.avgScore || b.completion - a.completion || a.avgSteps - b.avgSteps ; In plain words: highest average score wins. If two are tied, the one where more shoppers finished wins. If still tied, fewer steps wins. js const body = await winner.sandbox.files.read ${APP DIR}/${f} , { encoding: 'utf8' } ; await writeFile results/winner/${f} , body ; The winning files are copied from the sandbox to results/winner/ on your laptop. The program also writes results/report.md and results/report.json with all the scores and shopper quotes. Set up your key: cp .env.example .env open .env and add: OPENAI API KEY=sk-... set -a; source .env; set +a Then start the run: npm start The first time, you will see a Docker sign-in link. Open it, sign in, and the run continues. A normal run 2 fixes, 3 shoppers uses 3 small sandboxes and about 12 Codex calls. It takes around 5 to 10 minutes. Some handy options: npm start -- --variants 3 --personas 2 npm start -- --goal "Users can't tell which items are on sale" npm start -- --cleanup delete the sandboxes at the end npm start -- --no-dashboard skip the live web page While it runs, open http://localhost:4000 http://localhost:4000 . This is the live dashboard src/dashboard.ts . You will see each step light up, and a box for A, B and C showing scores as they come in. The terminal also prints a public link for each version. Open A, B and C in three tabs. You can watch B and C change while the builder is still working. When it finishes, you get a table like this: | Arm | Variant | Ease | Done | Steps | Lift | |---|---|---|---|---|---| | A | Baseline | 4.0 | 67% | 14.0 | - | | B winner | Search and filters | 8.3 | 100% | 4.0 | +108% | | C | Sort by price | 6.0 | 100% | 8.0 | +50% | Your numbers will be different. Every run is a little different. "Lift" means how much better the score is compared to A. To see exactly what the AI changed, compare the files: diff -u app/app.js results/winner/app.js That diff is basically your pull request. A quick note on ports, since there are a few: | URL | Where | What | |---|---|---| | http://localhost:4000 | your laptop | the live dashboard | | https://3000-....sbx.sandboxes-cloud.docker.com | the cloud | each version of the shop | | http://localhost:3000 | your laptop | only when you run npm run shop | By default, the sandboxes stay running so you can click around each version. When you are done: npm run cleanup This deletes every sandbox whose name starts with ab-. It uses client.allSandboxes to list them and delete { force: true } to remove each one. | What we wanted | SDK call | |---|---| | Sign in | oauth or pat | | Save the OpenAI key | client.secrets.create ... | | Create a sandbox | client.kits.launchAndWait 'codex', ... | | Copy files in and out | sandbox.files.mkdir / files.write / files.read | | Start a server in the background | sandbox.processes.start ... | | Run a quick command | sandbox.run ... | | Run a command and wait for it | sandbox.processes.run ... | | Get a public URL | sandbox.ports.create { number: 3000 } | | Clean up | client.allSandboxes , sandbox.delete ... | That is the whole SDK surface for this project. About ten calls. OPENAI API KEY . Check your .env and run set -a; source Promise.allSettled . npm run cleanup . @docker/sandboxes 0.53.0, so some names may change later. You now know how to: The shop is just an example. The same pattern works for many things: testing prompts, tuning settings, trying different copy. Anywhere you can pretend to be a user and give a score, you can use this loop. The big idea is simple. When a safe, throwaway computer is just one function call away, letting an agent "try it and see" stops being risky. Give it a try, and let me know what you build with it