Building an Autonomous A/B Testing Agent on Docker Cloud Sandboxes

Have you ever wondered if an AI agent could run an A/B test all by itself?

In this post, we will build one together. We will go slowly, one file at a time, so you can follow along even if you have never used Docker Cloud Sandboxes before.

Full code: github.com/ajeetraina/sbx-ab-agent

Think of a small online shop. Some people say it is hard to find things in it. We want to fix that, but we are not sure which fix is best.

So we do this:

Here is the same idea as a picture:

                ┌──────────── Docker Cloud Sandboxes ────────────┐
npm start ──►   │ A  original shop  → shoppers test → analyst    │
                │ B  new sandbox → builder adds fix B → re-test  │
                │ C  new sandbox → builder adds fix C → re-test  │
                └────────────────────────────────────────────────┘
                                   │
                                   ▼
                    winner + report + winning code

"A" is the original shop. "B" and "C" are the new versions. That is all A/B testing really means: try a few versions and see which one works better.

Our builder agent will write and run code that nobody has checked. You do not want that running on your laptop.

A Docker Cloud Sandbox is a small, throwaway computer in the cloud (a microVM). It gives us four things we need:

We control all of this from a small TypeScript program using the @docker/sandboxes SDK.

git clone https://github.com/ajeetraina/sbx-ab-agent
cd sbx-ab-agent
npm install

The project is small. Here is a map:

app/                 the demo shop
  server.mjs         a tiny web server
  index.html         the page
  app.js             shows the products
  styles.css         the look
  products.json      the product list
src/
  sbx.ts             helpers that talk to Docker Cloud Sandboxes
  prompts.ts         the instructions for each AI agent
  ab.ts              the main program that runs every step
  dashboard.ts       a live web page to watch the run
  cleanup.ts         deletes the sandboxes when you are done

A good reading order is: app/ first, then prompts.ts, then sbx.ts, then ab.ts. That is the order we will follow.

app/) The shop shows a long list of products. That is it. There is no search, no filter and no sort. This is on purpose. It is the problem we want our agents to find and fix.

Here is the code that draws the products, from app/app.js:

async function main() {
  const products = await (await fetch('/products.json')).json();
  render(products);
}

It loads every product and puts them all on the page. Nothing more.

Now look at the server in app/server.mjs. The key line is this one:

const body = await readFile(file);

The server reads the file from disk every time someone asks for it. It never keeps an old copy. This matters later: when the builder agent changes app.js, the live site changes right away. No restart needed.

Try the shop on your own machine:

npm run shop

src/prompts.ts) We have three roles. They all use the same AI (Codex). The only difference is the instructions (the prompt) we give each one.

We have three shoppers, each with a simple goal:

Shopper	Who they are	What they want
budget-student	a student with about $20	a gift under $20 for a developer friend
hoodie-hunter	a busy engineer	all hoodies in size L, then the cheapest one
sticker-collector	a sticker fan	see every sticker and compare prices

In code, a shopper is just a small object:

{
  id: 'hoodie-hunter',
  who: 'a busy engineer who already knows they want a hoodie in size L',
  task: 'find all hoodies available in size L and pick the cheapest one',
}

The shopper is told: "You are a real person, not a developer. Do not edit any files." It looks at the shop using curl and imagines the clicks and scrolling a real person would need.

Most important: the shopper must answer in JSON, not in a paragraph. That way our program can read the answer easily.

{
  "persona": "hoodie-hunter",
  "completed": true,
  "steps": 14,
  "score": 4,
  "friction": ["had to scroll the whole list; no size filter"],
  "quote": "I found it, but I had to look at everything."
}

score goes from 1 to 10. A 10 means the task was very easy.

The analyst reads all the shopper reports and suggests a few fixes. The rules we give it:

index.html, app.js and styles.css. It also answers in JSON:

{
  "summary": "Shoppers scroll the whole list. There is no way to narrow it down.",
  "variants": [
    { "id": "B", "title": "Search box and category filters", "hypothesis": "...", "spec": "..." },
    { "id": "C", "title": "Sort by price", "hypothesis": "...", "spec": "..." }
  ]
}

The builder gets one fix (for example "Sort by price") and writes the code. It has strict rules too:

curl. These rules keep every version fair. Each version is the same shop plus one small change.

src/sbx.ts) Now we start talking to Docker Cloud Sandboxes. Everything in this file is a small helper we will call later.

const auth = oauth({
  store: fileOAuthCredentialStore({ path: '.docker-sandboxes/oauth.json' }),
  onVerification: ({ verificationUriComplete, verificationUri, userCode }) => {
    console.log(`Sign in to Docker: ${verificationUriComplete ?? verificationUri}`);
    console.log(`Code: ${userCode}`);
  },
});
await auth.getAccessToken();
return new Sandboxes({ auth });

In plain words:

new Sandboxes({ auth }) gives us a client. We use this client for everything else. If you prefer no browser step, set DOCKER_USERNAME and DOCKER_PAT (a personal access token). The code then uses pat() instead.

const secret = await client.secrets.create({
  displayName: 'sbx-ab-agent-openai',
  serviceType: 'openai',
  token: { value: key },
});

This saves your OpenAI key in Docker, not inside any sandbox. Later we attach the secret to each sandbox. When Codex calls OpenAI, the sandbox adds the key to the request on the way out. Codex never sees the key itself.

One small thing to remember: the SDK does not attach your OpenAI secret for you. You must pass it in when you create a sandbox. We will do that in the next step.

launchShop) This one function does five small jobs. Let us take them one by one.

const sandbox = await client.kits.launchAndWait('codex', {
  displayName,
  resources: 'small',
  storage: { secrets: [secret] },
});

'codex' is a kit: a ready-made sandbox that already has Codex installed. launchAndWait creates it and waits until it is ready. Notice we pass our secret here.

await sandbox.files.mkdir(APP_DIR, { mode: 0o755, parents: true });
for (const f of await readdir(localAppDir)) {
  await sandbox.files.write(`${APP_DIR}/${f}`, await readFile(join(localAppDir, f), 'utf8'));
}

Make a folder in the sandbox, then copy each file from your app/ folder into it.

await sandbox.processes.start({
  args: ['node', 'server.mjs'],
  workingDir: APP_DIR,
  env: { PORT: '3000' },
  label: 'shop-server',
});

processes.start runs a command in the background. The server keeps running while we do other things.

The waitForHttp helper uses sandbox.run(...) to try the server once a second, up to 30 times. When it gets a good answer, we move on.

const port = await sandbox.ports.create({ number: 3000, protocol: 'tcp' });
return { sandbox, url: port.url };

This opens port 3000 to the internet and gives back a real HTTPS URL. You can open it in your browser.

That is it. One function, and we have a live shop running in the cloud.

runAgent) Every AI helper (shopper, analyst, builder) runs through this one function. It runs Codex inside the sandbox with no chat window, just a prompt in and an answer out.

const script =
  `printf '%s' "$1" | codex exec -C ${APP_DIR} --skip-git-repo-check --ephemeral ` +
  `--dangerously-bypass-approvals-and-sandbox -o ${outFile} -`;

const r = await sandbox.processes.run(
  { args: ['bash', '-lc', script, 'bash', prompt], workingDir: APP_DIR },
  { timeoutMs },
);

This looks scary, so let us break it down:

codex exec runs Codex once, without asking us questions.printf ... | ... - sends the prompt into Codex. The - at the end means "read the prompt from input". If you pass the prompt another way, Codex waits and then fails.--dangerously-bypass-approvals-and-sandbox turns off Codex's own safety box. That is fine here, because we are already inside a throwaway sandbox.-o ${outFile} saves only Codex's final answer to a file. After it finishes, we read the answer back:

return await sandbox.files.read(outFile, { encoding: 'utf8' });

AI answers are not always clean. Sometimes the JSON comes wrapped in a code block, or with a sentence before it. The extractJson helper finds the first { and the last } and reads what is between them. Simple, and it works well.

src/ab.ts) Now the main program. It just calls the helpers above in order. Read main() from top to bottom and you will see these steps:

const base = await launchShop(client, 'ab-A-baseline', secret, 'app');
js
const baseSessions = await simulate(base.sandbox, 'A', PERSONAS);

simulate runs all three shoppers at the same time with Promise.all. Each one returns its JSON report.

const plan = extractJson(
  await runAgent(base.sandbox, analystPrompt(GOAL, baseSessions, N_VARIANTS)),
);
js
const settled = await Promise.allSettled(
  variants.map((v) => runVariant(client, secret, v)),
);

For each fix, runVariant does three things:

We use Promise.allSettled instead of Promise.all. The difference: if one fix fails, the others still finish. One bad version does not ruin the whole run.

const ranked = [...arms].sort((a, b) =>
  b.avgScore - a.avgScore || b.completion - a.completion || a.avgSteps - b.avgSteps);

In plain words: highest average score wins. If two are tied, the one where more shoppers finished wins. If still tied, fewer steps wins.

const body = await winner.sandbox.files.read(`${APP_DIR}/${f}`, { encoding: 'utf8' });
await writeFile(`results/winner/${f}`, body);

The winning files are copied from the sandbox to results/winner/ on your laptop. The program also writes results/report.md and results/report.json with all the scores and shopper quotes.

Set up your key:

cp .env.example .env
set -a; source .env; set +a

Then start the run:

npm start

The first time, you will see a Docker sign-in link. Open it, sign in, and the run continues.

A normal run (2 fixes, 3 shoppers) uses 3 small sandboxes and about 12 Codex calls. It takes around 5 to 10 minutes.

Some handy options:

npm start -- --variants 3 --personas 2
npm start -- --goal "Users can't tell which items are on sale"
npm start -- --cleanup          # delete the sandboxes at the end
npm start -- --no-dashboard     # skip the live web page

While it runs, open http://localhost:4000. This is the live dashboard (src/dashboard.ts). You will see each step light up, and a box for A, B and C showing scores as they come in.

The terminal also prints a public link for each version. Open A, B and C in three tabs. You can watch B and C change while the builder is still working.

When it finishes, you get a table like this:

Arm	Variant	Ease	Done	Steps	Lift
A	Baseline	4.0	67%	14.0	-
B (winner)	Search and filters	8.3	100%	4.0	+108%
C	Sort by price	6.0	100%	8.0	+50%

(Your numbers will be different. Every run is a little different.)

"Lift" means how much better the score is compared to A.

To see exactly what the AI changed, compare the files:

diff -u app/app.js results/winner/app.js

That diff is basically your pull request.

A quick note on ports, since there are a few:

URL	Where	What
http://localhost:4000	your laptop	the live dashboard
https://3000-....sbx.sandboxes-cloud.docker.com	the cloud	each version of the shop
http://localhost:3000	your laptop	only when you run npm run shop

By default, the sandboxes stay running so you can click around each version. When you are done:

npm run cleanup

This deletes every sandbox whose name starts with ab-. It uses client.allSandboxes() to list them and delete({ force: true }) to remove each one.

What we wanted	SDK call
Sign in	oauth() orpat()
Save the OpenAI key	client.secrets.create(...)
Create a sandbox	client.kits.launchAndWait('codex', ...)
Copy files in and out	sandbox.files.mkdir /files.write /files.read
Start a server in the background	sandbox.processes.start(...)
Run a quick command	sandbox.run([...])
Run a command and wait for it	sandbox.processes.run(...)
Get a public URL	sandbox.ports.create({ number: 3000 })
Clean up	client.allSandboxes() ,sandbox.delete(...)

That is the whole SDK surface for this project. About ten calls.

OPENAI_API_KEY. Check your .env and run set -a; source Promise.allSettled. npm run cleanup.@docker/sandboxes 0.53.0, so some names may change later. You now know how to:

The shop is just an example. The same pattern works for many things: testing prompts, tuning settings, trying different copy. Anywhere you can pretend to be a user and give a score, you can use this loop.

The big idea is simple. When a safe, throwaway computer is just one function call away, letting an agent "try it and see" stops being risky.

Give it a try, and let me know what you build with it!

source & further reading

dev.to — original article Every "touch grass" app assumes you can walk. Mine asks how far you can go today. TerraGemma: An Offline Edge AI Trail Companion That Tells You to Put Your Phone Away TypeScript `TemplateStringsArray` and Tagged Templates: Building Type-Safe DSLs in 2026