{"slug": "building-reproducible-ai-evaluation-workflows-with-docker-sandboxes", "title": "Building Reproducible AI Evaluation Workflows with Docker Sandboxes", "summary": "The SBX AI Evaluation Kit, an open-source Docker Sandboxes Mixin Kit, enables reproducible AI evaluation workflows by executing configured commands in isolated sandboxes and recording runtime evidence such as stdout, stderr, exit code, and execution time. The kit separates evaluation definitions from execution environments, allowing the same workflow to run locally or via Docker Sandboxes without modification.", "body_md": "AI evaluation has never been easier to start. Reproducing it reliably is another story. Developers now have access to more benchmarks, evaluation libraries, model APIs, and agent frameworks than ever before. But keeping the prompt, model, and scoring method fixed doesn’t necessarily make a run reproducible. The execution environment matters too.\n\nPython dependencies change. Local tools drift. Setup steps go undocumented. A workflow that succeeds on one machine may behave differently on another. Most discussions about evaluation focus on *what* should be measured: benchmarks, scoring methods, or judge models. Much less attention is given to *how* those evaluations are executed. Yet that execution layer often determines whether someone else can reproduce the same workflow weeks or months later.\n\nWhen I started exploring Docker Sandboxes, I wasn’t trying to build another evaluation framework. I had a much smaller question.\n\nThat question eventually became the **SBX AI Evaluation Kit****,** an open-source Docker Sandboxes Mixin Kit focused on repeatable execution, structured evaluation records, and runtime evidence. The current *implementation *does not execute AI models or automatically derive evaluation judgments. Instead, it executes configured commands consistently and preserves evidence of what actually ran.\n\n## In Practice\n\nIn practice, the workflow starts by choosing where the evaluation command should run through the `execution`\n\nblock:\n\n```\nexecution:\n  executor: sbx\n  command:\n    - python3\n    - -c\n    - print(\"hello from sbx\")\n```\n\nWith `executor: sbx`\n\n, the runner delegates command execution to Docker Sandboxes and writes the runtime evidence into the resulting artifact.\n\nThe repository is also packaged as an SBX Mixin Kit, so it can be applied when starting a Claude sandbox:\n\n```\nsbx run claude --kit .\n```\n\nThe runner reads the configured executor and delegates the command to SBX, which executes it inside the sandbox.\n\n```\npython run_evaluation.py\n```\n\n## From Documentation to an Executable Workflow\n\nEach evaluation is defined in a YAML file that describes the evaluation and the command to run. The repository validates that definition, executes it, and produces a structured JSON record of the result. The difference is in what gets recorded. A written evaluation captures what someone intended to do. An execution-backed evaluation captures what actually happened.\n\n## Separating Evaluation from Execution\n\nI wanted the evaluation definition to stay independent of where it ran. A workflow written during local development shouldn’t need to change simply because it later executes inside Docker Sandboxes.\n\nTo keep those concerns separate, I introduced an executor abstraction. **The evaluation describes what should run; the executor determines where it runs.**\n\nWith the local executor, the configured command runs on the host. With the SBX executor, command execution is delegated to Docker Sandboxes. Switching between the two only requires changing the executor configuration, not rewriting the surrounding evaluation workflow.\n\n*Figure 1. Evaluation definitions remain independent of the execution environment. The same workflow can use either the local or SBX executor while producing runtime evidence in the same structure.*\n\n## Capturing Evidence Instead of Assumptions\n\nFor each execution, the runner records enough information to inspect what actually happened:\n\n- the selected executor,\n- the command that was executed,\n- standard output (\n`stdout`\n\n) and standard error (`stderr`\n\n), - the exit code,\n- and the execution time.\n\nThese details are stored in the evaluation artifact. The repository also generates a digest of the evaluation configuration. This creates a deterministic link between the evaluation configuration and the artifact it produced, without trying to replace full experiment-tracking systems.\n\n```\n{\n  \"executor\": \"sbx\",\n  \"command\": [\"python3\", \"-c\", \"print(\\\"hello from sbx\\\")\"],\n  \"stdout\": \"hello from sbx\\n\",\n  \"stderr\": \"\",\n  \"exit_code\": 0,\n  \"duration_ms\": 120.0\n}\n```\n\n## Scaling from One Evaluation to Many\n\nReal-world evaluation rarely consists of one isolated run. Teams compare prompts, validate behavior, measure regressions between releases, and test multiple scenarios. That led to evaluation suites.\n\nRather than changing how an individual evaluation works, a suite groups multiple evaluation definitions into a single repeatable workflow. Each evaluation still produces its own structured artifact, while the suite also generates an aggregated summary of the overall run.\n\n## Reusable SBX Kits Beyond Evaluation\n\nThe same pattern isn’t limited to evaluation. An SBX Kit can package more than a development environment; it can also package the setup an engineering workflow depends on. The same model could support regression testing, policy checks, security analysis, code-generation experiments, and other workflows that depend on consistent execution and inspectable results.\n\n## Conclusion\n\nThe SBX AI Evaluation Kit doesn’t replace evaluation frameworks, benchmarks, or scoring systems. Its job is narrower: execute configured evaluation workflows in a way that is easier to rerun and inspect.\n\nThe question I came away with is simple: before comparing benchmark scores or choosing a judge model, can someone else reliably run the same workflow under comparable conditions?\n\nYou can explore the code, experiment with custom evaluation YAMLs, and run the workflow yourself in the[ sbx-ai-eval-kit repository on GitHub](https://github.com/karanverma/sbx-ai-eval-kit).\n\n## Resources\n\n– Source code, example evaluation definitions, and the implementation described in this article.**SBX AI Evaluation Kit**– Official documentation for setting up and running Docker Sandboxes.** Docker Sandboxes documentation**– Official documentation for extending Docker Sandboxes with reusable Kits.** Customizing Docker Sandboxes with Kits**", "url": "https://wpnews.pro/news/building-reproducible-ai-evaluation-workflows-with-docker-sandboxes", "canonical_source": "https://www.docker.com/blog/building-reproducible-ai-evaluation-workflows-with-docker-sandboxes/", "published_at": "2026-09-02 13:00:00+00:00", "updated_at": "2026-09-02 13:53:39.098380+00:00", "lang": "en", "topics": ["ai-tools", "developer-tools", "mlops"], "entities": ["SBX AI Evaluation Kit", "Docker Sandboxes", "Claude"], "alternates": {"html": "https://wpnews.pro/news/building-reproducible-ai-evaluation-workflows-with-docker-sandboxes", "markdown": "https://wpnews.pro/news/building-reproducible-ai-evaluation-workflows-with-docker-sandboxes.md", "text": "https://wpnews.pro/news/building-reproducible-ai-evaluation-workflows-with-docker-sandboxes.txt", "jsonld": "https://wpnews.pro/news/building-reproducible-ai-evaluation-workflows-with-docker-sandboxes.jsonld"}}