The Agentic Scientist – SakanaAI

SakanaAI released Agentic Scientist, a pipeline that turns a task.md description into experiments, ablations, a sealed final evaluation, post-hoc analyses, and a written paper using coding agents. The tool drives claude, codex, cursor, opencode, gemini, and pi through Headless CLI, runs each agent phase inside Docker by default with full tool permissions (Headless --allow yolo), and requires Python 3.11 or higher with uv; GPU task presets are discovered via nvidia-smi. SakanaAI warns that the agents can run arbitrary commands and inherit supplied API keys, so runs should stay inside the Docker image and only trusted tasks and sources should be used.

Turn a task description into experiments, ablations, and a paper, driven by coding agents. Agentic Scientist treats coding agents as AI scientists. Starting from a task.md , the agents propose ideas, run experiments, evaluate outcomes against the task rubric, and iterate toward stronger programs. The pipeline then runs baselines and ablations, a sealed final evaluation, post-hoc analyses, and writes the paper, with every stage kept under one run directory. The recommended setup runs every agent phase inside Docker, which is the default for agsci run . Install Docker https://docs.docker.com/get-docker/ , then build the agent image: docker build -t agentic-scientist/agent-cli:latest -f src/docker/agent-cli/Dockerfile . If the build cannot reach package mirrors, add --network=host to the docker build command. Security note — Agentic Scientist launches coding agents with full tool permissions Headless --allow yolo . They can run arbitrary commands and inherit the API keys you give them, and the task files and papers they read steer what they do. Keep runs inside the Docker image rather than on your workstation, only run tasks and sources you trust, and review run directories before sharing them: agent logs can echo document contents and paths. - Python ≥ 3.11 with uv https://docs.astral.sh/uv/ . uv sync creates .venv and installs the agsci command: curl -LsSf https://astral.sh/uv/install.sh | sh uv sync source .venv/bin/activate - API keys in a .env file at the repo root ANTHROPIC API KEY , OPENAI API KEY , GEMINI API KEY , ... whichever your agent needs . The pipeline mounts it into the agent containers. Never commit it. - Agent CLIs are already inside the image. For native non-containerized runs, install the CLI of the agent you use plus Git and Node.js; versions and commands are in docs/install.md https://github.com/SakanaAI/AI-Scientist-v3/blob/main/docs/install.md . - GPUs are required only by the GPU task presets. The pipeline discovers them through nvidia-smi and refuses to start compute phases unless the visible GPUs match the preset. - Autoillustrate optional is built into the image for concept figures. To use it directly on the host, follow the single-command install in the Autoillustrate README https://github.com/SakanaAI/Autoillustrate readme . Run agsci doctor to check the host, or agsci doctor examples/<task preset=<task to check everything one specific run needs CLI, credentials, image, GPUs, task scaffold . Point agsci run at a task directory. This runs the default six phases task drafting → investigation → ablations → final evaluation → post-hoc analysis → paper on the CPU-only circle-packing example: agsci run examples/circpack Add --dry-run to validate the configuration and preview the exact commands without running anything. Every option is a Hydra key=value override; agsci run --help lists them all. Note — Agentic Scientist drives claude , codex , cursor , opencode , gemini , and pi through Headless CLI https://github.com/RobertTLange/headless-cli , so the same experiment can be repeated across agents and models: pick one with agent.name=<agent agent.model=<model . Task drafting only writes a scaffold when the task directory has none; every shipped example already includes its evaluator, so the drafting phase only snapshots it and the investigation phase builds and validates the task runtime. Drafting a task from a bare research brief is experimental see example 8 8-draft-a-task-from-a-research-brief-experimental . Each run writes one predictable envelope under results/ : results/<task /<agent /<run id / ├── task drafting/ drafting logs and validation audit ├── task/ immutable run-local task snapshot ├── investigation/ ideas, attempts, winner, and optimizer memory ├── ablations/ baselines and ablations ├── final evaluation/ sealed final evaluation ├── posthoc/ post-hoc analyses └── paper/ manuscript, PDF, manifest, and optional AI reviews Investigation plus the task snapshot is enough to write a paper; the other phases add evidence when available and are reported as degraded when they fail. Browse any run in the browser with agsci inspect start http://127.0.0.1:8765 . The default configuration is the one to start with: the claude agent, every phase in Docker, and the CPU-only examples/circpack task. It needs a Claude login, no GPUs, and exercises every phase through to a compiled paper. 1. Build the image and the Python environment as described in Install install . 2. Create .env at the repo root. Only the selected agent's keys are forwarded into its containers: CLAUDE CODE OAUTH TOKEN=... claude, billed to your Claude subscription ANTHROPIC API KEY for pay-per-token OPENAI API KEY=... recommended: concept figures in the paper and the review phase Other agents: codex reads OPENAI API KEY or CODEX API KEY , gemini reads GEMINI API KEY , cursor reads CURSOR API KEY , and Bedrock-backed Claude reads the AWS variables. Concept figures use OPENAI API KEY , fall back to GEMINI API KEY , and otherwise ship a plain export. 3. Export host variables only if you need them. Everything else host networking, the image, the .env mount is already the agsci run default: export AGENTIC TEMP ROOT=/path/to/scratch existing writable dir on a large disk; default: $TMPDIR, /mnt/localssd, or /tmp export AGENT CONTAINER GPUS=device=<UUID,... GPU presets only: which GPUs to use; default: the first container.gpu count x investigation workers from nvidia-smi 4. Check, preview, then run. Shrinking the ablation and post-hoc budgets makes the first end-to-end run finish faster without skipping a phase: agsci doctor examples/circpack host, image, credentials, task scaffold agsci run examples/circpack --dry-run print every phase command, run nothing agsci run examples/circpack \ investigation.execution.max idea cycles=2 \ ablations.execution.num replications=1 \ ablations.execution.max evals per variant=2 \ posthoc.execution.max selected analyses=1 The paper lands in results/circpack/claude/<run id /paper/paper.pdf ; agsci inspect start browses the run. Drop the overrides for the default budgets 3 idea cycles, 5 ablation replications, 3 post-hoc analyses , then move on to the GPU task preset and a pinned model, for example agsci run examples/grpo preset=grpo agent.model=claude-sonnet-5 . Task presets under configs/preset/ hold the campaign budgets, timeouts, paper format, and required GPUs. configs/preset/paper.yaml is the shared campaign preset and configs/preset/grpo.yaml composes it for the GRPO task two GPUs, harness-side GPU evaluation : agsci run examples/grpo preset=grpo Explicit overrides win over the preset: agsci run examples/grpo preset=grpo \ agent.name=opencode \ agent.model=amazon-bedrock/global.anthropic.claude-opus-5-5 \ investigation.execution.max idea cycles=20 Every phase resumes against the run directory. The pipeline recovers the original idea-cycle budget; pass the original agent and model again: Continue after investigation with ablations, post-hoc analysis, and the paper: agsci run examples/circpack execution.phases=ablations,posthoc,paper \ paths.run dir=results/circpack/claude/20260331 094917 Rewrite only the paper: agsci run examples/circpack execution.phases=paper \ paths.run dir=results/circpack/claude/20260331 094917 Adding the review phase sends the complete manuscript to the OpenAI-backed reviewer. Review failures are recorded but never discard the paper: agsci run examples/circpack paper.paper format=iclr \ execution.phases=task drafting,investigation,ablations,final evaluation,posthoc,paper,review The same reviewer runs standalone on any PDF with agsci review . It reads OPENAI API KEY from the environment or the repo .env and prints the review JSON: agsci review results/circpack/claude/<run id /paper/paper.pdf --output review.json Each phase composes its stage's own config, so any stage key is reachable as <phase .<key : agsci run examples/circpack \ investigation.execution.max idea cycles=5 \ ablations.execution.num replications=1 \ posthoc.execution.max selected analyses=2 \ paper.paper format=iclr paper.paper review rounds=1 agsci investigate src/investigation/cli.py runs the idea/experiment loop on its own, natively by default, optionally with parallel workers: agsci investigate agent.name=claude paths.task dir=examples/circpack \ execution.workers=2 execution.max idea cycles=4 See docs/usage.md https://github.com/SakanaAI/AI-Scientist-v3/blob/main/docs/usage.md for the standalone investigation loop, its container mode, and every argument. examples/grpo ships its own evaluator and needs two GPUs: vLLM sampling on one, training on the other. Fetch the model weights once about 3 GB, into examples/grpo/resources/ , then declare the GPUs the task needs: uv run examples/grpo/download model.py agsci run examples/grpo container.gpu count=2 \ investigation.execution.harness evaluated attempts=true \ investigation.execution.gpu evaluation=true To explore ideas in parallel, add investigation workers. Each worker gets its own container.gpu count slice, so two workers need four visible GPUs: agsci run examples/grpo container.gpu count=2 investigation.execution.workers=2 \ investigation.execution.harness evaluated attempts=true \ investigation.execution.gpu evaluation=true Each evaluation trains for 1-2 hours; raise investigation.execution.harness evaluation timeout seconds above its 7200 s default if the harness cuts runs short. A task directory holding only a task.md brief makes the drafting phase write the initial program, evaluators, and task contract itself: agsci run examples/circpack task drafting execution.phases=task drafting This works for simple tasks such as circle packing but is not yet robust: for larger tasks like the world-model or tabular distribution-shift examples it may fail to produce a scaffold that matches the full pipeline contract. Start from a shipped scaffold for anything beyond a small task, and check the drafting audit under task drafting/ when a draft is rejected. See docs/task drafting.md https://github.com/SakanaAI/AI-Scientist-v3/blob/main/docs/task drafting.md . Six phases run in sequence: Task drafting → Investigation → Ablations → Final evaluation → Post-hoc analysis → Paper , with optional Review and Revision phases afterwards. Task drafting experimental for tasks without a shipped scaffold turns task.md into a sealed evaluator and rubric; investigation is the agentic idea → experiment → evaluate loop; ablations compare the winning program against baselines and ablated variants; the final evaluation re-scores it in a sealed environment; post-hoc analysis mines the artifacts for extra evidence; and the paper phase writes, compiles, and audits the manuscript against that evidence. Each phase runs in its own container and only ever reads the run-local snapshot, so results stay reproducible and examples/ is never modified. See docs/pipeline.md https://github.com/SakanaAI/AI-Scientist-v3/blob/main/docs/pipeline.md for the details. - Installation https://github.com/SakanaAI/AI-Scientist-v3/blob/main/docs/install.md — agent CLI, Headless runtime, Docker build options, credentials, container options, Autoillustrate. - End-to-end pipeline https://github.com/SakanaAI/AI-Scientist-v3/blob/main/docs/pipeline.md — agsci run recipes, presets, GPUs, resuming, ablations, and cost output. - Investigation loop usage https://github.com/SakanaAI/AI-Scientist-v3/blob/main/docs/usage.md — standalone loop, parallel workers, containers, arguments, and checkpoints. - Task drafting https://github.com/SakanaAI/AI-Scientist-v3/blob/main/docs/task drafting.md — how tasks become sealed evaluators, qualification, and repair. - Variants pipeline https://github.com/SakanaAI/AI-Scientist-v3/blob/main/src/variants pipeline/README.md — stages, custom baselines, cross-task generalization, output layout. - Post-hoc pipeline https://github.com/SakanaAI/AI-Scientist-v3/blob/main/docs/posthoc pipeline.md — standalone architecture and resume flow. - Run Inspector https://github.com/SakanaAI/AI-Scientist-v3/blob/main/docs/run inspector.md — launch modes and overrides for the web UI. - Script reference https://github.com/SakanaAI/AI-Scientist-v3/blob/main/docs/scripts.md — the individual agent scripts and their outputs. - Agent billing https://github.com/SakanaAI/AI-Scientist-v3/blob/main/docs/agent billing.md , temp storage https://github.com/SakanaAI/AI-Scientist-v3/blob/main/docs/temporary storage.md , GPU quarantine https://github.com/SakanaAI/AI-Scientist-v3/blob/main/docs/gpu quarantine.md , evaluation evidence https://github.com/SakanaAI/AI-Scientist-v3/blob/main/docs/evaluation evidence.md , paper final outcomes https://github.com/SakanaAI/AI-Scientist-v3/blob/main/docs/paper final outcomes.md , no-winner studies https://github.com/SakanaAI/AI-Scientist-v3/blob/main/docs/no winner studies.md . If you use Agentic Scientist in your research, please cite our paper https://arxiv.org/abs/2610.18933 : @article{agentic scientist, title = {The Agentic Scientist: Orchestrating Coding Agents for Automated Scientific Co- Discovery}, author = {Lange, Robert Tjarko and Grillotti, Luca and Yamada, Yutaro and Charakorn, Rujikorn and Ha, David}, journal = {arXiv preprint arXiv:2610.18933}, year = {2026} }