An End-to-End Autoresearch Agent Framework
Paper (arXiv)
Chinese ScienceFlow is an end-to-end autoresearch agent framework for productive, stable, and goal-aligned research over hours or days. It organizes research around recoverable executable workspaces, coupling persistent state, adaptive exploration, and evidence-aware execution control so agents can continue, redirect, or recover without losing validated progress.
Across machine learning, scientific modeling, and mathematical optimization, ScienceFlow sustains effective long-horizon research and reaches 70.22 ± 1.18% Any-Medal on the full 75-task MLE-bench within a 24-hour budget, exceeding the strongest reported baseline by 4.92 percentage points.
Figure 1a. Full MLE-bench Any-Medal leaderboard. Mean ± SEM over three independent runs for ScienceFlow.
Recoverable executable state. Each persistent LNR worker advances research in an isolated executable workspace. An archived state binds that workspace to compact memory, validation evidence, and resource records.Stage Gate. A task-specific result signal invokesGateService
: the configured Evaluator produces normalized evidence, and the Gate policy decides admission. An accepted result materializes an immutable Stage with ledger facts and a recoverable workspace snapshot.ESTRA. At a research boundary, Executable-State Transition through Re-Anchoring makes a two-axis decision: a start point (the current workspace or an archived Stage) and an intent (continue
orredirect
). Selecting an archived start point restores its executable state before the next research segment.Persistent memory. Add records accepted Stage progress. Fold keeps recent, best-validated, and anchor-relevant evidence explicit while summarizing older records; Unfold/restore retrieves indexed evidence and state, and Assemble constructs the anchor-specific context for the next segment.Evidence-aware execution control. Research workers choose scientific routes, while the controller admits, leases, monitors, timeboxes, and stops physical jobs using resource availability, remaining budget, validated progress, and recoverability. Valid worker states are finalized undermerge/finals/final_*
.
Figure 2. ScienceFlow system architecture. Research workers operate over recoverable executable states and adapt long-horizon trajectories through boundary-triggered ESTRA transitions, while evidence-aware execution control coordinates physical resource allocation and runtime execution.
- LNR is
no-skill by default:
lnr_skill_tool_enabled: false
andlnr_skill_auto_read: false
. .scienceflow/skills/data_processing/
is retained only for the dedicated data-prep agent and validation-split workflow.auto
is the default evaluator backend and resolves registered tasks totask_package
;artifact_command
remains available for generic command-based evaluation.- Parallel runs bind CPU/GPU resources at task level, then split CPU capacity across workers. GPU leases support controlled sharing by multiple workers.
- Result signals may create Stages without a submission when the task contract permits it. Merge can only emit finals from candidates that carry the required artifact.
ScienceFlow/
├── scienceflow/ # Framework runtime
│ ├── core/ # Agent runtime, tools, memory, and execution
│ ├── solver/ # LNR, Stage lifecycle, ESTRA, resume, and merge
│ ├── gates/ # Stage Gate and Evaluator plugins
│ ├── safety/ # Evidence-aware resource and execution control
│ ├── ui/ # Monitor and trace interfaces
│ ├── config/ # Defaults and example manifests
│ ├── utils/ # Shared runtime utilities
│ └── cli.py # Command-line entry point
├── tasks/ # Task packages and evaluators
├── scripts/ # Maintained run and monitor manifests
├── .scienceflow/skills/data_processing/ # Data-preparation skills
└── doc/scienceflow/ # Detailed architecture documentation
Requirements: Python 3.11+ and uv.
curl -LsSf https://astral.sh/uv/install.sh | sh # or: pip install uv
git clone https://github.com/huawei-noah/noah-research.git
cd noah-research/ScienceFlow
cp env.example .env # then edit .env: set API_KEY and BASE_URL for your provider
uv sync
Notes:
uv sync
installs the lockeduv.lock
environment, including the in-repodeepcraft
subpackages, the officialmlebench
Git revision, and the full test/ML stack.- PyTorch wheels default to the
cu128 index (for CUDA 12.8-era drivers). Adjust[[tool.uv.index]]
inpyproject.toml
if you need a different CUDA build. - SciModelingBench support is an optional extra:
uv sync --extra scientific-design
. - Run commands either via
uv run ...
or by using.venv/bin/python
directly.
Start an interactive research REPL:
uv run python -m scienceflow.cli repl
Run the maintained two-worker Nomad2018 example:
uv run python -m scienceflow.cli parallel -m scripts/lnr.yaml -j 1
Monitor an existing run:
uv run python -m scienceflow.cli monitor --manifest scripts/lnr.yaml --refresh 5
Prepare a dataset with the dedicated data-prep agent:
uv run python -m scienceflow.cli parallel -m scripts/prep.yaml -j 1
Run the self-contained Circle Packing math-optimization example (no dataset or optional extra required):
uv run python tasks/opt_solver/_tools/prepare_math_opt_solver_tasks.py
uv run python -m scienceflow.cli parallel \
-m scienceflow/config/examples/tasks_circle_packing_example.yaml -j 1
The prepare step writes a tiny task package (problem.json
plus a valid baseline) under ./data/opt_solver/
. The agent then iteratively improves artifacts/best_solution.json
, and the system-side evaluator authoritatively validates each candidate and scores it by the sum of radii.
| Setting | Purpose |
|---|---|
lnr.num_workers |
|
| Number of persistent research workers inside one task. | |
task.cpu_list / task.gpu_list |
|
| Task-level CPU and GPU resource boundaries; LNR further splits CPU across workers. | |
lnr.omp_threads_cap |
|
| CPU thread cap for each worker slice. | |
lnr.wall_clock_budget_sec |
|
| Total wall-clock budget for the LNR process. | |
lnr.estra_enabled / estra_trigger_stage_count |
|
| Enables ESTRA and sets the trigger for boundary review and context folding. | |
lnr.resource_runtime_enabled |
|
| Enables evidence-aware resource and execution control. | |
resume_budget_policy |
|
Budget accounting for resumed runs; fresh adds this round's time_limit on top of accumulated time. |
|
evaluator.backend |
|
Selects auto (default), task_package , or artifact_command . |
|
evaluator.stage_source_mode |
|
Selects the shadow , adjudicate , or primary Stage source mode. |
|
evaluator.command.python_executable |
|
| Points a task at an isolated Python environment. | |
profile_overrides.<profile> |
|
| Overrides prompts, Evaluator, and resource behavior per task type. | |
tasks/**/task.yaml |
|
| Declares the task-level artifact, metric, provider/profile, Evaluator, and Gate policy. | |
metric.authoritative: true |
|
| Marks a metric as authoritative evidence eligible for high-trust selection. |
Architecture overview · Recoverable states and LNR · Evidence-aware execution control · Adding opt-solver tasks · Scientific modeling example
The paper evaluates the same ScienceFlow workflow across three classes of executable research tasks:
Machine learning engineering: all 75MLE-benchtasks through the pipeline-construction interface.Scientific modeling and design: 12SciModelingBench tasks on Hugging Facethrough the candidate-optimization interface.Mathematical and engineering optimization:Circle Packing,Ratio Minimization,Uncertainty Inequality, and the easy, medium, and hardSpOC4 KTTSPtracks through the candidate-optimization interface.
All task families share the Stage Gate and Evaluator contract. Each task.yaml
keeps provider/profile, artifact schema, metric direction, evaluator backend, authoritative status, and Gate policy outside the generic solver.
- MLE-bench tasks require the data root, task
exp_id
, andsubmission.csv
contract to be aligned. - Do not resume old workspaces across different task profiles, or prompts, datasets, or artifact dimensions may be inherited incorrectly.
stopped_by_user
marks a resumable terminal state from a manual stop, not a failure; a later resume should continue from the accumulated budget instate.json
and the workspace stages.
uv run pytest -q
The project uses Python 3.11+, Pydantic, OmegaConf, Click, and Rich. The official mlebench
dependency is pinned to a Git revision in uv.lock
; the optional scientific-design
extra provides SciModelingBench, Datasets, and PyArrow support. Command-based optimization tasks may use a separate Python environment injected through the evaluator configuration, with task-specific schema/scoring logic kept inside the tasks/<category>/...
task package.