{"slug": "how-acto-uses-marimo-to-bring-human-judgment-into-ai-evaluation", "title": "How Acto uses marimo to bring human judgment into AI evaluation", "summary": "Acto, a reinforcement-learning research lab founded by Viraj Tipnis, built an internal review application called Result Review in marimo to let domain experts inspect AI agent outputs alongside the evidence behind them, after an agent returned a correct FY27 EBITDA of $47.2 million by hard-coding a value in place of a formula and still passed the automated verifier. The application runs on Harbor, the Terminal-Bench team's open-source evaluation framework, and stores each expert judgment as an auditable review record linked to the specific run, separate from the automated score. Tipnis said marimo gave Acto \"a shared research surface: Python-native enough for researchers, accessible enough for domain review, and flexible enough to work directly with application state and task-specific evaluators.", "body_md": "*Viraj Tipnis**- Founder, Acto*\n\nIn a demonstration task from [Acto](https://www.actorl.com/), an AI agent receives a straightforward instruction: update a financial forecast using supplied assumptions while preserving the workbook’s structure and logic.\n\nThe agent returns a workbook with the expected FY27 earnings before interest, taxes, depreciation, and amortization (EBITDA) of $47.2 million. The automated check passes. But an expert inspecting the workbook finds that the agent replaced a formula with a hard-coded value. The answer is correct, but the workbook no longer calculates it as intended.\n\nFor Viraj Tipnis, founder of Acto, this example illustrates a central challenge in evaluating agents that perform real work. Automated checks can test specific requirements, but domain experts help determine whether those requirements capture what makes the work useful and trustworthy.\n\nActo uses marimo to connect those assessments with the evidence behind them. Researchers build review applications in Python, experts inspect the agent’s work, and their findings inform tests of the automated evaluators.\n\n## [#](#researchers-needed-expert-feedback-connected-to-the-evidence)Researchers needed expert feedback connected to the evidence\n\nActo is a reinforcement-learning research lab studying agents performing tasks in workplace software, including spreadsheets, documents, and email. Its [implementation](https://github.com/vrtnis/harbor-marimo) uses Harbor, the Terminal-Bench team’s open-source evaluation framework, to run tasks, collect outputs, and apply automated evaluators called verifiers.\n\nBefore marimo, researchers reconstructed cases from scattered files, screenshots, and logs. Expert feedback often arrived in a separate note or message, detached from the exact run it described. When a verifier changed, researchers struggled to compare its result with an expert’s earlier assessment.\n\nActo’s internal dashboard served research operations rather than focused expert review. Spreadsheets could not accommodate the different outputs and execution evidence. Traditional notebooks could combine code and evidence, but exposed unfamiliar tools to reviewers who did not regularly use Python. A custom application would become another product to maintain.\n\nThe team needed a way for researchers and experts to examine the same work and preserve the reasoning behind each judgment.\n\n## [#](#experts-can-inspect-the-task-output-and-supporting-evidence-together)Experts can inspect the task, output, and supporting evidence together\n\nActo’s research team built Result Review, a marimo application that places the agent’s task beside its output and scoring controls. The verifier result, reference solution, run configuration, and environment are available as supporting evidence.\n\n*Result Review presents the task and acceptance criteria, with controls for selecting the run and output to inspect. This example concerns a processor-performance model.*\n\nFor the forecast workbook, the expert can assess whether the agent followed the instruction to preserve its logic, rather than relying on the passing score. The expert records a judgment and rationale against the specific run.\n\nThat assessment remains separate from the automated score. Acto saves it as an auditable review record linked to the run, leaving the original output and verifier result unchanged. Researchers can then investigate the disagreement without losing either assessment.\n\n“marimo gave us a shared research surface: Python-native enough for researchers, accessible enough for domain review, and flexible enough to work directly with application state and task-specific evaluators,” says Viraj.\n\nWhen the output alone does not explain a failure, reviewers can inspect the recorded sequence of agent actions and observations. Result Review displays supported trajectories alongside the output, with screenshots when they were captured during execution. These are event-level traces rather than video replays. Experts can also compare two runs side by side and record which produced stronger work.\n\nThe same approach extends beyond spreadsheets. In a [processor-modeling task](https://github.com/harbor-framework/terminal-bench-science/tree/3ecc1515de0fa7ffc6b1f235e88dfa5d6b1d1c28/tasks/engineering-sciences/electrical-engineering/microarch-modeling), an expert assesses whether a model ranks promising CPU designs well enough to guide engineering decisions, alongside its numerical accuracy.\n\n## [#](#an-experts-finding-becomes-a-test-of-the-evaluator)An expert’s finding becomes a test of the evaluator\n\nThe workbook example makes the gap in the automated check concrete: checking that a file opens and returns the expected total does not establish that its formulas remain intact.\n\nActo uses another marimo application, Task Studio, to help researchers turn expert-defined acceptance criteria into verifiers and test them against known examples. A revised check can inspect the workbook’s formulas, rather than only its displayed result.\n\nResearchers can apply that check to preserved outputs without asking the agent to repeat its work. They test both known failures and expert-approved examples to determine whether the revision catches the problem without rejecting valid work.\n\nmarimo’s reactive execution updates related views when the author changes the task or acceptance checks. Running the verifier remains deliberate: the author selects “Test these examples” and examines the results. These tests do not overwrite the original run’s score.\n\n*In Task Studio, researchers specify examples that should pass or fail, then select “Test these examples” to check the verifier. This screenshot uses a scientific demonstration task.*\n\n“That lets us ask a better question than whether aggregate reward increased: is the evaluator aligned with the target behavior, and does it retain enough resolution to distinguish meaningful capability gains from noise or reward hacking?” says Viraj.\n\nExpert involvement also begins before execution. In Task Review, experts independently assess the task, its expected outputs, and its acceptance criteria before an agent attempts it. This helps researchers examine both the instructions they give agents and the standards used to judge their work.\n\nThe applications support evaluation and review, not automatic model training. Revised verifiers determine how subsequent evaluation runs are scored. Review records and verifier results can also supply signals to a separate downstream training system that updates model weights.\n\n## [#](#marimo-connects-python-inspection-tools-with-an-interface-experts-can-use)marimo connects Python inspection tools with an interface experts can use\n\nActo needs researchers to write task-specific code and domain experts to assess the evidence that code exposes. marimo supports both within the same application framework.\n\nResearchers can use Python to read a workbook, inspect a model, or load execution records directly. App mode presents the relevant evidence and controls to experts without requiring them to navigate notebook code. Reactive execution keeps related views consistent as inputs change.\n\n*Experts assess individual criteria and record their verdict, supporting evidence, rationale, and confidence in the review interface.*\n\nThe interface can also change with the task. Reviewing a workbook requires different evidence from reviewing a model, and researchers need to adjust how much they show.\n\n“The interface has to surface the minimum sufficient evidence for an expert to make a calibrated judgment,” says Viraj.\n\nWith marimo, the research team can adapt review interfaces to these kinds of tasks without maintaining a separate custom review product. Acto’s public [harbor-marimo repository](https://github.com/vrtnis/harbor-marimo) shares its evaluation and review infrastructure with other developers. Other frameworks can support a similar review interface by providing equivalent run data, outputs, and scores.\n\n## [#](#expert-findings-remain-useful-after-the-review-ends)Expert findings remain useful after the review ends\n\nWith marimo, Acto’s researchers no longer have to reconstruct each case by hand, and an expert’s finding does not end as a detached note. It remains linked to the task, the agent’s output, and the execution evidence. Researchers can return to that assessment, test revised checks against the preserved work, and use what they learn in subsequent evaluations.\n\n*Research context: Acto’s research includes task types represented by [GDPval](https://cdn.openai.com/pdf/d5eb7428-c4e9-4a33-bd86-86dd4bcf12ce/GDPval.pdf) and [Terminal-Bench](https://arxiv.org/abs/2601.11868). It builds on computer-use research including [OSWorld](https://arxiv.org/abs/2404.07972), [CUA-Gym](https://arxiv.org/abs/2605.25624), and [OpenComputer](https://arxiv.org/abs/2605.19769) ([repository](https://github.com/echo0715/OpenComputer)).*\n\n*marimo is the world’s best programming environment for working with data, and it’s all free and open source. Just `pip install marimo` or `uv add marimo` to get started.*", "url": "https://wpnews.pro/news/how-acto-uses-marimo-to-bring-human-judgment-into-ai-evaluation", "canonical_source": "https://marimo.io/blog/case-study-acto", "published_at": "2026-10-09 00:00:00+00:00", "updated_at": "2026-10-12 22:23:49.358937+00:00", "lang": "en", "topics": ["ai-safety", "ai-research", "ai-agents", "developer-tools"], "entities": ["Acto", "marimo", "Viraj Tipnis", "Harbor", "Terminal-Bench", "Result Review"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/how-acto-uses-marimo-to-bring-human-judgment-into-ai-evaluation", "markdown": "https://wpnews.pro/news/how-acto-uses-marimo-to-bring-human-judgment-into-ai-evaluation.md", "text": "https://wpnews.pro/news/how-acto-uses-marimo-to-bring-human-judgment-into-ai-evaluation.txt", "jsonld": "https://wpnews.pro/news/how-acto-uses-marimo-to-bring-human-judgment-into-ai-evaluation.jsonld"}}