Frontier agent research needs real computers #
ARIMLABS builds production-fidelity simulations and benchmarks for long-horizon AI agents. Frontier labs use these environments to see how models behave when a task requires tools, real systems, and decisions that unfold over hours or days.
This kind of evaluation is not a prompt followed by a score. Each agent needs an isolated computer where it can inspect systems, execute commands, use tools, and take actions. A serious benchmark may need many of those computers at once so researchers can run trials in parallel and compare the results.
ARIMLABS wants to know how models behave in realistic environments. To find out, it needs to reproduce those environments across thousands of isolated trials without turning its researchers into infrastructure operators.
From a small pilot to 500,000 executions a day #
"We started with a small pilot. Now we run evaluations 24/7, with roughly half a million executions per day."
Mykyta Mudryi, CEO of ARIMLABS
ARIMLABS initially ran its agent evaluations on local machines. That works when a researcher is testing a few ideas. It becomes restrictive when an evaluation calls for many isolated trials at the same time.
Harbor already organized ARIMLABS's evaluation runs, but the computers beneath those runs were still local. That tied capacity to researchers' machines and limited how many trials could run in parallel.
ARIMLABS moved that workload onto Islo. Each agent gets a dedicated cloud computer, and the team can start many trials in parallel without depending on a researcher's laptop.
"Islo lets us execute agents in the cloud instead of running them locally. That allows us to scale to a large number of trials with many parallel agents running at the same time."
Mykyta Mudryi, CEO of ARIMLABS
The change was not simply local compute replaced by cloud compute. ARIMLABS gained a repeatable way to give every trial the environment it needs, keep runs isolated from one another, and add capacity without rebuilding the runtime for every new batch.
Running Harbor evals on Islo #
ARIMLABS runs its benchmark workflow with Harbor, the open-source framework for running agent evaluations. ARIMLABS defines the models, agents, tasks, and evaluation logic. Islo's Harbor integration supplies and manages the computers beneath those runs.
That separation lets the researchers stay focused on what they want to measure. Harbor organizes the evaluation, while Islo creates the isolated environment in which each agent acts. A new batch can use the same infrastructure path without the team manually provisioning a fleet of machines first.
The environments run under network policies as well. For security-oriented evaluations, isolation is part of the benchmark design. An agent needs enough freedom to complete the task without receiving uncontrolled access outside its environment. Islo provides the sandbox and network-control layer, while ARIMLABS decides what each benchmark permits.
Every failed trial stays inspectable #
"It is easy to separate different trials, batches, and rollouts. If we encounter a problem with a sandbox or Docker image, we can open the logs for that specific sandbox, debug it, and rerun it."
Mykyta Mudryi, CEO of ARIMLABS
Parallelism creates a second problem after capacity: researchers need to know which run did what.
ARIMLABS uses Islo's dashboard to distinguish individual trials, batches of trials, and rollouts. When a sandbox or Docker image causes a problem, the team can go directly to the affected computer, inspect its logs, debug the failure, and run the trial again.
That trace from a batch down to one environment matters for evaluation quality. A failed run may reveal something about the model, the task, or the environment itself. Researchers need to separate those possibilities before they can trust the result. The computer behind each trial cannot be disposable in the sense of becoming unknowable once a run ends.
Islo gives ARIMLABS both layers it needs: enough parallel computers to run meaningful batches and enough observability to investigate one execution inside them.
More evaluations without becoming an infrastructure team #
"There's incredible demand from frontier labs for high-quality reinforcement learning environments. Labs like us need to employ every bit of focus to produce high-quality research. Any time we spend on infrastructure is wasted."
Mykyta Mudryi, CEO of ARIMLABS
The half-million daily executions are evidence of a more important change. ARIMLABS can sustain an around-the-clock evaluation workload without provisioning and operating the underlying environment infrastructure itself.
Research environments do not stay fixed. A new evaluation may require a different service, tool, policy, or capability inside the agent's computer. Islo works with ARIMLABS to add those capabilities as the research evolves, so the team does not have to choose between a better benchmark and weeks of infrastructure work.
For teams building agent evaluations, the computer layer shapes what can be measured. Agents need isolated environments where they can take real actions. Researchers need parallel capacity for meaningful batches, plus the ability to inspect one failure and rerun it with context intact. ARIMLABS started with a small pilot on local machines. It now runs evaluations continuously at roughly 500,000 executions per day. The researchers can spend their time designing better simulations and studying agent behavior because the computers beneath every trial are already there when the next batch begins.