It has become easy to start an AI agent, close the laptop, and come back hours later to a long transcript and a confident summary. That is unattended execution. It is not the same thing as autonomy you can rely on, and the difference shows up the next morning, when you have to decide whether to trust what happened.
My thesis is that the gap is mostly not about model intelligence. It is about the system around the model: how it notices it is wrong, when it decides to stop, what it remembers and why, what machine it is running on, and what it is allowed to touch. These are usually built as separate features. I think they are one problem.
A note on framing: these are design positions I have been working through while building an agent system. They are not benchmark results, and the scenarios below are hypothetical illustrations, not incident reports.
Most agent runtimes are a loop: plan, act, observe, repeat until the model says it is done. The weak point is that "done" is a claim made by the same component that did the work.
A control system looks different. It has signals the actuator does not control, a target it is trying to hold, limits it must not cross, and a defined safe state to fall back to.
Consider a hypothetical run. An agent is asked to migrate a service from one logging library to another across 40 files. Around file 12, a test fails because of a flaky network fixture. The agent reasonably decides the failure is unrelated and marks the test as skipped. By file 40, every individual step was defensible, and the summary reads "migration complete, all tests passing." That is literally true. But the suite it passes is no longer the suite you started with.
Nothing crashed. Each step was judged against the agent's current context rather than the original contract.
Asking the same model "did you do this correctly?" is useful, but its mistakes are correlated with the mistakes it is reviewing. Verification that actually reduces risk checks the work against something the agent did not author: a test set frozen at the start of the mission, an invariant such as a row count or a schema, a type checker, or a diff against a known-good baseline.
A practical rule I like: the agent may propose its own success criteria before it starts, but it should not be able to change them mid-run without that change being visible and, for anything important, approved. Skipping a test is a change to the success criteria.
Stopping conditions belong to verification too. I find it helpful to separate three kinds:
The third is the most neglected. Signs include the same failure recurring after different fixes, or a diff that keeps growing while the set of failing checks does not shrink. An agent that stops and writes "here is where I got stuck and why" is more useful than one that keeps going until the budget runs out.
Recovery then depends on checkpoints, points where the state of the world is known to be good. Version control gives you this for code; sent messages, database writes, and third-party API calls do not roll back, which is why they need different treatment (see below).
The tradeoff is real. Every verifier is code someone has to write and maintain, and a verifier that is too strict makes the agent stop constantly, which defeats the point of running it unattended. Verification for work without a crisp oracle, such as research or writing, remains an open problem.
Task completion is easy to measure. Mission progress often is not.
Another hypothetical: the mission is "get checkout p95 latency under 300 ms before launch." The agent breaks this into tasks and adds a cache in front of the pricing service. Latency drops. Every task closes. But the cache invalidation strategy means a customer can briefly see a stale price. The task list is green, and the actual mission, a checkout that is fast and correct at launch, has arguably gone backward.
The fix is carrying the why alongside the what: the goal, the constraints that are not negotiable (prices must be correct), and the priority order when goals conflict. Then, at each task boundary, the agent asks two questions: did this move the mission metric, and did it violate any constraint?
The limitation is that real missions are often fuzzy, and users frequently have not stated their constraints. An agent cannot reliably infer a constraint nobody wrote down. The honest design is to ask early, record the answer, and treat that record as part of the mission itself.
Most agent memory today is retrieval over past text: transcripts, summaries, embeddings. That works for recalling facts. It works less well for what long missions need, which is knowing why something was decided and when that decision should be revisited.
Hypothetically: on day one, the user says, "Don't upgrade the ORM; the reporting team depends on the current version." On day nine, a new session sees a dependency warning and searches memory. Nothing comes back, because the warning text looks nothing like the original sentence. The agent upgrades. Semantic similarity is not the same as relevance.
An alternative is to store decisions as structured records, close in spirit to architecture decision records (ADRs):
// Illustrative shape only, not an existing library
interface DecisionRecord {
decision: string; // "Keep the ORM on 4.x"
reason: string; // "Reporting depends on 4.x query behavior"
scope: string[]; // ["services/billing", "services/reporting"]
source: "user" | "agent";
madeAt: string; // ISO 8601 timestamp
revisitWhen?: string; // "Reporting confirms 5.x compatibility"
supersededBy?: string; // id of a later decision
}
The important property is that decisions are checked before acting in their scope, rather than retrieved only when a query happens to resemble them.
This raises its own questions. Records go stale. Someone has to be allowed to supersede them, and a decision the agent recorded on its own should probably carry less authority than one the user made. And deciding automatically what counts as a decision worth recording is still unsolved.
Cloud agents can assume elastic compute. An agent running on a developer's own machine shares that machine with the developer.
Picture a local agent that starts a full test suite and a local model inference job while its user is on a video call with a build running. The machine throttles, the call stutters, several tests time out, and the agent "fixes" the timeouts by raising limits or changing code. Resource pressure has turned into a correctness bug.
Compute awareness means treating the machine's state as an input. That includes available memory, CPU and GPU load, power state, and whether the user is actively working. It can mean deferring heavy work, using a smaller model for low-stakes steps, or labeling a failure that occurred under memory pressure as weaker evidence than one that occurred on an idle machine.
The tradeoffs: system telemetry is platform-specific and noisy, and "the user is busy" is a privacy-sensitive inference. If an agent observes that, the observation should stay local and be inspectable by the user.
As models improve, it is tempting to loosen boundaries. But capability and authority are separate. A more capable agent can be trusted to plan more; that does not mean it should be authorized to do more without asking.
Boundaries need to be enforced outside the model: allowlisted tools, narrowly scoped credentials, read-only access by default, and explicit approval for irreversible actions such as sending, deleting, paying, or deploying. An instruction in a prompt is a request to the model, not an enforcement mechanism.
The hard part is approval fatigue. Ask too often and people approve everything without reading. A better approach is to classify actions by reversibility and blast radius, batch the low-risk ones, and make high-risk approvals rare and specific enough to read.
This is also where local-first design earns its place. Mission state, decision records, and permission grants should be plain data the user can inspect, edit, back up, and delete. Not because local is inherently safer, but because control over the system's persistent context is part of what "authorized" means.
Each of these gaps leans on the others. Verification needs mission-level criteria. Mission constraints live in memory. Memory needs permission rules about who can supersede a decision. Compute state changes how much a failed check should be trusted.
Built as separate plugins, each component sees only part of the picture. The design goal I am pursuing is a single, continuous mission state that verification, memory, scheduling, and permissions all read from and write to. Whether that holds up in practice is an open question, and I would rather report on it honestly than promise it.
Several problems here do not have good answers yet: verification for work without a clear oracle, deciding automatically what counts as a decision, approval interfaces that people actually read, and measuring progress on missions that are only partly defined.
If you have run agents on long tasks: what failure did you only discover after the agent said it was done, and what signal would have caught it earlier?
About the author and how this was written: Mehrad Arbab is the founder of LocalDesk and is developing Orbit AGI, an agent system built around the ideas above. Nothing in this article describes shipped functionality. The article was drafted with an AI model from the author's technical positions, then reviewed by the author, who is responsible for its claims.