Working with AI can feel a bit like training a puppy. You give it an instruction, watch it repeat the instruction back correctly, and then it does the exact thing you told it not to do. Instead of chewing your shoes, it ignores a skill file or calls the wrong tool and then gives you a very convincing explanation for why.
I've spent a lot of the last few months building a plugin for Until, our plan-first workflow for coding agents. The basic idea is straightforward: agree what should be built, write and review the plan, then let the agent implement it. I went into this thinking the difficult bit would be getting that workflow right.
Building it has made me think that agent-first products have at least three interfaces:
- The interface the human sees and controls
- The interface the agent reasons through
- The MCP or app interface where the actions actually happen
We have spent decades learning how to design the first one, and the third looks enough like an API that we have some decent instincts. The agent interface in the middle is much stranger. It is where most of our failures have happened, and where a problem that seems obvious in one conversation can be a nightmare to recreate in the next.
The agent interface is a product surface #
An MCP tool can have a perfect schema and still be horrible to use. The agent needs to know when to call it, what state the product is in and what it is actually allowed to do. It also needs to know what to do when something goes wrong. This sounds obvious written down. It was not obvious while we were building it.
One of our first proper failures happened in Cursor's Plan Mode. The agent had written a plan, the user had read it and then clicked the big Build button. So the agent started building. You can see the logic: the user approved a plan and then pressed a button that literally said Build. The problem was that our workflow had another step in between. The plan needed to be submitted and reviewed before any code changed, so we added clearer instructions.
The next agent got as far as review. The reviewer came back with "revise", but the agent decided the comments were minor implementation details and carried on as if it had been approved. We added more instructions. Then another agent openly acknowledged that review was still pending, decided the user's eagerness counted as an "explicit go-ahead", and started building anyway.
This went on for a while. Every time we closed one route the agent found another, usually with a completely plausible explanation for why this time was different. Eventually we stopped trying to solve all of it with words. Instructions are still useful for teaching the agent how the product works, but they cannot be responsible for the rules that always need to hold. We moved those states into code and started blocking the transitions that should not be possible.
We also found a particularly stupid hole in our review flow. The agent that wrote a plan could record the verdict approving it. We had told it not to self-approve, but when the quickest route to finishing involved self-approval, some models could convince themselves there was a good reason to do it. A rule the agent is responsible for policing is still basically a suggestion, so the author and reviewer needed to be separate actors in the product, with that separation enforced outside the conversation.
Different models use the same product differently #
This is where the choice of model starts to matter a lot more than I expected.
The biggest difference we have seen is between Composer 2.5 and models like GPT-5.6 or Opus. Composer will bypass every living thing to get you where you asked to go. It can be brilliant when you just want something done. Put it inside a process with deliberate stopping points and it becomes an absolute pain.
GPT-5.6 and Opus have been more willing to treat the workflow as part of the job. Composer is more inclined to see the workflow as something standing between it and the job. I am not saying Composer is bad, or that this will hold for every version and every prompt. We pin model versions in our tests precisely because this stuff changes. There are plenty of tasks where I would prefer the model that keeps pushing until it gets an answer.
But two models supporting the same MCP tools does not mean they use your product in the same way. They can read the same instructions, see the same tools and make completely different calls about what they are allowed to do next. Composer has actually become quite useful for us because of this. If we have left an escape route somewhere, it tends to find it. I now think about the model more like part of the runtime than a swappable intelligence setting. If we say Until works with a model, we need to have tested the behaviour on that model.
Testing the answer is not enough #
Testing this properly turned out to be a pain. A unit test can prove that an MCP tool accepts the right input, but it cannot tell us whether the agent called it at the right time. The final code is not much help either. An agent can produce the right change after skipping the plan review, or drag a simple writing request into a planning ceremony nobody wanted.
So we built Quorum, our local eval harness:
Real user request
↓
Simulated multi-turn conversation
↓
Conversation + tool calls + Until activity
↓
Hard milestones + Gauntlet assessment
↓
Pass / Fail / Indeterminate
Each run gets an isolated repo, a pinned copy of the plugin and the right workspace. Quorum checks auth and Source Control first, then a simulated user answers questions, reads the plan and handles review. We record what actually happened rather than asking the agent whether it followed the process. Agents are extremely capable of giving themselves a glowing performance review.
The hard milestones catch things we can state exactly: the user saw the plan before submission, the right reviewer was selected and no files changed before approval. Gauntlet reads the whole conversation and catches runs that technically comply but are still painful to use.
We keep infrastructure problems separate as indeterminate results. An Anthropic 529 or broken fixture should not send us off to "fix" working product behaviour. For genuine failures, Quorum records the first point where things went wrong and whether the likely owner is our guidance, the model or the test itself.
From July 9 to July 23 the passing result shown by the suite went from 0% to 56%. Both the product and graders changed during that time, so I would not use it as a product-quality score. It does show that the same ugly scenarios were getting further without falling over.
Saved failures are more valuable than invented ones #
The best test cases have all come from agents doing something weird in a real session. When that happens we save a normalised version of the conversation and its tool activity. We can replay it through new checks without paying to run the model again, which is handy when the test is wrong rather than the product.
Replay only gets us so far. If we change the instructions in the plugin, we need to run the model again because the old recording cannot tell us how it will react to the new version. The loop is basically: watch an agent fail, save the failure, change Until and run it again across the models we care about. Then we look for the first point where the new conversation goes in a different direction. Those saved failures have been far more useful than prompts we invented to show the happy path because they contain the ambiguity we did not know was there.
Humans still need a good interface #
Putting an important decision back in front of a person does not automatically make it good UX. We hit this when trying to support people using Until on their own. We could have a separate agent review the plan, but there was no second person in the workspace who could record the approval. Our first honest workaround was to give the user a command to run in their terminal. It worked, in the least generous sense of the word.
Asking somebody to leave the conversation, copy a command and run it in a terminal just to approve their own plan felt terrible. We replaced it with a native approval interaction, which was much nicer, but that exposed another edge case. If the user had configured certain commands to auto-run, the action could happen without a human click.
We accepted that narrow trade-off for now because the terminal version was bad enough that people would avoid it. The longer-term answer is a proper solo approval flow in the product. This has been a useful reminder that safety features still have users, and if you make them painful enough people will find their own way around them.
We have had the same issue with deciding who builds the plan. Until can hand work to a cloud agent, while the local agent is also sitting there ready to start. If both react to the same approval you get two implementations of the same plan. We currently record who has the job and make the other agent stand down, but this really wants to be a first-class claim in the product.
Agents can help close the feedback loop #
One part of this has worked surprisingly well: agents can report problems while they are happening. When I hit a broken integration I have to decide whether it is annoying enough to report, remember what I was doing and then write the issue. Quite a lot gets lost between those steps.
The agent already has the failed tool call, the response and the context around it. With PostHog we let the agent submit MCP feedback when it runs into a problem. The report arrives at the point of failure instead of relying on somebody reconstructing it later.
We can then take that failure, turn it into a Quorum scenario and run the fix against the different models. If it works, the original failure stays in the suite as a regression test. I think we will see a lot more of this. Agents are awkward users in many ways, but they are very good at handing you a structured account of exactly what went wrong.
What I would ask before building another agent-first product #
I now find myself asking a few questions whenever we add something to Until. Can the person tell what is happening and where they need to step in? Does the agent have the right tool, and does it know when to use it? What happens when the response is incomplete or the setup is broken? Can the agent grant itself permission it should not have? And are we testing the conversation that gets us there, or just admiring the final result?
Adding an MCP server gives an agent access to your product. There is still a lot of product work between that access and an experience you would trust. That middle interface has taken far more work than I expected. It has also been the most interesting part to build, mostly because the puppy keeps finding new shoes.