I have a bad habit. When an agent fails a task, I turn the reasoning dial up. More thinking, more tokens, more chances to stumble into the right answer. It works just often enough to keep the habit alive, which is how bad habits survive in this field.
A paper I read this week argues pretty directly that I've been buying the wrong thing. OpenVLHarness: Real World Visual Agentic Tool-Use with Multimodal Memory runs a controlled comparison, and the controlled part is the whole story. Same model in every arm. One arm gets the harness at medium effort. The other arms get maximum effort: the bare model, a Codex-style code-as-action setup, and Visual Sketchpad. The harness arm wins, and it wins while spending a fraction of what the max-effort arms burn.
Sit with that for a second. The model that thought less, with better plumbing, beat the model that thought harder. Not by a rounding error — by a margin you'd notice in a product.
| Arm | What it optimizes | Where it leaks |
|---|---|---|
| Bare model, max effort | More sampling per step | Re-derives state every step from raw tool output |
| Codex-style code-as-action | Precise, composable actions | Great when the world is text; awkward when the observation is visual |
| Visual Sketchpad | Intermediate visual reasoning | The sketch is per-step, not a memory |
| Harness, medium effort | Model-readable, persistent observations | Needs per-tool renderers, which is real work |
The first three are all reasonable engineering. I've built versions of all three. What they share is that the model keeps paying to re-understand its own situation. Every step, the tool returns something — JSON, a screenshot, a diff, a log tail — and the model has to convert that into an internal representation before it can act. Then the next step arrives and it does the conversion again, because nothing carried over except a growing transcript.
Max effort is the expensive way to survive that. You're not giving the model more information. You're giving it more attempts at re-parsing the same ambiguous blob. That's a tax, not a capability.
Two things, and neither of them is clever.
It renders tool output into something the model can read directly. Not raw JSON with nested escaping, not a full-resolution screenshot where the relevant widget is forty pixels wide. A representation chosen per tool, at the granularity the next decision needs. This is unglamorous work. It's also most of the win.
It persists that representation across steps. The model's understanding of the screen, the document, the file — it lives in a memory the harness maintains, not in a context window that gets truncated at step twelve. So step thirteen starts from a state, not from an archaeology project.
Put those together and medium effort is enough, because the model isn't spending its budget reconstructing reality. It's spending it on the task.
Because it means the bottleneck was never the model. It was the interface between the model and its own tools, and that interface is something I control. I can't ship a better frontier model. I can absolutely ship a renderer that turns my scraper's output into three lines the model can act on instead of a 4KB HTML fragment it has to squint at.
I did the arithmetic on my own stack after reading this. A screenshot-heavy agent loop re-sends its history every step. A couple thousand tokens per image, thirty steps, and you're well into six figures of input tokens before the agent has accomplished anything you'd put in a changelog. Most of that spend is the model re-reading the same picture it already understood at step four. That's not reasoning. That's me paying for amnesia.
The harness is domain-specific. Someone had to write renderers for those tools, and those renderers encode assumptions about what matters in a visual observation. Change the domain and you rewrite them. That's the trade: you spend engineering time up front to stop spending tokens forever. Whether that's a good deal depends entirely on whether your agent runs once or ten thousand times. For a demo, max effort is fine. For anything with a bill attached, it isn't.
I also haven't run this myself, and visual agentic tasks are a specific slice of the world. Text-only tool use might not transfer cleanly — my tools return logs and diffs, not pixels, and the rendering problem looks different there. Maybe the persistence half is the bigger lever and the rendering half matters less outside vision. I genuinely don't know yet.
Three things, in order of how cheap they are.
Log every tool call with its full args and full output, before I need it. I already believe this and I still skip it on prototypes, and every time I skip it I spend an afternoon guessing.
Then write one renderer per tool that produces the smallest representation the next decision actually needs. Not a summary — a rendering. Deterministic, testable, versioned like any other code.
Then move state out of the transcript and into a store the harness owns, so the model reads current state instead of replaying history.
And measure the thing that matters: tokens per solved task, not tokens per step. Per-step cost rewards agents that give up quickly. Per-solved-task cost is the only number that shows up in a budget meeting.
The uncomfortable lesson from that comparison is that I've been treating reasoning budget as the knob. It's the knob that's easiest to turn and the one that pays back least. The harness is the boring answer, and boring answers are usually the ones that survive contact with prod.