{"slug": "navigating-the-solution-space-with-agents", "title": "Navigating the solution space with agents", "summary": "Coding agents dramatically lower the cost of exploring solution spaces but remain path-dependent and poor at deciding optimal endpoints, argues Mitchell Hashimoto and others in a debate about reading versus not reading code. The discussion highlights that models reward-hack by producing short-horizon, purely additive changes that create local optima, and that existing benchmarks are broken and prone to reward hacking, as noted by Cursor and OpenAI.", "body_md": "The “read the code vs don’t read the code\" debate has [been](https://x.com/mitchellh/status/2072738025344565262) [raging](https://x.com/bendee983/status/2081117964414435367?s=20) lately ([@dexhorthy](https://x.com/@dexhorthy) has the best take on it). But I think it's missing an important framing: The job we are doing isn’t merely about reading or writing code - it’s navigating the solution space. Coding agents have dramatically lowered the cost of exploring that space, but they’re still incredibly path dependent and poor at deciding where in it you should end up.\n\n## False Dichotomy\n\nThe debate is a false dichotomy for two reasons.\n\n**First of all, it's clearly context dependent**: scrutiny should scale with risk. So the first principle to use in my opinion is of concentric circles of risk - the closer you are to the \"core\" (where changes have a large blast radius) the more you need to understand what's going on. The further away you are to broad impact, the more you can accept risk and less maintainable code with the goal of moving faster.\n\n**Second, “reading the code” can mean wildly different things.** If any one means literally not reading a single line of code and only interacting through the agent - I can't understand how they'd be building any consequential software at that altitude. There's some amount of \"understanding the territory\" that can't avoid looking at code. Perhaps we'll get to a point of needing to soon, but we are not even close right now.\n\nIf they mean understanding the existing architecture, reviewing diffs critically, asking questions, digging into suspicious areas... well then I'd still call that \"reading the code\" (just with augmentation).\n\n## Reward Hacks\n\nAny one that's actively working on long term projects with coding agents will immediately notice a couple of things: models tend to be purely additive rarely refactoring, they tend to be overly conservative and maintain awkward backwards compatibility, and ultimately they love to reward hack.\n\nIf you understand how models are rewarded, you'll understand how they work. The benchmarks that models are trained on tend to be short horizon and don't adequately evaluate how maintainable the code they produce are long term. They will tend to learn to produce the simplest possible code changes that are sufficient to pass the narrow tests of the evals they're hillclimbing. In the context of real-world projects, these changes tend to be local optima.\n\nTo top it off, plenty of benchmarks that form the foundation of modern coding agents, are [flat out broken](https://openai.com/index/separating-signal-from-noise-coding-evaluations/) and [prone to reward hacking](https://cursor.com/blog/reward-hacking-coding-benchmarks?utm_campaign=ALL_ENG_Cursor-Blog-June-2026&utm_content=ALL_ENG_EM_Cursor-Digest-June-2026&utm_medium=email&utm_source=customerio). We probably need [new benchmarks](https://x.com/dexhorthy/status/2081797628552270027).\n\n## A real world example\n\nJust last night, I worked on a change that required 10+ rounds of iterations with Sol (other models were worse, I tried) to get to a net ~200 LOC that I was finally satisfied with.\n\n- The initial prompt was straightforward: \"Right now when the backend deployed is down the frontend just hangs in loading state forever\".\n- Right away I noticed the model was doing a bunch of work to maintain the \"legacy\" feature flag structure we had that I was able to tell it to remove altogether (all of our deployed apps since late last year have that endpoint - code cleanup!)\n- While asking the model about how the AbortSignal mechanism it introduced worked, it also suggested combining it with ReactQuery's own abort signal (why didn't it do that initially? but at least I got to learn more about how AbortSignals work!)\n- Then I realized that this change introduced rendering of visual components to a\n`FeatureFlagProvider`\n\nwhere it previously didn't exist. The agent tried injecting rendering logic to a React provider that previously was only about managing state. This added a much more complex set of rending states than we had before. - Only by looking at the code I realized we could actually push that down to the children and keep a similar rendering chain as we had before, simplifying things drastically, and making the user experience better (users can switch to a deployed app that isn't having an error, or logout)\n- Even tests require reading - during these changes the agent introduced some weird test scaffolding\n`FeatureFlagState`\n\nthat rendered different texts based on the feature flag the provider provided. This was super awkward - rather than rendering a fake visual component the app will never have, we can just use renderHook to assert the state computed directly - Finally - I noticed that the model had produced a new AppUnavailable component for rendering that specific error state.\n**We already had an Error component** for other error states, why not just use that and preserve visual consistency? The model had also exposed a bunch of machinery in the`FeatureFlagProvider`\n\ncoupled to that AppUnavailable component. Nudging the model to consolidate let us cut down a few more hundreds of lines of code.\n\nThat example is useful because it's a simple problem to fix but if I had stopped at any point along the iterations, the code and product would've been much worse off.\n\nThe models are extremely path dependent and tend to want to only add and special case things, never remove/simplify/question abstractions. At the same time, without the coding agent I probably would've produced much worse code myself (or never gotten around to doing it).\n\n## Guiding the ship\n\nThe best mental model I've come up with for where we're at is that **coding agents are amazing tools to help us traverse the solution space really rapidly.** They drastically bring down the cost of us trying things out, comparing results, asking lots of questions and learning along the way.\n\nBut ultimately, there's still **a lot of judgement and understanding necessary to guide them to a globally optimal location in the solution space**. And perhaps equally as important, taking an active role in that process is what lets you build a rich mental model of the solution - which compounds over time.", "url": "https://wpnews.pro/news/navigating-the-solution-space-with-agents", "canonical_source": "https://vimota.me/writing/navigating-the-solution-space-with-agents", "published_at": "2026-07-28 18:07:52+00:00", "updated_at": "2026-07-28 18:22:32.417480+00:00", "lang": "en", "topics": ["ai-agents", "ai-tools", "ai-research", "ai-ethics"], "entities": ["Mitchell Hashimoto", "Cursor", "OpenAI", "Sol", "ReactQuery"], "alternates": {"html": "https://wpnews.pro/news/navigating-the-solution-space-with-agents", "markdown": "https://wpnews.pro/news/navigating-the-solution-space-with-agents.md", "text": "https://wpnews.pro/news/navigating-the-solution-space-with-agents.txt", "jsonld": "https://wpnews.pro/news/navigating-the-solution-space-with-agents.jsonld"}}