# When Building Gets Cheap, Knowing What to Build Gets Expensive > Source: > Published: 2026-08-20 18:37:23+00:00 When building gets cheap, knowing what to build gets expensive. AI coding agents have made it surprisingly cheap to turn an idea into working code. That sounds unambiguously good, but it creates a new failure mode: we can now build the wrong thing much faster. We ran into this recently while working on AdaL. We had a strong feeling that AdaL was less token-efficient than Claude Code, and the initial investigation seemed to support it. We found architectural differences that could explain the gap, and agents quickly produced several plausible fixes. A year ago, those fixes might have been expensive enough to force a careful debate. Now an agent could implement some of them in hours. So instead of building them, we built the benchmark first. The benchmark told us our intuition was wrong. Adal is more token-efficient than Claude Code. Plausible ideas are getting dangerously cheap Our original observation was simple: AdaL felt more token-hungry than Claude Code in long sessions. The first breakdown showed that tool-call results occupied roughly 80% of the context window. That immediately suggested a mechanism: older tool output often contributes less to the current decision, so perhaps carrying all of it forward was an important source of inefficiency. We then compared how Claude Code, OpenCode, and Pi appeared to handle long histories and tool results. Agents were extremely useful here. They could inspect architectures, identify differences, and generate possible improvements much faster than we could manually. One idea looked especially compelling: compact old tool-call results after five or ten turns. The reasoning was straightforward. If old tool output dominates the context while contributing less over time, making it smaller should reduce token usage. It was exactly the kind of explanation that could turn into a feature before anyone had established that the original problem was real. A plausible explanation is not evidence that there is an opportunity worth solving. Cheap implementation makes that distinction more important, not less. Reverse the feature loop The usual product loop looks something like: idea → research → design → implementation → launch → measurement For ambiguous R&D, we've started preferring a different sequence: idea → observability → hypothesis → metrics → benchmark → decision → implementation The important change is where implementation happens. Instead of asking whether an agent can build a proposed improvement, we first ask whether we can demonstrate an opportunity worth building for. If we don't know how to measure the improvement, the first deliverable should be the measurement, not the feature. Only then could we ask the product question that mattered: is the gap large and consistent enough to deserve engineering time? The first benchmark rejected our hypothesis We first ran AdaL and Claude Code on a subset of SWE-bench, using the same model and effort setting and repeating the benchmark three times. We expected Claude Code to be more efficient. Instead, in the displayed run, both systems reached the same 50% pass rate — 25 out of 50 tasks — while AdaL reported lower total cost: $31.94 versus $42.09. AdaL was also slightly faster in this run, averaging 511 seconds per instance versus 563 seconds. Figure 1. SWE-bench subset comparison. Both harnesses reached a 50% pass rate (25/50); the displayed AdaL run reported $31.94 in total cost versus $42.09 for Claude Code. This didn't prove that AdaL was universally more efficient. The benchmark covered a limited task set and one style of usage. But it did reject the simplest version of our original belief: we could no longer claim that AdaL was obviously less token-efficient on long-running coding tasks. That left another possibility. SWE-bench is mostly headless: the agent receives a task and continues with little additional user input. Real developers use coding agents interactively, changing requirements, asking follow-up questions, requesting explanations, and redirecting work over many turns. Maybe AdaL performed well headlessly but became less efficient as conversations grew. So we tested that instead. We changed the hypothesis instead of defending it Using the SDKs exposed by AdaL and Claude Code, we simulated increasingly long interactive sessions: single-turn queries, short conversations, and longer multi-turn sessions. We kept the comparison as controlled as possible and inspected how much context each system re-fed as conversations grew. The result was much less dramatic than our intuition suggested. Across the displayed scenarios, context usage remained in roughly the same range: essentially identical in the one- and three-turn cases, lower for AdaL in the five-turn case, and about 1.1× higher in the ten-turn case. Figure 2. Selected interactive scenarios. Context growth remained broadly comparable across the displayed single-turn and multi-turn runs. CC means Claude Code. The aggregate picture told the same story. AdaL re-fed about 3.43M context tokens versus 3.26M for Claude Code, roughly a 1.1× difference overall. Claude Code did have better cache performance in this sample — a 99% average cache-hit rate versus 94% for AdaL — but we still weren't seeing the large efficiency gap that had motivated the investigation. Figure 3. Aggregate context and cache snapshot from the interactive benchmark. AdaL re-fed roughly 3.43M context tokens versus 3.26M for Claude Code, while Claude Code achieved the higher cache-hit rate in this sample. Again, the evaluation wasn't comprehensive enough to establish universal equivalence across every repository, model, prompt, or workload. But it was sufficient for the decision in front of us: there was no demonstrated easy win that justified immediately rebuilding AdaL's context-management behavior. At this point, the interesting result was no longer the token comparison. It was what the benchmark had prevented us from building. The obvious optimization might have made things worse Remember the attractive idea from the beginning: compact old tool-call results. Looking only at visible context size, it makes sense. If older tool output occupies most of the context and contributes less to the current response, making it smaller appears to be an obvious optimization. But visible context size isn't the whole system. Rewriting older turns also changes the existing prompt prefix, which can reduce or invalidate prompt-cache and KV-cache reuse. In our analysis, the next turn could become roughly ten times more expensive even though the prompt itself looked smaller. There was a second complication. A first-party client such as Claude Code can potentially coordinate with provider-side caching or context-management capabilities that independent harnesses cannot assume are available. AdaL, OpenCode, and Pi primarily operate at the harness layer; they don't own the model-serving layer. That means copying a visible behavior without the same underlying infrastructure can produce the opposite result. You can successfully reduce context size and still increase end-to-end cost. The agent had identified a legitimate technical idea. It had not established the opportunity, the guardrail, or the full system-level tradeoff. Agents reduce the cost of implementation. They do not reduce the cost of being wrong. In some cases they amplify it, because being wrong can now produce much more code. Not building can be the successful outcome Our token-efficiency investigation ended without the feature we expected to build. We began with a belief that AdaL had a meaningful disadvantage against Claude Code. We found convincing explanations for that belief, real product differences, and technically plausible improvements. Then we designed experiments that were capable of proving us wrong. They did. That saved us from spending engineering time on changes that might have produced little benefit — or actually increased cost by damaging cache reuse. This is an outcome AI-native engineering teams need to get more comfortable with. A benchmark isn't bureaucracy before the real work. In ambiguous R&D, sometimes the benchmark is the real work. Measure before you let agents build Our default process for ambiguous R&D is becoming simple: start with the signal, instrument the system, turn the intuition into a falsifiable hypothesis, define the metric and guardrails, and build a benchmark that is capable of proving the hypothesis wrong. Only then decide whether there is anything worth implementing. Agents are incredibly good at expanding the hypothesis space. Metrics and experiments narrow it. The combination lets us move quickly without confusing the ability to build something with evidence that it should exist. As implementation becomes cheaper, experiment design and opportunity sizing become more valuable. In a world where almost any plausible feature can be built, the scarce skill is knowing which code should exist at all. AdaL gives 7 days free trial ! curl -fsSL [https://adal.sylph.ai/install.sh](https://adal.sylph.ai/install.sh) | bash