You're building an AI agent to handle a real workflow—something that looks tactical on the surface but demands holding multiple constraints across ten steps ahead. You expect it to work well. You get 60%, and it doesn't improve. That's not a tuning problem. That's a ceiling.
I started thinking through what happens when an agent has to commit to a path before it sees the consequences. Most people assume agent failures are training failures—bad data, wrong reward signal, not enough examples. The real problem is stranger: even the best models are guessing better than chance but nowhere near reliably when the decision requires integrating evidence the system hasn't already packaged for itself. The gap widens the moment you ask the agent to predict which move will prove stronger ten steps ahead.
You should be comfortable with:
The figures this piece relies on — sources: arxiv.org.
A newly published benchmark called Taste-Bench tested 14 models on binary decision tasks where the agent had to choose between two technical paths without seeing the downstream effects. The best model, GPT-5.6 Sol, achieved 59.7% accuracy across 502 questions. Random guessing hits 25%. So the model is three times better than chance—but that's a floor, not a ceiling for production.
What matters is where that accuracy breaks: when the evidence was already in the system's context, the same model hit 79.7%. When the decision required substantial further investigation, it collapsed to 21.7%. That's not a model limitation you can fix with more compute. That's a constraint on how agents reason about futures they haven't yet explored.
Your job isn't to ignore this ceiling. It's to design your automation so the agent doesn't have to make decisions that sit in that 21.7% zone.
Not every step in your automation is equally hard. Some decisions are queries over existing state; others require the agent to hold multiple constraints and predict which path avoids future traps.
Draw your workflow as a chain. Mark each decision point with three questions:
The third category is where agents plateau. Focus your automation there.
The benchmark found a 58-percentage-point spread between decisions with evidence pre-loaded and decisions requiring investigation. That gap is your roadmap.
Instead of asking the agent to decide between two paths sight-unseen, ask it to gather evidence first. Decompose the multi-step decision into separate, simpler steps:
def choose_strategy(context):
paths = generate_candidates(context)
choice = agent.decide(paths) # ~60% accuracy
return execute(choice)
def choose_strategy_with_evidence(context):
paths = generate_candidates(context)
evidence = {}
for path in paths:
evidence[path] = agent.gather_facts(path, context)
choice = agent.decide_with_evidence(paths, evidence) # ~80% accuracy
return execute(choice)
The difference isn't the model—it's the information structure. When the agent sees consequences spelled out, it gets to the 79.7% zone. When it has to imagine them, it falls to 21.7%.
The agent isn't broken. It's just arrived at what it can actually do.
Agents are worse at "which future is better?" than "does this path violate constraints?" That's a different cognitive load.
Instead of asking your agent to predict and rank long-horizon outcomes, give it a set of constraints and ask it to enumerate paths that satisfy them all. This is weaker than true optimization—you're not finding the best path, you're finding safe paths—but it's more reliable.
choice = agent.rank_paths(paths)
constraints = [
"doesn't exceed resource budget",
"doesn't create circular dependencies",
"doesn't require tools we don't have access to"
]
safe_paths = agent.filter_by_constraints(paths, constraints)
if safe_paths:
choice = random.choice(safe_paths) # or take the first
else:
escalate()
Constraint satisfaction is something agents can do reliably. It's a bound problem, not an unbounded prediction.
The gap between 79.7% and 21.7% is the gap between "evidence is here" and "evidence hasn't been gathered yet." If you wait until after execution to surface consequences, you've already committed.
Inject feedback before the agent commits. Ask it to make a decision, then immediately check that decision against a simple rule or a past example. If the decision looks risky, escalate or ask the agent to justify its reasoning.
def decide_with_safety_check(agent, options, context):
choice = agent.decide(options)
if is_risky(choice, context):
reasoning = agent.explain_choice(choice, context)
if reasoning_is_weak(reasoning):
choice = options[1] # fallback to alternative
return choice
This doesn't make the agent smarter, but it prevents it from acting on weak reasoning.
You're not going to get agents to reliable performance on strategic decisions in the next year. The constraint is architectural, not a matter of scale. Plan your automation for 60% agent confidence and human override or escalation for the rest.
That sounds bad until you remember what you're replacing: manual work, which also gets tired and makes mistakes. A 60% agent that escalates on low-confidence cases is often better than a human doing it all, especially if the escalations are fast and cheap.
Decide upfront: what does your process look like when the agent can't decide? Who reviews it? How fast? That's not a safety mechanism—it's part of the design.
Waiting for bigger models. The benchmark tested 14 models, and the spread between the best and the median on long-horizon decisions was tight. Throwing more parameters at the problem doesn't move that ceiling. Focus on information structure instead.
Treating 60% as failure. It's failure only if you expected much higher accuracy. If you're replacing a process that was less reliable and high-friction, 60% agent with fast escalation is a win.
Ignoring the evidence gap. The difference between 79.7% and 21.7% isn't noise—it's the shape of the problem. If you're getting low accuracy, ask whether the agent actually sees the data it needs to decide.
Q: Can I improve this with few-shot examples?
Partially. Examples help when they show the agent what evidence to look for. They don't help when the agent has to imagine consequences it hasn't seen. That's why the evidence-gathering step matters—you're not asking the model to extrapolate into unknown territory.
Q: What if my workflow genuinely can't gather evidence upfront?
Then you're in the hard zone. Consider whether you can decompose the decision into smaller, lower-stakes choices, or whether the problem is a good fit for agentic automation at all. Sometimes the right move is to keep a human in the loop, not because the agent is broken, but because the decision structure demands foresight that current systems don't have reliable ways to automate yet.
Q: Does this apply to coding agents, planning agents, all agent types?
Yes. Ryan Peters, Director of AI Research at Heisenberg Research Labs, noted that "when an agent has to predict which technical path will prove stronger ten steps ahead—before feedback arrives—even frontier models are guessing better than chance but nowhere near reliably, and that gap widens the moment the decision requires integrating evidence the system hasn't already packaged for itself." This holds across domains. The constraint is about time horizons and incomplete information, not the task.