Why Your AI Agent Stops Working When Decisions Get Complex

A newly published benchmark, Taste-Bench, tested 14 models on binary decision tasks requiring them to choose between two technical paths without seeing downstream effects, with the best model, GPT-5.6 Sol, reaching 59.7% accuracy across 502 questions against a 25% random-guessing baseline. Accuracy split sharply by information structure: the same model hit 79.7% when evidence was already in context but collapsed to 21.7% when the decision required substantial further investigation, a 58-percentage-point gap. The writeup argues the shortfall is a ceiling on how agents reason about unexplored futures, and recommends decomposing decisions so agents gather evidence or check constraints before committing rather than ranking long-horizon outcomes blind.

You're building an AI agent to handle a real workflow—something that looks tactical on the surface but demands holding multiple constraints across ten steps ahead. You expect it to work well. You get 60%, and it doesn't improve. That's not a tuning problem. That's a ceiling. I started thinking through what happens when an agent has to commit to a path before it sees the consequences. Most people assume agent failures are training failures—bad data, wrong reward signal, not enough examples. The real problem is stranger: even the best models are guessing better than chance but nowhere near reliably when the decision requires integrating evidence the system hasn't already packaged for itself. The gap widens the moment you ask the agent to predict which move will prove stronger ten steps ahead. You should be comfortable with: The figures this piece relies on — sources: arxiv.org. A newly published benchmark called Taste-Bench tested 14 models on binary decision tasks where the agent had to choose between two technical paths without seeing the downstream effects. The best model, GPT-5.6 Sol, achieved 59.7% accuracy across 502 questions. Random guessing hits 25%. So the model is three times better than chance—but that's a floor, not a ceiling for production. What matters is where that accuracy breaks: when the evidence was already in the system's context, the same model hit 79.7%. When the decision required substantial further investigation, it collapsed to 21.7%. That's not a model limitation you can fix with more compute. That's a constraint on how agents reason about futures they haven't yet explored. Your job isn't to ignore this ceiling. It's to design your automation so the agent doesn't have to make decisions that sit in that 21.7% zone. Not every step in your automation is equally hard. Some decisions are queries over existing state; others require the agent to hold multiple constraints and predict which path avoids future traps. Draw your workflow as a chain. Mark each decision point with three questions: The third category is where agents plateau. Focus your automation there. The benchmark found a 58-percentage-point spread between decisions with evidence pre-loaded and decisions requiring investigation. That gap is your roadmap. Instead of asking the agent to decide between two paths sight-unseen, ask it to gather evidence first. Decompose the multi-step decision into separate, simpler steps: python Weak: agent guesses which path is better def choose strategy context : paths = generate candidates context choice = agent.decide paths ~60% accuracy return execute choice Stronger: agent gathers evidence, then decides def choose strategy with evidence context : paths = generate candidates context evidence = {} for path in paths: Ask agent to simulate or check, don't ask it to compare blindly evidence path = agent.gather facts path, context choice = agent.decide with evidence paths, evidence ~80% accuracy return execute choice The difference isn't the model—it's the information structure. When the agent sees consequences spelled out, it gets to the 79.7% zone. When it has to imagine them, it falls to 21.7%. The agent isn't broken. It's just arrived at what it can actually do. Agents are worse at "which future is better?" than "does this path violate constraints?" That's a different cognitive load. Instead of asking your agent to predict and rank long-horizon outcomes, give it a set of constraints and ask it to enumerate paths that satisfy them all. This is weaker than true optimization—you're not finding the best path, you're finding safe paths—but it's more reliable. Weak: "which path is optimal?" choice = agent.rank paths paths Stronger: "which paths don't violate these rules?" constraints = "doesn't exceed resource budget", "doesn't create circular dependencies", "doesn't require tools we don't have access to" safe paths = agent.filter by constraints paths, constraints if safe paths: choice = random.choice safe paths or take the first else: escalate—you hit a case the agent can't handle safely escalate Constraint satisfaction is something agents can do reliably. It's a bound problem, not an unbounded prediction. The gap between 79.7% and 21.7% is the gap between "evidence is here" and "evidence hasn't been gathered yet." If you wait until after execution to surface consequences, you've already committed. Inject feedback before the agent commits. Ask it to make a decision, then immediately check that decision against a simple rule or a past example. If the decision looks risky, escalate or ask the agent to justify its reasoning. python def decide with safety check agent, options, context : choice = agent.decide options Check if this choice matches patterns I've marked as risky if is risky choice, context : Don't execute; ask the agent to explain or pick again reasoning = agent.explain choice choice, context if reasoning is weak reasoning : choice = options 1 fallback to alternative return choice This doesn't make the agent smarter, but it prevents it from acting on weak reasoning. You're not going to get agents to reliable performance on strategic decisions in the next year. The constraint is architectural, not a matter of scale. Plan your automation for 60% agent confidence and human override or escalation for the rest. That sounds bad until you remember what you're replacing: manual work, which also gets tired and makes mistakes. A 60% agent that escalates on low-confidence cases is often better than a human doing it all, especially if the escalations are fast and cheap. Decide upfront: what does your process look like when the agent can't decide? Who reviews it? How fast? That's not a safety mechanism—it's part of the design. Waiting for bigger models. The benchmark tested 14 models, and the spread between the best and the median on long-horizon decisions was tight. Throwing more parameters at the problem doesn't move that ceiling. Focus on information structure instead. Treating 60% as failure. It's failure only if you expected much higher accuracy. If you're replacing a process that was less reliable and high-friction, 60% agent with fast escalation is a win. Ignoring the evidence gap. The difference between 79.7% and 21.7% isn't noise—it's the shape of the problem. If you're getting low accuracy, ask whether the agent actually sees the data it needs to decide. Q: Can I improve this with few-shot examples? Partially. Examples help when they show the agent what evidence to look for. They don't help when the agent has to imagine consequences it hasn't seen. That's why the evidence-gathering step matters—you're not asking the model to extrapolate into unknown territory. Q: What if my workflow genuinely can't gather evidence upfront? Then you're in the hard zone. Consider whether you can decompose the decision into smaller, lower-stakes choices, or whether the problem is a good fit for agentic automation at all. Sometimes the right move is to keep a human in the loop, not because the agent is broken, but because the decision structure demands foresight that current systems don't have reliable ways to automate yet. Q: Does this apply to coding agents, planning agents, all agent types? Yes. Ryan Peters, Director of AI Research at Heisenberg Research Labs, noted that "when an agent has to predict which technical path will prove stronger ten steps ahead—before feedback arrives—even frontier models are guessing better than chance but nowhere near reliably, and that gap widens the moment the decision requires integrating evidence the system hasn't already packaged for itself." This holds across domains. The constraint is about time horizons and incomplete information, not the task.