Your Agent Aced the Task. Will It Do It Again? IBM Research introduced consistency guidelines in its altk-evolve toolkit, built on a new Consistency Analyzer, to address a 24.4-point consistency gap in LLM agent reliability. A ReAct agent using GPT-4.1 on AppWorld's test_normal succeeded on 77.4% of runs on average (Mean@5) but succeeded on all five runs for only 53.0% of tasks (Pass^5). IBM Research reported that applying the guidelines halved the gap from 24.4 percentage points to 12.0 percentage points, with same-task Pass^5 up 16.0 points and similar-task up 13.0 points, without reducing average accuracy. Your Agent Aced the Task. Will It Do It Again? Enterprise Article https://huggingface.co/blog That is embarrassing onstage. In production, it is a reliability problem: a workflow that succeeded once may fail the next time a user makes the same request. For mission-critical work, such as reconciling a financial transaction or checking a contract for an obligation, that can be a showstopper. Most benchmarks hide this variability behind an average. On AppWorld, a ReAct agent using GPT-4.1 succeeded on 77.4% of runs across five repetitions. But it succeeded in all five runs for only 53.0% of tasks — a 24.4-point consistency gap . Most benchmarks report the first number. We built a way to measure the second — and improve it. In an earlier post, we introduced ALTK-Evolve https://huggingface.co/blog/ibm-research/altk-evolve — a system that turns an agent's own past trajectories into reusable guidelines, distilled automatically and injected back at inference time. It measurably improves task success, but those results only asked the average-case question too. This post introduces consistency guidelines , a new guideline type in altk-evolve https://github.com/AgentToolkit/altk-evolve built on top of a diagnostic tool we call the Consistency Analyzer , that targets this gap directly. TL;DR - Accuracy hides an unreliability problem. A ReAct agent GPT-4.1 on AppWorld test normal that succeeds 77.4% of the time on average succeeds on all 5 repeated runs for only 53.0% of tasks — a 24.4-point consistency gap . On hard tasks it reaches 30 points. - We built a diagnostic for exactly this. The Consistency Analyzer resamples an agent's own recorded trajectory to find flip-prone decision points — steps where the model was one token-sample away from doing something different. It needs one trace and no ground truth — it resamples each decision point in that trace with a single call requesting k completions k=5 by default , rather than re-running the task end-to-end. - Turning that diagnosis into guidelines halves the gap — from 24.4pp to 12.0pp same-task Pass⁵ +16.0pp, similar-task +13.0pp , without costing anything in average accuracy. - Full methodology and evaluations are in the technical report on arXiv https://arxiv.org/abs/2609.08832 . The Metric Almost Nobody Reports Standard agent evaluation reports Mean@k : run a benchmark k times, average the pass rate. Often k =3, sometimes just 1. It's the number on every leaderboard, and it's what "77% accurate" means in practice. Mean@k answers "how good is this agent, on average?" It does not answer the question a real user cares about: will it still be good if I ask this exact question again? For that you need Pass^k : the fraction of tasks where the agent succeeds on all k runs. ⚠️ Pass^k is not Pass@k. The familiar Pass@k is optimistic — it asks whether at least one of k attempts succeeded, the right question when you can verify and retry. Pass^k is its pessimistic mirror image: every attempt must succeed. Same letters, opposite question. Pass^k ≤ Mean@k ≤ Pass@k, always. A ReAct agent backed by GPT-4.1 posts a Mean@5 of 77.4% — genuinely strong. But Pass^5 is only 53.0% . Nearly a quarter of the benchmark consists of tasks the agent can sometimes solve and sometimes can't, with nothing about the task changing between runs. We call this gap — Mean@k minus Pass^k — the consistency gap . This isn't a capability problem you fix with a bigger model. It's an orthogonal axis: an agent can be capable and inconsistent at the same time. Why Agents Flip: Sharp Decisions vs. Flat Ones Every time an LLM agent decides something — which API to call, what argument to pass, whether to retry — that decision comes out of a probability distribution over next tokens. What matters is the shape of that distribution. A sharp one puts most of its mass on a single token: the runners-up are far behind, and the same choice comes out run after run. A flat one spreads comparable mass across several near-tied tokens, and which one wins is close to a coin flip. The shape decides how much noise it takes to change the outcome. Sharp distributions are resilient — GPU floating-point non-associativity, request batching, and other platform-side effects nudge the numbers slightly, but nowhere near enough to reorder a clear winner. Flat distributions are vulnerable to exactly that nudge: near-ties may reorder under small perturbations. And because a trajectory chains dozens of decisions, a small per-step chance of flipping compounds into a large chance that some run goes differently. That's where a 24-point gap comes from. This is also why the problem survives your decoding settings. Greedy decoding and a fixed seed both govern how a distribution gets turned into a token — they say nothing about the distribution itself. On a hosted endpoint the probabilities shift slightly from run to run, so the same prompt to the same model at temperature zero can still resolve a near-tie one way today and the other way tomorrow. Our setup: the ReAct agent runs at temperature 0.0 , so none of the variance above is ordinary sampling. Diagnose, Then Fix Which turns the problem into a search: which steps in a given trajectory were the flat ones — and what do you do about them once you know? Consistency guidelines come out of a two-stage pipeline that plugs into ALTK-Evolve's existing machinery — with a new source signal driving what gets written. 1. Detect — the Consistency Analyzer. Given one recorded trajectory, the analyzer replays each decision step through controlled resampling, measuring how much the model's output actually varies at that point. Concretely, that's one additional model call per decision step, done once offline — issued with the sampling parameter set to draw k completions at once k=5 by default — replayed against the already-recorded context, not new tool calls, not new environment interactions, and not a second end-to-end rollout of the task. This yields a consistency score per decision step that is written into a scorecard to pinpoint exactly which decisions are at risk of flipping on the next run. Detection is fully black-box — no logits, no model internals, no instrumentation beyond the trace you already have. 2. Generate — targeted guidelines. Every flagged step becomes a candidate consistency guideline in the standard ALTK-Evolve format, so it slots into the existing storage and retrieval pipeline. Here's a real example, generated by GPT-4.1 from a trajectory of the AppWorld task "How many activities are done in my bucket list as per my SimpleNote note?": Guideline 1 When counting checkbox-style markers in note content, use a line-anchored regex match rather than a plain substring count — note titles often repeat the marker symbol in a legend line. Guideline 2 Always verify search results for note queries by checking for multiple matches and confirming the correct note before proceeding. Nothing here is task-specific trivia. String-counting bugs and unverified search results are decision points that show up with high uncertainty across many AppWorld tasks. That's the point: the analyzer targets instability, not failure — so it catches steps the agent happened to get right this time but could easily get wrong next time. Watch the 2-minute demo https://www.youtube.com/watch?v=qlp7EzXe8Pg — five parallel runs of the agent split 3-2 on this task because of agent uncertainty about the counting strategy, then run again after these guidelines in context: all five agree. Results: Reducing the Gap Without Losing Accuracy We evaluated on AppWorld test normal 168 tasks with a ReAct agent on GPT-4.1, generating consistency guidelines from a single baseline trajectory per task and testing them on 5 fresh runs.