cd /news/ai-agents/grounded-vs-guessed-a-real-test-of-a… · home topics ai-agents article
[ARTICLE · art-123240] src=dev.to ↗ pub= topic=ai-agents verified=true sentiment=· neutral

Grounded vs. Guessed: A Real Test of AI Agent Context

A developer ran a controlled experiment comparing an AI coding agent's output with and without access to a persistent story map. The agent produced a generic user story from a ticket alone, but with tool access to the map, it resolved open questions about placement and scope, demonstrating the value of structured context for AI agents.

read7 min views14 publishedSep 8, 2026

I've started running a small, ongoing experiment on myself: how do I actually improve the quality of my own work, and more specifically, how do I hand an AI agent the kind of product context that makes it build the right thing instead of a confident guess dressed up as a spec. This piece is the first real result of that.

I want to show you something instead of just claiming it. So here's an actual experiment, run once, with the real output pasted in, not paraphrased or cleaned up after the fact.

The claim I keep making about story maps and AI coding agents is that a map gives an agent something a chat transcript can't: persistent, structured context. That's easy to say and hard to prove without just showing the receipts. So I ran the same feature request through the same agent twice, once blind, once with real tool access to a real map, and wrote down what actually came out both times.

One ticket:

"Add a gift wrapping option to checkout. Users should be able to opt to have their order gift wrapped before it ships."

Same underlying model both times. I want to be upfront about that, because it'd be more dramatic to say "two competing AI products" and it wouldn't be true. The variable isn't which vendor's model you use, it's whether that model can read a real, persistent source of context before it answers, or whether all it has is the words in the prompt. That's the actual thing worth testing, and it's the thing that changes regardless of which model you're running.

The map already existed before this experiment started: a checkout flow with four steps (Enter Shipping Address, Choose Payment Method, Review Order, Confirm Purchase), each with a handful of real cards under it, some already slotted into an MVP release, most sitting unscheduled. Nothing was staged for this test. It's the same structure from an earlier piece I wrote on story mapping basics.

Given only the ticket text, here's what came back:

As a customer, I want to add gift wrapping to my order at checkout so that I can send it directly as a gift.

Acceptance criteria: a gift wrap option appears somewhere in checkout, selecting it may add a fee (unclear), order confirmation should reflect the choice.

Open questions: where exactly does this go relative to shipping and payment? Is this a priority-one feature or something later? Per-item or per-order? Any specific persona this serves?

That's a perfectly serviceable user story. If you've used any coding agent to draft a spec from a one-line ticket, this will look familiar, it's fast, it's readable, and it's not wrong exactly. But look at the open questions. Every one of them is a real decision a developer will eventually have to make, and right now they're guesses dressed up as a to-do list.

-> [View the full size image](https://raw.githubusercontent.com/TamasTest/Article-Images/refs/heads/main/Add%20gift%20wrapping%20to%20checkout.png)

-> [Open the interactive story map for more details](https://tomiacademy.storiesonboard.com/storymap/reading-the-map-vs-guessing-gift-wrapping-experiment)

Same ticket, but this time the agent had tool access to the actual map: read the Checkout activity, read its four steps, read the real cards under each one, including which ones already sit in the MVP release. Three tool calls, all read-only, nothing modified. Here's what came back:

As a shopper, I want to add gift wrapping to my order from the review screen so that I can send it directly as a gift without doing the wrapping myself.

Almost the identical sentence to Condition A. That's worth sitting with for a second, because it would be more convincing for this article if the story itself came out dramatically better with context. It didn't. What changed was everything the story used to leave as an open question:

Placement stopped being a guess:

The Review Order step already held three cards: Order Summary, Cart Editing, Estimated Delivery Date, all order-level choices made right before confirming. Gift wrapping is obviously a fourth one. Condition A had "where does this go" as an open question. Condition B had it as a fact read off the map.

Priority stopped being a guess:

Every comparable optional extra already on this map (Apply Discount Code, Save Card for Later) sits in the unscheduled backlog, not the MVP release. That's a real pattern across two different steps, not a hunch. Gift wrapping followed it: reasonable to unschedule, not the walking-skeleton priority Condition A might have arbitrarily assigned.

No invented persona:

This map has zero personas configured. Condition A's blank "any specific persona?" question could easily have gotten filled in with something plausible-sounding but fabricated, "gift givers" or similar. Condition B just stated the actual fact: no persona split exists yet, so this serves the same general shopper as everything else on the map.

The user story barely moved between conditions, so I ran a second, narrower test on the same card: acceptance criteria. This is a better place to look for a real difference, because AC is where an agent has to commit to specifics, not just a placement decision.

Condition A, bare prompt, guessed AC:

  • Given a shopper is on checkout, when they select "Gift wrap this order," then the order is flagged as a gift.
  • Given the order is flagged as a gift, when checkout completes, then a gift wrap fee is added to the total (assumed).
  • Given the order is flagged as a gift, when it ships, then the packing slip excludes prices (assumed default).
  • Given the order contains only digital items, then the gift wrap option is hidden (assumed: this store sells both physical and digital products, and gift wrap doesn't apply to digital).

Four confident-sounding criteria. The last one is the interesting failure: it invents a fact about the product, that there's a physical/digital split, that was never established anywhere in the ticket. Nothing wrong with the sentence itself, it reads like something a real spec would say. It just isn't grounded in anything.

Condition B, grounded in the real map:

  • Given a shopper is on the Review Order screen, when they select "Gift wrap this order," then the order is flagged as a gift, alongside the other order-level choices already made on that screen.
  • Given the order is flagged as a gift, when Confirm Purchase completes and the existing Send Order Confirmation Email card fires, then the confirmation email reflects the gift wrap choice. Not a new integration point, it wires into a card that already exists later in this same activity.
  • Open, not guessed: whether gift wrapping applies to digital-only orders. Nothing on this map distinguishes physical from digital products, so this can't be answered from what's here.

Two grounded criteria, and one deliberately left open. That third line is the actual finding worth paying attention to. Condition A didn't leave anything open, it resolved every question, including one it had no basis to resolve. Condition B correctly identified which question the map genuinely can't answer and said so, instead of quietly making something up to look complete.

That's a better result than "the grounded version had more detail." A confidently wrong AC is worse than an honestly incomplete one, because the wrong one passes review and the incomplete one gets caught. Reading real context doesn't just fill in answers, it also tells the agent what it doesn't know yet.

The user story sentence took maybe two seconds to generate either way. That part was never the bottleneck. The bottleneck was always the surrounding decisions, where does this fit, how urgent is it, who is it for, and those are exactly the decisions a raw prompt has no way to answer honestly. It can guess convincingly, which is arguably worse than guessing badly, because a confident wrong guess is harder to catch in review than an obviously uncertain one.

A persistent, structured map doesn't make the agent smarter. It gives the agent something real to read instead of something to invent. As more of the first draft of a spec gets written by an agent rather than a person, that distinction stops being a nice-to-have and starts being the difference between a ticket that's technically answered and one that's actually grounded in the product.

You can see the real map from this experiment here: the Checkout activity, the cards that grounded Condition B's reasoning, and the Gift Wrapping card exactly where and how it landed. The map has since grown into a full journey example (Browse Products through Order Tracking), but the Checkout section and the reasoning behind it are unchanged: Reading the Map vs. Guessing.

── more in #ai-agents 4 stories · sorted by recency
sponsored brought to you by zahid.host 4,200+ EU-deployed projects
reading about agents? ship yours in a single git push.

Run your AI side-project on zahid.host

EU-based hosting, git-push deploys, automatic HTTPS, no cold starts. Free tier with a custom domain — perfect for shipping the agent you just read about.

$git push zahid main
Live at https://your-agent.zahid.host
Get free account → Pricing
from €0/mo · no card required
LIVE [news/grounded-vs-guessed-…] indexed:0 read:7min 2026-09-08 ·