{"slug": "the-difference-between-mentioned-and-answered", "title": "The difference between \"mentioned\" and \"answered\"", "summary": "A developer built 'intake', an AI scribe for community nurses, that adjudicates whether required form items were actually answered rather than merely mentioned. The system, built with Google ADK on Cloud Run and Gemini 3.6 Flash, uses an eval harness with 47 labelled cases and an asymmetric bar that defaults to 'insufficient' to avoid false positives. The project was created for the All Things Agentic Hackathon in August 2026.", "body_md": "*I built this for the All Things Agentic Hackathon (August 2026). Code:\ngithub.com/jwlai-cloud/intake.\nSet published: true in the front matter when you post it.*\n\nA community nurse has ninety minutes and a form she is legally required to\n\ncomplete. She asks about falls in the last twelve months. The answer is:\n\n\"Oh, I've had a couple of wobbles.\"\n\nEvery AI scribe on the market ticks that item. It was mentioned. It was never\n\nanswered. She finds the gap that evening, at her desk — and now it needs a phone\n\ncall, a guess, or a second visit. The person who knew the answer was sitting\n\nthree feet away an hour ago.\n\nThat gap is the entire product.\n\nI looked at what's shipping. Microsoft Teams' Facilitator marks a topic covered\n\n\"once the discussion for that topic has started\". Balto ticks when an item is\n\nmentioned. Otter's Live Assist checks off objectives. None of them adjudicate\n\nwhether a required item actually *received a real answer*, and none of them gate\n\nthe output on it.\n\nThat's not laziness. Mention-detection is a much easier problem. Deciding\n\nwhether *\"a couple of wobbles\"* satisfies a form's requirement for \"the number\n\nof falls and the circumstances of the most recent\" is a judgement call, and\n\njudgement calls are where LLM products quietly go wrong.\n\nSo the whole project reduces to one question: **can you make that judgement\nreliably enough to gate a report on it?**\n\nThe first thing I built wasn't the UI. It was an eval harness.\n\n`eval/`\n\nholds 47 labelled cases — for each required item, answers that should\n\ncount and answers that shouldn't. It scores the adjudicator against the live\n\nservice and **exits non-zero if any answer labelled insufficient was marked\nsufficient**. That failure is the one that destroys the product, because it's\n\nThe bar is asymmetric on purpose. A false *insufficient* costs one extra\n\nquestion. A false *sufficient* is a silently blank field in a legal document. So\n\nthe prompt's first rule is: **default to insufficient — if you're weighing it up,\nit's insufficient.**\n\nTwelve of those cases are adversarial, and writing them was the single\n\nhighest-value hour of the build. They found a real bug immediately:\n\n\"Three falls, and the last was in May on the stairs.\"\n\n\"No, hang on — I'm thinking of my sister. I've not actually fallen myself,\n\nnot that I can bring to mind.\"\n\nThe adjudicator read the retraction as a clean nil return and **ticked the\nitem**. The record now held a contradiction settled only by a hedge, which is a\n\nI'd never have found that by hand-testing. A set that scores 100% on its first\n\nrun can't tell you anything.\n\n**Google ADK 2.6.2 on Cloud Run**, calling **Gemini 3.6 Flash on Vertex AI**,\n\nwith session state in **Firestore**. One audio chunk in, one bounded turn out:\n\n```\ntranscribe → route → adjudicate (fanned out) → coach\n```\n\n**The slot state is the state, not the transcript.** The naive design\n\naccumulates the conversation and re-asks the model \"what's still missing?\" every\n\nturn. A forty-five minute interview becomes tens of thousands of growing tokens\n\n— expensive, and degrading as it grows. Instead each call gets the open items, a\n\nfixed-size struct of current values, and the new audio. A three-hour interview\n\ncosts the same per chunk as a ten-minute one.\n\n**Adjudication is one isolated call per open item, not one big prompt.** Three\n\nreasons, in order: a wrong verdict on one item can't corrupt another; each item\n\nis separately scoreable by the eval; and *k* concurrent calls cost about one\n\ncall's wall time. That last one surprises people — fanning out is usually the\n\nexpensive choice, and here it's close to free.\n\nBetween them sits a router. Without it, every open item was adjudicated against\n\nevery chunk, and each independently decided a vague remark was relevant to it —\n\nso *\"a couple of wobbles\"* attached itself to mobility, memory and low mood as\n\nwell as falls. One cheap classification call first cut items touched from seven\n\nto two, and cut cost with it.\n\n**The agent never authors domain content.** It tracks coverage against a\n\nhuman-authored form and quotes the span it relied on. It says *\"item M14 has no\nrecorded answer\"*, never\n\nBehavioural evaluation found the hole in that anyway. The schema forbids an\n\nanswer *field*; it doesn't forbid an interpretive *label*. The agent had emitted\n\na highlight titled *\"Formal decline to answer alcohol question\"* — a\n\ncharacterisation, not a quote. Titles are bare noun phrases now, and that exact\n\nstring is a test case.\n\n**No interviewee identity, ever.** Sessions are scoped to a job, not a person.\n\nPersistent memory is scoped to the *practitioner*.\n\nThe honest version of that claim is narrower than the slogan. Recorded answers\n\nare verbatim quotes, and a real interviewee says *\"my daughter Sarah drives me on\nMondays\"*. Redacting that would break adjudication, so identity is a\n\nThat last part was a real bug. A Vertex error response echoes the offending\n\nrequest, and the adjudicator's request body *is* the transcript — so a malformed\n\nchunk was writing interviewee speech into Cloud Logging, which outlives the\n\nsession document. Logs now carry the exception *type* only.\n\nIt gets better at helping one practitioner across her interviews. It learns\n\nnothing about the people she interviews.\n\nTwo things, both about the professional: a question phrasing that closed an item\n\non the first ask, and the item ids whose highlights she keeps dismissing. From\n\nher second interview the coach offers back a wording that worked, and stops\n\nproposing chips she's binned — while still asking the required question, because\n\nmuting a suggestion must never mute an obligation.\n\nThe version that would demo better is the one it refuses to build. *\"People like\nthis one usually under-report falls\"* would be useful and would permanently\n\nBefore a report is produced, every required item resolves into exactly one of\n\nthree states:\n\nAnswered |\nwith the transcript span it was drawn from |\nDeclined |\nformally recorded, and only where the form permits it |\nEscalated |\nthe agent drafts the follow-up itself and routes it |\n\nThe gate is a router, not a wall. A copilot that only says *no* is one\n\npractitioners switch off. When it refuses, the response carries what's\n\noutstanding, what's missing from each, and whether a decline is even permitted.\n\nThe escalation is the part I'd point at. Given an unresolved item, the agent\n\nwrites what's still not recorded in the form's own terms, why it couldn't be\n\nclosed, and picks a destination from a closed list. *\"Home access and hazards ·\nnot recorded during the visit → Occupational therapy queue.\"* Unprompted\n\n`GOOGLE_GENAI_USE_VERTEXAI=TRUE`\n\nis mandatory for ADK.`genai.Client(vertexai=True)`\n\nin the same process works.`SequentialAgent`\n\nis deprecated in 2.6.2`content`\n\n. `agents-cli eval generate`\n\nrejects any content-less event,\nso ADK's deprecated orchestrator is incompatible with ADK's current eval\ntooling. Then I measured the replacement: the graph `Workflow`\n\nemits\n`Event(output=…, content=None)`\n\nand `BaseAgent`\n\nis the only one of the three that can be evaluated.`BaseNode`\n\nsubclasses and compose fine.The general lesson, for an SDK this young: read signatures off the installed\n\npackage with `inspect`\n\n, not off documentation. Every API claim in this project\n\nwas verified that way, and the one time I trusted a note instead, it was wrong.\n\nTwo days before the deadline I asked someone to try the deployed app with a real\n\nmicrophone. Nothing visibly happened.\n\nChunks were arriving, HTTP 200, the ADK pipeline was running — and every turn\n\nfinished in about a tenth of a second having done nothing. A real turn takes six\n\nseconds.\n\nThe transcriber labels each turn `practitioner`\n\nor `interviewee`\n\n, and\n\nadjudication only looks at `interviewee`\n\nturns. **One person testing alone is a\nsingle voice, and the model reasonably labelled it practitioner.** Every chunk\n\nI'd tested the API with curl, the text path, and an automated browser capture.\n\nBut that capture runs Chromium with `--use-fake-device-for-media-stream`\n\n. I had\n\nverified a proxy for the product and called it the product.\n\nThe fix is one instruction, and *where* it went matters. My first attempt\n\nrelaxed the filter in the adjudicator, and it immediately broke a test asserting\n\nthat a nurse restating an answer must never close an item. Same input, two\n\nopposite correct answers — the adjudicator can't tell a lone tester from a\n\nprofessional summarising. The transcriber can: it's the only stage that hears\n\nhow many people are in the room. So it labels a lone voice as the interviewee,\n\nand the downstream guarantee is untouched.\n\n47 labelled cases, **100% precision on sufficient** — it has never once ticked\n\n`in`\n\n.A second template — insurance loss adjusting — runs on the same engine with no\n\ncode change, which is the test of whether the vertical is really just config.\n\nWhat I'd build next is per-user identity. Access control today is a capability\n\nmodel: one shared key, and session ids that are 128 bits of randomness. It\n\nholds, but it stops holding the moment an id reaches a log or a shared screen.", "url": "https://wpnews.pro/news/the-difference-between-mentioned-and-answered", "canonical_source": "https://dev.to/junwei_lai_71641e0a742b33/the-difference-between-mentioned-and-answered-3ekp", "published_at": "2026-08-31 07:58:08+00:00", "updated_at": "2026-08-31 08:22:26.803469+00:00", "lang": "en", "topics": ["artificial-intelligence", "ai-products", "ai-tools", "developer-tools", "ai-agents"], "entities": ["Google ADK", "Gemini 3.6 Flash", "Vertex AI", "Cloud Run", "Firestore", "Microsoft Teams Facilitator", "Balto", "Otter Live Assist"], "alternates": {"html": "https://wpnews.pro/news/the-difference-between-mentioned-and-answered", "markdown": "https://wpnews.pro/news/the-difference-between-mentioned-and-answered.md", "text": "https://wpnews.pro/news/the-difference-between-mentioned-and-answered.txt", "jsonld": "https://wpnews.pro/news/the-difference-between-mentioned-and-answered.jsonld"}}