The difference between "mentioned" and "answered"

A developer built 'intake', an AI scribe for community nurses, that adjudicates whether required form items were actually answered rather than merely mentioned. The system, built with Google ADK on Cloud Run and Gemini 3.6 Flash, uses an eval harness with 47 labelled cases and an asymmetric bar that defaults to 'insufficient' to avoid false positives. The project was created for the All Things Agentic Hackathon in August 2026.

I built this for the All Things Agentic Hackathon August 2026 . Code: github.com/jwlai-cloud/intake. Set published: true in the front matter when you post it. A community nurse has ninety minutes and a form she is legally required to complete. She asks about falls in the last twelve months. The answer is: "Oh, I've had a couple of wobbles." Every AI scribe on the market ticks that item. It was mentioned. It was never answered. She finds the gap that evening, at her desk — and now it needs a phone call, a guess, or a second visit. The person who knew the answer was sitting three feet away an hour ago. That gap is the entire product. I looked at what's shipping. Microsoft Teams' Facilitator marks a topic covered "once the discussion for that topic has started". Balto ticks when an item is mentioned. Otter's Live Assist checks off objectives. None of them adjudicate whether a required item actually received a real answer , and none of them gate the output on it. That's not laziness. Mention-detection is a much easier problem. Deciding whether "a couple of wobbles" satisfies a form's requirement for "the number of falls and the circumstances of the most recent" is a judgement call, and judgement calls are where LLM products quietly go wrong. So the whole project reduces to one question: can you make that judgement reliably enough to gate a report on it? The first thing I built wasn't the UI. It was an eval harness. eval/ holds 47 labelled cases — for each required item, answers that should count and answers that shouldn't. It scores the adjudicator against the live service and exits non-zero if any answer labelled insufficient was marked sufficient . That failure is the one that destroys the product, because it's The bar is asymmetric on purpose. A false insufficient costs one extra question. A false sufficient is a silently blank field in a legal document. So the prompt's first rule is: default to insufficient — if you're weighing it up, it's insufficient. Twelve of those cases are adversarial, and writing them was the single highest-value hour of the build. They found a real bug immediately: "Three falls, and the last was in May on the stairs." "No, hang on — I'm thinking of my sister. I've not actually fallen myself, not that I can bring to mind." The adjudicator read the retraction as a clean nil return and ticked the item . The record now held a contradiction settled only by a hedge, which is a I'd never have found that by hand-testing. A set that scores 100% on its first run can't tell you anything. Google ADK 2.6.2 on Cloud Run , calling Gemini 3.6 Flash on Vertex AI , with session state in Firestore . One audio chunk in, one bounded turn out: transcribe → route → adjudicate fanned out → coach The slot state is the state, not the transcript. The naive design accumulates the conversation and re-asks the model "what's still missing?" every turn. A forty-five minute interview becomes tens of thousands of growing tokens — expensive, and degrading as it grows. Instead each call gets the open items, a fixed-size struct of current values, and the new audio. A three-hour interview costs the same per chunk as a ten-minute one. Adjudication is one isolated call per open item, not one big prompt. Three reasons, in order: a wrong verdict on one item can't corrupt another; each item is separately scoreable by the eval; and k concurrent calls cost about one call's wall time. That last one surprises people — fanning out is usually the expensive choice, and here it's close to free. Between them sits a router. Without it, every open item was adjudicated against every chunk, and each independently decided a vague remark was relevant to it — so "a couple of wobbles" attached itself to mobility, memory and low mood as well as falls. One cheap classification call first cut items touched from seven to two, and cut cost with it. The agent never authors domain content. It tracks coverage against a human-authored form and quotes the span it relied on. It says "item M14 has no recorded answer" , never Behavioural evaluation found the hole in that anyway. The schema forbids an answer field ; it doesn't forbid an interpretive label . The agent had emitted a highlight titled "Formal decline to answer alcohol question" — a characterisation, not a quote. Titles are bare noun phrases now, and that exact string is a test case. No interviewee identity, ever. Sessions are scoped to a job, not a person. Persistent memory is scoped to the practitioner . The honest version of that claim is narrower than the slogan. Recorded answers are verbatim quotes, and a real interviewee says "my daughter Sarah drives me on Mondays" . Redacting that would break adjudication, so identity is a That last part was a real bug. A Vertex error response echoes the offending request, and the adjudicator's request body is the transcript — so a malformed chunk was writing interviewee speech into Cloud Logging, which outlives the session document. Logs now carry the exception type only. It gets better at helping one practitioner across her interviews. It learns nothing about the people she interviews. Two things, both about the professional: a question phrasing that closed an item on the first ask, and the item ids whose highlights she keeps dismissing. From her second interview the coach offers back a wording that worked, and stops proposing chips she's binned — while still asking the required question, because muting a suggestion must never mute an obligation. The version that would demo better is the one it refuses to build. "People like this one usually under-report falls" would be useful and would permanently Before a report is produced, every required item resolves into exactly one of three states: Answered | with the transcript span it was drawn from | Declined | formally recorded, and only where the form permits it | Escalated | the agent drafts the follow-up itself and routes it | The gate is a router, not a wall. A copilot that only says no is one practitioners switch off. When it refuses, the response carries what's outstanding, what's missing from each, and whether a decline is even permitted. The escalation is the part I'd point at. Given an unresolved item, the agent writes what's still not recorded in the form's own terms, why it couldn't be closed, and picks a destination from a closed list. "Home access and hazards · not recorded during the visit → Occupational therapy queue." Unprompted GOOGLE GENAI USE VERTEXAI=TRUE is mandatory for ADK. genai.Client vertexai=True in the same process works. SequentialAgent is deprecated in 2.6.2 content . agents-cli eval generate rejects any content-less event, so ADK's deprecated orchestrator is incompatible with ADK's current eval tooling. Then I measured the replacement: the graph Workflow emits Event output=…, content=None and BaseAgent is the only one of the three that can be evaluated. BaseNode subclasses and compose fine.The general lesson, for an SDK this young: read signatures off the installed package with inspect , not off documentation. Every API claim in this project was verified that way, and the one time I trusted a note instead, it was wrong. Two days before the deadline I asked someone to try the deployed app with a real microphone. Nothing visibly happened. Chunks were arriving, HTTP 200, the ADK pipeline was running — and every turn finished in about a tenth of a second having done nothing. A real turn takes six seconds. The transcriber labels each turn practitioner or interviewee , and adjudication only looks at interviewee turns. One person testing alone is a single voice, and the model reasonably labelled it practitioner. Every chunk I'd tested the API with curl, the text path, and an automated browser capture. But that capture runs Chromium with --use-fake-device-for-media-stream . I had verified a proxy for the product and called it the product. The fix is one instruction, and where it went matters. My first attempt relaxed the filter in the adjudicator, and it immediately broke a test asserting that a nurse restating an answer must never close an item. Same input, two opposite correct answers — the adjudicator can't tell a lone tester from a professional summarising. The transcriber can: it's the only stage that hears how many people are in the room. So it labels a lone voice as the interviewee, and the downstream guarantee is untouched. 47 labelled cases, 100% precision on sufficient — it has never once ticked in .A second template — insurance loss adjusting — runs on the same engine with no code change, which is the test of whether the vertical is really just config. What I'd build next is per-user identity. Access control today is a capability model: one shared key, and session ids that are 128 bits of randomness. It holds, but it stops holding the moment an id reaches a log or a shared screen.