Krasyn ships a tool called Note Check. Paste a visit transcript and the note any AI scribe drafted from it, and it labels each sentence Supported, Unsupported, Contradicted, Scaffolding or Unverified against the transcript, raises three pure-code flags (a number the transcript never contained, a denial about a topic raised and never denied, content filled over an inaudible marker) and lists the facts the note left out. It never edits the note. The clinician reads the whole note and signs it.
We publish no accuracy figure for it, because it has not been measured against a clinician-adjudicated reference set. What we can do is run it on open data that someone else has already labelled, publish the counts and the disagreements, and say which side looks right on reading. This is the first installment.
Omi Health publishes medical-note-eval under the MIT license: 300 synthetic primary-care dialogues, SOAP notes written for each by named frontier models, and labels from a panel of LLM judges. The 2025 benchmark ships per-note counts of unsupported claims from three cross-family judges; the 2026 benchmark publishes per-writer totals only.
We picked six dialogues with different presenting problems and took the notes from three 2025 writers (GPT-5, Claude Opus 4.5, Gemini 3 Pro Preview) and three 2026 writers (GPT-5.5, Claude Sonnet 4.6, Gemini 3.1 Pro Preview): 36 pairs. Sentence IDs were removed from transcripts and citation tags from notes; nothing else changed. Every pair went through the production Note Check service (engine notecheck-1, judge gpt-4o at temperature 0). Eight were also run in the live product with screenshots and a screen recording.
| Set | Pairs | Statements | Supported | Unsupported | Contradicted | Scaffolding | Rule flags | Omissions |
|---|---|---|---|---|---|---|---|---|
| 2025 notes (per-note labels exist) | 18 | 627 | 354 | 3 | 0 | 270 | 6 | 55 |
| 2026 notes (no per-note labels) | 18 | 774 | 422 | 0 | 0 | 352 | 10 | 30 |
For the 18 labelled notes, Omi's panel had counted a major unsupported claim (two or more judges agreeing) in 10. Note Check's judge labelled a statement Unsupported in one of those 10 and in none of the other nine. On the eight notes the panel had not flagged, Note Check also found nothing. The two tools agree on clean notes and disagree on almost every note the panel thought had a problem.
| Dialogue | Model | Omi majors per judge | Note Check Unsupported + Contradicted | Note Check flags | Reading |
|---|---|---|---|---|---|
| dialogue_0 | GPT-5 | 3, 3, 4 | 0 | 1 | Plan adds counselling nobody said. Flag is a rule misfire. |
| dialogue_0 | Claude Opus 4.5 | 2, 2, 2 | 0 | 0 | "Order laboratory studies" firms up "may need some labs". |
| dialogue_2 | GPT-5 | 1, 1, 1 | 0 | 2 | Assessment in a visit with none. One flag right, one wrong. |
| dialogue_2 | Claude Opus 4.5 | 1, 1, 1 | 0 | 0 | Same invented assessment. |
| dialogue_24 | GPT-5 | 1, 3, 1 | 3 | 0 | Hit three "None" placeholders, not the type label. |
| dialogue_24 | Claude Opus 4.5 | 1, 1, 0 | 0 | 0 | "Type 2" is never stated. |
| dialogue_48 | Gemini 3 Pro Preview | 1, 1, 0 | 0 | 3 | All three flags wrong; the patient did deny. |
| dialogue_57 | GPT-5 | 2, 3, 4 | 0 | 0 | The word diabetes never appears. |
| dialogue_57 | Claude Opus 4.5 | 3, 3, 1 | 0 | 0 | "Continue metformin" when the plan was to adjust. |
| dialogue_70 | GPT-5 | 1, 1, 0 | 0 | 0 | A differential and a symptom log nobody proposed. |
The eight rows where both sides found nothing, and the 18 notes from the 2026 writers, are in the full report.
In dialogue_48 the doctor asks the patient to confirm there is no history of blood clots, migraines with aura or uncontrolled blood pressure. The patient says, "Correct, none of those." Four models (Gemini 3 Pro Preview, GPT-5.5, Claude Sonnet 4.6, Gemini 3.1 Pro Preview) each wrote some form of "Denies history of blood clots, migraines with aura, or uncontrolled blood pressure." That is the right sentence.
Our pertinent-negative rule flagged all three items in all four notes as High: the topic was raised and, it claimed, nobody denied it. The rule found the question and did not recognise the answer as a denial. It is pure code, and it behaved identically every time. Twelve of the sixteen flags in this run are that one mistake. A thirteenth read "without sharp pain and intermittent foot swelling" as a denial of the swelling. Two flags were right: "weight change" recorded as negative in two notes when weight was asked about and never answered.
The larger miss is quieter. In dialogue_2 the visit is history only; it ends with the doctor saying the timeline is now clearer. Both the GPT-5 and Claude Opus 4.5 notes carry an assessment: cough "likely related to allergic etiology". All three of Omi's judges counted that as a major unsupported claim. Note Check's judge called it Supported. The same pattern covers smoking cessation counselling in dialogue_0, "Type 2 diabetes mellitus" where the doctor said "your diabetes" or nothing at all, and a differential of migraine versus transient ischemic attack in dialogue_70 that nobody voiced. Our judge prompt allows faithful clinical translation to count as Supported. On this corpus it let clinical conclusions through under that heading. Each is sensible, which is exactly why a signer would not catch them either.
The fingerprint for a pair was the same in the command-line run and in the live product. Fresh judge calls were not identical: verdicts matched on seven of eight pairs run both ways, and the omission list changed on four of eight. This is why the product stores a report under its fingerprint instead of recomputing it, and why the omission list is a prompt to look rather than a count.
Synthetic dialogues, short and clean. LLM judges on both sides, no clinician adjudication on either; the reading column is ours. Omi's labels are counts without statement text, so overlap is inferred. The notes come from general-purpose models under a benchmark prompt, not from any commercial scribe product. Six dialogues is a small set; we report counts and no rate.
The rule defect and the judge leniency are tracked. The next installment reruns this exact set on the next engine version. If you have a de-identified synthetic transcript and note pair, send it and we will run it and publish the result.
Dialogues and notes: Omi Health medical-note-eval, MIT license.
Note Check is free inside a Krasyn Scribe account: https://krasyn.com/note-check?utm_source=devto&utm_medium=longform&utm_campaign=scribe-redteam-2026. The faithfulness benchmark for Krasyn's own scribe is at https://krasyn.com/resources/ai-scribe-faithfulness-benchmark.