Auditing the Synthetic Memoir: Measuring Scene-Level Confabulation in LLM-Generated Autobiography Against the Documented Record of the Life It Describes

A new arXiv study (2608.23640v1) reports that 96.7% of days in an LLM-generated autobiography failed verification against the subject's documented record, with 354 of 366 days not rated VERIFIED (Wilson 95% CI 94.4-98.1%). The audit, conducted by the subject and author of the paper, found that only 12 days contained a corroborated scene, 19 days (5.2%) asserted claims actively contradicted by the record, and the dominant failure mode was grounded drift. Regenerating the same days with current named models reproduced 100% verification failure, while grounding generation in the subject's corpus improved the verification rate to 83.3% residual failure.

arXiv:2608.23640v1 Announce Type: new Abstract: When a large language model LLM is asked to write a person's life, how much of what it writes actually happened? We present a scene-level case-study audit - the first quantified audit of LLM-generated autobiography against a subject-specific ground-truth corpus that we are aware of, based on an unsystematic literature search. The subject and the author of this paper are the same person: a 366-day "page-a-day" book of first-person anecdotal entries was drafted with a conversational LLM whose documented inputs were a template, two exemplar days, and each day's quote - not her corpus - and every day was subsequently audited at the anecdote-scene level against an independent verification corpus using a four-level rubric fixed before analysis. We define the verification-failure rate as the share of days not rated VERIFIED scene positively corroborated : 354 of 366 days fail, 96.7% Wilson 95% CI 94.4-98.1% . Only 12 days contain a corroborated scene; 19 days 5.2% assert claims actively contradicted by the record; the dominant failure mode is grounded drift - real people, employers, and settings inside invented scenes - though its measured share varies across raters. Independent re-rating replicates the headline no evidence the original rate was inflated while showing that the four-way taxonomy has only fair-to-moderate reliability. Regenerating the same days with current named models reproduces 100% verification failure under the same inputs; grounding generation in the subject's corpus significantly improves the verification rate while leaving substantial residual failure 83.3% . We contribute the measurement, a reusable audit instrument whose WEAK/UNVERIFIED boundary we show to be unreliable, and a grounding remedy with quantified effect.