# Benchmark Says Top Multimodal Models Score Under 10% at Reading Words From Pen Sounds and Hand Motion > Source: > Published: 2026-08-18 06:29:43+00:00 ## What happened Researchers posted a paper describing The Unwritten Benchmark, which asks multimodal models to identify words being written using only the sound of a pen and video of a moving hand, with no visible ink. The abstract reports human ordered letter accuracy above 80% and leading models, including GPT-4o and Gemini 2.5-Pro, below 10%, and says combining the two modalities frequently degraded model performance rather than improving it. Three researchers — Garima Arya Yadav, Nilay Yilmaz and Yezhou Yang — have posted a paper introducing what they call The Unwritten Benchmark, a test of whether multimodal machine learning models can work out what a person is writing without ever seeing the writing itself. The paper is listed on arXiv as 2608.14558, filed under artificial intelligence with a cross-listing to computer vision and pattern recognition. The authors define the core task as "acousto-kinematic word inference": a model is given only the audio of pen scratches and video of hand movements, with no visible ink trace, and must decipher the word being written. The abstract says the task spans three different writing styles. The arXiv listing gives a submission date of 15 May 2026 and carries a comment stating the work is to be published in CVPR Findings 2026. The headline result is a gap. According to the abstract, human participants achieve high ordered letter accuracy — the authors say over 80% — while leading multimodal models fail to surpass 10%. The two models named in the abstract are GPT-4o and Gemini 2.5-Pro. The paper frames this not as a narrow recognition failure but as evidence of limits in cross-modal causal reasoning and in what the authors call the micro-kinematics of writing: the fine, fast hand and pen movements that carry information about which letter is being formed. The second reported finding is less expected. The authors describe a "paradoxical fusion effect," in which supplying both audio and video to a model often degrades its performance compared with providing a single modality. They read this as a breakdown in the models' ability to synthesise complementary perceptual cues. The abstract does not report per-modality scores, so the size of that degradation, and whether it appeared for every model tested, are not established by the material reviewed here. This account is drawn from the arXiv abstract page only; the full paper, its appendices and any released materials were not examined. Several details that would matter for judging the result are not stated there. The abstract does not give the number of words or clips in the dataset, the number of human participants or the conditions under which they were tested, the language or script involved, the exact definition of ordered letter accuracy, how the models were prompted, how video and audio were sampled and passed in, or whether the data and code will be published. The paper is also listed as forthcoming in a conference findings track rather than as a completed peer-reviewed publication, and no independent replication is known at this point. [Read the primary source: arxiv.org ↗](https://arxiv.org/abs/2608.14558) ## Why it matters Most multimodal evaluations test recognition of what is directly visible or audible. This one tests inference of something never shown, and reports that adding a second modality can hurt — a result that runs against the common assumption that fusing audio and video is additive. If it holds up, it points at a weakness in how current systems combine perceptual channels, not just at a hard task. Most widely used multimodal benchmarks test whether a system can name what is present in an image, a video or an audio clip. This one deliberately removes the answer from the input. The word is never shown; it has to be reconstructed from the physical process that produced it. That design targets a capability — inferring an unseen cause from dynamic evidence — that is closer to how people read a scene than to pattern matching over static content, and it is a capability current evaluation suites largely do not isolate. The fusion result is arguably the more consequential of the two claims. A great deal of product engineering assumes that giving a model more perceptual channels can only help: meeting assistants combine speech with screen video, accessibility tools combine camera and microphone input, and robotics stacks fuse several sensors on the assumption that redundancy is protective. If a second modality can reliably make a model worse on a task where both channels carry real signal, that assumption needs testing rather than trusting. The paper does not establish that this happens generally — it reports it on one task, with a small set of named models — but it is a concrete case where more input produced a worse answer. The task also sits near two practical areas. One is accessibility and digitisation: capturing handwriting from sound and motion, without a smart pen or a scanner, would be useful for note capture and for people who write on ordinary paper. The other is privacy. Inferring written content from incidental acoustic and visual signals is, in principle, a side-channel problem, and side-channel attacks on keyboards from audio have been studied for years. On the evidence in this abstract, current general-purpose models are far from being able to do this — under 10% is close to unusable — while humans reportedly can. That is a finding about today's model capabilities, not a durable guarantee. The limitations run in both directions and are worth stating plainly. Very low scores on a newly introduced benchmark are common, and by themselves they do not distinguish between a genuine reasoning deficit, a mismatch between the task format and how these models ingest video and audio, and an evaluation harness that was not tuned for the task. The two models named are general-purpose commercial systems, not systems built or fine-tuned for fine-grained motion analysis, and stronger or more recent models may not have been tested. The human baseline is reported as a single number without described conditions. And a benchmark authored by the same team that reports the gap has not yet been stress-tested by anyone else. ## What to watch next Whether the dataset, code and human-baseline protocol are released and independently reproduced; whether newer models or different prompting close the gap; and whether the reported fusion degradation shows up on other audio-video tasks or turns out to be specific to this benchmark. The first thing to watch is release and reproducibility. Whether the authors publish the dataset, the evaluation code and the exact human-study protocol will determine how quickly the headline numbers can be checked. The arXiv entry reviewed here does not indicate a code or data link. A benchmark that reports a 70-point human-machine gap invites independent replication, and the value of the claim depends heavily on whether other groups can reproduce both the model scores and the human baseline. The second is whether the gap survives contact with better setups. Scores on new multimodal tasks often move substantially with changes that have nothing to do with reasoning — frame rate, audio sampling and preprocessing, how many frames a model actually sees, prompt structure, or a modest amount of task-specific fine-tuning. If a small engineering change lifts models well above 10%, the result reads mainly as an input-pipeline problem. If careful tuning leaves them near the floor, the authors' stronger claim about cross-modal causal reasoning becomes harder to dismiss. Third is the fusion effect. The interesting question is whether degradation from adding a modality appears on other audio-video tasks, or whether it is specific to this one. If other groups find the same pattern elsewhere, it would point to something structural in how current models weight and combine channels rather than a quirk of pen scratches and hand motion. Per-modality ablations in the full paper, and attempts to reproduce them, are the place that gets settled. Finally, watch for adoption. Benchmarks matter when other labs report scores on them and when the numbers move over successive model generations. Whether this one appears in the CVPR 2026 findings track as stated, whether frontier labs include it in evaluation reports, and whether privacy and security researchers pick up the side-channel framing will all be visible over the coming months. It is also worth noting that a benchmark this far from saturation gives a clear signal for a while — until it does not, at which point the question becomes whether models learned the capability or the dataset.