We build our voice input method like an experiment Arion, a local-first voice and Pinyin input method, reported that wiring screen context into its polishing model added 5 to 6 passes on a frozen 30-case held-out set with the Qwen3 4B model but scored 9 of 30 in every context mode on its default Qwen3 1.7B model, which ignored the extra input entirely. In a second study of 154 test cases (924 measured generations on Qwen3 1.7B), an offline acoustic second pass with no LLM cut Mandarin character error rate from 8.17% to 5.82%, English word error rate from 7.63% to 3.97% and mixed error rate from 36.39% to 22.10%, while text-only LLM correction scored 8.42% CER and adding recognizer token scores degraded CER to 85.77%. The results led Arion to cap how much validation credit on-screen spellings can earn, switch from raw context sections to extracted terms to cut 4B median latency from 1.84 s to 0.86 s, and keep context polishing off by default on the 1.7B model. Blog https://arion.astrafoundry.si/blog/ Engineering Why we build our voice input method like an experiment When models and coding agents make prototypes cheap, the scarce thing is knowing whether a change helped. Starting from three experiments that contradicted our intuition, here is how Arion uses held-out sets, ablations and written rules to decide what ships. Suppose you are dictating a message about your colleague 张伟 Zhāng Wěi . While you speak, a chat window happens to be open on your screen from someone else: 张巍 Zhāng Wēi . Same Pinyin, different person. To help our local voice input method spell names and technical terms accurately, we wired screen context into the polishing model. It ran OCR over the active window and passed the visible text to a local LLM. It took an afternoon to build, and on the handful of sentences we tried by hand, it corrected every name we tested. If we had shipped based on that demo, the feature would have failed in two opposite ways at once. On Qwen3 4B, the model saw 张巍's message on screen and replaced the user's dictation about 张伟 with it. Worse, our output validator accepted the replacement. And on Qwen3 1.7B, the model we ship as the default, turning on screen context, cursor text and dictation history scored 9 out of 30 on our held-out test set. Turning all context off also scored 9 out of 30. 1 user-content-fn-context The same feature altered the speaker's meaning on the larger model, and did nothing at all on the default one. Three experiments where data contradicted intuition Arion is a local-first voice and Pinyin input method. You hold a key, speak Chinese, English or a mix of both, and speech recognition plus an optional local polishing model tidy the transcript turning "Wednesday, no wait, Thursday" into "Thursday" before inserting it at your cursor. Instead of judging context polishing by a few live tries, we froze a 30-case held-out set that included 7 negative cases: distracting windows, prompt injections on screen, near-homophone names and unrelated dictation history. Then we ran an ablation, testing each context source alone before combining them. 1 user-content-fn-context On this set, text around the cursor added 0 passes, and recent dictation history added 1 to 2. Screen text carried almost all the gain: paired with macOS Vision's accurate OCR 107 ms p50 , it added 5 to 6 passes on the 4B model, mostly names and terms visible in the window. Switching to Vision's fast mode 30 ms p50 lost 2 of those cases because it cannot read Chinese. Yet model size decided whether context helped at all. Our default 1.7B model scored 9 of 30 in every context mode, ignoring the extra input. Only the 4B model climbed from 9 to 16, at roughly twice the latency and memory. That 张伟 case led us to cap how much validation credit context spellings can earn, so on-screen text can no longer replace the main body of a dictation. Switching from raw context sections to extracted terms also reduced 4B median latency from 1.84 s to 0.86 s. And because the ablation showed 1.7B ignoring context, we avoided enabling a default setting that spends CPU without improving accuracy. A second study asked whether a text model can fix recognition errors when given pronunciation evidence. We tested Pinyin, universal IPA phones decoded from the audio, and the recognizer's own token confidence scores across 154 test cases 924 measured generations on Qwen3 1.7B . Next to those six LLM conditions, we included a control row that used no LLM at all: an offline acoustic second pass over the recorded audio. 2 user-content-fn-phonetic | Condition | Mandarin CER | English WER | Mixed MER | |---|---|---|---| | Raw streaming recognition | 8.17% | 7.63% | 36.39% | | Offline acoustic second pass control | 5.82% | 3.97% | 22.10% | | Text-only LLM correction Qwen3 1.7B | 8.42% | 7.53% | 36.39% | | Text + recognizer token scores | 85.77% | 39.27% | 231.27% | | Audio IPA phones only | 314.73% | 334.69% | 275.20% | Error rates above 100% in the bottom rows are not a bug in the scoring script. Our evaluation rules report insertions as measured rather than clamping error rates at 100% or replacing a failed model output with its source transcript. When we passed raw log-softmax token scores to an unadapted 1.7B model, it began repeating words, echoing prompt field names, and copying few-shot demonstrations until it hit the 256-token output limit. It corrupted 26 of the 60 utterances that streaming recognition had transcribed without error. Even the plain text-only editor changed almost nothing, and on all 4 pure-silence test cases it emitted a pair of quotation marks instead of an empty string. The control won across every metric. Re-decoding the audio with an offline acoustic model took 0.13 s at p50, lowered all three error rates, improved 60 utterances, and corrupted none of the originally correct ones. Without that control row, we could easily have spent weeks tuning prompts on a text model for a problem the acoustic pass already solved in a tenth of the time. A third study reproduced an external design. On the Pinyin keyboard side, a published LLM reranker for the Rime input method reported roughly 1 ms latency for reranking candidates with a local model. We reproduced the unchanged upstream Lua filter against 100 real native Rime candidate snapshots using Qwen3.5-0.8B on local oMLX. 3 user-content-fn-rime Top-1 accuracy fell from 56 of 100 to 45 of 100, with 3 rescues and 14 breakages. Because real Rime candidate lists mix full-word conversions with shorter prefix candidates that consume only part of the Pinyin, the prompt promoted single-character prefixes ahead of full words: 朋友 friend became 彭 , 学习 study became 学 , and 电脑 computer became 点 . Where did the 1 ms latency figure come from? In our run, HTTP calls to the local model server took 288.85 ms at the median. Reading the upstream Lua code explained the difference: its timer called os.clock , which measures process CPU time . While LuaSocket blocked waiting for the model server to respond, the Rime process was asleep, so its CPU clock barely moved 0.787 ms p50 while the user waited nearly a third of a second. Inspecting the raw responses revealed a second defect. The Lua parser placed multibyte UTF-8 Chinese punctuation inside byte-level character classes ^... . Whenever an ordinary Chinese character contained a matching byte, the parser split the word in half. Given the context 婚礼上许下 "vowed at the wedding" and Pinyin shi yan , the model returned the right candidate, 誓言 vow . The parser truncated the bytes and left 实验 experiment on screen. Out of 89 real model responses, 38 failed to parse. 3 user-content-fn-rime When code and demos are cheap Before we measured them systematically, all three ideas looked sound: testing a few windows by hand on the 4B model made screen context look ready to ship; inspecting individual text-correction examples without an acoustic control made prompt tuning look promising; and reading the os.clock log showed sub-millisecond numbers in the terminal. Working with AI coding agents makes this trap easier to fall into. Much of Arion's repository is written by coding agents and reviewed by engineers in pull requests. Writing a new prompt, swapping an ONNX model or adding a context source takes minutes, and every change comes with a convincing demo. Coding agents also optimize for passing whatever check is in front of them. If you ask an agent to run an evaluation or fix a failing pipeline, the shortest path to a green check is often to fall back to the source text when the model output fails validation, retry a dropped audio buffer, or mock a missing native driver. Code and demos got cheap, but finding out whether a change helps the next thousand dictations, or quietly hurts some of them, did not. To avoid fooling ourselves or being fooled by an agent's fallback logic, we treat product questions like lab experiments. How we run the experiments Write the benchmark before tuning the model Consider how we selected the live polishing model. Before tuning the prompt, we split the cases into two sets: a 24-case development set for iteration, and a 30-case held-out set written and SHA-256 hashed beforehand, used only to compare finished candidates.