A Transcript Is a Choice

Symphony, a steerable speech recognition model, nearly matches the purpose-built CrisperWhisper 2.0 on spoken disfluencies, scoring an F1 of 91.83% versus 92.46% on shared categories, according to a blog post by Ian Eaves. The post argues that transcription is a choice, as voice agents may need verbatim, clean, or structured transcripts, and that Symphony can adapt to each need.

Blog /blog - A Transcript Is a Choice A Transcript Is a Choice Ian Eaves Symphony nearly matches a purpose-built verbatim transcription model on spoken disfluencies, and can turn the same recording into the representation each part of a voice agent needs. A voice agent hears: "Um, can you move my Tuesday appointment to Thurs, Thursday at, uh, three thirty? Actually, make that four. Four works better." What should its transcript say? A verbatim transcript would preserve the fillers, the interrupted word, the abandoned time, and the eventual correction. A clean transcript would say something much simpler: the appointment should be moved to Thursday at 4:00. Agentic tasks might even need a structured interpretation explaining that 3:30 was proposed but ultimately replaced. All three outputs can be reasonably described as "accurate." They simply answer different questions. Disfluency is not the same as transcription error People rarely speak in finished sentences. Spontaneous speech contains fillers such as "um," repeated words, interrupted fragments, false starts, and corrections made halfway through an instruction. These features are collectively described as disfluencies , but the category includes more than meaningless noise. In the example above, the correction from 3:30 to 4:00 determines what the agent actually needs to do. A system can preserve what someone said verbatim, or it can produce the clean text they apparently intended. Most speech recognition systems do not expose that decision explicitly, which can lead to inconsistent behavior across similar utterances. Making matters worse, ordinary Word Error Rate WER does not capture this distinction cleanly. A filler counted as an error against a clean reference might be exactly what a verbatim system was asked to preserve. Purpose-built vs. general-purpose models Some companies, like Nyra Labs, are working to address this by purpose-building ASR models explicitly designed to handle disfluencies, and their published benchmarks show strong results. Steerable models like Symphony handle disfluencies among other issues as the caller needs, rather than being locked into a single purpose-built workflow. When we tested Symphony against Nyra's complete English benchmark using a simple verbatim-transcription instruction, Symphony performed almost identically to the frontier CrisperWhisper model. Sound-excluded disfluency metric, English | Metric | Symphony | CrisperWhisper 2.0 | Difference | |---|---|---|---| | Precision | 88.93% | 88.29% | +0.64 pp | | Recall | 94.93% | 97.03% | -2.10 pp | | F1 | 91.83% | 92.46% | approximately -0.6 pp | ::: callout{type="note" title="One qualification"} Symphony does not currently emit canonical vocal-sound labels such as laughter or breath under this configuration. We therefore recalculated the disfluency metric over the three categories both models produce: fillers, repetitions, and cut-offs. ::: On these shared categories, Symphony comes within roughly six-tenths of a percentage point of CrisperWhisper 2.0 on disfluency F1. The precision and recall split is informative. Symphony was slightly less likely to introduce a disfluency that was not present, while CrisperWhisper recovered slightly more of the disfluencies in the reference transcript. Overall, the models were near parity despite CrisperWhisper being purpose-built for controllable verbatim speech recognition. On the vagaries of transcription Recognizing disfluencies is useful, but a voice agent often needs more than a verbatim transcript. Take this example snippet from a hypothetical phone conversation Depending on application requirements an agent might need any of - A verbatim account of what was spoken. - A clean transcript of the speaker's final intended meaning. - A structured JSON interpretation of the corrections and uncertainty in the request. For applications which narrowly require a solution like 1 traditional ASR models can be great solutions. Likewise, applications which need a pure intent driven transcript like 2 can benefit from purpose built models like CrisperWhisper. The muddy middle is where things get difficult; what if you need structured outputs, or customization over how the transcript is processed? What if you need both a verbatim and cleaned transcript in a single request? For these sorts of tasks steerable multilingual models, like Symphony, can be a better match because the application can define it's own transcription protocol. In practice this actually ends up being much simpler than it sounds. If you need a single unadulturated transcript you can us something like this Instructions Transcribe this audio verbatim. Preserve filler words, repetitions, false starts, interrupted words, and self-corrections exactly as spoken. Return only the transcript. Transcript Um, can you move my Tuesday appointment to Thurs, Thursday at uh three thirty, Actually, make that four. Four works better. The output will respect the natural flow of the users real speech. Similarly, if you need disfluency handling you can use an instruction like this Instructions Produce a clean transcript of the speaker's final intended request. Remove incidental filler words, false starts, restarts, and abandoned values. Apply explicit self-corrections so that only the speaker's final resolved values remain. Preserve any uncertainty that the speaker does not resolve. Do not infer or add details that are not supported by the audio. Output only the clean intended transcript. Transcript Can you move my Tuesday appointment to Thursday at 4:00? It can even flexibly produce JSON output matching your specific transcription requirements. Instructions Interpret the attached voice request. Return one bare JSON object and nothing else. Use exactly these fields: - "final request": the speaker's final resolved request as a string - "corrections": an array of objects with string fields "abandoned" and "replacement" - "unresolved uncertainty": an array of strings - "material disfluencies": an array of short strings Identify explicit corrections and distinguish abandoned values from final values. Include only uncertainty that remains unresolved. In "material disfluencies", list each distinct audible filler, interrupted word or restart, repetition, and self-correction exactly once, in order of occurrence. Do not duplicate entries. Use no more than ten entries. Do not infer unsupported details. Do not include a transcript, explanation, or Markdown fence. Response { "final request": "Can you move my Tuesday appointment to Thursday at 4:00?", "corrections": { "abandoned": "3:30", "replacement": "4:00" } , "unresolved uncertainty": , "material disfluencies": "um", "uh", "thurs", "three thirty", "actually", "make that four", "four works better" } Under a strict output-format test, the cut-off notation was the one limitation. Symphony recognized that the speaker began and restarted the word, but it normalized the boundary rather than marking it with the requested symbol. That is better understood as a formatting limitation than a failure to recover the spoken repair. Nyra's own benchmark reports several cut-off metrics because exact fragments are acoustically ambiguous, and its headline cut-off F1 gives credit when a model preserves an aligned token at the interruption even without a dedicated cut-off marker Nyra Labs https://nyra-labs.com/research/nyra-verbatim-speech-benchmark . Not all repetition means the same thing We also tested whether Symphony could distinguish two recordings containing superficially similar repetition: "I really, really need this fixed today." and: "I, I need this fixed today." In both cases, a purely mechanical transcriber can observe that a word was repeated. The more useful question is why. For the first recording, Symphony preserved both instances of "really" and classified the repetition as emphasis. Its explanation treated the repetition as an intensifier rather than as an abandoned attempt to form the sentence. For the second recording, Symphony preserved both instances of "I" but classified the repetition as a speech repair. It interpreted the first "I" as a false start followed by an immediate restart. The words alone do not fully explain this distinction. The same surface behavior, a repeated token, can carry different functions depending on the word, its delivery, and the surrounding language. The transcript should fit the task On Nyra's English benchmark, Symphony nearly matches a purpose-built verbatim transcription model on fillers, repetitions, and cut-offs. In controlled tests, it also used those signals to recover the speaker's final request, identify a corrected value, and distinguish deliberate emphasis from a speech repair. Every transcription task has its own specific needs and requirements. Disfluency is an extremely important part of that story, but it is only one part. The broader problem is to build systems that can reliably preserve, remove, or interpret the features of the transcript required for its task. To get started with Symphony, create an account https://sprag.ai/auth/signup .