{"slug": "transcribing-a-lecture-and-transcribing-a-caller-are-not-the-same-problem", "title": "Transcribing a Lecture and Transcribing a Caller Are Not the Same Problem", "summary": "A developer who shipped LectureNotes AI and now builds voice agents argues that long-form lecture transcription and real-time call transcription are fundamentally different problems that fail, are evaluated, and are improved in different layers of the stack. The engineer reports that recording conditions and domain-vocabulary biasing delivered larger perceived accuracy gains than swapping speech-to-text models, and recommends measuring the specific terms or fields users care about rather than aggregate word error rate.", "body_md": "The first time I opened a Retell call transcript, I assumed I already knew this problem.\n\nI had shipped LectureNotes AI before that: a note-taking app that records a lecture and turns it into a summary and a revision outline. Audio in, text out, users happy. So when voice agents became most of my work, I filed speech recognition under solved and moved on to the parts I thought were hard.\n\nI was wrong in a way that cost me a few weeks. Transcribing a fifty minute lecture and transcribing a caller on a phone line share a model family and share almost nothing else. They fail differently, they are evaluated differently, and the thing that improves each one is in a completely different layer of the stack.\n\nHere is the honest split, after shipping both.\n\nLong-form transcription is a batch problem. A student hits record, sits through an hour, closes the laptop. The audio arrives as one file. Nobody is waiting on a socket for the next token. If a pass comes out badly you can run it again with different settings and nobody knows. You can spend three minutes of compute on an hour of audio and the user experiences that as \"it was ready when I looked\".\n\nA voice agent inverts every one of those properties. Audio arrives as a stream, the useful unit is a three to eight second turn, and the transcript is consumed immediately by a model that is about to say something out loud to a human being. There is no second pass and no running it again with a bigger model. The caller is on the line, and every millisecond you spend is silence they are listening to.\n\nOnce you frame it that way, the question stops being which STT provider is best and starts being which failure you can afford.\n\nThe single biggest accuracy win on the note-taking side had nothing to do with the transcription model. It was recording conditions.\n\nA lecture recording is a phone lying on a desk six rows back, in a room with hard walls and an air conditioner. The lecturer walks around. Against that, the gap between two good STT models is noise.\n\nWhat actually helped was unglamorous: handling the recording session properly so it survives the screen locking and the app being backgrounded, being honest in the UI about what a bad recording is going to produce, and getting the sample rate and format right at the source instead of repairing it downstream. If you upsample rubbish you get expensive rubbish.\n\nIf you are building anything that ingests audio, that is where I would spend the first week, not on provider benchmarks.\n\nWER treats every word as equal. Users do not.\n\nA lecture transcript that is broadly correct but mangles the lecturer's specific terminology is useless, because the terminology is the entire reason a student is reading it. Course jargon, people's names, formula names, abbreviations: those are exactly the tokens with the least redundancy in the surrounding sentence, so context cannot repair them. Meanwhile a transcript that quietly drops filler words and tidies false starts usually tests better than a faithful one.\n\nSo the useful evaluation was never an aggregate score. It was: did the terms survive? Feeding known domain vocabulary in as a bias list or prompt hint bought more perceived quality than any model swap I tried, because it targeted the small set of tokens carrying all the meaning.\n\nThe same shape shows up on the phone side with different tokens. Identifiers have no redundancy either: nothing around a postcode, a registration number or an email address tells the model what it should have been.\n\nPick the twenty terms or the five fields that actually matter to your user and measure those. An aggregate score will keep telling you things are fine while the only words anyone cares about are wrong.\n\nFor long-form, the transcript is not the deliverable. The summary and the outline are, and that is where the quality is genuinely won.\n\nThree things mattered more than I expected:\n\n**Chunk on the audio, not on the character count.** Cutting a transcript into fixed windows regularly slices a definition in half, and the summary then confidently loses it. Chunking around natural pauses, with overlap between chunks, produced noticeably better output for the same model and the same prompt.\n\n**Keep timestamps all the way through.** If a student cannot jump from a summary line back to the moment it came from, they cannot check it, and if they cannot check it they will not trust it. Traceability is a feature, not plumbing.\n\n**Structure is a product decision, not a prompt afterthought.** Students did not want prose. They wanted something scannable: headings, short lines, definitions pulled out. An excellent essay-shaped summary tested worse than a mediocre outline-shaped one.\n\nI run the same instinct on voice work now. The transcript is raw signal. Everything the client actually cares about, the booking, the CRM record, the outcome label, is a structured extraction sitting on top of it.\n\nThen there is the call.\n\nTelephony audio is narrowband. This is the part I wish someone had said to me plainly: it is not a lightly degraded version of good audio, it is audio with the top of the spectrum removed, and the frequencies it removes are the ones that distinguish consonants. That is why f and s collapse into each other, why m and n become a coin toss, and why five and nine are a permanent problem when you are taking a phone number. It is not the model failing. That information never reached the model.\n\nOn top of that you get callers in cars, on speaker, standing in a busy shop, with regional accents that a US-tuned recognition setting handles badly.\n\nBecause the audio is worse and the stakes are higher, the fix has to leave the STT layer entirely. You cannot make the line better. You can design a conversation that survives mishearing: confirm identifiers in small chunks as you take them, validate shape downstream in automation rather than hoping the prompt caught it, and build a repair ladder with an exit, so a bad capture becomes a text message or a transfer instead of a loop.\n\nIn batch, latency is a scheduling detail. In a call, it is the product.\n\nStreaming STT adds cost at the point where you are most sensitive. Endpointing has to decide when the caller has actually stopped talking, and that trades directly against being interrupted or leaving dead air. Set it aggressively and the agent talks over people. Set it conservatively and the agent feels slow. It is also the one setting I tune per moment rather than globally, because a caller reading out a phone number and a caller saying \"yes\" want opposite values.\n\nNobody has ever complained that a lecture summary arrived four hundred milliseconds late.\n\nThis is the difference I underrate least now.\n\nA bad lecture transcript is obvious. The user reads it, sees the mess, re-records or complains. The feedback loop is short and self-correcting.\n\nA bad phone capture is silent. The call sounds fine. The caller is polite. The agent confirms something plausible, the automation writes a record, and the failure surfaces days later when an email bounces or someone does not turn up to an appointment. Nobody watched it happen.\n\nThat is why I score voice agents per field rather than per call, and why I store the raw transcript and audio next to the extracted result. When a structured value is wrong, the only way to find out why is to go back to the signal.\n\nA quick heuristic, since founders ask me this often:\n\nThe uncomfortable conclusion after shipping both is that the model was rarely the interesting variable. Capture conditions, chunking, structured extraction and honest evaluation moved both products more than any provider comparison did, and all four are things you own rather than things you buy.\n\nI wrote a longer version of this on my own site, with more of the voice agent stack around it: [Speech-to-text for lectures vs phone calls](https://nabeelbaghoor.com/blog/speech-to-text-lectures-vs-phone-calls/).", "url": "https://wpnews.pro/news/transcribing-a-lecture-and-transcribing-a-caller-are-not-the-same-problem", "canonical_source": "https://dev.to/nabeelbaghoor/transcribing-a-lecture-and-transcribing-a-caller-are-not-the-same-problem-45j0", "published_at": "2026-09-12 21:44:02+00:00", "updated_at": "2026-09-12 22:25:08.275921+00:00", "lang": "en", "topics": ["natural-language-processing", "ai-products", "developer-tools", "ai-agents"], "entities": ["LectureNotes AI", "Retell"], "alternates": {"html": "https://wpnews.pro/news/transcribing-a-lecture-and-transcribing-a-caller-are-not-the-same-problem", "markdown": "https://wpnews.pro/news/transcribing-a-lecture-and-transcribing-a-caller-are-not-the-same-problem.md", "text": "https://wpnews.pro/news/transcribing-a-lecture-and-transcribing-a-caller-are-not-the-same-problem.txt", "jsonld": "https://wpnews.pro/news/transcribing-a-lecture-and-transcribing-a-caller-are-not-the-same-problem.jsonld"}}