Why Speech-to-Text Is More Than Calling an AI API

Echoryte, a workspace that turns recordings into editable transcripts, has detailed the engineering challenges of building speech-to-text products beyond simple API calls. The project emphasizes treating each processing step as a contract, inspecting media files with tools like ffprobe, and separating jobs, attempts, and publications to handle retries and failures robustly.

A speech-to-text demo can be wonderfully small: js const result = await provider.transcribe audioUrl ; return result.text; That code is enough to prove that a model can recognize speech. It is nowhere near enough to prove that a user can trust the result. Give the demo a two-hour interview instead of a 20-second clip. Let the upload lose its connection at 87%. Let the provider send the same webhook twice. Let two words overlap by 80 milliseconds, then let the next word arrive ten seconds out of order. Let the user correct a name while an AI summary is waiting in a queue. At that point, speech-to-text stops being an API integration. It becomes a document system, a media pipeline, and a distributed-systems problem wearing headphones. While building Echoryte https://echoryte.com , a workspace that turns recordings into editable, time-linked transcripts, we found a useful way to frame the problem: A speech API returns a hypothesis. A transcription product must turn that hypothesis into a durable, navigable, and explainable document. This article is not a comparison of speech models. Models and pricing change too quickly for that to age well. Instead, it is a map of the engineering boundaries that remain regardless of which provider you call. The naive architecture has two boxes: php recording - speech API - text A useful system looks closer to this: php upload - inspect the actual media - prepare a stable audio derivative - reserve work and enqueue a job - select a compatible provider/model - submit an attempt - wait through webhook and polling paths - validate and normalize the result - publish a transcript revision - edit, search, translate, summarize, and export Every arrow is a place where state can be lost, repeated, corrupted, or made ambiguous. The important shift is to treat each boundary as a contract rather than a convenient function call. A user uploads interview.mp3. The browser reports audio/mpeg. It is tempting to trust both. Neither is authoritative. Extensions can be wrong. MIME types are supplied by clients. Containers may hold several audio tracks, no audio track, damaged timestamps, or codecs your downstream provider cannot decode. Long recordings may also be variable-bitrate media whose reported duration is not what you expect. Before transcription, inspect the bytes with a media probe such as ffprobe and answer concrete questions: In Echoryte's pipeline, ingestion produces a provider-friendly audio derivative and a browser-friendly playback derivative. Doing this once creates a stable input for both recognition and later review. This also improves error messages. “The API failed” is not useful. “This file contains no audio stream” or “the measured duration exceeds your plan” tells the user what to do next. A good rule is: Validate business limits against inspected media, not against metadata declared by the client. This distinction sounds academic until the first retry. We model three separate things: type TranscriptionJob = { id: string; fileId: string; requestedTier: "fast" | "standard" | "precision"; languageHint?: string; diarization: boolean; }; type TranscriptionAttempt = { id: string; jobId: string; provider: string; model: string; providerJobId?: string; status: "submitted" | "waiting" | "succeeded" | "failed" | "ignored"; }; type TranscriptPublication = { jobId: string; version: number; revision: number; objectKey: string; }; The job represents user intent: “transcribe this recording with these capabilities.” An attempt represents one execution of that intent against one provider and model. A publication represents the result that users are allowed to see and edit. Separating them gives you several useful properties: The last point matters more than it appears. If a provider succeeds but your worker crashes before publication, the provider request happened, the cost happened, and the user-visible result did not. One status field cannot describe all three facts. Speech providers do not have one-dimensional capability. Support varies by model, language, automatic language detection, word timestamps, speaker diarization, latency, and sometimes region. A fallback that returns plain text when the product promises word-level navigation is not a fallback. It is a silent contract change. A better router first removes incompatible candidates and only then ranks the survivors: function isCompatible model: ModelCapability, request: RequestFeatures { const language = resolveLanguage model, request.languageHint ; return model.tiers.includes request.tier && language.supportsWordTimestamps && request.diarize || language.supportsDiarization && request.languageHint || language.supportsAutomaticDetection ; } const candidates = capabilitySnapshot.models .filter model = isCompatible model, request .sort rankByQualityLatencyAndCost ; Notice the capability snapshot. Provider documentation, model IDs, and language support change. Versioning the routing data lets you answer a difficult operational question later: “Why did this job choose that model on that day?” Failover should use the same compatibility filter and exclude capabilities already attempted. Otherwise, a retry loop can bounce between equivalent failures or quietly drop a requested feature. Long recordings do not belong in an HTTP request-response cycle. They need durable background work. That introduces at-least-once behavior almost everywhere: Our useful mental model is a state machine backed by the database: php queued - claimed - submitted - waiting - normalizing - published | | | | +------------+----------+------------+- failed / ignored The queue schedules work; it is not the source of truth. When a worker claims a job, it receives a lease token and expiry. It renews the lease while doing slow work. Every later write checks the same token. If the lease is lost, that execution can no longer publish. This prevents an old worker from waking up after a pause and overwriting the result produced by a newer worker. External operations also need stable identities. Creating an attempt, attaching a provider job ID, storing a raw result, reserving a transcript version, and publishing should all be safe to replay. For callback-based providers, the webhook handler should do very little: Polling remains useful as a recovery path when a callback never arrives. Webhooks reduce latency; polling closes the reliability gap. One more subtle point: progress is a user-interface estimate, not provider truth. A bar that moves smoothly to 73% does not mean 73% of the words exist. Show stages and historical time ranges, and label estimates as estimates. Even a successful provider response is external data. Parse it at runtime. A TypeScript interface cannot reject NaN, negative timestamps, missing fields, invalid confidence values, or a word whose start is after its end. A runtime schema can. After validation, normalize the result into a provider-independent structure: type Word = { text: string; startMs: number; endMs: number; confidence?: number; }; type Segment = { id: string; speakerId: string | null; startMs: number; endMs: number; words: Word ; }; type Transcript = { version: number; durationMs: number; language: string; speakers: Speaker ; segments: Segment ; }; Normalization is not just renaming fields. It is where you define what “valid time” means. For example, Echoryte's current normalizer tolerates a small, bounded word overlap by clipping the next start time. It rejects a large overlap or a word that reverses the timeline. It normalizes Unicode, removes control characters, and refuses words that become empty. Segments are then built around editing and reading behavior, not around arbitrary provider paragraphs. A speaker change forces a boundary. So does a meaningful silence. Very long segments are split at safe word boundaries. The principle is more important than the exact thresholds: Repair only what you can repair without changing meaning. Reject ambiguity before it becomes durable data. Silently sorting wildly disordered words may produce JSON that passes a schema, but it can make clicking a quote jump to the wrong moment. That is worse than an explicit failure. Keeping the original provider result separately is also valuable. You can re-run normalization after improving your rules, investigate disputes, or compare provider behavior without calling the API again. Plain text throws away the most valuable part of a transcript: its relationship to the recording. Once every word has time, several product behaviors become possible: But editing creates another problem. If you replace the transcript as one giant text blob, you lose stable identities and make concurrent changes difficult to reason about. A patch model is often a better fit: type TranscriptPatch = | { type: "replaceWordText"; segmentId: string; wordIndex: number; text: string } | { type: "setSegmentSpeaker"; segmentId: string; speakerId: string | null } | { type: "updateSpeakerLabel"; speakerId: string; label: string } | { type: "splitSegment"; segmentId: string; wordIndex: number }; Each accepted batch advances a revision. Periodically, patches can be compacted into a new snapshot. Revision numbers solve a quiet but serious race: The result is not necessarily wrong, but it is based on an older source. Store sourceRevision: 12 with the output and mark it as stale relative to revision 15. Do not pretend it reflects edits it never saw. The same rule applies to translations and exports: pin the input revision when the request is created. After transcription, it is natural to ask an LLM to add punctuation, produce chapters, extract action items, or answer questions. The dangerous shortcut is to let the model rewrite the canonical transcript freely. For punctuation enhancement, we use a smaller contract: the model may propose structured operations such as “add punctuation after word 42” or “insert a paragraph break after word 108.” It cannot replace the recognized words. Every index and punctuation mark is validated; invalid output falls back to the original transcript. For summaries and chat, the transcript is data, not trusted instructions. Delimit it clearly in the prompt. Validate time references against the recording duration. Sanitize rendered Markdown. Never allow text spoken inside an uploaded recording to redefine the system prompt. Translation needs similar honesty. Segment timing can remain attached to the source segment, but translated words do not magically receive forced-alignment-quality timestamps. If word timing is interpolated, label it as estimated. A useful hierarchy is: php recording evidence - normalized transcript - user edits - derived AI outputs Derived data should point back to its source. It should not quietly become the source. Transcription systems need rich diagnostics, but recordings often contain interviews, research, classes, customer calls, or other sensitive material. You can observe the pipeline without logging content. Useful fields include: Avoid logging filenames, transcript text, signed media URLs, webhook secrets, or user-supplied source URLs with query strings. This forces better operational design. “Provider returned 429 for attempt A” is searchable and actionable. A dump of the user's entire response payload is neither necessary nor safe. You do not need the full architecture for a weekend prototype. You do need to know which shortcuts you are taking. A reasonable progression is: Before calling the system production-ready, ask: If several answers are “no,” the speech model may still be excellent. The product is not finished. The API call is the impressive part of the demo. The surrounding contracts are the valuable part of the product. A trustworthy transcription system must preserve three kinds of truth at once: Keep those truths separate, connect them with stable identities, and make every transition safe to repeat. The speech API recognizes words. The system around it earns the user's trust. What failure mode surprised you the first time you built an AI pipeline?