{"slug": "why-speech-to-text-is-more-than-calling-an-ai-api", "title": "Why Speech-to-Text Is More Than Calling an AI API", "summary": "Echoryte, a workspace that turns recordings into editable transcripts, has detailed the engineering challenges of building speech-to-text products beyond simple API calls. The project emphasizes treating each processing step as a contract, inspecting media files with tools like ffprobe, and separating jobs, attempts, and publications to handle retries and failures robustly.", "body_md": "A speech-to-text demo can be wonderfully small:\n\n``` js\nconst result = await provider.transcribe(audioUrl);\nreturn result.text;\n```\n\nThat code is enough to prove that a model can recognize speech.\n\nIt is nowhere near enough to prove that a user can trust the result.\n\nGive the demo a two-hour interview instead of a 20-second clip. Let the upload lose its connection at 87%. Let the provider send the same webhook twice. Let two words overlap by 80 milliseconds, then let the next word arrive ten seconds out of order. Let the user correct a name while an AI summary is waiting in a queue.\n\nAt that point, speech-to-text stops being an API integration. It becomes a document system, a media pipeline, and a distributed-systems problem wearing headphones.\n\nWhile building [Echoryte](https://echoryte.com), a workspace that turns recordings into editable, time-linked transcripts, we found a useful way to frame the problem:\n\nA speech API returns a hypothesis. A transcription product must turn that hypothesis into a durable, navigable, and explainable document.\n\nThis article is not a comparison of speech models. Models and pricing change too quickly for that to age well. Instead, it is a map of the engineering boundaries that remain regardless of which provider you call.\n\nThe naive architecture has two boxes:\n\n``` php\nrecording -> speech API -> text\n```\n\nA useful system looks closer to this:\n\n``` php\nupload\n  -> inspect the actual media\n  -> prepare a stable audio derivative\n  -> reserve work and enqueue a job\n  -> select a compatible provider/model\n  -> submit an attempt\n  -> wait through webhook and polling paths\n  -> validate and normalize the result\n  -> publish a transcript revision\n  -> edit, search, translate, summarize, and export\n```\n\nEvery arrow is a place where state can be lost, repeated, corrupted, or made ambiguous.\n\nThe important shift is to treat each boundary as a contract rather than a convenient function call.\n\nA user uploads interview.mp3. The browser reports audio/mpeg. It is tempting to trust both.\n\nNeither is authoritative.\n\nExtensions can be wrong. MIME types are supplied by clients. Containers may hold several audio tracks, no audio track, damaged timestamps, or codecs your downstream provider cannot decode. Long recordings may also be variable-bitrate media whose reported duration is not what you expect.\n\nBefore transcription, inspect the bytes with a media probe such as ffprobe and answer concrete questions:\n\nIn Echoryte's pipeline, ingestion produces a provider-friendly audio derivative and a browser-friendly playback derivative. Doing this once creates a stable input for both recognition and later review.\n\nThis also improves error messages. “The API failed” is not useful. “This file contains no audio stream” or “the measured duration exceeds your plan” tells the user what to do next.\n\nA good rule is:\n\nValidate business limits against inspected media, not against metadata declared by the client.\n\nThis distinction sounds academic until the first retry.\n\nWe model three separate things:\n\n```\ntype TranscriptionJob = {\n  id: string;\n  fileId: string;\n  requestedTier: \"fast\" | \"standard\" | \"precision\";\n  languageHint?: string;\n  diarization: boolean;\n};\n\ntype TranscriptionAttempt = {\n  id: string;\n  jobId: string;\n  provider: string;\n  model: string;\n  providerJobId?: string;\n  status: \"submitted\" | \"waiting\" | \"succeeded\" | \"failed\" | \"ignored\";\n};\n\ntype TranscriptPublication = {\n  jobId: string;\n  version: number;\n  revision: number;\n  objectKey: string;\n};\n```\n\nThe job represents user intent: “transcribe this recording with these capabilities.”\n\nAn attempt represents one execution of that intent against one provider and model.\n\nA publication represents the result that users are allowed to see and edit.\n\nSeparating them gives you several useful properties:\n\nThe last point matters more than it appears. If a provider succeeds but your worker crashes before publication, the provider request happened, the cost happened, and the user-visible result did not. One status field cannot describe all three facts.\n\nSpeech providers do not have one-dimensional capability.\n\nSupport varies by model, language, automatic language detection, word timestamps, speaker diarization, latency, and sometimes region. A fallback that returns plain text when the product promises word-level navigation is not a fallback. It is a silent contract change.\n\nA better router first removes incompatible candidates and only then ranks the survivors:\n\n```\nfunction isCompatible(model: ModelCapability, request: RequestFeatures) {\n  const language = resolveLanguage(model, request.languageHint);\n\n  return (\n    model.tiers.includes(request.tier) &&\n    language.supportsWordTimestamps &&\n    (!request.diarize || language.supportsDiarization) &&\n    (request.languageHint || language.supportsAutomaticDetection)\n  );\n}\n\nconst candidates = capabilitySnapshot.models\n  .filter(model => isCompatible(model, request))\n  .sort(rankByQualityLatencyAndCost);\n```\n\nNotice the capability snapshot. Provider documentation, model IDs, and language support change. Versioning the routing data lets you answer a difficult operational question later:\n\n“Why did this job choose that model on that day?”\n\nFailover should use the same compatibility filter and exclude capabilities already attempted. Otherwise, a retry loop can bounce between equivalent failures or quietly drop a requested feature.\n\nLong recordings do not belong in an HTTP request-response cycle. They need durable background work.\n\nThat introduces at-least-once behavior almost everywhere:\n\nOur useful mental model is a state machine backed by the database:\n\n``` php\nqueued -> claimed -> submitted -> waiting -> normalizing -> published\n             |            |          |            |\n             +------------+----------+------------+-> failed / ignored\n```\n\nThe queue schedules work; it is not the source of truth.\n\nWhen a worker claims a job, it receives a lease token and expiry. It renews the lease while doing slow work. Every later write checks the same token. If the lease is lost, that execution can no longer publish.\n\nThis prevents an old worker from waking up after a pause and overwriting the result produced by a newer worker.\n\nExternal operations also need stable identities. Creating an attempt, attaching a provider job ID, storing a raw result, reserving a transcript version, and publishing should all be safe to replay.\n\nFor callback-based providers, the webhook handler should do very little:\n\nPolling remains useful as a recovery path when a callback never arrives. Webhooks reduce latency; polling closes the reliability gap.\n\nOne more subtle point: progress is a user-interface estimate, not provider truth. A bar that moves smoothly to 73% does not mean 73% of the words exist. Show stages and historical time ranges, and label estimates as estimates.\n\nEven a successful provider response is external data. Parse it at runtime.\n\nA TypeScript interface cannot reject NaN, negative timestamps, missing fields, invalid confidence values, or a word whose start is after its end. A runtime schema can.\n\nAfter validation, normalize the result into a provider-independent structure:\n\n```\ntype Word = {\n  text: string;\n  startMs: number;\n  endMs: number;\n  confidence?: number;\n};\n\ntype Segment = {\n  id: string;\n  speakerId: string | null;\n  startMs: number;\n  endMs: number;\n  words: Word[];\n};\n\ntype Transcript = {\n  version: number;\n  durationMs: number;\n  language: string;\n  speakers: Speaker[];\n  segments: Segment[];\n};\n```\n\nNormalization is not just renaming fields. It is where you define what “valid time” means.\n\nFor example, Echoryte's current normalizer tolerates a small, bounded word overlap by clipping the next start time. It rejects a large overlap or a word that reverses the timeline. It normalizes Unicode, removes control characters, and refuses words that become empty.\n\nSegments are then built around editing and reading behavior, not around arbitrary provider paragraphs. A speaker change forces a boundary. So does a meaningful silence. Very long segments are split at safe word boundaries.\n\nThe principle is more important than the exact thresholds:\n\nRepair only what you can repair without changing meaning. Reject ambiguity before it becomes durable data.\n\nSilently sorting wildly disordered words may produce JSON that passes a schema, but it can make clicking a quote jump to the wrong moment. That is worse than an explicit failure.\n\nKeeping the original provider result separately is also valuable. You can re-run normalization after improving your rules, investigate disputes, or compare provider behavior without calling the API again.\n\nPlain text throws away the most valuable part of a transcript: its relationship to the recording.\n\nOnce every word has time, several product behaviors become possible:\n\nBut editing creates another problem. If you replace the transcript as one giant text blob, you lose stable identities and make concurrent changes difficult to reason about.\n\nA patch model is often a better fit:\n\n```\ntype TranscriptPatch =\n  | { type: \"replaceWordText\"; segmentId: string; wordIndex: number; text: string }\n  | { type: \"setSegmentSpeaker\"; segmentId: string; speakerId: string | null }\n  | { type: \"updateSpeakerLabel\"; speakerId: string; label: string }\n  | { type: \"splitSegment\"; segmentId: string; wordIndex: number };\n```\n\nEach accepted batch advances a revision. Periodically, patches can be compacted into a new snapshot.\n\nRevision numbers solve a quiet but serious race:\n\nThe result is not necessarily wrong, but it is based on an older source. Store sourceRevision: 12 with the output and mark it as stale relative to revision 15. Do not pretend it reflects edits it never saw.\n\nThe same rule applies to translations and exports: pin the input revision when the request is created.\n\nAfter transcription, it is natural to ask an LLM to add punctuation, produce chapters, extract action items, or answer questions.\n\nThe dangerous shortcut is to let the model rewrite the canonical transcript freely.\n\nFor punctuation enhancement, we use a smaller contract: the model may propose structured operations such as “add punctuation after word 42” or “insert a paragraph break after word 108.” It cannot replace the recognized words. Every index and punctuation mark is validated; invalid output falls back to the original transcript.\n\nFor summaries and chat, the transcript is data, not trusted instructions. Delimit it clearly in the prompt. Validate time references against the recording duration. Sanitize rendered Markdown. Never allow text spoken inside an uploaded recording to redefine the system prompt.\n\nTranslation needs similar honesty. Segment timing can remain attached to the source segment, but translated words do not magically receive forced-alignment-quality timestamps. If word timing is interpolated, label it as estimated.\n\nA useful hierarchy is:\n\n``` php\nrecording evidence\n  -> normalized transcript\n      -> user edits\n          -> derived AI outputs\n```\n\nDerived data should point back to its source. It should not quietly become the source.\n\nTranscription systems need rich diagnostics, but recordings often contain interviews, research, classes, customer calls, or other sensitive material.\n\nYou can observe the pipeline without logging content.\n\nUseful fields include:\n\nAvoid logging filenames, transcript text, signed media URLs, webhook secrets, or user-supplied source URLs with query strings.\n\nThis forces better operational design. “Provider returned 429 for attempt A” is searchable and actionable. A dump of the user's entire response payload is neither necessary nor safe.\n\nYou do not need the full architecture for a weekend prototype. You do need to know which shortcuts you are taking.\n\nA reasonable progression is:\n\nBefore calling the system production-ready, ask:\n\nIf several answers are “no,” the speech model may still be excellent. The product is not finished.\n\nThe API call is the impressive part of the demo. The surrounding contracts are the valuable part of the product.\n\nA trustworthy transcription system must preserve three kinds of truth at once:\n\nKeep those truths separate, connect them with stable identities, and make every transition safe to repeat.\n\nThe speech API recognizes words. The system around it earns the user's trust.\n\nWhat failure mode surprised you the first time you built an AI pipeline?", "url": "https://wpnews.pro/news/why-speech-to-text-is-more-than-calling-an-ai-api", "canonical_source": "https://dev.to/loklok5/why-speech-to-text-is-more-than-calling-an-ai-api-418e", "published_at": "2026-09-04 04:41:04+00:00", "updated_at": "2026-09-04 04:53:57.617107+00:00", "lang": "en", "topics": ["developer-tools", "artificial-intelligence", "natural-language-processing", "ai-infrastructure"], "entities": ["Echoryte"], "alternates": {"html": "https://wpnews.pro/news/why-speech-to-text-is-more-than-calling-an-ai-api", "markdown": "https://wpnews.pro/news/why-speech-to-text-is-more-than-calling-an-ai-api.md", "text": "https://wpnews.pro/news/why-speech-to-text-is-more-than-calling-an-ai-api.txt", "jsonld": "https://wpnews.pro/news/why-speech-to-text-is-more-than-calling-an-ai-api.jsonld"}}