Whisper-powered transcription, explained: what actually happens to your audio

OpenAI's Whisper model processes audio by converting it into log-Mel spectrograms, which are then fed into an encoder-decoder transformer architecture. The model operates on 30-second audio chunks and uses special tokens to handle transcription, translation, and timestamping in a single forward pass. Trained on 680,000 hours of weakly supervised multilingual data, Whisper achieves robust speech recognition across 50+ languages.

If you have used any AI transcription tool in the last two years, there is a good chance a Whisper-family model did the heavy lifting under the hood. OpenAI's release in 2022 reset expectations for open speech recognition, and the architecture is worth understanding whether you are wiring it into a product or just curious why the output got so good so fast. This is a walk through what happens between "audio file" and "text," at the level of detail a developer actually wants. No math beyond what clarifies the idea. A model does not consume raw waveform samples directly. The first thing that happens is a conversion into a log-Mel spectrogram, which is a 2D representation of the audio: time on one axis, frequency on the other, intensity as the value. Whisper resamples everything to 16 kHz and computes an 80-channel log-Mel spectrogram using 25-millisecond windows with a 10-millisecond stride. In plain terms, it slices the audio into heavily overlapping short frames and, for each frame, measures how much energy sits in each of 80 frequency bands, on a scale tuned to roughly match how human hearing weights pitch. The result is an image-like tensor, and treating speech as an image is a large part of why convolutional and transformer machinery works on it at all. Whisper processes audio in 30-second chunks. That is a fixed architectural choice, not a preference. Shorter clips get padded to 30 seconds; longer files get split and stitched back together afterward. This detail explains a class of bugs you will hit if you build on raw Whisper. If a chunk boundary lands in the middle of a word, or in the middle of a sentence, the model can lose the thread at the seam. Production systems put a voice activity detection pass in front, so cuts happen during natural pauses rather than mid-syllable. If you have ever seen a repeated phrase or a dropped word at a suspiciously round timestamp, a naive 30-second split is a prime suspect. The spectrogram goes into an encoder: two convolutional layers, positional encoding, then a stack of transformer blocks. The encoder's job is to turn the acoustic picture into a sequence of internal representations that capture what was said, independent of who said it or in what language. A decoder then generates text tokens autoregressively, one at a time, attending to the encoder output. This is the same encoder-decoder transformer shape you have seen in translation models, which is not a coincidence. Whisper was trained partly on translation, so the machinery is shared. Here is the elegant part. Whisper does not run separate models for "detect the language," "transcribe," "translate," and "find the timestamps." It folds all of that into one decoder using special tokens at the start of the sequence. The decoder is prompted with tokens that say, in effect, "this is the transcription task, the language is Spanish, include timestamps," and it conditions its output on those instructions. Swap the task token and the same model translates instead of transcribing. This is how a single set of weights, trained on around 680,000 hours of weakly supervised multilingual audio, covers 50-plus languages and collapses what used to be a whole pipeline of separate components into one forward pass. That training approach matters too. "Weakly supervised" means it learned from a huge pile of imperfect, real-world transcripts scraped from the internet rather than a small, pristine, hand-labeled set. The messiness is a feature. It is why the model tolerates accents, background noise, and cross-talk far better than earlier systems trained on clean studio data. Raw Whisper gives you a transcript. Turning that into something a non-engineer wants involves several stages the base model does not handle: punctuation and casing cleanup, speaker diarization Whisper does not tell you who spoke , and a language-model layer that summarizes and pulls out structure. If you would rather consume the finished chain than assemble it, a tool like Vomo's audio to text https://vomo.ai/tools/audio-to-text wraps the whole thing: upload or paste a link, get a speaker-labeled transcript at 95%+ accuracy plus a generated summary, chapters, and action items, with the option to query the transcript conversationally. Useful as a reference implementation for what "done" looks like, even if you end up building your own. The base architecture is stable and well understood now. Most of the interesting engineering in 2026 has moved to the layers around it: cutting latency toward real time, and feeding domain vocabulary in so the model stops mangling your product names. The recognition problem is largely solved. Everything on top of it is still wide open.