Building a Speech Recognizer from Scratch in Rust with No Dependencies A new book, "Building a Speech Recognizer from Scratch," walks readers through building a speech-to-text command line tool in Rust with an empty [dependencies] section from the first page to the last. The book covers hand-written WAV parsing, two Fourier transform implementations, a dynamic time warping digit recognizer, and a from-scratch neural network library with backpropagation, convolutions, recurrent layers, Adam and CTC loss trained on a laptop CPU in about 40 minutes. The author reports the resulting recognizer knows 35 words and gets about one word in ten wrong on its own test set, and the final chapters add a hand-written inference engine for OpenAI's Whisper to transcribe ordinary English. From sound waves to words, in Rust. Welcome to Building a Speech Recognizer from Scratch. This is a book that walks you through building a program that listens to a person talking and writes down what they said. We start with a file full of numbers that happens to be a recording of somebody saying "seven", and we end with a command line tool that transcribes speech from a file or from your microphone. There is one rule, and we keep it for the whole book: we write everything ourselves. No PyTorch, no TensorFlow, no signal processing crates, no linear algebra crates, and no pretrained model until we have built and trained a small working recognizer of our own. The dependencies section of our Cargo.toml is empty on page one and it is still empty on the last page. If we need a Fourier transform, a matrix multiply, a random number generator, backpropagation, or a JSON parser, we write it, and we test it. You do not need to know any signal processing or any machine learning. You need to be comfortable reading Rust and you need the maths you had at sixteen. Everything else is built up from nothing, one piece at a time, and every piece is something you can run. A figure from chapter 4: the first third of the book in one picture. By then we have written every box. Four things, each one standing on the one before it. 1. Ears. Code that turns a WAV file into the picture a recognizer actually looks at. We parse the file by hand, cut the sound into short frames, write the Fourier transform twice slowly, then fast , and squeeze the result through a model of the human ear. Chapters 1 to 4. 2. A recognizer that learns nothing. Dynamic time warping: keep a few recordings of each digit, and compare a new recording against all of them. A short function and no training at all, and it recognizes spoken digits. This is our first "it works" moment and it arrives early on purpose. Chapter 5. 3. A recognizer that learns. A neural network library with backpropagation written by hand and checked numerically: linear layers, convolutions, recurrent layers, Adam, and the CTC loss that lets a network learn to spell without being told which sound goes with which letter. We train it on a laptop CPU in about forty minutes, and it spells out short phrases from files and live from a microphone. Chapters 6 to 11. 4. Somebody else's brain in our skull. Our model knows 35 words, because that is what a laptop and a free dataset can teach it. So in the last part we write an inference engine for OpenAI's Whisper and load the published weights into it. Same rule: we parse the weight file ourselves, write attention ourselves, and feed it with the kind of front end we built in part one, rebuilt to Whisper's constants. At that point our program transcribes ordinary English. Chapters 12 and 13. We try to be honest about numbers throughout. Our own recognizer gets about one word in ten wrong on its own test set, and it only knows 35 words. The book says so, shows where it fails, and explains what separates it from Whisper: a great deal of data and compute, and a small number of ideas attention, and a decoder that reads its own output that the last chapters run but do not train. In order. Each chapter assumes the ones before it. Part I: Hearing - Chapter 1 - Sound Is Just Numbers https://github.com/Ghvstcode/speech-recognizer-from-scratch/blob/main/chapters/01-sound-is-just-numbers.md - Chapter 2 - Slicing Time https://github.com/Ghvstcode/speech-recognizer-from-scratch/blob/main/chapters/02-slicing-time.md - Chapter 3 - The Fourier Transform, By Hand https://github.com/Ghvstcode/speech-recognizer-from-scratch/blob/main/chapters/03-the-fourier-transform-by-hand.md - Chapter 4 - Hearing Like A Human https://github.com/Ghvstcode/speech-recognizer-from-scratch/blob/main/chapters/04-hearing-like-a-human.md Part II: A Recognizer With No Learning Part III: Learning Part IV: Standing On Shoulders - Chapter 12 - Reading Someone Else's Weights https://github.com/Ghvstcode/speech-recognizer-from-scratch/blob/main/chapters/12-reading-someone-elses-weights.md - Chapter 13 - A Transformer That Listens https://github.com/Ghvstcode/speech-recognizer-from-scratch/blob/main/chapters/13-a-transformer-that-listens.md - Chapter 14 - Where To Go From Here https://github.com/Ghvstcode/speech-recognizer-from-scratch/blob/main/chapters/14-where-to-go-from-here.md The chapters show the code function by function, in the order you would write it, and the finished code for the whole book is in code/ https://github.com/Ghvstcode/speech-recognizer-from-scratch/blob/main/code . I would encourage you to type it rather than copy it. Every chapter ends with a few things to try yourself; they are where most of the learning happens. Some ideas are easier to play with than to read about, so several chapters link to a small interactive page. They are plain HTML files in interactive/ https://github.com/Ghvstcode/speech-recognizer-from-scratch/blob/main/interactive ; open interactive/index.html https://github.com/Ghvstcode/speech-recognizer-from-scratch/blob/main/interactive/index.html in a browser to see them all: sampling and aliasing, windowing, the Fourier transform as a spinning correlation, the mel filterbank, the DTW grid, the CTC lattice, and beam search. You need Rust 1.88 or newer installed with rustup https://rustup.rs . Nothing else for most of the book. Chapter 11 uses ffmpeg to read the microphone. git clone