{"slug": "it-s-the-amnesia-stupid", "title": "It's the Amnesia, Stupid", "summary": "Voice AI startups are wasting compute and adding latency by regenerating identical responses and speech on every call, according to Luke Miller, co-founder of an unnamed company, who argues that the industry's failure to cache repeated phrases and structural decisions is the root cause of poor user experience. Miller says most of an agent's output is templated with only a few variables, yet every turn round-trips a frontier model and text-to-speech engine, making latency the product's defining flaw.", "body_md": "Infrastructure\n\nAugust 5, 2026\n\nIt's the amnesia, stupid. Voice AI's missing cache layer.\n\nEvery expensive computation in computing history got a cache in front of it. Voice AI is the exception. Why the cascade's seams are the product.\n\nLuke Miller\n\nCo-founder\n\nIn voice, latency is the product.\n\nEvery other interface gets a cheap way to hide its delays. A webpage renders progressively, a chatbot shows a typing indicator, an app puts up a spinner and you forgive it for ten seconds, free of charge. A phone call gets a filler phrase like \"let me just pull that up for you\" and that's the whole toolkit.\n\nIt buys two seconds, maybe three, before it curdles into stalling. Humans run turn-taking on a clock measured in milliseconds; a pause of two hundred feels attentive, a pause of two thousand feels broken, and one filler phrase is all the cushioning the medium allows. The caller doesn't think \"high P99.\" They think nobody's there, and then they hang up.\n\nThe entire user experience of a voice agent comes down to who controls the silence.\n\nRight now the answer, for almost every developer, is: NOT you. Every turn of every conversation round-trips a frontier model in someone else's region, behind someone else's queue, and your product's feel on any given afternoon depends on that company's load that afternoon. You don't have a latency problem. You have a latency landlord.\n\nNow look at where the wait and the bill actually come from. In a cascade turn, two meters dominate: the language model generating the response, and the synthesis speaking it.\n\nAnd audit what they're generating. The LLM is produced, token by token, at full compute, with the entire conversation re-fed as input every single turn - confirmations, readbacks, templated responses: sentences that are 90% the same sentence, call after call.\n\nThen the TTS renders that mostly-identical speech fresh, as if it had never been spoken before.\n\nThe industry is quietly admitting the point.\n\nYes, the savvy teams already hand-cache their greetings and filler phrases, and good for them. But a cached greeting is where the reuse ends, because the moment a sentence has a variable in it, exact-match caching is useless.\n\n\"Your appointment is Tuesday the 14th at 3:15\" gets said, in shape, millions of times a day, one slot changing, everything else identical, and every single one is generated fresh and synthesized fresh, twice through the meter. That's most of what an agent actually says.\n\nAnd around those two meters, the same expensive loop is also picking tools and validating inputs - decisions a lookup table could make.\n\nA CPU from 2009 handles the structural work without noticing. We put all of it on the scarcest silicon on earth, in the one position where every millisecond is audible.\n\nIt's amnesia. Very expensive amnesia.\n\nTo be precise about the patient: I'm talking about voice agents on the cascade architecture. You have an orchestrator coordinating speech-to-text, a language model, and text-to-speech, which is how nearly every production voice agent runs today. There are other architectures, and choosing between them is a discussion for another day.\n\nThe moment this clicked for us wasn't a latency graph. It was watching our customers fight non-deterministic models for reliability while all of them were wrestling with the same two problems:\n\n- Getting model calls to behave dependably across the stack,\n- Getting the same question to produce the same answer twice.\n\nMost of the agent is not a model.\n\nSomewhere in the middle of that it landed: the industry's reflex is to throw a model at every problem, when for most of the work there are tools that are faster, cheaper, more reliable and already extraordinarily distributed, in a way high-end GPUs very much are not.\n\nWe over-attribute agents to models the way we once over-attributed websites to servers — the behavior looks like intelligence, so we assume it's all inference.\n\nIt isn't.\n\nMost of it is structure, and structure runs anywhere. A voice agent shouldn't outsource all its thinking to a non-deterministic LLM. We just need to pull in some older, less sexy tooling.\n\nThis argument is older than the web browser.\n\nI didn't come up with the underlying law and neither did anyone in voice. Computing has now had to relearn it four times.\n\nIn 1993 the web was two years old and arguing about whether HTML should have an image tag. In the very thread where [Marc Andreessen proposed the tag](http://1997.webhistory.org/www.lists/www-talk.1993q1/0260.html), he was already defending what he called the single-hop principle i.e. a page should arrive in one round trip, against proposals that would piece pages together on the fly. The document web's first architecture fight was a latency fight.\n\nIn 1996, as dial-up brought the internet to consumers, Stuart Cheshire wrote the [rant that named the law](https://www.stuartcheshire.org/rants/latency.html): \"It's the latency, stupid.\" Bandwidth improves forever, latency has a floor set by the speed of light, and the floor doesn't negotiate. The industry was marketing fatter pipes. The pipe was never the problem.\n\nIn 2014 Guillermo Rauch dragged the law into the application web, in an essay that examined [JavaScript purely through UX](https://rauchg.com/2014/7-principles-of-rich-web-applications) which he defined as minimizing the time it takes a user to get what they came for. Pre-render what can be known before the request. Act on input before the network answers. Predict what's coming; he compressed that one into two words, \"negative latency.\" And he called out, by name, the then-fashionable tools whose whole trick was making the round trip feel shorter while leaving it in place.\n\nHe founded Vercel the next year, and the doctrine — compute the heavy things once, ahead of time, push the result next to the user, spend live computation only on what genuinely can't be known in advance — won the web so completely that we forgot it was ever contested. Even the sensitive stuff obeyed: the static shell caches everywhere precisely because your account details were never baked into it. They inject at request time into holes the precomputed layer left open.\n\nVoice Agents in 2026 is the web in 2015.\n\nThe expensive resource sits in the hot path of every single request, the precompute doctrine is available and unclaimed, and the medium is less forgiving than the web ever was.\n\nThe web's spinner was free and patient; voice's costs money and expires in three seconds. And we've already rebuilt the school Rauch dismissed. The industry's consensus answer to voice latency is streaming. Overlap the pipeline stages so the wait feels shorter.\n\nStreaming is genuinely useful and I'm not telling anyone to turn it off. But it polishes the round trip. It never asks why the round trip is there.\n\nSort your agent\n\nTake the agent apart and put every piece of work in one of two piles: needs the frontier, live, or just needs to have been done already.\n\nSort honestly and the second pile dwarfs the first. Routing and deterministic logic run on the CPU in microseconds. The state sits in memory, in place.\n\nSpeech — this is the one people resist, then can't unsee — is overwhelmingly assembled from pieces that could have been generated once, on GPUs, offline, at leisure, and spliced together at delivery the way a page assembles from static assets. The sensitive data gets injected at the splice point, never baked into the reusable layer. Same dynamic hole the web used, transposed to audio.\n\nWhat's left over is reasoning that genuinely can't be precomputed or structured. It's what frontier models are for, and it's a fraction of the work, and even its output can usually be consumed off the hot path: prefetched, masked behind speech, arriving while the conversation stays local. CPU in real time, GPU asynchronously.\n\nThe rule: local CPU by default, frontier GPU by exception.\n\nThis is not an anti-model position. It's the opposite. It treats frontier inference as the most valuable computation available, which is exactly why you don't waste it in a position where its queue time is your dead air.\n\nA deterministic path doesn't hallucinate, doesn't time out, doesn't emit malformed JSON, and doesn't wait in a frontier lab's queue.\n\nA word on the industry's proposed alternative, which is not to fix the cascade but to dissolve it, fold everything into one realtime speech-to-speech model.\n\nThe naturalness gains are real. Prosody, duplex conversation, genuine wins for expressive use cases, and I'd be lying if I said the demos weren't impressive.\n\nBut look at what dissolving the seams removes: every place structure could live. No text boundary means no splice point, so nothing is ever reusable. No redaction point, so everything the caller says goes into the model.\n\nThe meter runs on every millisecond, on GPU, by design. The cascade's supposed weakness is that it's stitched together, but it’s also what makes it fixable. The seams are the product.\n\nThe missing cache.\n\nHere's the strangest thing about this industry, and the fastest way to see how young its architecture is. This solution is about a lot more than caching, however caching is bizarrely underused.\n\nEvery expensive computation in computing's history got a cache put in front of it. Your CPU has three layers of them. Your DNS resolver answers a name once and serves it from memory until the TTL expires.\n\nDatabases build materialized views so the expensive query runs once and gets read forever. Code gets compiled once and executed a billion times — nobody has ever paid a per-run compile fee.\n\nThe web's entire modern shape is a cache.\n\nRender once, serve twenty thousand times for the price of bytes; HTTP has a status code, 304, whose whole meaning is \"you already have this answer, pay nothing.\" It's not even a computing pattern, really. A song is recorded once and streamed a billion times. A die is machined once and stamps a million parts.\n\nEvery expensive computation ever invented got a cache in front of it. Except this one. Voice AI runs the most expensive computation in history, across a range of modalities, millions of times a day, identically, at full price, with no reuse tier anywhere in the stack.\n\nAnd before anyone points at prompt caching: that's a cache on the seller's side of the meter, sold back to you as a discount. The cached tokens still get billed, and the generation still runs fresh every time.\n\nThe builders already know.\n\nProduction teams [hand-roll greeting caches](https://quantumautomations.ai/blog/tts-caching-voice-agents.html) and filler-phrase caches around their TTS providers; one healthcare deployment reports [90% cache hit rates](https://vexyl.ai/elevenlabs-tts-latency-test-2026-real-world-results/) on common phrases; there's an [open feature request](https://github.com/pipecat-ai/pipecat/issues/2629) on a leading framework asking, more or less politely, why there's no built-in way to reuse already-synthesized speech across calls.\n\nMany vendors whose synthesis is being cached publishes a how-to guide for building the workaround yourself. Everyone has admitted the layer should exist except the people who'd bill less if it did. But that's the next essay.\n\nThis isn't a thought experiment: we've been building it.\n\nA customer- and use-case-specific corpus of reusable components, scoped to their governance standards, generated ahead of time and spliced at delivery with the live data points — the appointment time, the name, the number — so that much of what an agent repeats can come from memory, and synthesis is saved for what's genuinely new.\n\nIt's live. Check the [Docs](https://docs.slng.ai/execution-layer).\n\nAnd when a turn genuinely needs a frontier model, the request fires early and the answer arrives while the agent is still talking — the GPU round trip happens behind the speech, never in the silence.\n\nRauch had a name for the web's version of this: negative latency. The answer exists before the question finishes. Voice's version works the same way, for the same reason.\n\nSo how much is reusable?\n\nDon't take our word for it. That healthcare team hand-rolling their own cache hits 90% on common phrases, and the teams pre-caching greetings measured the answer-rate difference in points, not milliseconds.\n\nWith the layer built in rather than bolted on, we've seen over 90% reuse across LLM and TTS in exceptionally tuned use cases, and most customers get to 50% across both in a few hours.\n\nA sh*tload, in other words. There is no polite number for buying the same answer 20,000 times a day.\n\nYou're not paying for intelligence. You're paying for amnesia.\n\nSo ask the question the way a buyer should ask it, holding the invoice:\n\nWhy do I buy the same answer 20,000 times a day?\n\nBecause the architecture refuses to remember.\n\nGet this right and three walls fall over, mostly for free.\n\n-\nLatency stops being a benchmark war and becomes physics you own. Nothing GPU-shaped in the hot path, nothing queued, nothing intercontinental — your UX stops depending on another company's peak traffic.\n\n-\nCost stops scaling with conversation length and starts scaling with the genuinely hard fraction of the work. The meter only runs on the exception. And notice the reliability that comes bundled in, because deterministic paths don't have bad days.\n\n-\nAnd the boundary problem — data locked inside hospital systems, bank cores, telco stacks, intelligence locked out in a handful of GPU regions — mostly dissolves, because an agent that is mostly structure doesn't need to cross the boundary.\n\nStructure runs anywhere, including inside the walls, next to the data that can't move. CPUs exist everywhere. H100s don't. The frontier gets consulted by exception, redacted questions out, answers back. The data stays.\n\nWrapping up\n\nI might be wrong about how fast this happens. I don't think I'm wrong about the direction, because the direction has a thirty-year undefeated record and voice would have to be the first medium in computing history to permanently exempt itself from it.\n\nThe industry's best engineers are streaming the wait into smaller and smaller pieces. Overlapping it, masking it, benchmarking the remains.\n\nAll of it is polishing the proverbial. The answer was never faster inference.\n\nIndex\n\nIn voice, latency is the product.\n\nThe entire user experience of a voice agent comes down to who controls the silence.\n\nThe industry is quietly admitting the point.\n\nIt's amnesia. Very expensive amnesia.\n\nMost of the agent is not a model.\n\nThis argument is older than the web browser.\n\nVoice Agents in 2026 is the web in 2015.\n\nSort your agent\n\nThe rule: local CPU by default, frontier GPU by exception.\n\nThe missing cache.\n\nThe web's entire modern shape is a cache.\n\nThe builders already know.\n\nThis isn't a thought experiment: we've been building it.\n\nSo how much is reusable?\n\nYou're not paying for intelligence. You're paying for amnesia.\n\nGet this right and three walls fall over, mostly for free.\n\nWrapping up\n\nOther posts", "url": "https://wpnews.pro/news/it-s-the-amnesia-stupid", "canonical_source": "https://slng.ai/posts/voice-ai-missing-cache-layer", "published_at": "2026-08-05 18:01:30+00:00", "updated_at": "2026-08-05 18:06:38.417330+00:00", "lang": "en", "topics": ["artificial-intelligence", "generative-ai", "ai-infrastructure", "ai-products"], "entities": ["Luke Miller"], "alternates": {"html": "https://wpnews.pro/news/it-s-the-amnesia-stupid", "markdown": "https://wpnews.pro/news/it-s-the-amnesia-stupid.md", "text": "https://wpnews.pro/news/it-s-the-amnesia-stupid.txt", "jsonld": "https://wpnews.pro/news/it-s-the-amnesia-stupid.jsonld"}}