# Stanford and Berkeley Researchers Build TokenGate to Cut LLM Prefill Costs > Source: > Published: 2026-10-26 10:38:39+00:00 *A new method out of Stanford and UC Berkeley promises to make the most expensive part of running long-context AI, the first few seconds before a model says anything, up to 11 times faster without hurting accuracy.* Prefill is the part nobody talks about at dinner parties, but it's where the money goes. Before a large language model generates a single word of output, it has to read and process every token you fed it, the whole document, the whole chat history, every tool schema an agent might call. That reading pass is called prefill, and for long-context, agent-heavy applications it can eat more compute than the actual answer. A team from Stanford and UC Berkeley just published a fix for it, and the numbers are hard to ignore. The paper, titled "TokenGate: Adapting LLMs for Fast and Memory-Efficient Prefill with Learned Token Pruning," appeared on arXiv on October 25 under the identifier 2610.26754. The authors are Liana Patel, Rishabh Ranjan, Ion Stoica, Matei Zaharia, and Carlos Guestrin, a lineup that matters: Stoica and Zaharia co-founded Databricks, and Guestrin has spent years building systems research at the intersection of machine learning and infrastructure. This isn't a side project. It's a systems paper from people who've built the infrastructure other people's AI products run on. According to the paper and a write-up from MarkTechPost, TokenGate delivers 1.4x to 11.3x faster time-to-first-token and 1.5x to 3.4x lower peak memory during prefill, with no accuracy loss, across a range of tasks. Compared with the best existing token-pruning methods, it still comes out ahead: up to 6.4x faster prefill and 2x lower peak memory. The core idea borrows from Mixture-of-Experts models. TokenGate inserts a small, trainable "gate" partway through the network. Below the gate, every token gets processed normally. At the gate, each token receives an importance score, and only the top-scoring ones get carried forward through the remaining layers. The trick is making that selection process differentiable end to end, since picking a fixed top-K set of tokens is normally a hard, non-trainable operation. TokenGate solves that by re-weighting attention logits with the importance scores themselves, letting gradients flow through the whole pipeline during training. The timing is not an accident. Context windows have stretched from a few thousand tokens to hundreds of thousands, and the applications pushing hardest on that ceiling, retrieval-augmented generation and multi-step AI agents, are exactly the ones where prefill dominates the bill. A ReAct-style agent chaining ten tool calls can burn 800,000 input tokens to produce 500 tokens of actual output, because every single call drags along the full system prompt, every tool schema, and the entire conversation history up to that point. Each of those calls triggers its own prefill pass, and at tens of thousands of tokens a pass, that adds up in both latency and dollars, on every single step, not once per conversation. [Hobbyists Are Squeezing Trillion-Parameter AI Models Onto Laptops](https://startupfortune.com/hobbyists-are-squeezing-trillion-parameter-ai-models-onto-laptops/) DeepSeek V4 Pro, a 1.6 trillion parameter model, ran on a 128GB MacBook M5 Max via SSD streaming this year, managing about 2 tokens per second, while H100 rental prices climbed toward $3.28 an hour. - [how to run trillion parameter models on laptop](https://startupfortune.com/hobbyists-are-squeezing-trillion-parameter-ai-models-onto-laptops/) - [running large language models on gaming PC hardware](https://startupfortune.com/hobbyists-are-squeezing-trillion-parameter-ai-models-onto-laptops/) That cost structure is also why long-context prompting and RAG keep getting weighed against each other instead of one simply winning. Dumping a full one-million-token knowledge base into context instead of retrieving a handful of relevant chunks can cost roughly 300 times more per request at current frontier pricing, which is why most teams still reach for RAG once a knowledge base crosses a few hundred thousand tokens. TokenGate doesn't eliminate that tradeoff, but it moves the line. If prefill gets cheaper and faster, the corpus size at which long context stops making economic sense gets pushed further out, and agent frameworks that already pay the prefill tax on every tool call get real relief. TokenGate also isn't the only approach chasing this problem, and it's worth being honest about where it sits. Speculative decoding attacks a different stage, the token-by-token generation that follows prefill, by having a small draft model propose tokens for a larger model to verify. KV cache compression shrinks the memory footprint of attention keys and values so a model can hold more context without running out of GPU memory, but it doesn't necessarily speed up the initial read-through. TokenGate is squarely aimed at that first pass, the one agentic and RAG-heavy workloads hit over and over. The paper notes it works as a lightweight adaptation on top of existing transformer architectures, including Qwen3, and hybrid-attention models like Qwen3.5, which means labs wouldn't need to retrain a model from scratch to bolt it on. Whether OpenAI, Anthropic, or Google adopt something like it is an open question the paper doesn't answer, and frankly, academic speedup numbers and production deployment are two different animals. But the direction is the one every infra team serving long-context, tool-using models is already straining toward: cut the cost of reading before you even start writing. The code for TokenGate is public on GitHub under the token-gate-llms organization, which means the industry gets to test that gap between benchmark and production itself. **Also read:** [A hobbyist project called CrowdGPT wants volunteers to train an AI model at home](https://startupfortune.com/a-hobbyist-project-called-crowdgpt-wants-volunteers-to-train-an-ai-model-at-home/) • [Grok Releases Seven Minutes of Its AI-Made Odyssey Series](https://startupfortune.com/grok-releases-seven-minutes-of-its-ai-made-odyssey-series/) • [Nvidia Floats Charging $2,000 a Year Per GPU for Its AI Agent Software](https://startupfortune.com/nvidia-floats-charging-2000-a-year-per-gpu-for-its-ai-agent-software/) *This article is posted in [AI News](https://startupfortune.com/category/ai/), check it out for more related stories.* ## Join the discussion [Open in the community →](https://startupfortune.com/community/) Almost there. Sign in and your reply posts straight away.