Turn one of a reorder conversation on FundlyMart looks nothing like turn six.
Turn one is a pharmacist typing something like “crocin 20 strips.” Small message, small context, cheap to serve: 1,140 tokens, system prompt and tool schemas included. By turn six, after an order-history fetch, a stock check across a handful of SKUs, one ambiguity-resolution round trip, and a cart build, the same conversation is carrying 4,780 tokens into the model.
The task didn’t get harder. The pharmacist still wants twenty strips of Crocin. What changed is everything sitting behind the message that the model never asked for and can’t ignore.
I’ve been staring at this shape of problem for months, building FundlyMart, a WhatsApp B2B pharmacy commerce bot, at Ardour Analytics. So when Business Insider reported that Google slipped Gemini 3.5 Pro’s general availability from June to July over, among other things, token-efficiency concerns surfaced by early testers on agentic tasks, I didn’t read it as a surprising story. I read it as a frontier lab running into a wall I hit at a much smaller scale a while ago.
The instinct, when an agent gets expensive or slow, is to blame the model. Wrong reasoning, wrong tool choice, not smart enough. In my experience that’s rarely where the problem lives. The problem lives in what you’re re-sending on every single turn.
A tool-calling loop built on the Vercel AI SDK’s generateText with tools doesn't get one shot at the context window. It gets one shot per iteration, and every iteration re-sends the full accumulated state. Take a real FundlyMart flow: a retailer asks to reorder "same as last time." The agent has to fetch order history, resolve any ambiguous SKUs, check current stock, and build a cart before it can respond. That's four to six tool calls minimum, and each one hands its raw output back to the model so it can decide the next step.
Here’s what’s actually stacking up across those calls:
None of these individually looks expensive. A tool output here, a clarification there. What makes it a real cost problem is that they’re additive and non-decaying. Nothing falls out of context on its own. By the time a six-step reorder flow reaches its last call, the model isn’t reasoning over “what does this pharmacist want.” It’s reasoning over that plus a full transcript of how it figured out what the pharmacist wanted last time it asked.
That distinction, content growth versus reasoning growth, is the entire story. The model’s job stayed the same size. The bill for doing that job did not.
I didn’t fix this by making prompts shorter or switching models. The fixes that held up were architectural, not stylistic.
The first was separating what the LLM needs to reason over from what the system needs to persist. WhatsApp’s 24-hour session window forced this early: conversation state is ephemeral and session-scoped, cart state is persistent and user-scoped, and they don’t share a context. That split turned out to matter for tokens too. Cart state lives in a database, not in a message the model has to re-read every turn. The agent asks for exactly the slice it needs (current cart contents) instead of carrying the running history of how the cart got built.
The second was refusing to let raw tool output become the default payload shape. A fetch_order_history call returns full product metadata: category trees, image URLs, timestamps, internal flags, none of which the next reasoning step touches. I strip that down to SKU, quantity, and last unit price before it goes back into context. That took an average order-history payload from around 640 tokens to about 190. Whatever the mechanism, the principle was the same: a tool result answers a question, it isn't a transcript of the database row it came from.
The number that actually tells you whether this worked isn’t the total token count. It’s the shape of the growth curve across a conversation. On FundlyMart, growth isn’t linear. It front-loads hard at the order-history and stock-check calls, then flattens once the cart is built and confirmation is the only step left. Even after trimming, tool output still accounts for a little under 60% of total context by the last turn of a six-step flow. That’s the metric I’d trust over a single flashy percentage, because token-per-turn growth is what determines whether a bot degrades gracefully at turn ten or falls over.
None of this is WhatsApp-specific, and it isn’t small-team-specific either.
Google previewed Gemini 3.5 Pro at I/O on May 19, with Sundar Pichai telling the audience it would ship “next month.” June came and went. According to Business Insider’s reporting, picked up widely since, Google pushed general availability into July while gathering more feedback from early testers on its Antigravity platform and the LMArena benchmarking site, and while folding in lessons from the earlier Gemini 3.5 Flash release, where some users reported the model burning through tokens faster than expected on long prompts and extended workflows. The reporting points to three linked reasons for the slip: token-efficiency concerns, coding performance not yet at flagship standard, and long-horizon multi-step reasoning falling short of the bar Google set at I/O. As of this week, Gemini 3.5 Pro still has no confirmed public GA date.
I want to be careful here, because it’s tempting to read this as “Google’s model isn’t smart enough yet,” and that’s not what’s being reported. The gaps are specifically in long-horizon, multi-step, agentic execution, the exact category of task where context doesn’t reset between steps and every prior action’s output has to be carried forward. That’s not an intelligence gap. That’s the same accumulation problem I hit building a WhatsApp reorder flow, playing out at a scale where the cost of getting it wrong is a delayed frontier launch instead of a slower bot response.
One piece of context makes this concrete: enterprise teams evaluating frontier models this year have started treating token efficiency as a procurement criterion in its own right, separate from raw benchmark scores or context window size. A model that burns more tokens to reach the same answer is simply a more expensive model to run at scale, regardless of how capable it is per token. Google tuning cost profile before shipping a flagship isn’t a sign of a weak model. It’s a sign the lab is measuring the thing that actually determines whether agentic deployment is viable: spend per completed task, not IQ per parameter.
If there’s one thing I’d want another engineer building agents to take from this, it’s to stop treating context growth as a byproduct of the tool loop and start treating it as the thing you’re actually engineering. Model upgrades get the attention because they’re easy to announce. Context discipline doesn’t get a launch post. But the failure mode is the same whether you’re a final-year intern shipping a WhatsApp bot or a frontier lab shipping a flagship model. The agent’s job stays constant in size, and everything else, history, tool output, retries, re-injected state, keeps growing unless something is actively stopping it. Model IQ was never the ceiling. Uncontrolled accumulation was.
Gemini 3.5 Pro will ship eventually, and it’ll probably be a good model. Whether it’s a good agent depends on whether Google solved the same problem I had to solve at a much smaller scale: making sure the tenth turn of a conversation isn’t paying for the memory of the first nine.
Tags: LLM, Agentic AI, Machine Learning, Software Engineering, AI
Token Bloat, Not Model IQ was originally published in Towards AI on Medium, where people are continuing the conversation by highlighting and responding to this story.