smolagents replays its whole memory every step: the O(n ) token bill nobody mentions

Hugging Face's smolagents framework incurs a quadratic token cost because its CodeAgent re-sends the entire accumulated memory of every previous step at each reasoning step, rather than sending only new information. An audit of the code reveals that the write_memory_to_messages method rebuilds the full message list from scratch each step, leading to an overpay factor of up to 9.5× for the default 20-step budget. The fix involves a single hook to cap or summarize the growing memory.

Cost-audit series, episode 5. This series began with an AI agent that burned 136M tokens overnight →. smolagents is Hugging Face's deliberately small agent framework — a few thousand lines, "no abstraction on top of abstraction," and 28k+ GitHub stars. Its CodeAgent is genuinely elegant: the model writes Python, the runtime executes it, the result comes back, repeat until a final answer. The elegance hides a cost curve. A smolagents run that takes n reasoning steps does not cost n times a single step. On input tokens it costs closer to n²/2 , because every step re-sends the entire accumulated memory of every previous step . The default step budget is 20. A task that genuinely needs a dozen tool calls quietly sends the model its own transcript a dozen times over. This audit shows the exact lines, gives you a formula you can evaluate on your own workload, and shows the one-hook fix. Every step, the agent rebuilds the full message list it sends to the model. Here is the method that does it, verbatim agents.py:758-770, v1.26.0 https://github.com/huggingface/smolagents/blob/v1.26.0/src/smolagents/agents.py L758-L770 : php def write memory to messages self, summary mode: bool = False, - list ChatMessage : """ Reads past llm outputs, actions, and observations or errors from the memory into a series of messages that can be used as input to the LLM. Adds a number of keywords such as PLAN, error, etc to help the LLM. """ messages = self.memory.system prompt.to messages summary mode=summary mode for memory step in self.memory.steps: messages.extend memory step.to messages summary mode=summary mode return messages Read the loop: it walks every entry in self.memory.steps and appends its messages. memory.steps only ever grows — it is a plain list initialised empty and appended to, never trimmed, except by an explicit reset between runs memory.py:230 https://github.com/huggingface/smolagents/blob/v1.26.0/src/smolagents/memory.py L230 , memory.py:232-234 That method is called at the top of every action step , with no summary mode , so the full history is replayed each time agents.py:1284-1286 https://github.com/huggingface/smolagents/blob/v1.26.0/src/smolagents/agents.py L1284-L1286 : memory messages = self.write memory to messages ... input messages = memory messages.copy The default ceiling on how many times this can happen is 20 agents.py:300, max steps: int = 20 https://github.com/huggingface/smolagents/blob/v1.26.0/src/smolagents/agents.py L300 . Nothing in the default path caps or summarises the growing memory — summary mode=True is used only for agents.py:684 agents.py:886 Let: P = tokens in the system prompt fixed, sent every step s = tokens each completed step At step k 1-indexed the input the model receives is P + k-1 ·s — the prompt plus everything the previous k-1 steps left behind. Summed over an n -step run, cumulative input tokens are: Σ k=1..n P + k-1 ·s = n·P + s · n n-1 /2 The s · n n-1 /2 term is quadratic in n . Compare it to the intuition most people price with — " n steps ≈ n × one step" = n· P + s . The history you re-pay for is n n-1 /2 · s instead of n · s : Steps n | History replays × s , naive | History replays × s , actual | Overpay factor | |---|---|---|---| | 4 | 4 | 6 | 1.5× | | 8 | 8 | 28 | 3.5× | | 12 | 12 | 66 | 5.5× | | 20 max | 20 | 190 | 9.5× | Table is illustrative of the formula above — it is the closed form n n-1 /2 vs n, not a measured run. Plug in your own P and s to get dollars. The observation size s is where it bites hardest. If a tool returns a chunk of a web page, a file, or a dataframe, that payload is now re-sent on every subsequent step for the rest of the run. Long, tool-heavy tasks are exactly the ones that hit max steps , so the worst tasks pay the worst multiplier. Good news: smolagents already measures this for you. Every ActionStep carries a token usage field with input/output token counts memory.py:63 https://github.com/huggingface/smolagents/blob/v1.26.0/src/smolagents/memory.py L63 . After a run, sum step.token usage.input tokens across agent.memory.steps and you will see the curve directly. The problem is that by the time you read it, you have already paid.smolagents gives you the exact hook you need. The agent accepts step callbacks — callables invoked at the end of each step, and you can register them per step-type agents.py:282 https://github.com/huggingface/smolagents/blob/v1.26.0/src/smolagents/agents.py L282 , agents.py:304 setup step callbacks , agents.py:416-425 memory.steps is just a list you own, a callback can cap how much history survives into the next write memory to messages call: python from smolagents import CodeAgent, ActionStep KEEP LAST = 6 replay only the most recent N action steps def trim memory step, agent : action steps = s for s in agent.memory.steps if isinstance s, ActionStep for stale in action steps :-KEEP LAST : collapse the bulky observation; keep a short marker so the model still knows the step happened stale.observations = " trimmed to control context cost " agent = CodeAgent tools= ... , model=..., step callbacks= trim memory Illustrative usage of the real step callbacks API — tune KEEP LAST and what you collapse to your task. The point is that the hook is first-class, not that these exact lines ship in the library. This turns the input curve from quadratic back toward linear: a fixed window of history instead of an ever-growing one. You trade some long-range recall for a bounded bill — for most tool-loop tasks that is the right trade, and you make it deliberately instead of discovering it on an invoice. Other levers, in order of bluntness: lower max steps from the default 20 so a wandering run can't rack up 190× history replays; truncate large tool return values before they enter memory; and use the planning/summary path smolagents already has for long-horizon tasks. The pattern in this series is always the same: the framework is fine, the default is expensive, and the cost is invisible until it shows up on the invoice. smolagents is the most honest case yet — it even hands you token usage — but you still have to run the task, at full quadratic cost, to see it. That is the gap tokenscope https://www.npmjs.com/package/@wartzar-bee/tokenscope closes — it shows what a run actually cost, and estimates a source tree's token footprint npx @wartzar-bee/tokenscope --demo Then run it on your own most-recent Claude Code session just npx @wartzar-bee/tokenscope , or estimate a directory's token cost before a run — the static check that powers the guardrail: npx @wartzar-bee/tokenscope scan --dir . If you want that check enforced automatically — a bot that comments the predicted token-cost delta on the responsible files in every pull request and can block a regression — that is what we build the ci-guardrail GitHub Action https://github.com/wartzar-bee/ci-guardrail for. Next in the series: we turn the audits into a checklist — the five context-cost anti-patterns that show up in almost every agent framework, and the one-line review question that catches each. Follow @wartzarbee so you don't miss it. Found an error in this audit? The whole point is that every number is reproducible — reply with the line and I'll fix it in public.