A Guide to Saving Token Usage with Multi-Agent AI

A new guide outlines four strategies for reducing token usage in multi-agent AI systems: static instruction caching, semantic caching, just-in-time tooling, and task escalation. The guide, published by an unnamed source, emphasizes that scaling multi-agent architectures does not have to proportionally increase costs if these techniques are properly implemented.

A Guide to Saving Token Usage with Multi-Agent AI Scaling up and streamlining a multi-agent architecture doesn't necessarily entail escalated costs if you know how to properly implement these four strategies for saving token usage. Introduction When multiple AI agents are strung together to cooperate and address complex workflows, the sheer volume of tokens — text elements or units, so to speak — may easily escalate. Everything adds up: from memory logs to detailed tool specifications, system instructions, and so on. Eventually, this leads to dragged down speed of executions and computing budget exhaustion. Consequently, managing token usage is vital for today's AI developers and practitioners as a whole. There's good news, though: scaling up and streamlining a multi-agent architecture doesn't necessarily entail equal scaling of costs if you know how to properly implement some strategies for saving token usage. This article introduces and shows four of them in action. Four Key Strategies for Saving Token Usage Below are four commonly adopted best practices to streamline multi-agent AI solutions while optimizing the use of tokens. Strategies for saving token usage in multi-agent AI systems // 1. Using Static Instruction Caching Prefix-Match Caching Consider this one as a "don't repeat yourself" rule. Large language models LLMs , an indispensable part of modern AI agents, invest much processing energy re-reading the same system prompts in successive turns. Prefix caching, which consists of storing key-value pairs, helps address this issue by storing such static, long instructions as a reference guide prepared ahead of time. Rather than having the model re-read the whole "how-to-act-as-this-agent" instruction manual every time, the model bookmarks a summarized state upon receiving a query. In subsequent turns, the model only needs to open that bookmark and go straight to processing the new prompt. As a result, latency due to preparation is significantly cut down, and so are the associated token costs. // 2. Using Semantic Caching: Intent-Based Recall If an AI agent has already solved a specific problem before, why ask it to generate a brand new response from scratch? This strategy leverages embeddings — numerical, vector-based representations of text that "retain" semantic properties — and uses them to quickly identify similar past intents. For instance, two distinct users' prompts like " How can I reset my router? " and " What are the steps to restart my wifi box? " would be recognized as the same intent based on semantic caching. In certain cases, this opens up the opportunity of entirely bypassing the LLM while still providing the right answer. // 3. Using Just-in-Time Tooling Also known as lazy loading, this technique is designed to tackle a frequent pitfall in AI agent building: front-loading context windows with huge "reference manuals" of every single API, tool, and database schema within their reach. Of course, that would be the perfect recipe for bloated, noisy prompts and excessive token consumption. Instead, why not give the agent a high-level, lean directory of its capabilities? Only when the agent identifies a specific task required at a given moment does it trigger the fetching of the detailed, fine-grained instructions and parameters needed for that specific tool. // 4. Using Task Escalation: Cost-Efficient Model Routing Not all user prompts require a massive, heavy-hitting model to be properly addressed. Effective multi-agent AI architectures are designed to act as triage centers, endowed with a routing layer that analyzes every incoming task based on its nature and complexity. Accordingly, simpler tasks like formatting data, summarizing text, or classifying intent are routed to lightweight, often free models capable of successfully running these tasks locally. Meanwhile, heavy, compute-intensive models where token consumption matters are "reserved" exclusively for complex tasks like those requiring deep reasoning or orchestration across multiple steps. Implementation Example in a Nutshell Now that we've covered four practical strategies that can help optimize token usage in multi-agent AI applications, how about illustrating how some of them work through a high-level example? This code snippet illustrates how to combine two of them: semantic caching and model routing. The code uses an actual model — a sentence transformer — to convert text into the embeddings needed for semantic caching. The calls to LLMs are mocked, but you can easily replace the code especially for the lightweight model part with an actual, free-weights model like those available at Groq , as shown in this article https://machinelearningmastery.com/building-an-end-to-end-sentiment-analysis-pipeline-with-scikit-llm/ , for instance. python import numpy as np from sentence transformers import SentenceTransformer Loading a free, local model to convert text into embeddings embedder = SentenceTransformer 'all-MiniLM-L6-v2' In-memory semantic cache and similarity threshold 0.90 = 90% similar semantic cache = {} SIMILARITY THRESHOLD = 0.90 def cosine similarity vec1, vec2 : """Calculates how closely related two queries are.""" return np.dot vec1, vec2 / np.linalg.norm vec1 np.linalg.norm vec2 def route and respond user query : 1. Converting the current query into an embedding vector query vector = embedder.encode user query 2. Semantic Caching: Check if a similar problem was solved recently for cached vector, past response in semantic cache.values : if cosine similarity query vector, cached vector = SIMILARITY THRESHOLD: return f" Served from Cache {past response}" 3. Model Routing: Triage the task based on complexity Simple tasks get routed to a free, locally hosted model e.g. Llama 3 via Ollama This routing logic is illustrative-only; not be used in production if "summarize" in user query.lower or len user query < 100: response = call free local agent user query else: Complex multi-step reasoning escalates to a larger orchestration agent response = call heavy reasoning agent user query 4. Save the new vector and response to our cache for future users semantic cache user query = query vector, response return response --- Mocking Agent Functions for Illustration: no actual LLMs invoked here --- def call free local agent prompt : return "Action completed by local, zero-cost model." def call heavy reasoning agent prompt : return "Action completed by complex orchestration agent." Example Usage: mocking the alternate use of different agents/models Comment/uncomment to try both examples and try your own print route and respond "Summarize today's server logs" print route and respond "Draft an optimal one-month itinerary for my upcoming Japan trip. Take into consideration the set of documents, public transport timetables and other documents provided, along with real-time API information" The complexity of the task requested in the prompt passed to route and respond will determine which model type is used. The output of executing this code will be either one of the two return messages in these functions: python def call free local agent prompt : return "Action completed by local, zero-cost model." def call heavy reasoning agent prompt : return "Action completed by complex orchestration agent." Wrapping Up This article described four key strategies to be aware of when implementing multi-agent AI applications and architectures, with emphasis on optimizing token usage and reducing costs and latency. Through a practical, mock-based example, we reinforced our understanding of applying two of them in combination: semantic caching and model routing. is a leader, writer, speaker, and adviser in AI, machine learning, deep learning & LLMs. He trains and guides others in harnessing AI in the real world. Iván Palomares Carrascosa https://www.linkedin.com/in/ivanpc/