The Agentic Loop: Three loops in a trench coat

Building autonomous AI agents involves three nested loops: an inference loop for LLM chat completions, a tool loop for executing tool calls, and a third loop for managing state and context. The author argues that oversimplifying agent loops as a single loop misses the complexity required for production systems.

The Agentic Loop: Three loops in a trench coat The building blocks for autonomous agents aren't as simple as they seem. Agent loops are often oversimplified. They’re presented as a single loop, when really it’s three loops in a trench coat that make up an “agentic” experience for a customer. I’m here to write yes, I wrote this, insane right? yet-another-blog about agent loops. The example code blocks are also pseudo-code and for illustrating these ideas. Also I’ve omitted streaming, which complicates the post but the shape of these stays the same. The Inference Loop The most reductive way to explain Large Language Models is that they take text, and predict the next charact... err... tokens. Unlike your ex, they really do complete your sentences. This is achieved with an inference loop. Your inference loop has three responsibilities: Make chat completion API calls infer the next words Pass a tool usage request to your tool loop more later Manage chat history persistence of tool results, or more user messages When you’re building an agent, the first “outer” loop you’re creating is the inference loop. This is the loop that sends a system prompt, user and assistant messages, and available tools to an LLM’s “chat completion” endpoint. Anthropic has one https://platform.claude.com/docs/en/api/go/beta/messages/create . OpenAI has one https://developers.openai.com/api/reference/resources/chat/subresources/completions/methods/create . OpenRouter makes all of them easy to use https://openrouter.ai/docs/api/api-reference/chat/create-a-chat-completion . Once an LLM returns its messages, it’s your responsibility to append them to a “chat history” so that a conversation can be continued. This can be an array, a database table, Redis keys, whatever. Your prerogative. chatMessages = { role: "system", content: "You are a couples therapist. You help people either make it work, or make them realize it will never work. The user you're speaking with is named Tom." }, { role: "user", content: "I kinda miss Laney, what should I write to her?" } continueInferenceLoop = true while continueInferenceLoop inferred = aiClient.completeChat messages: chatMessages chatMessages.push { role: "assistant", content: inferred.message.content, toolCalls: inferred.toolCalls, } if inferred.toolCalls.length == 0 continueInferenceLoop = false else Handle tools see more later end end finalResult = chatMessages.last.content puts "Assistant responded with: {finalResult}" The API design of most Large Language Model providers is a stateless design. This means the model provider has no idea what your previous conversation was with it. You need to provide the entire conversation every time. This is why you see the warning of “Large conversations use tokens faster” — yes I typed an em-dash — because you are sending the tome of messages you’ve built up about whether you should reach out to your ex. Don’t do it, Tom. The Tool Loop LLMs are brains in a jar. They provide no functional value on their own. The tools you give an LLM are what make it an agent. When you tell a model “here are the tools you have” in your outer inference loop, the model may try to “use” them in its inference response . This is the same thing as a brain sending an electrical signal telling your index finger to hover over the enter key of the email you desperately want to send Laney. Tom, we need to set boundaries my man. The separate tool definitions you include in your API request are usually serialized into the system prompt field of the token stream the model processes. And it may infer the usage of multiple tools in one turn. Hence: Tool Loop . chatMessages = { role: "system", content: "You are a couples therapist. You help people either make it work, or make them realize it will never work. The user you're speaking with is named Tom." }, { role: "user", content: "I kinda miss Laney, what should I write to her?" } tools = { type: "function", function: { name: "send ex girlfriend an email", parameters: { type: "object", properties: { email message: {type: "string"} }, required: "email message" } } } continueInferenceLoop = true while continueInferenceLoop inferred = aiClient.completeChat messages: chatMessages, tools: tools chatMessages.push { role: "assistant", content: inferred.message.content, toolCalls: inferred.toolCalls, } if inferred.toolCalls.length == 0 continueInferenceLoop = false else Behold... the Tool Loop while tool = inferred.toolCalls.shift Do tool things in here like sending an email You might have a switch/case statement to control which tool based on the name. Tool calls also have an ID toolResult = "...see next section..." Tool result shapes vary by an provider's API chatMessages.push { role: "user", content: { type: "tool result", toolCallId: tool.toolCallId, content: toolResult } } end end end Your tool loop must look up the tool the model inferred it should use, and call your own function with the parameters the model provided as well. But keep in mind a few things: Because tool calls are also inferred text - it means a tool name, or function parameters, can be hallucinated. You should be defensive to these bogus tool calls with something like Tool Not Found: "call my ex girlfriend" The API responds with a tool call id or tool use id if you’re using Anthropic . This ID is used for the API & Model to correlate calls with responses. If you send a subsequent completion request without the tool call’s ID, the API will error.For some providers, tool results don’t have error codes or error states - The resulting content is the error. Use XML tags to denote an error like <tool call error Phone number blocked</tool call error Anthropic does have an is error field, though. The Human Loop I debated the name of this loop. This loop doesn’t need to exist, really. It also doesn’t need to be a human. But I believe AI should benefit humans, so I’m sticking with it. If you’d like to be a contrarian you could also call it “The Safety Loop” or “The Sanity Loop” – both would work. Assuming you like control over agentic implementations, you’ll need to implement your third loop to approve/deny a tool, or steer the model another direction. I also must concede that this “loop” is not a programmatic loop. It is more of a blocking function call, with someone or something looping on their own whether or not to proceed with the tool call. The loop is not in the bounds of your code, it resides within the bounds of the approver. while tool = inferred.toolCalls.shift result = "" case tool.name when "send ex girlfriend an email" Block this loop. Maybe this sends an SMS, Email, or displays a button with the decision. decision = wait for approval tool.parameters if decision.approved? send email tool.parameters "email message" result = "Email sent" elsif decision.denied with instructions? result = "<tool call denied with instructions {decision.instructions} </tool call denied with instructions " else result = "<tool call error Tool call was not approved</tool call error " end else result = "<tool call error The tool ' {tool.name}' was not found</tool call error " end chatMessages.push { role: "user", content: { type: "tool result", toolCallId: tool.toolCallId, content: result } } end The Human Loop is arguably the hardest part to implement in agentic systems. You can’t have a piece of code block for hours. What if the server restarts? What if you have thousands of other requests coming in you need to respond to? The first two loops inference and tool are simple enough. The human loop ups the ante of difficulty. This is why durable execution frameworks exist, like Temporal. But the human loop is necessary, because it’s the only thing stopping Tom from actually sending that message to Laney. IT WAS TWO YEARS AGO TOM, MOVE ON Looping Back Putting it all together: Inference Loop - Calls a chat completion API, and delegates tool calls to your.... Tool Loop - Which handles the tool usage request the model is attempting to make and hands off approval to your... Human Loop - to ask for approval, or a new direction. The result of this propagates back the tool’s result. These three loops are the building blocks of agentic systems. They are used for RAG, progressive discovery, automatic tool approval checks, and more. Now go build one