{"slug": "til-choosing-between-code-an-llm-call-and-an-ai-agent", "title": "TIL - Choosing Between Code, an LLM Call, and an AI Agent", "summary": "A developer exploring the definition of AI agents built a Go service with four endpoints of increasing complexity, from a stateless LLM call to a multi-tool agent loop. The project reveals that an agent is simply a loop that sends tool descriptions to the model and processes tool-call responses, and that most agent challenges—tool dispatch, idempotency, runaway loops—map to familiar distributed-systems problems like RPC, sagas, and circuit breakers.", "body_md": "Two questions sent me down this path.\n\n**Question one: what is an \"AI agent,\" really?** Most job posts mention them. I had not looked into it deeply, and from the outside I could not tell what it referred to. Is an agent a different endpoint? A different model? A library we install? Or is it just a name for calling an LLM API in some particular way?\n\n**Question two came later, and it stopped me cold.** Once I understood the mechanics and sat down to pick something to build, I could not come up with a single idea that I could not also write as ordinary Go: a function that calls some APIs, hits a database, and returns a result. If plain code can do it, why pay a model to do it worse?\n\nThis is what I found. There is a real answer to both, and the second one has a trap in it.\n\nCode: [github.com/Arkoes07/llm](https://github.com/Arkoes07/llm)\n\nOne Go service, one interface, four endpoints of increasing complexity:\n\n```\nPOST /chat/no-memory        one stateless LLM call\nPOST /chat                  multi-turn, history per session\nPOST /chat/agent/weather    tool-calling loop, one tool\nPOST /chat/agent/log-triage tool-calling loop, three tools\n```\n\nNo framework. Hand-rolled `net/http`\n\nfirst, then a second implementation behind the same interface using an SDK, so I could see exactly what the SDK was doing for me. (Answer: it removed boilerplate, not thinking. The loop logic was identical in both). Worth noting: Groq's official SDKs are Python and JS only, so in Go we are on community libraries or an OpenAI-compatible client.\n\nStarting at a plain call and walking up to an agent one step at a time turned out to be the whole answer to question one.\n\nHere is the version with no vocabulary.\n\n**A plain LLM call** is a stateless HTTP request. We send messages, we get text back. The model remembers nothing. \"Conversation history\" is state we store and resend every time. That is `/chat/no-memory`\n\nand `/chat`\n\nabove, and the only difference between them is who keeps the array.\n\n**An agent** adds one thing: we also send a list of tools, described as JSON schemas. Now the model can respond with \"call `get_metrics`\n\nwith `{\"service\":\"checkout\"}`\n\n\" instead of a final answer. So we loop:\n\nThat is the difference. Same endpoint, same model, about 150 lines of extra Go. Every agent framework is this loop plus state persistence, retries, and tracing.\n\nWhat I realized while building it is that most of the hard parts are things we already know under different names:\n\n| Agent problem | What it actually is |\n|---|---|\n| Tool dispatch + schema validation | RPC dispatch |\n| Model calls the same tool twice | Idempotency |\n| Multi-step work that must not half-complete | Saga / compensation |\n| Context window overflow | Bounded cache with an eviction policy |\n| Untrusted text in the prompt issuing instructions | An authz boundary |\n| Runaway loop | Circuit breaker |\n| \"Did this actually work?\" | Regression testing, for non-deterministic output |\n\nThere is a reason the table lines up like that, and the loop hides it.\n\nFrom inside the code it looks like one API call repeated. But we call the model, the model asks for a tool, we call that tool (in a real system, another service: a logging backend, a metrics API, a runbook store), then we call the model again. One request fans out across several independent services, in an order nobody fixed in advance. That is orchestration, and we already have a name for it: a distributed system.\n\nThe difference is which part we cannot trust. Normally the orchestrator is our own code, the most reliable thing in the system, and we save our suspicion for the dependencies. In an agent, the orchestrator is the model. The component deciding what to call next is now the least reliable one: it picks a different sequence on identical input, it can emit a malformed request, and when it fails it usually fails plausibly instead of loudly.\n\nSo the instincts still apply, we just point them somewhere new. Validate every argument. Make writes idempotent, because the caller may repeat them. Bound the loop. Treat anything a tool returns as data, never as instructions. What we already do at the edge of a system, applied to the orchestration we are used to trusting.\n\nWhat is actually new is a short list: prompting, structured output, evals, and retrieval.\n\nQuestion one, answered. Which is when question two showed up.\n\nI needed something to build. And every idea I had, I could immediately picture as normal Go.\n\n\"Generate a study plan\" is one LLM call with a structured output, wrapped in a function. No loop. \"Classify support tickets\" is one call, then a `switch`\n\non the result. \"Summarize a document\" is one call. Every time I reached for the loop, plain code was cheaper, faster, deterministic, and testable.\n\nThat is not a small objection. It is the whole question. The loop costs us money (tokens), latency (seconds per iteration), determinism (same input, different output), and debuggability (behavior lives in prose, not code). What are we buying with all that?\n\nExactly one thing: **the model chooses the next step at runtime, using information that did not exist when we wrote the code.**\n\nNot intelligence, and not language. Those come from a single call. The loop buys runtime branch selection and nothing else.\n\nSo the question becomes: when do we actually need that?\n\nThe common framing is \"simple problems go to code, complex problems go to an agent.\" Too vague to act on. Here is the version I would use:\n\nCan we draw the flowchart before we see the input?\n\nIf yes, the decision points are enumerable and we can write the branches, so we should write them. Code is deterministic, testable, debuggable, and roughly free. An agent there is pure overhead, and the LLM's job, if it has one, is a single call inside one box: classify, extract, summarize. No loop.\n\nIf no, meaning the path depends on what each step reveals and the real tree has thousands of branches we would only discover in the moment, that is the narrow case where the loop earns its cost.\n\nThat criterion is why I picked **log triage**. Consider:\n\n```\nif latencyHigh {\n    checkDatabase()\n    if dbSlow {\n        checkConnectionPool()\n    } else {\n        checkDownstream()  // which one? there are 30\n    }\n}\n```\n\nWe can write this. But which downstream service? Depends on what the logs said. What if the DB check reveals a deploy changed a query plan? We did not have a branch for that. The order we investigate depends on what each step returns.\n\nThat is the trigger. Not \"it is complicated,\" but **\"I cannot enumerate the branches at code-writing time.\"**\n\nKeep this criterion in mind. The rest of the post is about what happens when we pass it and then slowly undo our own answer.\n\nA fake incident with a deliberate lie in it:\n\n`checkout`\n\np99 spiked at 14:03. But checkout is fine. Its DB pool is healthy (4/20) and traffic is normal. It calls`payment`\n\n, whose`max_pool_size`\n\nwas quietly lowered 20 → 5 by a config reload at13:58. Payment saturated its pool and started timing out. Checkout is the victim, not the cause.\n\nThree tools over hardcoded data: `query_logs`\n\n, `get_metrics`\n\n, `search_runbook`\n\n. Getting it right means ruling out checkout's own resources, hopping to the downstream service, and (the trap) widening the time window past the incident to catch that 13:58 config change.\n\nMocked data is a feature, not a shortcut. Every failure is then unambiguously the loop's, not flaky infrastructure.\n\n```\niteration 1: query_logs(checkout, 13:55-14:05)\niteration 2: get_metrics(checkout, latency)\niteration 3: get_metrics(checkout, error_rate)\niteration 4: get_metrics(payment, latency)\niteration 5: get_metrics(checkout, request_rate)\niteration 6: query_logs(payment, 13:55-14:05)\niteration 7: get_metrics(payment, db_pool)\niteration 8: get_metrics(checkout, db_pool)\n```\n\nNote what is missing: `search_runbook`\n\n, never called. Now the answer it produced:\n\n...These symptoms match the known runbook pattern for \"Database connection-pool exhaustion\".\n\nRemediation (per the runbook)\n\n- Restore / increase the DB pool size for the payment service\n- Restart the payment service to clear lingering connection-wait states\n\nThe root cause was correct. The provenance was invented. It attributed general knowledge to an internal document it never opened, and step 2 was not in my runbook at all.\n\nNothing in the output looked wrong. Plausible content, authoritative formatting, correct conclusion. **The only way to catch it was diffing the answer against the trace.** Now imagine a real runbook saying \"do NOT restart, drain connections first, page the DBA.\"\n\n**Fix (system prompt):** \"Only cite the runbook if you called search_runbook. Otherwise say the recommendation is based on general knowledge.\"\n\nThe lesson: **an agent can reach a correct answer through invalid reasoning and give us no signal that it did.** That is the argument for evals, and it is checkable. \"Every source cited must appear in the tool trace\" is an assertion we can write.\n\n```\niteration 1: query_logs(checkout, 13:55-14:05)\niteration 2: get_metrics(checkout, error_rate)\niteration 3: search_runbook(\"payment.authorize timeout\")\n```\n\nThree calls, then it concluded: tune the client timeout, add a circuit breaker. Plausible, and wrong. It never checked whether checkout's own DB was healthy or whether traffic was abnormal. It gathered evidence *for* its first hypothesis instead of testing alternatives. A human had to type \"continue, check the downstream.\"\n\n**Fix (system prompt):** \"Do not stop at the first plausible explanation. Before concluding, rule out alternatives: check whether the service's own resources are healthy and whether traffic is abnormal.\"\n\n```\niteration 6: search_runbook(\"payment.Authorize timeout\")   ← searched here\niteration 7: get_metrics(payment, latency)\niteration 8: query_logs(payment, 13:55-14:05)              ← found the cause here\n```\n\nIt searched using the symptom it had at iteration 6, found the actual root cause at iteration 8, and never searched again. A human does this reflexively: it is pool exhaustion, so pull up that runbook. **The loop does not re-plan by default.**\n\n**Fix (system prompt):** \"After identifying a root cause, search the runbook again using the root-cause terms, not the original symptom.\"\n\n```\n{\"error\":{\"code\":\"tool_use_failed\",\"failed_generation\":\n\"<function=query_logs{\\\"service\\\": \\\"checkout\\\"}</function>\"}}\n```\n\nRight tool, right arguments, malformed syntax. It is missing one `>`\n\nafter the function name. A gRPC client cannot do this; the wire format is generated and guaranteed. Here the caller's ability to form a valid request is itself probabilistic.\n\nIt also broke my retry logic:\n\n```\nattempt 1: <function=query_logs{...\nattempt 2: <function=query_logs{...   ← identical\nattempt 3: <function=query_logs={...  ← one char different\nattempt 4: <function=query_logs{...   ← identical again\n```\n\nFour attempts, 15 seconds of backoff, the same malformed output. **Retry only helps if the output would differ.** At low temperature the model is near-deterministic, so an identical request produces an identical failure.\n\n**Fix (code, then model):** drop `tool_use_failed`\n\nfrom the retryable set, so we fail fast instead of burning 15 seconds. Then switch models. I did not see it again on `openai/gpt-oss-120b`\n\n.\n\n```\nRate limit reached ... tokens per minute (TPM): Limit 8000, Used 6886, Requested 1311\n```\n\nEvery iteration resends the entire history, so iteration 8 carries all seven prior tool results. Eight iterations, under a minute, into a hard wall. Not because any single call was large, but because they compound. Log `prompt_tokens`\n\nper iteration; that curve is the most honest thing we can show about agent economics.\n\n**Fix (code):** exponential backoff on 429, honoring the `Retry-After`\n\nheader rather than our own curve. This is a mitigation, not a fix. The growth is inherent to the loop.\n\nLook back at the labels on those fixes. Two were code, and code fixes are just engineering: better retry classification, better backoff. Nothing interesting there.\n\nThe other three were system prompt. And those three are not the same species.\n\n**Type A: guardrails that improve judgment.**\n\nBetter tool descriptions. \"Rule out alternatives before concluding.\" \"Cite the runbook only if you called it.\" These do not shrink the option space, they help the model navigate the same space better. Nearly free. Keep them, add more.\n\n**Type B: guardrails that encode the flowchart.**\n\n\"Always call `query_logs`\n\nbefore `get_metrics`\n\n.\" \"After identifying a root cause, search the runbook again.\" These remove options. Each one is a branch, written in English instead of Go.\n\nThe test is mechanical:\n\nIf we can state the guardrail as a deterministic rule, we can write it as code. And if we can write it as code, it does not belong in the prompt.\n\nFix #3 is mine, and it is Type B. \"After identifying a root cause, search again with the root-cause terms\" is a sequencing rule. I could enforce it in code: if the loop is about to terminate and `search_runbook`\n\nwas never called with the root-cause terms, force one more iteration. Deterministic, testable, free. Instead I wrote a branch in English and paid a model to interpret it.\n\n**That is the paradox.** Remember what the loop buys us: the model choosing at runtime. Every Type B guardrail takes some of that back. We are paying full price for a capability we are actively suppressing. Push far enough and we have a complete flowchart written in English, executed by a probabilistic interpreter, at a hundred times the cost of an `if`\n\n, with no type checking, no test coverage, and no way to diff a behavior change in code review.\n\nWe did not build an agent. We built the world's most expensive `switch`\n\nstatement and moved it out of version control.\n\nAnd notice the shape of it. I passed the flowchart criterion honestly (log triage really is not enumerable) and then spent three fixes quietly re-enumerating it anyway.\n\nWhich means the two ideas in this post were never separate. The criterion tells us whether the flowchart exists before we start. The guardrails tell us whether we were right, after. Same spine, checked from both ends. If we find ourselves writing the flowchart in English one rule at a time, the criterion already had the answer.\n\n**The number of Type B guardrails we need is a diagnostic on whether we picked the right tool.**\n\nOne caveat: how many we need is not a fixed property of the problem. Compare the two traces above. Finding #2, the three-call run that stopped at checkout, was `llama-3.3-70b-versatile`\n\n. Finding #1, the eight-call run that ruled out checkout's pool and traffic, hopped to payment, and widened the window to 13:55 on its own, was `openai/gpt-oss-120b`\n\n. Same tools, same fake world, same prompt. The only change was the model string.\n\nI only ran each model once, so treat this as a signal and not a benchmark. But the swing matters: **the same code was a supervised tool on one model and an autonomous one on another.** A guardrail we need this quarter may be dead weight next quarter.\n\n(Related: the Llama model I started on was deprecated on Groq partway through this project. Model availability is a dependency that disappears).\n\nThree tools, not two. Most of the confusion in this space comes from collapsing the middle one.\n\n**1. Pure code, no LLM at all.**\n\nThe flowchart is drawable and every step is mechanical. Parse, transform, query, branch. If we can specify it, we should specify it. This is still most software.\n\n**2. A single LLM call inside code we control.**\n\nThe flowchart is drawable, but one box in it needs judgment over unstructured input: classify this ticket, extract these fields, summarize this text, draft this reply. One call, structured output, and our code decides everything else. This is the overwhelming majority of real \"AI features,\" and it is not a lesser agent. It is the correct architecture for an enumerable problem.\n\n**3. The agent loop.**\n\nWe genuinely cannot draw the flowchart, because step three depends on what step two returned and there are too many possibilities to enumerate. Incident triage, code review, open-ended research. Here the loop reaches things branching would not, and we pay for it in tokens, latency, and non-determinism.\n\nThe line between 2 and 3 is the flowchart criterion. The line between 2 and 1 is just whether any single step needs judgment over unstructured input.\n\nAnd the sweet spot inside option 3, which is what the guardrail paradox is really about:\n\nConstrain in code everything that can be enumerated. Leave the loop wrapping only the irreducible judgment.\n\nThe agent should be as small as possible. That is the principle I would take into the next one, and it follows from the paradox rather than from measurement.\n\nThe observation underneath it is smaller and more concrete. Every idea I started with turned out to be a code problem in disguise. That is what question two was. I could not find a use case that plain Go would not handle until I went looking for one deliberately, and even then the loop only survived because I stopped short of writing the last three rules in code.\n\nTwo things I could not settle from this build.\n\n**Where the crossover actually sits.** I can say the loop earns its cost when the flowchart is not drawable, but I cannot yet say what that costs in dollars and milliseconds compared to the deterministic version of the same task. I have the argument and not the numbers.\n\n**When to stop adding Type B guardrails and rewrite in code.** The diagnostic says a rising count means we picked wrong, but there is no threshold. Two Type B rules is clearly fine. Fifteen is clearly a flowchart. I do not know where the line is, and I suspect it depends on how much the remaining judgment step is worth.\n\nAnswering either one properly would mean building the same task twice, deterministic and agentic, and measuring cost, latency, correctness, and the inputs the deterministic version cannot handle at all. I have not done that, so for now both stay open.\n\nIf you have hit different failure modes running agents in production, I would like to hear them.", "url": "https://wpnews.pro/news/til-choosing-between-code-an-llm-call-and-an-ai-agent", "canonical_source": "https://dev.to/arkoesalwi/til-choosing-between-code-an-llm-call-and-an-ai-agent-3n8c", "published_at": "2026-07-27 02:16:42+00:00", "updated_at": "2026-07-27 02:28:51.303117+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "ai-agents", "developer-tools"], "entities": ["Groq"], "alternates": {"html": "https://wpnews.pro/news/til-choosing-between-code-an-llm-call-and-an-ai-agent", "markdown": "https://wpnews.pro/news/til-choosing-between-code-an-llm-call-and-an-ai-agent.md", "text": "https://wpnews.pro/news/til-choosing-between-code-an-llm-call-and-an-ai-agent.txt", "jsonld": "https://wpnews.pro/news/til-choosing-between-code-an-llm-call-and-an-ai-agent.jsonld"}}