{"slug": "lessons-from-building-production-agentic-systems-what-actually-matters-beyond", "title": "Lessons from building production agentic systems — what actually matters beyond the LLM", "summary": "A practitioner with 18 months of experience building production agentic systems reports that deterministic guardrails, structured tool interfaces, and trajectory evals matter more than the LLM itself, and that a task with 90% per-step reliability across 5 chained steps lands around 59% end-to-end. The author, who used hosted APIs and open models including Llama 3.x and Qwen 2.5, found that fine-tuning helped less than better tool design and that single well-instrumented agents beat multi-agent prototypes in most cases.", "body_md": "I’ve spent the last year and a half moving projects from “chatbot with a system prompt” to actual agentic systems — ones that plan, call tools, and complete multi-step tasks without a human babysitting every turn. Most tutorials make this look like a weekend project. It isn’t. Here’s the architecture and the failure modes nobody warns you about.\n\n**What “agentic” actually requires**\n\nStrip away the hype and an agent is four components in a loop:\n\n**A reasoning core** — the LLM itself. We’ve used both hosted APIs and open models (Llama 3.x and Qwen 2.5 fine-tunes served via TGI/vLLM). For tool-heavy workloads, a smaller model with good function-calling accuracy beats a bigger model that hallucinates arguments. Benchmark on *your* tool schemas, not leaderboards.\n**Tool interface layer** — where 70% of real engineering lives. Every tool needs a strict schema, input validation, timeouts, and a machine-readable error format the model can recover from. Our biggest early mistake: returning raw stack traces as tool output. The model would “apologize” and retry the identical broken call forever. Now every tool returns structured errors with a `retryable`\n\nflag and a hint field, and loop behavior improved dramatically.\n**Planning + control flow** — we started with free-form ReAct-style loops. They work in demos and drift in production. What stabilized things: an explicit state machine around the model. The LLM proposes actions; deterministic code decides what’s allowed from the current state. Think of the model as a proposal engine, not the executor. Max-iteration caps and budget ceilings are non-negotiable — an agent that can loop is an agent that *will* loop.\n**Memory** — three separate problems people conflate: (a) in-context working memory (summarize aggressively — after ~15 tool calls, transcripts poison the context), (b) episodic memory across sessions (we use embedding retrieval over past task outcomes), (c) long-term structured state (just use a database; not everything needs vectors).\n\n**The eval problem**\n\nThis is where most agent projects quietly die. Single-turn evals tell you almost nothing about multi-step behavior. What we landed on:\n\n**Trajectory evals**: record full action sequences on a fixed task suite, score terminal state (“did the refund actually get issued?”), not the politeness of the final message.\n**Regression traps**: every production failure becomes a permanent test case.\n**LLM-as-judge for intermediate steps**, but only after we calibrated the judge against human labels — uncalibrated judges gave us confident garbage.\n\nRough numbers from our experience: a task with 90% per-step reliability across 5 chained steps lands around 59% end-to-end. That math is the whole reason agent architecture matters. You claw reliability back through validation layers, retries with modified prompts, and checkpoints where a human can intervene — not through prompt wizardry.\n\n**Things I’d tell past-me**\n\n- Start with\n**one narrow workflow**, fully instrumented, before generalizing. “General-purpose agent” as v1 is how projects die.\n- Deterministic guardrails > prompt instructions. If an action is dangerous, gate it in code. “Please don’t delete records” in a system prompt is a wish, not a control.\n- Log every model call with full context. Agent debugging without traces is archaeology.\n- Human-in-the-loop isn’t a compromise — for anything irreversible (payments, data mutation, external comms), approval gates are what make deployment survivable.\n- Fine-tuning helped us far less than better tool design. We assumed we’d need custom models; mostly we needed better schemas and error messages.\n\n**Where I’m still unsure**\n\nMulti-agent setups. We’ve experimented with planner/executor splits and reviewer agents, and honestly the added coordination overhead ate most of the gains for our use cases. Single well-instrumented agent + good tools has beaten our multi-agent prototypes almost every time. But I see strong claims in the other direction from people doing research-style workloads.\n\nSo, question for this community: **for those running agents in production — has anyone found a multi-agent architecture that consistently beats a single-agent loop on reliability, not just on benchmark tasks? And what’s your eval setup for multi-step trajectories?** Genuinely curious what’s working outside my bubble.", "url": "https://wpnews.pro/news/lessons-from-building-production-agentic-systems-what-actually-matters-beyond", "canonical_source": "https://discuss.huggingface.co/t/lessons-from-building-production-agentic-systems-what-actually-matters-beyond-the-llm/178656#post_1", "published_at": "2026-08-14 07:22:21+00:00", "updated_at": "2026-08-14 07:42:43.432727+00:00", "lang": "en", "topics": ["artificial-intelligence", "ai-agents", "ai-infrastructure", "ai-research"], "entities": ["Llama 3.x", "Qwen 2.5", "TGI", "vLLM"], "alternates": {"html": "https://wpnews.pro/news/lessons-from-building-production-agentic-systems-what-actually-matters-beyond", "markdown": "https://wpnews.pro/news/lessons-from-building-production-agentic-systems-what-actually-matters-beyond.md", "text": "https://wpnews.pro/news/lessons-from-building-production-agentic-systems-what-actually-matters-beyond.txt", "jsonld": "https://wpnews.pro/news/lessons-from-building-production-agentic-systems-what-actually-matters-beyond.jsonld"}}