{"slug": "agentic-search-architecture-five-patterns-that-survive-production", "title": "'Agentic Search Architecture: Five Patterns That Survive Production'", "summary": "A developer outlined five production patterns for agentic search architectures, arguing the control loop around the reasoning model — typed tools, context isolation, budgets, and an evaluator that can send work back — matters more than the model itself. The writeup reports an 8.7-point nDCG@10 gain from query formulation and iteration with the embedding model held constant, at a cost of 107.4 seconds per query versus 0.67 seconds and 764.1K input tokens per question. It also recommends caching at the normalized sub-query level plus index version, rather than at the answer level, to keep costs manageable.", "body_md": "**Summary**: The reasoning loop is the cheap part. What separates a working agentic search system from a demo is the control loop around it: typed tools, context isolation, budgets, and an evaluator that can send work back.\n\nClassic retrieval is one pass: embed the query, fetch top-k, rerank, answer. Agentic search adds a decision layer — the system decides what to search, how many times, and whether the evidence it got is good enough to answer.\n\nThat decision layer buys generalisation. On out-of-domain tasks, fixed retrieval pipelines degrade while the same agentic loop stays competitive across two different leaderboards. It also buys a specific, measurable quality gain: the 8.7-point nDCG at 10 improvement holds with the embedding model held constant, which means the gain comes from query formulation and iteration, not from a better encoder.\n\nThe price is visible in the same measurement: 107.4 seconds per query against 0.67 seconds, and 764.1K input tokens with 5.8K output tokens. Two orders of magnitude in latency and a five-figure token bill per question is not a rounding error — it changes what the system can be used for, and it is the reason architecture matters more than the model choice.\n\n| Pattern | Shape | Best for | Fails when | \n|---|---|---|---|\n| Plan-and-Execute | One planning step produces typed sub-queries, then parallel execution | Multi-part questions, dashboards, compliance checks | Sub-queries depend on each other's results | \n| ReAct loop | Think-act-observe repeated until the answer is supported | Exploratory research, unknown corpora | Latency budget under five seconds | \n| Hierarchical workers | Host decomposes, managers aggregate, workers search in parallel with isolated context | Large corpora, long sessions | Orchestration overhead exceeds retrieval time | \n| Retrieval-as-tool | The model calls search, filters, and SQL as typed functions | Mixed structured and unstructured data | Tool schemas drift from the index schema | \n| Evaluator-gated | A checker scores evidence, rejects and re-plans | High-stakes answers, regulated domains | The evaluator shares the planner's blind spots | \n\nThe first pattern is the workhorse. When a question decomposes into independent parts, planning once and executing in parallel keeps latency near a small multiple of standard retrieval instead of letting it grow with the number of reasoning steps.\n\nReAct is better when you genuinely do not know what to look for. It is also the pattern most likely to blow a latency budget, because each observation costs a model call.\n\nHierarchical designs exist for a boring reason: context saturation. If every sub-query's results land in one window, the later reasoning degrades. Isolating worker context and aggregating at a manager layer keeps the final synthesis clean, and parallelism absorbs part of the added cost.\n\nRetrieval-as-tool is what most teams actually ship, because it lets the same agent answer \"what is the policy\" and \"how many tickets breached SLA last month\" without two pipelines. The failure mode is schema drift: the tool description says one thing, the index says another, and the agent silently searches the wrong field.\n\nEvaluator-gated designs add a second model that scores whether the retrieved evidence supports an answer. Useful in regulated settings, but only if the evaluator is trained or prompted on a different failure distribution than the planner — otherwise both miss the same thing.\n\nOne more pattern deserves a name because it is what actually keeps cost sane: caching at the sub-query level rather than the answer level. Two questions that share a party name and a date range can reuse the same retrieval result if the cache key is the normalised sub-query plus the index version. Answer-level caching fails because the final answer differs on every phrasing; sub-query-level caching hits often enough to matter, and it is invalidated cleanly when the index changes.\n\nThe counter-argument is that cached retrieval makes the loop lazy. It does, in the sense that the planner reuses a prior result instead of reformulating — which is the behaviour you want, because reformulating a working sub-query burns tokens for no gain.\n\nPoint three is where most teams under-invest, and it is cheap. A prompt that says \"be efficient\" is not a budget; a counter that terminates the loop is.\n\nA representative case: an enterprise contract repository, 1,400,000 documents, average query a legal question with two or three conditions. The first agentic version used a ReAct loop and hit a 9-second p95 with a token spend that made the finance team ask questions.\n\nWe replaced it with Plan-and-Execute. One planning call produced three to five typed sub-queries (party, clause type, date range, jurisdiction), executed in parallel against the same hybrid index, then a single synthesis step. p95 fell to 3.4 seconds, and nDCG at 10 improved because the sub-queries split the conditions instead of hoping one embedding captured all of them.\n\nTwo things we did not expect. First, the planner needed a hard cap of five sub-queries — beyond that, recall stopped improving and latency grew linearly. Second, the biggest win came from the evidence ledger, not from the loop: once answers cited passage IDs, the legal reviewers' correction rate dropped, because they could check the exact clause instead of re-reading the answer.\n\nA note on tool schemas, because this is the failure that hides longest. When a tool is described in prose (\"search contracts by counterparty\"), the model will pass a free-text counterparty and the index will match it against the wrong field — a party name against a document title, say. Define the tool with an enum of index fields and a validated type per field, and let the model choose among them. Prose tools produce plausible queries against the wrong fields, which is worse than a failed call, because the evidence looks relevant.\n\nA last structural point about evaluation: score the loop on task success and evidence coverage, not only on ranking metrics. A system that returns a beautifully ranked set of passages which do not jointly answer the question has improved nDCG and failed the task. Track both, and treat a rise in ranking quality with a flat task success rate as a sign that the planner is producing sub-queries the index can rank but cannot satisfy.\n\n**Q:** Do we need an agent at all?\n\n**A:** No, if one retrieval pass answers the question. Measure the share of queries that need a second sub-query; below 15%, a better reranker is cheaper.\n\n**Q:** Which pattern first?\n\n**A:** Plan-and-Execute with typed tools. It is the one that scales across query types with predictable latency.\n\n**Q:** How do we cap cost?\n\n**A:** Hard token and time budgets in the orchestrator, plus a per-tenant monthly ceiling with a graceful degradation path.\n\n**Q:** Is context isolation worth the complexity?\n\n**A:** Yes above roughly 20 retrieved passages per session; below that, a single window with a good summariser is fine.\n\n**Q:** How do we evaluate the loop?\n\n**A:** Fixed query set with graded relevance, nDCG at 10 for ranking quality, plus task success and per-step latency for the loop.\n\n**Q:** What about security?\n\n**A:** Tool-level authorisation, not prompt-level. The agent should be unable to call a tool the user's role cannot use.\n\nTengence Search ships agentic retrieval as a governed capability: typed tools, hard budgets, an evidence ledger and per-step observability, so the loop's quality gain survives production latency targets.", "url": "https://wpnews.pro/news/agentic-search-architecture-five-patterns-that-survive-production", "canonical_source": "https://dev.to/tengence/agentic-search-architecture-five-patterns-that-survive-production-o1l", "published_at": "2026-11-15 08:39:29+00:00", "updated_at": "2026-11-15 08:50:36.549866+00:00", "lang": "en", "topics": ["ai-agents", "ai-search", "ai-tools", "artificial-intelligence"], "entities": [], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/agentic-search-architecture-five-patterns-that-survive-production", "markdown": "https://wpnews.pro/news/agentic-search-architecture-five-patterns-that-survive-production.md", "text": "https://wpnews.pro/news/agentic-search-architecture-five-patterns-that-survive-production.txt", "jsonld": "https://wpnews.pro/news/agentic-search-architecture-five-patterns-that-survive-production.jsonld"}}