Building a grounded LLM pipeline A developer building a grounded LLM pipeline for RoleReady's Company Research feature found that most engineering effort must go outside the prompt, with the model doing as little deterministic work as possible. The pipeline uses anti-hallucination as a cross-layer contract, a retrieval pipeline that does not trust the model with source identity, and a constrained markdown dialect the UI can rely on. I recently shipped an AI feature called Company Research inside RoleReady. It takes a saved job posting and produces a structured research brief about the company and role: three sections of labeled bullets, each backed by cited web sources. Building it taught me a single lesson worth writing down. In a production LLM feature, most of the engineering that matters happens outside the prompt, and the model’s job is to do as little of the deterministic work as possible. What follows is a writeup of the patterns that held up — anti-hallucination as a cross-layer contract, a retrieval pipeline that doesn’t trust the model with source identity, a discriminated resilience policy, tiered model routing across two providers, and a constrained markdown dialect the UI can rely on. I’ll speak in patterns, not file paths; the value is in the architecture. Background: what the feature does A user saves a job posting in RoleReady — title, company, location, raw description. They click Research. A worker runs an asynchronous pipeline: it plans retrieval queries, calls external search APIs, hydrates and deduplicates the results, assembles a prompt, calls a standard-tier LLM, parses the output, and writes a structured brief that the UI renders as a checklist. The brief has three sections. A Company Snapshot headquarters, industry, stage, funding, employee count, recent news . A Role Reality what the day-to-day looks like, what technologies are implied, what the team seems to own . And Risks & Unknowns layoff signals, funding stress, the things the model couldn’t verify . Each bullet is a - Label: value line. Each factual claim is followed by a superscript citation n referring to a numbered source. That last detail — the citation marker — is where the architecture starts to get interesting. The model is told to write Unknown , and the UI is designed around it Hallucination is the obvious failure mode for a feature like this. A research brief about a company the model has never heard of is the highest-risk surface for confident fabrication: the model knows what a generic Series-B SaaS company looks like and will happily emit those facts. A user acting on invented funding or layoff signals is the worst-case outcome. The standard remedy — adding “do not hallucinate” to the prompt — is necessary but not sufficient. The interesting work is making that instruction end-to-end , across four layers. The prompt opens with an accuracy block, not a format block. The first thing the model sees after the assembled context is a dedicated ACCURACY RULES critical : section: never invent information; if you don’t know, say so; derive answers only from explicitly stated facts in the job posting or the provided web sources; for unknown companies, summarize only what the posting explicitly states, and never infer a team or org from the tech stack. Order matters. “Don’t invent” is the first instruction, before any rendering contract. The format rules reinforce it. The format spec requires that if a fact is unknown, the model writes Unknown or Not stated instead of inventing it. The format spec isn’t separate from the accuracy spec — it expresses the same constraint at the rendering layer. The worker threads a failure flag into the prompt when retrieval produced nothing. When every retrieval path returns empty, the model isn’t left to guess whether the empty context is a bug. It’s told, in plain text, that no external sources are available and that it should generate the brief using only the job posting, explicitly flagging every unverifiable field. The parser treats Unknown as a first-class value. This is the part most teams miss. Downstream, the brief parser and the UI’s view-model recognize Unknown , Not stated , n/a , and unclear as real states with their own rendering rules. The UI surfaces “important” unknowns — Funding, Headquarters, Employee count — because that’s signal the user came here to find. It silently drops noisy unknown filler. The model is taught to admit ignorance and the UI is designed around the assumption that it will . A model that writes Unknown is only useful if every layer below treats Unknown as a real value rather than a parse failure. The model is not allowed to emit sources The most consequential design decision in the pipeline is also the most counterintuitive: the LLM’s return type contains only the markdown body. There is no sources field. The naive RAG pattern asks the model to return both prose and a structured sources: { url, title, snippet } list. The model is bad at this. It invents plausible-looking URLs that 404, truncates them, hallucinates titles, copies the wrong snippet. Every one of those bugs becomes a bug in your UI’s citation surface. The pipeline inverts the responsibility. TypeScript owns retrieval and source identity. The worker retrieves sources from external search APIs, numbers them deterministically, dedupes by URL, drops anything without a domain, and persists them in the same transaction that writes the brief. The model owns only the markdown-rendering contract. Its sole citation responsibility is to write a superscript n referencing a source the TS layer already numbered. It cannot introduce a new source, fabricate a URL, or rename one — because the return type doesn’t let it. The prompt reinforces this in prose. “Do not emit a separate structured sources list — sources are tracked separately by the caller.” This was a deliberate reversal. The return type used to have a sources field. An explicit audit pass removed it because the worker was already persisting TS-merged sources and the model-emitted list was never being read. Forcing the model to emit structured sources wasted tokens and risked hallucinated URLs. The model can’t fabricate URLs because it never has the chance . The general pattern: don’t trust the model with deterministic work. Anything the system can compute — the list of sources, source numbering, dedup, URL validity — should be computed by code, and the model should be restricted to the irreducibly generative part: writing prose that references those pre-computed identifiers. Multi-source retrieval, with a discriminated resilience policy The retrieval layer sits in front of paid external APIs and has to handle four distinct failure modes: transient rate limits, partial failures some queries error, some succeed , silent failures everything “succeeds” but returns nothing useful , and cached-stale failures a transient blip gets locked into a long-TTL cache . The interesting part isn’t any single mechanism. It’s that resilience is a policy, not a feature. “Retry on errors” and “swallow errors” are both wrong in the absolute. The right answer is per-error-class. Expected transient errors rate limits, API-level search failures are skippable. The planner catches the typed search error and continues to the next query, so one flaky query doesn’t kill the whole batch. Unexpected errors abort, timeout, network rethrow. The same planner explicitly does not swallow these. Silently masking real network failures hid legitimate problems in earlier iterations; the policy was inverted to surface them. Total-empty results get a dedicated flag. When every retrieval path returns nothing, the worker logs at error level not warn and threads the failure flag into the prompt so the brief self-flags unverifiable fields. Around that discriminated core, four targeted mechanisms harden the path. Single-retry 429 back-pressure. A shared helper retries exactly once on HTTP 429, honoring Retry-After delta-seconds or HTTP-date with a 30-second cap so a pathological Retry-After: 86400 can’t wedge the worker. It composes the per-attempt timeout with the caller’s outer budget using AbortSignal.any ... , so both fire — a 2026 idiom, where older codebases typically have only one or the other. A 40-second retrieval latency budget , threaded through every retrieval step. Bounds the entire retrieval chain instead of leaving two sequential calls unbounded. A quality-aware cache. A dedicated table holds per-domain retrieval results for seven days. The TTL drops to one hour when the entry was produced by a degraded path — the primary provider failed and a lower-quality fallback fired. A transient blip self-heals in an hour instead of stranding every job at that company on the fallback for a week. A concurrent-worker lock. When two workers research the same company simultaneously, a conditional UPDATE ... WHERE refreshStartedAt IS NULL OR < stale-cutoff ensures only one of them runs retrieval; the other reads the cached result. This arc — ship a blanket retry, discover it masks failures, replace it with a per-error-class policy — is the part I’d want any team shipping an LLM feature to internalize. Resilience isn’t a switch you flip on; it’s a policy you refine. Tiered models, two providers, no fallback chain The pipeline uses a tiered model abstraction. Three tiers — fast , standard , premium — each map to exactly one configured model. The mapping is the codebase’s own taxonomy: fast = extraction, standard = intelligence, premium = personalized. The brief function pins itself to the standard tier with a written justification in the prompt file: rich markdown brief generation is the documented standard-tier use case. The next engineer to touch the file doesn’t have to guess why this isn’t on the fast chain. Provider routing is abstracted behind the tier. Production runs on GCP Vertex AI as the primary, with OpenRouter as a configured alternative. The switch is environmental; the prompt layer never knows which vendor answered. Both providers expose the same frontier-grade standard-tier models, so swapping is an operational decision rather than a product one. Two non-obvious decisions are worth calling out. There is no model-level fallback chain. Each tier trusts its single configured model’s uptime, with a BAML-level retry policy two retries, exponential backoff, 300ms initial delay, 10-second cap handling transient failures. A fallback chain would make latency and behavior non-deterministic, which is the wrong trade for a feature whose entire value proposition is grounded and predictable output. Provider concerns stop at the env layer. The model never participates in deciding where it runs. Anything the system can handle deterministically — retry, backoff, provider swap — stays out of the prompt. A constrained markdown dialect, not JSON-mode Free-form LLM markdown is hostile to UI: the model decides what’s a heading, what’s a list, where emphasis goes, and you can’t render a consistent checklist from it. The alternative — locking the model to JSON — gives you structure but loses prose quality. Try writing a “Risks & Unknowns” section as a JSON object without sounding like a form. The pipeline takes a middle path: a narrow markdown dialect the model writes inside, a forgiving parser that extracts the canonical contract, and a strict validator that fails closed when the contract is violated. The dialect is small. Three fixed section headings — Company Snapshot, Role Reality, Risks & Unknowns. Key/value bullets in the exact form - Label: value , so the UI can render a consistent checklist. For clean-fact labels Headquarters, Industry, Employee count, Funding, Company stage the value must be the clean value only — no parenthetical qualifiers. "San Francisco, CA" , not "San Francisco, CA implied by filings " . Company-global facts stay in Snapshot; role-specific facts drawn from the posting stay in Role Reality, and the prompt forbids mixing them twice. A bounded ==highlight== marker the model can use for emphasis, capped at under eight per document and banned from headings, code, and links. The parser then post-processes what the model actually emitted: strips