# Designing LLM Inference Pipelines for GxP-Regulated Environments: An Architecture Reference > Source: > Published: 2026-10-04 09:26:01+00:00 *A reference pipeline for LLM inference where correctness under failure matters more than raw speed.* When an LLM pipeline handles clinical documents, “it worked” isn’t good enough, you need to know it’ll still be correct after a crash, a retry, or a duplicate message. This pipeline (SQS → Fargate → Amazon Bedrock, two dependent stages) is built around that one idea: nothing gets lost, nothing gets processed twice, and every result can be traced back to exactly what the model saw. Here’s how, with the code. Treat an LLM pipeline like a typical microservice and you optimize for the wrong thing. A synchronous API, parallel calls, aggressive retries all correct choices for a chatbot, **all wrong for a clinical document pipeline.** In a GxP regulated system, three failure modes matter more than response time: 2. A worker crashes between calling the model and writing the result, and the document is now neither pending nor processed, just gone. 3. Six months later, a QA reviewer asks what exactly the model saw and returned for a given document, and the answer depends on log lines that were never designed to survive that question. None of these show up in a demo. They show up during validation, or during an incident review, when it is too late to redesign around them. The rest of this article is about the specific choices that close each of these gaps, not about making inference faster. Two ECS Fargate workers, each consuming its own SQS queue, both calling Amazon Bedrock (Nova): There is no REST API anywhere in this system. Documents enter only through a ‘SendMessage’ call to the extraction queue. Classification never touches a document until extraction has durably committed its result for that ‘doc_id’. The pipeline is intentionally two dependent stages, not two independent services that happen to share a database. Both workers run on Fargate behind no load balancer and no public entry point, inside private isolated subnets reachable only through VPC endpoints. Scaling is driven by queue depth, since both workers spend most of their time waiting on a network call to Bedrock rather than consuming CPU. Every decision above buys safety by spending time. A document is not considered done after one model call and one write. It goes through two separate stages, each with its own database write and its own audit log entry, plus a queue hop in between before classification even starts. A synchronous version of this same pipeline, one API call in, one answer out, would be noticeably faster for any single document. That slowdown is accepted on purpose, for three concrete reasons. First, speed was never the requirement here. The actual requirement is that a result can be trusted as its usually its behind a complex workflow, reproduced, and explained later, and that only comes from durable writes and audit logs, both of which take time. Second, the time that is spent is deliberately sized, not wasted. The five minute visibility timeout and the autoscaling thresholds are set based on how long Bedrock actually takes, so the system is exactly as slow as it needs to be to avoid duplicate processing, and no slower. Third, this is a one time cost per document, not a cost that grows. Each document pays for two writes and two audit logs regardless of load, so throughput scales by adding more workers against queue depth, latency per document stays roughly constant. So the honest summary is this. A single document takes longer to finish here than it would in a simpler, faster pipeline. In exchange, you get a system where a crash never silently loses a document, a retry never silently duplicates one, and every finished result comes with proof of exactly what the model was given and what it returned. At this scale, a single shared queue per stage with per message idempotency is enough. At significantly higher volume, per document locking would need to move from an implicit property of the data model to an explicit mechanism, since concurrent overwrites of the same `doc_id` could reorder under load. The audit log currently lives only in CloudWatch. For a system approaching real validation, that log needs a defined retention and export policy decided up front, not added later once compliance asks for it. Pinning the model version solves reproducibility for a given deployment, but does not by itself solve the harder problem of proving the model’s behavior was validated against the version it is pinned to. That validation step lives outside this architecture and has to be tracked separately. Before building something similar, answer three questions for your own pipeline. What happens if this exact message gets processed twice? What happens if the process dies between the model call and the write? Can you prove what the model saw and returned, without that proof becoming a liability of its own? Answer those first. The latency numbers that come after are the ones that will actually hold up under review. Note: This project is a simplified version of a real production architecture I worked on. The scale, data, and infrastructure details below are rebuilt independently to be safe to share publicly, but the core design decisions are the same ones that mattered in the original system. Full code and architecture: [https://github.com/h-swathi-shenoy/clinical-doc-pipeline.git](https://github.com/h-swathi-shenoy/clinical-doc-pipeline.git) [Designing LLM Inference Pipelines for GxP-Regulated Environments: An Architecture Reference](https://blog.devgenius.io/designing-llm-inference-pipelines-for-gxp-regulated-environments-an-architecture-reference-4e7cdf50d823) was originally published in [Dev Genius](https://blog.devgenius.io) on Medium, where people are continuing the conversation by highlighting and responding to this story.