Beyond the Chatbot: Building Production AI Systems on AWS

A developer detailed the architecture of production AI systems on AWS, emphasizing that the core challenge is building reliable systems around LLM API calls rather than making the calls themselves. The post maps common problems to AWS services such as Amazon Bedrock, S3, and OpenSearch, and outlines strategies for handling failures like timeouts and hallucinations, including retries, fallback models, and guardrails.

AI apps have moved past simple chat boxes. Today's AI systems need agents, tools, memory, data, security, monitoring, and scale. The hard part is not calling an LLM API. The hard part is building a reliable system around that API call. A demo is simple: php flowchart LR A Prompt -- B Model -- C Response A real production system looks very different: php flowchart TD U User -- API API API -- APP Application Layer APP -- ORCH AI Orchestration ORCH -- LLM LLM ORCH -- TOOLS Tools ORCH -- RAG RAG ORCH -- MEM Memory ORCH -- GUARD Guardrails ORCH -- DATA Data + Infrastructure DATA -- OBS Observability Each box matters. If you skip Guardrails , bad input can hijack your system. If you skip Memory , every message re-explains itself and costs more tokens. If you skip Observability , you won't know why the system failed until a user tells you. The rest of this article walks through each box. Instead of listing AWS services, let's match each one to a real problem. | Problem | AWS Service | Why | |---|---|---| | Need a foundation model | Amazon Bedrock | Managed access to multiple LLMs, no infra to run | | Store documents and files | S3 | Cheap, durable, scales easily | | Store app data | RDS / Aurora / DynamoDB | Structured data, users, sessions, transactions | | Search by meaning retrieval | OpenSearch / pgvector | Vector search for RAG | | Run code | Lambda / ECS | Serverless or container compute for your app logic | | Handle async work | SQS / EventBridge | Queue jobs, decouple slow tasks, avoid lost requests | | Watch the system | CloudWatch | Logs, metrics, alarms | | Keep it secure | IAM / Secrets Manager | Access control and safe storage of keys | flowchart LR subgraph Compute L Lambda / ECS end subgraph Data S3 S3 DB RDS / DynamoDB VEC OpenSearch / pgvector end subgraph AI BR Bedrock end subgraph Ops CW CloudWatch SEC IAM / Secrets Manager end L -- BR L -- S3 L -- DB L -- VEC L -- CW L -- SEC An agent doesn't just answer — it plans, calls tools, and acts in steps . sequenceDiagram participant U as User participant A as Agent participant T as Tool participant M as Memory U- A: Ask a question A- M: Load context A- A: Plan next step A- T: Call tool T-- A: Tool result A- A: Decide: done or retry? A-- U: Final answer This changes the design in a few key ways: RAG Retrieval-Augmented Generation has a full pipeline, not just one step: php flowchart TD D Documents -- I Ingestion I -- C Chunking C -- E Embeddings E -- V Vector Store V -- R Retrieval R -- RR Reranking RR -- LLM LLM LLM -- RES Response Things that break in production: This is where most "demo-only" AI systems fail. A production system must handle: | Failure | Fix | |---|---| | Model timeout | Retry with backoff, set a timeout limit | | API rate limit | Queue requests, add backpressure | | Hallucination | Add a validation/guardrail step, don't trust blindly | | Duplicate job runs | Use idempotency keys | | Queue failures | Dead-letter queues, alerts | | Full outage | Fallback model or cached response | php flowchart LR REQ Request -- TRY{Call Model} TRY -- |Success| OK Return Response TRY -- |Timeout/Error| RETRY Retry with Backoff RETRY -- |Still Failing| FALLBACK Fallback Model / Cached Response FALLBACK -- OK Normal app monitoring is not enough. A slow API call is easy to see. A wrong but confident answer is not. Track these: php flowchart LR USER User Input -- FILTER Input Guardrail FILTER -- MODEL LLM DOC Retrieved Document -- FILTER2 Content Guardrail FILTER2 -- MODEL MODEL -- OUT Output Guardrail OUT -- RESPONSE Safe Response Production AI has two cost buckets: The design choices you make affect both. For example: Here is a full production AI system, combining everything above: php flowchart TD U User -- API API Gateway API -- APP Application Layer - Lambda/ECS APP -- ORCH AI Orchestration ORCH -- BR Bedrock - LLM ORCH -- AGENT Agent + Tools ORCH -- RAGF RAG Pipeline ORCH -- MEMD DynamoDB - Memory/State ORCH -- GUARDF Guardrails RAGF -- S3D S3 - Documents RAGF -- VEC OpenSearch/pgvector ORCH -- QUEUE SQS/EventBridge - Async Jobs QUEUE -- WORKER Background Worker APP -- DBD RDS/Aurora - App Data ORCH -- CWD CloudWatch - Observability APP -- SECD IAM/Secrets Manager - Security Building an AI app is no longer just connecting to an LLM. The real engineering work starts when the system needs to be reliable, observable, secure, scalable, and affordable . This space is still changing fast — new agent frameworks, new observability tools, and new AWS features arrive often. The core idea will stay the same: the model is a small part of the system. The rest is real engineering.