{"slug": "what-actually-happens-when-you-build-a-production-rag-chatbot-on-amazon-bedrock", "title": "What Actually Happens When You Build a Production RAG Chatbot on Amazon Bedrock", "summary": "A developer built a production RAG document Q&A chatbot on Amazon Bedrock using NestJS, PostgreSQL with pgvector, hybrid search with reciprocal rank fusion, prompt caching, and Claude, and documented the architecture and the failure modes it addresses. The writeup argues that most RAG degradation is silent — stale documents, mismatched error codes, and ineffective caching — and that retrieval architecture, not prompt engineering, determines success. The system splits ingestion and query paths, using Cohere embeddings with distinct input types for documents and queries and a two-level chunker with header injection.", "body_md": "*How we built a document Q&A system with NestJS, PostgreSQL, pgvector, Hybrid Search (RRF), Prompt Caching, and Claude on Amazon Bedrock.*\n\nBuilding a demo chatbot is easy.\n\nBuilding one that consistently answers correctly after weeks of document updates, changing requirements, and real user traffic is where things become interesting.\n\nMost Retrieval-Augmented Generation (RAG) failures don't look like failures.\n\nThe API returns 200.\n\nThe model responds.\n\nInfrastructure dashboards stay green.\n\nYet answer quality gradually gets worse.\n\nYou ask about error code `E-4021` and receive documentation for `E-4012`.\n\nA recently updated document never appears in search results.\n\nPrompt caching is enabled, but costs don't go down.\n\nHybrid search is implemented, but retrieval quality barely improves.\n\nNothing crashes.\n\nNothing raises an alert.\n\nThe chatbot simply becomes less useful over time.\n\nFor teams involved in **[application software development](https://relipa.global/app-development-services-company/)**, the real work begins after the first successful response. Retrieval quality, chunking strategy, embedding consistency, prompt caching, document lifecycle management, and hallucination control quickly become more important than model selection itself.\n\nThis article walks through a complete production-oriented RAG implementation built with:\n\nMore importantly, it explains why each component exists, what problem it solves, and which production issues it prevents.\n\n**RAG (Retrieval-Augmented Generation)** does not ask the LLM directly. Before each question, the system:\n\nThe consequence of this design:\n\nThe problem \"how does the LLM know my data\" becomes a **search** problem. If retrieval is weak, the chatbot is weak, because no prompt can produce information that isn't in the context.\n\nThis distinction is easy to underestimate.\n\nMany teams spend weeks experimenting with prompts while retrieval quality remains unchanged.\n\nIn reality, most enterprise application software development projects that incorporate AI succeed or fail based on retrieval architecture rather than prompt engineering.\n\nThat is why most of this article is about **chunking documents** and **search**, not about prompts.\n\nCompared with the alternatives:\n\nA RAG system has three stages:\n\n*The three stages of RAG: documents are indexed ahead of time, then each question goes through retrieval and then generation*\n\n```\n ══ OFFLINE ══════════════════════════════════════════════════════════\n   data/in/*.md|txt|json\n        │\n        ▼\n   LoaderService ──► ChunkerService ──► Bedrock (Cohere Embed) ──► Postgres\n   read files        2-level split +    input_type =               chunks table\n                     header injection   search_document            (vector + tsvector)\n\n ══ ONLINE ═══════════════════════════════════════════════════════════\n   POST /api/chat { question }\n        │\n        ▼\n   SearchService.hybridSearch\n        ├─ embedQuery (Cohere, input_type = search_query)\n        ├─ DenseSearch   (HNSW index)  ─┐  run in parallel\n        ├─ KeywordSearch (GIN index)   ─┘\n        └─ reciprocalRankFusion ──► top 5 chunks\n        │\n        ▼\n   PromptBuilder ── [system + few-shot: CACHED] + [context + question: not cached]\n        │\n        ▼\n   Bedrock (Claude, InvokeModel)\n        │\n        ▼\n   { answer, citations, usage }\n```\n\nFolder layout, one folder per stage:\n\n```\nsrc/\n├── config/     all config, read from env\n├── database/   schema.sql + pg.Pool wrapper (no ORM)\n├── llm/        LlmService interface + Bedrock implementation\n├── ingest/     loader → chunker → repository → ingest.service\n├── search/     dense · keyword · rrf · search.service\n├── prompt/     system prompt · few-shot · prompt builder\n└── chat/       wires the three steps into one RAG loop + HTTP API\nscripts/        db-init · ingest · search (CLI)\ntest/           unit tests for chunker, rrf, prompt builder\n```\n\nThis architecture may look surprisingly simple.\n\nThat is intentional.\n\nEvery stage has a single responsibility and can be tested independently.\n\nWhen retrieval quality drops, we can inspect retrieval.\n\nWhen generation quality drops, we can inspect prompts.\n\nWhen ingest breaks, we can inspect indexing.\n\nThe separation becomes extremely valuable once the system grows beyond a proof of concept.\n\nThe diagram in section 2 shows the blocks. This section slows down: it takes one concrete document and one concrete question and follows them through each function, each SQL query, and each Bedrock call. Once you have read this section, the code in the later steps is just detail.\n\nThe sample data used throughout is `data/in/warranty.md`:\n\n```\n# Warranty Policy\n\n## Warranty period\nAll products are covered by a warranty of 12 months from the date of purchase.\n\n## Returns\nProducts with a manufacturing defect can be exchanged within the first 7 days.\n```\n\nThis flow runs offline via `npm run ingest`, not over HTTP. Re-run it whenever documents change.\n\n```\nnpm run ingest [-- <folder>]\n   │\n   ▼\nscripts/ingest.ts\n   │  NestFactory.createApplicationContext(AppModule)   (no HTTP port opened)\n   ▼\nIngestService.ingestDirectory(dir = INGEST_DIR)\n   │\n   ▼\n[1] LoaderService.loadDirectory ─────────► RawDocument[]\n   │    recursive scan; .md/.txt = 1 document; .json = n documents; other extensions skipped\n   │\n   │    for EACH RawDocument:\n   ▼\n[2] ChunkerService.chunk ────────────────► Chunk[]\n   │    split by heading, split long sections (1200 chars, 120 overlap), prepend heading\n   ▼\n[3] BedrockService.embedDocuments ───────► one 1024-dim vector per chunk\n   │    up to 96 texts per call, Cohere input_type = search_document\n   ▼\n[4] ChunkRepository.upsert (per chunk) ──► chunks table\n   │    INSERT ... ON CONFLICT (chunk_id) DO UPDATE\n   │    Postgres computes search_vector and updates HNSW + GIN indexes\n   ▼\n[5] ChunkRepository.deleteStale ─────────► delete leftover chunks of this document\n   │\n   ▼\nIngestReport { documents, chunks, deleted }  →  printed to console, pool closed\n```\n\n**[1] Read files.** `LoaderService` walks the folder recursively. A markdown file becomes one `RawDocument`:\n\n```\n{\n  \"source\": \"data/in/warranty.md\",\n  \"title\": \"Warranty Policy\",\n  \"text\": \"# Warranty Policy\\n\\n## Warranty period\\nAll products are covered...\"\n}\n```\n\n`source` is the file path, and `title` comes from the first H1 (or the file name if there is none). For `.json` files, each element with a `text` field becomes its own document.\n\n**[2] Chunk.** `splitByHeading` reads line by line and starts a new section whenever a line begins with `#`. The sample file produces 3 sections:\n\n```\nsection A  heading = \"Warranty Policy\"   text = \"\"                  → no chunk\nsection B  heading = \"Warranty period\"   text = \"All products are...\"\nsection C  heading = \"Returns\"           text = \"Products with a...\"\n```\n\nSection A has a heading but no body, so it is dropped. Sections B and C are shorter than 1200 characters, so they are kept whole. A longer section would be split further by `splitLongText`: paragraph boundaries first, then sentences, with 120 characters of overlap between consecutive pieces.\n\nAfter prepending the heading (header injection), we get 2 chunks:\n\n```\n[\n  {\n    \"chunkId\": \"data/in/warranty.md#0\",\n    \"title\": \"Warranty period\",\n    \"content\": \"Warranty period\\n\\nAll products are covered by a warranty of 12 months from the date of purchase.\",\n    \"metadata\": { \"heading\": \"Warranty period\" }\n  },\n  {\n    \"chunkId\": \"data/in/warranty.md#1\",\n    \"title\": \"Returns\",\n    \"content\": \"Returns\\n\\nProducts with a manufacturing defect can be exchanged within the first 7 days.\",\n    \"metadata\": { \"heading\": \"Returns\" }\n  }\n]\n```\n\n`chunkId` has the form `<source>#<position>`. It is the key that lets the next ingest update the existing row instead of adding a new one.\n\n**[3] Embed.** Both `content` strings go to Cohere in a single `InvokeModel` call:\n\n```\n{ \"texts\": [\"Warranty period\\n\\nAll products are...\", \"Returns\\n\\nProducts with a...\"],\n  \"input_type\": \"search_document\", \"truncate\": \"END\" }\n```\n\nThe result is 2 vectors of 1024 floats each. Because the heading is part of `content`, chunk #0's vector carries the meaning \"warranty period\", not just the sentence about 12 months.\n\n**[4] Store.** Each chunk is one `INSERT ... ON CONFLICT (chunk_id) DO UPDATE`. The application only writes `content` and `embedding`. Postgres does the rest:\n\n`search_vector` column is recomputed from `title + content` with `to_tsvector('simple', ...)`. Chunk #0 yields the tokens `warranty`, `period`, `all`, `products`, `are`, `covered`, `by`, `a`, `of`, `12`, `months`, `from`, `the`, `date`, `purchase`.` embedding`) and the GIN index (on **[5] Clean up leftovers.** `deleteStale('data/in/warranty.md', ['...#0', '...#1'])` deletes every row of this file whose `chunk_id` is not in the list just written. On the first run there is nothing to delete. If the file later shrinks from 5 chunks to 2, rows `#2`, `#3` and `#4` are deleted here.\n\nAt the end, the console prints a summary: number of documents, chunks, and stale chunks deleted.\n\nThree things worth knowing about this flow:\n\n`metadata` and skip chunks whose hash hasn't changed.`source` is the path exactly as scanned.`npm run ingest` (giving `data/in/...`) and later `npm run ingest -- /absolute/path` produces a different `ChunkRepository.deleteBySource`.\nThis flow runs online, once per request. One question costs **2 Bedrock calls** (1 embed, 1 generate) and **2 SQL queries run in parallel**.\n\n```\nClient\n   │  POST /api/chat  { \"question\": \"How long is the product warranty?\" }\n   ▼\n[1] ValidationPipe + AskDto ─────────────► 400 if empty, > 2000 chars, or unknown fields\n   │\n   ▼\nChatController.ask  →  ChatService.ask\n   │\n   ▼\n[2] SearchService.hybridSearch(question)\n   │\n   ├─ [2a] BedrockService.embedQuery ────► question vector (Cohere, input_type = search_query)\n   │\n   ├─ [2b] in parallel (Promise.all):\n   │      ├─ DenseSearch   : ORDER BY embedding <=> $vec LIMIT 50   → HNSW index\n   │      └─ KeywordSearch : search_vector @@ websearch_to_tsquery  → GIN index\n   │\n   └─ [2c] reciprocalRankFusion(k = 60) ─► top 5 chunks, with denseRank / keywordRank\n   │\n   ▼\n[3] PromptBuilder.build ─────────────────► { system: [...], user: [...] }\n   │      system = SYSTEM_PROMPT + few-shot                    (cached)\n   │      user   = <context>[1]..[5]</context> + <question>    (not cached)\n   ▼\n[4] BedrockService.generate ─────────────► InvokeModel Claude\n   │      cache_control on the few-shot block; retry on throttling\n   ▼\n[5] ChatService assembles the result\n   │      answer = all \"text\" blocks joined\n   │      citations = the chunks that were sent, numbered to match [n]\n   │      usage = input/output tokens + cache write/read tokens\n   ▼\nClient receives { answer, citations, usage }\n```\n\n**[1] Guard the input.** `ValidationPipe` runs with `whitelist` and `forbidNonWhitelisted`, so the body may contain only the `question` field. `AskDto` requires `question` to be a non-empty string of at most 2000 characters. Invalid requests get a 400 immediately, before a single cent is spent on Bedrock.\n\n**[2a] Embed the question.** The question goes to Cohere with `input_type: \"search_query\"`. This mirrors step [3] of the ingest flow: documents use `search_document`, questions use `search_query`, same model. Get either one wrong and question vectors and document vectors no longer live on the same \"map\".\n\n**[2b] Two search branches run at the same time.**\n\nThe dense branch compares the question vector with every chunk vector by cosine; thanks to HNSW it does not scan the whole table. With our 2-chunk sample corpus:\n\n```\ndense:   1. ...#0  \"Warranty period\"   highest cosine\n         2. ...#1  \"Returns\"\n```\n\nThe keyword branch passes the raw question to `websearch_to_tsquery('simple', ...)`:\n\n```\n'how' & 'long' & 'is' & 'the' & 'product' & 'warranty'\n```\n\nThis function joins words with **AND**. Chunk #0 contains none of `how`, `long`, `is`, and has `products` rather than `product` (the `simple` dictionary does not stem), so the keyword branch returns **0 results** for this question. That is the real behavior of the code, and the reason the dense branch matters: for long natural-language questions the keyword branch usually stays silent. It shines on short queries containing codes or names, like `E-4021` or `warranty 12 months`.\n\n**[2c] Merge with RRF.** Each chunk gets `1 / (60 + rank)` from every branch it appears in:\n\n```\n...#0  rrf = 1/61 = 0.01639   denseRank = 1   keywordRank = null\n...#1  rrf = 1/62 = 0.01613   denseRank = 2   keywordRank = null\n```\n\nIf the query were `warranty 12 months`, chunk #0 would appear in both branches, score `1/61 + 1/61 = 0.03279`, and pull far ahead of chunks found by only one branch. The list is cut to the top 5 (`TOP_K`).\n\n**[3] Build the prompt.** `PromptBuilder` numbers the chunks from 1 and wraps them in tags (the repository uses Vietnamese prompt text and tag names; English equivalents are shown throughout this article, see section 12):\n\n```\n<context>\n[1] Warranty period\nWarranty period\n\nAll products are covered by a warranty of 12 months from the date of purchase.\n\n[2] Returns\nReturns\n\nProducts with a manufacturing defect can be exchanged within the first 7 days.\n</context>\n<question>How long is the product warranty?</question>\n```\n\nThe heading appears twice because `title` is printed first while `content` already contains the heading from header injection. It's harmless, but you can drop the `title` line when rendering to save tokens.\n\nIf neither branch finds anything, the context is `(no relevant documents found)` and the LLM is still called. The system prompt and few-shot example #2 teach the model to say it could not find the information instead of inventing an answer.\n\n**[4] Call Claude.** The body sent via `InvokeModel`:\n\n```\n{\n  \"anthropic_version\": \"bedrock-2023-05-31\",\n  \"max_tokens\": 2000,\n  \"system\": [\n    { \"type\": \"text\", \"text\": \"You are a Q&A assistant that answers from the provided documents...\" },\n    { \"type\": \"text\", \"text\": \"Examples of good answers: ...\",\n      \"cache_control\": { \"type\": \"ephemeral\" } }\n  ],\n  \"messages\": [\n    { \"role\": \"user\", \"content\": [\n      { \"type\": \"text\", \"text\": \"<context>\\n[1] Warranty period\\n...\\n</context>\" },\n      { \"type\": \"text\", \"text\": \"<question>How long is the product warranty?</question>\" }\n    ] }\n  ]\n}\n```\n\nEverything before `cache_control` (system prompt + few-shot) is identical on every request, so from the second request on it is read from cache. `messages` changes with every question and is always billed at full price. If Bedrock returns `ThrottlingException` or `ServiceUnavailableException`, the request is retried after 500ms and then 1s; any other error is thrown immediately.\n\n**[5] Return the result.** Claude returns a `content` array. The code keeps only `type: \"text\"` blocks and joins them, ignoring other block types. `citations` is the list of chunks put into the prompt, numbered to match `[n]` in the context, so the client can turn `[1]` in the answer into a link to `source`:\n\n```\n{\n  \"answer\": \"All products are covered by a 12-month warranty from the date of purchase [1].\",\n  \"citations\": [\n    { \"index\": 1, \"chunkId\": \"data/in/warranty.md#0\", \"source\": \"data/in/warranty.md\", \"title\": \"Warranty period\" },\n    { \"index\": 2, \"chunkId\": \"data/in/warranty.md#1\", \"source\": \"data/in/warranty.md\", \"title\": \"Returns\" }\n  ],\n  \"usage\": { \"inputTokens\": 180, \"outputTokens\": 25, \"cacheWriteTokens\": 0, \"cacheReadTokens\": 650 }\n}\n```\n\n(The `usage` values above are illustrative.) Note that `citations` contains every chunk that was sent, not only the ones the model actually cited. The client should filter by the `[n]` numbers that appear in `answer`.\n\nAccuracy doesn't come from one place. It accumulates across both flows:\n\n`input_type`:`[n]`, and knows how to say \"not found\".\nWhat these two flows **don't** do yet, covered in the \"Upgrade path\" section at the end: Unicode NFC normalization for both documents and questions, rewriting questions using conversation history (the API is currently stateless), a confidence threshold to skip the LLM when results are too weak, and checking in code that the `[n]` the model cites actually exist.\n\nOne design goal guided every technology choice: **keep the architecture understandable.**\n\nEvery component should be replaceable without rewriting the rest of the system.\n\n`useClass` change.`cohere.embed-multilingual-v3`, 1024 dimensions): multilingual, which matters when documents mix a local language with English UI labels.`pg`, no ORM`@aws-sdk/client-bedrock-runtime`` InvokeModel`.\nThat's the entire runtime dependency list:\n\n```\n\"dependencies\": {\n  \"@aws-sdk/client-bedrock-runtime\": \"^3.696.0\",\n  \"@nestjs/common\": \"^10.4.0\",\n  \"@nestjs/core\": \"^10.4.0\",\n  \"@nestjs/platform-express\": \"^10.4.0\",\n  \"class-transformer\": \"^0.5.1\",\n  \"class-validator\": \"^0.14.1\",\n  \"dotenv\": \"^16.4.5\",\n  \"pg\": \"^8.12.0\",\n  \"reflect-metadata\": \"^0.2.2\",\n  \"rxjs\": \"^7.8.1\"\n}\n```\n\n`ap-northeast-1`).` anthropic.` prefix, for example `anthropic.claude-opus-5-5`. Some regions require calling through a The application only needs `bedrock:InvokeModel`:\n\n```\n{\n  \"Version\": \"2012-10-17\",\n  \"Statement\": [\n    {\n      \"Effect\": \"Allow\",\n      \"Action\": [\"bedrock:InvokeModel\"],\n      \"Resource\": [\n        \"arn:aws:bedrock:*::foundation-model/*\",\n        \"arn:aws:bedrock:*:*:inference-profile/*\"\n      ]\n    }\n  ]\n}\n```\n\nIn production, narrow `Resource` down to the exact ARNs of the two models you use.\n\n**About credentials:** static access keys work, but it's better to leave them empty and let the AWS SDK pick up `AWS_PROFILE`, SSO, or the EC2/ECS IAM role. The code below supports both.\n\n```\nnpm i -g @nestjs/cli\nnest new rag-chatbot --package-manager npm --skip-git\ncd rag-chatbot\n\nnpm i @aws-sdk/client-bedrock-runtime pg dotenv class-validator class-transformer\nnpm i -D @types/pg ts-node\n```\n\nAdd these scripts to `package.json`:\n\n```\n\"scripts\": {\n  \"build\": \"nest build\",\n  \"start:dev\": \"nest start --watch\",\n  \"start:prod\": \"node dist/main\",\n  \"db:init\": \"ts-node scripts/db-init.ts\",\n  \"ingest\": \"ts-node scripts/ingest.ts\",\n  \"search\": \"ts-node scripts/search.ts\",\n  \"test\": \"jest\",\n  \"lint\": \"eslint \\\"{src,scripts,test}/**/*.ts\\\"\"\n}\n```\n\nEvery setting is read from env, with sensible defaults so it runs out of the box. No per-project profile files.\n\n`.env.example`:\n\n```\n# ── App\nPORT=8000\n\n# ── PostgreSQL + pgvector\nDB_HOST=127.0.0.1\nDB_PORT=5432\nDB_NAME=ragdb\nDB_USER=postgres\nDB_PASSWORD=postgres\n# Must match the embedding model's dimensions and the vector(N) column in the schema.\n# cohere.embed-multilingual-v3 → 1024. Changing it = re-run db:init + re-embed everything.\nEMBEDDING_DIMENSIONS=1024\n\n# ── AWS Bedrock\nAWS_REGION=ap-northeast-1\nAWS_ACCESS_KEY_ID=\nAWS_SECRET_ACCESS_KEY=\nAWS_SESSION_TOKEN=\n# Leave the keys above empty to let the SDK use AWS_PROFILE / SSO / instance role.\n\nBEDROCK_EMBEDDING_MODEL_ID=cohere.embed-multilingual-v3\nBEDROCK_GENERATION_MODEL_ID=anthropic.claude-opus-5-5\n\n# ── Ingest / chunking\nINGEST_DIR=./data/in\nCHUNK_MAX_CHARS=1200\nCHUNK_OVERLAP_CHARS=120\nEMBED_BATCH_SIZE=96\n\n# ── Search\nTOP_K=5            # number of chunks finally placed in the prompt\nCANDIDATE_LIMIT=50 # candidates fetched by EACH branch before fusion\nRRF_K=60           # RRF constant k, 60 is the value from the original paper\n\n# ── Generation\nMAX_TOKENS=2000\nENABLE_PROMPT_CACHE=true\n```\n\n`src/config/config.ts` (abridged; the full version has an `AppConfig` interface documenting each field):\n\n``` js\nconst num = (v: string | undefined, fallback: number) =>\n  v === undefined || v === '' ? fallback : Number(v);\n\nexport function loadConfig(): AppConfig {\n  return {\n    port: num(process.env.PORT, 8000),\n    database: {\n      host: process.env.DB_HOST || '127.0.0.1',\n      port: num(process.env.DB_PORT, 5432),\n      name: process.env.DB_NAME || 'ragdb',\n      user: process.env.DB_USER || 'postgres',\n      password: process.env.DB_PASSWORD || 'postgres',\n      embeddingDimensions: num(process.env.EMBEDDING_DIMENSIONS, 1024),\n    },\n    bedrock: {\n      region: process.env.AWS_REGION || 'ap-northeast-1',\n      accessKeyId: process.env.AWS_ACCESS_KEY_ID,\n      secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY,\n      sessionToken: process.env.AWS_SESSION_TOKEN,\n      embeddingModelId: process.env.BEDROCK_EMBEDDING_MODEL_ID || 'cohere.embed-multilingual-v3',\n      generationModelId: process.env.BEDROCK_GENERATION_MODEL_ID || 'anthropic.claude-opus-5-5',\n    },\n    ingest: {\n      dir: process.env.INGEST_DIR || './data/in',\n      chunkMaxChars: num(process.env.CHUNK_MAX_CHARS, 1200),\n      chunkOverlapChars: num(process.env.CHUNK_OVERLAP_CHARS, 120),\n      embedBatchSize: num(process.env.EMBED_BATCH_SIZE, 96),\n    },\n    search: {\n      topK: num(process.env.TOP_K, 5),\n      candidateLimit: num(process.env.CANDIDATE_LIMIT, 50),\n      rrfK: num(process.env.RRF_K, 60),\n    },\n    generation: {\n      maxTokens: num(process.env.MAX_TOKENS, 2000),\n      enableCache: process.env.ENABLE_PROMPT_CACHE !== 'false',\n    },\n  };\n}\n\nexport const CONFIG = Symbol('CONFIG');\n```\n\nRegister `CONFIG` as a global provider in `AppConfigModule` (`{ provide: CONFIG, useFactory: loadConfig }`) so every service can inject it with `@Inject(CONFIG)`.\n\n```\nname: rag-core\n\nservices:\n  postgres:\n    image: pgvector/pgvector:pg16   # pgvector 0.7+, needed for the HNSW index\n    container_name: rag-postgres\n    environment:\n      POSTGRES_USER: ${DB_USER:-postgres}\n      POSTGRES_PASSWORD: ${DB_PASSWORD:-postgres}\n      POSTGRES_DB: ${DB_NAME:-ragdb}\n    ports:\n      - \"${DB_PORT:-5432}:5432\"\n    volumes:\n      - pgvector_data:/var/lib/postgresql/data\n    healthcheck:\n      test: [\"CMD-SHELL\", \"pg_isready -U ${DB_USER:-postgres} -d ${DB_NAME:-ragdb}\"]\n      interval: 5s\n      timeout: 5s\n      retries: 10\n\nvolumes:\n  pgvector_data:\n```\n\n`name: rag-core` is there for a reason: without it, Compose uses the folder name as the project name, and two checkouts with the same folder name on one machine will silently recreate each other's containers.\n\n`src/database/schema.sql`:\n\n```\nCREATE EXTENSION IF NOT EXISTS vector;\n\nCREATE TABLE IF NOT EXISTS chunks (\n  id           BIGSERIAL PRIMARY KEY,\n  -- Logical key \"<source>#<index>\": re-ingesting the same file upserts the same row.\n  chunk_id     TEXT NOT NULL UNIQUE,\n  source       TEXT NOT NULL,\n  chunk_index  INT  NOT NULL,\n  title        TEXT,\n  content      TEXT NOT NULL,\n  metadata     JSONB NOT NULL DEFAULT '{}'::jsonb,\n  embedding    vector(1024),\n\n  -- Auto-generated full-text column, 'simple' dictionary.\n  search_vector tsvector GENERATED ALWAYS AS (\n    to_tsvector('simple', coalesce(title, '') || ' ' || content)\n  ) STORED,\n\n  created_at   TIMESTAMPTZ NOT NULL DEFAULT NOW(),\n  updated_at   TIMESTAMPTZ NOT NULL DEFAULT NOW()\n);\n\n-- Vector search: HNSW + cosine\nCREATE INDEX IF NOT EXISTS chunks_embedding_idx\n  ON chunks USING hnsw (embedding vector_cosine_ops);\n\n-- Full-text search\nCREATE INDEX IF NOT EXISTS chunks_fts_idx\n  ON chunks USING gin (search_vector);\n\nCREATE INDEX IF NOT EXISTS chunks_source_idx ON chunks (source);\n```\n\nThree decisions worth noting:\n\n`chunk_id` is a unique logical key.`content` changes, so application code never has to keep it in sync.`'simple'` dictionary instead of `'english'`.` simple` does no stemming and removes no language-specific stopwords. For languages like Vietnamese, the `english` config would chop word endings in nonsensical ways. `scripts/db-init.ts` reads the SQL file, swaps in the vector dimension from `EMBEDDING_DIMENSIONS`, and runs it. Safe to run any number of times (every statement is `IF NOT EXISTS`):\n\n``` js\nconst sql = await fs.readFile(path.join(__dirname, '../src/database/schema.sql'), 'utf8');\nawait client.connect();\nawait client.query(sql.replace('vector(1024)', `vector(${config.database.embeddingDimensions})`));\nawait client.end();\n```\n\n`pg.Pool`\n\n```\n@Injectable()\nexport class DatabaseService implements OnModuleDestroy {\n  private readonly pool: Pool;\n\n  constructor(@Inject(CONFIG) config: AppConfig) {\n    this.pool = new Pool({\n      host: config.database.host,\n      port: config.database.port,\n      database: config.database.name,\n      user: config.database.user,\n      password: config.database.password,\n    });\n  }\n\n  async query<T extends QueryResultRow>(sql: string, params: unknown[] = []): Promise<T[]> {\n    const result = await this.pool.query<T>(sql, params);\n    return result.rows;\n  }\n\n  /** pgvector accepts a vector as the literal '[1,2,3]' cast to ::vector. */\n  static toVector(values: number[]): string {\n    return `[${values.join(',')}]`;\n  }\n\n  async onModuleDestroy(): Promise<void> {\n    await this.pool.end();\n  }\n}\n```\n\nThe rest of the codebase only knows this interface, not Bedrock:\n\n``` js\nexport const LLM = Symbol('LLM');\n\nexport interface PromptBlock {\n  text: string;\n  /** true = block belongs to the static prefix and gets a cache_control marker. */\n  cacheable?: boolean;\n}\n\nexport interface GenerateRequest {\n  system: PromptBlock[]; // persona, rules, few-shot: the cached part\n  user: PromptBlock[];   // retrieved context + question: changes every turn\n  maxTokens?: number;\n}\n\nexport interface Usage {\n  inputTokens: number;\n  outputTokens: number;\n  cacheWriteTokens: number;\n  cacheReadTokens: number;\n}\n\nexport interface LlmService {\n  embedDocuments(texts: string[]): Promise<number[][]>;\n  embedQuery(text: string): Promise<number[]>;\n  generate(request: GenerateRequest): Promise<{ text: string; usage: Usage }>;\n}\n// src/llm/llm.module.ts: switching providers = changing this one useClass line\n@Global()\n@Module({\n  providers: [{ provide: LLM, useClass: BedrockService }],\n  exports: [LLM],\n})\nexport class LlmModule {}\n@Injectable()\nexport class BedrockService implements LlmService {\n  private readonly client: BedrockRuntimeClient;\n\n  constructor(@Inject(CONFIG) private readonly config: AppConfig) {\n    const { region, accessKeyId, secretAccessKey, sessionToken } = config.bedrock;\n    this.client = new BedrockRuntimeClient({\n      region,\n      // Don't pass credentials when there are no static keys,\n      // so the SDK uses its own chain (AWS_PROFILE, SSO, instance role).\n      ...(accessKeyId && secretAccessKey\n        ? { credentials: { accessKeyId, secretAccessKey, sessionToken } }\n        : {}),\n    });\n  }\n\n  private async invoke(modelId: string, body: unknown): Promise<Record<string, any>> {\n    let lastError: unknown;\n\n    for (let attempt = 1; attempt <= 3; attempt++) {\n      try {\n        const response = await this.client.send(\n          new InvokeModelCommand({\n            modelId,\n            contentType: 'application/json',\n            accept: 'application/json',\n            body: JSON.stringify(body),\n          }),\n        );\n        return JSON.parse(new TextDecoder().decode(response.body));\n      } catch (err) {\n        lastError = err;\n        const name = (err as { name?: string }).name ?? '';\n        const retryable = name === 'ThrottlingException' || name === 'ServiceUnavailableException';\n        if (!retryable || attempt === 3) break;\n        await new Promise((r) => setTimeout(r, 2 ** attempt * 250)); // 500ms, 1s\n      }\n    }\n    throw lastError;\n  }\n```\n\nOnly **transient** errors (throttling, service unavailable) are retried, with exponential backoff. A `ValidationException` or `AccessDeniedException` won't change no matter how many times you retry, so those are thrown immediately.\n\n```\n  private async embed(\n    texts: string[],\n    inputType: 'search_document' | 'search_query',\n  ): Promise<number[][]> {\n    const out: number[][] = [];\n    const batchSize = Math.min(this.config.ingest.embedBatchSize, 96); // Cohere: max 96 texts per call\n\n    for (let i = 0; i < texts.length; i += batchSize) {\n      const body = await this.invoke(this.config.bedrock.embeddingModelId, {\n        texts: texts.slice(i, i + batchSize),\n        input_type: inputType,\n        truncate: 'END',\n      });\n      out.push(...(body.embeddings as number[][]));\n    }\n    return out;\n  }\n\n  async embedDocuments(texts: string[]) {\n    return texts.length ? this.embed(texts, 'search_document') : [];\n  }\n\n  async embedQuery(text: string) {\n    const [embedding] = await this.embed([text], 'search_query');\n    return embedding;\n  }\n```\n\n`input_type` directly affects results. Cohere only projects documents and queries into the same space when each side declares its role correctly: documents are `search_document`, questions are `search_query`. Using the same value for both still runs and reports no error; recall just drops a few percent, and hardly anyone notices.\n\nThe body sent to Claude via `InvokeModel` follows the Anthropic Messages format, with `anthropic_version: 'bedrock-2023-05-31'`:\n\n```\ntype TextBlock = { type: 'text'; text: string; cache_control?: { type: 'ephemeral' } };\n\n  /** Put cache_control on the LAST block of the static prefix. */\n  private toBlocks(blocks: PromptBlock[]): TextBlock[] {\n    const out: TextBlock[] = blocks\n      .filter((b) => b.text.trim())\n      .map((b) => ({ type: 'text' as const, text: b.text }));\n\n    if (!this.config.generation.enableCache) return out;\n\n    const lastCacheable = blocks.reduce(\n      (acc, b, i) => (b.cacheable && b.text.trim() ? i : acc),\n      -1,\n    );\n    if (lastCacheable >= 0 && out[lastCacheable]) {\n      out[lastCacheable].cache_control = { type: 'ephemeral' };\n    }\n    return out;\n  }\n\n  async generate(request: GenerateRequest) {\n    const body = await this.invoke(this.config.bedrock.generationModelId, {\n      anthropic_version: 'bedrock-2023-05-31',\n      max_tokens: request.maxTokens ?? this.config.generation.maxTokens,\n      system: this.toBlocks(request.system),\n      messages: [{ role: 'user', content: this.toBlocks(request.user) }],\n    });\n\n    // Keep only 'text' blocks. Newer Claude models may return other block\n    // types (e.g. thinking); without this filter they leak into the answer.\n    const text = ((body.content ?? []) as Array<{ type: string; text?: string }>)\n      .filter((block) => block.type === 'text')\n      .map((block) => block.text ?? '')\n      .join('')\n      .trim();\n\n    const usage = body.usage ?? {};\n    return {\n      text,\n      usage: {\n        inputTokens: usage.input_tokens ?? 0,\n        outputTokens: usage.output_tokens ?? 0,\n        cacheWriteTokens: usage.cache_creation_input_tokens ?? 0,\n        cacheReadTokens: usage.cache_read_input_tokens ?? 0,\n      },\n    };\n  }\n}\n```\n\nWhy the cache marker goes on the last block of the static part: see Step 7.\n\nThe pipeline has four classes, each with one job:\n\n```\nLoaderService  ──►  ChunkerService  ──►  LlmService.embedDocuments  ──►  ChunkRepository\n(read sources)      (chunking)            (batches of 96)                 (upsert + clean up)\n                         ▲\n                  IngestService only orchestrates, no logic of its own\nexport interface RawDocument {\n  source: string;   // file path, URL, CMS id...\n  title?: string;\n  text: string;\n  metadata?: Record<string, unknown>;\n}\n```\n\n`LoaderService` scans the folder recursively:\n\n`.md` / `.txt`: reads the whole file. Uses the first H1 as the title, or the file name if there is none.`.json`: an array of objects with a `text` field (optional `title`, \n\n``` js\nasync loadTextFile(file: string): Promise<RawDocument> {\n  const text = await fs.readFile(file, 'utf8');\n  return {\n    source: file,\n    title: /^#\\s+(.+)$/m.exec(text)?.[1]?.trim() ?? path.basename(file),\n    text,\n  };\n}\n```\n\nTo add S3, Confluence, Notion and so on, add one method that returns `RawDocument[]`. Chunking and embedding stay the same.\n\nThis step has a big impact on quality but is often done carelessly.\n\nThe codebase's strategy:\n\n`CHUNK_MAX_CHARS``\\n\\n`), then by sentence, with an \n\n```\n@Injectable()\nexport class ChunkerService {\n  constructor(@Inject(CONFIG) private readonly config: AppConfig) {}\n\n  chunk(doc: RawDocument): Chunk[] {\n    const { chunkMaxChars, chunkOverlapChars } = this.config.ingest;\n    const pieces: Array<{ heading?: string; text: string }> = [];\n\n    for (const section of this.splitByHeading(doc.text)) {\n      for (const part of this.splitLongText(section.text, chunkMaxChars, chunkOverlapChars)) {\n        pieces.push({ heading: section.heading, text: part });\n      }\n    }\n\n    return pieces.map((piece, index) => ({\n      chunkId: `${doc.source}#${index}`,\n      source: doc.source,\n      chunkIndex: index,\n      title: piece.heading ?? doc.title,\n      // Header injection: the heading goes into the embedding and FTS with the content.\n      content: piece.heading ? `${piece.heading}\\n\\n${piece.text}` : piece.text,\n      metadata: { ...(doc.metadata ?? {}), heading: piece.heading ?? null },\n    }));\n  }\n\n  private splitByHeading(text: string) {\n    const sections: Array<{ heading?: string; text: string[] }> = [];\n    let current: { heading?: string; text: string[] } = { text: [] };\n\n    for (const line of text.split('\\n')) {\n      const heading = /^(#{1,6})\\s+(.+)$/.exec(line);\n      if (heading) {\n        if (current.text.join('').trim() || current.heading) sections.push(current);\n        current = { heading: heading[2].trim(), text: [] };\n      } else {\n        current.text.push(line);\n      }\n    }\n    sections.push(current);\n\n    return sections\n      .map((s) => ({ heading: s.heading, text: s.text.join('\\n').trim() }))\n      .filter((s) => s.text || s.heading);\n  }\n\n  private splitLongText(text: string, maxChars: number, overlapChars: number): string[] {\n    if (text.length <= maxChars) return text ? [text] : [];\n\n    // Prefer paragraph boundaries, then sentences (Japanese punctuation included).\n    const units = text\n      .split(/\\n{2,}/)\n      .flatMap((p) => (p.length <= maxChars ? [p] : p.split(/(?<=[.!?。？！])\\s+/)))\n      .filter((u) => u.trim());\n\n    const out: string[] = [];\n    let buffer = '';\n\n    const flush = () => {\n      if (!buffer.trim()) return;\n      out.push(buffer.trim());\n      buffer = overlapChars > 0 ? buffer.slice(-overlapChars) : ''; // overlap\n    };\n\n    for (const unit of units) {\n      if (buffer && buffer.length + unit.length + 2 > maxChars) flush();\n\n      if (unit.length > maxChars) {\n        // The smallest unit is still too long, no boundary left to respect.\n        for (let i = 0; i < unit.length; i += maxChars - overlapChars) {\n          out.push(unit.slice(i, i + maxChars).trim());\n        }\n        buffer = '';\n        continue;\n      }\n      buffer = buffer ? `${buffer}\\n\\n${unit}` : unit;\n    }\n\n    if (buffer.trim()) out.push(buffer.trim());\n    return out.filter(Boolean);\n  }\n}\n```\n\nIf you can make only **one** chunking improvement, make it header injection: it costs almost nothing (string concatenation) and improves **both** the vector branch and the keyword branch.\n\n**A note on size:** `CHUNK_MAX_CHARS=1200` counts characters. For the same content, Vietnamese takes roughly 1.5x as many tokens as English, and Japanese roughly 3x. For Japanese documents, lower this limit so chunks stay within the embedding model's limit (anything beyond it is silently cut by `truncate: 'END'`).\n\n```\nasync upsert(chunk: Chunk, embedding: number[]): Promise<void> {\n  await this.db.query(\n    `\n    INSERT INTO chunks (chunk_id, source, chunk_index, title, content, metadata, embedding)\n    VALUES ($1, $2, $3, $4, $5, $6::jsonb, $7::vector)\n    ON CONFLICT (chunk_id) DO UPDATE SET\n      source = EXCLUDED.source, chunk_index = EXCLUDED.chunk_index,\n      title = EXCLUDED.title, content = EXCLUDED.content,\n      metadata = EXCLUDED.metadata, embedding = EXCLUDED.embedding,\n      updated_at = NOW()\n    `,\n    [chunk.chunkId, chunk.source, chunk.chunkIndex, chunk.title ?? null,\n     chunk.content, JSON.stringify(chunk.metadata), DatabaseService.toVector(embedding)],\n  );\n}\n\nasync deleteStale(source: string, keepChunkIds: string[]): Promise<number> {\n  const rows = await this.db.query<{ chunk_id: string }>(\n    `DELETE FROM chunks WHERE source = $1 AND chunk_id <> ALL($2::text[]) RETURNING chunk_id`,\n    [source, keepChunkIds],\n  );\n  return rows.length;\n}\n```\n\nUpsert alone is **not enough** for safe re-ingestion. Say a document used to have 10 chunks and the new version has 6. Upsert updates the first 6 rows, but the last 4 stay there with their old embeddings and keep being retrieved. The chatbot ends up answering from content that no longer exists.\n\n`deleteStale` deletes by **exclusion** after the upsert finishes, instead of \"delete everything, then write again\". That way search never sees a window where the document is missing.\n\nAbridged version (in the repo, embed + upsert live in a separate `embedAndStore` method):\n\n``` js\nasync ingestDocuments(docs: RawDocument[]): Promise<IngestReport> {\n  const report = { documents: 0, chunks: 0, deleted: 0 };\n\n  for (const doc of docs) {\n    const chunks = this.chunker.chunk(doc);\n    if (!chunks.length) continue;\n\n    // Embed in batches, then upsert each chunk\n    for (let i = 0; i < chunks.length; i += this.config.ingest.embedBatchSize) {\n      const batch = chunks.slice(i, i + this.config.ingest.embedBatchSize);\n      const embeddings = await this.llm.embedDocuments(batch.map((c) => c.content));\n      for (const [j, chunk] of batch.entries()) await this.repo.upsert(chunk, embeddings[j]);\n    }\n\n    report.deleted += await this.repo.deleteStale(doc.source, chunks.map((c) => c.chunkId));\n    report.documents += 1;\n    report.chunks += chunks.length;\n  }\n  return report;\n}\n```\n\nThe CLI script `scripts/ingest.ts` boots Nest as an application context (no HTTP server) and calls the service:\n\n``` js\nconst app = await NestFactory.createApplicationContext(AppModule, { logger: ['log', 'warn', 'error'] });\nconst report = await app.get(IngestService).ingestDirectory(process.argv[2]);\nconsole.log(`done: ${report.documents} documents, ${report.chunks} chunks, ${report.deleted} stale chunks deleted`);\nawait app.close();\n```\n\nEach branch fails exactly where the other is strong:\n\n`E-4021` and `E-4012` are almost identical in embedding space.\nYou combine the two because they cover for each other, not because more branches are always better.\n\n```\nasync search(queryVector: number[], limit: number): Promise<Candidate[]> {\n  const rows = await this.db.query<any>(\n    `\n    SELECT chunk_id AS \"chunkId\", source, title, content, metadata,\n           1 - (embedding <=> $1::vector) AS score\n    FROM chunks\n    WHERE embedding IS NOT NULL\n    ORDER BY embedding <=> $1::vector\n    LIMIT $2::int\n    `,\n    [DatabaseService.toVector(queryVector), limit],\n  );\n  return rows.map((r) => ({ ...r, score: Number(r.score) }));\n}\n```\n\n`<=>` is pgvector's cosine distance operator, so `1 - distance` is cosine similarity. The query deliberately has just `ORDER BY embedding <=> $1 LIMIT n`, because that is the shape the planner needs to use the HNSW index. If you add filters, check again with `EXPLAIN`, since it is very easy to turn this into a full table scan.\n\n``` js\nasync search(queryText: string, limit: number): Promise<Candidate[]> {\n  const rows = await this.db.query<any>(\n    `\n    WITH tsq AS (SELECT websearch_to_tsquery('simple', $1) AS q)\n    SELECT c.chunk_id AS \"chunkId\", c.source, c.title, c.content, c.metadata,\n           ts_rank(c.search_vector, tsq.q) AS score\n    FROM chunks c, tsq\n    WHERE c.search_vector @@ tsq.q\n    ORDER BY score DESC\n    LIMIT $2::int\n    `,\n    [queryText, limit],\n  );\n  return rows.map((r) => ({ ...r, score: Number(r.score) }));\n}\n```\n\n`websearch_to_tsquery` accepts the syntax users already know (`\"exact phrase\"`, `a OR b`, `-exclude`) and `ts_rank` is The \"obvious\" approach is `score = 0.7 × cosine + 0.3 × ts_rank`. It's wrong:\n\n`[0, 1]`, while `0.0x`. Add them directly and the keyword branch contributes almost nothing. You think hybrid search is running, but it is really just vector search.\n**Reciprocal Rank Fusion (RRF)** throws away the raw scores and uses only **ranks**:\n\n```\nRRF(d) = Σ        1 / (k + rank_branch(d))        k = 60\n        branch\n```\n\nA worked example:\n\n```\nDense:   1. chunk_A   2. chunk_B   3. chunk_C\nKeyword: 1. chunk_C   2. chunk_A   3. chunk_D\n\nchunk_A: 1/61 + 1/62 = 0.03252   ← rank 1\nchunk_C: 1/63 + 1/61 = 0.03226   ← rank 2\nchunk_B: 1/62        = 0.01613\nchunk_D: 1/63        = 0.01587\n```\n\n`chunk_A` wins without being first in either branch, because it ranks well in both. Agreement between two independent methods is a stronger signal than a high score from one.\n\nThe constant `k` flattens the curve: with `k = 60`, the gap between rank 1 and rank 2 is only about 1.6%, so **appearing** in several lists matters more than the exact rank.\n\n```\nexport function reciprocalRankFusion(\n  branches: { dense: Candidate[]; keyword: Candidate[] },\n  k = 60,\n  topK = 5,\n): SearchResult[] {\n  const byId = new Map<string, SearchResult>();\n\n  const ingest = (candidates: Candidate[], branch: 'denseRank' | 'keywordRank') => {\n    candidates.forEach((candidate, i) => {\n      const rank = i + 1;\n      const existing = byId.get(candidate.chunkId);\n\n      if (existing) {\n        existing.rrfScore += 1 / (k + rank);\n        existing[branch] = rank;\n        return;\n      }\n\n      const { score: _ignored, ...rest } = candidate;\n      byId.set(candidate.chunkId, {\n        ...rest,\n        rrfScore: 1 / (k + rank),\n        denseRank: null,\n        keywordRank: null,\n        [branch]: rank,\n      } as SearchResult);\n    });\n  };\n\n  ingest(branches.dense, 'denseRank');\n  ingest(branches.keyword, 'keywordRank');\n\n  return [...byId.values()].sort((a, b) => b.rrfScore - a.rrfScore).slice(0, topK);\n}\n```\n\nThe result keeps each chunk's `denseRank` and `keywordRank`. When debugging, one look tells you whether a result came from semantics, keywords, or both.\n\n```\nasync hybridSearch(query: string, options: SearchOptions = {}): Promise<SearchResult[]> {\n  const topK = options.topK ?? this.config.search.topK;            // 5\n  const limit = options.candidateLimit ?? this.config.search.candidateLimit; // 50\n\n  const queryVector = await this.llm.embedQuery(query);\n\n  const [dense, keyword] = await Promise.all([\n    this.dense.search(queryVector, limit),\n    this.keyword.search(query, limit),\n  ]);\n\n  return reciprocalRankFusion({ dense, keyword }, this.config.search.rrfK, topK);\n}\n```\n\nTwo details decide the quality:\n\nAdding a third branch (multi-query, or real BM25) is just one more array passed to RRF. No score normalization, no weight tuning.\n\nThe repository ships this prompt in Vietnamese (with `<ngu_canh>` / `<cau_hoi>` tags); the English equivalent is shown here.\n\n``` js\nexport const SYSTEM_PROMPT = `You are a Q&A assistant that answers from the provided documents.\n\nAnswering rules:\n- Answer only from the CONTEXT provided. Do not guess, and do not use knowledge\n  outside the context.\n- If the context is not enough to answer, say plainly that the information was\n  not found. A wrong answer is more dangerous than \"I don't know\".\n- Cite sources with [number] matching the position of the context passage you\n  relied on.\n- Answer in the same language as the question.\n- Be concise and get straight to the point. Do not start with \"Based on the documents...\".`;\n```\n\nThree rules for this file:\n\n`if (x) prompt += ...`). Every combination of conditions is a different prefix, i.e. a different cache entry.\nThe rule \"answer in the same language as the question\" is essential: when documents are in English and the user asks in another language, without this rule the model tends to answer in the language of the context.\n\n``` js\nexport const EXAMPLES: Example[] = [\n  {\n    question: 'How long is the product warranty?',\n    context:\n      '[1] Warranty policy\\nAll products are covered by a 12-month warranty from the date of purchase.\\n\\n' +\n      '[2] Returns\\nDefective products can be exchanged within the first 7 days.',\n    answer: 'Products are covered by a 12-month warranty from the date of purchase [1].',\n  },\n  {\n    // Negative example: there is context, but no passage answers the question.\n    question: 'Does the company have a branch in Da Nang?',\n    context: '[1] Warranty policy\\nAll products are covered by a 12-month warranty from the date of purchase.',\n    answer: 'The available documents do not mention any branch information.',\n  },\n];\n```\n\nExamples are the strongest signal in a prompt: the `[1]` citation format is easy to misread when described in words, but obvious from a single example.\n\n**The refusal example matters as much as the answer example.** If every example is a successful answer, the model learns that there must always be an answer. Example #2 teaches it to decline when the context is irrelevant.\n\nPrompt caching is a **prefix match**: the system hashes the prompt from the start up to the `cache_control` marker. Changing one byte at position N invalidates the cache for everything from N onward. Consequence: content must be ordered **from most stable to most volatile**.\n\n```\n┌─ system ─────────────────────────────────── CACHED ─┐\n│ [0] SYSTEM_PROMPT   persona + rules, never changes  │\n│ [1] EXAMPLES        few-shot, never changes         │ ← cache_control\n└─────────────────────────────────────────────────────┘\n┌─ user ───────────────────────────────── NOT CACHED ─┐\n│ [0] CONTEXT         retrieved chunks, per question  │\n│ [1] QUESTION        the question, every turn        │\n└─────────────────────────────────────────────────────┘\n@Injectable()\nexport class PromptBuilder {\n  build(question: string, results: SearchResult[]): GenerateRequest {\n    const system: PromptBlock[] = [\n      { text: SYSTEM_PROMPT, cacheable: true },\n      { text: renderExamples(), cacheable: true },\n    ];\n\n    const user: PromptBlock[] = [\n      { text: `<context>\\n${this.renderContext(results)}\\n</context>` },\n      { text: `<question>${question}</question>` },\n    ];\n\n    return { system, user };\n  }\n\n  /** Number from 1 so the model cites [1], [2]... consistent with the few-shot. */\n  private renderContext(results: SearchResult[]): string {\n    if (!results.length) return '(no relevant documents found)';\n    return results\n      .map((r, i) => `[${i + 1}] ${r.title ? `${r.title}\\n` : ''}${r.content}`)\n      .join('\\n\\n');\n  }\n}\n```\n\n**Why is the retrieved context NOT in the cached region?** Each question retrieves a different set of chunks. Put it before the marker and every request creates a new cache entry that no later request can read. You end up paying the cache **write** premium (about 1.25x the input price) without ever getting the cache **read** price (about 0.1x or lower, depending on the model).\n\n**Cache economics:** with a 5-minute TTL, you break even on the second request (1.25 + 0.1 = 1.35 versus 2x without caching). A chatbot with a steady stream of users is the ideal case.\n\n**Minimum length:** a prefix shorter than the model's minimum cacheable length is not cached, and no error tells you so. The minimum varies by model (from a few hundred to a few thousand tokens). The system prompt + few-shot above is fairly short, so check `cacheWriteTokens` on the first request. If it is 0, the prefix is too short: add business rules, add few-shot examples, or accept running without cache.\n\nWhy `<context>` and `<question>` tags: XML tags mark a clear boundary between data and question, which helps the model avoid treating document content as instructions.\n\n```\n@Injectable()\nexport class ChatService {\n  constructor(\n    @Inject(LLM) private readonly llm: LlmService,\n    private readonly search: SearchService,\n    private readonly prompt: PromptBuilder,\n  ) {}\n\n  async ask(question: string): Promise<ChatResponse> {\n    const results = await this.search.hybridSearch(question);    // 1. retrieve\n    const request = this.prompt.build(question, results);        // 2. build prompt\n    const { text, usage } = await this.llm.generate(request);    // 3. generate\n\n    return {\n      answer: text,\n      citations: results.map((r, i) => ({\n        index: i + 1,            // matches [n] in the answer\n        chunkId: r.chunkId,\n        source: r.source,\n        title: r.title,\n      })),\n      usage,\n    };\n  }\n}\nclass AskDto {\n  @IsString()\n  @IsNotEmpty()\n  @MaxLength(2000)\n  question!: string;\n}\n\n@Controller('api')\nexport class ChatController {\n  constructor(\n    private readonly chat: ChatService,\n    private readonly search: SearchService,\n    private readonly repo: ChunkRepository,\n    private readonly db: DatabaseService,\n  ) {}\n\n  @Get('health')\n  async health() {\n    return { status: (await this.db.ping()) ? 'ok' : 'degraded' };\n  }\n\n  @Post('chat')\n  ask(@Body() dto: AskDto) {\n    return this.chat.ask(dto.question);\n  }\n\n  /** Inspect retrieval results directly, without the LLM. Useful when tuning. */\n  @Get('search')\n  search_(@Query('q') q: string, @Query('mode') mode?: string) {\n    if (mode === 'dense') return this.search.denseSearch(q);\n    if (mode === 'keyword') return this.search.keywordSearch(q);\n    return this.search.hybridSearch(q);\n  }\n\n  @Get('stats')\n  stats() {\n    return this.repo.stats();\n  }\n}\n```\n\nBesides validation, `@MaxLength(2000)` stops users from sending huge questions that inflate token costs.\n\n`main.ts`\n\n``` js\nimport 'dotenv/config';\n\nasync function bootstrap(): Promise<void> {\n  const app = await NestFactory.create(AppModule);\n  const config = app.get<AppConfig>(CONFIG);\n\n  app.useGlobalPipes(\n    new ValidationPipe({ whitelist: true, forbidNonWhitelisted: true, transform: true }),\n  );\n  // Let the pool close cleanly on SIGTERM instead of cutting off running queries.\n  app.enableShutdownHooks();\n\n  await app.listen(config.port);\n}\nbootstrap();\ncp .env.example .env      # fill in AWS credentials (or use AWS_PROFILE)\ndocker compose up -d      # Postgres + pgvector\nnpm install\nnpm run db:init           # create extension, table, indexes\n```\n\nPut a few markdown files in `data/in/`, for example `data/in/warranty.md`:\n\n```\n# Warranty Policy\n\n## Warranty period\nAll products are covered by a warranty of 12 months from the date of purchase.\n\n## Returns\nProducts with a manufacturing defect can be exchanged within the first 7 days.\nnpm run ingest\nnpm run start:dev\n```\n\n**Check retrieval before spending generation tokens.** Make this a habit from day one when tuning RAG:\n\n```\nnpm run search -- \"warranty 12 months\"            # hybrid, with per-branch ranks\nnpm run search -- \"warranty 12 months\" dense      # vector only\nnpm run search -- \"warranty 12 months\" keyword    # full-text only\n```\n\nHybrid output prints `rrf=...  dense=1  keyword=2  <source>` for each chunk, so you can see at a glance which branch a chunk came from.\n\n**Ask the chatbot:**\n\n```\ncurl -s -X POST http://localhost:8000/api/chat \\\n  -H 'Content-Type: application/json' \\\n  -d '{\"question\": \"How long is the product warranty?\"}'\n```\n\nThe response looks like this (illustrative values):\n\n```\n{\n  \"answer\": \"All products are covered by a 12-month warranty from the date of purchase [1].\",\n  \"citations\": [\n    { \"index\": 1, \"chunkId\": \"data/in/warranty.md#0\", \"source\": \"data/in/warranty.md\", \"title\": \"Warranty period\" },\n    { \"index\": 2, \"chunkId\": \"data/in/warranty.md#1\", \"source\": \"data/in/warranty.md\", \"title\": \"Returns\" }\n  ],\n  \"usage\": {\n    \"inputTokens\": 180,\n    \"outputTokens\": 25,\n    \"cacheWriteTokens\": 0,\n    \"cacheReadTokens\": 650\n  }\n}\n```\n\nSend the same question twice: the first request has `cacheWriteTokens > 0`, the second has `cacheReadTokens > 0`. If `cacheReadTokens` stays 0 across many requests, something is breaking the cache. It's usually a dynamic value that slipped into the system prompt, or a prefix below the minimum length.\n\nOther endpoints:\n\n`GET /api/search?q=...&mode=hybrid|dense|keyword`: inspect retrieval results.` GET /api/stats`: chunk count per source.` GET /api/health`: health check (used by Docker).\nThe chunker and RRF are pure functions, so their tests need no database and no AWS:\n\n```\nit('a chunk ranked well in both branches beats a chunk ranked first in only one', () => {\n  const results = reciprocalRankFusion(\n    {\n      dense: [candidate('dense-only'), candidate('both')],\n      keyword: [candidate('keyword-only'), candidate('both')],\n    },\n    60,\n    3,\n  );\n  expect(results[0].chunkId).toBe('both');\n});\n\nit('splits by heading and prepends the heading to chunk content', () => {\n  const chunks = makeChunker(1000).chunk({\n    source: 'doc.md',\n    text: '# Title\\n\\nIntro.\\n\\n## Part A\\n\\nContent A.\\n\\n## Part B\\n\\nContent B.',\n  });\n  expect(chunks).toHaveLength(3);\n  expect(chunks[1].title).toBe('Part A');\n  expect(chunks[1].content).toContain('Part A');\n});\nFROM node:20-alpine AS builder\nWORKDIR /app\nCOPY package*.json ./\nRUN npm ci\nCOPY tsconfig.json tsconfig.build.json nest-cli.json ./\nCOPY src ./src\nRUN npm run build\nRUN npm prune --omit=dev\n\nFROM node:20-alpine AS runtime\nWORKDIR /app\nENV NODE_ENV=production\nCOPY --from=builder --chown=node:node /app/node_modules ./node_modules\nCOPY --from=builder --chown=node:node /app/dist ./dist\nCOPY --chown=node:node package.json ./\nUSER node\nEXPOSE 8000\nHEALTHCHECK --interval=30s --timeout=5s --start-period=20s --retries=3 \\\n  CMD node -e \"require('http').get('http://127.0.0.1:'+(process.env.PORT||8000)+'/api/health',r=>process.exit(r.statusCode===200?0:1)).on('error',()=>process.exit(1))\"\nCMD [\"node\", \"dist/main\"]\n```\n\n`npm ci` layer is separate so the Docker cache survives source edits.`node` user, not root.`http`, so there's no need to install `curl`.\n`docker-compose.prod.yml` layers on top of the dev file, adds the `app` service, hides the Postgres port, and makes every sensitive value mandatory:\n\n```\nservices:\n  postgres:\n    environment:\n      POSTGRES_PASSWORD: ${DB_PASSWORD:?DB_PASSWORD is required}\n    ports: !reset []   # Postgres not exposed outside the network\n\n  app:\n    build: .\n    environment:\n      DB_HOST: postgres\n      DB_PASSWORD: ${DB_PASSWORD:?DB_PASSWORD is required}\n      BEDROCK_GENERATION_MODEL_ID: ${BEDROCK_GENERATION_MODEL_ID}\n      # ...\n    depends_on:\n      postgres:\n        condition: service_healthy\ndocker compose -f docker-compose.yml -f docker-compose.prod.yml up -d\n```\n\nThe `${VAR:?message}` syntax makes the deploy **fail immediately** when a variable is missing, instead of silently running with the default `postgres` password.\n\n```\nname: CI\non:\n  push:\n    branches: [main]\n  pull_request:\n\njobs:\n  check:\n    runs-on: ubuntu-latest\n    steps:\n      - uses: actions/checkout@v4\n      - uses: actions/setup-node@v4\n        with:\n          node-version: 20\n          cache: npm\n      - run: npm ci\n      - run: npm run lint\n      - run: npm test\n      - run: npm run build\n```\n\nNone of these throw an exception. The chatbot just answers worse, and the cause is very hard to track down.\n\n`search_document` for questions → recall drops a few percent, and no log tells you.`EMBEDDING_DIMENSIONS` and `vector(N)` → inserts fail (the lucky case, at least there's an error). Switching to a model with the same dimensions without re-embedding the corpus → old and new vectors share one index, results are meaningless, and `ề` has two valid encodings that look identical on screen but are not equal to a computer. Documents copied from PDFs or created on macOS are often NFD → full-text search doesn't match. Fix: `text.normalize('NFC')` at `kiểm` + `kê`, and also matches documents containing \"kê khai\" (declaration). Japanese has no spaces at all → the keyword branch is nearly useless; you need word segmentation in the application layer (MeCab, Sudachi) or an extension such as PGroonga.`cacheReadTokens`.\nBy this point, the system is already capable of answering questions over a document collection.\n\nThe next challenge is not functionality.\n\nThe next challenge is reliability.\n\nMost production RAG systems evolve through a series of small improvements rather than a single architectural rewrite.\n\nThe following upgrades are listed roughly in order of value delivered relative to implementation effort.\n\nThe codebase above is **Advanced RAG at its minimum**: hybrid search + RRF + header injection + prompt caching. When you put it in front of real customers, these are the next steps, ordered by benefit per unit of effort:\n\n`k` results even when the corpus has no answer, and that is the most common source of hallucination. Note that the RRF score `citations` as a list of `stop_reason`.` max_tokens` = the answer was truncated; some newer models may return `refusal`). Return an appropriate message instead of an empty or half-finished string.`InvokeModelWithResponseStreamCommand`) to reduce perceived waiting time.\nFor multi-tenant setups (several businesses on shared infrastructure), filtering by `tenant_id` from the schema up is a matter of **right/wrong**, not quality. Without it, data leaks between customers.\n\nAfter building several generations of RAG systems, one lesson keeps repeating itself:\n\nRAG is fundamentally a retrieval problem.\n\nMost production issues are not caused by the language model.\n\nThey come from:\n\nThe architecture in this article intentionally avoids unnecessary complexity.\n\nThere is no external vector database.\n\nNo orchestration framework.\n\nNo hidden abstraction layers.\n\nJust PostgreSQL, pgvector, Amazon Bedrock, Hybrid Search, and a clear separation between indexing, retrieval, and generation.\n\nFor teams involved in application software development, that simplicity is often an advantage rather than a limitation. A system that can be understood, measured, and debugged is significantly easier to improve than one hidden behind multiple abstraction layers.\n\nA system that can be understood, measured, and debugged is much easier to improve than one that depends on layers of abstraction.\n\nIf you're building a RAG chatbot today, invest in retrieval first.\n\nEverything else becomes easier afterwards.", "url": "https://wpnews.pro/news/what-actually-happens-when-you-build-a-production-rag-chatbot-on-amazon-bedrock", "canonical_source": "https://dev.to/hieu_le/what-actually-happens-when-you-build-a-production-rag-chatbot-on-amazon-bedrock-4l2", "published_at": "2026-10-23 07:00:47+00:00", "updated_at": "2026-10-23 07:11:39.107390+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "generative-ai", "ai-agents", "mlops"], "entities": ["Amazon Bedrock", "NestJS", "PostgreSQL", "pgvector", "Claude", "Cohere", "Relipa"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/what-actually-happens-when-you-build-a-production-rag-chatbot-on-amazon-bedrock", "markdown": "https://wpnews.pro/news/what-actually-happens-when-you-build-a-production-rag-chatbot-on-amazon-bedrock.md", "text": "https://wpnews.pro/news/what-actually-happens-when-you-build-a-production-rag-chatbot-on-amazon-bedrock.txt", "jsonld": "https://wpnews.pro/news/what-actually-happens-when-you-build-a-production-rag-chatbot-on-amazon-bedrock.jsonld"}}