How we built a document Q&A system with NestJS, PostgreSQL, pgvector, Hybrid Search (RRF), Prompt Caching, and Claude on Amazon Bedrock.
Building a demo chatbot is easy.
Building one that consistently answers correctly after weeks of document updates, changing requirements, and real user traffic is where things become interesting.
Most Retrieval-Augmented Generation (RAG) failures don't look like failures.
The API returns 200.
The model responds.
Infrastructure dashboards stay green.
Yet answer quality gradually gets worse.
You ask about error code E-4021 and receive documentation for E-4012.
A recently updated document never appears in search results.
Prompt caching is enabled, but costs don't go down.
Hybrid search is implemented, but retrieval quality barely improves.
Nothing crashes.
Nothing raises an alert.
The chatbot simply becomes less useful over time.
For teams involved in application software development, the real work begins after the first successful response. Retrieval quality, chunking strategy, embedding consistency, prompt caching, document lifecycle management, and hallucination control quickly become more important than model selection itself.
This article walks through a complete production-oriented RAG implementation built with:
More importantly, it explains why each component exists, what problem it solves, and which production issues it prevents.
RAG (Retrieval-Augmented Generation) does not ask the LLM directly. Before each question, the system:
The consequence of this design:
The problem "how does the LLM know my data" becomes a search problem. If retrieval is weak, the chatbot is weak, because no prompt can produce information that isn't in the context.
This distinction is easy to underestimate.
Many teams spend weeks experimenting with prompts while retrieval quality remains unchanged.
In reality, most enterprise application software development projects that incorporate AI succeed or fail based on retrieval architecture rather than prompt engineering.
That is why most of this article is about chunking documents and search, not about prompts.
Compared with the alternatives:
A RAG system has three stages:
The three stages of RAG: documents are indexed ahead of time, then each question goes through retrieval and then generation
ββ OFFLINE ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
data/in/*.md|txt|json
β
βΌ
Service βββΊ ChunkerService βββΊ Bedrock (Cohere Embed) βββΊ Postgres
read files 2-level split + input_type = chunks table
header injection search_document (vector + tsvector)
ββ ONLINE βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
POST /api/chat { question }
β
βΌ
SearchService.hybridSearch
ββ embedQuery (Cohere, input_type = search_query)
ββ DenseSearch (HNSW index) ββ run in parallel
ββ KeywordSearch (GIN index) ββ
ββ reciprocalRankFusion βββΊ top 5 chunks
β
βΌ
PromptBuilder ββ [system + few-shot: CACHED] + [context + question: not cached]
β
βΌ
Bedrock (Claude, InvokeModel)
β
βΌ
{ answer, citations, usage }
Folder layout, one folder per stage:
src/
βββ config/ all config, read from env
βββ database/ schema.sql + pg.Pool wrapper (no ORM)
βββ llm/ LlmService interface + Bedrock implementation
βββ ingest/ β chunker β repository β ingest.service
βββ search/ dense Β· keyword Β· rrf Β· search.service
βββ prompt/ system prompt Β· few-shot Β· prompt builder
βββ chat/ wires the three steps into one RAG loop + HTTP API
scripts/ db-init Β· ingest Β· search (CLI)
test/ unit tests for chunker, rrf, prompt builder
This architecture may look surprisingly simple.
That is intentional.
Every stage has a single responsibility and can be tested independently.
When retrieval quality drops, we can inspect retrieval.
When generation quality drops, we can inspect prompts.
When ingest breaks, we can inspect indexing.
The separation becomes extremely valuable once the system grows beyond a proof of concept.
The diagram in section 2 shows the blocks. This section slows down: it takes one concrete document and one concrete question and follows them through each function, each SQL query, and each Bedrock call. Once you have read this section, the code in the later steps is just detail.
The sample data used throughout is data/in/warranty.md:
## Warranty period
All products are covered by a warranty of 12 months from the date of purchase.
## Returns
Products with a manufacturing defect can be exchanged within the first 7 days.
This flow runs offline via npm run ingest, not over HTTP. Re-run it whenever documents change.
npm run ingest [-- <folder>]
β
βΌ
scripts/ingest.ts
β NestFactory.createApplicationContext(AppModule) (no HTTP port opened)
βΌ
IngestService.ingestDirectory(dir = INGEST_DIR)
β
βΌ
[1] Service.loadDirectory ββββββββββΊ RawDocument[]
β recursive scan; .md/.txt = 1 document; .json = n documents; other extensions skipped
β
β for EACH RawDocument:
βΌ
[2] ChunkerService.chunk βββββββββββββββββΊ Chunk[]
β split by heading, split long sections (1200 chars, 120 overlap), prepend heading
βΌ
[3] BedrockService.embedDocuments ββββββββΊ one 1024-dim vector per chunk
β up to 96 texts per call, Cohere input_type = search_document
βΌ
[4] ChunkRepository.upsert (per chunk) βββΊ chunks table
β INSERT ... ON CONFLICT (chunk_id) DO UPDATE
β Postgres computes search_vector and updates HNSW + GIN indexes
βΌ
[5] ChunkRepository.deleteStale ββββββββββΊ delete leftover chunks of this document
β
βΌ
IngestReport { documents, chunks, deleted } β printed to console, pool closed
[1] Read files. Service walks the folder recursively. A markdown file becomes one RawDocument:
{
"source": "data/in/warranty.md",
"title": "Warranty Policy",
"text": "# Warranty Policy\n\n## Warranty period\nAll products are covered..."
}
source is the file path, and title comes from the first H1 (or the file name if there is none). For .json files, each element with a text field becomes its own document.
[2] Chunk. splitByHeading reads line by line and starts a new section whenever a line begins with #. The sample file produces 3 sections:
section A heading = "Warranty Policy" text = "" β no chunk
section B heading = "Warranty period" text = "All products are..."
section C heading = "Returns" text = "Products with a..."
Section A has a heading but no body, so it is dropped. Sections B and C are shorter than 1200 characters, so they are kept whole. A longer section would be split further by splitLongText: paragraph boundaries first, then sentences, with 120 characters of overlap between consecutive pieces.
After prepending the heading (header injection), we get 2 chunks:
[
{
"chunkId": "data/in/warranty.md#0",
"title": "Warranty period",
"content": "Warranty period\n\nAll products are covered by a warranty of 12 months from the date of purchase.",
"metadata": { "heading": "Warranty period" }
},
{
"chunkId": "data/in/warranty.md#1",
"title": "Returns",
"content": "Returns\n\nProducts with a manufacturing defect can be exchanged within the first 7 days.",
"metadata": { "heading": "Returns" }
}
]
chunkId has the form <source>#<position>. It is the key that lets the next ingest update the existing row instead of adding a new one.
[3] Embed. Both content strings go to Cohere in a single InvokeModel call:
{ "texts": ["Warranty period\n\nAll products are...", "Returns\n\nProducts with a..."],
"input_type": "search_document", "truncate": "END" }
The result is 2 vectors of 1024 floats each. Because the heading is part of content, chunk #0's vector carries the meaning "warranty period", not just the sentence about 12 months.
[4] Store. Each chunk is one INSERT ... ON CONFLICT (chunk_id) DO UPDATE. The application only writes content and embedding. Postgres does the rest:
search_vector column is recomputed from title + content with to_tsvector('simple', ...). Chunk #0 yields the tokens warranty, period, all, products, are, covered, by, a, of, 12, months, from, the, date, purchase. embedding) and the GIN index (on [5] Clean up leftovers. deleteStale('data/in/warranty.md', ['...#0', '...#1']) deletes every row of this file whose chunk_id is not in the list just written. On the first run there is nothing to delete. If the file later shrinks from 5 chunks to 2, rows #2, #3 and #4 are deleted here.
At the end, the console prints a summary: number of documents, chunks, and stale chunks deleted.
Three things worth knowing about this flow:
metadata and skip chunks whose hash hasn't changed.source is the path exactly as scanned.npm run ingest (giving data/in/...) and later npm run ingest -- /absolute/path produces a different ChunkRepository.deleteBySource.
This flow runs online, once per request. One question costs 2 Bedrock calls (1 embed, 1 generate) and 2 SQL queries run in parallel.
Client
β POST /api/chat { "question": "How long is the product warranty?" }
βΌ
[1] ValidationPipe + AskDto ββββββββββββββΊ 400 if empty, > 2000 chars, or unknown fields
β
βΌ
ChatController.ask β ChatService.ask
β
βΌ
[2] SearchService.hybridSearch(question)
β
ββ [2a] BedrockService.embedQuery βββββΊ question vector (Cohere, input_type = search_query)
β
ββ [2b] in parallel (Promise.all):
β ββ DenseSearch : ORDER BY embedding <=> $vec LIMIT 50 β HNSW index
β ββ KeywordSearch : search_vector @@ websearch_to_tsquery β GIN index
β
ββ [2c] reciprocalRankFusion(k = 60) ββΊ top 5 chunks, with denseRank / keywordRank
β
βΌ
[3] PromptBuilder.build ββββββββββββββββββΊ { system: [...], user: [...] }
β system = SYSTEM_PROMPT + few-shot (cached)
β user = <context>[1]..[5]</context> + <question> (not cached)
βΌ
[4] BedrockService.generate ββββββββββββββΊ InvokeModel Claude
β cache_control on the few-shot block; retry on throttling
βΌ
[5] ChatService assembles the result
β answer = all "text" blocks joined
β citations = the chunks that were sent, numbered to match [n]
β usage = input/output tokens + cache write/read tokens
βΌ
Client receives { answer, citations, usage }
[1] Guard the input. ValidationPipe runs with whitelist and forbidNonWhitelisted, so the body may contain only the question field. AskDto requires question to be a non-empty string of at most 2000 characters. Invalid requests get a 400 immediately, before a single cent is spent on Bedrock.
[2a] Embed the question. The question goes to Cohere with input_type: "search_query". This mirrors step [3] of the ingest flow: documents use search_document, questions use search_query, same model. Get either one wrong and question vectors and document vectors no longer live on the same "map".
[2b] Two search branches run at the same time.
The dense branch compares the question vector with every chunk vector by cosine; thanks to HNSW it does not scan the whole table. With our 2-chunk sample corpus:
dense: 1. ...#0 "Warranty period" highest cosine
2. ...#1 "Returns"
The keyword branch passes the raw question to websearch_to_tsquery('simple', ...):
'how' & 'long' & 'is' & 'the' & 'product' & 'warranty'
This function joins words with AND. Chunk #0 contains none of how, long, is, and has products rather than product (the simple dictionary does not stem), so the keyword branch returns 0 results for this question. That is the real behavior of the code, and the reason the dense branch matters: for long natural-language questions the keyword branch usually stays silent. It shines on short queries containing codes or names, like E-4021 or warranty 12 months.
[2c] Merge with RRF. Each chunk gets 1 / (60 + rank) from every branch it appears in:
...#0 rrf = 1/61 = 0.01639 denseRank = 1 keywordRank = null
...#1 rrf = 1/62 = 0.01613 denseRank = 2 keywordRank = null
If the query were warranty 12 months, chunk #0 would appear in both branches, score 1/61 + 1/61 = 0.03279, and pull far ahead of chunks found by only one branch. The list is cut to the top 5 (TOP_K).
[3] Build the prompt. PromptBuilder numbers the chunks from 1 and wraps them in tags (the repository uses Vietnamese prompt text and tag names; English equivalents are shown throughout this article, see section 12):
<context>
[1] Warranty period
Warranty period
All products are covered by a warranty of 12 months from the date of purchase.
[2] Returns
Returns
Products with a manufacturing defect can be exchanged within the first 7 days.
</context>
<question>How long is the product warranty?</question>
The heading appears twice because title is printed first while content already contains the heading from header injection. It's harmless, but you can drop the title line when rendering to save tokens.
If neither branch finds anything, the context is (no relevant documents found) and the LLM is still called. The system prompt and few-shot example #2 teach the model to say it could not find the information instead of inventing an answer.
[4] Call Claude. The body sent via InvokeModel:
{
"anthropic_version": "bedrock-2023-05-31",
"max_tokens": 2000,
"system": [
{ "type": "text", "text": "You are a Q&A assistant that answers from the provided documents..." },
{ "type": "text", "text": "Examples of good answers: ...",
"cache_control": { "type": "ephemeral" } }
],
"messages": [
{ "role": "user", "content": [
{ "type": "text", "text": "<context>\n[1] Warranty period\n...\n</context>" },
{ "type": "text", "text": "<question>How long is the product warranty?</question>" }
] }
]
}
Everything before cache_control (system prompt + few-shot) is identical on every request, so from the second request on it is read from cache. messages changes with every question and is always billed at full price. If Bedrock returns ThrottlingException or ServiceUnavailableException, the request is retried after 500ms and then 1s; any other error is thrown immediately.
[5] Return the result. Claude returns a content array. The code keeps only type: "text" blocks and joins them, ignoring other block types. citations is the list of chunks put into the prompt, numbered to match [n] in the context, so the client can turn [1] in the answer into a link to source:
{
"answer": "All products are covered by a 12-month warranty from the date of purchase [1].",
"citations": [
{ "index": 1, "chunkId": "data/in/warranty.md#0", "source": "data/in/warranty.md", "title": "Warranty period" },
{ "index": 2, "chunkId": "data/in/warranty.md#1", "source": "data/in/warranty.md", "title": "Returns" }
],
"usage": { "inputTokens": 180, "outputTokens": 25, "cacheWriteTokens": 0, "cacheReadTokens": 650 }
}
(The usage values above are illustrative.) Note that citations contains every chunk that was sent, not only the ones the model actually cited. The client should filter by the [n] numbers that appear in answer.
Accuracy doesn't come from one place. It accumulates across both flows:
input_type:[n], and knows how to say "not found".
What these two flows don't do yet, covered in the "Upgrade path" section at the end: Unicode NFC normalization for both documents and questions, rewriting questions using conversation history (the API is currently stateless), a confidence threshold to skip the LLM when results are too weak, and checking in code that the [n] the model cites actually exist.
One design goal guided every technology choice: keep the architecture understandable.
Every component should be replaceable without rewriting the rest of the system.
useClass change.cohere.embed-multilingual-v3, 1024 dimensions): multilingual, which matters when documents mix a local language with English UI labels.pg, no ORM@aws-sdk/client-bedrock-runtime`` InvokeModel.
That's the entire runtime dependency list:
"dependencies": {
"@aws-sdk/client-bedrock-runtime": "^3.696.0",
"@nestjs/common": "^10.4.0",
"@nestjs/core": "^10.4.0",
"@nestjs/platform-express": "^10.4.0",
"class-transformer": "^0.5.1",
"class-validator": "^0.14.1",
"dotenv": "^16.4.5",
"pg": "^8.12.0",
"reflect-metadata": "^0.2.2",
"rxjs": "^7.8.1"
}
ap-northeast-1). anthropic. prefix, for example anthropic.claude-opus-5-5. Some regions require calling through a The application only needs bedrock:InvokeModel:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": ["bedrock:InvokeModel"],
"Resource": [
"arn:aws:bedrock:*::foundation-model/*",
"arn:aws:bedrock:*:*:inference-profile/*"
]
}
]
}
In production, narrow Resource down to the exact ARNs of the two models you use.
About credentials: static access keys work, but it's better to leave them empty and let the AWS SDK pick up AWS_PROFILE, SSO, or the EC2/ECS IAM role. The code below supports both.
npm i -g @nestjs/cli
nest new rag-chatbot --package-manager npm --skip-git
cd rag-chatbot
npm i @aws-sdk/client-bedrock-runtime pg dotenv class-validator class-transformer
npm i -D @types/pg ts-node
Add these scripts to package.json:
"scripts": {
"build": "nest build",
"start:dev": "nest start --watch",
"start:prod": "node dist/main",
"db:init": "ts-node scripts/db-init.ts",
"ingest": "ts-node scripts/ingest.ts",
"search": "ts-node scripts/search.ts",
"test": "jest",
"lint": "eslint \"{src,scripts,test}/**/*.ts\""
}
Every setting is read from env, with sensible defaults so it runs out of the box. No per-project profile files.
.env.example:
PORT=8000
DB_HOST=127.0.0.1
DB_PORT=5432
DB_NAME=ragdb
DB_USER=postgres
DB_PASSWORD=postgres
EMBEDDING_DIMENSIONS=1024
AWS_REGION=ap-northeast-1
AWS_ACCESS_KEY_ID=
AWS_SECRET_ACCESS_KEY=
AWS_SESSION_TOKEN=
BEDROCK_EMBEDDING_MODEL_ID=cohere.embed-multilingual-v3
BEDROCK_GENERATION_MODEL_ID=anthropic.claude-opus-5-5
INGEST_DIR=./data/in
CHUNK_MAX_CHARS=1200
CHUNK_OVERLAP_CHARS=120
EMBED_BATCH_SIZE=96
TOP_K=5 # number of chunks finally placed in the prompt
CANDIDATE_LIMIT=50 # candidates fetched by EACH branch before fusion
RRF_K=60 # RRF constant k, 60 is the value from the original paper
MAX_TOKENS=2000
ENABLE_PROMPT_CACHE=true
src/config/config.ts (abridged; the full version has an AppConfig interface documenting each field):
const num = (v: string | undefined, fallback: number) =>
v === undefined || v === '' ? fallback : Number(v);
export function loadConfig(): AppConfig {
return {
port: num(process.env.PORT, 8000),
database: {
host: process.env.DB_HOST || '127.0.0.1',
port: num(process.env.DB_PORT, 5432),
name: process.env.DB_NAME || 'ragdb',
user: process.env.DB_USER || 'postgres',
password: process.env.DB_PASSWORD || 'postgres',
embeddingDimensions: num(process.env.EMBEDDING_DIMENSIONS, 1024),
},
bedrock: {
region: process.env.AWS_REGION || 'ap-northeast-1',
accessKeyId: process.env.AWS_ACCESS_KEY_ID,
secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY,
sessionToken: process.env.AWS_SESSION_TOKEN,
embeddingModelId: process.env.BEDROCK_EMBEDDING_MODEL_ID || 'cohere.embed-multilingual-v3',
generationModelId: process.env.BEDROCK_GENERATION_MODEL_ID || 'anthropic.claude-opus-5-5',
},
ingest: {
dir: process.env.INGEST_DIR || './data/in',
chunkMaxChars: num(process.env.CHUNK_MAX_CHARS, 1200),
chunkOverlapChars: num(process.env.CHUNK_OVERLAP_CHARS, 120),
embedBatchSize: num(process.env.EMBED_BATCH_SIZE, 96),
},
search: {
topK: num(process.env.TOP_K, 5),
candidateLimit: num(process.env.CANDIDATE_LIMIT, 50),
rrfK: num(process.env.RRF_K, 60),
},
generation: {
maxTokens: num(process.env.MAX_TOKENS, 2000),
enableCache: process.env.ENABLE_PROMPT_CACHE !== 'false',
},
};
}
export const CONFIG = Symbol('CONFIG');
Register CONFIG as a global provider in AppConfigModule ({ provide: CONFIG, useFactory: loadConfig }) so every service can inject it with @Inject(CONFIG).
name: rag-core
services:
postgres:
image: pgvector/pgvector:pg16 # pgvector 0.7+, needed for the HNSW index
container_name: rag-postgres
environment:
POSTGRES_USER: ${DB_USER:-postgres}
POSTGRES_PASSWORD: ${DB_PASSWORD:-postgres}
POSTGRES_DB: ${DB_NAME:-ragdb}
ports:
- "${DB_PORT:-5432}:5432"
volumes:
- pgvector_data:/var/lib/postgresql/data
healthcheck:
test: ["CMD-SHELL", "pg_isready -U ${DB_USER:-postgres} -d ${DB_NAME:-ragdb}"]
interval: 5s
timeout: 5s
retries: 10
volumes:
pgvector_data:
name: rag-core is there for a reason: without it, Compose uses the folder name as the project name, and two checkouts with the same folder name on one machine will silently recreate each other's containers.
src/database/schema.sql:
CREATE EXTENSION IF NOT EXISTS vector;
CREATE TABLE IF NOT EXISTS chunks (
id BIGSERIAL PRIMARY KEY,
-- Logical key "<source>#<index>": re-ingesting the same file upserts the same row.
chunk_id TEXT NOT NULL UNIQUE,
source TEXT NOT NULL,
chunk_index INT NOT NULL,
title TEXT,
content TEXT NOT NULL,
metadata JSONB NOT NULL DEFAULT '{}'::jsonb,
embedding vector(1024),
-- Auto-generated full-text column, 'simple' dictionary.
search_vector tsvector GENERATED ALWAYS AS (
to_tsvector('simple', coalesce(title, '') || ' ' || content)
) STORED,
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
updated_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
);
-- Vector search: HNSW + cosine
CREATE INDEX IF NOT EXISTS chunks_embedding_idx
ON chunks USING hnsw (embedding vector_cosine_ops);
-- Full-text search
CREATE INDEX IF NOT EXISTS chunks_fts_idx
ON chunks USING gin (search_vector);
CREATE INDEX IF NOT EXISTS chunks_source_idx ON chunks (source);
Three decisions worth noting:
chunk_id is a unique logical key.content changes, so application code never has to keep it in sync.'simple' dictionary instead of 'english'. simple does no stemming and removes no language-specific stopwords. For languages like Vietnamese, the english config would chop word endings in nonsensical ways. scripts/db-init.ts reads the SQL file, swaps in the vector dimension from EMBEDDING_DIMENSIONS, and runs it. Safe to run any number of times (every statement is IF NOT EXISTS):
const sql = await fs.readFile(path.join(__dirname, '../src/database/schema.sql'), 'utf8');
await client.connect();
await client.query(sql.replace('vector(1024)', `vector(${config.database.embeddingDimensions})`));
await client.end();
pg.Pool
@Injectable()
export class DatabaseService implements OnModuleDestroy {
private readonly pool: Pool;
constructor(@Inject(CONFIG) config: AppConfig) {
this.pool = new Pool({
host: config.database.host,
port: config.database.port,
database: config.database.name,
user: config.database.user,
password: config.database.password,
});
}
async query<T extends QueryResultRow>(sql: string, params: unknown[] = []): Promise<T[]> {
const result = await this.pool.query<T>(sql, params);
return result.rows;
}
/** pgvector accepts a vector as the literal '[1,2,3]' cast to ::vector. */
static toVector(values: number[]): string {
return `[${values.join(',')}]`;
}
async onModuleDestroy(): Promise<void> {
await this.pool.end();
}
}
The rest of the codebase only knows this interface, not Bedrock:
export const LLM = Symbol('LLM');
export interface PromptBlock {
text: string;
/** true = block belongs to the static prefix and gets a cache_control marker. */
cacheable?: boolean;
}
export interface GenerateRequest {
system: PromptBlock[]; // persona, rules, few-shot: the cached part
user: PromptBlock[]; // retrieved context + question: changes every turn
maxTokens?: number;
}
export interface Usage {
inputTokens: number;
outputTokens: number;
cacheWriteTokens: number;
cacheReadTokens: number;
}
export interface LlmService {
embedDocuments(texts: string[]): Promise<number[][]>;
embedQuery(text: string): Promise<number[]>;
generate(request: GenerateRequest): Promise<{ text: string; usage: Usage }>;
}
// src/llm/llm.module.ts: switching providers = changing this one useClass line
@Global()
@Module({
providers: [{ provide: LLM, useClass: BedrockService }],
exports: [LLM],
})
export class LlmModule {}
@Injectable()
export class BedrockService implements LlmService {
private readonly client: BedrockRuntimeClient;
constructor(@Inject(CONFIG) private readonly config: AppConfig) {
const { region, accessKeyId, secretAccessKey, sessionToken } = config.bedrock;
this.client = new BedrockRuntimeClient({
region,
// Don't pass credentials when there are no static keys,
// so the SDK uses its own chain (AWS_PROFILE, SSO, instance role).
...(accessKeyId && secretAccessKey
? { credentials: { accessKeyId, secretAccessKey, sessionToken } }
: {}),
});
}
private async invoke(modelId: string, body: unknown): Promise<Record<string, any>> {
let lastError: unknown;
for (let attempt = 1; attempt <= 3; attempt++) {
try {
const response = await this.client.send(
new InvokeModelCommand({
modelId,
contentType: 'application/json',
accept: 'application/json',
body: JSON.stringify(body),
}),
);
return JSON.parse(new TextDecoder().decode(response.body));
} catch (err) {
lastError = err;
const name = (err as { name?: string }).name ?? '';
const retryable = name === 'ThrottlingException' || name === 'ServiceUnavailableException';
if (!retryable || attempt === 3) break;
await new Promise((r) => setTimeout(r, 2 ** attempt * 250)); // 500ms, 1s
}
}
throw lastError;
}
Only transient errors (throttling, service unavailable) are retried, with exponential backoff. A ValidationException or AccessDeniedException won't change no matter how many times you retry, so those are thrown immediately.
private async embed(
texts: string[],
inputType: 'search_document' | 'search_query',
): Promise<number[][]> {
const out: number[][] = [];
const batchSize = Math.min(this.config.ingest.embedBatchSize, 96); // Cohere: max 96 texts per call
for (let i = 0; i < texts.length; i += batchSize) {
const body = await this.invoke(this.config.bedrock.embeddingModelId, {
texts: texts.slice(i, i + batchSize),
input_type: inputType,
truncate: 'END',
});
out.push(...(body.embeddings as number[][]));
}
return out;
}
async embedDocuments(texts: string[]) {
return texts.length ? this.embed(texts, 'search_document') : [];
}
async embedQuery(text: string) {
const [embedding] = await this.embed([text], 'search_query');
return embedding;
}
input_type directly affects results. Cohere only projects documents and queries into the same space when each side declares its role correctly: documents are search_document, questions are search_query. Using the same value for both still runs and reports no error; recall just drops a few percent, and hardly anyone notices.
The body sent to Claude via InvokeModel follows the Anthropic Messages format, with anthropic_version: 'bedrock-2023-05-31':
type TextBlock = { type: 'text'; text: string; cache_control?: { type: 'ephemeral' } };
/** Put cache_control on the LAST block of the static prefix. */
private toBlocks(blocks: PromptBlock[]): TextBlock[] {
const out: TextBlock[] = blocks
.filter((b) => b.text.trim())
.map((b) => ({ type: 'text' as const, text: b.text }));
if (!this.config.generation.enableCache) return out;
const lastCacheable = blocks.reduce(
(acc, b, i) => (b.cacheable && b.text.trim() ? i : acc),
-1,
);
if (lastCacheable >= 0 && out[lastCacheable]) {
out[lastCacheable].cache_control = { type: 'ephemeral' };
}
return out;
}
async generate(request: GenerateRequest) {
const body = await this.invoke(this.config.bedrock.generationModelId, {
anthropic_version: 'bedrock-2023-05-31',
max_tokens: request.maxTokens ?? this.config.generation.maxTokens,
system: this.toBlocks(request.system),
messages: [{ role: 'user', content: this.toBlocks(request.user) }],
});
// Keep only 'text' blocks. Newer Claude models may return other block
// types (e.g. thinking); without this filter they leak into the answer.
const text = ((body.content ?? []) as Array<{ type: string; text?: string }>)
.filter((block) => block.type === 'text')
.map((block) => block.text ?? '')
.join('')
.trim();
const usage = body.usage ?? {};
return {
text,
usage: {
inputTokens: usage.input_tokens ?? 0,
outputTokens: usage.output_tokens ?? 0,
cacheWriteTokens: usage.cache_creation_input_tokens ?? 0,
cacheReadTokens: usage.cache_read_input_tokens ?? 0,
},
};
}
}
Why the cache marker goes on the last block of the static part: see Step 7.
The pipeline has four classes, each with one job:
Service βββΊ ChunkerService βββΊ LlmService.embedDocuments βββΊ ChunkRepository
(read sources) (chunking) (batches of 96) (upsert + clean up)
β²
IngestService only orchestrates, no logic of its own
export interface RawDocument {
source: string; // file path, URL, CMS id...
title?: string;
text: string;
metadata?: Record<string, unknown>;
}
Service scans the folder recursively:
.md / .txt: reads the whole file. Uses the first H1 as the title, or the file name if there is none..json: an array of objects with a text field (optional title,
async loadTextFile(file: string): Promise<RawDocument> {
const text = await fs.readFile(file, 'utf8');
return {
source: file,
title: /^#\s+(.+)$/m.exec(text)?.[1]?.trim() ?? path.basename(file),
text,
};
}
To add S3, Confluence, Notion and so on, add one method that returns RawDocument[]. Chunking and embedding stay the same.
This step has a big impact on quality but is often done carelessly.
The codebase's strategy:
CHUNK_MAX_CHARS``\n\n), then by sentence, with an
@Injectable()
export class ChunkerService {
constructor(@Inject(CONFIG) private readonly config: AppConfig) {}
chunk(doc: RawDocument): Chunk[] {
const { chunkMaxChars, chunkOverlapChars } = this.config.ingest;
const pieces: Array<{ heading?: string; text: string }> = [];
for (const section of this.splitByHeading(doc.text)) {
for (const part of this.splitLongText(section.text, chunkMaxChars, chunkOverlapChars)) {
pieces.push({ heading: section.heading, text: part });
}
}
return pieces.map((piece, index) => ({
chunkId: `${doc.source}#${index}`,
source: doc.source,
chunkIndex: index,
title: piece.heading ?? doc.title,
// Header injection: the heading goes into the embedding and FTS with the content.
content: piece.heading ? `${piece.heading}\n\n${piece.text}` : piece.text,
metadata: { ...(doc.metadata ?? {}), heading: piece.heading ?? null },
}));
}
private splitByHeading(text: string) {
const sections: Array<{ heading?: string; text: string[] }> = [];
let current: { heading?: string; text: string[] } = { text: [] };
for (const line of text.split('\n')) {
const heading = /^(#{1,6})\s+(.+)$/.exec(line);
if (heading) {
if (current.text.join('').trim() || current.heading) sections.push(current);
current = { heading: heading[2].trim(), text: [] };
} else {
current.text.push(line);
}
}
sections.push(current);
return sections
.map((s) => ({ heading: s.heading, text: s.text.join('\n').trim() }))
.filter((s) => s.text || s.heading);
}
private splitLongText(text: string, maxChars: number, overlapChars: number): string[] {
if (text.length <= maxChars) return text ? [text] : [];
// Prefer paragraph boundaries, then sentences (Japanese punctuation included).
const units = text
.split(/\n{2,}/)
.flatMap((p) => (p.length <= maxChars ? [p] : p.split(/(?<=[.!?γοΌοΌ])\s+/)))
.filter((u) => u.trim());
const out: string[] = [];
let buffer = '';
const flush = () => {
if (!buffer.trim()) return;
out.push(buffer.trim());
buffer = overlapChars > 0 ? buffer.slice(-overlapChars) : ''; // overlap
};
for (const unit of units) {
if (buffer && buffer.length + unit.length + 2 > maxChars) flush();
if (unit.length > maxChars) {
// The smallest unit is still too long, no boundary left to respect.
for (let i = 0; i < unit.length; i += maxChars - overlapChars) {
out.push(unit.slice(i, i + maxChars).trim());
}
buffer = '';
continue;
}
buffer = buffer ? `${buffer}\n\n${unit}` : unit;
}
if (buffer.trim()) out.push(buffer.trim());
return out.filter(Boolean);
}
}
If you can make only one chunking improvement, make it header injection: it costs almost nothing (string concatenation) and improves both the vector branch and the keyword branch.
A note on size: CHUNK_MAX_CHARS=1200 counts characters. For the same content, Vietnamese takes roughly 1.5x as many tokens as English, and Japanese roughly 3x. For Japanese documents, lower this limit so chunks stay within the embedding model's limit (anything beyond it is silently cut by truncate: 'END').
async upsert(chunk: Chunk, embedding: number[]): Promise<void> {
await this.db.query(
`
INSERT INTO chunks (chunk_id, source, chunk_index, title, content, metadata, embedding)
VALUES ($1, $2, $3, $4, $5, $6::jsonb, $7::vector)
ON CONFLICT (chunk_id) DO UPDATE SET
source = EXCLUDED.source, chunk_index = EXCLUDED.chunk_index,
title = EXCLUDED.title, content = EXCLUDED.content,
metadata = EXCLUDED.metadata, embedding = EXCLUDED.embedding,
updated_at = NOW()
`,
[chunk.chunkId, chunk.source, chunk.chunkIndex, chunk.title ?? null,
chunk.content, JSON.stringify(chunk.metadata), DatabaseService.toVector(embedding)],
);
}
async deleteStale(source: string, keepChunkIds: string[]): Promise<number> {
const rows = await this.db.query<{ chunk_id: string }>(
`DELETE FROM chunks WHERE source = $1 AND chunk_id <> ALL($2::text[]) RETURNING chunk_id`,
[source, keepChunkIds],
);
return rows.length;
}
Upsert alone is not enough for safe re-ingestion. Say a document used to have 10 chunks and the new version has 6. Upsert updates the first 6 rows, but the last 4 stay there with their old embeddings and keep being retrieved. The chatbot ends up answering from content that no longer exists.
deleteStale deletes by exclusion after the upsert finishes, instead of "delete everything, then write again". That way search never sees a window where the document is missing.
Abridged version (in the repo, embed + upsert live in a separate embedAndStore method):
async ingestDocuments(docs: RawDocument[]): Promise<IngestReport> {
const report = { documents: 0, chunks: 0, deleted: 0 };
for (const doc of docs) {
const chunks = this.chunker.chunk(doc);
if (!chunks.length) continue;
// Embed in batches, then upsert each chunk
for (let i = 0; i < chunks.length; i += this.config.ingest.embedBatchSize) {
const batch = chunks.slice(i, i + this.config.ingest.embedBatchSize);
const embeddings = await this.llm.embedDocuments(batch.map((c) => c.content));
for (const [j, chunk] of batch.entries()) await this.repo.upsert(chunk, embeddings[j]);
}
report.deleted += await this.repo.deleteStale(doc.source, chunks.map((c) => c.chunkId));
report.documents += 1;
report.chunks += chunks.length;
}
return report;
}
The CLI script scripts/ingest.ts boots Nest as an application context (no HTTP server) and calls the service:
const app = await NestFactory.createApplicationContext(AppModule, { logger: ['log', 'warn', 'error'] });
const report = await app.get(IngestService).ingestDirectory(process.argv[2]);
console.log(`done: ${report.documents} documents, ${report.chunks} chunks, ${report.deleted} stale chunks deleted`);
await app.close();
Each branch fails exactly where the other is strong:
E-4021 and E-4012 are almost identical in embedding space.
You combine the two because they cover for each other, not because more branches are always better.
async search(queryVector: number[], limit: number): Promise<Candidate[]> {
const rows = await this.db.query<any>(
`
SELECT chunk_id AS "chunkId", source, title, content, metadata,
1 - (embedding <=> $1::vector) AS score
FROM chunks
WHERE embedding IS NOT NULL
ORDER BY embedding <=> $1::vector
LIMIT $2::int
`,
[DatabaseService.toVector(queryVector), limit],
);
return rows.map((r) => ({ ...r, score: Number(r.score) }));
}
<=> is pgvector's cosine distance operator, so 1 - distance is cosine similarity. The query deliberately has just ORDER BY embedding <=> $1 LIMIT n, because that is the shape the planner needs to use the HNSW index. If you add filters, check again with EXPLAIN, since it is very easy to turn this into a full table scan.
async search(queryText: string, limit: number): Promise<Candidate[]> {
const rows = await this.db.query<any>(
`
WITH tsq AS (SELECT websearch_to_tsquery('simple', $1) AS q)
SELECT c.chunk_id AS "chunkId", c.source, c.title, c.content, c.metadata,
ts_rank(c.search_vector, tsq.q) AS score
FROM chunks c, tsq
WHERE c.search_vector @@ tsq.q
ORDER BY score DESC
LIMIT $2::int
`,
[queryText, limit],
);
return rows.map((r) => ({ ...r, score: Number(r.score) }));
}
websearch_to_tsquery accepts the syntax users already know ("exact phrase", a OR b, -exclude) and ts_rank is The "obvious" approach is score = 0.7 Γ cosine + 0.3 Γ ts_rank. It's wrong:
[0, 1], while 0.0x. Add them directly and the keyword branch contributes almost nothing. You think hybrid search is running, but it is really just vector search.
Reciprocal Rank Fusion (RRF) throws away the raw scores and uses only ranks:
RRF(d) = Ξ£ 1 / (k + rank_branch(d)) k = 60
branch
A worked example:
Dense: 1. chunk_A 2. chunk_B 3. chunk_C
Keyword: 1. chunk_C 2. chunk_A 3. chunk_D
chunk_A: 1/61 + 1/62 = 0.03252 β rank 1
chunk_C: 1/63 + 1/61 = 0.03226 β rank 2
chunk_B: 1/62 = 0.01613
chunk_D: 1/63 = 0.01587
chunk_A wins without being first in either branch, because it ranks well in both. Agreement between two independent methods is a stronger signal than a high score from one.
The constant k flattens the curve: with k = 60, the gap between rank 1 and rank 2 is only about 1.6%, so appearing in several lists matters more than the exact rank.
export function reciprocalRankFusion(
branches: { dense: Candidate[]; keyword: Candidate[] },
k = 60,
topK = 5,
): SearchResult[] {
const byId = new Map<string, SearchResult>();
const ingest = (candidates: Candidate[], branch: 'denseRank' | 'keywordRank') => {
candidates.forEach((candidate, i) => {
const rank = i + 1;
const existing = byId.get(candidate.chunkId);
if (existing) {
existing.rrfScore += 1 / (k + rank);
existing[branch] = rank;
return;
}
const { score: _ignored, ...rest } = candidate;
byId.set(candidate.chunkId, {
...rest,
rrfScore: 1 / (k + rank),
denseRank: null,
keywordRank: null,
[branch]: rank,
} as SearchResult);
});
};
ingest(branches.dense, 'denseRank');
ingest(branches.keyword, 'keywordRank');
return [...byId.values()].sort((a, b) => b.rrfScore - a.rrfScore).slice(0, topK);
}
The result keeps each chunk's denseRank and keywordRank. When debugging, one look tells you whether a result came from semantics, keywords, or both.
async hybridSearch(query: string, options: SearchOptions = {}): Promise<SearchResult[]> {
const topK = options.topK ?? this.config.search.topK; // 5
const limit = options.candidateLimit ?? this.config.search.candidateLimit; // 50
const queryVector = await this.llm.embedQuery(query);
const [dense, keyword] = await Promise.all([
this.dense.search(queryVector, limit),
this.keyword.search(query, limit),
]);
return reciprocalRankFusion({ dense, keyword }, this.config.search.rrfK, topK);
}
Two details decide the quality:
Adding a third branch (multi-query, or real BM25) is just one more array passed to RRF. No score normalization, no weight tuning.
The repository ships this prompt in Vietnamese (with <ngu_canh> / <cau_hoi> tags); the English equivalent is shown here.
export const SYSTEM_PROMPT = `You are a Q&A assistant that answers from the provided documents.
Answering rules:
- Answer only from the CONTEXT provided. Do not guess, and do not use knowledge
outside the context.
- If the context is not enough to answer, say plainly that the information was
not found. A wrong answer is more dangerous than "I don't know".
- Cite sources with [number] matching the position of the context passage you
relied on.
- Answer in the same language as the question.
- Be concise and get straight to the point. Do not start with "Based on the documents...".`;
Three rules for this file:
if (x) prompt += ...). Every combination of conditions is a different prefix, i.e. a different cache entry.
The rule "answer in the same language as the question" is essential: when documents are in English and the user asks in another language, without this rule the model tends to answer in the language of the context.
export const EXAMPLES: Example[] = [
{
question: 'How long is the product warranty?',
context:
'[1] Warranty policy\nAll products are covered by a 12-month warranty from the date of purchase.\n\n' +
'[2] Returns\nDefective products can be exchanged within the first 7 days.',
answer: 'Products are covered by a 12-month warranty from the date of purchase [1].',
},
{
// Negative example: there is context, but no passage answers the question.
question: 'Does the company have a branch in Da Nang?',
context: '[1] Warranty policy\nAll products are covered by a 12-month warranty from the date of purchase.',
answer: 'The available documents do not mention any branch information.',
},
];
Examples are the strongest signal in a prompt: the [1] citation format is easy to misread when described in words, but obvious from a single example.
The refusal example matters as much as the answer example. If every example is a successful answer, the model learns that there must always be an answer. Example #2 teaches it to decline when the context is irrelevant.
Prompt caching is a prefix match: the system hashes the prompt from the start up to the cache_control marker. Changing one byte at position N invalidates the cache for everything from N onward. Consequence: content must be ordered from most stable to most volatile.
ββ system βββββββββββββββββββββββββββββββββββ CACHED ββ
β [0] SYSTEM_PROMPT persona + rules, never changes β
β [1] EXAMPLES few-shot, never changes β β cache_control
βββββββββββββββββββββββββββββββββββββββββββββββββββββββ
ββ user βββββββββββββββββββββββββββββββββ NOT CACHED ββ
β [0] CONTEXT retrieved chunks, per question β
β [1] QUESTION the question, every turn β
βββββββββββββββββββββββββββββββββββββββββββββββββββββββ
@Injectable()
export class PromptBuilder {
build(question: string, results: SearchResult[]): GenerateRequest {
const system: PromptBlock[] = [
{ text: SYSTEM_PROMPT, cacheable: true },
{ text: renderExamples(), cacheable: true },
];
const user: PromptBlock[] = [
{ text: `<context>\n${this.renderContext(results)}\n</context>` },
{ text: `<question>${question}</question>` },
];
return { system, user };
}
/** Number from 1 so the model cites [1], [2]... consistent with the few-shot. */
private renderContext(results: SearchResult[]): string {
if (!results.length) return '(no relevant documents found)';
return results
.map((r, i) => `[${i + 1}] ${r.title ? `${r.title}\n` : ''}${r.content}`)
.join('\n\n');
}
}
Why is the retrieved context NOT in the cached region? Each question retrieves a different set of chunks. Put it before the marker and every request creates a new cache entry that no later request can read. You end up paying the cache write premium (about 1.25x the input price) without ever getting the cache read price (about 0.1x or lower, depending on the model).
Cache economics: with a 5-minute TTL, you break even on the second request (1.25 + 0.1 = 1.35 versus 2x without caching). A chatbot with a steady stream of users is the ideal case.
Minimum length: a prefix shorter than the model's minimum cacheable length is not cached, and no error tells you so. The minimum varies by model (from a few hundred to a few thousand tokens). The system prompt + few-shot above is fairly short, so check cacheWriteTokens on the first request. If it is 0, the prefix is too short: add business rules, add few-shot examples, or accept running without cache.
Why <context> and <question> tags: XML tags mark a clear boundary between data and question, which helps the model avoid treating document content as instructions.
@Injectable()
export class ChatService {
constructor(
@Inject(LLM) private readonly llm: LlmService,
private readonly search: SearchService,
private readonly prompt: PromptBuilder,
) {}
async ask(question: string): Promise<ChatResponse> {
const results = await this.search.hybridSearch(question); // 1. retrieve
const request = this.prompt.build(question, results); // 2. build prompt
const { text, usage } = await this.llm.generate(request); // 3. generate
return {
answer: text,
citations: results.map((r, i) => ({
index: i + 1, // matches [n] in the answer
chunkId: r.chunkId,
source: r.source,
title: r.title,
})),
usage,
};
}
}
class AskDto {
@IsString()
@IsNotEmpty()
@MaxLength(2000)
question!: string;
}
@Controller('api')
export class ChatController {
constructor(
private readonly chat: ChatService,
private readonly search: SearchService,
private readonly repo: ChunkRepository,
private readonly db: DatabaseService,
) {}
@Get('health')
async health() {
return { status: (await this.db.ping()) ? 'ok' : 'degraded' };
}
@Post('chat')
ask(@Body() dto: AskDto) {
return this.chat.ask(dto.question);
}
/** Inspect retrieval results directly, without the LLM. Useful when tuning. */
@Get('search')
search_(@Query('q') q: string, @Query('mode') mode?: string) {
if (mode === 'dense') return this.search.denseSearch(q);
if (mode === 'keyword') return this.search.keywordSearch(q);
return this.search.hybridSearch(q);
}
@Get('stats')
stats() {
return this.repo.stats();
}
}
Besides validation, @MaxLength(2000) stops users from sending huge questions that inflate token costs.
main.ts
import 'dotenv/config';
async function bootstrap(): Promise<void> {
const app = await NestFactory.create(AppModule);
const config = app.get<AppConfig>(CONFIG);
app.useGlobalPipes(
new ValidationPipe({ whitelist: true, forbidNonWhitelisted: true, transform: true }),
);
// Let the pool close cleanly on SIGTERM instead of cutting off running queries.
app.enableShutdownHooks();
await app.listen(config.port);
}
bootstrap();
cp .env.example .env # fill in AWS credentials (or use AWS_PROFILE)
docker compose up -d # Postgres + pgvector
npm install
npm run db:init # create extension, table, indexes
Put a few markdown files in data/in/, for example data/in/warranty.md:
## Warranty period
All products are covered by a warranty of 12 months from the date of purchase.
## Returns
Products with a manufacturing defect can be exchanged within the first 7 days.
npm run ingest
npm run start:dev
Check retrieval before spending generation tokens. Make this a habit from day one when tuning RAG:
npm run search -- "warranty 12 months" # hybrid, with per-branch ranks
npm run search -- "warranty 12 months" dense # vector only
npm run search -- "warranty 12 months" keyword # full-text only
Hybrid output prints rrf=... dense=1 keyword=2 <source> for each chunk, so you can see at a glance which branch a chunk came from.
Ask the chatbot:
curl -s -X POST http://localhost:8000/api/chat \
-H 'Content-Type: application/json' \
-d '{"question": "How long is the product warranty?"}'
The response looks like this (illustrative values):
{
"answer": "All products are covered by a 12-month warranty from the date of purchase [1].",
"citations": [
{ "index": 1, "chunkId": "data/in/warranty.md#0", "source": "data/in/warranty.md", "title": "Warranty period" },
{ "index": 2, "chunkId": "data/in/warranty.md#1", "source": "data/in/warranty.md", "title": "Returns" }
],
"usage": {
"inputTokens": 180,
"outputTokens": 25,
"cacheWriteTokens": 0,
"cacheReadTokens": 650
}
}
Send the same question twice: the first request has cacheWriteTokens > 0, the second has cacheReadTokens > 0. If cacheReadTokens stays 0 across many requests, something is breaking the cache. It's usually a dynamic value that slipped into the system prompt, or a prefix below the minimum length.
Other endpoints:
GET /api/search?q=...&mode=hybrid|dense|keyword: inspect retrieval results. GET /api/stats: chunk count per source. GET /api/health: health check (used by Docker).
The chunker and RRF are pure functions, so their tests need no database and no AWS:
it('a chunk ranked well in both branches beats a chunk ranked first in only one', () => {
const results = reciprocalRankFusion(
{
dense: [candidate('dense-only'), candidate('both')],
keyword: [candidate('keyword-only'), candidate('both')],
},
60,
3,
);
expect(results[0].chunkId).toBe('both');
});
it('splits by heading and prepends the heading to chunk content', () => {
const chunks = makeChunker(1000).chunk({
source: 'doc.md',
text: '# Title\n\nIntro.\n\n## Part A\n\nContent A.\n\n## Part B\n\nContent B.',
});
expect(chunks).toHaveLength(3);
expect(chunks[1].title).toBe('Part A');
expect(chunks[1].content).toContain('Part A');
});
FROM node:20-alpine AS builder
WORKDIR /app
COPY package*.json ./
RUN npm ci
COPY tsconfig.json tsconfig.build.json nest-cli.json ./
COPY src ./src
RUN npm run build
RUN npm prune --omit=dev
FROM node:20-alpine AS runtime
WORKDIR /app
ENV NODE_ENV=production
COPY --from=builder --chown=node:node /app/node_modules ./node_modules
COPY --from=builder --chown=node:node /app/dist ./dist
COPY --chown=node:node package.json ./
USER node
EXPOSE 8000
HEALTHCHECK --interval=30s --timeout=5s --start-period=20s --retries=3 \
CMD node -e "require('http').get('http://127.0.0.1:'+(process.env.PORT||8000)+'/api/health',r=>process.exit(r.statusCode===200?0:1)).on('error',()=>process.exit(1))"
CMD ["node", "dist/main"]
npm ci layer is separate so the Docker cache survives source edits.node user, not root.http, so there's no need to install curl.
docker-compose.prod.yml layers on top of the dev file, adds the app service, hides the Postgres port, and makes every sensitive value mandatory:
services:
postgres:
environment:
POSTGRES_PASSWORD: ${DB_PASSWORD:?DB_PASSWORD is required}
ports: !reset [] # Postgres not exposed outside the network
app:
build: .
environment:
DB_HOST: postgres
DB_PASSWORD: ${DB_PASSWORD:?DB_PASSWORD is required}
BEDROCK_GENERATION_MODEL_ID: ${BEDROCK_GENERATION_MODEL_ID}
depends_on:
postgres:
condition: service_healthy
docker compose -f docker-compose.yml -f docker-compose.prod.yml up -d
The ${VAR:?message} syntax makes the deploy fail immediately when a variable is missing, instead of silently running with the default postgres password.
name: CI
on:
push:
branches: [main]
pull_request:
jobs:
check:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-node@v4
with:
node-version: 20
cache: npm
- run: npm ci
- run: npm run lint
- run: npm test
- run: npm run build
None of these throw an exception. The chatbot just answers worse, and the cause is very hard to track down.
search_document for questions β recall drops a few percent, and no log tells you.EMBEDDING_DIMENSIONS and vector(N) β inserts fail (the lucky case, at least there's an error). Switching to a model with the same dimensions without re-embedding the corpus β old and new vectors share one index, results are meaningless, and α» has two valid encodings that look identical on screen but are not equal to a computer. Documents copied from PDFs or created on macOS are often NFD β full-text search doesn't match. Fix: text.normalize('NFC') at kiα»m + kΓͺ, and also matches documents containing "kΓͺ khai" (declaration). Japanese has no spaces at all β the keyword branch is nearly useless; you need word segmentation in the application layer (MeCab, Sudachi) or an extension such as PGroonga.cacheReadTokens.
By this point, the system is already capable of answering questions over a document collection.
The next challenge is not functionality.
The next challenge is reliability.
Most production RAG systems evolve through a series of small improvements rather than a single architectural rewrite.
The following upgrades are listed roughly in order of value delivered relative to implementation effort.
The codebase above is Advanced RAG at its minimum: hybrid search + RRF + header injection + prompt caching. When you put it in front of real customers, these are the next steps, ordered by benefit per unit of effort:
k results even when the corpus has no answer, and that is the most common source of hallucination. Note that the RRF score citations as a list of stop_reason. max_tokens = the answer was truncated; some newer models may return refusal). Return an appropriate message instead of an empty or half-finished string.InvokeModelWithResponseStreamCommand) to reduce perceived waiting time.
For multi-tenant setups (several businesses on shared infrastructure), filtering by tenant_id from the schema up is a matter of right/wrong, not quality. Without it, data leaks between customers.
After building several generations of RAG systems, one lesson keeps repeating itself:
RAG is fundamentally a retrieval problem.
Most production issues are not caused by the language model.
They come from:
The architecture in this article intentionally avoids unnecessary complexity.
There is no external vector database.
No orchestration framework.
No hidden abstraction layers.
Just PostgreSQL, pgvector, Amazon Bedrock, Hybrid Search, and a clear separation between indexing, retrieval, and generation.
For teams involved in application software development, that simplicity is often an advantage rather than a limitation. A system that can be understood, measured, and debugged is significantly easier to improve than one hidden behind multiple abstraction layers.
A system that can be understood, measured, and debugged is much easier to improve than one that depends on layers of abstraction.
If you're building a RAG chatbot today, invest in retrieval first.
Everything else becomes easier afterwards.