In the previous article, we built a working RAG pipeline. Now let's step back and ask why we made each design decision — and what alternatives exist when your requirements change.
Here's what we built:
Ingest phase
Text → gemini-embedding-001 (RETRIEVAL_DOCUMENT, 768 dims)
→ pgvector (HNSW index, cosine similarity)
Query phase
Question → gemini-embedding-001 (RETRIEVAL_QUERY, 768 dims)
→ pgvector search (top-k)
→ Gemini 2.5 Flash (answer generation)
Every element in this diagram was a choice. Let's examine each one.
We used pgvector, a PostgreSQL extension, rather than a purpose-built vector database like Pinecone, Weaviate, or Qdrant.
Why pgvector works here:
category
, join with other tables, all in one round-tripWhen to consider a dedicated vector DB:
| Signal | Consider moving to |
|---|---|
| > 10M documents | Pinecone, Weaviate |
| Multi-modal search (text + image) | Weaviate, Qdrant |
| Managed cloud with SLA | Pinecone |
| On-premise, full control | Qdrant |
For most enterprise RAG applications at typical document volumes, pgvector is the right starting point. Migrate when you hit actual limits, not anticipated ones.
gemini-embedding-001
outputs 3072 dimensions by default. We set output_dimensionality=768
.
The constraint: pgvector's HNSW index has a hard limit of 2000 dimensions.
Why not 2000? We chose 768 because:
Dimension vs. quality trade-off:
| Dimensions | Index build | Storage | Retrieval quality |
|---|---|---|---|
| 256 | Fastest | Smallest | Noticeably lower |
| 768 | Fast | Small | Near full quality |
| 1536 | Moderate | Moderate | Full quality |
| 3072 | Slow | Largest | Full quality (no HNSW) |
task_type
We used different task_type
values for ingestion and querying:
config=types.EmbedContentConfig(task_type="RETRIEVAL_DOCUMENT", ...)
config=types.EmbedContentConfig(task_type="RETRIEVAL_QUERY", ...)
Why this matters: Gemini's embedding model is trained with asymmetric objectives. A document and a query about the same topic are represented differently in embedding space — the model learns to map queries toward relevant documents, not to the same point. Using the same task type for both degrades retrieval accuracy.
This is analogous to how you'd phrase a document differently from a search query in natural language: "F1 Score is the harmonic mean of Precision and Recall" (document) vs. "how to calculate F1" (query).
pgvector supports two index types. We chose HNSW.
| HNSW | IVFFlat | |
|---|---|---|
| Query speed | Fast | Moderate |
| Build time | Moderate | Fast |
| Memory | Higher | Lower |
| Accuracy at scale | Higher | Lower |
| Requires training data | No | Yes (needs VACUUM after inserts) |
HNSW is the better default for production. IVFFlat is worth considering only when you have very tight memory constraints and can afford slower queries.
HNSW parameter guide:
WITH (
m = 16, -- max connections per node
ef_construction = 64 -- search width during build
)
m
: higher = better recall, more memory. Range: 4–64. Default 16 works for most cases.ef_construction
: higher = better index quality, slower build. Range: 16–512. Default 64 is a good production starting point.We used gemini-2.5-flash
rather than the more capable gemini-opus
models.
Reasoning:
When to upgrade the generation model:
When to upgrade the embedding model:
The embedding model matters more for retrieval quality. The generation model matters more for answer quality. Optimize them independently.
This architecture scales predictably:
Phase 1 (now): pgvector local → works to ~1M docs
Phase 2: pgvector + Supabase → managed PostgreSQL, easy scaling
Phase 3: pgvector + read replicas → horizontal query scaling
Phase 4: Dedicated vector DB → if you genuinely outgrow pgvector
Most teams never reach Phase 4. Start at Phase 1, move when you have evidence you need to.
Chunking strategy matters more than model choice. If your documents are long (PDFs, reports), how you split them into chunks dramatically affects retrieval quality. A naive split at 512 tokens often breaks context mid-sentence. Consider semantic chunking or overlap.
Don't embed the question alone. For complex questions, consider HyDE (Hypothetical Document Embedding): generate a hypothetical answer to the question, embed that, then search. This often retrieves better documents than embedding the raw question.
Reranking improves precision. After vector search returns top-k candidates, a cross-encoder reranker (like Cohere Rerank) re-scores them for precision. Add this when recall is good but final answer quality is inconsistent.
In the next article, we'll give the LLM the ability to call these search functions autonomously using Tool Use.
Full source code: github.com/qameqame/pgvector-tutorial