RAG projects have a way of collecting infrastructure before they collect evidence.
A database gets provisioned. A vector store appears. Then Redis, object storage, a parser service, a queue worker, and a few dashboards. By the time the first PDF is imported, there are enough moving parts that a bad search result can mean almost anything.
Was the parser wrong? Did chunking lose the useful context? Was the vector index empty? Did the query just not match the document?
For early work, I would rather make the problem smaller. Use a handful of documents people already depend on. Ask questions they have actually asked before. Look at the passages returned by search and decide whether they are useful. That is what I mean by starting local.
A knowledge base becomes interesting once it contains the annoying files: a PDF with a table in the middle, a spreadsheet someone edited last quarter, a product manual with the same heading on every page, a contract with a number that people need to find exactly.
Those files are much more informative than a clean benchmark dataset.
When I test retrieval, I usually want a short list of questions with different failure modes:
The last question matters more than it sounds. A system that returns nothing useful should make that clear. Otherwise, people start blaming the model for an answer that retrieval never supported in the first place.
I also want every result to point back somewhere specific: the source document, its revision, and a page, row, or other usable anchor. “The search found something” is not enough when someone needs to check it.
Chinese is often where a retrieval setup reveals what it has been getting away with.
Vector search can be good at questions with enough semantic context. It is less reassuring when the query is a product code, a contract number, a person’s name, or an internal abbreviation. Those queries are often better served by lexical matching.
The opposite problem is also real. Keyword search alone does poorly when the source and the question use different wording.
That is why I prefer hybrid retrieval: keep a semantic path and a lexical path, then let both participate in recall.
The local version does not need to look exactly like production. SQLite with FTS5, a vector extension, and Chinese tokenization can be enough to test the idea. A larger deployment might move to PostgreSQL, pgvector, and a Chinese full-text extension. The components change, but the question stays the same: can this query find the passage a person would expect to see?
I wrote more about that trade-off in this hybrid-search guide. I do not think SQLite is a substitute for a production database, and I would not run a shared, business-critical knowledge service from a developer machine.
Local mode has a different job. It lets a team test document processing and retrieval before the environment becomes the main thing being tested.
A single binary with a local database is useful when you want to import some files, try a few searches, reset the library, and repeat. There is no need to bring up PostgreSQL, Redis, Docker, and object storage just to learn that the documents were chunked badly.
The boundary changes when people start relying on the system:
At that point, adding production infrastructure is not overengineering. It is responding to an operating need you can now name.
Once local retrieval is working, connecting it to an agent is usually simpler than building a chat interface around it.
The knowledge service can expose search and document operations over an API or MCP. The agent asks for relevant evidence when it needs it; it does not carry a whole library through every conversation.
That distinction matters when a second agent needs the same documents. Copying the files into another application may be quick, but it creates a second versioning and retrieval problem. Sharing a retrieval boundary is often the cleaner move.
I have been building Langhuan around that boundary: document ingestion and retrieval on one side, agents and application workflows on the other. Its standalone mode uses SQLite for local validation, while the production path can use PostgreSQL and Redis when the workload earns them.
If I were starting a new RAG project tomorrow, I would begin with five messy documents and a short list of real questions. I would not add the next service until I could explain which problem it solves.