Building an AI Document Intelligence System: Architecture, LangChain, and Production Lessons

How I built a provider-agnostic RAG platform for document Q&A - and what years of delivering document-heavy digital services taught me about doing it properly.

Every organisation I've worked with - public sector, regulated enterprise, or otherwise - has the same quiet bottleneck: information locked inside PDFs, DOCX files, and scanned reports that nobody has time to read in full. A caseworker needs one clause from a 40-page policy document. An analyst needs a number buried on page 22 of a compliance report. The document exists, technically "available," but functionally unsearchable.

Having spent several years delivering large, document-heavy digital services, I saw this pattern constantly, across departments and industries: teams either paid for a closed-source document AI product with a vague pricing model and a data-residency question mark, or they lived with the manual search.

So I built AI-DocumentIntelligence - an open-source, self-hostable, provider-agnostic RAG (Retrieval-Augmented Generation) platform for document upload, processing, and natural-language Q&A. The goal wasn't to build another wrapper around an API. It was to build something I'd be comfortable recommending to any delivery team handling sensitive documents: transparent, swappable, and auditable.

At its core, the app does four things:

pgvector That last point matters more than it sounds. Most tutorial-grade RAG projects hardcode a single LLM provider. In a real procurement context - any regulated department or enterprise - that's a non-starter - you need to be able to swap providers for cost, compliance, or availability reasons without a rewrite. So LLM_PROVIDER is a config flag, not an architectural decision baked into the code.

I deliberately kept this to a stack that's boring in the best sense - nothing here requires a team to learn a new deployment paradigm. If your organisation can already run a Node.js service and a PostgreSQL database, you can run this.

 ┌────────────┐      ┌──────────────────┐      ┌────────────────────┐
 │  React UI   │ ───► │  Express API      │ ───► │  Document Processor │
 │ (upload+chat)│      │ (TypeScript)       │      │ (chunking / split)   │
 └────────────┘      └──────────────────┘      └────────────────────┘
                                │                          │
                                ▼                          ▼
                      ┌──────────────────┐      ┌────────────────────┐
                      │  LangChain Layer   │ ◄──► │ PostgreSQL + pgvector│
                      │ (OpenAI / Claude)    │      │  (embeddings store)   │
                      └──────────────────┘      └────────────────────┘

The chunking and embedding pipeline runs once per upload. Query time is a straightforward retrieve-then-generate flow: embed the question, do a similarity search against pgvector, pass the top-k chunks to the selected LLM with a grounding prompt, and return an answer plus the chat history for that document.

Nothing exotic. That's intentional - the value isn't in a clever novel architecture, it's in getting the boring parts (provider abstraction, chunk quality, error handling around missing credentials) actually right.

Switching LLM providers is a config read, not a code branch buried three files deep:

// backend/src/llm/provider.ts (simplified)
const provider = process.env.LLM_PROVIDER; // "openai" | "anthropic"

export function getChatModel() {
  if (provider === "anthropic") {
    return new ChatAnthropic({
      apiKey: process.env.ANTHROPIC_API_KEY,
      model: process.env.ANTHROPIC_MODEL,
    });
  }
  return new ChatOpenAI({
    apiKey: process.env.OPENAI_API_KEY,
  });
}

Chunking uses LangChain's recursive splitter, tuned for document Q&A rather than raw ingestion:

const splitter = new RecursiveCharacterTextSplitter({
  chunkSize: 1000,
  chunkOverlap: 150,
});
const chunks = await splitter.splitDocuments(rawDocs);

The overlap matters more than most tutorials admit - too little, and you clip an answer in half across a chunk boundary; too much, and you're paying to embed and search the same sentence three times. 150 on a 1000-character chunk was the balance that held up best across the mixed PDF/DOCX test set I used.

Retrieval and generation stay separated, so either half is independently testable - I can validate that retrieval returns the right chunks without spending API credits on generation, which matters a lot when you're iterating on an unpaid OpenAI tier.

What went right:

What was genuinely hard:

docker compose up".env.example, explicit DB_NAME expectations, a documented port-conflict fallback - took longer than the RAG logic itself, and it's the part that actually determines whether anyone else can use the project. I'm building this alongside a small portfolio of open-source AI engineering projects - a policy Q&A RAG system for citizen-facing services, an agentic PR-review bot, a voice-to-Agile-user-story generator - all reflecting the same thread: applying LLM orchestration (LangChain/LangGraph) to real workflow problems I've encountered across large-scale digital delivery, not toy demos. If that combination - production delivery experience plus applied open-source AI engineering - is useful context for anyone evaluating this work, that's very much the point of publishing it.

The repo is open source and set up for local development in a few commands:

git clone https://github.com/Srameshgitnow/AI-DocumentIntelligence.git
cd AI-DocumentIntelligence
docker compose up --build

Full setup instructions, environment variables, and troubleshooting are in the README.

If you try it, hit a bug, or have thoughts on the provider-abstraction approach, I'd genuinely like to hear them - issues and PRs are open. And if the project is useful to you, a ⭐ on the repo goes a long way toward other people finding it:

👉 github.com/Srameshgitnow/AI-DocumentIntelligence

I'm a full-stack / AI engineer (React, Node.js, LangChain/LangGraph) with a background delivering large-scale digital services. I write about applied AI engineering and open-source tooling - follow along for the next post in this series.