How I Built a Custom AI Document Assistant That Understands 1000s of PDFs and Talks Like a Human

A developer built a custom multimodal Retrieval-Augmented Generation (RAG) system that reads and understands thousands of PDFs using open-source AI, extracting text and images, chunking content, and embedding vectors with sentence-transformers and FAISS for fast search. The system, which scales from 10 to 10,000 documents, uses CLIP to index diagrams and answers questions like a domain expert, addressing the limitations of keyword-based search.

Forget basic search. I designed a Retrieval-Augmented Generation RAG system that reads technical documents, extracts diagrams, interprets text, and answers questions like a domain expert all using open-source AI. Ever tried finding a specific piece of info from 500+ PDFs? Even with filenames like Report Final v2 NEW Latest.pdf, good luck. Search tools don’t understand content. They match keywords. That’s not enough for: That’s when I decided to build something smarter a multimodal RAG AI that reads, understands, and answers questions from thousands of PDFs. Organizing everything properly up front helped me scale this from 10 docs to 10,000. ai-doc-assistant/├── ingest/│ ├── extract text.py│ ├── extract images.py├── process/│ ├── chunk text.py│ ├── embed chunks.py├── index/│ └── vector store.py├── backend/│ ├── qa chain.py│ └── server.py├── interface/│ └── ui.py Each module had one responsibility. Ingest → Process → Index → Answer → Display. Text is the gold mine. So I extracted it page by page, preserving metadata. python import fitzdef extract text file path : doc = fitz.open file path pages = for i, page in enumerate doc : text = page.get text pages.append { "file": file path, "page": i + 1, "text": text } return pages This gave me full control: filenames, page references, and selective inclusion. Visuals carry meaning especially in research and product manuals. So I extracted all embedded images. python def extract images pdf path, output dir : doc = fitz.open pdf path for page index in range len doc : images = doc page index .get images full=True for img index, img in enumerate images : xref = images img index 0 base image = doc.extract image xref image bytes = base image "image" image filename = f"{output dir}/{page index} {img index}.png" with open image filename, "wb" as img file: img file.write image bytes Later, these diagrams were embedded using CLIP and stored alongside text. Large LLMs can’t take full documents so we chunk them. python def chunk text text, size=500, overlap=100 : chunks = for i in range 0, len text , size - overlap : chunk = text i:i + size chunks.append chunk return chunks Overlap ensures that context doesn’t break between chunks. It’s essential for coherent answers. Now, I transformed the chunks into vectors using sentence-transformers. python from sentence transformers import SentenceTransformermodel = SentenceTransformer 'all-MiniLM-L6-v2' def embed chunks chunks : return model.encode chunks These vectors represent meaning , not just keywords. So later, we can retrieve the most relevant concepts, even if phrased differently. 🐍Most tutorials teach you syntax. This one teaches you how professionals actually write Python that survives production. Read it now. Python Starter Guide for Non Programmers https://abdulahad28.gumroad.com/l/irehc?source=post page-----10aca15b4487--------------------------------------- I used FAISS to store and search embeddings efficiently. python import faissimport numpy as npdef build index embeddings : dim = embeddings.shape 1 index = faiss.IndexFlatL2 dim index.add embeddings return index Once built, this allowed me to run 10,000+ doc searches in under a second. To make the assistant multimodal, I indexed diagrams using CLIP. python from transformers import CLIPProcessor, CLIPModelfrom PIL import Imageclip model = CLIPModel.from pretrained "openai/clip-vit-base-patch32" processor = CLIPProcessor.from pretrained "openai/clip-vit-base-patch32" def embed image image path : image = Image.open image path inputs = processor images=image, return tensors="pt" with torch.no grad : image features = clip model.get image features inputs return image features.squeeze .numpy Now the bot could answer queries like “show me the process diagram for system reboot”. Once the top documents were retrieved, I fed them into an LLM. python from langchain.chains import RetrievalQAfrom langchain.vectorstores import FAISSfrom langchain.llms import Ollamaretriever = FAISS.load local "index", embeddings=model llm = Ollama model="mistral" qa = RetrievalQA.from chain type llm=llm, retriever=retriever Query it: qa.run "What’s the warranty coverage of Product X?" Boom human-like answers with cited documents. To increase trust, I appended sources to every answer. python def format response answer, docs : refs = "\n".join f"{doc.metadata 'file' } p{doc.metadata 'page' } " for doc in docs return f"{answer}\n\nSources:\n{refs}" This made it feel more like a lawyer than a chatbot. Every claim was traceable. I wrapped the whole pipeline in FastAPI: python from fastapi import FastAPIfrom pydantic import BaseModelclass Query BaseModel : question: strapp = FastAPI @app.post "/ask" def ask query: Query : response = qa.run query.question return {"answer": response} I could now connect this backend to web apps, Slack, or even voice. To demo it to clients and teammates, I created a slick UI: python import gradio as grdef answer question q : return qa.run q gr.Interface fn=answer question, inputs="text", outputs="text", title="AI PDF Assistant" .launch They asked real questions from company manuals and the bot nailed it . Users could drop new PDFs on the UI, and they were instantly processed and indexed. python @app.post "/upload" async def upload pdf file: UploadFile : save path = f"./docs/{file.filename}" with open save path, "wb" as f: f.write await file.read Extract, chunk, embed, and update FAISS index This made the assistant self-updating . New docs = new knowledge. For full control, I containerized everything. Dockerfile: FROM python:3.10WORKDIR /appCOPY . .RUN pip install -r requirements.txtCMD "uvicorn", "backend.server:app", "--host", "0.0.0.0", "--port", "8000" Now it runs on any laptop, without the cloud. I gave the assistant memory: python import sqlite3def save interaction question, answer : conn = sqlite3.connect "history.db" conn.execute "INSERT INTO log q, a VALUES ?, ? ", question, answer conn.commit Later, I used this to retrain the model and improve responses. This wasn’t just search. This was reasoning over documents, charts, and audio. It could: I didn’t just build a bot. I built an AI teammate . 🐍 Tired of digging through hundreds of new Python releases every week just to find the ones worth your time? Python Weekly Brief hands you the good stuff on a plate, every single week. How I Built a Custom AI Document Assistant That Understands 1000s of PDFs and Talks Like a Human https://blog.stackademic.com/how-i-built-a-custom-ai-document-assistant-that-understands-1000s-of-pdfs-and-talks-like-a-human-e6489a4b8e20 was originally published in Stackademic https://blog.stackademic.com on Medium, where people are continuing the conversation by highlighting and responding to this story.