Forget basic search. I designed a Retrieval-Augmented Generation (RAG) system that reads technical documents, extracts diagrams, interprets text, and answers questions like a domain expert all using open-source AI.
Ever tried finding a specific piece of info from 500+ PDFs?
Even with filenames like Report_Final_v2_NEW_Latest.pdf, good luck.
Search tools don’t understand content. They match keywords. That’s not enough for:
That’s when I decided to build something smarter a multimodal RAG AI that reads, understands, and answers questions from thousands of PDFs.
Organizing everything properly up front helped me scale this from 10 docs to 10,000.
ai-doc-assistant/├── ingest/│ ├── extract_text.py│ ├── extract_images.py├── process/│ ├── chunk_text.py│ ├── embed_chunks.py├── index/│ └── vector_store.py├── backend/│ ├── qa_chain.py│ └── server.py├── interface/│ └── ui.py
Each module had one responsibility.
Ingest → Process → Index → Answer → Display.
Text is the gold mine. So I extracted it page by page, preserving metadata.
import fitzdef extract_text(file_path): doc = fitz.open(file_path) pages = [] for i, page in enumerate(doc): text = page.get_text() pages.append({ "file": file_path, "page": i + 1, "text": text }) return pages
This gave me full control: filenames, page references, and selective inclusion.
Visuals carry meaning especially in research and product manuals.
So I extracted all embedded images.
def extract_images(pdf_path, output_dir): doc = fitz.open(pdf_path) for page_index in range(len(doc)): images = doc[page_index].get_images(full=True) for img_index, img in enumerate(images): xref = images[img_index][0] base_image = doc.extract_image(xref) image_bytes = base_image["image"] image_filename = f"{output_dir}/{page_index}_{img_index}.png" with open(image_filename, "wb") as img_file: img_file.write(image_bytes)
Later, these diagrams were embedded using CLIP and stored alongside text.
Large LLMs can’t take full documents so we chunk them.
def chunk_text(text, size=500, overlap=100): chunks = [] for i in range(0, len(text), size - overlap): chunk = text[i:i + size] chunks.append(chunk) return chunks
Overlap ensures that context doesn’t break between chunks.
It’s essential for coherent answers.
Now, I transformed the chunks into vectors using sentence-transformers.
from sentence_transformers import SentenceTransformermodel = SentenceTransformer('all-MiniLM-L6-v2')def embed_chunks(chunks): return model.encode(chunks)
These vectors represent meaning, not just keywords.
So later, we can retrieve the most relevant concepts, even if phrased differently.
🐍Most tutorials teach you syntax. This one teaches you how professionals actually write Python that survives production. Read it now.
Python Starter Guide for Non Programmers
I used FAISS to store and search embeddings efficiently.
import faissimport numpy as npdef build_index(embeddings): dim = embeddings.shape[1] index = faiss.IndexFlatL2(dim) index.add(embeddings) return index
Once built, this allowed me to run 10,000+ doc searches in under a second.
To make the assistant multimodal, I indexed diagrams using CLIP.
from transformers import CLIPProcessor, CLIPModelfrom PIL import Imageclip_model = CLIPModel.from_pretrained("openai/clip-vit-base-patch32")processor = CLIPProcessor.from_pretrained("openai/clip-vit-base-patch32")def embed_image(image_path): image = Image.open(image_path) inputs = processor(images=image, return_tensors="pt") with torch.no_grad(): image_features = clip_model.get_image_features(**inputs) return image_features.squeeze().numpy()
Now the bot could answer queries like “show me the process diagram for system reboot”.
Once the top documents were retrieved, I fed them into an LLM.
from langchain.chains import RetrievalQAfrom langchain.vectorstores import FAISSfrom langchain.llms import Ollamaretriever = FAISS.load_local("index", embeddings=model)llm = Ollama(model="mistral")qa = RetrievalQA.from_chain_type(llm=llm, retriever=retriever)
Query it:
qa.run("What’s the warranty coverage of Product X?")
Boom human-like answers with cited documents.
To increase trust, I appended sources to every answer.
def format_response(answer, docs): refs = "\n".join([f"{doc.metadata['file']} (p{doc.metadata['page']})" for doc in docs]) return f"{answer}\n\nSources:\n{refs}"
This made it feel more like a lawyer than a chatbot.
Every claim was traceable.
I wrapped the whole pipeline in FastAPI:
from fastapi import FastAPIfrom pydantic import BaseModelclass Query(BaseModel): question: strapp = FastAPI()@app.post("/ask")def ask(query: Query): response = qa.run(query.question) return {"answer": response}
I could now connect this backend to web apps, Slack, or even voice.
To demo it to clients and teammates, I created a slick UI:
import gradio as grdef answer_question(q): return qa.run(q)gr.Interface(fn=answer_question, inputs="text", outputs="text", title="AI PDF Assistant").launch()
They asked real questions from company manuals and the bot nailed it.
Users could drop new PDFs on the UI, and they were instantly processed and indexed.
@app.post("/upload")async def upload_pdf(file: UploadFile): save_path = f"./docs/{file.filename}" with open(save_path, "wb") as f: f.write(await file.read()) # Extract, chunk, embed, and update FAISS index
This made the assistant self-updating.
New docs = new knowledge.
For full control, I containerized everything.
Dockerfile:
FROM python:3.10WORKDIR /appCOPY . .RUN pip install -r requirements.txtCMD ["uvicorn", "backend.server:app", "--host", "0.0.0.0", "--port", "8000"]
Now it runs on any laptop, without the cloud.
I gave the assistant memory:
import sqlite3def save_interaction(question, answer): conn = sqlite3.connect("history.db") conn.execute("INSERT INTO log (q, a) VALUES (?, ?)", (question, answer)) conn.commit()
Later, I used this to retrain the model and improve responses.
This wasn’t just search. This was reasoning over documents, charts, and audio.
It could:
I didn’t just build a bot. I built an AI teammate.
🐍 Tired of digging through hundreds of new Python releases every week just to find the ones worth your time? Python Weekly Brief hands you the good stuff on a plate, every single week.
How I Built a Custom AI Document Assistant That Understands 1000s of PDFs and Talks Like a Human was originally published in Stackademic on Medium, where people are continuing the conversation by highlighting and responding to this story.