# How I Built a Custom AI Document Assistant That Understands 1000s of PDFs and Talks Like a Human

> Source: <https://blog.stackademic.com/how-i-built-a-custom-ai-document-assistant-that-understands-1000s-of-pdfs-and-talks-like-a-human-e6489a4b8e20?source=rss----d1baaa8417a4---4>
> Published: 2026-08-12 17:01:03+00:00

**Forget basic search. I designed a Retrieval-Augmented Generation (RAG) system that reads technical documents, extracts diagrams, interprets text, and answers questions like a domain expert all using open-source AI.**

Ever tried finding a specific piece of info from 500+ PDFs?

Even with filenames like Report_Final_v2_NEW_Latest.pdf, good luck.

Search tools don’t **understand** content. They match keywords. That’s not enough for:

That’s when I decided to build something smarter a **multimodal RAG AI** that reads, understands, and answers questions from thousands of PDFs.

Organizing everything properly up front helped me scale this from 10 docs to 10,000.

```
ai-doc-assistant/├── ingest/│   ├── extract_text.py│   ├── extract_images.py├── process/│   ├── chunk_text.py│   ├── embed_chunks.py├── index/│   └── vector_store.py├── backend/│   ├── qa_chain.py│   └── server.py├── interface/│   └── ui.py
```

Each module had one responsibility.

Ingest → Process → Index → Answer → Display.

Text is the gold mine. So I extracted it page by page, preserving metadata.

``` python
import fitzdef extract_text(file_path):    doc = fitz.open(file_path)    pages = []    for i, page in enumerate(doc):        text = page.get_text()        pages.append({            "file": file_path,            "page": i + 1,            "text": text        })    return pages
```

This gave me full control: filenames, page references, and selective inclusion.

Visuals carry meaning especially in research and product manuals.

So I extracted all embedded images.

``` python
def extract_images(pdf_path, output_dir):    doc = fitz.open(pdf_path)    for page_index in range(len(doc)):        images = doc[page_index].get_images(full=True)        for img_index, img in enumerate(images):            xref = images[img_index][0]            base_image = doc.extract_image(xref)            image_bytes = base_image["image"]            image_filename = f"{output_dir}/{page_index}_{img_index}.png"            with open(image_filename, "wb") as img_file:                img_file.write(image_bytes)
```

Later, these diagrams were embedded using CLIP and stored alongside text.

Large LLMs can’t take full documents so we chunk them.

``` python
def chunk_text(text, size=500, overlap=100):    chunks = []    for i in range(0, len(text), size - overlap):        chunk = text[i:i + size]        chunks.append(chunk)    return chunks
```

Overlap ensures that context doesn’t break between chunks.

It’s essential for coherent answers.

Now, I transformed the chunks into vectors using sentence-transformers.

``` python
from sentence_transformers import SentenceTransformermodel = SentenceTransformer('all-MiniLM-L6-v2')def embed_chunks(chunks):    return model.encode(chunks)
```

These vectors represent **meaning**, not just keywords.

So later, we can retrieve the most relevant concepts, even if phrased differently.

🐍Most tutorials teach you syntax. This one teaches you how professionals actually write Python that survives production. Read it now.

[Python Starter Guide for Non Programmers](https://abdulahad28.gumroad.com/l/irehc?source=post_page-----10aca15b4487---------------------------------------)

I used FAISS to store and search embeddings efficiently.

``` python
import faissimport numpy as npdef build_index(embeddings):    dim = embeddings.shape[1]    index = faiss.IndexFlatL2(dim)    index.add(embeddings)    return index
```

Once built, this allowed me to run 10,000+ doc searches in under a second.

To make the assistant multimodal, I indexed diagrams using CLIP.

``` python
from transformers import CLIPProcessor, CLIPModelfrom PIL import Imageclip_model = CLIPModel.from_pretrained("openai/clip-vit-base-patch32")processor = CLIPProcessor.from_pretrained("openai/clip-vit-base-patch32")def embed_image(image_path):    image = Image.open(image_path)    inputs = processor(images=image, return_tensors="pt")    with torch.no_grad():        image_features = clip_model.get_image_features(**inputs)    return image_features.squeeze().numpy()
```

Now the bot could answer queries like “show me the process diagram for system reboot”.

Once the top documents were retrieved, I fed them into an LLM.

``` python
from langchain.chains import RetrievalQAfrom langchain.vectorstores import FAISSfrom langchain.llms import Ollamaretriever = FAISS.load_local("index", embeddings=model)llm = Ollama(model="mistral")qa = RetrievalQA.from_chain_type(llm=llm, retriever=retriever)
```

Query it:

```
qa.run("What’s the warranty coverage of Product X?")
```

Boom human-like answers with cited documents.

To increase trust, I appended sources to every answer.

``` python
def format_response(answer, docs):    refs = "\n".join([f"{doc.metadata['file']} (p{doc.metadata['page']})" for doc in docs])    return f"{answer}\n\nSources:\n{refs}"
```

This made it feel more like a lawyer than a chatbot.

Every claim was traceable.

I wrapped the whole pipeline in FastAPI:

``` python
from fastapi import FastAPIfrom pydantic import BaseModelclass Query(BaseModel):    question: strapp = FastAPI()@app.post("/ask")def ask(query: Query):    response = qa.run(query.question)    return {"answer": response}
```

I could now connect this backend to web apps, Slack, or even voice.

To demo it to clients and teammates, I created a slick UI:

``` python
import gradio as grdef answer_question(q):    return qa.run(q)gr.Interface(fn=answer_question, inputs="text", outputs="text", title="AI PDF Assistant").launch()
```

They asked real questions from company manuals and the bot *nailed it*.

Users could drop new PDFs on the UI, and they were instantly processed and indexed.

``` python
@app.post("/upload")async def upload_pdf(file: UploadFile):    save_path = f"./docs/{file.filename}"    with open(save_path, "wb") as f:        f.write(await file.read())    # Extract, chunk, embed, and update FAISS index
```

This made the assistant **self-updating**.

New docs = new knowledge.

For full control, I containerized everything.

Dockerfile:

```
FROM python:3.10WORKDIR /appCOPY . .RUN pip install -r requirements.txtCMD ["uvicorn", "backend.server:app", "--host", "0.0.0.0", "--port", "8000"]
```

Now it runs on any laptop, without the cloud.

I gave the assistant memory:

``` python
import sqlite3def save_interaction(question, answer):    conn = sqlite3.connect("history.db")    conn.execute("INSERT INTO log (q, a) VALUES (?, ?)", (question, answer))    conn.commit()
```

Later, I used this to retrain the model and improve responses.

This wasn’t just search. This was **reasoning** over documents, charts, and audio.

It could:

I didn’t just build a bot. I built an **AI teammate**.

🐍 Tired of digging through hundreds of new Python releases every week just to find the ones worth your time? Python Weekly Brief hands you the good stuff on a plate, every single week.

[How I Built a Custom AI Document Assistant That Understands 1000s of PDFs and Talks Like a Human](https://blog.stackademic.com/how-i-built-a-custom-ai-document-assistant-that-understands-1000s-of-pdfs-and-talks-like-a-human-e6489a4b8e20) was originally published in [Stackademic](https://blog.stackademic.com) on Medium, where people are continuing the conversation by highlighting and responding to this story.