{"slug": "how-i-built-a-custom-ai-document-assistant-that-understands-1000s-of-pdfs-and-a", "title": "How I Built a Custom AI Document Assistant That Understands 1000s of PDFs and Talks Like a Human", "summary": "A developer built a custom multimodal Retrieval-Augmented Generation (RAG) system that reads and understands thousands of PDFs using open-source AI, extracting text and images, chunking content, and embedding vectors with sentence-transformers and FAISS for fast search. The system, which scales from 10 to 10,000 documents, uses CLIP to index diagrams and answers questions like a domain expert, addressing the limitations of keyword-based search.", "body_md": "**Forget basic search. I designed a Retrieval-Augmented Generation (RAG) system that reads technical documents, extracts diagrams, interprets text, and answers questions like a domain expert all using open-source AI.**\n\nEver tried finding a specific piece of info from 500+ PDFs?\n\nEven with filenames like Report_Final_v2_NEW_Latest.pdf, good luck.\n\nSearch tools don’t **understand** content. They match keywords. That’s not enough for:\n\nThat’s when I decided to build something smarter a **multimodal RAG AI** that reads, understands, and answers questions from thousands of PDFs.\n\nOrganizing everything properly up front helped me scale this from 10 docs to 10,000.\n\n```\nai-doc-assistant/├── ingest/│   ├── extract_text.py│   ├── extract_images.py├── process/│   ├── chunk_text.py│   ├── embed_chunks.py├── index/│   └── vector_store.py├── backend/│   ├── qa_chain.py│   └── server.py├── interface/│   └── ui.py\n```\n\nEach module had one responsibility.\n\nIngest → Process → Index → Answer → Display.\n\nText is the gold mine. So I extracted it page by page, preserving metadata.\n\n``` python\nimport fitzdef extract_text(file_path):    doc = fitz.open(file_path)    pages = []    for i, page in enumerate(doc):        text = page.get_text()        pages.append({            \"file\": file_path,            \"page\": i + 1,            \"text\": text        })    return pages\n```\n\nThis gave me full control: filenames, page references, and selective inclusion.\n\nVisuals carry meaning especially in research and product manuals.\n\nSo I extracted all embedded images.\n\n``` python\ndef extract_images(pdf_path, output_dir):    doc = fitz.open(pdf_path)    for page_index in range(len(doc)):        images = doc[page_index].get_images(full=True)        for img_index, img in enumerate(images):            xref = images[img_index][0]            base_image = doc.extract_image(xref)            image_bytes = base_image[\"image\"]            image_filename = f\"{output_dir}/{page_index}_{img_index}.png\"            with open(image_filename, \"wb\") as img_file:                img_file.write(image_bytes)\n```\n\nLater, these diagrams were embedded using CLIP and stored alongside text.\n\nLarge LLMs can’t take full documents so we chunk them.\n\n``` python\ndef chunk_text(text, size=500, overlap=100):    chunks = []    for i in range(0, len(text), size - overlap):        chunk = text[i:i + size]        chunks.append(chunk)    return chunks\n```\n\nOverlap ensures that context doesn’t break between chunks.\n\nIt’s essential for coherent answers.\n\nNow, I transformed the chunks into vectors using sentence-transformers.\n\n``` python\nfrom sentence_transformers import SentenceTransformermodel = SentenceTransformer('all-MiniLM-L6-v2')def embed_chunks(chunks):    return model.encode(chunks)\n```\n\nThese vectors represent **meaning**, not just keywords.\n\nSo later, we can retrieve the most relevant concepts, even if phrased differently.\n\n🐍Most tutorials teach you syntax. This one teaches you how professionals actually write Python that survives production. Read it now.\n\n[Python Starter Guide for Non Programmers](https://abdulahad28.gumroad.com/l/irehc?source=post_page-----10aca15b4487---------------------------------------)\n\nI used FAISS to store and search embeddings efficiently.\n\n``` python\nimport faissimport numpy as npdef build_index(embeddings):    dim = embeddings.shape[1]    index = faiss.IndexFlatL2(dim)    index.add(embeddings)    return index\n```\n\nOnce built, this allowed me to run 10,000+ doc searches in under a second.\n\nTo make the assistant multimodal, I indexed diagrams using CLIP.\n\n``` python\nfrom transformers import CLIPProcessor, CLIPModelfrom PIL import Imageclip_model = CLIPModel.from_pretrained(\"openai/clip-vit-base-patch32\")processor = CLIPProcessor.from_pretrained(\"openai/clip-vit-base-patch32\")def embed_image(image_path):    image = Image.open(image_path)    inputs = processor(images=image, return_tensors=\"pt\")    with torch.no_grad():        image_features = clip_model.get_image_features(**inputs)    return image_features.squeeze().numpy()\n```\n\nNow the bot could answer queries like “show me the process diagram for system reboot”.\n\nOnce the top documents were retrieved, I fed them into an LLM.\n\n``` python\nfrom langchain.chains import RetrievalQAfrom langchain.vectorstores import FAISSfrom langchain.llms import Ollamaretriever = FAISS.load_local(\"index\", embeddings=model)llm = Ollama(model=\"mistral\")qa = RetrievalQA.from_chain_type(llm=llm, retriever=retriever)\n```\n\nQuery it:\n\n```\nqa.run(\"What’s the warranty coverage of Product X?\")\n```\n\nBoom human-like answers with cited documents.\n\nTo increase trust, I appended sources to every answer.\n\n``` python\ndef format_response(answer, docs):    refs = \"\\n\".join([f\"{doc.metadata['file']} (p{doc.metadata['page']})\" for doc in docs])    return f\"{answer}\\n\\nSources:\\n{refs}\"\n```\n\nThis made it feel more like a lawyer than a chatbot.\n\nEvery claim was traceable.\n\nI wrapped the whole pipeline in FastAPI:\n\n``` python\nfrom fastapi import FastAPIfrom pydantic import BaseModelclass Query(BaseModel):    question: strapp = FastAPI()@app.post(\"/ask\")def ask(query: Query):    response = qa.run(query.question)    return {\"answer\": response}\n```\n\nI could now connect this backend to web apps, Slack, or even voice.\n\nTo demo it to clients and teammates, I created a slick UI:\n\n``` python\nimport gradio as grdef answer_question(q):    return qa.run(q)gr.Interface(fn=answer_question, inputs=\"text\", outputs=\"text\", title=\"AI PDF Assistant\").launch()\n```\n\nThey asked real questions from company manuals and the bot *nailed it*.\n\nUsers could drop new PDFs on the UI, and they were instantly processed and indexed.\n\n``` python\n@app.post(\"/upload\")async def upload_pdf(file: UploadFile):    save_path = f\"./docs/{file.filename}\"    with open(save_path, \"wb\") as f:        f.write(await file.read())    # Extract, chunk, embed, and update FAISS index\n```\n\nThis made the assistant **self-updating**.\n\nNew docs = new knowledge.\n\nFor full control, I containerized everything.\n\nDockerfile:\n\n```\nFROM python:3.10WORKDIR /appCOPY . .RUN pip install -r requirements.txtCMD [\"uvicorn\", \"backend.server:app\", \"--host\", \"0.0.0.0\", \"--port\", \"8000\"]\n```\n\nNow it runs on any laptop, without the cloud.\n\nI gave the assistant memory:\n\n``` python\nimport sqlite3def save_interaction(question, answer):    conn = sqlite3.connect(\"history.db\")    conn.execute(\"INSERT INTO log (q, a) VALUES (?, ?)\", (question, answer))    conn.commit()\n```\n\nLater, I used this to retrain the model and improve responses.\n\nThis wasn’t just search. This was **reasoning** over documents, charts, and audio.\n\nIt could:\n\nI didn’t just build a bot. I built an **AI teammate**.\n\n🐍 Tired of digging through hundreds of new Python releases every week just to find the ones worth your time? Python Weekly Brief hands you the good stuff on a plate, every single week.\n\n[How I Built a Custom AI Document Assistant That Understands 1000s of PDFs and Talks Like a Human](https://blog.stackademic.com/how-i-built-a-custom-ai-document-assistant-that-understands-1000s-of-pdfs-and-talks-like-a-human-e6489a4b8e20) was originally published in [Stackademic](https://blog.stackademic.com) on Medium, where people are continuing the conversation by highlighting and responding to this story.", "url": "https://wpnews.pro/news/how-i-built-a-custom-ai-document-assistant-that-understands-1000s-of-pdfs-and-a", "canonical_source": "https://blog.stackademic.com/how-i-built-a-custom-ai-document-assistant-that-understands-1000s-of-pdfs-and-talks-like-a-human-e6489a4b8e20?source=rss----d1baaa8417a4---4", "published_at": "2026-08-12 17:01:03+00:00", "updated_at": "2026-08-12 17:25:06.422297+00:00", "lang": "en", "topics": ["artificial-intelligence", "machine-learning", "generative-ai", "ai-tools", "ai-infrastructure"], "entities": ["FAISS", "CLIP", "sentence-transformers", "PyMuPDF", "OpenAI"], "alternates": {"html": "https://wpnews.pro/news/how-i-built-a-custom-ai-document-assistant-that-understands-1000s-of-pdfs-and-a", "markdown": "https://wpnews.pro/news/how-i-built-a-custom-ai-document-assistant-that-understands-1000s-of-pdfs-and-a.md", "text": "https://wpnews.pro/news/how-i-built-a-custom-ai-document-assistant-that-understands-1000s-of-pdfs-and-a.txt", "jsonld": "https://wpnews.pro/news/how-i-built-a-custom-ai-document-assistant-that-understands-1000s-of-pdfs-and-a.jsonld"}}