{"slug": "building-a-grounded-document-q-a-app-with-gemini-langgraph-and-chroma", "title": "Building a Grounded Document Q&A App with Gemini, LangGraph, and Chroma", "summary": "A developer built Doc Processing AI, a full-stack retrieval-augmented generation app that chunks and embeds uploaded PDF, TXT, and Markdown files with Google Gemini, stores the vectors in Chroma, and answers questions through either a classic RAG pipeline or a LangGraph tool-calling agent, streaming cited responses to a React UI over Server-Sent Events. The app splits text with RecursiveCharacterTextSplitter at chunk_size=1000 and chunk_overlap=150, uses deterministic chunk IDs of the form {doc_id}_{index} with old vectors deleted before re-ingestion, and runs indexing in a FastAPI BackgroundTask through a pending → indexing → ready | failed state machine. RAG mode retrieves the top 4 chunks under a strict context-only system prompt, while agent mode uses a two-node LangGraph StateGraph with a single retrieve_documents tool, and a golden-set eval harness scores every change.", "body_md": "Most “chat with your PDF” demos stop at the happy path: embed a file, ask a question, get a plausible answer. The hard parts come later. Can you tell *where* the answer came from? What does the user see while the model is thinking? And how do you know retrieval still works after you change something?\n\nI built **Doc Processing AI** to answer those questions end to end. It’s a full-stack RAG app: you upload documents, they get chunked and embedded with Google Gemini, stored in Chroma, and queried through either a classic RAG pipeline or a LangGraph tool-calling agent. Answers stream to a React UI over Server-Sent Events, with citations attached. There’s also a golden-set eval harness that scores every change.\n\nHere’s how it’s put together and what I learned along the way.\n\nEverything runs locally with one API key. No Docker, no hosted vector DB, no paid tier.\n\nWhen you upload a PDF, TXT, or Markdown file, the API writes it to disk, creates a Document row with status pending, and returns right away. The actual indexing runs in a FastAPI BackgroundTask, so the upload request never blocks on embedding calls.\n\nThe ingest worker moves the document through a small state machine (pending → indexing → ready | failed), which the UI shows as status chips:\n\n```\nsplitter = RecursiveCharacterTextSplitter(chunk_size=1000, chunk_overlap=150)chunks = splitter.split_text(text)lc_docs = [    LCDocument(        page_content=chunk,        metadata={\"document_id\": doc.id, \"filename\": doc.filename, \"chunk_index\": i},    )    for i, chunk in enumerate(chunks)]store.delete(where={\"document_id\": doc.id})  # re-ingest safetystore.add_documents(lc_docs, ids=[f\"{doc.id}_{i}\" for i in range(len(lc_docs))])\n```\n\nTwo details matter here. First, every chunk carries document_id and filename metadata. That's what makes citations and per-document filtering possible later. Second, chunk IDs are deterministic ({doc_id}_{index}), and old vectors are deleted before re-adding, so re-ingesting a document never leaves duplicates behind.\n\nIf extraction fails (a scanned PDF with no text layer, for example), the error message is stored on the row and shown in the UI instead of failing silently.\n\nRetrieval is a thin wrapper over Chroma’s similarity_search_with_score. In the chat UI you can optionally limit a question to specific documents, which becomes a metadata filter:\n\n```\nfilt = {\"document_id\": {\"$in\": document_ids}} if document_ids else Nonereturn store.similarity_search_with_score(query, k=k, filter=filt)\n```\n\nChroma’s filter syntax has changed between versions, so there’s a fallback: if the filtered query throws, the code over-fetches (k * 3) unfiltered results and filters them in Python. It's less efficient, but chat keeps working.\n\nThe chat page has a **RAG / Agent** toggle, and both modes use the same index.\n\n**RAG mode** is the deterministic pipeline: retrieve the top 4 chunks, format them into a numbered context block with filenames and scores, and send them to Gemini with a strict system prompt:\n\n*Answer ONLY using the provided context excerpts. If the context is insufficient, say you don’t have enough information in the uploaded documents. Do not invent facts.*\n\nTokens are streamed straight from llm.astream() to the client.\n\n**Agent mode** uses a LangGraph StateGraph with two nodes. One is an agent node that calls Gemini with tools bound, and the other is a ToolNode that exposes a single retrieve_documents tool. A conditional edge loops back to the agent whenever the model makes a tool call:\n\n```\ngraph = StateGraph(AgentState)graph.add_node(\"agent\", agent_node)graph.add_node(\"tools\", tool_node)graph.set_entry_point(\"agent\")graph.add_conditional_edges(\"agent\", should_continue, {\"tools\": \"tools\", END: END})graph.add_edge(\"tools\", \"agent\")\n```\n\nThe difference shows up on harder questions. The agent can rephrase its query, search more than once, or combine evidence from several documents before it answers. RAG mode is faster and more predictable. Having both side by side over the same corpus makes the tradeoff easy to see.\n\nThe single POST /chat endpoint returns a text/event-stream. Every data: line is a JSON object with an event field:\n\nBoth RAG and the agent are written as async generators that yield (event_name, payload) tuples, so the router stays tiny and any exception reaches the client as an error event instead of a dropped connection.\n\nThe status events are a small UX detail that made a big difference. On the free Gemini tier, time to first token can be a few seconds. Without feedback the app feels broken. With \"Retrieving documents…\" followed by \"Streaming answer…\", it feels responsive.\n\nOn the client, I skipped EventSource because it only supports GET. Instead the app uses fetch with a ReadableStream reader, splits on \\n\\n, and dispatches each event to typed handlers (onToken, onCitation, onStatus, and so on). Citations render as chips under each answer, so you can see exactly which file supported it.\n\nThis is the part most RAG tutorials skip. The repo ships with three sample documents (a benefits handbook, a project brief, and a refund/shipping policy) and a golden_set.json of 8 questions with known answers. Running python -m evals.run_evals scores each case on three axes:\n\nThese three metrics isolate different failure modes. A hit@k miss means retrieval or chunking is broken. A hit with no match on “contains” means the model ignored or misread good context. A faithfulness failure means it made something up. When you change chunk size, the prompt, or the model, you know which layer broke.\n\nThe current baseline is 100% across all three metrics on the golden set. With only 8 cases that number doesn’t say much about real-world quality, but it’s a regression baseline: if a change drops a score, you find out before your users do.\n\nIf you add a LangSmith key to .env, every RAG call, agent step, and tool invocation gets traced automatically. Each run is tagged with mode:rag or mode:agent, and eval runs also log hit_at_k, answer_contains, and faithfulness as feedback scores. That way you can open a failing eval case and see the exact prompt, retrieved chunks, and latency.\n\nNatural next steps are migrating to pgvector for multi-user deployments, adding auth, trying hybrid (BM25 + vector) retrieval, and growing the golden set into something statistically meaningful.\n\nThe full source is on GitHub: [github.com/Eric-Souza/gemini-rag-doc-processing](https://github.com/Eric-Souza/gemini-rag-doc-processing). Clone it, add a free Gemini key, drop in the sample docs, and you’ll have grounded, cited answers in about five minutes.\n\nIf you build something on top of it, or have opinions about RAG vs. agentic retrieval, I’d love to hear about it in the comments.\n\n[Building a Grounded Document Q&A App with Gemini, LangGraph, and Chroma](https://blog.stackademic.com/building-a-grounded-document-q-a-app-with-gemini-langgraph-and-chroma-07a1b230b9d5) was originally published in [Stackademic](https://blog.stackademic.com) on Medium, where people are continuing the conversation by highlighting and responding to this story.", "url": "https://wpnews.pro/news/building-a-grounded-document-q-a-app-with-gemini-langgraph-and-chroma", "canonical_source": "https://blog.stackademic.com/building-a-grounded-document-q-a-app-with-gemini-langgraph-and-chroma-07a1b230b9d5?source=rss----d1baaa8417a4---4", "published_at": "2026-10-26 09:51:23+00:00", "updated_at": "2026-10-26 10:13:23.079858+00:00", "lang": "en", "topics": ["artificial-intelligence", "generative-ai", "ai-agents", "ai-tools", "large-language-models"], "entities": ["Doc Processing AI", "Google Gemini", "LangGraph", "Chroma", "FastAPI", "React", "RecursiveCharacterTextSplitter"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/building-a-grounded-document-q-a-app-with-gemini-langgraph-and-chroma", "markdown": "https://wpnews.pro/news/building-a-grounded-document-q-a-app-with-gemini-langgraph-and-chroma.md", "text": "https://wpnews.pro/news/building-a-grounded-document-q-a-app-with-gemini-langgraph-and-chroma.txt", "jsonld": "https://wpnews.pro/news/building-a-grounded-document-q-a-app-with-gemini-langgraph-and-chroma.jsonld"}}