Building a Multimodal RAG Pipeline with NVIDIA NeMo Retriever, Hosted NIMs, LanceDB, Reranking, and Grounded Generation

NVIDIA's NeMo Retriever enables a multimodal retrieval-augmented generation pipeline that extracts text offline with PDFium, detects page elements, tables, charts, and infographics via hosted NVIDIA NIM endpoints, generates dense embeddings, stores content in LanceDB, and performs dense retrieval, vision-language reranking, metadata-filtered search, and grounded response generation with inline citations. The pipeline includes a recall-at-k evaluation to validate retrieval quality across multimodal document content.

In this tutorial, we build an advanced multimodal retrieval-augmented generation pipeline with NVIDIA NeMo Retriever https://github.com/NVIDIA/NeMo-Retriever . We begin by configuring a Python 3.12 environment, installing the required packages, and performing offline PDF text extraction without relying on a GPU or external API key. We then extend the workflow with hosted NVIDIA NIM endpoints to detect page elements, extract tables, charts, and infographics, generate dense vector embeddings, and store the processed content in LanceDB. Finally, we implement dense retrieval, vision-language reranking, metadata-filtered search, grounded response generation with inline citations, and a lightweight recall-at-k evaluation to validate retrieval quality across multimodal document content. python import sys, os, subprocess, textwrap, json, time, warnings warnings.filterwarnings "ignore" assert sys.version info :2 == 3, 12 , f"nemo-retriever requires Python 3.12.x found {sys.version.split 0 } . " "Colab's default runtime is 3.12; if you changed it, switch back." def sh cmd : print f"$ {cmd}" subprocess.run cmd, shell=True, check=False try: import nemo retriever print "nemo-retriever already installed" except ImportError: sh "pip install -q --ignore-installed PyJWT nemo-retriever openai" import nemo retriever print "nemo-retriever version:", nemo retriever. version from nemo retriever import create ingestor try: from nemo retriever.io import to markdown, to markdown by page except ImportError: from nemo retriever.common.io import to markdown, to markdown by page try: from nemo retriever.retriever import Retriever except ImportError: from nemo retriever.graph.retriever import Retriever import pandas as pd pd.set option "display.max colwidth", 160 DOC = "multimodal test.pdf" if not os.path.exists DOC : sh f"curl -sL -o {DOC} " "https://raw.githubusercontent.com/NVIDIA/NeMo-Retriever/main/data/multimodal test.pdf" print "document:", DOC, os.path.getsize DOC , "bytes" DOCS = DOC print "\n=== STAGE 1: offline text extraction no API key ===" offline = create ingestor run mode="inprocess", allow no gpu=True .files DOCS .extract extract text=True, extract tables=False, extract charts=False, extract images=False, extract infographics=False, use page elements=False, extract page as image=False, method="pdfium", df offline = offline.ingest print "rows:", df offline.shape, "\ncolumns:", list df offline.columns print "\npage 1 text preview:\n", df offline.iloc 0 "text" :400 We configure the Python 3.12 environment, install NVIDIA NeMo Retriever, and import the required ingestion and retrieval components. We download the sample multimodal PDF and define it as the input document for the pipeline. We then perform CPU-based offline text extraction with PDFium and inspect the extracted rows, columns, and page content. python from getpass import getpass if not os.environ.get "NVIDIA API KEY" : try: from google.colab import userdata os.environ "NVIDIA API KEY" = userdata.get "NVIDIA API KEY" except Exception: os.environ "NVIDIA API KEY" = getpass "NVIDIA API KEY nvapi-... : " .strip API KEY = os.environ.get "NVIDIA API KEY", "" .strip HAVE KEY = API KEY.startswith "nvapi-" print "API key present:", HAVE KEY PAGE ELEMENTS URL = "https://ai.api.nvidia.com/v1/cv/nvidia/nemotron-page-elements-v3" OCR URL = "https://ai.api.nvidia.com/v1/cv/nvidia/nemotron-ocr-v1" TABLE STRUCT URL = "https://ai.api.nvidia.com/v1/cv/nvidia/nemotron-table-structure-v1" GRAPHIC ELEM URL = "https://ai.api.nvidia.com/v1/cv/nvidia/nemotron-graphic-elements-v1" EMBED URL = "https://integrate.api.nvidia.com/v1/embeddings" RERANK URL = "https://ai.api.nvidia.com/v1/retrieval/nvidia/llama-nemotron-rerank-vl-1b-v2/reranking" CHAT URL = "https://integrate.api.nvidia.com/v1" EMBED MODEL = "nvidia/llama-nemotron-embed-1b-v2" RERANK MODEL = "nvidia/llama-nemotron-rerank-vl-1b-v2" LLM MODEL = "nvidia/llama-3.3-nemotron-super-49b-v1.5" LANCEDB URI, TABLE = "./lancedb", "colab demo" df = df offline if HAVE KEY: print "\n=== STAGE 2: multimodal ingest via hosted NIMs ===" ing = create ingestor run mode="inprocess", allow no gpu=True, error policy="collect", .files DOCS .extract extract text=True, extract tables=True, extract charts=True, extract infographics=True, extract images=False, method="pdfium", dpi=200, table output format="markdown", page elements invoke url=PAGE ELEMENTS URL, ocr invoke url=OCR URL, table structure invoke url=TABLE STRUCT URL, graphic elements invoke url=GRAPHIC ELEM URL, api key=API KEY, request timeout s=120.0, split config={"text": {"max tokens": 512, "overlap tokens": 64}}, .dedup content hash=True, bbox iou=True, iou threshold=0.45 .embed embedding endpoint=EMBED URL, model name=EMBED MODEL, embed model name=EMBED MODEL, api key=API KEY, input type="passage", inference batch size=16, nim http max concurrent=8, .vdb upload vdb op="lancedb", vdb kwargs={ "uri": LANCEDB URI, "table name": TABLE, "overwrite": True, "create index": True, "index type": "IVF HNSW SQ", "metric": "l2", }, t0 = time.time df = ing.ingest show progress=True print f"ingested in {time.time -t0:.1f}s - {df.shape}" We securely load the NVIDIA API key and define the hosted NIM endpoints for layout detection, OCR, table extraction, graphic analysis, embedding, reranking, and generation. We create a multimodal ingestion pipeline that extracts text, tables, charts, and infographics while applying token-aware chunking and content deduplication. We generate embeddings for the extracted content and upload the resulting vectors and metadata to a LanceDB table. print "\n=== Extraction inspection ===" for col in "tables", "charts", "infographics", "images" : if col in df.columns: n = int df col .apply lambda v: len v if isinstance v, list, tuple else 0 .sum print f" {col:<14} {n}" pages = to markdown by page df print "\npages rendered to markdown:", list pages.keys print "\n--- page 1 markdown first 900 chars ---\n", pages min pages :900 full md = to markdown df if full md: with open "extracted.md", "w" as f: f.write full md print "\nfull document markdown - extracted.md" if HAVE KEY: print "\n=== STAGE 3: dense retrieval ===" retriever = Retriever run mode="service", top k=5, rerank=False, vdb kwargs={"uri": LANCEDB URI, "table name": TABLE}, embed kwargs={ "embedding endpoint": EMBED URL, "model name": EMBED MODEL, "embed model name": EMBED MODEL, "api key": API KEY, "input type": "query", }, QUERIES = "Given their activities, which animal is responsible for the typos in my documents?", "What is the most expensive gadget and how much does it cost?", "Which animal is at the beach?", def show hits, label="" : print f"\n--- {label} ---" for i, h in enumerate hits, 1 : meta = h.get "metadata" if isinstance meta, str : try: meta = json.loads meta except Exception: meta = {} page = meta or {} .get "page number", "?" score = h.get " distance", h.get "rerank score", "" body = " ".join str h.get "text", "" .split :180 print f" {i}. p{page} score={score} {body}" show retriever.query QUERIES 0 , "single query" for q, hits in zip QUERIES, retriever.queries QUERIES, top k=3 : show hits, q :60 We inspect the extracted multimodal elements and convert the processed document into page-level and full-document Markdown. We configure a dense retriever that embeds user queries and searches the LanceDB vector index for the most relevant document chunks. We test both individual and batched queries while displaying page numbers, similarity scores, and retrieved text previews. if HAVE KEY: print "\n=== STAGE 4: retrieve + VL rerank ===" reranking = Retriever run mode="service", top k=5, rerank=True, vdb kwargs={"uri": LANCEDB URI, "table name": TABLE}, embed kwargs={ "embedding endpoint": EMBED URL, "model name": EMBED MODEL, "embed model name": EMBED MODEL, "api key": API KEY, "input type": "query", }, rerank kwargs={ "model name": RERANK MODEL, "invoke url": RERANK URL, "api key": API KEY, "refine factor": 4, "batch size": 16, }, try: show reranking.query QUERIES 0 , "reranked" except Exception as e: print "rerank unavailable, dense results stand:", type e . name , str e :160 if HAVE KEY: print "\n=== STAGE 5: filtered retrieval ===" try: hits = retriever.query "gadget costs", top k=5, vdb kwargs={"where": "text LIKE '%Cost%'"}, show hits, "where: text LIKE '%Cost%'" except Exception as e: print "filter push-down failed:", type e . name , str e :160 import lancedb tbl = lancedb.connect LANCEDB URI .open table TABLE print "\nrows in LanceDB:", tbl.count rows print tbl.to pandas "text" .head 3 .to string We create a vision-language reranking pipeline that retrieves a wider candidate set and reorders the results according to semantic relevance. We also apply a text-based filter to narrow retrieval results to chunks containing specific content from the document. We directly inspect the LanceDB table to verify the number of stored records and examine the indexed text. python if HAVE KEY: print "\n=== STAGE 6: RAG answer ===" from openai import OpenAI client = OpenAI base url=CHAT URL, api key=API KEY def rag question, k=5 : hits = retriever.query question, top k=k ctx = for i, h in enumerate hits, 1 : meta = h.get "metadata" if isinstance meta, str : try: meta = json.loads meta except Exception: meta = {} ctx.append f" {i} page { meta or {} .get 'page number','?' } \n{h.get 'text','' }" prompt = textwrap.dedent f"""\ Answer the question using ONLY the numbered context below. Cite the sources you used as 1 , 2 , etc. If the context is insufficient, say so plainly. Context: {chr 10 .join ctx } Question: {question} """ r = client.chat.completions.create model=LLM MODEL, messages= {"role": "user", "content": prompt} , temperature=0.0, max tokens=512, return r.choices 0 .message.content, hits for q in QUERIES :2 : try: ans, = rag q print f"\nQ: {q}\nA: {ans}\n" + "-" 70 except Exception as e: print "generation failed:", type e . name , str e :200 if HAVE KEY: print "\n=== Recall@k check ===" GOLD = "which animal is jumping onto a laptop", "Cat" , "what does the chart show", "Gadgets" , "which animal is at the beach", "Giraffe" , K = 5 hit lists = retriever.queries q for q, in GOLD , top k=K got = sum any exp.lower in str h.get "text", "" .lower for h in hits for , exp , hits in zip GOLD, hit lists print f"recall@{K} = {got}/{len GOLD } = {got/len GOLD :.2f}" print "\nDone. Artifacts: ./lancedb vector table , ./extracted.md markdown ." We combine the retrieved document chunks with a hosted Nemotron language model to generate answers grounded only in the supplied context. We include numbered source references and page metadata so the generated responses remain traceable to the original document. We conclude by calculating recall at k for a small set of expected answers and report the final vector database and Markdown artifacts. In conclusion, we created a complete multimodal RAG system that transforms structured and unstructured PDF content into searchable, citation-ready knowledge. We used NeMo Retriever to coordinate extraction, deduplication, chunking, embedding, vector database indexing, retrieval, and reranking while keeping the Colab runtime lightweight by delegating model inference to hosted NVIDIA NIM services. We also generated grounded answers with a Nemotron language model and measured retrieval effectiveness with a simple recall-at-k test. By completing this workflow, we established a reusable foundation for building document intelligence applications that process text, tables, charts, and visual elements through a unified retrieval pipeline. Check out the FULL CODES here. Also, feel free to follow us on and don’t forget to join our Twitter https://x.com/intent/follow?screen name=marktechpost and Subscribe to 150k+ML SubReddit https://www.reddit.com/r/machinelearningnews/ . Wait are you on telegram? our Newsletter https://www.aidevsignals.com/ now you can join us on telegram as well. https://t.me/machinelearningresearchnews Need to partner with us for promoting your GitHub Repo OR Hugging Face Page OR Product Release OR Webinar etc.? Connect with us https://forms.gle/wbash1wF6efRj8G58 Sana Hassan, a consulting intern at Marktechpost and dual-degree student at IIT Madras, is passionate about applying technology and AI to address real-world challenges. With a keen interest in solving practical problems, he brings a fresh perspective to the intersection of AI and real-life solutions.