{"slug": "how-to-build-a-multi-modal-rag-pipeline-for-images-and-documents-in-python", "title": "How to Build a Multi-Modal RAG Pipeline for Images and Documents in Python", "summary": "A KDnuggets tutorial demonstrates building a multi-modal retrieval-augmented generation pipeline in Python that parses PDFs into separate text and image elements with the unstructured library, summarizes each extracted image using GPT-4o, and stores both in a multi-vector retrieval system queried via LangChain and the OpenAI API. The pipeline uses partition_pdf with extract_image_block_types set to \"Image\", infer_table_structure enabled, and a by_title chunking strategy capped at 4000 characters, requiring the poppler-utils system dependency to extract images. The approach targets documents mixing prose, tables, and embedded charts that text-only chunking handles poorly.", "body_md": "In this article, you will learn how to build a multi-modal RAG pipeline in Python that retrieves and reasons over both text and images extracted from complex documents.\n\nTopics we will cover include:\n\n- How to parse a PDF into separate text and image elements using the `unstructured` library.\n- How to generate searchable text summaries for extracted images using GPT-4o, and store everything in a multi-vector retrieval system.\n- How to query the pipeline with a multimodal prompt that passes both text chunks and raw images to GPT-4o for a grounded final answer.\n\nStandard retrieval-augmented generation works great for plain text. But feed it a PDF packed with embedded charts, scanned diagrams, or data tables, and things fall apart fast. Text-based chunking destroys table formatting and skips images, so your pipeline ends up answering questions from an incomplete picture of the document.\n\nA multi-modal RAG pipeline fixes this by treating images as first-class retrievable objects alongside text. In this tutorial, we’ll build one in Python using [LangChain](https://www.langchain.com/), the [unstructured](https://docs.unstructured.io/) library, and the [OpenAI API](https://platform.openai.com/docs/overview). We’ll parse a complex PDF into separate text and image elements, generate searchable summaries for each image, store everything in a multi-vector retrieval system, and query it with a prompt that returns both text and visual evidence to GPT-4o for a final answer.\n\nAs covered in [The Multimodal AI Guide on KDnuggets](https://www.kdnuggets.com/the-multimodal-ai-guide-vision-voice-text-and-beyond), modern AI systems are designed to reason across content types natively. This tutorial applies that principle to a practical problem: documents that mix prose, tables, and embedded images. Where [the MLM multi-modal RAG implementation](https://machinelearningmastery.com/implementing-multi-modal-rag-systems/) takes a local PyTorch approach suited to audio and image-heavy workloads, this guide uses an API-driven architecture that runs on standard hardware and fits business document use cases well.\n\n## Prerequisites and Setup\n\nYou’ll need an OpenAI API key and the following packages installed. The `poppler-utils` system dependency is required by `unstructured` to extract images from PDFs.\n\n```\npip install langchain langchain-openai unstructured[pdf] chromadb pillow\napt-get install -y poppler-utils\n\n12\n\npip install langchain langchain-openai unstructured[pdf] chromadb pillowapt-get install -y poppler-utils\n```\n\nSet your API key as an environment variable before running any code:\n\n``` python\nimport os\nos.environ[\"OPENAI_API_KEY\"] = \"your-api-key-here\"\n\n12\n\nimport osos.environ[\"OPENAI_API_KEY\"] = \"your-api-key-here\"\n```\n\nThe pipeline has four stages: parse, summarize, store, and retrieve. Each one maps to one of the code blocks below.\n\n## Step 1: Parsing the Document into Text and Image Elements\n\nThe `unstructured` library does the heavy lifting here. Its `partition_pdf()` function reads the document and classifies each element by type. It separates paragraphs, tables, and embedded images instead of treating the whole file as a flat stream of characters.\n\n``` python\nfrom unstructured.partition.pdf import partition_pdf\n\nraw_elements = partition_pdf(\n    filename=\"report.pdf\",\n    extract_image_block_types=[\"Image\"],\n    extract_image_block_output_dir=\"./extracted_images\",\n    infer_table_structure=True,\n    chunking_strategy=\"by_title\",\n    max_characters=4000,\n)\n\n12345678910\n\nfrom unstructured.partition.pdf import partition_pdf raw_elements = partition_pdf(    filename=\"report.pdf\",    extract_image_block_types=[\"Image\"],    extract_image_block_output_dir=\"./extracted_images\",    infer_table_structure=True,    chunking_strategy=\"by_title\",    max_characters=4000,)\n```\n\n`extract_image_block_output_dir` tells `unstructured` where to save extracted image files, while `extract_image_block_types` controls which element types get pulled out. Next, separate the returned elements into text and image lists:\n\n```\ntext_elements = []\nimage_paths = []\n\nfor el in raw_elements:\n    if \"CompositeElement\" in str(type(el)):\n        text_elements.append(str(el))\n    elif \"Image\" in str(type(el)):\n        image_paths.append(el.metadata.image_path)\n\nprint(f\"Extracted {len(text_elements)} text chunks, {len(image_paths)} images\")\n\n12345678910\n\ntext_elements = []image_paths = [] for el in raw_elements:    if \"CompositeElement\" in str(type(el)):        text_elements.append(str(el))    elif \"Image\" in str(type(el)):        image_paths.append(el.metadata.image_path) print(f\"Extracted {len(text_elements)} text chunks, {len(image_paths)} images\")\n```\n\nThe loop separates composite text chunks from image elements, storing the image file paths for the next step.\n\n## Step 2: Generating Searchable Summaries for Each Image\n\nText-based vector databases can’t index raw images, so each image needs a text representation that captures its content in a way that’s retrievable by a natural language query. This step passes every extracted image to GPT-4o with a prompt designed to produce dense, searchable descriptions.\n\n``` python\nimport base64\nfrom openai import OpenAI\n\nclient = OpenAI()\n\n1234\n\nimport base64from openai import OpenAI client = OpenAI()\n```\n\nWith the client ready, define a function that base64-encodes each image and sends it to GPT-4o with a retrieval-focused prompt:\n\n``` python\ndef summarize_image(image_path):\n    with open(image_path, \"rb\") as f:\n        image_data = base64.b64encode(f.read()).decode(\"utf-8\")\n    response = client.chat.completions.create(\n        model=\"gpt-4o\",\n        messages=[{\"role\": \"user\", \"content\": [\n            {\"type\": \"image_url\",\n             \"image_url\": {\"url\": f\"data:image/png;base64,{image_data}\"}},\n            {\"type\": \"text\",\n             \"text\": \"Describe this image in detail for use in a search system.\"}\n        ]}]\n    )\n    return response.choices[0].message.content\n\n12345678910111213\n\ndef summarize_image(image_path):    with open(image_path, \"rb\") as f:        image_data = base64.b64encode(f.read()).decode(\"utf-8\")    response = client.chat.completions.create(        model=\"gpt-4o\",        messages=[{\"role\": \"user\", \"content\": [            {\"type\": \"image_url\",             \"image_url\": {\"url\": f\"data:image/png;base64,{image_data}\"}},            {\"type\": \"text\",             \"text\": \"Describe this image in detail for use in a search system.\"}        ]}]    )    return response.choices[0].message.content\n```\n\nTo run it across all extracted images:\n\n```\nimage_summaries = [summarize_image(p) for p in image_paths]\n\n1\n\nimage_summaries = [summarize_image(p) for p in image_paths]\n```\n\nEach summary becomes the searchable proxy for its source image. The raw image file is kept separately and retrieved by reference when a query matches its summary.\n\nA note on costs: every extracted image triggers a GPT-4o API call at indexing time. If you’re working with large documents full of embedded logos or icons, consider filtering out images below a minimum file size before passing them to `summarize_image()`.\n\n## Step 3: Building the Multi-Vector Store\n\nThis is where the architecture diverges from standard RAG. A `MultiVectorRetriever` maintains two stores: a vector store that holds embedded text chunks and image summaries for similarity search, and a document store that holds the raw source content linked to those embeddings through unique IDs.\n\n``` python\nimport uuid\nfrom langchain_openai import OpenAIEmbeddings\nfrom langchain_chroma import Chroma\nfrom langchain.storage import InMemoryStore\nfrom langchain.retrievers.multi_vector import MultiVectorRetriever\nfrom langchain_core.documents import Document\n\nembeddings = OpenAIEmbeddings(model=\"text-embedding-3-small\")\nvectorstore = Chroma(embedding_function=embeddings)\ndocstore = InMemoryStore()\nretriever = MultiVectorRetriever(vectorstore=vectorstore, docstore=docstore)\ntext_ids = [str(uuid.uuid4()) for _ in text_elements]\nimage_ids = [str(uuid.uuid4()) for _ in image_summaries]\n\n12345678910111213\n\nimport uuidfrom langchain_openai import OpenAIEmbeddingsfrom langchain_chroma import Chromafrom langchain.storage import InMemoryStorefrom langchain.retrievers.multi_vector import MultiVectorRetrieverfrom langchain_core.documents import Document embeddings = OpenAIEmbeddings(model=\"text-embedding-3-small\")vectorstore = Chroma(embedding_function=embeddings)docstore = InMemoryStore()retriever = MultiVectorRetriever(vectorstore=vectorstore, docstore=docstore)text_ids = [str(uuid.uuid4()) for _ in text_elements]image_ids = [str(uuid.uuid4()) for _ in image_summaries]\n```\n\nThis initializes both stores and generates a unique ID for each element. The vector store holds the searchable summaries; the document store holds the raw content. With the IDs in place, populate both:\n\n```\nretriever.vectorstore.add_documents(\n    [Document(page_content=t, metadata={\"doc_id\": i})\n     for t, i in zip(text_elements + image_summaries, text_ids + image_ids)]\n)\nraw_docs = (\n    [Document(page_content=t, metadata={\"type\": \"text\"}) for t in text_elements] +\n    [Document(page_content=p, metadata={\"type\": \"image\"}) for p in image_paths]\n)\nretriever.docstore.mset(list(zip(text_ids + image_ids, raw_docs)))\n\n123456789\n\nretriever.vectorstore.add_documents(    [Document(page_content=t, metadata={\"doc_id\": i})     for t, i in zip(text_elements + image_summaries, text_ids + image_ids)])raw_docs = (    [Document(page_content=t, metadata={\"type\": \"text\"}) for t in text_elements] +    [Document(page_content=p, metadata={\"type\": \"image\"}) for p in image_paths])retriever.docstore.mset(list(zip(text_ids + image_ids, raw_docs)))\n```\n\nWrapping raw content in `Document` objects with an explicit `type` field keeps the retrieval logic in Step 4 clean. Each ID links an embedded summary in the vector store to its corresponding raw `Document` in the document store.\n\n## Step 4: Querying the Pipeline with Multimodal Retrieval\n\nAt query time, the retriever embeds the question and finds the closest matches across both text and image summaries. If a match points to an image path, the raw file is loaded and passed to GPT-4o alongside any retrieved text. GPT-4o then generates an answer grounded in both content types.\n\n``` python\ndef query_pipeline(question):\n    docs = retriever.invoke(question)\n    text_content, image_content = [], []\n    for doc in docs:\n        if doc.metadata.get(\"type\") == \"image\":\n            with open(doc.page_content, \"rb\") as f:\n                img_b64 = base64.b64encode(f.read()).decode(\"utf-8\")\n            image_content.append({\"type\": \"image_url\",\n                \"image_url\": {\"url\": f\"data:image/png;base64,{img_b64}\"}})\n        else:\n            text_content.append({\"type\": \"text\", \"text\": doc.page_content})\n\n1234567891011\n\ndef query_pipeline(question):    docs = retriever.invoke(question)    text_content, image_content = [], []    for doc in docs:        if doc.metadata.get(\"type\") == \"image\":            with open(doc.page_content, \"rb\") as f:                img_b64 = base64.b64encode(f.read()).decode(\"utf-8\")            image_content.append({\"type\": \"image_url\",                \"image_url\": {\"url\": f\"data:image/png;base64,{img_b64}\"}})        else:            text_content.append({\"type\": \"text\", \"text\": doc.page_content})\n```\n\nThe `metadata` check routes each retrieved `Document` cleanly: image paths get opened and base64-encoded, while text chunks pass through as-is. The function then assembles both into a single multimodal GPT-4o call:\n\n```\n    response = client.chat.completions.create(\n        model=\"gpt-4o\",\n        messages=[{\"role\": \"user\",\n            \"content\": text_content + image_content +\n                [{\"type\": \"text\", \"text\": question}]}]\n    )\n    return response.choices[0].message.content\n\n1234567\n\n    response = client.chat.completions.create(        model=\"gpt-4o\",        messages=[{\"role\": \"user\",            \"content\": text_content + image_content +                [{\"type\": \"text\", \"text\": question}]}]    )    return response.choices[0].message.content\n```\n\nCall the function with any question about the document:\n\n```\nanswer = query_pipeline(\"What does the revenue chart show for Q3?\")\nprint(answer)\n\n12\n\nanswer = query_pipeline(\"What does the revenue chart show for Q3?\")print(answer)\n```\n\nGPT-4o reasons across all retrieved evidence — both text and images — before returning a final answer.\n\n## Conclusion\n\nThis pipeline handles what standard RAG can’t: documents where the most important information lives in charts, diagrams, or embedded graphics. The four stages are: parse with `unstructured`, summarize images with GPT-4o, store with a multi-vector retriever, and query with a multimodal prompt. This pattern scales to research papers, financial reports, product manuals, and any document type that mixes text with visuals.\n\nFrom here, you can extend the pipeline by adding re-ranking to improve retrieval precision, persisting the [Chroma](https://www.trychroma.com/) vector store to disk for production use, or exploring vision-language models like [ColPali](https://github.com/illuin-tech/colpali) that skip the parsing step by embedding document pages as images.", "url": "https://wpnews.pro/news/how-to-build-a-multi-modal-rag-pipeline-for-images-and-documents-in-python", "canonical_source": "https://machinelearningmastery.com/how-to-build-a-multi-modal-rag-pipeline-for-images-and-documents-in-python/", "published_at": "2026-10-21 11:00:00+00:00", "updated_at": "2026-10-21 18:10:55.513538+00:00", "lang": "en", "topics": ["artificial-intelligence", "generative-ai", "large-language-models", "ai-tools", "computer-vision"], "entities": ["KDnuggets", "LangChain", "unstructured", "OpenAI", "GPT-4o", "ChromaDB", "Python"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/how-to-build-a-multi-modal-rag-pipeline-for-images-and-documents-in-python", "markdown": "https://wpnews.pro/news/how-to-build-a-multi-modal-rag-pipeline-for-images-and-documents-in-python.md", "text": "https://wpnews.pro/news/how-to-build-a-multi-modal-rag-pipeline-for-images-and-documents-in-python.txt", "jsonld": "https://wpnews.pro/news/how-to-build-a-multi-modal-rag-pipeline-for-images-and-documents-in-python.jsonld"}}