Building an End-to-End Document Intelligence Pipeline with deepDoctection

DeepDoctection 1.2.x enables an end-to-end document intelligence pipeline combining layout detection, table structure recognition, OCR, reading-order reconstruction, annotation linking, and structured export, as demonstrated in a tutorial that uses DocLayNet-based layout detection, Table Transformer structure recognition, and DocTR OCR. The pipeline processes PDFs and images, extends the framework with custom object types and a PipelineComponent for extracting monetary and date entities, and serializes pages into JSONL chunks for downstream RAG systems.

In this tutorial, we implement a document intelligence pipeline with deepDoctection 1.2.x https://github.com/deepdoctection/deepdoctection that combines layout detection, table structure recognition, OCR, reading-order reconstruction, annotation linking, and structured export in a single workflow. We configure the analyzer explicitly with DocLayNet-based layout detection, Table Transformer structure recognition, and DocTR OCR, then inspect the resulting Page objects to understand how deepDoctection represents text, figures, tables, relationships, provenance, and reading order. We also extend the framework by registering custom object types and implementing our own PipelineComponent for extracting monetary and date entities while classifying documents by their tabular characteristics. Finally, we assemble a custom pipeline manually with ServiceFactory, explore filtering and service rollback, serialize processed pages, and transform document annotations into ordered JSONL chunks suitable for downstream RAG and retrieval systems. pip install -q "deepdoctection" "transformers =5.2.0" "timm" "python-doctr" "pdfplumber" "networkx" "lxml" import os os.environ "DD USE TORCH" = "True" os.environ "DPI" = "200" os.environ "LOG LEVEL" = "INFO" os.environ "ENABLE DYNAMIC OBJECT TYPES" = "False" import json, re, textwrap from pathlib import Path from collections import Counter import numpy as np import matplotlib.pyplot as plt from IPython.display import HTML, display import deepdoctection as dd print "deepdoctection:", dd. version import transformers.integrations.peft as hf peft if hf peft.is peft available : hf peft.is peft available = lambda: False print "patched: PEFT adapter lookup disabled for from pretrained" mkdir -p /content/docs /content/imgs wget -q -O /content/docs/paper.pdf \ Click to access 2312.13560.pdf wget -q -O /content/imgs/finance.png \ https://raw.githubusercontent.com/deepdoctection/notebooks/main/sample/finance/1bcac3899c9cb1c0b0f650b1431d3d52 7.png PDF = Path "/content/docs/paper.pdf" PNG = Path "/content/imgs/finance.png" OUT = Path "/content/out" ; OUT.mkdir exist ok=True def show img, w=16 : if img is None: return plt.figure figsize= w, w 1.3 ; plt.axis "off" ; plt.imshow img ; plt.show def analyze any pipe, path, kw : """ Dispatch correctly for a directory, a PDF, or a single image file. DoctectionPipe can stream a directory or a PDF from disk, but a single image has no reader — path= only supplies the file name / provenance, and the pixels must be handed in via bytes=. Without this you get: ValueError: When passing a path to a single image, bytes of the image must be passed """ path = Path path if path.is dir : kw.setdefault "file type", ".jpg", ".png", ".jpeg", ".tif" return pipe.analyze path=path, kw if path.suffix.lower == ".pdf": return pipe.analyze path=path, kw if path.suffix.lower in ".png", ".jpg", ".jpeg", ".tif" : return pipe.analyze path=path, bytes=path.read bytes , kw raise ValueError f"unsupported input: {path}" We install the required deepDoctection dependencies, configure its runtime environment, and apply a compatibility patch for Transformers and PEFT. We download the sample PDF and image files that we use throughout the tutorial and prepare our output directory. We also define helper functions to visualize images and consistently analyze directories, PDFs, and individual image files. dd.print model infos add description=False, add config=False, add categories=False profile = dd.ModelCatalog.get profile "Aryn/deformable-detr-DocLayNet/model.safetensors" print "\nlayout model categories:", profile.categories print "is registered:", dd.ModelCatalog.is registered "Aryn/deformable-detr-DocLayNet/model.safetensors" config overwrite = "USE ROTATOR=False", "USE LAYOUT=True", "USE LAYOUT NMS=True", "USE TABLE SEGMENTATION=True", "USE TABLE REFINEMENT=False", "USE PDF MINER=False", "USE OCR=True", "USE LAYOUT LINK=True", "LAYOUT.WEIGHTS=Aryn/deformable-detr-DocLayNet/model.safetensors", "ITEM.WEIGHTS=deepdoctection/tatr tab struct v2/model.safetensors", "ITEM.FILTER= 'table' ", "OCR.USE DOCTR=True", "OCR.USE TESSERACT=False", "OCR.USE TEXTRACT=False", "OCR.WEIGHTS.DOCTR WORD=doctr/db resnet50/db resnet50-ac60cadc.pt", "OCR.WEIGHTS.DOCTR RECOGNITION=doctr/crnn vgg16 bn/crnn vgg16 bn-0417f351.pt", "SEGMENTATION.THRESHOLD ROWS=0.4", "SEGMENTATION.THRESHOLD COLS=0.4", "SEGMENTATION.FULL TABLE TILING=True", "WORD MATCHING.RULE=ioa", "WORD MATCHING.THRESHOLD=0.3", "WORD MATCHING.MAX PARENT ONLY=True", "TEXT ORDERING.INCLUDE RESIDUAL TEXT CONTAINER=True", "TEXT ORDERING.PARAGRAPH BREAK=0.035", "TEXT ORDERING.BROKEN LINE TOLERANCE=0.003", "LAYOUT LINK.PARENTAL CATEGORIES= 'figure','table' ", "LAYOUT LINK.CHILD CATEGORIES= 'caption' ", analyzer = dd.get dd analyzer config overwrite=config overwrite print "\n--- pipeline ---" for sid, name in analyzer.get pipeline info .items : print f"{sid} {name}" print "\n--- what this pipeline produces ---" print analyzer.get meta annotation We inspect deepDoctection’s model registry to verify the layout model and its supported document categories. We explicitly configure the analyzer to combine layout detection, table segmentation, DocTR OCR, word matching, reading-order reconstruction, and layout linking. We then initialize the analyzer and inspect its pipeline components and the annotation types that it produces. df = analyze any analyzer, PDF, session id="tutorial01", max datapoints=3 df.reset state pages = list df print f"\nparsed {len pages } pages" page = pages 0 show page.viz show figures=True, show residual layouts=True, show table structure=True print "== narrative text ==" print textwrap.fill page.text :900 , 110 print "\n== layout blocks in reading order ==" for doc id, img id, pno, ann id, order, cat, txt in page.chunks :12 : print f" {order: 3} {str cat :<15} {txt :70 r}" print "\n== category histogram ==" print Counter a.category name for a in page.get annotation for fig in page.figures: linked = fig.get relationship "layout link" print "figure", fig.annotation id :8 , "- caption ids:", i :8 for i in linked if page.words: w = page.words 0 print "\nword:", w.characters, "| service:", w.service id, "| model:", w.model id, "| bbox:", round x for x in w.bbox tbl pages = p for p in pages if p.tables if tbl pages: t = tbl pages 0 .tables 0 print f"table {t.number of rows}x{t.number of columns}, " f"max row span={t.max row span}, max col span={t.max col span}" display HTML t.html for row in t.csv :5 : print c :22 for c in row for c in t.cells :5 : print f" r{c.row number} c{c.column number} " f" span {c.row span}x{c.column span} {c.text :40 r}" else: print "no table on these pages — the finance.png sample below has one" We run the configured analyzer on the sample PDF and materialize the resulting pages from the lazy data flow. We inspect narrative text, reading-order chunks, annotation categories, figure-caption relationships, word provenance, and bounding boxes. We also access detected tables through HTML, CSV, and individual cell representations to examine their structured output. @dd.object types registry.register "CustomKey" class CustomKey dd.ObjectTypes : """Custom summary keys — must be registered to be serialisable.""" MONEY MENTIONS = "money mentions" DATE MENTIONS = "date mentions" DOC FLAVOUR = "doc flavour" @dd.object types registry.register "FlavourLabel" class FlavourLabel dd.ObjectTypes : TABULAR = "tabular" NARRATIVE = "narrative" MIXED = "mixed" MONEY = re.compile r" ?: $€£ \s?\d \d,. |\d \d,. \s? ?:USD|EUR|GBP|million|bn " DATE = re.compile r"\b ?:\d{1,2} /- \d{1,2} /- \d{2,4}|\d{4}-\d{2}-\d{2}|" r" ?:Jan|Feb|Mar|Apr|May|Jun|Jul|Aug|Sep|Oct|Nov|Dec \w \s+\d{1,2},?\s+\d{4} \b" class EntityAndFlavourService dd.PipelineComponent : def init self, name="entity flavour", tabular ratio=0.25 : self.tabular ratio = tabular ratio super . init name def serve self, dp: dd.Image - None: page = dd.Page.from image dp, text container=dd.LayoutLabel.WORD text = page.text no line break money = sorted set MONEY.findall text dates = sorted set DATE.findall text tables = page.tables table area = sum b 2 - b 0 b 3 - b 1 for b in t.bbox for t in tables ratio = table area / float page.width page.height or 1 flavor = FlavourLabel.TABULAR if ratio self.tabular ratio else FlavourLabel.NARRATIVE if not tables else FlavourLabel.MIXED self.dp manager.set summary annotation summary key=CustomKey.MONEY MENTIONS, summary name=CustomKey.MONEY MENTIONS, summary value=money self.dp manager.set summary annotation summary key=CustomKey.DATE MENTIONS, summary name=CustomKey.DATE MENTIONS, summary value=dates self.dp manager.set summary annotation summary key=CustomKey.DOC FLAVOUR, summary name=flavour, summary score=round ratio, 4 def clone self : return self. class self.name, self.tabular ratio def get meta annotation self - dd.MetaAnnotation: return dd.MetaAnnotation image annotations= , sub categories={}, relationships={}, summaries= CustomKey.MONEY MENTIONS, CustomKey.DATE MENTIONS, CustomKey.DOC FLAVOUR , for k in CustomKey.MONEY MENTIONS, CustomKey.DATE MENTIONS, CustomKey.DOC FLAVOUR : dd.Page.add attribute name k We register custom object types for extracted monetary mentions, date mentions, and document flavor classifications. We implement a custom deepDoctection pipeline component that analyzes page text and table coverage to generate these page-level summaries. We then expose the custom summary fields as Page attributes so that we can access them directly from processed documents. python from deepdoctection.analyzer import cfg, ServiceFactory cfg.freeze False cfg.USE TABLE SEGMENTATION = True cfg.freeze True components = layout detector = ServiceFactory.build layout detector cfg, mode="LAYOUT" components.append ServiceFactory.build layout service cfg, detector=layout detector, mode="LAYOUT" components.append ServiceFactory.build layout nms service cfg item detector = ServiceFactory.build layout detector cfg, mode="ITEM" components.append ServiceFactory.build sub image service cfg, detector=item detector, mode="ITEM" components.append ServiceFactory.build table segmentation service cfg, detector=item detector word detector = ServiceFactory.build doctr word detector cfg components.append ServiceFactory.build doctr word detector service word detector components.append ServiceFactory.build text extraction service cfg, ServiceFactory.build ocr detector cfg components.append ServiceFactory.build word matching service cfg components.append ServiceFactory.build text order service cfg components.append EntityAndFlavourService custom pipe = dd.DoctectionPipe pipeline component list=components print "\ncustom pipeline:", list custom pipe.get pipeline info .values df2 = analyze any custom pipe, PNG df2.reset state fin page = next iter df2 print "flavour :", fin page.doc flavour print "money :", fin page.money mentions :10 print "dates :", fin page.date mentions :10 show fin page.viz show table structure=True , w=13 def skip if no table dp: dd.Image - bool: return "table" not in {a.category name for a in dp.get annotation } components -1 .set inbound filter skip if no table det sid = next sid for sid, n in analyzer.get pipeline info .items if n.startswith "image doctr" det comp = analyzer.get pipeline component service id=det sid df undo = det comp.undo dd.DataFromList p.base image for p in pages df undo.reset state undone = list df undo print "annotations before/after undo:", len pages 0 .get annotation , len dd.Page.from image undone 0 .get annotation We manually assemble a deepDoctection pipeline with ServiceFactory, combining layout analysis, table processing, OCR, text ordering, and our custom component. We execute this custom pipeline on the financial document image and inspect the detected flavor, monetary values, dates, and table structure. We also apply an inbound filter and demonstrate how we undo the annotations produced by a selected DocTR service. for i, p in enumerate pages : p.save image to json=False, path=OUT / f"page {i}.json" restored = dd.Page.from file str OUT / "page 0.json" print "round-trip:", len restored.get annotation , "of", len pages 0 .get annotation , "annotations restored" records = for p in pages: for doc id, img id, pno, ann id, order, cat, txt in p.chunks: if txt and txt.strip : records.append {"document id": doc id, "page": pno, "order": order, "category": str cat , "annotation id": ann id, "text": txt} for t in p.tables: records.append {"document id": p.document id, "page": p.page number, "order": -1, "category": "table html", "annotation id": t.annotation id, "text": t.html} OUT / "chunks.jsonl" .write text "\n".join json.dumps r for r in records print f"\n{len records } chunks - {OUT/'chunks.jsonl'}" print json.dumps records 0 , indent=2 :400 We serialize each processed page to JSON while preserving its structural annotations without embedding the original image data. We reload a saved page and compare annotation counts to verify that the structural information survives serialization. We finally transform narrative chunks and table HTML into JSONL records that we can use directly in RAG, retrieval, and downstream document-processing pipelines. In conclusion, we developed a practical understanding of how deepDoctection orchestrates multiple document-analysis models and rule-based services into a configurable processing pipeline. We moved beyond simply running a predefined analyzer by inspecting model registrations, controlling individual services, accessing structured page-level annotations, extracting tables, creating custom summary metadata, and composing our own pipeline stages. We also examined how service filtering and undo operations affect annotations, giving us finer control over complex document-processing workflows. Finally, we serialized the processed document structure. We generated RAG-ready chunks, giving us a reusable foundation for building document search, knowledge extraction, retrieval-augmented generation, and other production-oriented document AI applications. Check out the FULL CODES here https://github.com/MARKTECHPOST-AI-MEDIA-INC/AI-Agents-Projects-Tutorials/blob/main/Computer%20Vision/deepdoctection advanced document intelligence pipeline Marktechpost.ipynb . Also, feel free to follow us on Twitter and don’t forget to join our and Subscribe to 150k+ML SubReddit https://www.reddit.com/r/machinelearningnews/ . Wait are you on telegram? our Newsletter https://magic.beehiiv.com/v1/f5e63dd4-5653-4f09-83e2-321a8b1ba526?email={{email}} now you can join us on telegram as well. https://t.me/machinelearningresearchnews Need to partner with us for promoting your GitHub Repo OR Hugging Face Page OR Product Release OR Webinar etc.? Connect with us https://forms.gle/wbash1wF6efRj8G58 Sana Hassan, a consulting intern at Marktechpost and dual-degree student at IIT Madras, is passionate about applying technology and AI to address real-world challenges. With a keen interest in solving practical problems, he brings a fresh perspective to the intersection of AI and real-life solutions.