{"slug": "building-an-end-to-end-document-intelligence-pipeline-with-deepdoctection", "title": "Building an End-to-End Document Intelligence Pipeline with deepDoctection", "summary": "DeepDoctection 1.2.x enables an end-to-end document intelligence pipeline combining layout detection, table structure recognition, OCR, reading-order reconstruction, annotation linking, and structured export, as demonstrated in a tutorial that uses DocLayNet-based layout detection, Table Transformer structure recognition, and DocTR OCR. The pipeline processes PDFs and images, extends the framework with custom object types and a PipelineComponent for extracting monetary and date entities, and serializes pages into JSONL chunks for downstream RAG systems.", "body_md": "In this tutorial, we implement a document intelligence pipeline with [ deepDoctection 1.2.x](https://github.com/deepdoctection/deepdoctection) that combines layout detection, table structure recognition, OCR, reading-order reconstruction, annotation linking, and structured export in a single workflow. We configure the analyzer explicitly with DocLayNet-based layout detection, Table Transformer structure recognition, and DocTR OCR, then inspect the resulting Page objects to understand how deepDoctection represents text, figures, tables, relationships, provenance, and reading order. We also extend the framework by registering custom object types and implementing our own PipelineComponent for extracting monetary and date entities while classifying documents by their tabular characteristics. Finally, we assemble a custom pipeline manually with ServiceFactory, explore filtering and service rollback, serialize processed pages, and transform document annotations into ordered JSONL chunks suitable for downstream RAG and retrieval systems.\n\n```\n!pip install -q \"deepdoctection\" \"transformers>=5.2.0\" \"timm\" \"python-doctr\" \"pdfplumber\" \"networkx\" \"lxml\"\nimport os\nos.environ[\"DD_USE_TORCH\"]  = \"True\"\nos.environ[\"DPI\"]           = \"200\"\nos.environ[\"LOG_LEVEL\"]     = \"INFO\"\nos.environ[\"ENABLE_DYNAMIC_OBJECT_TYPES\"] = \"False\"\nimport json, re, textwrap\nfrom pathlib import Path\nfrom collections import Counter\nimport numpy as np\nimport matplotlib.pyplot as plt\nfrom IPython.display import HTML, display\nimport deepdoctection as dd\nprint(\"deepdoctection:\", dd.__version__)\nimport transformers.integrations.peft as _hf_peft\nif _hf_peft.is_peft_available():\n   _hf_peft.is_peft_available = lambda: False\n   print(\"patched: PEFT adapter lookup disabled for from_pretrained\")\n!mkdir -p /content/docs /content/imgs\n!wget -q -O /content/docs/paper.pdf \\\n Click to access 2312.13560.pdf\n\n!wget -q -O /content/imgs/finance.png \\\n https://raw.githubusercontent.com/deepdoctection/notebooks/main/sample/finance/1bcac3899c9cb1c0b0f650b1431d3d52_7.png\nPDF = Path(\"/content/docs/paper.pdf\")\nPNG = Path(\"/content/imgs/finance.png\")\nOUT = Path(\"/content/out\"); OUT.mkdir(exist_ok=True)\ndef show(img, w=16):\n   if img is None: return\n   plt.figure(figsize=(w, w * 1.3)); plt.axis(\"off\"); plt.imshow(img); plt.show()\ndef analyze_any(pipe, path, **kw):\n   \"\"\"\n   Dispatch correctly for a directory, a PDF, or a single image file.\n   DoctectionPipe can stream a directory or a PDF from disk, but a *single*\n   image has no reader — path= only supplies the file name / provenance, and\n   the pixels must be handed in via bytes=. Without this you get:\n     ValueError: When passing a path to a single image, bytes of the image\n                 must be passed\n   \"\"\"\n   path = Path(path)\n   if path.is_dir():\n       kw.setdefault(\"file_type\", [\".jpg\", \".png\", \".jpeg\", \".tif\"])\n       return pipe.analyze(path=path, **kw)\n   if path.suffix.lower() == \".pdf\":\n       return pipe.analyze(path=path, **kw)\n   if path.suffix.lower() in (\".png\", \".jpg\", \".jpeg\", \".tif\"):\n       return pipe.analyze(path=path, bytes=path.read_bytes(), **kw)\n   raise ValueError(f\"unsupported input: {path}\")\n```\n\nWe install the required deepDoctection dependencies, configure its runtime environment, and apply a compatibility patch for Transformers and PEFT. We download the sample PDF and image files that we use throughout the tutorial and prepare our output directory. We also define helper functions to visualize images and consistently analyze directories, PDFs, and individual image files.\n\n```\ndd.print_model_infos(add_description=False, add_config=False, add_categories=False)\nprofile = dd.ModelCatalog.get_profile(\"Aryn/deformable-detr-DocLayNet/model.safetensors\")\nprint(\"\\nlayout model categories:\", profile.categories)\nprint(\"is registered:\", dd.ModelCatalog.is_registered(\"Aryn/deformable-detr-DocLayNet/model.safetensors\"))\nconfig_overwrite = [\n   \"USE_ROTATOR=False\",\n   \"USE_LAYOUT=True\",\n   \"USE_LAYOUT_NMS=True\",\n   \"USE_TABLE_SEGMENTATION=True\",\n   \"USE_TABLE_REFINEMENT=False\",\n   \"USE_PDF_MINER=False\",\n   \"USE_OCR=True\",\n   \"USE_LAYOUT_LINK=True\",\n   \"LAYOUT.WEIGHTS=Aryn/deformable-detr-DocLayNet/model.safetensors\",\n   \"ITEM.WEIGHTS=deepdoctection/tatr_tab_struct_v2/model.safetensors\",\n   \"ITEM.FILTER=['table']\",\n   \"OCR.USE_DOCTR=True\",\n   \"OCR.USE_TESSERACT=False\",\n   \"OCR.USE_TEXTRACT=False\",\n   \"OCR.WEIGHTS.DOCTR_WORD=doctr/db_resnet50/db_resnet50-ac60cadc.pt\",\n   \"OCR.WEIGHTS.DOCTR_RECOGNITION=doctr/crnn_vgg16_bn/crnn_vgg16_bn-0417f351.pt\",\n   \"SEGMENTATION.THRESHOLD_ROWS=0.4\",\n   \"SEGMENTATION.THRESHOLD_COLS=0.4\",\n   \"SEGMENTATION.FULL_TABLE_TILING=True\",\n   \"WORD_MATCHING.RULE=ioa\",\n   \"WORD_MATCHING.THRESHOLD=0.3\",\n   \"WORD_MATCHING.MAX_PARENT_ONLY=True\",\n   \"TEXT_ORDERING.INCLUDE_RESIDUAL_TEXT_CONTAINER=True\",\n   \"TEXT_ORDERING.PARAGRAPH_BREAK=0.035\",\n   \"TEXT_ORDERING.BROKEN_LINE_TOLERANCE=0.003\",\n   \"LAYOUT_LINK.PARENTAL_CATEGORIES=['figure','table']\",\n   \"LAYOUT_LINK.CHILD_CATEGORIES=['caption']\",\n]\nanalyzer = dd.get_dd_analyzer(config_overwrite=config_overwrite)\nprint(\"\\n--- pipeline ---\")\nfor sid, name in analyzer.get_pipeline_info().items():\n   print(f\"{sid}  {name}\")\nprint(\"\\n--- what this pipeline produces ---\")\nprint(analyzer.get_meta_annotation())\n```\n\nWe inspect deepDoctection’s model registry to verify the layout model and its supported document categories. We explicitly configure the analyzer to combine layout detection, table segmentation, DocTR OCR, word matching, reading-order reconstruction, and layout linking. We then initialize the analyzer and inspect its pipeline components and the annotation types that it produces.\n\n```\ndf = analyze_any(analyzer, PDF, session_id=\"tutorial01\", max_datapoints=3)\ndf.reset_state()\npages = list(df)\nprint(f\"\\nparsed {len(pages)} pages\")\npage = pages[0]\nshow(page.viz(show_figures=True, show_residual_layouts=True, show_table_structure=True))\nprint(\"== narrative text ==\")\nprint(textwrap.fill(page.text[:900], 110))\nprint(\"\\n== layout blocks in reading order ==\")\nfor doc_id, img_id, pno, ann_id, order, cat, txt in page.chunks[:12]:\n   print(f\"[{order:>3}] {str(cat):<15} {txt[:70]!r}\")\nprint(\"\\n== category histogram ==\")\nprint(Counter(a.category_name for a in page.get_annotation()))\nfor fig in page.figures:\n   linked = fig.get_relationship(\"layout_link\")\n   print(\"figure\", fig.annotation_id[:8], \"-> caption ids:\", [i[:8] for i in linked])\nif page.words:\n   w = page.words[0]\n   print(\"\\nword:\", w.characters, \"| service:\", w.service_id,\n         \"| model:\", w.model_id, \"| bbox:\", [round(x) for x in w.bbox])\ntbl_pages = [p for p in pages if p.tables]\nif tbl_pages:\n   t = tbl_pages[0].tables[0]\n   print(f\"table {t.number_of_rows}x{t.number_of_columns}, \"\n         f\"max_row_span={t.max_row_span}, max_col_span={t.max_col_span}\")\n   display(HTML(t.html))\n   for row in t.csv[:5]:\n       print([c[:22] for c in row])\n   for c in t.cells[:5]:\n       print(f\"  r{c.row_number} c{c.column_number} \"\n             f\"(span {c.row_span}x{c.column_span}) {c.text[:40]!r}\")\nelse:\n   print(\"no table on these pages — the finance.png sample below has one\")\n```\n\nWe run the configured analyzer on the sample PDF and materialize the resulting pages from the lazy data flow. We inspect narrative text, reading-order chunks, annotation categories, figure-caption relationships, word provenance, and bounding boxes. We also access detected tables through HTML, CSV, and individual cell representations to examine their structured output.\n\n```\n@dd.object_types_registry.register(\"CustomKey\")\nclass CustomKey(dd.ObjectTypes):\n   \"\"\"Custom summary keys — must be registered to be serialisable.\"\"\"\n   MONEY_MENTIONS = \"money_mentions\"\n   DATE_MENTIONS  = \"date_mentions\"\n   DOC_FLAVOUR    = \"doc_flavour\"\n@dd.object_types_registry.register(\"FlavourLabel\")\nclass FlavourLabel(dd.ObjectTypes):\n   TABULAR   = \"tabular\"\n   NARRATIVE = \"narrative\"\n   MIXED     = \"mixed\"\nMONEY = re.compile(r\"(?:[$€£]\\s?\\d[\\d,.]*|\\d[\\d,.]*\\s?(?:USD|EUR|GBP|million|bn))\")\nDATE  = re.compile(r\"\\b(?:\\d{1,2}[/-]\\d{1,2}[/-]\\d{2,4}|\\d{4}-\\d{2}-\\d{2}|\"\n                  r\"(?:Jan|Feb|Mar|Apr|May|Jun|Jul|Aug|Sep|Oct|Nov|Dec)\\w*\\s+\\d{1,2},?\\s+\\d{4})\\b\")\nclass EntityAndFlavourService(dd.PipelineComponent):\n   def __init__(self, name=\"entity_flavour\", tabular_ratio=0.25):\n       self.tabular_ratio = tabular_ratio\n       super().__init__(name)\n   def serve(self, dp: dd.Image) -> None:\n       page = dd.Page.from_image(dp, text_container=dd.LayoutLabel.WORD)\n       text = page.text_no_line_break\n       money = sorted(set(MONEY.findall(text)))\n       dates = sorted(set(DATE.findall(text)))\n       tables = page.tables\n       table_area = sum((b[2] - b[0]) * (b[3] - b[1]) for b in (t.bbox for t in tables))\n       ratio = table_area / float(page.width * page.height or 1)\n       flavor = (FlavourLabel.TABULAR if ratio > self.tabular_ratio\n                  else FlavourLabel.NARRATIVE if not tables\n                  else FlavourLabel.MIXED)\n       self.dp_manager.set_summary_annotation(\n           summary_key=CustomKey.MONEY_MENTIONS, summary_name=CustomKey.MONEY_MENTIONS,\n           summary_value=money)\n       self.dp_manager.set_summary_annotation(\n           summary_key=CustomKey.DATE_MENTIONS, summary_name=CustomKey.DATE_MENTIONS,\n           summary_value=dates)\n       self.dp_manager.set_summary_annotation(\n           summary_key=CustomKey.DOC_FLAVOUR, summary_name=flavour,\n           summary_score=round(ratio, 4))\n   def clone(self):\n       return self.__class__(self.name, self.tabular_ratio)\n   def get_meta_annotation(self) -> dd.MetaAnnotation:\n       return dd.MetaAnnotation(\n           image_annotations=(),\n           sub_categories={},\n           relationships={},\n           summaries=(CustomKey.MONEY_MENTIONS, CustomKey.DATE_MENTIONS, CustomKey.DOC_FLAVOUR),\n       )\nfor k in (CustomKey.MONEY_MENTIONS, CustomKey.DATE_MENTIONS, CustomKey.DOC_FLAVOUR):\n   dd.Page.add_attribute_name(k)\n```\n\nWe register custom object types for extracted monetary mentions, date mentions, and document flavor classifications. We implement a custom deepDoctection pipeline component that analyzes page text and table coverage to generate these page-level summaries. We then expose the custom summary fields as Page attributes so that we can access them directly from processed documents.\n\n``` python\nfrom deepdoctection.analyzer import cfg, ServiceFactory\ncfg.freeze(False)\ncfg.USE_TABLE_SEGMENTATION = True\ncfg.freeze(True)\ncomponents = []\nlayout_detector = ServiceFactory.build_layout_detector(cfg, mode=\"LAYOUT\")\ncomponents.append(ServiceFactory.build_layout_service(cfg, detector=layout_detector, mode=\"LAYOUT\"))\ncomponents.append(ServiceFactory.build_layout_nms_service(cfg))\nitem_detector = ServiceFactory.build_layout_detector(cfg, mode=\"ITEM\")\ncomponents.append(ServiceFactory.build_sub_image_service(cfg, detector=item_detector, mode=\"ITEM\"))\ncomponents.append(ServiceFactory.build_table_segmentation_service(cfg, detector=item_detector))\nword_detector = ServiceFactory.build_doctr_word_detector(cfg)\ncomponents.append(ServiceFactory.build_doctr_word_detector_service(word_detector))\ncomponents.append(ServiceFactory.build_text_extraction_service(cfg, ServiceFactory.build_ocr_detector(cfg)))\ncomponents.append(ServiceFactory.build_word_matching_service(cfg))\ncomponents.append(ServiceFactory.build_text_order_service(cfg))\ncomponents.append(EntityAndFlavourService())\ncustom_pipe = dd.DoctectionPipe(pipeline_component_list=components)\nprint(\"\\ncustom pipeline:\", list(custom_pipe.get_pipeline_info().values()))\ndf2 = analyze_any(custom_pipe, PNG)\ndf2.reset_state()\nfin_page = next(iter(df2))\nprint(\"flavour  :\", fin_page.doc_flavour)\nprint(\"money    :\", fin_page.money_mentions[:10])\nprint(\"dates    :\", fin_page.date_mentions[:10])\nshow(fin_page.viz(show_table_structure=True), w=13)\ndef skip_if_no_table(dp: dd.Image) -> bool:\n   return \"table\" not in {a.category_name for a in dp.get_annotation()}\ncomponents[-1].set_inbound_filter(skip_if_no_table)\ndet_sid = next(sid for sid, n in analyzer.get_pipeline_info().items()\n              if n.startswith(\"image_doctr\"))\ndet_comp = analyzer.get_pipeline_component(service_id=det_sid)\ndf_undo = det_comp.undo(dd.DataFromList([p.base_image for p in pages]))\ndf_undo.reset_state()\nundone = list(df_undo)\nprint(\"annotations before/after undo:\",\n     len(pages[0].get_annotation()),\n     len(dd.Page.from_image(undone[0]).get_annotation()))\n```\n\nWe manually assemble a deepDoctection pipeline with ServiceFactory, combining layout analysis, table processing, OCR, text ordering, and our custom component. We execute this custom pipeline on the financial document image and inspect the detected flavor, monetary values, dates, and table structure. We also apply an inbound filter and demonstrate how we undo the annotations produced by a selected DocTR service.\n\n```\nfor i, p in enumerate(pages):\n   p.save(image_to_json=False, path=OUT / f\"page_{i}.json\")\nrestored = dd.Page.from_file(str(OUT / \"page_0.json\"))\nprint(\"round-trip:\", len(restored.get_annotation()), \"of\",\n     len(pages[0].get_annotation()), \"annotations restored\")\nrecords = []\nfor p in pages:\n   for doc_id, img_id, pno, ann_id, order, cat, txt in p.chunks:\n       if txt and txt.strip():\n           records.append({\"document_id\": doc_id, \"page\": pno, \"order\": order,\n                           \"category\": str(cat), \"annotation_id\": ann_id, \"text\": txt})\n   for t in p.tables:\n       records.append({\"document_id\": p.document_id, \"page\": p.page_number,\n                       \"order\": -1, \"category\": \"table_html\",\n                       \"annotation_id\": t.annotation_id, \"text\": t.html})\n(OUT / \"chunks.jsonl\").write_text(\"\\n\".join(json.dumps(r) for r in records))\nprint(f\"\\n{len(records)} chunks -> {OUT/'chunks.jsonl'}\")\nprint(json.dumps(records[0], indent=2)[:400])\n```\n\nWe serialize each processed page to JSON while preserving its structural annotations without embedding the original image data. We reload a saved page and compare annotation counts to verify that the structural information survives serialization. We finally transform narrative chunks and table HTML into JSONL records that we can use directly in RAG, retrieval, and downstream document-processing pipelines.\n\nIn conclusion, we developed a practical understanding of how deepDoctection orchestrates multiple document-analysis models and rule-based services into a configurable processing pipeline. We moved beyond simply running a predefined analyzer by inspecting model registrations, controlling individual services, accessing structured page-level annotations, extracting tables, creating custom summary metadata, and composing our own pipeline stages. We also examined how service filtering and undo operations affect annotations, giving us finer control over complex document-processing workflows. Finally, we serialized the processed document structure. We generated RAG-ready chunks, giving us a reusable foundation for building document search, knowledge extraction, retrieval-augmented generation, and other production-oriented document AI applications.\n\nCheck out the [FULL CODES here](https://github.com/MARKTECHPOST-AI-MEDIA-INC/AI-Agents-Projects-Tutorials/blob/main/Computer%20Vision/deepdoctection_advanced_document_intelligence_pipeline_Marktechpost.ipynb)*.* Also, feel free to follow us on ** Twitter** and don’t forget to join our\n\n**and Subscribe to**\n\n[150k+ML SubReddit](https://www.reddit.com/r/machinelearningnews/)**. Wait! are you on telegram?**\n\n[our Newsletter](https://magic.beehiiv.com/v1/f5e63dd4-5653-4f09-83e2-321a8b1ba526?email={{email}})\n\n[now you can join us on telegram as well.](https://t.me/machinelearningresearchnews)Need to partner with us for promoting your GitHub Repo OR Hugging Face Page OR Product Release OR Webinar etc.? [Connect with us](https://forms.gle/wbash1wF6efRj8G58)\n\nSana Hassan, a consulting intern at Marktechpost and dual-degree student at IIT Madras, is passionate about applying technology and AI to address real-world challenges. With a keen interest in solving practical problems, he brings a fresh perspective to the intersection of AI and real-life solutions.", "url": "https://wpnews.pro/news/building-an-end-to-end-document-intelligence-pipeline-with-deepdoctection", "canonical_source": "https://www.marktechpost.com/2026/08/23/building-an-end-to-end-document-intelligence-pipeline-with-deepdoctection/", "published_at": "2026-08-23 07:51:22+00:00", "updated_at": "2026-08-23 08:13:21.524128+00:00", "lang": "en", "topics": ["machine-learning", "computer-vision", "natural-language-processing", "developer-tools"], "entities": ["deepDoctection", "DocLayNet", "Table Transformer", "DocTR", "Transformers", "PEFT", "ServiceFactory"], "alternates": {"html": "https://wpnews.pro/news/building-an-end-to-end-document-intelligence-pipeline-with-deepdoctection", "markdown": "https://wpnews.pro/news/building-an-end-to-end-document-intelligence-pipeline-with-deepdoctection.md", "text": "https://wpnews.pro/news/building-an-end-to-end-document-intelligence-pipeline-with-deepdoctection.txt", "jsonld": "https://wpnews.pro/news/building-an-end-to-end-document-intelligence-pipeline-with-deepdoctection.jsonld"}}