A Deep Dive into the Microsoft Foundry Document Intelligence SDK: From PDF to Structured Data

A developer published a hands-on deep dive into Microsoft's Document Intelligence SDK (formerly Form Recognizer), demonstrating how to convert scanned invoices, contracts, and receipts into structured data for RAG pipelines. The walkthrough covers using the DocumentIntelligenceClient and DocumentIntelligenceAdministrationClient with prebuilt models like prebuilt-layout, prebuilt-invoice, and prebuilt-receipt, extracting layout as GitHub-flavored markdown, pulling named fields with confidence scores, classifying documents, and training custom extraction models on labeled data.

Most "RAG over PDFs" pipelines have a step nobody talks about much: something has to turn a scanned invoice, a multi-column contract, or a photographed receipt into text a model can actually reason over. On Microsoft's stack, that something is usually the Document Intelligence SDK, formerly Form Recognizer, and it's worth understanding on its own terms rather than treating it as a black box that happens before the interesting part starts. This is a hands-on deep dive into that SDK specifically. Not a tour of every Foundry Tools SDK, Vision and Speech and Content Safety each deserve their own treatment, but a real build using Document Intelligence: extracting layout as clean markdown, pulling structured fields out of a known document type, classifying documents before routing them, and training a custom extraction model on your own labeled data. Two clients, and three kinds of model, cover almost everything this SDK does: DocumentIntelligenceClient begin analyze document , and a model id parameter decides what kind of analysis happens. It's a long-running operation, so every call returns a poller. DocumentIntelligenceAdministrationClient prebuilt-layout , prebuilt-invoice , prebuilt-receipt , prebuilt-idDocument , prebuilt-read , and others handle common, well-known document shapes out of the box. No training required. pip install azure-ai-documentintelligence azure-identity python from azure.ai.documentintelligence import DocumentIntelligenceClient from azure.core.credentials import AzureKeyCredential endpoint = "https://YOUR-RESOURCE.cognitiveservices.azure.com" client = DocumentIntelligenceClient endpoint=endpoint, credential=AzureKeyCredential "YOUR-KEY" For anything past local experimentation, swap the key for DefaultAzureCredential and an RBAC role scoped to the resource, the same pattern every other Foundry-adjacent SDK in this series has used. This is the single most useful call in the whole SDK if your end goal is feeding documents into a RAG pipeline. prebuilt-layout doesn't just extract text, it understands headings, tables, and section structure, and it can hand all of that back as GitHub-flavored markdown instead of a flat text blob. python from azure.ai.documentintelligence.models import AnalyzeDocumentRequest, DocumentContentFormat with open "contract.pdf", "rb" as f: poller = client.begin analyze document "prebuilt-layout", AnalyzeDocumentRequest bytes source=f.read , output content format=DocumentContentFormat.MARKDOWN, result = poller.result print result.content :500 result.content is now a markdown string, headings as , tables as GFM pipe tables, page structure preserved. That matters more than it sounds like it should: a table flattened into plain text loses its row and column relationships, and a model reasoning over that text has to reconstruct structure it was never actually given. Markdown output keeps the structure intact. For document types Document Intelligence already knows, invoices are the clearest example, you get named fields back with a confidence score per field, not just raw text. with open "invoice.pdf", "rb" as f: poller = client.begin analyze document "prebuilt-invoice", AnalyzeDocumentRequest bytes source=f.read result = poller.result for doc in result.documents: vendor = doc.fields.get "VendorName" total = doc.fields.get "InvoiceTotal" if vendor: print f"Vendor: {vendor.value string} confidence: {vendor.confidence:.2f} " if total: print f"Total: {total.value currency.amount} confidence: {total.confidence:.2f} " That confidence score isn't decoration. It's the field you should actually branch on in production code, more on that in the production section below. A few optional capabilities aren't on by default, since they add processing cost, but are worth turning on deliberately rather than discovering you needed them after the fact: python from azure.ai.documentintelligence.models import AnalyzeDocumentRequest, DocumentAnalysisFeature with open "shipping-label.pdf", "rb" as f: poller = client.begin analyze document "prebuilt-layout", AnalyzeDocumentRequest bytes source=f.read , features= DocumentAnalysisFeature.BARCODES, DocumentAnalysisFeature.FORMULAS , BARCODES extracts barcode and QR code payloads directly, useful for shipping labels and inventory documents where the barcode carries the actual identifier the text doesn't repeat. FORMULAS pulls out mathematical expressions as LaTeX, relevant if you're processing scientific or financial documents where a formula matters more than the surrounding prose. There's also a high-resolution mode for documents where small print matters, at the cost of slower processing. Real intake pipelines rarely receive one document type. A classifier solves the "what am I even looking at" problem before you commit to an extraction model. python from azure.ai.documentintelligence import DocumentIntelligenceAdministrationClient from azure.ai.documentintelligence.models import BuildDocumentClassifierRequest, ClassifierDocumentTypeDetails, AzureBlobContentSource, admin client = DocumentIntelligenceAdministrationClient endpoint=endpoint, credential=AzureKeyCredential "YOUR-KEY" poller = admin client.begin build classifier BuildDocumentClassifierRequest classifier id="support-doc-classifier", doc types={ "invoice": ClassifierDocumentTypeDetails azure blob source=AzureBlobContentSource container url="<SAS-url-to-invoices-container " , "contract": ClassifierDocumentTypeDetails azure blob source=AzureBlobContentSource container url="<SAS-url-to-contracts-container " , }, classifier = poller.result You need at least five sample documents per category to train a classifier at all, and more than that for anything you'd trust in production. Once it's built, classifying an incoming document is a single call: with open "unknown.pdf", "rb" as f: poller = client.begin classify document "support-doc-classifier", AnalyzeDocumentRequest bytes source=f.read result = poller.result for doc in result.documents: print f"Classified as: {doc.doc type} confidence: {doc.confidence:.2f} " When a document type isn't invoices, receipts, or any of the other prebuilt shapes, train your own. This needs a set of labeled training documents in Blob Storage, produced through the labeling tool in Foundry's document intelligence studio or programmatically. from azure.ai.documentintelligence.models import BuildDocumentModelRequest, AzureBlobContentSource, DocumentBuildMode, poller = admin client.begin build document model BuildDocumentModelRequest model id="acme-service-agreement-v1", build mode=DocumentBuildMode.TEMPLATE, azure blob source=AzureBlobContentSource container url="<SAS-url-to-training-container " , description="Extraction model for Acme's standard service agreement template.", model = poller.result Two build modes matter here, and they're not interchangeable. TEMPLATE mode is faster to train and works well when your documents follow a consistent visual layout, the same form filled out differently each time. NEURAL mode handles structural variation better, different layouts that still represent the same document type, at the cost of needing more training examples and longer build time. Start with TEMPLATE unless your documents genuinely vary in structure, not just content. One naming constraint worth knowing before you hit it: a custom model ID can't start with prebuilt- , since that prefix is reserved for Microsoft's own models across every resource. This is the detail that trips people up once they've also worked with the Foundry SDK or Agent Framework elsewhere in this series: Document Intelligence doesn't go through your Foundry project endpoint at all. It has its own resource, its own endpoint resource.cognitiveservices.azure.com , and its own authentication scope. That's what "Foundry Tools SDK" actually means as a category, prebuilt AI services with tool-specific endpoints, distinct from the Foundry SDK's unified project endpoint that Agent Framework and the Responses API build on. The practical upshot is the pipeline most teams actually want: run prebuilt-layout over incoming documents, get markdown back, and hand that markdown to a Foundry IQ Knowledge Base as a File Knowledge Source. Document Intelligence handles turning the PDF into clean, structured text. Foundry IQ handles chunking, embedding, and retrieval on top of it. Neither service needs to know the other exists, they just happen to compose well because markdown is a reasonable interchange format for both. TEMPLATE vs NEURAL is a real tradeoff, not a default to leave unexamined. The Document Intelligence SDK is easy to undersell because the interesting part of most AI applications feels like it's happening somewhere else, in the model, in the retrieval layer, in the agent's reasoning. But the quality ceiling of everything downstream is set right here, at the point where a physical or scanned document either does or doesn't become text a model can actually use well. Layout extraction to markdown, confidence-aware field extraction, classifiers for mixed intake, and custom models for your own document shapes cover the large majority of real document-processing needs, and all four are a few lines of SDK code once you know which one you need. The judgment call was never really about the API. It's about matching the right one of these four tools to what's actually in your inbound documents.