Building a Document Intelligence Platform on Snowflake with Cortex AI

A document intelligence platform built on Snowflake using Cortex AI functions enables enterprises to parse, classify, extract, summarize, and search PDFs and images natively within Snowflake's governance perimeter, processing documents through a multi-stage pipeline that keeps data inside Snowflake. The platform uses AI_PARSE_DOCUMENT, AI_CLASSIFY, AI_EXTRACT, AI_COMPLETES, CORTEX.EMBED, and CORTEX.SEARCH_PREVIEW to handle OCR, classification, structured extraction, summarization, vector embeddings, and semantic search, organized into four schemas for raw ingestion, processed documents, analytics, and application objects.

Your business runs on documents. Invoices from vendors, contracts with customers, financial reports from partners, medical intake forms — thousands of PDFs flowing through your organization daily. Yet the data trapped inside them remains invisible to your analytics layer. Documents are the context layer for almost every business activity — planning, approvals, deep analysis, execution — all produce a massive trail of text, slides, and sheets. Yet traditional data stacks struggle to make sense of them at the accuracy enterprises require to trust automated processes. Traditional approaches demand stitching together OCR services, custom NLP models, orchestration engines, and external storage — creating brittle pipelines that break at scale and introduce governance blind spots. What if you could parse, classify, extract, summarize, and search documents natively inside Snowflake ? That’s exactly what we built. Our platform implements the three core document intelligence patterns that Snowflake identifies as driving real business outcomes: Our Document Intelligence Platform processes documents through a multi-stage pipeline that keeps everything within Snowflake’s governance perimeter: Key Cortex AI functions used: FunctionPurposeAI PARSE DOCUMENTOCR and layout-aware text extraction from PDFs/imagesAI CLASSIFYAutomatic document type routingAI EXTRACTStructured field extraction into JSONAI COMPLETESummarization, sentiment analysis, comparisonCORTEX.EMBEDVector embeddings for semantic search and theme mappingCORTEX.SEARCH PREVIEWFull-text semantic search over document corpus We organized the platform into four schemas reflecting the data lifecycle: CREATE DATABASE DOC INTEL;-- Raw ingestion layerCREATE SCHEMA DOC INTEL.RAW;-- Processed & classified documentsCREATE SCHEMA DOC INTEL.PROCESSED;-- Analytics, themes, and searchCREATE SCHEMA DOC INTEL.ANALYTICS;-- UDFs, secrets, and application objectsCREATE SCHEMA DOC INTEL.APP; The staging area receives documents via file upload or automated feeds: CREATE STAGE DOC INTEL.RAW.DOCUMENTS STAGE DIRECTORY = ENABLE = TRUE ENCRYPTION = TYPE = 'SNOWFLAKE SSE' ; A file log tracks every document through the pipeline: CREATE TABLE DOC INTEL.RAW.FILE LOG FILE ID VARCHAR DEFAULT UUID STRING , RELATIVE PATH VARCHAR, FILE NAME VARCHAR, FILE EXTENSION VARCHAR, PROCESSING STATUS VARCHAR DEFAULT 'PENDING', CLASSIFICATION VARCHAR, UPLOAD TS TIMESTAMP NTZ DEFAULT CURRENT TIMESTAMP ; The first pipeline stage converts raw PDFs and images into machine-readable text while preserving layout structure: CREATE OR REPLACE DYNAMIC TABLE DOC INTEL.PROCESSED.DT DOC FULL TEXT TARGET LAG = '10 minutes' WAREHOUSE = DOC INTEL WHASSELECT fl.FILE ID, fl.RELATIVE PATH, fl.FILE NAME, fl.CLASSIFICATION AS DOC TYPE, AI PARSE DOCUMENT BUILD SCOPED FILE URL @DOC INTEL.RAW.DOCUMENTS STAGE, fl.RELATIVE PATH , {'mode': 'LAYOUT'} :content::VARCHAR AS FULL TEXT, CURRENT TIMESTAMP AS PARSED ATFROM DOC INTEL.RAW.FILE LOG flWHERE fl.PROCESSING STATUS IN 'CLASSIFIED', 'COMPLETED' ; AI PARSE DOCUMENT handles PDFs, scanned images, TIFF, and Office documents. The LAYOUT mode preserves the original structure — reading order across multi-column layouts, table structures, visual hierarchy, and embedded chart images. This is critical because for enterprise documents like earnings reports with tables, charts, and footnotes or contracts spanning hundreds of pages, structure contains meaning. A flattened text stream loses the context that downstream AI systems need for accurate retrieval and reasoning. Instead of manual tagging, every document is automatically routed to the correct processing pipeline: -- Classification happens in the pipeline, but here's the native approach:SELECT FILE NAME, AI CLASSIFY FULL TEXT, 'invoice', 'contract', 'financial report', 'medical form', 'correspondence', 'purchase order', 'resume' AS CLASSIFICATIONFROM DOC INTEL.PROCESSED.DT DOC FULL TEXT; The classification output includes confidence scores, enabling automatic routing for high-confidence results and human review queues for edge cases. This pattern — where AI CLASSIFY acts as a traffic controller routing different documents to specialized extraction pipelines — eliminates the manual triage that becomes a bottleneck at enterprise scale: CREATE TABLE DOC INTEL.PROCESSED.REVIEW QUEUE FILE ID VARCHAR, DOC TYPE VARCHAR, EXTRACTION CONFIDENCE FLOAT, RAW EXTRACT VARIANT, REVIEW STATUS VARCHAR DEFAULT 'PENDING' ; Documents below a confidence threshold we use 0.7 are flagged for manual review, while everything above flows directly to extraction. This is where the real value materializes. AI EXTRACT pulls structured fields directly from unstructured text — no regex, no templates, no maintenance: CREATE OR REPLACE DYNAMIC TABLE DOC INTEL.PROCESSED.DT EXTRACT INVOICES TARGET LAG = 'DOWNSTREAM' WAREHOUSE = DOC INTEL WHASSELECT ft.FILE ID, ft.FILE NAME, AI EXTRACT ft.FULL TEXT, OBJECT CONSTRUCT 'vendor name', 'Name of the vendor or supplier', 'invoice number', 'Invoice or reference number', 'invoice date', 'Invoice date in YYYY-MM-DD format', 'total amount', 'Total amount due as a number', 'currency', 'Currency code USD, EUR, etc. ', 'payment terms', 'Payment terms e.g. Net 30 ', 'line items', 'Array of line items with description and amount' AS RAW EXTRACT, RAW EXTRACT:response:total amount::FLOAT / NULLIF RAW EXTRACT:response:total amount::FLOAT, 0 AS EXTRACTION CONFIDENCE, CURRENT TIMESTAMP AS EXTRACTED ATFROM DOC INTEL.PROCESSED.DT DOC FULL TEXT ftWHERE ft.DOC TYPE = 'invoice'; CREATE OR REPLACE DYNAMIC TABLE DOC INTEL.PROCESSED.DT EXTRACT CONTRACTS TARGET LAG = 'DOWNSTREAM' WAREHOUSE = DOC INTEL WHASSELECT ft.FILE ID, ft.FILE NAME, AI EXTRACT ft.FULL TEXT, OBJECT CONSTRUCT 'contract title', 'Title of the agreement', 'parties', 'Names of all parties as array ', 'effective date', 'Start date YYYY-MM-DD ', 'expiration date', 'End date YYYY-MM-DD ', 'governing law', 'Jurisdiction', 'total value', 'Total contract value with currency', 'auto renewal', 'Whether contract auto-renews true/false ' AS RAW EXTRACT, CURRENT TIMESTAMP AS EXTRACTED ATFROM DOC INTEL.PROCESSED.DT DOC FULL TEXT ftWHERE ft.DOC TYPE = 'contract'; The extracted data lands as VARIANT JSON in Snowflake, immediately queryable: -- "Which vendors invoiced us more than $50K this quarter?"SELECT RAW EXTRACT:response:vendor name::STRING AS vendor, SUM RAW EXTRACT:response:total amount::NUMBER AS totalFROM DOC INTEL.PROCESSED.DT EXTRACT INVOICESWHERE RAW EXTRACT:response:invoice date::DATE = DATEADD 'quarter', -1, CURRENT DATE GROUP BY vendorHAVING total 50000ORDER BY total DESC; Every document gets an executive summary and sentiment classification using dynamic tables that auto-refresh: CREATE OR REPLACE DYNAMIC TABLE DOC INTEL.ANALYTICS.DT DOC SUMMARIES TARGET LAG = 'DOWNSTREAM' REFRESH MODE = INCREMENTAL WAREHOUSE = DOC INTEL WHASSELECT ft.FILE ID, ft.RELATIVE PATH, ft.FILE NAME, ft.DOC TYPE, SNOWFLAKE.CORTEX.COMPLETE 'mistral-large2', 'You are a document analyst. Summarize this ' || ft.DOC TYPE || ' document in exactly 2-3 concise sentences. Focus on key facts, amounts, and parties involved.\n\nDocument:\n' || LEFT ft.FULL TEXT, 8000 AS EXECUTIVE SUMMARY, TRIM SNOWFLAKE.CORTEX.COMPLETE 'mistral-7b', 'Classify the overall sentiment of the following text. Respond with exactly ONE word: positive, negative, neutral, or mixed.\n\n' || LEFT ft.FULL TEXT, 4000 AS SENTIMENT, SNOWFLAKE.CORTEX.EMBED 'snowflake-arctic-embed-m-v2.0', LEFT ft.FULL TEXT, 512 AS SUMMARY VEC, ft.PARSED AT AS SUMMARIZED ATFROM DOC INTEL.PROCESSED.DT DOC FULL TEXT ft; The SUMMARY VEC embedding powers downstream theme mapping and semantic similarity queries. We discover document themes using embedding cosine similarity rather than keyword matching: CREATE OR REPLACE DYNAMIC TABLE DOC INTEL.ANALYTICS.DT DOC THEME MAP TARGET LAG = '1 hour' WAREHOUSE = DOC INTEL WHASSELECT ds.FILE ID, ds.FILE NAME, ds.DOC TYPE, ct.THEME ID, ct.THEME NAME, VECTOR COSINE SIMILARITY ds.SUMMARY VEC::VECTOR FLOAT, 768 , ct.THEME VEC::VECTOR FLOAT, 768 AS SIMILARITY SCOREFROM DOC INTEL.ANALYTICS.DT DOC SUMMARIES dsCROSS JOIN DOC INTEL.ANALYTICS.CORPUS THEMES ctWHERE ds.SUMMARY VEC IS NOT NULLQUALIFY ROW NUMBER OVER PARTITION BY ds.FILE ID ORDER BY SIMILARITY SCORE DESC <= 3; This enables automatic theme detection: “Which documents are about financial operations?” becomes a vector similarity query, not a fragile text search. Users search across their entire document corpus using natural language: CREATE OR REPLACE CORTEX SEARCH SERVICE DOC INTEL.ANALYTICS.DOC SEARCH SERVICE ON CHUNK TEXT ATTRIBUTES DOC TYPE, FILE ID, PAGE NUMBER, RELATIVE PATH WAREHOUSE = DOC INTEL WH TARGET LAG = '1 hour'ASSELECT ft.FILE ID, ft.DOC TYPE, c.VALUE::VARCHAR AS CHUNK TEXT, c.INDEX + 1 AS PAGE NUMBER, ft.RELATIVE PATHFROM DOC INTEL.PROCESSED.DT DOC FULL TEXT ft, LATERAL FLATTEN SPLIT ft.FULL TEXT, '\n\n' cWHERE LENGTH c.VALUE::VARCHAR 50; Querying from the Streamlit app: results = session.sql """ SELECT PARSE JSON SNOWFLAKE.CORTEX.SEARCH PREVIEW 'DOC INTEL.ANALYTICS.DOC SEARCH SERVICE', ?, 'CHUNK TEXT', 'DOC TYPE', 'FILE ID', 'PAGE NUMBER', 'RELATIVE PATH' , 10 :results AS RESULTS""", params= user query .collect This returns semantically relevant document chunks — not just keyword matches — with faceted filtering by document type. We built a six-page Streamlit in Snowflake application as the user-facing layer: stats = session.sql """ SELECT SELECT COUNT FROM DOC INTEL.PROCESSED.DT DOC CLASSIFIED AS TOTAL DOCS, SELECT ROUND AVG EXTRACTION CONFIDENCE , 1 FROM DOC INTEL.PROCESSED.DT EXTRACT INVOICES AS AVG CONF, SELECT COUNT FROM DOC INTEL.ANALYTICS.CORPUS THEMES AS THEME COUNT, SELECT COUNT FROM DOC INTEL.PROCESSED.REVIEW QUEUE WHERE REVIEW STATUS = 'PENDING' AS PENDING REVIEWS""" .collect 0 with st.container horizontal=True : st.metric "Total documents", f"{stats 'TOTAL DOCS' :,}", border=True st.metric "Avg confidence", f"{stats 'AVG CONF' }%", border=True st.metric "Themes discovered", stats 'THEME COUNT' , border=True st.metric "Pending reviews", stats 'PENDING REVIEWS' , border=True Users drag-and-drop documents directly into Snowflake’s internal stage: uploaded files = st.file uploader "Drag and drop files here", type= "pdf", "png", "jpg", "jpeg", "tiff", "docx", "xlsx", "pptx" , accept multiple files=True, if uploaded files: progress = st.progress 0, text="Uploading..." for i, f in enumerate uploaded files : session.file.put stream f, f"@DOC INTEL.RAW.DOCUMENTS STAGE/{f.name}", auto compress=False progress.progress i + 1 / len uploaded files , text=f"Uploaded {f.name}" st.toast f"{len uploaded files } file s uploaded successfully." The Compare feature sends documents to Groq or Cortex AI for side-by-side analysis: Via Groq UDF External Access Integration live = session.sql """ SELECT ft.FILE NAME, DOC INTEL.APP.GROQ SUMMARIZE ft.FULL TEXT, ft.DOC TYPE :executive summary::VARCHAR AS EXECUTIVE SUMMARY, DOC INTEL.APP.GROQ SUMMARIZE ft.FULL TEXT, ft.DOC TYPE :sentiment::VARCHAR AS SENTIMENT FROM DOC INTEL.PROCESSED.DT DOC FULL TEXT ft WHERE ft.FILE NAME IN ?, ? """, params= doc a, doc b .to pandas The Groq UDF is a Python UDF with External Access Integration: CREATE OR REPLACE FUNCTION DOC INTEL.APP.GROQ SUMMARIZE doc text VARCHAR, doc type VARCHAR RETURNS VARIANTLANGUAGE PYTHONRUNTIME VERSION = '3.11'PACKAGES = 'requests' HANDLER = 'summarize'EXTERNAL ACCESS INTEGRATIONS = GROQ ACCESS INTEGRATION SECRETS = 'groq key' = DOC INTEL.APP.GROQ API KEY AS $$import json, snowflake, requestsdef summarize doc text, doc type : api key = snowflake.get generic secret string 'groq key' resp = requests.post "https://api.groq.com/openai/v1/chat/completions", headers={"Authorization": f"Bearer {api key}"}, json={ "model": "llama-3.3-70b-versatile", "messages": {"role": "user", "content": f"Summarize this {doc type}:\n{doc text :8000 }"} , "temperature": 0.1, "response format": {"type": "json object"} }, timeout=30 return json.loads resp.json "choices" 0 "message" "content" $$; Natural language queries over the entire document corpus using Cortex Agents: if prompt := st.chat input "Ask about your documents..." : request body = json.dumps {"messages": {"role": "user", "content": prompt} } result = session.sql """ SELECT SNOWFLAKE.CORTEX.DATA AGENT RUN 'DOC INTEL.APP.DOC INTEL AGENT', PARSE JSON ? AS RESPONSE """, params= request body .collect The monitoring dashboard tracks pipeline health, extraction quality, throughput, and credit consumption: -- Health check: detect stale pipeline stagesCREATE OR REPLACE VIEW DOC INTEL.MONITORING.V PIPELINE HEALTH ASSELECT TABLE NAME, TIMESTAMPDIFF 'MINUTE', DATA TIMESTAMP, CURRENT TIMESTAMP AS MINUTES SINCE LAST RECORDFROM TABLE INFORMATION SCHEMA.DYNAMIC TABLE REFRESH HISTORY WHERE SCHEMA NAME IN 'PROCESSED', 'ANALYTICS' QUALIFY ROW NUMBER OVER PARTITION BY TABLE NAME ORDER BY DATA TIMESTAMP DESC = 1; For development, testing, or accounts without Cortex AI access, we built a hybrid pipeline that calls Groq’s free-tier API externally and writes results back to Snowflake: python def summarize document doc type, file name, text : prompt = f"""Analyze this {doc type} document and summarize it.Document: {file name}Text: {text :8000 }Respond ONLY with JSON:{{ "executive summary": "2-3 sentence summary", "key findings": "finding1", "finding2" , "sentiment": "positive|negative|neutral|mixed"}}""" return call groq prompt Write back to Snowflakedef write summary conn, file id, file name, path, doc type, summary : cur = conn.cursor cur.execute """ MERGE INTO DOC INTEL.ANALYTICS.DT DOC SUMMARIES tgt USING SELECT %s AS FILE ID src ON tgt.FILE ID = src.FILE ID WHEN MATCHED THEN UPDATE SET EXECUTIVE SUMMARY = %s, SENTIMENT = %s, SUMMARIZED AT = CURRENT TIMESTAMP WHEN NOT MATCHED THEN INSERT FILE ID, RELATIVE PATH, FILE NAME, DOC TYPE, EXECUTIVE SUMMARY, SENTIMENT, SUMMARIZED AT VALUES %s, %s, %s, %s, %s, %s, CURRENT TIMESTAMP """, file id, summary "executive summary" , summary "sentiment" , file id, path, file name, doc type, summary "executive summary" , summary "sentiment" conn.commit This means you can prototype immediately with Groq’s free tier Llama 3.3 70B , then switch to Cortex AI in production — same schema, same app, zero code changes in the Streamlit layer. Proving a document use case works on 10 files is straightforward. The real enterprise challenge is processing hundreds of thousands of documents daily without your pipeline collapsing. Dynamic tables solve this by letting you declare the transformation as SQL — Snowflake handles scheduling, refresh orchestration, dependency ordering, and automatic catch-up after failures. No task graph to debug, no DAG orchestrator to maintain. Setting REFRESH MODE = INCREMENTAL guarantees each incoming document is processed exactly once, enabling predictable cost at scale. Every user-facing SQL uses bind parameters params= ... instead of f-string interpolation. This prevents SQL injection — critical when file names come from user uploads. Not every extraction is perfect. Low-confidence results below 70% are automatically flagged for human review rather than silently propagating bad data downstream. Keyword-based classification breaks when document language evolves. Vector similarity finds conceptual matches — a “liability clause” and a “risk mitigation section” are recognized as related even with zero keyword overlap. After deploying this platform: Youtube Demo: Please find the complete source code on Github https://github.com/satishkumarai/snowflake-doc-intelligence Built on Snowflake. Powered by Groq/Cortex AI. Governed by default. 👏 Give it a clap if it added value 🔗 Share it with your team ➕ Follow for more 📘 Medium: 🔗 LinkedIn: satishkumar-snowflake https://www.linkedin.com/in/satishkumar-snowflake/ See you in the next one 👋 Building a Document Intelligence Platform on Snowflake with Cortex AI https://pub.towardsai.net/building-a-document-intelligence-platform-on-snowflake-with-cortex-ai-a3ab59d8239a was originally published in Towards AI https://pub.towardsai.net on Medium, where people are continuing the conversation by highlighting and responding to this story.