{"slug": "knowhere-hit-4000-github-stars-here-is-how-we-built-progressive-pdf-ingestion", "title": "Knowhere hit 4,000 GitHub stars! Here is how we built progressive PDF ingestion", "summary": "The open-source document memory project Knowhere, which crossed 4,000 GitHub stars, added a Progressive Ingestion mode that publishes a searchable baseline of a PDF within seconds and streams structural hierarchy and background enrichment to AI agents asynchronously, rather than blocking until the entire file is parsed. The team stress-tested the pipeline on 11,945 ML conference papers, positioning progressive mode for latency-sensitive chat-with-PDF and interactive agent workloads while full ingestion remains the default for batch jobs.", "body_md": "Hey DEV! 👋\n\nOur open-source document memory project, Knowhere, just crossed 4,000 stars on GitHub! 🎉\n\nWhen we started building Knowhere to wrangle messy PDFs, complex tables, and engineering blueprints for AI agents, we weren't sure how many other devs were fighting the exact same battles. Hitting this milestone means the world to our team—thank you to everyone who has starred the repo, opened issues, or submitted PRs!\n\nTo celebrate, I want to share a look under the hood at our latest architectural update—Progressive Ingestion—and show you what happened when we stress-tested the pipeline on 11,945 ML conference papers.\n\nUntil recently, document ingestion worked like boiling a pot of water: until it hits 100°C, you just sit there and wait.\n\nWhen you pass a PDF into a standard RAG or parsing pipeline, a ton of heavy lifting happens under the hood:\n\nHistorically, a document isn't queryable until every single one of those steps finishes.\n\nFor a 5-page PDF? Waiting a few seconds is fine.\n\nFor a 500-page technical manual, a financial report, or a dense bundle of academic papers? The UX completely falls apart.\n\nEven if the first 200 pages are already parsed, the entire file stays locked in a \"processing\" state as long as a few complex pages are still sitting in the queue. If a user—or an AI agent—just wants to look up a single part number on Page 12, they are stuck waiting for Page 480 to finish.\n\nFor AI agents, this all-or-nothing blocking is a dealbreaker. Agents work iteratively—they search, read, and gather context on the fly. Forcing an agent to wait until a document is 100% pre-processed kills real-time responsiveness before the workflow even starts.\n\nTo kill this latency bottleneck, we built Progressive Ingestion into Knowhere.\n\nWhen you set ingestion_mode=progressive, Knowhere stops holding your documents hostage until the entire pipeline finishes. Instead, it streams context to your agent in three distinct stages:\n\n**Stage 1: Instant Baseline (Readable & Searchable in Seconds)**\n\nKnowhere immediately publishes a baseline version of the document so your agent can open, read, and run searches within seconds of upload.\n\n**Stage 2: Structural Hierarchy (TOC & Section Trees)**\n\nNext, Knowhere reconstructs outlines and section hierarchies, turning a flat pile of pages into a structured document tree that agents can navigate via a Table of Contents.\n\n**Stage 3: On-Demand Background Enrichment**\n\nDeep enrichment and heavy parsing continue asynchronously in the background. Crucially, fine-grained nodes are prioritized on-demand whenever an agent actively queries those specific sections.\n\nThink of it like dining at a restaurant: instead of starving at the table until every dish for the entire party is cooked, appetizers hit the table immediately while the main courses arrive as they’re ready.\n\nWe aren't cutting corners or dropping parsing steps—we're just slashing time-to-first-use while finishing the heavy lifting in the background.\n\n💡 When should you use which mode?\n\nProgressive mode is built for latency-sensitive apps (live chat-with-PDF, interactive agents).\n\nFull Ingestion remains the default for batch workloads where you only want documents exposed after 100% of processing is done.\n\nBack in Knowhere 2.0, we introduced two core pillars for complex documents:\n\nVISION-MAP (Visual Parsing): Visually dense files (scanned PDFs, slide decks, engineering blueprints) are routed directly to vision models as full pages to preserve layout fidelity. Clean files (Word, Excel) go through native text parsing. Both converge into one unified document memory.\n\nAgent-Native Retrieval: Instead of rigid, one-shot chunk retrieval, the agent inspects the document map, calls navigation tools, and plans its own multi-step retrieval route.\n\nProgressive Ingestion completes this puzzle:\n\nYou can't benchmark a document engine by running demos on three clean PDFs. We wanted a stress test with massive scale, gnarly layouts, and real utility for the AI community.\n\nAcademic papers checked every box: double-column text scrambles reading order, equations and tables float across pages, and a casual \"see Appendix C\" might point to a table 15 pages away.\n\nSo we ran all 11,945 accepted papers from ACL, CVPR, ICML, and NeurIPS 2025 through Knowhere.\n\nInstead of dumping flat text into an LLM for a generic summary, our pipeline worked structurally:\n\nParsed each paper into chunks tagged with exact section paths.\n\nReconstructed the full section hierarchy tree (Methodology, Experiments, Appendix, etc.).\n\nExtracted 28 structured fields per paper (research problem, core innovations, baselines, experimental results, etc.) while keeping the structural context intact.\n\nPackaged the structured fields, grounded source snippets, and section trees into a single verifiable dataset.\n\nWhy does this matter? Because claims stay verifiable. If a record says \"Method X beats the baseline by 5%,\" a Research Agent can trace the section path straight back to the exact paragraph to verify the dataset, experimental setup, and ablation tables.\n\n(Disclaimer: Since this dataset was generated via an automated pipeline without manual paper-by-paper proofreading, definitely double-check the original PDF before citing numbers in formal papers!)\n\n**Grab the Open-Source Datasets on Hugging Face 🤗**\n\nWe’ve open-sourced the full 2025 dataset (plus 2023, 2024, and Embodied AI collections) on Hugging Face:\n\n2025 Top ML Papers (11,945 papers): huggingface.co/datasets/JensCS/top-ml-conference-papers-2025\n\n2024 Edition: huggingface.co/datasets/JensCS/top-ml-conference-papers-2024\n\n2023 Edition: huggingface.co/datasets/JensCS/top-ml-conference-papers-2023\n\nEmbodied AI Papers (2023–2025): huggingface.co/datasets/JensCS/embodied-ai-conference-papers-2023-2025\n\nWe're on a mission to get messy, real-world documents into agent workflows without the friction. If you're building RAG pipelines, enterprise knowledge bases, or Research Agents, give Knowhere a spin:\n\n  🔗 [Website](https://knowhereto.ai) |\n  📄 [Docs](https://docs.knowhereto.ai/) |\n  🏠 [Self-Host](https://github.com/Ontos-AI/knowhere-self-hosted) |\n  🖥️ [Dashboard](https://github.com/Ontos-AI/knowhere-dashboard)\n\n**Knowhere is a document intelligence and retrieval infrastructure that turns complex, dirty files into persistent, navigable memory for AI agents.**\n\nIt ingests unstructured documents and produces persistent, navigable memory: parsing, hierarchy reconstruction, multi-modal structuring, and graph construction in a single pipeline. Every result stays connected to its document, section, source pages, and related assets, making the output a natural fit for *Agentic RAG*, *vector-based RAG*, or any LLM workflow.\n\nKnowhere parses files along their native hierarchy (headings, if any) and uses frontier vision models to cut the extraction errors of traditional parsers. Documents can be ingested progressively: readable and searchable first, enriched in the background. This progressive feature enables fast response in agent systems.\n\nNote\n\n**Get started in seconds with Knowhere Cloud.**\nAvoid the complexity of self-deployment. Use our managed API at [knowhereto.ai](https://knowhereto.ai) and enjoy **$5**…\n\nWebsite: [knowhereto.ai](https://knowhereto.ai/?utm_source=dev)\n\nI'd love to hear from you in the comments: What’s the gnarliest document format or latency bottleneck you've run into while building RAG or agent pipelines? Let's geek out below! 👇", "url": "https://wpnews.pro/news/knowhere-hit-4000-github-stars-here-is-how-we-built-progressive-pdf-ingestion", "canonical_source": "https://dev.to/sylvie_reed/knowhere-hit-4000-github-stars-here-is-how-we-built-progressive-pdf-ingestion-2p20", "published_at": "2026-11-14 18:46:47+00:00", "updated_at": "2026-11-14 18:50:48.546032+00:00", "lang": "en", "topics": ["ai-agents", "ai-tools", "artificial-intelligence", "developer-tools"], "entities": ["Knowhere", "GitHub"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/knowhere-hit-4000-github-stars-here-is-how-we-built-progressive-pdf-ingestion", "markdown": "https://wpnews.pro/news/knowhere-hit-4000-github-stars-here-is-how-we-built-progressive-pdf-ingestion.md", "text": "https://wpnews.pro/news/knowhere-hit-4000-github-stars-here-is-how-we-built-progressive-pdf-ingestion.txt", "jsonld": "https://wpnews.pro/news/knowhere-hit-4000-github-stars-here-is-how-we-built-progressive-pdf-ingestion.jsonld"}}