Hey DEV! 👋
Our open-source document memory project, Knowhere, just crossed 4,000 stars on GitHub! 🎉
When we started building Knowhere to wrangle messy PDFs, complex tables, and engineering blueprints for AI agents, we weren't sure how many other devs were fighting the exact same battles. Hitting this milestone means the world to our team—thank you to everyone who has starred the repo, opened issues, or submitted PRs!
To celebrate, I want to share a look under the hood at our latest architectural update—Progressive Ingestion—and show you what happened when we stress-tested the pipeline on 11,945 ML conference papers.
Until recently, document ingestion worked like boiling a pot of water: until it hits 100°C, you just sit there and wait.
When you pass a PDF into a standard RAG or parsing pipeline, a ton of heavy lifting happens under the hood:
Historically, a document isn't queryable until every single one of those steps finishes.
For a 5-page PDF? Waiting a few seconds is fine.
For a 500-page technical manual, a financial report, or a dense bundle of academic papers? The UX completely falls apart.
Even if the first 200 pages are already parsed, the entire file stays locked in a "processing" state as long as a few complex pages are still sitting in the queue. If a user—or an AI agent—just wants to look up a single part number on Page 12, they are stuck waiting for Page 480 to finish.
For AI agents, this all-or-nothing blocking is a dealbreaker. Agents work iteratively—they search, read, and gather context on the fly. Forcing an agent to wait until a document is 100% pre-processed kills real-time responsiveness before the workflow even starts. To kill this latency bottleneck, we built Progressive Ingestion into Knowhere.
When you set ingestion_mode=progressive, Knowhere stops holding your documents hostage until the entire pipeline finishes. Instead, it streams context to your agent in three distinct stages:
Stage 1: Instant Baseline (Readable & Searchable in Seconds)
Knowhere immediately publishes a baseline version of the document so your agent can open, read, and run searches within seconds of upload.
Stage 2: Structural Hierarchy (TOC & Section Trees) Next, Knowhere reconstructs outlines and section hierarchies, turning a flat pile of pages into a structured document tree that agents can navigate via a Table of Contents.
Stage 3: On-Demand Background Enrichment
Deep enrichment and heavy parsing continue asynchronously in the background. Crucially, fine-grained nodes are prioritized on-demand whenever an agent actively queries those specific sections.
Think of it like dining at a restaurant: instead of starving at the table until every dish for the entire party is cooked, appetizers hit the table immediately while the main courses arrive as they’re ready.
We aren't cutting corners or dropping parsing steps—we're just slashing time-to-first-use while finishing the heavy lifting in the background.
💡 When should you use which mode?
Progressive mode is built for latency-sensitive apps (live chat-with-PDF, interactive agents). Full Ingestion remains the default for batch workloads where you only want documents exposed after 100% of processing is done.
Back in Knowhere 2.0, we introduced two core pillars for complex documents:
VISION-MAP (Visual Parsing): Visually dense files (scanned PDFs, slide decks, engineering blueprints) are routed directly to vision models as full pages to preserve layout fidelity. Clean files (Word, Excel) go through native text parsing. Both converge into one unified document memory.
Agent-Native Retrieval: Instead of rigid, one-shot chunk retrieval, the agent inspects the document map, calls navigation tools, and plans its own multi-step retrieval route.
Progressive Ingestion completes this puzzle:
You can't benchmark a document engine by running demos on three clean PDFs. We wanted a stress test with massive scale, gnarly layouts, and real utility for the AI community.
Academic papers checked every box: double-column text scrambles reading order, equations and tables float across pages, and a casual "see Appendix C" might point to a table 15 pages away.
So we ran all 11,945 accepted papers from ACL, CVPR, ICML, and NeurIPS 2025 through Knowhere.
Instead of dumping flat text into an LLM for a generic summary, our pipeline worked structurally:
Parsed each paper into chunks tagged with exact section paths.
Reconstructed the full section hierarchy tree (Methodology, Experiments, Appendix, etc.).
Extracted 28 structured fields per paper (research problem, core innovations, baselines, experimental results, etc.) while keeping the structural context intact.
Packaged the structured fields, grounded source snippets, and section trees into a single verifiable dataset.
Why does this matter? Because claims stay verifiable. If a record says "Method X beats the baseline by 5%," a Research Agent can trace the section path straight back to the exact paragraph to verify the dataset, experimental setup, and ablation tables.
(Disclaimer: Since this dataset was generated via an automated pipeline without manual paper-by-paper proofreading, definitely double-check the original PDF before citing numbers in formal papers!)
Grab the Open-Source Datasets on Hugging Face 🤗
We’ve open-sourced the full 2025 dataset (plus 2023, 2024, and Embodied AI collections) on Hugging Face:
2025 Top ML Papers (11,945 papers): huggingface.co/datasets/JensCS/top-ml-conference-papers-2025
2024 Edition: huggingface.co/datasets/JensCS/top-ml-conference-papers-2024
2023 Edition: huggingface.co/datasets/JensCS/top-ml-conference-papers-2023
Embodied AI Papers (2023–2025): huggingface.co/datasets/JensCS/embodied-ai-conference-papers-2023-2025
We're on a mission to get messy, real-world documents into agent workflows without the friction. If you're building RAG pipelines, enterprise knowledge bases, or Research Agents, give Knowhere a spin:
🔗 [Website](https://knowhereto.ai) |
📄 [Docs](https://docs.knowhereto.ai/) |
🏠 [Self-Host](https://github.com/Ontos-AI/knowhere-self-hosted) |
🖥️ [Dashboard](https://github.com/Ontos-AI/knowhere-dashboard)
Knowhere is a document intelligence and retrieval infrastructure that turns complex, dirty files into persistent, navigable memory for AI agents.
It ingests unstructured documents and produces persistent, navigable memory: parsing, hierarchy reconstruction, multi-modal structuring, and graph construction in a single pipeline. Every result stays connected to its document, section, source pages, and related assets, making the output a natural fit for Agentic RAG, vector-based RAG, or any LLM workflow.
Knowhere parses files along their native hierarchy (headings, if any) and uses frontier vision models to cut the extraction errors of traditional parsers. Documents can be ingested progressively: readable and searchable first, enriched in the background. This progressive feature enables fast response in agent systems.
Note
Get started in seconds with Knowhere Cloud. Avoid the complexity of self-deployment. Use our managed API at knowhereto.ai and enjoy $5…
Website: knowhereto.ai I'd love to hear from you in the comments: What’s the gnarliest document format or latency bottleneck you've run into while building RAG or agent pipelines? Let's geek out below! 👇