{"slug": "show-hn-markdown-is-40x-more-token-efficient-with-sqlite", "title": "Show HN: Markdown is 40× more token-efficient with SQLite", "summary": "Native founder Richard reported that AI agents used 40× fewer tokens per correct answer when querying a SQLite-backed Markdown vault than a plain Markdown vault, based on a 120-trial evaluation over 11,000 records. Richard said the benchmark dataset remains private because it draws on two months of his own work, though he is considering open-sourcing it, and that the core engine evaluated is open-source on GitHub under withnative/native. He also noted that over two-thirds of the token bill on long agent sessions came from cached input tokens.", "body_md": "Hi HN, I'm Richard, founder of Native. I'm working on open-source context ownership, to stop AI centralisation: [https://github.com/withnative/native](https://github.com/withnative/native)\n\nMy first focus has been making Markdown vaults powerful enough so that teams can self-host all of their context (messaging, docs and projects), without it being externally fragmented or rate-limited.\n\nI'm sharing some early results: Markdown vaults can be 40× more efficient with SQLite!\n\n[https://www.withnative.ai/research-and-posts/markdown-contex...](https://www.withnative.ai/research-and-posts/markdown-context-benchmark)\n\nHere's some of the main ideas:\n\n- SQLite or Markdown folders, you pick. You can project the same authoritative event log into either form; more projection formats will follow (e.g. maybe neo4j; we do use a graph structure).\n\n- Empirically: SQLite is better for agents! On a 120 trial evaluation for 11,000 records, agents use 40× fewer tokens with SQLite vs Markdown, per correct answer.\n\n- Everything is a file or SQLite row. Messages, documents and projects: all can be modelled in your vault. You get local-first collaboration with teammates, synced to a cloud of your choice.\n\n- You can render your vault with arbitrary UI. Your WhatsApp-like UI speaks to your teammate's Slack-like UI; both are just views over the same database.\n\nA few things I've learned building this:\n\n- Agents do pretty well even without SQL if you give them strongly typed Markdown frontmatter. This was one arm of the trial I ran. I didn't have any clear expectations, but it was cool to watch: agents ended up writing Python scripts to do SQL-like queries.\n\n- Cumulatively, cached input tokens can get expensive. I feel silly in hindsight here. I'd gotten used to letting my sessions run long, \"It's all cached so it's very cheap\". But when I looked at my results here, over 2/3 of the token bill on long sessions was cached input...\n\n- \"Markdown vaults don't scale as well as databases\" isn't as self-evident as I think. This is why I decided to benchmark it... I've had some very blank stares when I try to explain how Markdown files might break at true agent-scale volume.\n\nUnfortunately, the dataset is private, because it's two months of my own work. I am considering open-sourcing it even so, but there is definitely some embarrassing stuff in there. TBD... I know that independent result verification matters!\n\nThe core engine that I evaluated is open-source, with more to be released soon: [https://github.com/withnative/native](https://github.com/withnative/native)\n\nIf you're interested in trying connected context on an open engine, you can get updates from our website or email me directly: richard@withnative.ai\n\nComments URL: [https://news.ycombinator.com/item?id=50258875](https://news.ycombinator.com/item?id=50258875)\n\nPoints: 2\n\n# Comments: 0", "url": "https://wpnews.pro/news/show-hn-markdown-is-40x-more-token-efficient-with-sqlite", "canonical_source": "https://www.withnative.ai/research-and-posts/markdown-context-benchmark", "published_at": "2026-10-27 13:23:49+00:00", "updated_at": "2026-10-27 13:41:56.255211+00:00", "lang": "en", "topics": ["ai-agents", "ai-tools", "ai-infrastructure", "developer-tools", "artificial-intelligence"], "entities": ["Native", "Richard", "SQLite", "Markdown", "withnative/native", "GitHub"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/show-hn-markdown-is-40x-more-token-efficient-with-sqlite", "markdown": "https://wpnews.pro/news/show-hn-markdown-is-40x-more-token-efficient-with-sqlite.md", "text": "https://wpnews.pro/news/show-hn-markdown-is-40x-more-token-efficient-with-sqlite.txt", "jsonld": "https://wpnews.pro/news/show-hn-markdown-is-40x-more-token-efficient-with-sqlite.jsonld"}}