I've been building data2prompt for a few months now. It takes a data-heavy project (CSVs, SQL dumps, notebooks, Excel files) and turns it into a single file an LLM can read, instead of the model choking on raw 200MB CSVs or a generic repo-packager just skipping them. This week I shipped v0.5.0, and the real work in that release wasn't a new parser or a flag. It was a document I wrote called docs/output-contract.md
, and writing it forced me to think about a failure mode I'd been quietly ignoring for months.
Here's the problem. To fit a large table into a context window, data2prompt samples it, say, 15 random rows out of 1.2 million. That's necessary and fine. But once you hand that sample to an LLM without being extremely explicit about what it's looking at, the model will happily treat 15 rows as the dataset. It'll compute an "average" from them. It'll describe a "trend." It'll answer "how many customers churned" using a number that only exists because of your random seed. The sample doesn't just lose information, it actively invites the model to hallucinate as if it had everything.
That's the actual design problem behind data2prompt: the output isn't documentation for a person, it's an input for a model, and models fail differently than people do. A person skimming a CSV preview instinctively knows it's a preview. A model doesn't have that instinct unless you build it in.
The rule I landed on: any time data gets reduced, the full count has to be captured before the reduction happens, and it has to travel with the sample as a notice.
total_rows = len(df) # capture first
sample = df.sample(n=config.csv_sample_size, random_state=config.seed).sort_index()
Every sampled table, every truncated SQL insert block, every notebook cell whose output got cut off, all of it carries a line like:
-- [Sample: random 15 of 1,234,567 rows] --
-- [CSV truncated: Showing random 15 of 1,234,567 rows to save context] --
-- [Table data truncated: Showing random 15 of 200 buffered rows to save context] --
Same grammar every time: -- [Category: detail] --
. Not a *Note:*
in italics, no emoji, no "heads up!" One shape, always, on purpose. The system prompt at the top of the generated file teaches the model this exact pattern once, and after that the model can reliably tell "this line is the tool talking to me" apart from "this line is file content." Mix in even one differently-formatted note and that separation gets fuzzy.
This is the rule that actually matters, and it applies past just numeric sampling. A .parquet
file with no pyarrow
installed doesn't vanish. It shows up with Skipped (No pyarrow)
and an install command. An SQL table's schema-only mode still prints CREATE TABLE
, just with the rows replaced by -- [N data row(s) omitted: schema-only] --
. A file excluded by .gitignore
still gets a row in the File Index with status Omitted
. If a file was scanned, it's accounted for somewhere in the document, even if the answer is "not shown, here's why."
The alternative is silently dropping things, which is how you get an LLM confidently telling you a project has no config file when it does. It just didn't make the cut.
Writing the contract wasn't just theory, it surfaced real bugs while I was auditing the codebase against it:
The Excel visual-element check (are there charts or images in this sheet the tool can't extract) was reading sheet._images
off worksheets opened in read_only=True
mode. Read-only worksheets in openpyxl never parse drawing parts, and the attribute on regular worksheets is _charts
, not charts
, anyway. So that check could never fire. It always said "no visuals," even when a sheet was covered in charts. Fixed it by treating an .xlsx
as what it actually is, a zip file, and checking for xl/media/
and xl/charts/
in the archive listing directly. No workbook load needed, and now it's actually correct instead of silently wrong.
The other one is worse in a different way: bare .env
files have an empty file suffix, so they were falling through the extension-based skip list and getting parsed by the generic text handler, which just dumps file contents. A tool built specifically to keep secrets out of your LLM context was leaking them because the routing was extension-based and .env
doesn't have an extension. Now .env
files (and .env.local
, prod.env
, etc.) are matched by name before extension dispatch even runs, and the parser only ever emits KEY=<redacted>
.
Neither of these were found by "testing the feature." They were found by writing down what the system was supposed to guarantee and then checking the code against that sentence.
I'm a data scientist, not a software engineer by trade, and I build data2prompt by directing coding agents through the implementation rather than hand-writing every line myself. That workflow has a specific failure mode: without something written down, every session re-derives its own idea of "how should this notice be worded" or "does this new parser need to update the preamble too," and consistency erodes one plausible-looking PR at a time. The contract doc exists so that decision doesn't have to be re-litigated: format parity across markdown/xml, one notice grammar, one canonical path key, controlled vocabularies only, fixed anchors at the top and bottom of the document. Seven invariants, plus a checklist for each kind of change. Any session, human or agent, that touches the output code reads that first.
If you're building anything that generates content for an LLM to consume rather than a human, not just data2prompt, any RAG pipeline, any agent tool output, I'd push back on treating "human-readable" and "model-readable" as the same design target. They're not. A human notices when something looks off. A model narrates whatever's in front of it as if it's the truth, unless you've engineered the document to stop it from doing that.
Repo's here if you want to see the actual contract doc or poke at the code: github.com/arianmokhtariha/data2prompt. It's MIT licensed, on PyPI as data2prompt
. Happy to talk through any of the tradeoffs above in the comments.