A CSV sample that looks like the whole dataset is worse than no CSV at all

A developer building data2prompt, a tool that converts data-heavy projects into LLM-readable files, identified a critical failure mode: when sampling large datasets, LLMs treat the sample as the full dataset, leading to hallucinations. The fix requires capturing total row counts before reduction and embedding explicit notices in the output, ensuring models recognize truncated data. The v0.5.0 release also fixed a bug in Excel visual-element detection by directly inspecting the zip archive instead of relying on openpyxl's read-only mode.

I've been building data2prompt https://github.com/arianmokhtariha/data2prompt for a few months now. It takes a data-heavy project CSVs, SQL dumps, notebooks, Excel files and turns it into a single file an LLM can read, instead of the model choking on raw 200MB CSVs or a generic repo-packager just skipping them. This week I shipped v0.5.0, and the real work in that release wasn't a new parser or a flag. It was a document I wrote called docs/output-contract.md , and writing it forced me to think about a failure mode I'd been quietly ignoring for months. Here's the problem. To fit a large table into a context window, data2prompt samples it, say, 15 random rows out of 1.2 million. That's necessary and fine. But once you hand that sample to an LLM without being extremely explicit about what it's looking at, the model will happily treat 15 rows as the dataset. It'll compute an "average" from them. It'll describe a "trend." It'll answer "how many customers churned" using a number that only exists because of your random seed. The sample doesn't just lose information, it actively invites the model to hallucinate as if it had everything. That's the actual design problem behind data2prompt: the output isn't documentation for a person, it's an input for a model, and models fail differently than people do. A person skimming a CSV preview instinctively knows it's a preview. A model doesn't have that instinct unless you build it in. The rule I landed on: any time data gets reduced, the full count has to be captured before the reduction happens, and it has to travel with the sample as a notice. total rows = len df capture first sample = df.sample n=config.csv sample size, random state=config.seed .sort index -- Sample: random 15 of 1,234,567 rows -- Every sampled table, every truncated SQL insert block, every notebook cell whose output got cut off, all of it carries a line like: -- Sample: random 15 of 1,234,567 rows -- -- CSV truncated: Showing random 15 of 1,234,567 rows to save context -- -- Table data truncated: Showing random 15 of 200 buffered rows to save context -- Same grammar every time: -- Category: detail -- . Not a Note: in italics, no emoji, no "heads up " One shape, always, on purpose. The system prompt at the top of the generated file teaches the model this exact pattern once, and after that the model can reliably tell "this line is the tool talking to me" apart from "this line is file content." Mix in even one differently-formatted note and that separation gets fuzzy. This is the rule that actually matters, and it applies past just numeric sampling. A .parquet file with no pyarrow installed doesn't vanish. It shows up with Skipped No pyarrow and an install command. An SQL table's schema-only mode still prints CREATE TABLE , just with the rows replaced by -- N data row s omitted: schema-only -- . A file excluded by .gitignore still gets a row in the File Index with status Omitted . If a file was scanned, it's accounted for somewhere in the document, even if the answer is "not shown, here's why." The alternative is silently dropping things, which is how you get an LLM confidently telling you a project has no config file when it does. It just didn't make the cut. Writing the contract wasn't just theory, it surfaced real bugs while I was auditing the codebase against it: The Excel visual-element check are there charts or images in this sheet the tool can't extract was reading sheet. images off worksheets opened in read only=True mode. Read-only worksheets in openpyxl never parse drawing parts, and the attribute on regular worksheets is charts , not charts , anyway. So that check could never fire. It always said "no visuals," even when a sheet was covered in charts. Fixed it by treating an .xlsx as what it actually is, a zip file, and checking for xl/media/ and xl/charts/ in the archive listing directly. No workbook load needed, and now it's actually correct instead of silently wrong. The other one is worse in a different way: bare .env files have an empty file suffix, so they were falling through the extension-based skip list and getting parsed by the generic text handler, which just dumps file contents. A tool built specifically to keep secrets out of your LLM context was leaking them because the routing was extension-based and .env doesn't have an extension. Now .env files and .env.local , prod.env , etc. are matched by name before extension dispatch even runs, and the parser only ever emits KEY=<redacted . Neither of these were found by "testing the feature." They were found by writing down what the system was supposed to guarantee and then checking the code against that sentence. I'm a data scientist, not a software engineer by trade, and I build data2prompt by directing coding agents through the implementation rather than hand-writing every line myself. That workflow has a specific failure mode: without something written down, every session re-derives its own idea of "how should this notice be worded" or "does this new parser need to update the preamble too," and consistency erodes one plausible-looking PR at a time. The contract doc exists so that decision doesn't have to be re-litigated: format parity across markdown/xml, one notice grammar, one canonical path key, controlled vocabularies only, fixed anchors at the top and bottom of the document. Seven invariants, plus a checklist for each kind of change. Any session, human or agent, that touches the output code reads that first. If you're building anything that generates content for an LLM to consume rather than a human, not just data2prompt, any RAG pipeline, any agent tool output, I'd push back on treating "human-readable" and "model-readable" as the same design target. They're not. A human notices when something looks off. A model narrates whatever's in front of it as if it's the truth, unless you've engineered the document to stop it from doing that. Repo's here if you want to see the actual contract doc or poke at the code: github.com/arianmokhtariha/data2prompt https://github.com/arianmokhtariha/data2prompt . It's MIT licensed, on PyPI as data2prompt . Happy to talk through any of the tradeoffs above in the comments.