{"slug": "show-hn-cli-tool-that-packages-data-heavy-projects-into-llm-context-windows", "title": "Show HN: CLI tool that packages data-heavy projects into LLM context windows", "summary": "A new open-source CLI tool called data2prompt packages data-heavy projects into LLM context windows by profiling tables and sampling rows instead of dumping raw files. Developer Arian Mokhtariha released data2prompt on GitHub, claiming it reduces a 22 MB generic packer output to 241 KB while preserving schema, statistics, and representative samples. The tool supports CSV, Parquet, Excel, SQLite, SQL, Jupyter notebooks, and .env files, and can fit output to a specified token budget.", "body_md": "**Turn data-heavy projects into LLM context that actually fits — and actually informs.**\n\nOne command turns a project full of CSVs, notebooks, Excel workbooks and SQLite\ndatabases into a single, structured, LLM-ready document — sampled, profiled,\nredacted, and fitted to your context window.\n\nGeneric repo-to-prompt tools break down on data projects — and not by a margin\nyou can optimize away. Point them at a directory with a few real datasets and\nthe output is tens of megabytes: no amount of trimming fits that into any\ncontext window that exists. To a generic packer, a CSV is just a large text\nfile, so the only choices it offers are *dump it raw* or *skip it entirely* —\nand both lose.\n\n**data2prompt** treats data files as data. Every table is profiled — schema,\ndtypes, per-column statistics, and missing-value counts computed on the\n*complete* dataset — then represented by a seeded random sample of real rows.\nThe model doesn't get a shrunken view of your data; it gets a statistical\nunderstanding of it that raw rows alone could never provide, plus enough real\nrows to see formats, ranges, and quirks. Every intervention is disclosed in a\nuniform notice grammar, so the LLM knows exactly what it is looking at and what\nwas left out.\n\nThe same data-heavy project, packed by three tools with default settings:\n\nRead that chart as feasibility, not savings. A 22 MB dump is millions of tokens — several times larger than the largest context window on the market, at any price. On a real data project, a generic packer doesn't produce an expensive prompt; it produces an impossible one. The 241 KB document is the only output of the three that a model can actually be handed.\n\nAnd the reduction is *representation*, not truncation. Every table still\ncontributes its full schema, per-column statistics computed on the complete\ndataset, and a seeded random sample of real rows. The model often ends up\nknowing *more* about your data than it would from pages of raw rows:\ndistributions, missingness, and dtypes are stated outright instead of left to\nbe inferred from whichever rows happened to fit. The model loses row noise,\nnot information.\n\n```\n# Recommended: install as a global CLI tool\npipx install data2prompt\n\n# Or into an active virtual environment\npip install data2prompt\n```\n\nRun it from your project root:\n\n```\ndata2prompt                    # → PROMPT.md (markdown, default settings)\ndata2prompt -b 100k -c        # fit the output into 100k tokens, copy to clipboard\ndata2prompt -f xml --schema-only   # XML format, schemas only, zero data rows\n```\n\n**Parquet / Feather / Arrow support** (optional extra)\n\nColumnar formats need [pyarrow](https://arrow.apache.org/docs/python/), which is\nnot bundled by default:\n\n```\npipx install \"data2prompt[parquet]\"     # fresh install\npipx inject data2prompt pyarrow          # already installed via pipx\npip install \"data2prompt[parquet]\"      # pip equivalent\n```\n\nWithout pyarrow these files still appear in the output with an inline note explaining why they were skipped.\n\n**Install from source**\n\n```\ngit clone https://github.com/arianmokhtariha/data2prompt.git\ncd data2prompt\npip install -e .\n```\n\nEvery file type gets a strategy, not a dump:\n\n| File type | Strategy | What the LLM sees |\n|---|---|---|\n`.csv` |\nSeeded random sampling | Column schema, full-dataset stats, N sampled rows |\n`.parquet` `.feather` `.arrow` |\nSame, via pyarrow | Schema + stats + sample — identical treatment to CSV |\n`.xlsx` `.xls` `.xlsm` |\nPer-sheet extraction | Each sheet as its own schema + stats + sample section |\n`.db` `.sqlite` `.sqlite3` |\nRead-only stdlib `sqlite3` |\nPer-table `CREATE TABLE` DDL (keys, FKs, indexes) + stats + sampled rows |\n`.sql` |\nStatement-aware parsing | Schema statements kept intact, `INSERT` floods capped |\n`.ipynb` |\nCell-level cleaning | Code, markdown and text outputs — base64 images and HTML dumps stripped |\n`.env` |\nName-only redaction | `KEY=<redacted>` — variable names, never values |\n| Binary files | Null-byte detection | Skipped, listed in the file index |\n| Everything else | Size-aware reading | Full text, or first 10 KB past `--max-file-size` |\n\nTwo details make the samples trustworthy:\n\n**Statistics are computed on the full dataset, not the sample.** Dtypes, missing counts/percentages, and a`describe()`\n\nsummary are extracted before sampling, so the model sees true data quality even from 15 rows.**Every intervention is disclosed.** Sampling, truncation, redaction, and skips all surface as uniform`-- [...] --`\n\nnotices inside the document — the model is never left guessing why content looks incomplete.\n\nState the outcome you want instead of tuning knobs:\n\n```\ndata2prompt --budget 100k\n```\n\n`--budget`\n\nruns a **de-escalation ladder** — halve CSV/SQL sample sizes, trim\nnotebook outputs, drop the stats blocks, switch to schema-only, and, as a last\nresort, omit the heaviest remaining files — re-rendering and re-counting the\n*actual* document after every step until it fits. No estimates: the number that\nis checked is the number you ship.\n\n- Accepts\n`50000`\n\n,`100k`\n\n,`1.5m`\n\n— commas and underscores welcome. - A\n**budget report** is embedded in the document and shown in the terminal report: every parameter change and every omitted file, stated explicitly. - If the budget is infeasible even at the ladder's floor,\n**nothing is written** and the process exits non-zero with the minimum achievable count — you will never silently receive an over-budget file.\n\nWhat sets data2prompt apart from a generic repo-to-text dumper isn't any one\nflag — it's that the whole pipeline is built around two rules: **never\nsilently lie about what the model is seeing**, and **never guess when the\nreal number can be measured**. Those two rules are why `--budget`\n\nre-renders\nand re-counts instead of estimating, why every reduction gets a `-- [...] --`\n\nnotice instead of vanishing quietly, and why the same command on the same\nproject produces byte-identical output a year later.\n\n**A full statistical profile on every table**— before a single row is sampled, each CSV, Parquet file, Excel sheet, and SQLite table is profiled on the*complete*dataset: per-column dtype, missing count and percentage, and the full`describe()`\n\nbattery (count, unique, top, freq, mean, std, min, quartiles, max) rendered as one unified schema table. Fifteen sampled rows plus this block tell the model more than a thousand raw rows would.**Samples that behave like the data**— sampling is seeded, and the drawn rows are re-sorted to original file order, so time series stay chronological and IDs stay ascending. Every sample notice cites the true size —`-- [Sample: random 15 of 1,234,567 rows] --`\n\n— captured*before*sampling, so the model can never mistake the sample for the dataset.**Exact dtypes from native schemas**— Parquet/Feather/Arrow columns carry pyarrow's native type strings (`int64`\n\n,`utf8`\n\n,`timestamp[us, tz=UTC]`\n\n) and SQLite columns their declared types from`PRAGMA table_info`\n\n, overriding pandas' lossier type inference.**Databases handled like databases**— SQLite files are verified by magic bytes, opened strictly read-only (`mode=ro`\n\n+`PRAGMA query_only`\n\n), and every table rendered with its full`CREATE`\n\nDDL — keys, foreign keys, indexes. Tables past 100k rows are`LIMIT`\n\n-read so a pathological database can never stall a run, and their stats block is honestly omitted rather than computed on a partial scan and passed off as truth.\n\n**The document teaches the model to read itself**— it opens with a reading contract: layout, structural conventions, the notice grammar, and explicit anti-hallucination accuracy rules. And the preamble is context-aware: conventions for notebooks, Excel, SQLite, or env files only appear when those file types were actually scanned this run.**An authoritative File Index**— one row per scanned file with its type and a controlled inclusion-status vocabulary (`Full`\n\n,`Sampled`\n\n,`Schema Only`\n\n,`Redacted`\n\n,`Omitted`\n\n, …). Every file the scan touched is accounted for; nothing silently vanishes from the document.**One notice grammar**— every tool intervention (sampling, truncation, redaction, skip, error) is a uniform`-- [Category: detail] --`\n\nline, taught once in the preamble, so the model can always separate what the tool says from what your files say.**Dynamic fence wrappers**— before any content is embedded, the longest backtick run inside it is measured and the enclosing fence is made one backtick longer. A file that itself contains`````\n\nfences — a README, a notebook, generated docs — can never break the document's structure.**One canonical path per file**— project-relative, forward-slashed, and byte-identical across the File Index, the file headers, and every notice, so the model can cross-reference sections by literal string match.**Two output formats, one contract**—`markdown`\n\n(default) and`xml`\n\nfor stronger structural anchoring in long contexts. Both are logically identical — same information, same order, same vocabulary — and governed by a written[output contract](/arianmokhtariha/data2prompt/blob/main/docs/output-contract.md). In XML mode, attributes carrying user data are properly quoted while file content stays verbatim — no`<`\n\nentity noise inflating tokens.**A closing recency anchor**— the document ends with an explicit end-of-codebase recap restating the accuracy rules, so the model knows the snapshot is complete and nothing follows.\n\n**Offline, exact token counting**— a bundled`o200k_base`\n\nBPE (tiktoken) counts the fully rendered document, scaffolding included. No network call, ever; a pure-regex fallback keeps counts flowing where the encoding cannot load.**Deterministic by default**—`--seed 42`\n\nmeans the exact same rows get sampled on every run. Regenerate a prompt for a diff, a reproducible eval, or a bug report and get the identical document back, not a new random draw.**Fails loud, never truncates silently**— if`--budget`\n\nis infeasible even at the ladder's floor,**nothing is written**. You get a non-zero exit and the minimum achievable token count — never a file that quietly blew past what you asked for.**Every budget decision is on the record**— a`--budget`\n\nrun embeds a Budget Report in the document itself: every parameter it tightened and every file it omitted, so both you and the model know exactly what was traded away to fit.\n\n**Per-file error containment**— every parser degrades a corrupt file, a locked file, or a truncated database to an inline error note for that file alone. One bad file — or one bad Excel sheet, or one bad SQLite table — never crashes the run.**Secret-safe by design**—`.env`\n\nvalues never reach the output (variable names only, values redacted); long lines are truncated to neutralize prompt-injection padding; binary content is detected and excluded.**Scan hygiene**— respects`.gitignore`\n\nwith real per-directory scoping (a`src/.gitignore`\n\napplies under`src/`\n\n, exactly like git), honors a project-level`.data2promptignore`\n\n, ships hardened core ignore lists (`.git`\n\n,`node_modules`\n\n, caches), and recognizes its own previously generated outputs by an embedded marker so it never packs itself.**Straight to clipboard**—`--clipboard`\n\npipes the result to your OS clipboard via native tools (`clip`\n\n/`pbcopy`\n\n/`xclip`\n\n/`wl-copy`\n\n) with a file fallback — UTF-16 on Windows so non-ASCII content round-trips intact.**A terminal UI that earns its place**— an animated glitch-sweep banner, a transient progress bar, and a final report with a token gauge against a 200K context window, a per-type composition chart, attention badges, and the heaviest files each with a token-share bar. Animations disable themselves on non-interactive output, and every quantity in the report doubles as a proportional bar.\n\nThe flags you will actually reach for:\n\n| Flag | Default | Purpose |\n|---|---|---|\n`-o` , `--output` |\n`PROMPT` |\nBase name of the generated file |\n`-f` , `--format` |\n`markdown` |\nOutput format: `markdown` or `xml` |\n`-b` , `--budget` |\noff | Target token budget (`50000` , `100k` , `1.5m` ) |\n`-c` , `--clipboard` |\noff | Copy to clipboard instead of writing a file |\n`-s` , `--csv-sample-size` |\n`15` |\nRows sampled per tabular file |\n`--seed` |\n`42` |\nSampling seed — identical output across runs |\n`--schema-only` |\noff | Schemas and dtypes only, zero data rows |\n`--max-lines` |\n`40` |\nOutput lines kept per notebook cell |\n`--max-sheets` |\n`10` |\nSheets processed per Excel workbook |\n`--max-tables` |\n`25` |\nTables processed per SQLite database |\n`--max-file-size` |\n`70` |\nKB threshold before plain files are head-truncated |\n`--no-stats-summary` |\nstats on | Drop the per-table stats block |\n`--no-env-keys` |\nredact | Skip `.env` files entirely instead of redacting |\n`--no-gitignore` |\nrespect | Ignore `.gitignore` rules while scanning |\n`--ignore-folders` / `--ignore-files` / `--skip-exts` |\n— | Additional exclusions, merged with the core ignore sets |\n\nFull reference with validation rules and edge cases: [docs/cli.md](/arianmokhtariha/data2prompt/blob/main/docs/cli.md)\n\nSmall, single-responsibility modules under an orchestration layer — parsing, output generation, scanning, token budgeting, and UI never bleed into each other:\n\n``` php\ngraph LR\n    CLI[cli.py] --> Main[main.py]\n    Main -->|Registry| Parsers[parsers.py]\n    Main -->|Strategy| Output[output.py]\n    Main -->|Scan + tokens| Utils[utils.py]\n    Main -->|Feedback| UI[ui.py]\n    Main -->|--budget| Budget[budget.py]\n    Budget --> Output\n```\n\n- A\n**parser registry** maps extensions to specialized parsers; new file types plug in without touching the pipeline. - An\n**output strategy** keeps markdown and XML generation interchangeable and contract-bound. - Fully typed (PEP 484), stdlib-first, with per-file error containment — one corrupt file degrades to an inline error note, never a crashed run.\n\nEvery module has a matching deep-dive document:\n\n|\n\n[Parsers](/arianmokhtariha/data2prompt/blob/main/docs/parsers.md)[Budget](/arianmokhtariha/data2prompt/blob/main/docs/budget.md)`--budget`\n\nde-escalation ladder, end to end[Output](/arianmokhtariha/data2prompt/blob/main/docs/output.md)·[Output Contract](/arianmokhtariha/data2prompt/blob/main/docs/output-contract.md)[CLI](/arianmokhtariha/data2prompt/blob/main/docs/cli.md)·[UI](/arianmokhtariha/data2prompt/blob/main/docs/ui.md)·[Installation](/arianmokhtariha/data2prompt/blob/main/docs/installation.md)\n\n```\npip install -e .[dev]\npytest\n```\n\nContributions are welcome — new file-type parsers are the highest-leverage place\nto start (the registry makes them self-contained). Open an issue first for\nanything that changes the generated document, and read\n[docs/output-contract.md](/arianmokhtariha/data2prompt/blob/main/docs/output-contract.md) before touching output code.\n\n*If data2prompt saved you time and tokens, a ⭐ helps other data people find it.*", "url": "https://wpnews.pro/news/show-hn-cli-tool-that-packages-data-heavy-projects-into-llm-context-windows", "canonical_source": "https://github.com/arianmokhtariha/data2prompt", "published_at": "2026-07-21 12:48:31+00:00", "updated_at": "2026-07-21 13:13:45.101843+00:00", "lang": "en", "topics": ["developer-tools", "large-language-models"], "entities": ["data2prompt", "Arian Mokhtariha", "GitHub", "pyarrow"], "alternates": {"html": "https://wpnews.pro/news/show-hn-cli-tool-that-packages-data-heavy-projects-into-llm-context-windows", "markdown": "https://wpnews.pro/news/show-hn-cli-tool-that-packages-data-heavy-projects-into-llm-context-windows.md", "text": "https://wpnews.pro/news/show-hn-cli-tool-that-packages-data-heavy-projects-into-llm-context-windows.txt", "jsonld": "https://wpnews.pro/news/show-hn-cli-tool-that-packages-data-heavy-projects-into-llm-context-windows.jsonld"}}