# Show HN: CLI tool that packages data-heavy projects into LLM context windows

> Source: <https://github.com/arianmokhtariha/data2prompt>
> Published: 2026-07-21 12:48:31+00:00

**Turn data-heavy projects into LLM context that actually fits — and actually informs.**

One command turns a project full of CSVs, notebooks, Excel workbooks and SQLite
databases into a single, structured, LLM-ready document — sampled, profiled,
redacted, and fitted to your context window.

Generic repo-to-prompt tools break down on data projects — and not by a margin
you can optimize away. Point them at a directory with a few real datasets and
the output is tens of megabytes: no amount of trimming fits that into any
context window that exists. To a generic packer, a CSV is just a large text
file, so the only choices it offers are *dump it raw* or *skip it entirely* —
and both lose.

**data2prompt** treats data files as data. Every table is profiled — schema,
dtypes, per-column statistics, and missing-value counts computed on the
*complete* dataset — then represented by a seeded random sample of real rows.
The model doesn't get a shrunken view of your data; it gets a statistical
understanding of it that raw rows alone could never provide, plus enough real
rows to see formats, ranges, and quirks. Every intervention is disclosed in a
uniform notice grammar, so the LLM knows exactly what it is looking at and what
was left out.

The same data-heavy project, packed by three tools with default settings:

Read that chart as feasibility, not savings. A 22 MB dump is millions of tokens — several times larger than the largest context window on the market, at any price. On a real data project, a generic packer doesn't produce an expensive prompt; it produces an impossible one. The 241 KB document is the only output of the three that a model can actually be handed.

And the reduction is *representation*, not truncation. Every table still
contributes its full schema, per-column statistics computed on the complete
dataset, and a seeded random sample of real rows. The model often ends up
knowing *more* about your data than it would from pages of raw rows:
distributions, missingness, and dtypes are stated outright instead of left to
be inferred from whichever rows happened to fit. The model loses row noise,
not information.

```
# Recommended: install as a global CLI tool
pipx install data2prompt

# Or into an active virtual environment
pip install data2prompt
```

Run it from your project root:

```
data2prompt                    # → PROMPT.md (markdown, default settings)
data2prompt -b 100k -c        # fit the output into 100k tokens, copy to clipboard
data2prompt -f xml --schema-only   # XML format, schemas only, zero data rows
```

**Parquet / Feather / Arrow support** (optional extra)

Columnar formats need [pyarrow](https://arrow.apache.org/docs/python/), which is
not bundled by default:

```
pipx install "data2prompt[parquet]"     # fresh install
pipx inject data2prompt pyarrow          # already installed via pipx
pip install "data2prompt[parquet]"      # pip equivalent
```

Without pyarrow these files still appear in the output with an inline note explaining why they were skipped.

**Install from source**

```
git clone https://github.com/arianmokhtariha/data2prompt.git
cd data2prompt
pip install -e .
```

Every file type gets a strategy, not a dump:

| File type | Strategy | What the LLM sees |
|---|---|---|
`.csv` |
Seeded random sampling | Column schema, full-dataset stats, N sampled rows |
`.parquet` `.feather` `.arrow` |
Same, via pyarrow | Schema + stats + sample — identical treatment to CSV |
`.xlsx` `.xls` `.xlsm` |
Per-sheet extraction | Each sheet as its own schema + stats + sample section |
`.db` `.sqlite` `.sqlite3` |
Read-only stdlib `sqlite3` |
Per-table `CREATE TABLE` DDL (keys, FKs, indexes) + stats + sampled rows |
`.sql` |
Statement-aware parsing | Schema statements kept intact, `INSERT` floods capped |
`.ipynb` |
Cell-level cleaning | Code, markdown and text outputs — base64 images and HTML dumps stripped |
`.env` |
Name-only redaction | `KEY=<redacted>` — variable names, never values |
| Binary files | Null-byte detection | Skipped, listed in the file index |
| Everything else | Size-aware reading | Full text, or first 10 KB past `--max-file-size` |

Two details make the samples trustworthy:

**Statistics are computed on the full dataset, not the sample.** Dtypes, missing counts/percentages, and a`describe()`

summary are extracted before sampling, so the model sees true data quality even from 15 rows.**Every intervention is disclosed.** Sampling, truncation, redaction, and skips all surface as uniform`-- [...] --`

notices inside the document — the model is never left guessing why content looks incomplete.

State the outcome you want instead of tuning knobs:

```
data2prompt --budget 100k
```

`--budget`

runs a **de-escalation ladder** — halve CSV/SQL sample sizes, trim
notebook outputs, drop the stats blocks, switch to schema-only, and, as a last
resort, omit the heaviest remaining files — re-rendering and re-counting the
*actual* document after every step until it fits. No estimates: the number that
is checked is the number you ship.

- Accepts
`50000`

,`100k`

,`1.5m`

— commas and underscores welcome. - A
**budget report** is embedded in the document and shown in the terminal report: every parameter change and every omitted file, stated explicitly. - If the budget is infeasible even at the ladder's floor,
**nothing is written** and the process exits non-zero with the minimum achievable count — you will never silently receive an over-budget file.

What sets data2prompt apart from a generic repo-to-text dumper isn't any one
flag — it's that the whole pipeline is built around two rules: **never
silently lie about what the model is seeing**, and **never guess when the
real number can be measured**. Those two rules are why `--budget`

re-renders
and re-counts instead of estimating, why every reduction gets a `-- [...] --`

notice instead of vanishing quietly, and why the same command on the same
project produces byte-identical output a year later.

**A full statistical profile on every table**— before a single row is sampled, each CSV, Parquet file, Excel sheet, and SQLite table is profiled on the*complete*dataset: per-column dtype, missing count and percentage, and the full`describe()`

battery (count, unique, top, freq, mean, std, min, quartiles, max) rendered as one unified schema table. Fifteen sampled rows plus this block tell the model more than a thousand raw rows would.**Samples that behave like the data**— sampling is seeded, and the drawn rows are re-sorted to original file order, so time series stay chronological and IDs stay ascending. Every sample notice cites the true size —`-- [Sample: random 15 of 1,234,567 rows] --`

— captured*before*sampling, so the model can never mistake the sample for the dataset.**Exact dtypes from native schemas**— Parquet/Feather/Arrow columns carry pyarrow's native type strings (`int64`

,`utf8`

,`timestamp[us, tz=UTC]`

) and SQLite columns their declared types from`PRAGMA table_info`

, overriding pandas' lossier type inference.**Databases handled like databases**— SQLite files are verified by magic bytes, opened strictly read-only (`mode=ro`

+`PRAGMA query_only`

), and every table rendered with its full`CREATE`

DDL — keys, foreign keys, indexes. Tables past 100k rows are`LIMIT`

-read so a pathological database can never stall a run, and their stats block is honestly omitted rather than computed on a partial scan and passed off as truth.

**The document teaches the model to read itself**— it opens with a reading contract: layout, structural conventions, the notice grammar, and explicit anti-hallucination accuracy rules. And the preamble is context-aware: conventions for notebooks, Excel, SQLite, or env files only appear when those file types were actually scanned this run.**An authoritative File Index**— one row per scanned file with its type and a controlled inclusion-status vocabulary (`Full`

,`Sampled`

,`Schema Only`

,`Redacted`

,`Omitted`

, …). Every file the scan touched is accounted for; nothing silently vanishes from the document.**One notice grammar**— every tool intervention (sampling, truncation, redaction, skip, error) is a uniform`-- [Category: detail] --`

line, taught once in the preamble, so the model can always separate what the tool says from what your files say.**Dynamic fence wrappers**— before any content is embedded, the longest backtick run inside it is measured and the enclosing fence is made one backtick longer. A file that itself contains`````

fences — a README, a notebook, generated docs — can never break the document's structure.**One canonical path per file**— project-relative, forward-slashed, and byte-identical across the File Index, the file headers, and every notice, so the model can cross-reference sections by literal string match.**Two output formats, one contract**—`markdown`

(default) and`xml`

for stronger structural anchoring in long contexts. Both are logically identical — same information, same order, same vocabulary — and governed by a written[output contract](/arianmokhtariha/data2prompt/blob/main/docs/output-contract.md). In XML mode, attributes carrying user data are properly quoted while file content stays verbatim — no`<`

entity noise inflating tokens.**A closing recency anchor**— the document ends with an explicit end-of-codebase recap restating the accuracy rules, so the model knows the snapshot is complete and nothing follows.

**Offline, exact token counting**— a bundled`o200k_base`

BPE (tiktoken) counts the fully rendered document, scaffolding included. No network call, ever; a pure-regex fallback keeps counts flowing where the encoding cannot load.**Deterministic by default**—`--seed 42`

means the exact same rows get sampled on every run. Regenerate a prompt for a diff, a reproducible eval, or a bug report and get the identical document back, not a new random draw.**Fails loud, never truncates silently**— if`--budget`

is infeasible even at the ladder's floor,**nothing is written**. You get a non-zero exit and the minimum achievable token count — never a file that quietly blew past what you asked for.**Every budget decision is on the record**— a`--budget`

run embeds a Budget Report in the document itself: every parameter it tightened and every file it omitted, so both you and the model know exactly what was traded away to fit.

**Per-file error containment**— every parser degrades a corrupt file, a locked file, or a truncated database to an inline error note for that file alone. One bad file — or one bad Excel sheet, or one bad SQLite table — never crashes the run.**Secret-safe by design**—`.env`

values never reach the output (variable names only, values redacted); long lines are truncated to neutralize prompt-injection padding; binary content is detected and excluded.**Scan hygiene**— respects`.gitignore`

with real per-directory scoping (a`src/.gitignore`

applies under`src/`

, exactly like git), honors a project-level`.data2promptignore`

, ships hardened core ignore lists (`.git`

,`node_modules`

, caches), and recognizes its own previously generated outputs by an embedded marker so it never packs itself.**Straight to clipboard**—`--clipboard`

pipes the result to your OS clipboard via native tools (`clip`

/`pbcopy`

/`xclip`

/`wl-copy`

) with a file fallback — UTF-16 on Windows so non-ASCII content round-trips intact.**A terminal UI that earns its place**— an animated glitch-sweep banner, a transient progress bar, and a final report with a token gauge against a 200K context window, a per-type composition chart, attention badges, and the heaviest files each with a token-share bar. Animations disable themselves on non-interactive output, and every quantity in the report doubles as a proportional bar.

The flags you will actually reach for:

| Flag | Default | Purpose |
|---|---|---|
`-o` , `--output` |
`PROMPT` |
Base name of the generated file |
`-f` , `--format` |
`markdown` |
Output format: `markdown` or `xml` |
`-b` , `--budget` |
off | Target token budget (`50000` , `100k` , `1.5m` ) |
`-c` , `--clipboard` |
off | Copy to clipboard instead of writing a file |
`-s` , `--csv-sample-size` |
`15` |
Rows sampled per tabular file |
`--seed` |
`42` |
Sampling seed — identical output across runs |
`--schema-only` |
off | Schemas and dtypes only, zero data rows |
`--max-lines` |
`40` |
Output lines kept per notebook cell |
`--max-sheets` |
`10` |
Sheets processed per Excel workbook |
`--max-tables` |
`25` |
Tables processed per SQLite database |
`--max-file-size` |
`70` |
KB threshold before plain files are head-truncated |
`--no-stats-summary` |
stats on | Drop the per-table stats block |
`--no-env-keys` |
redact | Skip `.env` files entirely instead of redacting |
`--no-gitignore` |
respect | Ignore `.gitignore` rules while scanning |
`--ignore-folders` / `--ignore-files` / `--skip-exts` |
— | Additional exclusions, merged with the core ignore sets |

Full reference with validation rules and edge cases: [docs/cli.md](/arianmokhtariha/data2prompt/blob/main/docs/cli.md)

Small, single-responsibility modules under an orchestration layer — parsing, output generation, scanning, token budgeting, and UI never bleed into each other:

``` php
graph LR
    CLI[cli.py] --> Main[main.py]
    Main -->|Registry| Parsers[parsers.py]
    Main -->|Strategy| Output[output.py]
    Main -->|Scan + tokens| Utils[utils.py]
    Main -->|Feedback| UI[ui.py]
    Main -->|--budget| Budget[budget.py]
    Budget --> Output
```

- A
**parser registry** maps extensions to specialized parsers; new file types plug in without touching the pipeline. - An
**output strategy** keeps markdown and XML generation interchangeable and contract-bound. - Fully typed (PEP 484), stdlib-first, with per-file error containment — one corrupt file degrades to an inline error note, never a crashed run.

Every module has a matching deep-dive document:

|

[Parsers](/arianmokhtariha/data2prompt/blob/main/docs/parsers.md)[Budget](/arianmokhtariha/data2prompt/blob/main/docs/budget.md)`--budget`

de-escalation ladder, end to end[Output](/arianmokhtariha/data2prompt/blob/main/docs/output.md)·[Output Contract](/arianmokhtariha/data2prompt/blob/main/docs/output-contract.md)[CLI](/arianmokhtariha/data2prompt/blob/main/docs/cli.md)·[UI](/arianmokhtariha/data2prompt/blob/main/docs/ui.md)·[Installation](/arianmokhtariha/data2prompt/blob/main/docs/installation.md)

```
pip install -e .[dev]
pytest
```

Contributions are welcome — new file-type parsers are the highest-leverage place
to start (the registry makes them self-contained). Open an issue first for
anything that changes the generated document, and read
[docs/output-contract.md](/arianmokhtariha/data2prompt/blob/main/docs/output-contract.md) before touching output code.

*If data2prompt saved you time and tokens, a ⭐ helps other data people find it.*