{"slug": "datasette-arrows-0-1a0-streaming-apache-arrow-exports-in-datasette", "title": "datasette-arrows 0.1a0: Streaming Apache Arrow Exports in Datasette", "summary": "Simon Willison released datasette-arrows 0.1a0 on November 18, 2026, a reference plugin for Datasette's new streaming export hook that adds Apache Arrow IPC (.arrows) downloads from tables, views and SQL queries without loading an entire SQLite file into RAM. The plugin, installable via `datasette install datasette-arrows`, requires datasette>=1.0a43 and pyarrow, serves the application/vnd.apache.arrow.stream MIME type, and streams one record batch per page when `?_stream=1` is set, with `allow_stream` and `max_stream_mb` settings applying the same guardrails as CSV streaming. Willison wrote that he \"never releases a new plugin feature without a plugin that uses it,\" and noted that Claude Fable 5.2 suggested Arrow streaming and implemented most of the package in one shot.", "body_md": "**datasette-arrows 0.1a0** shipped on **November 18, 2026** as a **reference plugin** for Datasette's new **streaming export hook**. It adds **Apache Arrow IPC** (`.arrows`) downloads from **tables, views, and SQL queries** without loading an entire SQLite file into RAM. Sources: [Simon Willison's release note](https://simonwillison.net/2026/Nov/18/datasette-arrows/?ref=explainx), [PyPI](https://pypi.org/project/datasette-arrows/?ref=explainx), and the [datasette/datasette-arrows README](https://github.com/datasette/datasette-arrows?ref=explainx).\n\nWillison wrote that he **\"never releases a new plugin feature without a plugin that uses it.\"** Here, **Claude Fable 5.2** suggested **Arrow streaming** and implemented most of the package in **one shot**—a useful data point next to [Datasette 1.0a43's streaming CSV work](https://explainx.ai/blog/datasette-1-0a43-stored-query-advisory-streaming-2026) we covered earlier in November.\n\n## TL;DR\n\n| Piece | Detail | \n|---|---|\n| Version | 0.1a0 on PyPI, 2026-11-18 | \n| Install | `datasette install datasette-arrows` | \n| URLs | `/db/table.arrows` ,`/db/-/query.arrows?sql=...` | \n| Stream all rows | `?_stream=1` or Export dialog checkbox | \n| MIME | `application/vnd.apache.arrow.stream` | \n| Python deps | `datasette>=1.0a43` ,`pyarrow` | \n| License | Apache-2.0 | \n\n## Why Arrow after CSV and TSV streaming\n\nDatasette export dialog showing the new Apache Arrow streaming option\n*Image: [Simon Willison](https://simonwillison.net/2026/Nov/18/datasette-arrows/?ref=explainx)*\n\nDatasette already streamed **CSV and TSV** because those formats **chunk cleanly**. Arrow IPC adds **typed columns** and **zero-copy** paths into **pandas, Polars, and PyArrow** workflows—closer to what **[LanceDB multimodal pipelines](https://explainx.ai/blog/lancedb-realtime-engine-memwal-52ms-november-2026)** want than **string parsing**.\n\nFor publishers exposing **public-interest SQLite**—government logs, leaks, research drops—Arrow means a **data journalist** can `read_all()` in **Python** while a **million-row table** still **streams from the server**.\n\n## URL patterns and streaming semantics\n\nAdapter pipes meeting through a junction evoking typed columnar export streams beside CSV\n\nFrom the README:\n\n- **Paginated default:**`/database/table.arrows` returns** one page** like HTML.\n- **Full stream:**`/database/table.arrows?_stream=1` emits** one record batch per page** until complete.\n- **SQL queries:**`/database/-/query.arrows?sql=select+*+from+table`\n- **Download header:**`?_dl=1` forces file download.\n- **Governance:**`allow_stream` and`max_stream_mb` Datasette settings apply—same guardrails as CSV streaming.\n\nFilters, sort orders, and `?_col=` projections apply to exports, so **Arrow is not a bypass** around **Datasette's SQL safety**.\n\n## Column typing rules that bite in production\n\nSQL query funnel into a typed table, evoking SQLite columns mapped to Arrow types\n\nSQLite allows **mixed types**; Arrow requires **one type per column**. The plugin infers from **declared schema** plus **first page values**:\n\n| SQLite values seen | Arrow type | \n|---|---|\n| Integers | int64 | \n| Floats or int/float mix | float64 | \n| Text or text/number mix | string | \n| Blobs or mixed with blobs | binary | \n| All null, no declared type | string | \n\nIf a later page introduces an **incompatible value**—text in an INTEGER column—the export **drops the connection** so clients never treat a **truncated stream** as complete. Fix with explicit SQL casts, e.g. `select cast(value as text) as value from ...`.\n\nThat behavior is **stricter than CSV**, which happily emits **mangled strings**. For ML exporters, strictness prevents **silent label corruption**.\n\n## Reading streams in Python\n\nThe README documents the minimal consumer:\n\n``` python\nimport urllib.request\nimport pyarrow as pa\n\nwith urllib.request.urlopen(\n    \"http://localhost:8001/data/dogs.arrows?_stream=1\"\n) as response:\n    table = pa.ipc.open_stream(response).read_all()\n```\n\nFrom there, zero-copy handoff to **Polars**, ** DuckDB**, or **training pipelines** is standard. Compare to [Willison's Pyodide Datasette experiment](https://explainx.ai/blog/datasette-1-0a43-stored-query-advisory-streaming-2026) themes: **SQLite in the browser** for demos, **Arrow on the wire** for **analytics scale**.\n\n## UI and release cadence caveats\n\nWillison's screenshot uses **Datasette main**, where **streaming formats** will soon sit beside CSV/TSV under **\"All rows in this table\"** in the export dialog. **0.1a0** shipped before that UI grouping landed in a **numbered alpha**, so early adopters may need **main** or the **next alpha** for the polished dialog.\n\nThe plugin **requires streaming output renderers**—present on **main**, not necessarily on the **latest tagged release** you run in production. Pin versions explicitly in **Docker images**.\n\n## How this fits explainx.ai's tool coverage\n\ndatasette-arrows is **small** but **exemplary**: it shows how **plugin hooks** should ship with **reference implementations**. The same week brought **[Vercel Board](https://explainx.ai/blog/vercel-board-agent-message-board-november-2026)** for **agent coordination** and **[CodeRabbit Verify](https://explainx.ai/blog/coderabbit-verify-agentic-qa-public-beta-november-2026)** for **runtime QA**—different layers, same lesson: **agents need contracts**, not just **models**.\n\nFor **local AI**, Arrow exports pair with **[AirLLM-style constrained inference](https://explainx.ai/blog/airllm-run-70b-llm-4gb-gpu-inference-2026)** when you are featurizing **public tables** on **modest GPUs**—download **features**, not **giant CSV text**.\n\n## Security and publishing notes\n\nStreaming exports can **exfiltrate large tables** if `allow_stream` is on for **sensitive databases**. Treat `.arrows` endpoints like **CSV download** permissions. **Row-level auth plugins** still apply; Arrow does not magically **redact** columns.\n\nIf you expose Datasette on the **public internet**, keep **[stored-query advisory fixes](https://explainx.ai/blog/datasette-1-0a43-stored-query-advisory-streaming-2026)** patched—**streaming does not fix SQL injection** in saved queries.\n\n## Try it in fifteen minutes\n\n1. Install **Datasette main** or**>=1.0a43** with streaming renderers enabled.\n2. `datasette install datasette-arrows`\n3. `datasette serve data.db`\n4. Open a table, confirm **Arrow** in**Export data** .\n5. `curl -o out.arrows 'http://localhost:8001/db/table.arrows?_stream=1'`\n6. Load with **pyarrow** and compare row count to`select count(*)` .\n\nOpen an issue on **datasette/datasette-arrows** if **type coercion** fails on real-world messy columns—that feedback shapes **1.0**.\n\n## Hook mechanics: register_output_renderer()\n\nDatasette **1.0a43** extended plugins so authors can register **streaming output renderers** alongside built-in CSV and TSV. The core idea: **fetch a page of rows**, **serialize a chunk**, **repeat**—never materialize the whole table in the Datasette process. datasette-arrows is the **reference implementation** for **binary columnar** output instead of **delimited text**.\n\nPlugin authors should copy its patterns:\n\n- Honor **`?_stream=1`** and dialog**Include all rows** .\n- Respect **`allow_stream`** and**` max_stream_mb`** caps.\n- Fail **hard** on type conflicts rather than emit**silent corruption** .\n\nIf you maintain **custom SQLite** packs for newsrooms or NGOs, shipping **Arrow** lowers the bar for **partner universities** that already standardize on **PyArrow**.\n\n## Integration sketches beyond pyarrow\n\n**Polars** can read Arrow streams with `pl.read_ipc_stream`. **DuckDB** can attach Arrow streams for **SQL transforms** before **Parquet** archival. **Training stacks** that expect **HF datasets** can still ingest Arrow, convert once to **sharded Parquet**, and push to **[LanceDB](https://explainx.ai/blog/lancedb-realtime-engine-memwal-52ms-november-2026)** if you later need **vector search**.\n\nFor **agents**, a coding agent that **knows Arrow schemas** can validate **column dtypes** before **fine-tune**—fewer **silent NaN** bugs than **CSV guesswork**.\n\n## Comparison to CSV for million-row tables\n\nCSV wins on **universal tool support** and **human grep**. Arrow wins when:\n\n- Columns are **numeric-heavy** (counts, lat/lon, embeddings serialized as lists).\n- You need **stable dtypes** across**pages** .\n- You pipeline into **Python ML** without**parse overhead** .\n\nBenchmark on your table: time **`table.csv?_stream=1`** versus **` table.arrows?_stream=1`** into **Polars** on the same laptop. Arrow should dominate when **columns > 20** or **rows > 1M**.\n\n## Maintainer and CI notes\n\nThe GitHub repo ships **`uv run pytest`**. Depend on **` datasette>=1.0a43`** and **` pyarrow`**. Because streaming renderers lagged **tagged releases** when **0.1a0** shipped, pin **Docker images** to **`datasette:main`** only if you accept **moving targets**—otherwise wait for the **next alpha** Willison promised for **export dialog grouping**.\n\nWatch **[Datasette plugins directory](https://datasette.io/plugins?ref=explainx)** for **composition**: combine **datasette-arrows** with **label plugins** for **foreign keys in Arrow exports** via `?_labels=on`.\n\n## When this release is not news for you\n\nIf you do not run **Datasette**, skip this plugin. If you only publish **500-row tables**, CSV is fine. If you are on **Datasette** versions before **1.0a43**, upgrade **[stored-query patches](https://explainx.ai/blog/datasette-1-0a43-stored-query-advisory-streaming-2026)** before experimenting.\n\n## Observability and abuse monitoring\n\nOperators should log **`.arrows?_stream=1` hits** the same way they log **large CSV exports**. A sudden spike may mean **scraping**, not legitimate research. Combine **`max_stream_mb`** with **rate limits** at the reverse proxy when tables contain **PII**.\n\nFor **multitenant Datasette Cloud** (if you run one), partition **plugins per deployment** so **Arrow** cannot become a **covert exfil channel** from **restricted databases**.\n\n## Roadmap signals from Willison's release cadence\n\nWillison's post notes the **export dialog UI** will group **streaming formats** in the **next alpha** after **0.1a0**. That means early adopters may field **support tickets** from users who do not see **Include all rows** beside Arrow yet. Document **`?_stream=1` deep links** in your **data handbook** until the UI catches up.\n\nFuture plugins may add **Parquet stream**, **GeoArrow**, or **embedding-specific** layouts using the same hook—datasette-arrows is the **template**, not the **final catalog**.\n\n## Ethics of publishing queryable public data\n\nArrow makes **re-identification attacks** faster when tables include **quasi-identifiers**. If you publish **census microdata** or **leaked corp tables**, review **[k-anonymity](https://en.wikipedia.org/wiki/K-anonymity?ref=explainx)** basics even when the format is modern. Streaming does not remove **duty of care**.\n\nJournalists pairing Datasette with **[AI Overviews traffic loss stories](https://explainx.ai/blog/all-about-berlin-google-ai-overviews-traffic-loss-november-2026)** should remember: **machines read Arrow faster than humans read HTML**, changing **scraper economics** overnight.\n\n## Troubleshooting common failures\n\n**Empty stream after 200 OK:** Check whether you forgot **`?_stream=1`** on a multi-million-row table; the first response may be **only one page**.\n\n**Connection reset mid-download:** Often **type coercion** failure on a later page. Re-run export with a SQL query that **casts messy columns** to text.\n\n**Plugin not listed:** Confirm **`uv run datasette plugins`** shows datasette-arrows and that you are on a Datasette build with **streaming renderers** enabled.\n\n**pyarrow ImportError:** Install pyarrow in the **client environment** reading the stream, not only on the server.\n\n**Foreign keys missing labels:** Enable **`?_labels=on`** on the URL or verify the **labels plugin** chain you expect is active.\n\n## Why Willison shipped Arrow before Parquet stream\n\nArrow IPC is **row-batch oriented** for **incremental HTTP**, matching Datasette's **page loop**. Parquet wants **column stripes** and is **heavier to stream** without custom footers. If Parquet streaming arrives later, expect **different memory tradeoffs**—Arrow remains the **low-latency wire format** for **Python analytics**.\n\n## Ecosystem placement next to Lance and DuckDB\n\nTeams exporting **SQLite snapshots** to **Lance** for **vector search** can treat **datasette-arrows** as **step one** (fast extract) before **offline convert**. Teams on **DuckDB** can **`ATTACH`** Arrow streams for **JOIN** against **CSV dimensions** without **double disk use**.\n\nNone of that is automatic—you still write **glue**—but you avoid **CSV parse tax** on **wide tables**.", "url": "https://wpnews.pro/news/datasette-arrows-0-1a0-streaming-apache-arrow-exports-in-datasette", "canonical_source": "https://explainx.ai/blog/datasette-arrows-apache-arrow-streaming-export-2026?utm_source=rss&utm_medium=feed&utm_campaign=blog", "published_at": "2026-11-19 00:00:00+00:00", "updated_at": "2026-11-19 06:42:23.775641+00:00", "lang": "en", "topics": ["developer-tools", "ai-tools", "ai-agents"], "entities": ["datasette-arrows", "Datasette", "Simon Willison", "Apache Arrow", "pyarrow", "Claude Fable 5.2", "PyPI", "Polars"], "also_reported_by": [], "takeaway": "Datasette 1.0a43's streaming export hook now supports Apache Arrow IPC via datasette-arrows 0.1a0, which streams one record batch per page with `?_stream=1` and drops the connection on type-incompatible later pages rather than emitting a truncated stream.", "recommendation": null, "alternates": {"html": "https://wpnews.pro/news/datasette-arrows-0-1a0-streaming-apache-arrow-exports-in-datasette", "markdown": "https://wpnews.pro/news/datasette-arrows-0-1a0-streaming-apache-arrow-exports-in-datasette.md", "text": "https://wpnews.pro/news/datasette-arrows-0-1a0-streaming-apache-arrow-exports-in-datasette.txt", "jsonld": "https://wpnews.pro/news/datasette-arrows-0-1a0-streaming-apache-arrow-exports-in-datasette.jsonld"}}