# datasette-arrows 0.1a0: Streaming Apache Arrow Exports in Datasette

> Source: <https://explainx.ai/blog/datasette-arrows-apache-arrow-streaming-export-2026?utm_source=rss&utm_medium=feed&utm_campaign=blog>
> Published: 2026-11-19 00:00:00+00:00

**Key takeaway:** Datasette 1.0a43's streaming export hook now supports Apache Arrow IPC via datasette-arrows 0.1a0, which streams one record batch per page with `?_stream=1` and drops the connection on type-incompatible later pages rather than emitting a truncated stream.

**datasette-arrows 0.1a0** shipped on **November 18, 2026** as a **reference plugin** for Datasette's new **streaming export hook**. It adds **Apache Arrow IPC** (`.arrows`) downloads from **tables, views, and SQL queries** without loading an entire SQLite file into RAM. Sources: [Simon Willison's release note](https://simonwillison.net/2026/Nov/18/datasette-arrows/?ref=explainx), [PyPI](https://pypi.org/project/datasette-arrows/?ref=explainx), and the [datasette/datasette-arrows README](https://github.com/datasette/datasette-arrows?ref=explainx).

Willison wrote that he **"never releases a new plugin feature without a plugin that uses it."** Here, **Claude Fable 5.2** suggested **Arrow streaming** and implemented most of the package in **one shot**—a useful data point next to [Datasette 1.0a43's streaming CSV work](https://explainx.ai/blog/datasette-1-0a43-stored-query-advisory-streaming-2026) we covered earlier in November.

## TL;DR

| Piece | Detail | 
|---|---|
| Version | 0.1a0 on PyPI, 2026-11-18 | 
| Install | `datasette install datasette-arrows` | 
| URLs | `/db/table.arrows` ,`/db/-/query.arrows?sql=...` | 
| Stream all rows | `?_stream=1` or Export dialog checkbox | 
| MIME | `application/vnd.apache.arrow.stream` | 
| Python deps | `datasette>=1.0a43` ,`pyarrow` | 
| License | Apache-2.0 | 

## Why Arrow after CSV and TSV streaming

Datasette export dialog showing the new Apache Arrow streaming option
*Image: [Simon Willison](https://simonwillison.net/2026/Nov/18/datasette-arrows/?ref=explainx)*

Datasette already streamed **CSV and TSV** because those formats **chunk cleanly**. Arrow IPC adds **typed columns** and **zero-copy** paths into **pandas, Polars, and PyArrow** workflows—closer to what **[LanceDB multimodal pipelines](https://explainx.ai/blog/lancedb-realtime-engine-memwal-52ms-november-2026)** want than **string parsing**.

For publishers exposing **public-interest SQLite**—government logs, leaks, research drops—Arrow means a **data journalist** can `read_all()` in **Python** while a **million-row table** still **streams from the server**.

## URL patterns and streaming semantics

Adapter pipes meeting through a junction evoking typed columnar export streams beside CSV

From the README:

- **Paginated default:**`/database/table.arrows` returns** one page** like HTML.
- **Full stream:**`/database/table.arrows?_stream=1` emits** one record batch per page** until complete.
- **SQL queries:**`/database/-/query.arrows?sql=select+*+from+table`
- **Download header:**`?_dl=1` forces file download.
- **Governance:**`allow_stream` and`max_stream_mb` Datasette settings apply—same guardrails as CSV streaming.

Filters, sort orders, and `?_col=` projections apply to exports, so **Arrow is not a bypass** around **Datasette's SQL safety**.

## Column typing rules that bite in production

SQL query funnel into a typed table, evoking SQLite columns mapped to Arrow types

SQLite allows **mixed types**; Arrow requires **one type per column**. The plugin infers from **declared schema** plus **first page values**:

| SQLite values seen | Arrow type | 
|---|---|
| Integers | int64 | 
| Floats or int/float mix | float64 | 
| Text or text/number mix | string | 
| Blobs or mixed with blobs | binary | 
| All null, no declared type | string | 

If a later page introduces an **incompatible value**—text in an INTEGER column—the export **drops the connection** so clients never treat a **truncated stream** as complete. Fix with explicit SQL casts, e.g. `select cast(value as text) as value from ...`.

That behavior is **stricter than CSV**, which happily emits **mangled strings**. For ML exporters, strictness prevents **silent label corruption**.

## Reading streams in Python

The README documents the minimal consumer:

``` python
import urllib.request
import pyarrow as pa

with urllib.request.urlopen(
    "http://localhost:8001/data/dogs.arrows?_stream=1"
) as response:
    table = pa.ipc.open_stream(response).read_all()
```

From there, zero-copy handoff to **Polars**, ** DuckDB**, or **training pipelines** is standard. Compare to [Willison's Pyodide Datasette experiment](https://explainx.ai/blog/datasette-1-0a43-stored-query-advisory-streaming-2026) themes: **SQLite in the browser** for demos, **Arrow on the wire** for **analytics scale**.

## UI and release cadence caveats

Willison's screenshot uses **Datasette main**, where **streaming formats** will soon sit beside CSV/TSV under **"All rows in this table"** in the export dialog. **0.1a0** shipped before that UI grouping landed in a **numbered alpha**, so early adopters may need **main** or the **next alpha** for the polished dialog.

The plugin **requires streaming output renderers**—present on **main**, not necessarily on the **latest tagged release** you run in production. Pin versions explicitly in **Docker images**.

## How this fits explainx.ai's tool coverage

datasette-arrows is **small** but **exemplary**: it shows how **plugin hooks** should ship with **reference implementations**. The same week brought **[Vercel Board](https://explainx.ai/blog/vercel-board-agent-message-board-november-2026)** for **agent coordination** and **[CodeRabbit Verify](https://explainx.ai/blog/coderabbit-verify-agentic-qa-public-beta-november-2026)** for **runtime QA**—different layers, same lesson: **agents need contracts**, not just **models**.

For **local AI**, Arrow exports pair with **[AirLLM-style constrained inference](https://explainx.ai/blog/airllm-run-70b-llm-4gb-gpu-inference-2026)** when you are featurizing **public tables** on **modest GPUs**—download **features**, not **giant CSV text**.

## Security and publishing notes

Streaming exports can **exfiltrate large tables** if `allow_stream` is on for **sensitive databases**. Treat `.arrows` endpoints like **CSV download** permissions. **Row-level auth plugins** still apply; Arrow does not magically **redact** columns.

If you expose Datasette on the **public internet**, keep **[stored-query advisory fixes](https://explainx.ai/blog/datasette-1-0a43-stored-query-advisory-streaming-2026)** patched—**streaming does not fix SQL injection** in saved queries.

## Try it in fifteen minutes

1. Install **Datasette main** or**>=1.0a43** with streaming renderers enabled.
2. `datasette install datasette-arrows`
3. `datasette serve data.db`
4. Open a table, confirm **Arrow** in**Export data** .
5. `curl -o out.arrows 'http://localhost:8001/db/table.arrows?_stream=1'`
6. Load with **pyarrow** and compare row count to`select count(*)` .

Open an issue on **datasette/datasette-arrows** if **type coercion** fails on real-world messy columns—that feedback shapes **1.0**.

## Hook mechanics: register_output_renderer()

Datasette **1.0a43** extended plugins so authors can register **streaming output renderers** alongside built-in CSV and TSV. The core idea: **fetch a page of rows**, **serialize a chunk**, **repeat**—never materialize the whole table in the Datasette process. datasette-arrows is the **reference implementation** for **binary columnar** output instead of **delimited text**.

Plugin authors should copy its patterns:

- Honor **`?_stream=1`** and dialog**Include all rows** .
- Respect **`allow_stream`** and**` max_stream_mb`** caps.
- Fail **hard** on type conflicts rather than emit**silent corruption** .

If you maintain **custom SQLite** packs for newsrooms or NGOs, shipping **Arrow** lowers the bar for **partner universities** that already standardize on **PyArrow**.

## Integration sketches beyond pyarrow

**Polars** can read Arrow streams with `pl.read_ipc_stream`. **DuckDB** can attach Arrow streams for **SQL transforms** before **Parquet** archival. **Training stacks** that expect **HF datasets** can still ingest Arrow, convert once to **sharded Parquet**, and push to **[LanceDB](https://explainx.ai/blog/lancedb-realtime-engine-memwal-52ms-november-2026)** if you later need **vector search**.

For **agents**, a coding agent that **knows Arrow schemas** can validate **column dtypes** before **fine-tune**—fewer **silent NaN** bugs than **CSV guesswork**.

## Comparison to CSV for million-row tables

CSV wins on **universal tool support** and **human grep**. Arrow wins when:

- Columns are **numeric-heavy** (counts, lat/lon, embeddings serialized as lists).
- You need **stable dtypes** across**pages** .
- You pipeline into **Python ML** without**parse overhead** .

Benchmark on your table: time **`table.csv?_stream=1`** versus **` table.arrows?_stream=1`** into **Polars** on the same laptop. Arrow should dominate when **columns > 20** or **rows > 1M**.

## Maintainer and CI notes

The GitHub repo ships **`uv run pytest`**. Depend on **` datasette>=1.0a43`** and **` pyarrow`**. Because streaming renderers lagged **tagged releases** when **0.1a0** shipped, pin **Docker images** to **`datasette:main`** only if you accept **moving targets**—otherwise wait for the **next alpha** Willison promised for **export dialog grouping**.

Watch **[Datasette plugins directory](https://datasette.io/plugins?ref=explainx)** for **composition**: combine **datasette-arrows** with **label plugins** for **foreign keys in Arrow exports** via `?_labels=on`.

## When this release is not news for you

If you do not run **Datasette**, skip this plugin. If you only publish **500-row tables**, CSV is fine. If you are on **Datasette** versions before **1.0a43**, upgrade **[stored-query patches](https://explainx.ai/blog/datasette-1-0a43-stored-query-advisory-streaming-2026)** before experimenting.

## Observability and abuse monitoring

Operators should log **`.arrows?_stream=1` hits** the same way they log **large CSV exports**. A sudden spike may mean **scraping**, not legitimate research. Combine **`max_stream_mb`** with **rate limits** at the reverse proxy when tables contain **PII**.

For **multitenant Datasette Cloud** (if you run one), partition **plugins per deployment** so **Arrow** cannot become a **covert exfil channel** from **restricted databases**.

## Roadmap signals from Willison's release cadence

Willison's post notes the **export dialog UI** will group **streaming formats** in the **next alpha** after **0.1a0**. That means early adopters may field **support tickets** from users who do not see **Include all rows** beside Arrow yet. Document **`?_stream=1` deep links** in your **data handbook** until the UI catches up.

Future plugins may add **Parquet stream**, **GeoArrow**, or **embedding-specific** layouts using the same hook—datasette-arrows is the **template**, not the **final catalog**.

## Ethics of publishing queryable public data

Arrow makes **re-identification attacks** faster when tables include **quasi-identifiers**. If you publish **census microdata** or **leaked corp tables**, review **[k-anonymity](https://en.wikipedia.org/wiki/K-anonymity?ref=explainx)** basics even when the format is modern. Streaming does not remove **duty of care**.

Journalists pairing Datasette with **[AI Overviews traffic loss stories](https://explainx.ai/blog/all-about-berlin-google-ai-overviews-traffic-loss-november-2026)** should remember: **machines read Arrow faster than humans read HTML**, changing **scraper economics** overnight.

## Troubleshooting common failures

**Empty stream after 200 OK:** Check whether you forgot **`?_stream=1`** on a multi-million-row table; the first response may be **only one page**.

**Connection reset mid-download:** Often **type coercion** failure on a later page. Re-run export with a SQL query that **casts messy columns** to text.

**Plugin not listed:** Confirm **`uv run datasette plugins`** shows datasette-arrows and that you are on a Datasette build with **streaming renderers** enabled.

**pyarrow ImportError:** Install pyarrow in the **client environment** reading the stream, not only on the server.

**Foreign keys missing labels:** Enable **`?_labels=on`** on the URL or verify the **labels plugin** chain you expect is active.

## Why Willison shipped Arrow before Parquet stream

Arrow IPC is **row-batch oriented** for **incremental HTTP**, matching Datasette's **page loop**. Parquet wants **column stripes** and is **heavier to stream** without custom footers. If Parquet streaming arrives later, expect **different memory tradeoffs**—Arrow remains the **low-latency wire format** for **Python analytics**.

## Ecosystem placement next to Lance and DuckDB

Teams exporting **SQLite snapshots** to **Lance** for **vector search** can treat **datasette-arrows** as **step one** (fast extract) before **offline convert**. Teams on **DuckDB** can **`ATTACH`** Arrow streams for **JOIN** against **CSV dimensions** without **double disk use**.

None of that is automatic—you still write **glue**—but you avoid **CSV parse tax** on **wide tables**.
