datasette-arrows 0.1a0: Streaming Apache Arrow Exports in Datasette Simon Willison released datasette-arrows 0.1a0 on November 18, 2026, a reference plugin for Datasette's new streaming export hook that adds Apache Arrow IPC (.arrows) downloads from tables, views and SQL queries without loading an entire SQLite file into RAM. The plugin, installable via `datasette install datasette-arrows`, requires datasette>=1.0a43 and pyarrow, serves the application/vnd.apache.arrow.stream MIME type, and streams one record batch per page when `?_stream=1` is set, with `allow_stream` and `max_stream_mb` settings applying the same guardrails as CSV streaming. Willison wrote that he "never releases a new plugin feature without a plugin that uses it," and noted that Claude Fable 5.2 suggested Arrow streaming and implemented most of the package in one shot. datasette-arrows 0.1a0 shipped on November 18, 2026 as a reference plugin for Datasette's new streaming export hook . It adds Apache Arrow IPC .arrows downloads from tables, views, and SQL queries without loading an entire SQLite file into RAM. Sources: Simon Willison's release note https://simonwillison.net/2026/Nov/18/datasette-arrows/?ref=explainx , PyPI https://pypi.org/project/datasette-arrows/?ref=explainx , and the datasette/datasette-arrows README https://github.com/datasette/datasette-arrows?ref=explainx . Willison wrote that he "never releases a new plugin feature without a plugin that uses it." Here, Claude Fable 5.2 suggested Arrow streaming and implemented most of the package in one shot —a useful data point next to Datasette 1.0a43's streaming CSV work https://explainx.ai/blog/datasette-1-0a43-stored-query-advisory-streaming-2026 we covered earlier in November. TL;DR | Piece | Detail | |---|---| | Version | 0.1a0 on PyPI, 2026-11-18 | | Install | datasette install datasette-arrows | | URLs | /db/table.arrows , /db/-/query.arrows?sql=... | | Stream all rows | ? stream=1 or Export dialog checkbox | | MIME | application/vnd.apache.arrow.stream | | Python deps | datasette =1.0a43 , pyarrow | | License | Apache-2.0 | Why Arrow after CSV and TSV streaming Datasette export dialog showing the new Apache Arrow streaming option Image: Simon Willison https://simonwillison.net/2026/Nov/18/datasette-arrows/?ref=explainx Datasette already streamed CSV and TSV because those formats chunk cleanly . Arrow IPC adds typed columns and zero-copy paths into pandas, Polars, and PyArrow workflows—closer to what LanceDB multimodal pipelines https://explainx.ai/blog/lancedb-realtime-engine-memwal-52ms-november-2026 want than string parsing . For publishers exposing public-interest SQLite —government logs, leaks, research drops—Arrow means a data journalist can read all in Python while a million-row table still streams from the server . URL patterns and streaming semantics Adapter pipes meeting through a junction evoking typed columnar export streams beside CSV From the README: - Paginated default: /database/table.arrows returns one page like HTML. - Full stream: /database/table.arrows? stream=1 emits one record batch per page until complete. - SQL queries: /database/-/query.arrows?sql=select+ +from+table - Download header: ? dl=1 forces file download. - Governance: allow stream and max stream mb Datasette settings apply—same guardrails as CSV streaming. Filters, sort orders, and ? col= projections apply to exports, so Arrow is not a bypass around Datasette's SQL safety . Column typing rules that bite in production SQL query funnel into a typed table, evoking SQLite columns mapped to Arrow types SQLite allows mixed types ; Arrow requires one type per column . The plugin infers from declared schema plus first page values : | SQLite values seen | Arrow type | |---|---| | Integers | int64 | | Floats or int/float mix | float64 | | Text or text/number mix | string | | Blobs or mixed with blobs | binary | | All null, no declared type | string | If a later page introduces an incompatible value —text in an INTEGER column—the export drops the connection so clients never treat a truncated stream as complete. Fix with explicit SQL casts, e.g. select cast value as text as value from ... . That behavior is stricter than CSV , which happily emits mangled strings . For ML exporters, strictness prevents silent label corruption . Reading streams in Python The README documents the minimal consumer: python import urllib.request import pyarrow as pa with urllib.request.urlopen "http://localhost:8001/data/dogs.arrows? stream=1" as response: table = pa.ipc.open stream response .read all From there, zero-copy handoff to Polars , DuckDB , or training pipelines is standard. Compare to Willison's Pyodide Datasette experiment https://explainx.ai/blog/datasette-1-0a43-stored-query-advisory-streaming-2026 themes: SQLite in the browser for demos, Arrow on the wire for analytics scale . UI and release cadence caveats Willison's screenshot uses Datasette main , where streaming formats will soon sit beside CSV/TSV under "All rows in this table" in the export dialog. 0.1a0 shipped before that UI grouping landed in a numbered alpha , so early adopters may need main or the next alpha for the polished dialog. The plugin requires streaming output renderers —present on main , not necessarily on the latest tagged release you run in production. Pin versions explicitly in Docker images . How this fits explainx.ai's tool coverage datasette-arrows is small but exemplary : it shows how plugin hooks should ship with reference implementations . The same week brought Vercel Board https://explainx.ai/blog/vercel-board-agent-message-board-november-2026 for agent coordination and CodeRabbit Verify https://explainx.ai/blog/coderabbit-verify-agentic-qa-public-beta-november-2026 for runtime QA —different layers, same lesson: agents need contracts , not just models . For local AI , Arrow exports pair with AirLLM-style constrained inference https://explainx.ai/blog/airllm-run-70b-llm-4gb-gpu-inference-2026 when you are featurizing public tables on modest GPUs —download features , not giant CSV text . Security and publishing notes Streaming exports can exfiltrate large tables if allow stream is on for sensitive databases . Treat .arrows endpoints like CSV download permissions. Row-level auth plugins still apply; Arrow does not magically redact columns. If you expose Datasette on the public internet , keep stored-query advisory fixes https://explainx.ai/blog/datasette-1-0a43-stored-query-advisory-streaming-2026 patched— streaming does not fix SQL injection in saved queries. Try it in fifteen minutes 1. Install Datasette main or =1.0a43 with streaming renderers enabled. 2. datasette install datasette-arrows 3. datasette serve data.db 4. Open a table, confirm Arrow in Export data . 5. curl -o out.arrows 'http://localhost:8001/db/table.arrows? stream=1' 6. Load with pyarrow and compare row count to select count . Open an issue on datasette/datasette-arrows if type coercion fails on real-world messy columns—that feedback shapes 1.0 . Hook mechanics: register output renderer Datasette 1.0a43 extended plugins so authors can register streaming output renderers alongside built-in CSV and TSV. The core idea: fetch a page of rows , serialize a chunk , repeat —never materialize the whole table in the Datasette process. datasette-arrows is the reference implementation for binary columnar output instead of delimited text . Plugin authors should copy its patterns: - Honor ? stream=1 and dialog Include all rows . - Respect allow stream and max stream mb caps. - Fail hard on type conflicts rather than emit silent corruption . If you maintain custom SQLite packs for newsrooms or NGOs, shipping Arrow lowers the bar for partner universities that already standardize on PyArrow . Integration sketches beyond pyarrow Polars can read Arrow streams with pl.read ipc stream . DuckDB can attach Arrow streams for SQL transforms before Parquet archival. Training stacks that expect HF datasets can still ingest Arrow, convert once to sharded Parquet , and push to LanceDB https://explainx.ai/blog/lancedb-realtime-engine-memwal-52ms-november-2026 if you later need vector search . For agents , a coding agent that knows Arrow schemas can validate column dtypes before fine-tune —fewer silent NaN bugs than CSV guesswork . Comparison to CSV for million-row tables CSV wins on universal tool support and human grep . Arrow wins when: - Columns are numeric-heavy counts, lat/lon, embeddings serialized as lists . - You need stable dtypes across pages . - You pipeline into Python ML without parse overhead . Benchmark on your table: time table.csv? stream=1 versus table.arrows? stream=1 into Polars on the same laptop. Arrow should dominate when columns 20 or rows 1M . Maintainer and CI notes The GitHub repo ships uv run pytest . Depend on datasette =1.0a43 and pyarrow . Because streaming renderers lagged tagged releases when 0.1a0 shipped, pin Docker images to datasette:main only if you accept moving targets —otherwise wait for the next alpha Willison promised for export dialog grouping . Watch Datasette plugins directory https://datasette.io/plugins?ref=explainx for composition : combine datasette-arrows with label plugins for foreign keys in Arrow exports via ? labels=on . When this release is not news for you If you do not run Datasette , skip this plugin. If you only publish 500-row tables , CSV is fine. If you are on Datasette versions before 1.0a43 , upgrade stored-query patches https://explainx.ai/blog/datasette-1-0a43-stored-query-advisory-streaming-2026 before experimenting. Observability and abuse monitoring Operators should log .arrows? stream=1 hits the same way they log large CSV exports . A sudden spike may mean scraping , not legitimate research. Combine max stream mb with rate limits at the reverse proxy when tables contain PII . For multitenant Datasette Cloud if you run one , partition plugins per deployment so Arrow cannot become a covert exfil channel from restricted databases . Roadmap signals from Willison's release cadence Willison's post notes the export dialog UI will group streaming formats in the next alpha after 0.1a0 . That means early adopters may field support tickets from users who do not see Include all rows beside Arrow yet. Document ? stream=1 deep links in your data handbook until the UI catches up. Future plugins may add Parquet stream , GeoArrow , or embedding-specific layouts using the same hook—datasette-arrows is the template , not the final catalog . Ethics of publishing queryable public data Arrow makes re-identification attacks faster when tables include quasi-identifiers . If you publish census microdata or leaked corp tables , review k-anonymity https://en.wikipedia.org/wiki/K-anonymity?ref=explainx basics even when the format is modern. Streaming does not remove duty of care . Journalists pairing Datasette with AI Overviews traffic loss stories https://explainx.ai/blog/all-about-berlin-google-ai-overviews-traffic-loss-november-2026 should remember: machines read Arrow faster than humans read HTML , changing scraper economics overnight. Troubleshooting common failures Empty stream after 200 OK: Check whether you forgot ? stream=1 on a multi-million-row table; the first response may be only one page . Connection reset mid-download: Often type coercion failure on a later page. Re-run export with a SQL query that casts messy columns to text. Plugin not listed: Confirm uv run datasette plugins shows datasette-arrows and that you are on a Datasette build with streaming renderers enabled. pyarrow ImportError: Install pyarrow in the client environment reading the stream, not only on the server. Foreign keys missing labels: Enable ? labels=on on the URL or verify the labels plugin chain you expect is active. Why Willison shipped Arrow before Parquet stream Arrow IPC is row-batch oriented for incremental HTTP , matching Datasette's page loop . Parquet wants column stripes and is heavier to stream without custom footers. If Parquet streaming arrives later, expect different memory tradeoffs —Arrow remains the low-latency wire format for Python analytics . Ecosystem placement next to Lance and DuckDB Teams exporting SQLite snapshots to Lance for vector search can treat datasette-arrows as step one fast extract before offline convert . Teams on DuckDB can ATTACH Arrow streams for JOIN against CSV dimensions without double disk use . None of that is automatic—you still write glue —but you avoid CSV parse tax on wide tables .