HEY, FRIEND 👋 #
I hope you're doing well. I'm Simon, and I am happy to share another monthly newsletter with highlights and the latest updates about DuckDB, delivered straight to your inbox.
In this August issue, I gathered the usual 10 updates (+2 bonus) and news highlights from DuckDB's ecosystem. Please enjoy reading about giving agents their own databases, having a single database connection that grants access to over 30 external DBs, a sneak peek into the upcoming v2 async feature, and how DuckDB is used for graph analytics and as a data layer for multiple tools.
Kyle Cheung
Kyle is co-founder at Greybeam and the author of the DuckDB Internals series. Part 1 walked through the high-level design decisions that make DuckDB fast and went viral on Hacker News. He just released Part 2 on vectorized execution, which you will also find in the links below.
Thanks, Kyle, for these deep technical dives. In a world of short content, this is much appreciated.
Connect with Kyle on [LinkedIn](https://www.linkedin.com/in/cheung-kyle/).
[To Every Agent Its Own Database](https://joereis.substack.com/p/to-every-agent-its-own-database)
TL;DR: Joe proposes an agent-native architecture where each agent leverages its own embedded DuckDB instance for direct, peer-to-peer analytical data exchange, moving computation closer to the agents.
Joe explored an alternative to centralized data platforms for AI agents by giving each agent (peer) a dedicated, embedded DuckDB instance using Quack. Quack peers communicate directly via loopback TCP ports, exchanging "immutable analytical slices". Each slice is uniquely identified by a SliceRef
dataclass with catalog_id
, dataset
, DuckLake snapshot_id
, a contract_digest
of its Malloy-defined semantic model, and a slice_digest
. This approach decentralizes computation and analytical state, addressing concurrency and rapid chaining of results in agent workloads.
💡 One bonus article as related: Martin wrote about [Your coding agent keeps a diary](https://www.sfrt.io/your-coding-agent-keeps-a-diary/), with a separate DuckDB for AI with dlt extraction.
[Announcing the DuckDB ADBC Extension](https://columnar.tech/blog/announcing-duckdb-adbc-extension/)
TL;DR: The new DuckDB ADBC extension enables DuckDB to connect directly to over 30 external databases via the Arrow Database Connectivity (ADBC) API, leveraging Apache Arrow for efficient data transfer.
Sam explained that this extension offers a unified interface to systems like Snowflake, Databricks, and PostgreSQL, moving beyond individual vendor extensions. It utilizes Apache Arrow for fast, zero-copy, columnar data transfers, bypassing the performance bottlenecks of row-based APIs. Users can execute queries directly with read_adbc('profile://mydb', 'SELECT * FROM games')
or use `ATTACH 'profile://mydb' AS mydb (TYPE adbc)`
for persistent connections supporting `SELECT`
, INSERT
, COPY
, and CTAS
. ATTACH
currently lacks predicate and projection pushdown, requiring read_adbc
for optimized large dataset operations, but this is a planned enhancement.
[Asynchronous I/O in DuckDB: Work, Thread, Work](https://duckdb.org/2026/07/31/asynchronous-io)
TL;DR: A big one for the upcoming DuckDB v2.0 (Fall 2026), introducing asynchronous I/O for Parquet and CSV files, significantly improving query performance by better utilizing remote storage bandwidth.
Pedro explains the implementation involves separate REGULAR
worker and ASYNC
I/O thread pools (up to 256 threads), coupled with a read-ahead queue and asynchronous memory governance to proactively fetch data and prevent out-of-memory issues. Benchmarks show dramatic speedups such as TPC-H Q6 SF100 on S3 with Parquet improved 3x (from 8.23s to 2.84s), or saturating network bandwidth, while CSV queries saw a nearly 20x speedup (from 878s to 45s), even under concurrent workloads and memory constraints.
Webduck: A self-hosted DuckDB-as-a-Service server with REST API and Web UI TL;DR: WebDuck is designed for web hosting providers who want to offer their customers a ready-to-use administration interface for DuckDB databases and data analytics out of the box, just like the database admin tools that come with any hosting package.
As the name says, Webduck provides a UI with an overview dashboard, projects, and browsing views. It implements a DuckDB storage engine with per-file reader-writer locks to ensure safe concurrent access, aligning with DuckDB's single-writer model. Mike, the creator, highlighted the off of blocking engine and auth calls to a threadpool via asyncio.to_thread
in api/db.py
and api/admin.py
.
duckdb-gql: An extension to run graph queries and algorithms using ISO GQL TL;DR: DuckGQL, a C++17 extension, integrates a growing ISO GQL subset into DuckDB, allowing graph pattern queries and mutations with DuckDB's native relational storage and execution engine, and provides an explicit CSR layer for graph algorithms.
DuckGQL leverages typed vertex and edge tables, supporting bulk import and GQL operations. The extension uses dynamically built, connection-local CSR projections for algorithms, automatically invalidated by mutations, and property indexes utilize native DuckDB ART indexes, extending DuckDB's capabilities for direct graph analytics.
Perspective 5.0.0 featuring DuckDB pushdown
TL;DR: Perspective, a visualization and analytics component for large and/or streaming datasets, ships a DuckDB Virtual Server in v5.0.0 that replaces its built-in query engine with pushdown SQL, so <perspective-viewer>
drives DuckDB directly instead of ingesting data.
It adds DuckDBHandler
, a VirtualServerHandler
implementation for both Python and browser duckdb-wasm
. ViewerConfig
options like group_by
, filters, sorts and expressions are translated into SQL, with results read back via Apache Arrow IPC. Because DuckDB owns the query layer, its aggregates and expression syntax surface in the UI rather than perspective-server
's: Andrew's screen recordings show 6mm rows from NYC Open Data and native time_bucket
used inside the Column Expression Editor to build time-series bins. NYC Open Data Demo, Browser demo.
DuckDB Internals: Why is DuckDB Fast? (Part 2 Vectorized Execution) TL;DR: Kyle follows up on his highly shared Part 1, highlighting that DuckDB's vectorized execution processes data in 2048-row batches, reducing function call overhead and improving CPU IPC over the row-at-a-time Volcano model.
Key elements include DataChunk
s and four Vector
types (Flat, Constant, Dictionary, Sequence), unified by a UnifiedVectorFormat
for streamlined operator logic and delayed materialization. Filters leverage selection vectors to avoid data copying. Kyle explains how DuckDB is using interpreted execution with precompiled functions, optimizes inner loops with validity masks and C++ compiler-driven SIMD. The transition to a push-based execution model enhances parallel processing, enabling independent pipelines for sources, operators, and sinks, crucial for UNION ALL
.
datafusion-ducklake-provider TL;DR: A Rust crate that enables Apache DataFusion to interact with DuckLake tables via various catalog backends, including SQLite, DuckDB-file, Postgres, and client-only DuckDB Quack transports.
The core integrates as a DataFusion TableProvider
and catalog provider, facilitating WRITE
, DELETE
, UPDATE
, and maintenance operations. Developers can leverage the DuckLakeSessionContext
to execute DuckDB-shaped SQL commands like ATTACH 'ducklake:sqlite:metadata.sqlite' AS lake (DATA_PATH 'data/')
for seamless DataFusion integration with DuckLake catalogs.
[Developer Dad Does Disney: Vacationing in the Personal Software Era](https://motherduck.com/blog/developer-dad-does-disney/)
TL;DR: Example of how to use MotherDuck's MCP, Dives, and Flights to build a personal Disney World ride planner, dynamically integrating live and historical data for optimal decision-making.
Alex detailed how remote MCP-enabled AI agents, like Claude, can orchestrate database operations and app construction. Data was fetched from live APIs using read_json
for parallel querying across park endpoints. Historical wait times were regularly refreshed via a MotherDuck Flight, the new agent-native data pipeline. The solutions provide a live ride-and-show planner and a live ride planner. Check out the blog above if you go there or want to see how these look and how to build your own for another open API you have access to.
Self-hosting DuckDB: the road to production TL;DR: Self-hosting DuckDB scales from local to cloud, but each step adds operational complexity, especially for concurrency and persistence.
Mehdi highlights a practical map for self-hosting DuckDB, detailing the increasing operational "boxes" at each stage. An initial local setup on NVMe is the fastest, but moving data to S3 introduces network bottlenecks and authorization overhead. Scaling compute to the cloud (e.g., EC2) can mitigate this, and distributed queries can be achieved via fan-out with Lambda functions for partial GROUP BY
and SUM/COUNT
. For concurrent readers, dedicated compute per user flattens tail latency, while DuckDB-WASM pushes compute to the browser. He mentions that many writers require either the new quack_serve
protocol for serialized writes or a DuckLake setup with a managed Postgres catalog for ACID transactions.
His core takeaway is to self-host, but don't accidentally build a database company.
[Beyond Copilots: We're Building a Data Stack Live with AI Agents](https://luma.com/data-stack-live)
**2026-08-18. h: 08:00. Online**
You've seen a hundred demos of AI writing SQL. The query was never the hard part. dltHub, MotherDuck, and Lightdash build the whole stack live in under an hour: one schema travels from AI-generated ingestion to dbt models and a governed semantic layer.
[Do AI Agents Need a Semantic Layer?](https://luma.com/mother-72zw)
**2026-08-26. h: 09:00. Online**
Half the industry says AI agents are useless without a semantic layer. The other half ships agents that run fine on a pile of markdown files. Jacob Matson and Alex Monahan ran the evals and share what plain-text context vs a Malloy semantic layer showed.
[Time Travel & Schema Evolution: DuckLake Handles Change](https://luma.com/mother-4vcr)
**2026-09-03. h: 09:00. Online**
Schemas drift, bad loads slip in, and someone always needs last Tuesday's numbers. Alex Monahan, co-author of DuckLake: The Definitive Guide, joins Hoyt Emerson (Early Signal) to walk through time travel, schema evolution, and the change feed live.
[dbt Summit](https://www.getdbt.com/dbt-summit)
**2026-09-15. h: 09:00. Las Vegas, NV**
The world's largest gathering of dbt users, 2,200+ data leaders shaping the future of analytics and AI. MotherDuck is sponsoring again, so hope to catch you there.
[Big Data London](https://www.bigdataldn.com/)
**2026-09-23. h: 09:00. Olympia, London, UK**
The UK's largest data, analytics, and AI conference at Olympia London: two days, 400+ sessions, and exhibitors from Snowflake and Databricks to early-stage startups. Come find us at booth P60.