Polars vs Pandas 2.0 vs DuckDB: Arrow In-Memory Computation & Multi-Threaded DataFrames A developer benchmarked Pandas 2.0, Polars 1.x, and DuckDB 1.x on identical AWS c7i.8xlarge hardware against the H2O.ai groupby/join suite (12 GB CSV, ~300M rows) and TPC-H SF=10, finding Polars fastest on DataFrame operations (5.1 s groupby vs 48.2 s for Pandas, 3.9 s for a 50M-row join) while DuckDB led the 22-query TPC-H run at 38 s versus Polars' 46 s and Pandas' 217 s. The writeup attributes Pandas' gap to its single-threaded, eager execution model with no query optimizer, and recommends keeping it only as an interoperability layer for the roughly 450k libraries that consume pd.DataFrame. All three engines now speak Apache Arrow, but they occupy fundamentally different points in the execution-space: Pandas 2.0 is a single-threaded, block-manager-backed library with an optional Arrow backend; Polars is a Rust-native, multi-threaded OLAP DataFrame engine with lazy query optimization; DuckDB is an in-process analytical SQL database with a vectorized, morsel-driven executor. The benchmarks below were run on identical hardware AWS c7i.8xlarge, 32 vCPU Intel Sapphire Rapids, 64 GB DDR5-4800, gp3 volume at 16,000 baseline IOPS against the H2O.ai groupby/join benchmark suite 12 GB CSV, ~300M rows plus a custom TPC-H SF=10 run. | Metric c7i.8xlarge, 12 GB dataset | Pandas 2.0 Arrow backend | Polars 1.x lazy, streaming | DuckDB 1.x in-memory | |---|---|---|---| | Groupby-agg wall time | 48.2 s | 5.1 s | 6.8 s | | 50M-row join int64 key | 31.4 s | 3.9 s | 4.4 s | | Peak RSS during groupby | 41 GB | 19 GB | 22 GB | | Threads saturated at p99 | 1 | 32 | 32 morsel-driven | | CSV read throughput | 0.31 GB/s | 1.9 GB/s | 1.6 GB/s | | Parquet read row groups, no projection | 0.8 GB/s | 2.4 GB/s | 2.2 GB/s | | TPC-H SF=10 total 22 queries, warm | 217 s | 46 s | 38 s | | GC/alloc jitter p99 − p50 latency, per query | 380 ms | 9 ms | 14 ms | | Memory overhead over raw Arrow data | 1.9–3.1× | 1.15× | 1.05× buffer-managed | | Ecosystem maturity PyPI deps reverse count | 1 baseline, ~450k dependents | ~2,400 | ~8,900 | | Learning curve SQL-native teams | High its own API | Medium expressions API | Low ANSI SQL | Decision Matrix in One Sentence: Use Polars if your workload is Python-native DataFrame transformations on datasets that fit in RAM at 1.2× raw size; use DuckDB if you can express the logic in SQL, need out-of-core spill-to-disk, or query Parquet files directly; keep Pandas 2.0 only as the interoperability lingua franca for the 450k libraries that consume pd.DataFrame — never as the compute engine above ~2 GB. Pandas 2.x introduced dtype backend="pyarrow" , which changes column storage from NumPy ndarrays + object blocks to Arrow arrays. This fixes the two worst historical sins — object-dtype strings 8-byte pointer + heap-allocated Python str per value and the copy-on-... well, lack thereof — but the execution model is untouched: single-threaded, eager, row-at-a-time-in-batches vectorization via NumPy/Arrow kernels . There is no query optimizer, no predicate pushdown, no join reordering. Every operation materializes an intermediate. A 5-step pipeline on 10 GB of data allocates and frees roughly 40 GB of cumulative intermediates. The GIL is irrelevant here Arrow kernels release it , but there is simply no internal parallelism. Adding parallel pandas or Dask on top reintroduces serialization overhead that Polars avoids with shared-memory Arrow chunks and a work-stealing scheduler Tokio-backed . Polars parses expressions into a logical plan, runs rule-based optimization predicate pushdown, projection pruning, join reordering via CBO estimates, common subplan elimination , then executes on a rayon thread pool — one thread per core by default, each processing morsels of ~50k rows. The streaming engine default since 1.x fragments the physical plan into stages that process data in bounded-memory batches, letting you exceed RAM at reduced throughput. python import polars as pl Lazy plan: predicate/projection pushdown happens at .collect , only 3 columns and matched rows ever leave the scan layer. lf = pl.scan parquet "s3://bucket/events/ .parquet" .filter pl.col "ts" = pl.datetime 2026, 1, 1 .group by "user id", "region" .agg pl.col "latency ms" .quantile 0.99 .alias "p99" , pl.col "bytes" .sum , .sort "p99", descending=True result = lf.collect engine="streaming" bounded RSS ~ 1.2x batch size print lf.explain optimized=True inspect pushed-down predicates Critical caveat: the streaming engine historically did not support every operation e.g., certain window functions and sorted-key joins fall back to non-streaming, spiking RSS . Audit explain optimized=True output; "STREAMING" tags on every node are the contract. DuckDB is a genuine database: MVCC, ACID, a persistent single-file storage format, a buffer manager that pages cold blocks to disk true out-of-core execution, not just streaming , and a morsel-driven parallelism model where worker threads claim fixed-size morsels ~120k tuples of pipeline-local work. It ingests Arrow zero-copy via duckdb.arrow / register , so Arrow data parked in Polars or pandas can be queried without serialization. -- Query Parquet directly from S3; only projected columns + pushed -- predicates cross the network. Requires httpfs extension. INSTALL httpfs; LOAD httpfs; SET s3 region='us-east-1'; SET enable object cache = true; SELECT user id, region, quantile cont latency ms, 0.99 AS p99, sum bytes AS total bytes FROM read parquet 's3://bucket/events/ .parquet', hive partitioning=true WHERE ts = TIMESTAMP '2026-01-01' GROUP BY user id, region ORDER BY p99 DESC LIMIT 100; flowchart LR subgraph Pandas "Pandas 2.0" A1 Python Interpreter