We got early access to a machine with an NVIDIA Vera CPU. This was a dual-socket ARM machine with 176 cores (352 threads) and 1536 GiB of RAM. NVIDIA calls it its two-threads-per-core design “Spatial Multithreading”. You can read about the full specs here.
Last week we released Polars 2.0 together with the announcement that we will treat SQL as first class citizen. We benchmarked Polars SQL against DuckDB and DataFusion with stunning results.
Now that we have access to the Vera CPU, we decided to run that benchmark again and see how we scale. To see how it compares we also ran the benchmark on an Intel Xeon 6 dual-socket machine on AWS (m8id.metal-96xl) with 192 cores (384 threads) and 1536 GiB of RAM.
Benchmark configuration #
We ran the benchmark on Polars 2.0, DuckDB 2.0 alpha (2.0.0.dev2610011535) and DataFusion 54.0.0 from Python. All tools were downloaded from the published PyPi releases. Similar to last time, every query ran 5 times in a hot setting, with a separate process per query and a 60 second timeout. The file cache was cleared between each engine/benchmark (not between queries). For every query we take the best of the 5 runs. We compare engines on both the sum and the geometric mean of those query times <sup>1</sup>.
The data and query generation was similar to the previous 2.0 benchmark. The raw benchmark results are posted in our benchmark repo: https://github.com/pola-rs/polars-2.0-benchmark.
Results #
Every engine ran twice on each machine: once with all threads (352 on Vera, 384 on the Xeon 6) and once limited to the number of cores of a single socket (88 on Vera, 96 on the Xeon 6). The charts below show the runtime of each engine on the TPC-H and TPC-DS<sup>2</sup> derived benchmarks at SF100 and SF1000 in seconds (lower is better), split by machine and ordered from fastest to slowest. Overall
NVIDIA Vera (176 cores, 352 threads, 1536 GiB)
Intel Xeon 6, m8id.metal-96xl (192 cores, 384 threads, 1536 GiB)
Polars and DuckDB completed all queries on both machines. Queries that DataFusion didn’t finish are counted as the 60 second timeout, so its totals are lower bounds (marked with ≥). With all threads, DataFusion didn’t finish TPC-DS Q72 at SF100 on either machine. At SF1000 it didn’t finish 1 TPC-H and 6 TPC-DS queries on Vera, and 10 TPC-H and 20 TPC-DS queries on the Xeon 6. Hover over a bar for the exact count.
Vera vs Xeon 6
The chart below shows the sum of query times of each engine on the Xeon 6 (open) and on Vera (filled), on a log scale. The label is how much faster the engine ran on Vera.
Footnotes #
One bug was spotted halfway through the process where a timeout on a single iteration would mark that entire query as a timeout. This only affected six queries for DataFusion on the Vera machine and three queries in Polars on the m8id.metal before it was spotted. Those three Polars queries were re-ran with the fixed script, but we could not re-run DataFusion on the Vera machine (as our preview access expired). ↩ 2. These benchmarks are derived from TPC-H and TPC-DS Benchmarks and as such any results obtained are not comparable to published TPC-H and TPC-DS Benchmark results, as the results obtained do not comply with the TPC-H and TPC-DS Benchmarks. ↩