Profile your Polars queries

Polars released its query profiler for open source users for free, enabling profiling via a single line, `pl.Config.enable_monitoring()`, after installing `polars_cloud`. The profiler hooks into the streaming engine, sending the optimized logical and physical plans plus per-node progress telemetry every 5 seconds to https://cloud.pola.rs/portal/, while query data stays in the user's environment; Polars says an MCP for agents is coming soon.

TL;DR; Add pl.Config.enable monitoring to your python environment to enable profiling of your query allowing you to view your query’s progress live and find & optimize bottlenecks. Our query profiler is now available for open source Polars users for free<sup 1 user-content-fn-1 </sup . This enables users to debug & profile queries running on their own local infrastructure using an intuitive interface and soon an MCP for your agents. The profiler gets you the most detailed information of what Polars during execution. You get a glimpse under the hood of the query engine. You can see rows flowing through the query, see how many rows are filtered which join produces most data and much more. With this information you can point out bottle necks in your queries, recommend optimizations using an agent and act as a live progress indicator for long running queries. The query is executed locally and sends live telemetry data to the platform. This functionality is available for the streaming and distributed engine. Getting started Using the query profiler is a one line change in your data pipelines. First pip install polars cloud in your python environment. This enables the functionality to send telemetry data to the platform. Second, enable the flag pl.Config.enable monitoring in the root of your script. Try it out today using the script below: python from datetime import date import polars as pl pl.Config.enable monitoring BASE = "s3://polars-public-datasets/tpch/sf1" OPTS = {"aws skip signature": "true", "aws region": "eu-west-1"} lineitem = pl.scan parquet f"{BASE}/lineitem/ .parquet", storage options=OPTS orders = pl.scan parquet f"{BASE}/orders/ .parquet", storage options=OPTS late = lineitem.filter pl.col "l commitdate" < pl.col "l receiptdate" result = orders.join late, left on="o orderkey", right on="l orderkey", how="semi" .filter pl.col "o orderdate" .is between date 1993, 7, 1 , date 1993, 10, 1 , closed="left" .group by "o orderpriority" .agg pl.len .alias "order count" .sort "o orderpriority" .collect How it works The profiler hooks into the streaming engine and sends back telemetry data at key points: - After query planning & optimization the optimized plan logical & physical is sent - During query execution the progress of each node is sent at fixed intervals currently every 5s - Once the query is finished a final flush is done to ensure accurate metrics and timings Security & Privacy Only the query plan is shared with the platform, the data never leaves your environment. Analyzing your Query If you are in charge of running data pipelines, the query profiler is a great tool for analyzing and optimizing historical queries. It allows you to observe where time is spent, identify bottlenecks and optimize your queries. During a query or for historical queries you can view the progress at https://cloud.pola.rs/portal/ https://cloud.pola.rs/portal/ under queries. Clicking on the query shows the high level details and both plans. The logical plan contains the query plan after optimizations and the physical plan contains the detailed execution nodes with performance metrics included. In the query above we can see that 24.9 MiB + 15 MiB ~ 40 MiB were loaded from S3 and the majority of the CPU time was spent on the join. This query was bound by I/O speed as the CPU time of the join was low ~103ms compared to total query time ~5s indicating the join was waiting on data loaded from S3. For more details on how to use the query profiler go to our user guide https://docs.cloud.pola.rs/polars-cloud/run/query-profile . Try it out Run your queries and let us know your experience by commenting on our Discord https://discord.gg/4UfP5cfBE7 or adding feature requests to our issue tracker https://github.com/pola-rs/polars/issues . Up Next We are already working on the next big release for the query profiler. In the upcoming weeks you can expect the release of our MCP server which allows your agents to profile queries. Additionally we are adding node specific metrics to our plans to increase the capabilities to analyze your queries. Footnotes 1. Usage is limited to fair use to prevent excessive platform use. ↩ user-content-fnref-1