TL;DR: Distributed Polars now runs on your own Ray cluster. Install polars-nanook-ray on your Ray nodes and your existing Polars queries run across Ray actors, while the engine takes care of partitioning, shuffles and scaling. Get started at https://nanook.pola.rs/portal.
Ray is an open-source framework for scaling Python over a cluster of machines, and has become a common foundation for machine learning platforms: training, hyperparameter tuning, serving and batch inference all run on the same Ray cluster. Before any model sees data, that data has to be prepared. Cleaning, joining and aggregating raw tables into features is where many teams use Polars for its performance and ergonomics.
As datasets grow, that preprocessing step outgrows a single machine, while the Ray cluster next to it has plenty of capacity. With polars-nanook-ray your Polars preprocessing can use that capacity. It runs the distributed engine behind Polars Nanook directly inside the Ray cluster you already operate.
Why not just @ray.remote? #
You can run open-source Polars on Ray today: split your data into partitions, wrap your Polars logic in @ray.remote tasks and fan them out over the cluster. For a simple map over files this works fine.
It stops working as soon as your query needs data from more than one partition. A join or a group_by needs all rows with the same key in the same place, so now you are writing the shuffle yourself: choosing a partitioning scheme, re-partitioning between stages, and wiring tasks together. You end up maintaining a small, hand-written distributed query engine next to your actual pipeline. Aside from the maintenance burden, you also miss out on many distributed optimizations Polars Nanook does.
We think that should be the engine’s job, so with polars-nanook-ray you write a single Polars query and the engine plans it, partitions the data and shuffles between stages on its own. Ray does what it is good at: placing the Polars scheduler and workers as actors on your nodes and keeping them alive.
Quickstart #
Already have a Ray cluster running? Install the package in the environment of every Ray node, with the same ray version as your cluster:
uv venv
source .venv/bin/activate
uv pip install polars-nanook-ray ray==<X.X.X>
Next, create an account at https://nanook.pola.rs/portal and export your service account credentials. The snippet below starts a cluster with two workers of 4 CPUs and 16 GiB of memory each:
import polars as pl
import ray
from polars_nanook_ray import RayClusterContext, ServiceAccount
ray.init(address="<RAY_CLUSTER_URI>")
ctx = RayClusterContext(
license=ServiceAccount(
workspace_id="<WORKSPACE_ID>",
client_id="<SERVICE_ACCOUNT_ID>",
client_secret="<SERVICE_ACCOUNT_SECRET>",
),
workers=2,
cpus=4,
memory=16,
)
ctx.start()
The context is also what you send queries to. Take a regular Polars query, here on public TPC-H data, and instead of collecting it locally, call .remote(ctx):
from datetime import date
BASE = "s3://polars-public-datasets/tpch/sf1000"
lineitem = pl.scan_parquet(f"{BASE}/lineitem/*.parquet")
orders = pl.scan_parquet(f"{BASE}/orders/*.parquet")
result = (
orders.join(lineitem, left_on="o_orderkey", right_on="l_orderkey")
.filter(pl.col("o_orderdate") >= date(1995, 1, 1))
.group_by("o_orderpriority")
.agg(revenue=pl.col("l_extendedprice").sum())
.sort("o_orderpriority")
.remote(ctx)
.execute()
)
print(result.head)
That join and group_by is the part you would otherwise shuffle by hand. When you are done, stop the cluster:
ctx.stop()
ray.shutdown()
Why run Polars on Ray #
No partitioning code
The query above is the same query you would run on your laptop. It doesn’t change when you go from 2 workers to 20, and there is no partitioning scheme or task graph to keep in sync with it. The distributed engine is built on our open-source streaming engine, so the semantics, from null handling to ordering, are the same as the Polars you already know.
Scale to zero
A Polars cluster doesn’t need to run at full size all day. With autoscale, you can keep a tiny cluster around, down to just the scheduler and zero workers, and let it scale up when queries come in:
from polars_nanook_ray import Autoscale
ctx = RayClusterContext(
license=...,
workers=0,
autoscale=Autoscale(min_workers=0, max_workers=16),
)
ctx.start()
A query asks for the workers it needs, and the cluster starts them:
query.remote(ctx).distributed(min_workers=8).execute()
Polars decides how many workers a query needs and Ray decides where to place them, and if your Ray cluster autoscales too, Ray adds the nodes these new workers need.
See every query
Aside from managing the distribution of your Polars code, Polars Nanook also ships with a dashboard with query insights that are tailored to Polars.
Distributed engines are often a black box until the query finishes, which is why every cluster ships with a dashboard, at ctx.dashboard_addr, that shows each operation of a running query, how many rows flow through it and how much data is shuffled between workers. All queries also show up in your workspace at https://nanook.pola.rs/portal, so you can still profile a pipeline run after the cluster is gone.
Fits the Ray cluster you run
There is no second cluster manager to operate, because the scheduler and workers are ordinary Ray actors: Ray places them, spreads them over your nodes, and restarts them when a node is lost. They show up in the Ray dashboard next to the rest of your workloads.
Try it out! #
Sign up for Polars Nanook, go to Infrastructure and select Ray to get started. You can also visit our user guide or the GitHub repo. We are happy to hear your feedback.