cd /news/ai-infrastructure/distributed-polars-on-ray · home › topics › ai-infrastructure › article
[ARTICLE · art-171860] src=pola.rs ↗ pub= topic=ai-infrastructure verified=true sentiment=↑ positive

Distributed Polars on Ray

Polars released polars-nanook-ray, a package that runs the distributed engine behind Polars Nanook inside an existing Ray cluster, letting standard Polars queries execute across Ray actors with the engine handling partitioning and shuffles. Users install polars-nanook-ray on every Ray node with a matching ray version, authenticate with a Nanook service account, and call .remote(ctx) on a query instead of collecting locally; the example starts a cluster with 2 workers of 4 CPUs and 16 GiB of memory each. The package targets preprocessing joins and group_bys that outgrow a single machine, replacing hand-written shuffle code built on @ray.remote tasks.

key takeaway

Install polars-nanook-ray with a ray version matching the cluster, then call .remote(ctx) on an existing Polars query to run joins and group_bys across Ray actors without writing shuffle code yourself.

by read4 min views2 publishedNov 18, 2026
Distributed Polars on Ray
Image: Pola (auto-discovered)

TL;DR: Distributed Polars now runs on your own Ray cluster. Install polars-nanook-ray on your Ray nodes and your existing Polars queries run across Ray actors, while the engine takes care of partitioning, shuffles and scaling. Get started at https://nanook.pola.rs/portal.

Ray is an open-source framework for scaling Python over a cluster of machines, and has become a common foundation for machine learning platforms: training, hyperparameter tuning, serving and batch inference all run on the same Ray cluster. Before any model sees data, that data has to be prepared. Cleaning, joining and aggregating raw tables into features is where many teams use Polars for its performance and ergonomics.

As datasets grow, that preprocessing step outgrows a single machine, while the Ray cluster next to it has plenty of capacity. With polars-nanook-ray your Polars preprocessing can use that capacity. It runs the distributed engine behind Polars Nanook directly inside the Ray cluster you already operate.

Why not just @ray.remote? #

You can run open-source Polars on Ray today: split your data into partitions, wrap your Polars logic in @ray.remote tasks and fan them out over the cluster. For a simple map over files this works fine.

It stops working as soon as your query needs data from more than one partition. A join or a group_by needs all rows with the same key in the same place, so now you are writing the shuffle yourself: choosing a partitioning scheme, re-partitioning between stages, and wiring tasks together. You end up maintaining a small, hand-written distributed query engine next to your actual pipeline. Aside from the maintenance burden, you also miss out on many distributed optimizations Polars Nanook does.

We think that should be the engine’s job, so with polars-nanook-ray you write a single Polars query and the engine plans it, partitions the data and shuffles between stages on its own. Ray does what it is good at: placing the Polars scheduler and workers as actors on your nodes and keeping them alive.

Quickstart #

Already have a Ray cluster running? Install the package in the environment of every Ray node, with the same ray version as your cluster:

uv venv
source .venv/bin/activate
uv pip install polars-nanook-ray ray==<X.X.X>

Next, create an account at https://nanook.pola.rs/portal and export your service account credentials. The snippet below starts a cluster with two workers of 4 CPUs and 16 GiB of memory each:

import polars as pl
import ray

from polars_nanook_ray import RayClusterContext, ServiceAccount

ray.init(address="<RAY_CLUSTER_URI>")

ctx = RayClusterContext(
    license=ServiceAccount(
        workspace_id="<WORKSPACE_ID>",
        client_id="<SERVICE_ACCOUNT_ID>",
        client_secret="<SERVICE_ACCOUNT_SECRET>",
    ),
    workers=2,
    cpus=4,  
    memory=16,  
)
ctx.start()

The context is also what you send queries to. Take a regular Polars query, here on public TPC-H data, and instead of collecting it locally, call .remote(ctx):

from datetime import date

BASE = "s3://polars-public-datasets/tpch/sf1000"
lineitem = pl.scan_parquet(f"{BASE}/lineitem/*.parquet")
orders = pl.scan_parquet(f"{BASE}/orders/*.parquet")

result = (
    orders.join(lineitem, left_on="o_orderkey", right_on="l_orderkey")
    .filter(pl.col("o_orderdate") >= date(1995, 1, 1))
    .group_by("o_orderpriority")
    .agg(revenue=pl.col("l_extendedprice").sum())
    .sort("o_orderpriority")
    .remote(ctx)
    .execute()
)
print(result.head)

That join and group_by is the part you would otherwise shuffle by hand. When you are done, stop the cluster:

ctx.stop()
ray.shutdown()

Why run Polars on Ray #

No partitioning code

The query above is the same query you would run on your laptop. It doesn’t change when you go from 2 workers to 20, and there is no partitioning scheme or task graph to keep in sync with it. The distributed engine is built on our open-source streaming engine, so the semantics, from null handling to ordering, are the same as the Polars you already know.

Scale to zero

A Polars cluster doesn’t need to run at full size all day. With autoscale, you can keep a tiny cluster around, down to just the scheduler and zero workers, and let it scale up when queries come in:

from polars_nanook_ray import Autoscale

ctx = RayClusterContext(
    license=...,
    workers=0,
    autoscale=Autoscale(min_workers=0, max_workers=16),
)
ctx.start()

A query asks for the workers it needs, and the cluster starts them:

query.remote(ctx).distributed(min_workers=8).execute()

Polars decides how many workers a query needs and Ray decides where to place them, and if your Ray cluster autoscales too, Ray adds the nodes these new workers need.

See every query

Aside from managing the distribution of your Polars code, Polars Nanook also ships with a dashboard with query insights that are tailored to Polars.

Distributed engines are often a black box until the query finishes, which is why every cluster ships with a dashboard, at ctx.dashboard_addr, that shows each operation of a running query, how many rows flow through it and how much data is shuffled between workers. All queries also show up in your workspace at https://nanook.pola.rs/portal, so you can still profile a pipeline run after the cluster is gone.

Fits the Ray cluster you run

There is no second cluster manager to operate, because the scheduler and workers are ordinary Ray actors: Ray places them, spreads them over your nodes, and restarts them when a node is lost. They show up in the Ray dashboard next to the rest of your workloads.

Try it out! #

Sign up for Polars Nanook, go to Infrastructure and select Ray to get started. You can also visit our user guide or the GitHub repo. We are happy to hear your feedback.

── more in #ai-infrastructure 4 stories · sorted by recency
── more on @polars 3 stories trending now
sponsored brought to you by zahid.host 4,200+ EU-deployed projects
reading about agents? ship yours in a single git push.

Run your AI side-project on zahid.host

EU-based hosting, git-push deploys, automatic HTTPS, no cold starts. Free tier with a custom domain — perfect for shipping the agent you just read about.

$git push zahid main
→ Live at https://your-agent.zahid.host ✓
Get free account → Pricing
from €0/mo · no card required
LIVE [news/distributed-polars-o…] indexed:0 read:4min 2026-11-18 · —