{"slug": "distributed-polars-on-ray", "title": "Distributed Polars on Ray", "summary": "Polars released polars-nanook-ray, a package that runs the distributed engine behind Polars Nanook inside an existing Ray cluster, letting standard Polars queries execute across Ray actors with the engine handling partitioning and shuffles. Users install polars-nanook-ray on every Ray node with a matching ray version, authenticate with a Nanook service account, and call .remote(ctx) on a query instead of collecting locally; the example starts a cluster with 2 workers of 4 CPUs and 16 GiB of memory each. The package targets preprocessing joins and group_bys that outgrow a single machine, replacing hand-written shuffle code built on @ray.remote tasks.", "body_md": "TL;DR: Distributed Polars now runs on your own Ray cluster. Install `polars-nanook-ray` on your Ray nodes and your existing Polars queries run across Ray actors, while the engine takes care of partitioning, shuffles and scaling. Get started at [https://nanook.pola.rs/portal](https://nanook.pola.rs/portal).\n\n[Ray](https://www.ray.io/) is an open-source framework for scaling Python over a cluster of machines, and has become a common foundation for machine learning platforms: training, hyperparameter tuning, serving and batch inference all run on the same Ray cluster. Before any model sees data, that data has to be prepared. Cleaning, joining and aggregating raw tables into features is where many teams use Polars for its performance and ergonomics.\n\nAs datasets grow, that preprocessing step outgrows a single machine, while the Ray cluster next to it has plenty of capacity. With [`polars-nanook-ray`](https://github.com/polars-inc/polars-nanook-ray) your Polars preprocessing can use that capacity. It runs the distributed engine behind [Polars Nanook](https://pola.rs/nanook/) directly inside the Ray cluster you already operate.\n\n## Why not just `@ray.remote`?\n\nYou can run open-source Polars on Ray today: split your data into partitions, wrap your Polars logic in `@ray.remote` tasks and fan them out over the cluster. For a simple `map` over files this works fine.\n\nIt stops working as soon as your query needs data from more than one partition. A join or a `group_by` needs all rows with the same key in the same place, so now you are writing the shuffle yourself: choosing a partitioning scheme, re-partitioning between stages, and wiring tasks together. You end up maintaining a small, hand-written distributed query engine next to your actual pipeline. Aside from the maintenance burden, you also miss out on many distributed optimizations Polars Nanook does.\n\nWe think that should be the engine’s job, so with `polars-nanook-ray` you write a single Polars query and the engine plans it, partitions the data and shuffles between stages on its own. Ray does what it is good at: placing the Polars scheduler and workers as actors on your nodes and keeping them alive.\n\n## Quickstart\n\nAlready have a Ray cluster running? Install the package in the environment of every Ray node, with the same `ray` version as your cluster:\n\n```\nuv venv\nsource .venv/bin/activate\nuv pip install polars-nanook-ray ray==<X.X.X>\n```\n\nNext, create an account at [https://nanook.pola.rs/portal](https://nanook.pola.rs/portal) and export your service account credentials. The snippet below starts a cluster with two workers of 4 CPUs and 16 GiB of memory each:\n\n``` python\nimport polars as pl\nimport ray\n\nfrom polars_nanook_ray import RayClusterContext, ServiceAccount\n\nray.init(address=\"<RAY_CLUSTER_URI>\")\n\nctx = RayClusterContext(\n    license=ServiceAccount(\n        workspace_id=\"<WORKSPACE_ID>\",\n        client_id=\"<SERVICE_ACCOUNT_ID>\",\n        client_secret=\"<SERVICE_ACCOUNT_SECRET>\",\n    ),\n    workers=2,\n    cpus=4,  \n    memory=16,  \n)\nctx.start()\n```\n\nThe context is also what you send queries to. Take a regular Polars query, here on public TPC-H data, and instead of collecting it locally, call `.remote(ctx)`:\n\n``` python\nfrom datetime import date\n\nBASE = \"s3://polars-public-datasets/tpch/sf1000\"\nlineitem = pl.scan_parquet(f\"{BASE}/lineitem/*.parquet\")\norders = pl.scan_parquet(f\"{BASE}/orders/*.parquet\")\n\nresult = (\n    orders.join(lineitem, left_on=\"o_orderkey\", right_on=\"l_orderkey\")\n    .filter(pl.col(\"o_orderdate\") >= date(1995, 1, 1))\n    .group_by(\"o_orderpriority\")\n    .agg(revenue=pl.col(\"l_extendedprice\").sum())\n    .sort(\"o_orderpriority\")\n    .remote(ctx)\n    .execute()\n)\nprint(result.head)\n```\n\nThat join and `group_by` is the part you would otherwise shuffle by hand. When you are done, stop the cluster:\n\n```\nctx.stop()\nray.shutdown()\n```\n\n## Why run Polars on Ray\n\n### No partitioning code\n\nThe query above is the same query you would run on your laptop. It doesn’t change when you go from 2 workers to 20, and there is no partitioning scheme or task graph to keep in sync with it. The distributed engine is built on our open-source streaming engine, so the semantics, from null handling to ordering, are the same as the Polars you already know.\n\n### Scale to zero\n\nA Polars cluster doesn’t need to run at full size all day. With `autoscale`, you can keep a tiny cluster around, down to just the scheduler and zero workers, and let it scale up when queries come in:\n\n``` python\nfrom polars_nanook_ray import Autoscale\n\nctx = RayClusterContext(\n    license=...,\n    workers=0,\n    autoscale=Autoscale(min_workers=0, max_workers=16),\n)\nctx.start()\n```\n\nA query asks for the workers it needs, and the cluster starts them:\n\n```\nquery.remote(ctx).distributed(min_workers=8).execute()\n```\n\nPolars decides how many workers a query needs and Ray decides where to place them, and if your Ray cluster autoscales too, Ray adds the nodes these new workers need.\n\n### See every query\n\nAside from managing the distribution of your Polars code, Polars Nanook also ships with a dashboard with query insights that are tailored to Polars.\n\nDistributed engines are often a black box until the query finishes, which is why every cluster ships with a dashboard, at `ctx.dashboard_addr`, that shows each operation of a running query, how many rows flow through it and how much data is shuffled between workers. All queries also show up in your workspace at [https://nanook.pola.rs/portal](https://nanook.pola.rs/portal), so you can still profile a pipeline run after the cluster is gone.\n\n### Fits the Ray cluster you run\n\nThere is no second cluster manager to operate, because the scheduler and workers are ordinary Ray actors: Ray places them, spreads them over your nodes, and restarts them when a node is lost. They show up in the Ray dashboard next to the rest of your workloads.\n\n## Try it out!\n\n[Sign up for Polars Nanook](https://nanook.pola.rs/api/redirects/register), go to **Infrastructure** and select **Ray** to get started. You can also visit our [user guide](https://docs.nanook.pola.rs/deploy/ray/) or the [GitHub repo](https://github.com/polars-inc/polars-nanook-ray). We are happy to hear your feedback.", "url": "https://wpnews.pro/news/distributed-polars-on-ray", "canonical_source": "https://pola.rs/posts/ray-actors/", "published_at": "2026-11-18 00:00:00+00:00", "updated_at": "2026-11-18 15:15:45.615769+00:00", "lang": "en", "topics": ["ai-infrastructure", "developer-tools", "machine-learning", "mlops"], "entities": ["Polars", "Polars Nanook", "polars-nanook-ray", "Ray", "polars-inc"], "also_reported_by": [], "takeaway": "Install polars-nanook-ray with a ray version matching the cluster, then call .remote(ctx) on an existing Polars query to run joins and group_bys across Ray actors without writing shuffle code yourself.", "recommendation": null, "alternates": {"html": "https://wpnews.pro/news/distributed-polars-on-ray", "markdown": "https://wpnews.pro/news/distributed-polars-on-ray.md", "text": "https://wpnews.pro/news/distributed-polars-on-ray.txt", "jsonld": "https://wpnews.pro/news/distributed-polars-on-ray.jsonld"}}