Ray is easy to describe badly.
Call it a "distributed Python framework," and it sounds like a faster multiprocessing
. Call it an "AI platform," and it sounds like it should manage users, approvals, datasets, and model releases. Call it a "cluster manager," and people understandably ask why they still need Kubernetes.
None of those labels is sufficient by itself.
A more useful description is this:
Ray is a distributed compute layer for Python and AI workloads. It gives you low-level primitives for running Python across processes and machines, plus higher-level libraries for data processing, training, tuning, reinforcement learning, and serving.
That distinction matters. Once you understand which problems Ray solves—and which problems remain yours—the project becomes much easier to reason about.
This article builds that mental model from the smallest possible Ray program, then follows it through an end-to-end machine learning workflow. It closes by examining the boundary between Ray's compute runtime and the lifecycle contracts an application or platform may add above it.
Version scope: This article targets Ray 2.55.1, the compatibility baseline for the SDK case study at the end. API details and behavior can change, so use the versioned Ray documentation for the environment you deploy.
The task and actor examples use small in-memory inputs so you can run them locally. They adapt the task and actor patterns from the versioned Ray Core documentation to one small scoring example. The rest of the article focuses on how Ray's libraries and boundaries fit together rather than presenting a complete application.
Suppose you have several data partitions to score. A normal Python program might process them one at a time, with one process owning execution order, memory, errors, and return values.
Distributing the same work introduces a surprising number of questions:
Ray's value starts here. It lets your code describe work and resource requirements while the runtime handles scheduling, process management, and distributed object movement.
The smallest useful Ray vocabulary contains three nouns:
Here is the earlier loop as Ray tasks:
import ray
ray.init()
@ray.remote(num_cpus=1)
def score_partition(partition: list[int]) -> dict:
return {"count": len(partition), "sum": sum(partition)}
partitions = [[1, 2, 3], [4, 5], [6, 7, 8, 9]]
result_refs = [score_partition.remote(partition) for partition in partitions]
results = ray.get(result_refs)
The @ray.remote
decorator turns the function into a task definition. Calling .remote()
submits work asynchronously and immediately returns an ObjectRef
. Calling ray.get()
resolves the references and waits only when the concrete values are needed.
This is more important than the decorator makes it look. The driver no longer chooses a process or machine. It declares work, and Ray's scheduler places that work according to the resources available in the cluster.
There is also a common performance trap hidden in this example. If you call ray.get()
immediately inside the submission loop, you serialize independent work again. Submit independent calls first; synchronize later.
Tasks are a good fit for independent transformations. They are less convenient when initialization is expensive or state must survive across calls. A model server, database connection pool, or simulation environment often fits the actor model better:
import ray
ray.init()
@ray.remote(num_cpus=1)
class ModelWorker:
def __init__(self, bias: float = 0.0):
self.bias = bias
def predict(self, batch: list[float]) -> list[float]:
return [value + self.bias for value in batch]
worker = ModelWorker.remote(0.5)
prediction_ref = worker.predict.remote([1.0, 2.0, 3.0])
predictions = ray.get(prediction_ref)
The actor's state is initialized once inside the actor process and reused by later method calls. A real model worker can use the same pattern to load a model once and keep it warm. Ray AI libraries build many of their higher-level abstractions from the same task, actor, and object primitives.
An ObjectRef
is best understood as a location-independent claim ticket. The value can live in a node's shared-memory object store rather than inside the driver's Python heap. Tasks can pass object references to other tasks without forcing the driver to download and re-upload every value.
That does not make data movement free. Large datasets still consume object-store memory and network bandwidth. Ray removes much of the coordination burden; it does not repeal physics.
There is a useful progression in Ray's API design.
With Core primitives, you control more of how work runs: which calls become tasks, which state lives in actors, how many calls may remain pending, when to wait, and where object references flow. That flexibility is valuable when an application does not fit a fixed data-processing pattern.
With an AI library, you usually declare more of what you want: transform this dataset in batches, start this training worker group, explore this search space, or serve this deployment. The library translates that declaration into tasks, actors, object transfers, scheduling requests, and recovery behavior.
Neither level is universally better. Start at the highest level that expresses the workload correctly, then drop to Core only when you genuinely need tighter execution control. This avoids rebuilding batching, backpressure, actor-pool management, or retry logic that a Ray library already owns.
Ray's official overview presents the framework in three layers. Thinking in those layers prevents many category errors.
At the top, Ray AI Libraries solve recognizable workload problems. In the middle, Ray Core supplies distributed tasks, actors, objects, scheduling, resource declarations, and runtime environments. At the bottom, Ray Clusters provide the processes and machines on which the work runs.
The same Python application can start a local Ray runtime on a laptop or connect to an existing cluster. The code may stay similar, but production operation does not happen automatically: networking, storage, observability, security, dependency images, and failure policy still need deliberate design.
Ray's native libraries share a runtime, but they do not form one mandatory pipeline. You can use Data without Train, Tune with a custom function, or Serve without training the model in Ray.
| Library | The question it answers | Core abstraction |
|---|---|---|
| Ray Data | ||
| How do I read, transform, and write distributed tabular or tensor data while streaming blocks through execution? | Lazy Dataset plans executed through tasks or actor pools |
|
| Ray Train | ||
| How do I coordinate distributed training workers? | A training function, workers, ScalingConfig , and a Trainer |
|
| Ray Tune | ||
| How do I run and manage many experiments efficiently? | Trainables, search spaces, search algorithms, schedulers, trials, and Tuner |
|
| Ray Serve | ||
| How do I expose Python and models as scalable online services? | Controller, proxies, deployments, replicas, and deployment handles | |
| RLlib | ||
| How do I scale reinforcement-learning sampling and learning? | Algorithms, EnvRunners, Learners, environments, and RLModules |
A Ray Dataset
represents a distributed collection split into blocks. Operations such as read_parquet()
, filter()
, and select_columns()
are lazy: Ray first builds a logical plan, then optimizes and converts it into physical operators when the dataset is consumed.
The point is not merely parallel file reading. Ray Data can stream blocks through operators so a pipeline does not have to materialize its full intermediate state at every step. For vectorized transforms and model inference, map_batches()
is usually the key API. Callable classes can run in an actor pool, which allows each actor to load a model once and reuse it across batches.
Ray Data is not a data catalog, data-governance product, or universal transaction layer. The selected data source still defines consistency, snapshot, schema-evolution, and commit behavior.
Ray Train organizes distributed training around four ideas: a user training function, worker processes, a scaling configuration, and a Trainer. In a typical PyTorch integration, TorchTrainer
launches the worker group from a train_loop_per_worker
function and uses ScalingConfig
to request the required workers and GPUs.
Ray creates the worker group, configures the framework's distributed environment, and runs the function on every worker. PyTorch still owns tensor operations, gradients, and the model. Ray owns distributed execution around that training code.
This separation is one of Ray's strongest design choices: it generally integrates with the Python ML ecosystem instead of asking you to replace it.
Training one model faster is only part of the problem. Model development often means running dozens or thousands of trials with different hyperparameters.
Tune separates several concerns that are frequently mixed together:
ResultGrid
.The function API reports intermediate metrics with tune.report()
. The important distinction is that Tune selects and manages experiments; it does not by itself turn the best trial into a production release.
Tune can launch distributed Ray Train runs as trials, but a tuning result is not automatically a production model release. Selecting the best trial, reproducing it, validating artifacts, and publishing a deployable model remain lifecycle decisions.
Serve runs on Ray actors. A controller manages deployments; HTTP or gRPC proxies receive requests; replicas execute application code; deployment handles connect components inside a composed application.
That architecture gives Serve several useful properties:
Serve is a serving runtime, not a model-governance system. It does not decide which model version passed your approval process or whether a schema change is safe.
Reinforcement learning has two expensive loops: collecting experience from environments and learning from that experience. RLlib can scale those axes independently with EnvRunner and Learner actors. It also provides algorithms, multi-agent support, offline data paths, and model abstractions through RLModules.
RLlib belongs in the overall Ray map even if your organization never uses reinforcement learning. It demonstrates why general-purpose tasks and actors are valuable: a complex domain-specific runtime can be assembled on top of a small distributed foundation.
The word job is often confused with task.
A Ray task is one remote function call. A Ray Job is the application entrypoint and all tasks, actors, and objects created recursively by that entrypoint.
For a remote cluster, the Ray Jobs API is a common boundary for submitting an application. A submission contains an entrypoint command and a runtime environment describing code and package dependencies. The CLI form is ray job submit --no-wait --working-dir . -- python train.py
; with --no-wait
, the client can disconnect while the cluster-owned job continues. The Jobs API also exposes status, logs, and stop operations.
KubeRay solves a different problem. It is a Kubernetes operator that manages RayCluster, RayJob, and RayService resources. Ray schedules Python work inside the Ray cluster; Kubernetes and KubeRay manage the cluster's pods and infrastructure lifecycle.
The broader AI stack becomes clearer when we separate its scheduling grains. PyTorch and vLLM optimize model execution. Ray coordinates the distributed processes and stages inside an application. Kubernetes or Slurm allocates infrastructure resources across applications and users. A company platform or SDK adds the domain policy that none of those lower layers can guess.
| Layer | Examples | Primary responsibility |
|---|---|---|
| Application and lifecycle contracts | ||
| Internal ML platforms, workflow SDKs | Valid requests, provenance, artifacts, release semantics, organization policy | |
| Model runtimes and frameworks | ||
| PyTorch, XGBoost, vLLM, Hugging Face | Tensor computation, model logic, model parallelism, engine-level optimization | |
| Distributed compute engine | ||
| Ray Core and Ray AI Libraries | Tasks, actors, objects, data movement, worker coordination, workload-aware recovery | |
| Infrastructure orchestration | ||
| Kubernetes, KubeRay, Slurm, cloud VMs | Nodes, containers, multi-tenant resource allocation, infrastructure lifecycle |
The boundaries are compositional rather than competitive. Ray Train can coordinate PyTorch workers. Ray Serve can orchestrate vLLM replicas. KubeRay can manage the Kubernetes resources that host both. In October 2025, Ray became a PyTorch Foundation-hosted project, alongside projects including PyTorch, vLLM, and DeepSpeed—an organizational reflection of this layered technical relationship.
Imagine a customer-churn system trained from daily Parquet data.
A sensible flow might look like this:
Ray can execute every compute-heavy box in that picture. However, the arrows between the boxes contain product-specific rules:
Those are not merely scheduling questions. They are lifecycle-contract questions.
Production stories are useful because they reveal both Ray's strengths and the engineering that still surrounds it. The following systems include company-specific extensions; they should not be read as a list of features available automatically in an unmodified Ray installation.
ByteDance described using Ray for large-scale audio and video data pipelines serving multimodal-model development. Its teams initially used Ray Core to distribute pipeline nodes, then adopted Ray Data for automatic block management, data , common transforms, and actor-pool scaling. This is the how-to-what progression in a real system: Core made the migration possible, while Data removed infrastructure code that did not differentiate the application.
The case also exposes an important boundary. Moving large video objects through the object store introduced serialization and spilling costs, so one packaging path fused download, processing, Parquet writing, and upload inside multithreaded actors. On preemptible infrastructure, the team added higher-level task reassignment and lineage handling around its particular workload. The lesson is not "always fuse operators" or "Ray handles every failure." It is to profile object movement and assign recovery ownership at the correct layer. See the Ray Summit 2024 case summary and recorded talk.
WeChat's AI workloads include search, recommendation, content understanding, and generative-media processing. Its published Ray platform work emphasizes fine-grained CPU/GPU resource declarations and the ability to express a multi-model application as Python rather than a mesh of separately deployed services. At its scale, however, the team built substantial platform machinery for heterogeneous resources, rapid failure routing, runtime distribution, and federated clusters. That is strong evidence for Ray as a compute foundation—and equally strong evidence that Ray is not the whole enterprise platform.
A later Tencent Hunyuan pipeline architecture makes another useful point: many "data pipelines" in modern AI are really batch-inference systems. Table metadata may identify the input, but most compute is spent decoding multimodal objects and running CPU or GPU models. A streaming-batch execution model can keep heterogeneous stages busy without turning the workload into an online service. See the streaming-batch research paper, WeChat's large-scale Ray practice, and Tencent Hunyuan's heterogeneous pipeline architecture.
Spotify built its Hendrix ML platform and a cloud development environment around Kubernetes, Ray, PyTorch, and internal SDKs. The surrounding platform standardized environments, integrated company data services, exposed remote compute, and added access control, telemetry, availability, and cost-management behavior.
This is the clearest answer to a common question: if Ray already scales Python, why build another SDK? Because a compute runtime does not know how an organization names datasets, packages models, authorizes users, or defines a successful release. Spotify's platform work illustrates that upper-layer problem. See the Spotify Engineering account of its Ray platform.
Ray is often most useful when you treat it as a compute substrate rather than expecting it to become your entire ML platform.
| Ray provides | Your application or platform still decides |
|---|---|
| Tasks, actors, objects, scheduling, and retries | Business-level state transitions and idempotency |
| Dataset execution | Source governance, credentials, snapshot guarantees, and schema policy |
| Training workers and checkpoints | A portable, validated model-delivery contract |
| Tune trials and result selection tools | Whether and how a selected trial becomes a formal release |
| Serve deployments and autoscaling | Approval, promotion, canary, rollback, and tenant policy |
| Cluster resource requests | Organizational quota, RBAC, cost ownership, and infrastructure security |
This boundary is healthy. A general compute framework should not guess every organization's governance model.
It also creates room for higher-level frameworks that add opinionated contracts while continuing to delegate distributed execution to Ray.
Ray is worth evaluating when several of these statements are true:
Ray may be unnecessary or counterproductive when:
"Can Ray run it?" is usually the wrong first question. Ask whether the workload has enough parallelism, state, or heterogeneous resource demand to justify a distributed runtime.
ray.init()
is excellent for learning and local development. Before production, test the actual submission, dependency, storage, and failure boundaries you will use on the cluster. A local success does not prove that workers have the right packages or network access.
Ray schedules declared resources. If a task consumes four CPUs but requests one, the scheduler cannot make a good placement decision. The same applies to GPUs, custom resources, and memory-sensitive actors.
The driver should coordinate work, not collect every large intermediate result into its heap. Prefer distributed transformations and pass object references between workers where possible.
A retried task, restarted actor, failed training worker, partially published artifact, and timed-out HTTP request are different failures. Define which layer owns retry, cleanup, idempotency, and the final source of truth.
Runtime environments can ship code and Python packages, but dynamic installation is not a substitute for a tested runtime image in sensitive or large deployments. Pin versions and verify that the driver and workers see the same dependency closure.
Distribution adds serialization, coordination, object-store pressure, and network transfer. Profile the single-node path first. More machines help only when useful parallel work exceeds those costs.
Profile the whole pipeline as well as the model kernel. GPU utilization can remain low because CPU decoding, tokenization, network reads, or serialization cannot feed the accelerator quickly enough. Conversely, sending very large intermediate objects through many actor boundaries can make the object store or spill path the bottleneck. Tune stage concurrency, batch size, CPU-to-GPU ratios, and object lifetime together rather than optimizing each operator in isolation.
Ray becomes much less mysterious when you keep four boundaries in mind:
Disclosure: I maintain Tributo. It is an Apache-2.0, Ray-native SDK shown here as one implementation of the upper layer described above, not as a required part of Ray. Its current compatibility baseline is Ray 2.55.1. Its design keeps distributed execution inside Ray while adding typed requests, provider routing, provenance, validated model Bundles, and explicit batch or online inference contracts.
The project deliberately does not provision Kubernetes, implement a custom scheduler, manage tenants or quotas, or provide approval and rollout workflows. Its support matrix separates verified, beta, alpha, adapter-only, and unsupported paths. The broader lesson is independent of this project: keep policy and delivery semantics above Ray, and keep physical distributed execution inside the runtime that owns it.
Disclosure: This article was drafted with AI assistance for research organization, structure, language, and diagram production. I reviewed the technical claims against the linked Ray documentation and the public Tributo source tree. The human author remains responsible for rechecking every claim and code sample before publication.