[Feature] KubeRay Federation: Multi-Cluster RayCluster Deployment and AutoScaling

[yuchen-ecnu]

Search before asking

• I had searched in the issues and found no similar feature requirement.

Description

Summary

We propose adding KubeRay Federation capability to enable RayCluster deployment and auto-scaling across multiple Kubernetes clusters. This feature allows organizations to unify fragmented compute resources (especially GPUs) distributed across different availability zones, cloud providers, or physical locations into a single logical Ray cluster.

Motivation

The Problem

In large-scale production environments, compute resources are often distributed across:

• Multiple availability zones (AZs) within a single cloud provider
• Multiple cloud vendors (for supply assurance and cost optimization)
• On-premise and cloud hybrid environments

Current KubeRay is limited to single Kubernetes cluster deployment, which creates several operational challenges:

1. Fragmented GPU Resources

Organizations procure GPUs from multiple cloud vendors to ensure supply. These resources are physically isolated across different K8s clusters, making it impossible to:

• create a single RayCluster spanning multiple AZs
• dynamically scale workers across cluster boundaries
• efficiently utilize all available resources as a unified pool (for scheduling user ray job within a single Ray cluster)

2. Operational Overhead with Current Workarounds

To work around single-cluster limitations, teams currently:

• Deploy multiple smaller Ray clusters (e.g., 20 clusters × 10 GPUs instead of one cluster × 200 GPUs) and split datasets into multiple Ray Data jobs for large-scale computing.
  • These jobs are isolated and cannot be balanced across Ray clusters, which may lead to long-tail performance issues.
• Manually start/stop clusters across different AZs when resources are unavailable

3. Virtual Kubelet Limitations

Some organizations use Virtual Kubelet to aggregate resources into a single entry-point K8s cluster. However, this approach has scalability issues:

• The entry-point K8s control plane must handle all pods, creating bottlenecks
• During peak scaling (e.g., scaling from 10K to 400K+ cores within an hour), the control plane may fail to keep up
• Single cluster pod capacity limits require multiple entry-point clusters, increasing deployment complexity

Use case

Use Cases: Large-Scale Elastic Batch Inference

In large-scale batch inference scenarios, workloads often require hundreds or even thousands of GPUs to process massive datasets within tight time windows. However, due to GPUs are procured from multiple cloud vendors and availability zones, these GPU resources are scattered across different Kubernetes clusters.

Current Pain Points:

1. Forced Multi-Cluster Deployment

Users are forced to create multiple RayCluster instances across different K8s clusters and manually partition datasets and submit jobs separately to each cluster.

2. Dynamic Resource Availability Creates Imbalance

In elastic resource environments, idle capacity curves vary across cloud vendors and AZs over time — compute power is constantly fluctuating.

• At time T1: Cloud-A/AZ1 has 200 idle GPUs, Cloud-B/AZ1 has 50 idle GPUs
• At time T2: Cloud-A/AZ1 drops to 80 GPUs (preemption), Cloud-B/AZ1 scales to 300 GPUs
  Once data is partitioned and distributed to separate clusters, Ray Data cannot perform cross-cluster load balancing. The cluster with fewer resources becomes a bottleneck while others sit idle, which causes severe long-tail effects.

3. Operational Complexity

Managing multiple RayClusters creates significant burden:

• Deployment complexity: Manually create/configure clusters across different K8s environments
• Data partitioning: Estimate resource availability and split data accordingly (often inaccurate)
• Monitoring overhead: Track job progress across multiple dashboards
• Failure handling: Manually redistribute work when one cluster fails or is preempted

Desired State with KubeRay Federation:

• Single job submission: one RayCluster, one dataset, one job
• Automatic load balancing: Ray Data dynamically schedules tasks to available resources across all AZs
• Preemption resilience: When GPUs are reclaimed in one AZ, work automatically shifts to others
• Optimal resource utilization: No idle waiting; work flows to wherever capacity exists
• Simplified operations: Single cluster to deploy, monitor, and maintain

Scope & Suitable Workloads

Federated KubeRay is designed to enable unified Ray clusters across multiple Kubernetes clusters, AZs, or cloud providers. While it brings significant benefits, it is important to clarify when it is suitable and when it is not.

Suitable Scenarios

1. Compute-Intensive Workloads

• Model inference (OCR, ASR, embeddings, LLM inference)
• Large-scale batch processing where computation dominates communication
• Workloads where single jobs require hundreds to thousands of GPUs across clusters

2. Elastic / Multi-Cloud Resource Environments

• Resources are scattered across multiple cloud vendors or availability zones
• GPU availability fluctuates due to preemption or autoscaling
• Desire to dynamically balance workloads with Ray-level task / actor across multiple Kubernetes clusters

Scenarios Where Federation May Not Be Ideal

1. IO-Bound or Lightweight Tasks

• Simple filtering, text transformations, or small-scale ETL
• When network latency dominates execution time, local execution is more efficient

2. Workloads with Large-Scale Shuffle

• Applications that require frequent all-to-all data exchanges between tasks
• Examples: large-scale joins, sorts, or other shuffle-heavy distributed algorithms

Feel free to suggest additional scenarios.

Design Considerations & Community Concerns

1. Cross-AZ Communication Overhead

A key concern is the potential communication overhead when Ray workers are distributed across multiple availability zones (AZs) or Kubernetes clusters.
In practice, the impact depends heavily on the workload characteristics.

For compute-heavy workloads such as, the computation time per data block is significantly larger than the data transfer cost. And in the current LLM era, compute-intensive workloads are becoming the dominant use case, which motivates exploring cross-cluster federation.

Prior Art & References

We noticed that Tencent may have implemented a similar approach, which serves as a useful reference: ray-forward-2025

We're happy to share more details about our production use cases. Looking forward to the community's feedback!

Related issues

No response

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

[yuchen-ecnu]

cc @Future-Outlier

[Future-Outlier]

Hi, @yuchen-ecnu
is skypilot similar to this solution?

[yuchen-ecnu]

Hi, @yuchen-ecnu is skypilot similar to this solution?

Hi @Future-Outlier , I’ve added two figures to better illustrate our use case. While SkyPilot provides cross-Kubernetes management capabilities, our scenario focuses on federating resources from multiple Kubernetes clusters into a single unified Ray cluster.

[andrewsykim]

Multi Kueue (https://kueue.sigs.k8s.io/docs/concepts/multikueue/) can be another alternative solution, however I don't think it would support federating Ray-level task / actor across multiple Kubernetes clusters like you're proposing. Multi-kueue is more at the custom resource level (RayCluster, RayJob, RayService)

[siyuanfoundation]

While this is technically possible, we need to think about the implications of communication overhead across AZs.
@yuchen-ecnu are you proposing this for Ray Serve use case or general use cases?
For the more general use cases, Ray head would need be aware of the locations of the workers and distribute data & tasks very carefully to not degrade the performance significantly. So I think this would need substantial work both in Ray and KubeRay.

An alternative might be a Ray job delegator/proxy that delegates jobs to different underlying ray clusters, and aggregates the job status across instead of a single ray cluster. Have you considered that?

[andrewsykim]

I think to some extent it would be on the user to ensure there is not heavy data transfer across AZs (label selectors that include the AZ would help here)

[andrewsykim]

Ray Data does seem like a very good use-case here because batch inference is localized to the local GPU and you can horizontally scale and store the results in a central store (GCS, S3 ,etc).

[siyuanfoundation]

Ray Data does seem like a very good use-case here because batch inference is localized to the local GPU and you can horizontally scale and store the results in a central store (GCS, S3 ,etc).

I agree it would be useful for very limited scenarios: Ray Data that does not involve any reshuffling, and possibly Ray Serve. I suggest calling it out explicitly in the proposal.

[siyuanfoundation]

Multi Kueue (kueue.sigs.k8s.io/docs/concepts/multikueue) can be another alternative solution, however I don't think it would support federating Ray-level task / actor across multiple Kubernetes clusters like you're proposing. Multi-kueue is more at the custom resource level (RayCluster, RayJob, RayService)

Multikueue does support pod level scheduling and status update (https://kueue.sigs.k8s.io/docs/tasks/run/multikueue/plain_pods/). In theory, we can deploy the RayCluster in one cluster, and only delegate the worker pods to multikueue, as long as they know the "public address" of the head pod and have access.

[yuchen-ecnu]

While this is technically possible, we need to think about the implications of communication overhead across AZs. @yuchen-ecnu are you proposing this for Ray Serve use case or general use cases? For the more general use cases, Ray head would need be aware of the locations of the workers and distribute data & tasks very carefully to not degrade the performance significantly. So I think this would need substantial work both in Ray and KubeRay.

An alternative might be a Ray job delegator/proxy that delegates jobs to different underlying ray clusters, and aggregates the job status across instead of a single ray cluster. Have you considered that?

Hi @siyuanfoundation , thanks for the thoughtful feedback.

The communication overhead across AZs is indeed an important concern. In our production evaluation, we found that the impact depends heavily on the workload type.

1. Ray Data workloads

Compute-intensive workloads

For workloads such as model inference (OCR, ASR, Embeddings, LLM inference), the computation cost is significantly higher than the communication cost.
For example:

• A typical Ray Data block is 128 MB (~7000 records).
• Running inference with a 7B model on a single H20 GPU may take ~7 hours to process this block.

This results in an average bandwidth requirement of roughly: 128 MB / 7 hours ≈ 5 KB/s
In addition, Ray Data already supports prefetching blocks, which allows the next block to be transferred while the current block is being processed.
(For example, via Ray Data’s max_tasks_in_flight_per_actor)

Because of this, we believe the communication overhead is relatively small compared with the compute cost, and the bandwidth usage is acceptable.

IO-intensive workloads

For IO-bound tasks (e.g., simple data filtering or text transformations), running them close to the dataset is indeed more efficient. These workloads typically require only small amounts of compute and may not benefit from a federated cluster setup.

However, in the current LLM era, we are observing that:

• compute-intensive workloads are becoming increasingly common
• single jobs are growing significantly in scale

This trend is one of the motivations for exploring federated KubeRay clusters.

2. Ray Serve workloads

In Ray Serve scenarios, users send requests via HTTP or RPC.

The bandwidth costs mainly come from two sources:

User request traffic

The network traffic generated by user requests is unavoidable and must be supported regardless of deployment topology.

Ray internal communication

In many Ray Serve deployments, replicas across clusters are horizontally scaled instances of the same deployment.
Replicas typically do not communicate with each other directly, so cross-cluster communication overhead is limited.
For applications with multiple deployments within a single application, placement strategies such as label selectors (as mentioned by @andrewsykim ) could help control scheduling locality.

Regarding the alternative approach (job delegator / proxy)

We have considered the idea of a Ray job delegator that submits jobs to multiple underlying Ray clusters. However, this approach still leaves several issues unresolved:

1. No cross-cluster load balancing
   Once jobs are launched on separate Ray clusters, the datasets are already partitioned and cannot be dynamically rebalanced.
   In elastic environments, resource availability constantly changes. Without a unified cluster, it becomes impossible to rebalance workloads among actors/tasks dynamically, which can lead to long-tail effects.

2. Operational overhead
   Users must manually:

• estimate available resources per cluster
• partition datasets
• submit multiple jobs
• monitor multiple clusters
• redistribute work after failures (even repartition datasets for load banlance)

While some tooling can reduce the operational friction (like Multi Kueue (https://kueue.sigs.k8s.io/docs/concepts/multikueue/)), it does not fundamentally solve the resource utilization and scheduling imbalance problem.

I’ll also update the proposal later to reflect these considerations. Thanks again to @siyuanfoundation and @andrewsykim for raising these important questions.

[yuchen-ecnu]

Multi Kueue (kueue.sigs.k8s.io/docs/concepts/multikueue) can be another alternative solution, however I don't think it would support federating Ray-level task / actor across multiple Kubernetes clusters like you're proposing. Multi-kueue is more at the custom resource level (RayCluster, RayJob, RayService)

Multikueue does support pod level scheduling and status update (https://kueue.sigs.k8s.io/docs/tasks/run/multikueue/plain_pods/). In theory, we can deploy the RayCluster in one cluster, and only delegate the worker pods to multikueue, as long as they know the "public address" of the head pod and have access.

You’re right. We have actually experimented with a similar approach in our environment by combining Virtual Kubelet with a modified version of KubeRay to build a cross-Kubernetes RayCluster.

Our setup works roughly as follows:

1. We first create the Ray head pod in an entry-point Kubernetes cluster by kuberay operator.
2. We then retrieve the head pod IP address in the kuberay operator.
3. When creating worker pods, we render the ray start parameters using the head pod IP instead of a Kubernetes Service address (since the svc only works within a single Kubernetes cluster).
4. The worker pods are then scheduled onto Virtual Kubelet nodes, which route them to other Kubernetes clusters.

With this setup, workers in different clusters can successfully connect to the head node and form a single Ray cluster.

However, we encountered an important issue with this approach: head node failure.

If the head pod crashes and gets recreated, its IP address may change. When that happens, all worker pods must be restarted with the new head IP so they can reconnect to the cluster. This makes the system fragile, so we don’t think it is a good long-term design.

We believe a more robust mechanism for stable cross-cluster head discovery would be needed.

Regarding MultiKueue, the plain_pods mode seems conceptually similar to the architecture we described, but without relying on Virtual Kubelet.