Terrill Dicki Aug 25, 2026 16:49

Anyscale pushes Ray Core scalability, achieving 10,000-node clusters for AI training and inference. Key improvements include faster actor launches and reduced bottlenecks.

Anyscale has announced substantial advancements in scaling Ray Core, the distributed runtime layer of the Ray framework, to support AI workloads on clusters as large as 10,000 nodes. These upgrades improve performance across batch inference, reinforcement learning (RL), and pre-/post-training tasks, addressing key bottlenecks in scheduling and resource management.

Key metrics from the latest enhancements include a 23% faster end-to-end performance for batch inference on 500-node clusters, a 24% speedup in Ray Data shuffle operations, and a 6.5x improvement in actor launch times at 2,000 nodes. Notably, Ray Core now scales to 40,000 actors on 10,000-node clusters, a milestone previously unattainable.

These optimizations are crucial as Ray increasingly underpins large-scale AI systems. For instance, distributed training workloads often demand low-latency actor scheduling and fault recovery processes, particularly when managing GPUs constrained by network topology in data center setups. By addressing these scaling challenges, Ray Core positions itself as a foundational tool for AI infrastructure, particularly in cloud and hybrid environments.

Breaking Down the Bottlenecks

Historically, three primary issues limited Ray’s scalability:

• Lock contention between threads, which hindered task throughput.
• Overloaded threads managing asynchronous operations, delaying task scheduling.
• Reliance on stale resource views, leading to inefficient task placement.

To resolve these, Anyscale introduced multithreading in Ray’s Global Control Service (GCS), reducing idle load on the system. Placement group scheduling, a critical feature for topology-aware workloads, saw a dramatic reduction in setup time—from hours to seconds at 10,000 nodes. Moreover, optimizations to actor lifecycle management and resource synchronization further minimized delays, especially during large-scale restarts caused by node failures.

Strategic Implications for AI Infrastructure

The scalability gains align with Anyscale’s broader strategy following its July 2026 announcement of a definitive agreement to join Nscale. The company plans to "double down" on Ray development, optimizing the framework for emerging accelerator and data-center architectures while maintaining its open-source governance under the PyTorch Foundation.

These improvements also reflect market trends emphasizing production-scale AI workloads. For example, Google Cloud and Anyscale reported up to 5x higher throughput and 8x lower latency for Ray Serve on GKE earlier this year, highlighting Ray’s role in serving large language models (LLMs) and other inference-heavy applications. The latest advancements make Ray even more competitive in distributed AI workloads, particularly for organizations scaling across Kubernetes and hybrid cloud environments.

What’s Next?

With these updates already available in Ray nightly builds, Anyscale plans further investments in actor and driver scalability. Upcoming developments include transitioning to a more centralized scheduling mechanism and relocating actor lifecycle management directly to the owner rather than the GCS. These changes aim to further enhance scalability and reduce latency for workloads at the extreme edge of cluster size.

As AI workloads continue to grow in complexity and scale, Ray’s ability to handle clusters at 10,000 nodes and beyond positions it as a critical enabler for next-generation AI infrastructure. Users can contribute to or test these advancements via Ray’s GitHub repository and Slack channels.

Image source: Shutterstock

read more