Ray Serve: Advancing scalability and flexibility | Ray Summit 2025

At Ray Summit 2025, Abrar Sheikh and Alexander Yang from Anyscale share the major advancements in Ray Serve, now one of the most widely used libraries for powering modern AI applications across industries.

They begin by highlighting why Ray Serve stands apart from traditional online inference frameworks: it is natively built for multi-model serving, supports any hardware and accelerator, and integrates cleanly with any inference engine—from vLLM and TensorRT-LLM to custom model runtimes.

The session then dives into the most significant improvements Ray Serve has delivered over the past year, including:

Greater flexibility for complex inference patterns
Expanded APIs and routing capabilities make it easier to serve multi-stage pipelines, ensemble models, agentic systems, and inference graphs.

Higher performance at scale
Under-the-hood optimizations, improved scheduling, and faster data movement enable Ray Serve to handle massive request volumes with lower latency and higher throughput.

Multi-cloud inference support
New features make it easier to deploy Ray Serve clusters across multiple cloud providers—supporting hybrid inference, failover strategies, and portable deployment architectures.

Abrar and Alexander demonstrate how Ray Serve continues to evolve to meet the needs of cutting-edge AI systems—from LLMs to multimodal workloads—and offer a look at what's next for the framework.

Attendees will learn how to take advantage of Ray Serve’s newest capabilities to build flexible, performant, and cloud-agnostic inference platforms at scale.

Liked this video? Check out other Ray Summit breakout session recordings    • Ray Summit 2025 - Breakout Sessions  

Subscribe to our YouTube channel to stay up-to-date on the future of AI!    / anyscale  

🔗 Connect with us:
LinkedIn:   / joinanyscale  
X: https://x.com/anyscalecompute
Website: https://www.anyscale.com/