How Coinbase Uses Ray, vLLM & LiteLLM to Power Secure LLM Services | Ray Summit 2025

At Ray Summit 2025, Wenyue Liu and Akshit Trehan from Coinbase share how the Coinbase Machine Learning Platform (MLP) team built trusted, production-grade LLM services using Ray, vLLM, and LiteLLM—supporting one of the world’s most security-sensitive environments and reinforcing Coinbase’s mission to remain the most trusted crypto exchange.

They begin by outlining the unique challenges of building LLM infrastructure inside a financial institution, where trust, security, and reliability are non-negotiable. To meet these requirements, Coinbase engineered an LLM serving stack that seamlessly integrates:

Ray for distributed orchestration and scaling

vLLM for high-throughput, low-latency inference

LiteLLM for routing, abstraction, and multi-provider reliability

The speakers then take a deep dive into the technical architecture behind Coinbase’s internal LLM services, including:

User authentication and authorization patterns tailored for secure LLM access

Service-to-service (s2s) trust models that allow safe and auditable communication between internal systems

LiteLLM distribution strategies to balance throughput, reliability, and fallback behavior

How vLLM and Ray work together to power scalable, production-grade LLM serving APIs

Systems built to support high-volume internal LLM traffic, ensuring consistent performance under load

The session walks through the full end-to-end story of how Coinbase uses Ray and vLLM to deliver trustworthy, secure, and efficient LLM services—meeting the strict reliability requirements of a top global crypto exchange.

Liked this video? Check out other Ray Summit breakout session recordings    • Ray Summit 2025 - Breakout Sessions  

Subscribe to our YouTube channel to stay up-to-date on the future of AI!    / anyscale  

🔗 Connect with us:
LinkedIn:   / joinanyscale