Summary
In this episode Robert Nishihara, co-founder of Anyscale and co-creator of Ray, talks about maximizing hardware utilization for AI and data-intensive workloads. He explores Ray’s evolution alongside Kubernetes and PyTorch, and why consolidation at these layers has enabled a new generation of complex, heterogeneous workloads. Robert explains how data preparation has shifted to GPU- and inference-heavy, multimodal pipelines; where Ray fits compared to Spark and workflow orchestrators; and why Ray excels at composing heterogeneous pools of compute, handling failures, and scaling complex systems like multi-node LLM inference and reinforcement learning. He digs into practical strategies for boosting GPU utilization across training and inference, elasticity and prioritization of workloads, topology-aware scheduling, and the importance of fast failure recovery as hardware scales from nodes to racks. If you’re wrestling with expensive GPUs, multimodal data curation, or cross-node LLM inference, this conversation offers concrete mental models and architectural guidance.
Announcements
* Hello and welcome to the Data Engineering Podcast, the show about modern data management
* Your host is Tobias Macey and today I'm interviewing Robert Nishihara about the challenges of maximizing the utility of your available hardware for AI applications
Interview
* Introduction
* How did you get involved in the area of data management?
* Can you start by giving an overview of the major contributors to wasted or idle compute?
* Why does it matter if the available compute isn't being maximized?
* What are some of the typical ad-hoc methods that teams might use to try to get the most out of their available hardware (especially GPUs)?
* What are the most interesting, innovative, or unexpected ways that you have seen Ray used?
* What are the most interesting, unexpected, or challenging lessons that you have learned while working on Ray and distributed compute for data and AI?
* When is Ray the wrong choice?
* What do you have planned for the future of Ray?
Contact Info
* LinkedIn
Parting Question
* From your perspective, what is the biggest gap in the tooling or technology for data management today?
Closing Announcements
* Thank you for listening! Don't forget to check out our other shows. Podcast.__init__ covers the Python language, its community, and the innovative ways it is being used. The AI Engineering Podcast is your guide to the fast-moving world of building AI systems.
* Visit the site to subscribe to the show, sign up for the mailing list, and read the show notes.
* If you've learned something or tried out a project from the show then tell us about it! Email hosts@dataengineeringpodcast.com with your story.
Links
* AnyScale
* Ray
* Deep Learning
* Computer Vision
* Kubernetes
* Cursor
* Claude Code
* Kube-Ray
* PyTorch
* Tensorflow
* Theano
* Caffe
* vLLM
* SGLang
* Ray Tune
* Neural Network
* Learning Rates
* Reinforcement Learning
* AlphaGo
* Cursor Composer 2
* ImageNet
* Transformer Architecture
* Stochastic Gradient Descent
* Airflow
* Dagster
* Flyte
* Mixture of Experts
* Prefill
* Temporal
* Actor Framework
* RDMA == Remote Direct Memory Access
* Neoclouds
* AI Engineering Podcast Episode
The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA