Scaling Python for Machine Learning: Beyond Data Parallelism • Holden Karau • GOTO 2023

This presentation was recorded at GOTO Chicago 2023. #GOTOcon #GOTOchgo
https://gotochgo.com

Holden Karau - Open Source Engineer at Netflix ‪@HoldenKarau‬

RESOURCES
http://www.holdenkarau.com
  / holdenkarau  
  / holdenkarau  
https://tech.lgbt/@holden
https://www.distributedcomputing4kids...

ABSTRACT
Data Parallelism can be amazing and it frees us from so many fiddly complicated tasks (like dealing with locks). On the other hand, as training large machine learning models becomes increasingly popular, we're seeing the need to move beyond purely data-parallel techniques. Depending on recompute exclusively for failure is no longer sufficient as our operations are not idempotent.

In this talk we will look at Spark, Dask, and Ray in the context of scaling machine learning models and how you can take advantage of other types of distributed parallelism (including the actor model for managing model weights during training). [...]

TIMECODES
00:00 Intro
04:16 Probable (relevant) biases
06:14 Quick refresher on data parallelism
11:52 What do (distributed) tasks look like?
12:49 Dask distributed tasks
13:42 Ray distributed tasks
14:03 How are they different & same?
18:41 Task fault tolerance
21:53 Does Spark have tasks & actors?
27:39 Ray Diagram
31:33 Ray actor fault tolerance
33:43 What's up with Ray & Netflix?
34:39 Dask actor fault tolerance
37:28 Outro

Download slides and read the full abstract here:
https://gotochgo.com/2023/sessions/2596

RECOMMENDED BOOKS
Holden Karau • Distributed Computing 4 Kids • https://www.distributedcomputing4kids...
Holden Karau • Scaling Python with Dask • https://www.oreilly.com/library/view/...
Holden Karau & Boris Lublinsky • Scaling Python with Ray • https://amzn.to/44GU6cC
Holden Karau & Rachel Warren • High Performance Spark • https://amzn.to/3v2eLbn
Holden Karau, Konwinski, Wendell & Zaharia • Learning Spark • https://amzn.to/397e2NE
Holden Karau & Krishna Sankar • Fast Data Processing with Spark 2nd Edition • https://amzn.to/3xKhXKu
Holden Karau • Fast Data Processing with Spark 1st Edition • https://amzn.to/3rHQgOu

  / gotocon  
  / goto-  
  / gotoconferences  
#MachineLearning #PythonProgramming #Parallelism #DataParallelism #HoldenKarau #PythonForML #ApacheRay #ApacheDask #ApacheSpark #Joblib #FaultTolerance

Looking for a unique learning experience?
Attend the next GOTO conference near you! Get your ticket at https://gotopia.tech
Sign up for updates and specials at https://gotopia.tech/newsletter

SUBSCRIBE TO OUR CHANNEL - new videos posted almost daily.
https://www.youtube.com/user/GotoConf...