PyTorch Symmetric Memory: A New Paradigm for Programming Distributed AI

Speakers: Ke Wen and Natalia Gimelshein

Recent model advancements have highlighted the need for customized communication. In response, PyTorch introduces Symmetric Memory, a distributed programming model that creates a global address space for data spanning multiple GPUs' memory.

In this talk, we will demonstrate how developers can author their own communication kernels at the device level. Additionally, we will show how to interleave communication and computation within the same kernel using popular languages like Triton, achieving the finest-grained fusion possible. We will also discuss key network technologies for scaling symmetric memory across nodes.

Learn more here: https://atscaleconference.com/