Resurrecting Recurrent Neural Networks for Long Sequences | Razvan Pascanu

ICARL Seminar Series - 2023 Winter

Resurrecting Recurrent Neural Networks for Long Sequences
Seminar by Razvan Pascanu

Abstract:
In this talk, Razvan Pascanu will focus on State Space Models (SSM), a recently introduced family of sequential models and specifically discuss the relationship between SSMs and recurrent neural networks. He will start with a short history of architecture design for language modelling, which he will use as a motivating task. This will allow to provide some insights in the evolution of RNN architectures, and why some choices behind the SSM architecture seemed counter-intuitive. Most of the talk will focus on introducing the Linear Recurrent Unit architecture, explaining the role of the various modifications from traditional non-linear recurrent models.

The talk will conclude with some open questions about the role recurrent architectures could or should play, and potentially the less well understood relationship between these SSM models and transformer like architectures.

About the Speaker

Razvan Pascanu has been a research scientist at Google DeepMind since 2014. Before this, he did his PhD in Montréal with prof. Yoshua Bengio, working on understanding deep networks, recurrent models and optimization. Since he joined DeepMind he has also had significant contributions in deep reinforcement learning, continual learning, meta-learning, graph neural networks as well as continuing his research agenda of understanding deep learning, recurrent models and optimization. Please see his scholar page for specific contributions. He is also actively promoting AI research and education as a main organizer of Conference on Life-long Learning Agents (CoLLAs) lifelong-ml.cc , Eastern European Machine Learning Summer School (EEML) www.eeml.eu and www.workshops.eeml.eu as well as different workshops at NeurIPS, ICML and ICLR.

——————————————————
Links
Razvan Pascanu
Site: https://sites.google.com/view/razp

ICARL
Site: icarl.doc.ic.ac.uk
Twitter: twitter.com/ic_arl
YouTube: @ICARLSeminars
——————————————————