Mamba - a replacement for Transformers?

Mamba is a new neural network architecture proposed by Albert Gu and Tri Dao.

Timestamps:
00:00 - Mamba - a replacement for Transformers?
00:19 - The Long Range Arena benchmark
01:20 - Legendre Memory Units
02:07 - HiPPO: Recurrent Memory with Optimal Polynomial Projections
02:38 - Combining Recurrent, Convolutional and Continuous-time Models with Linear State-Space Layers
03:28 - Efficiently Modeling Long Sequences with Structured State Spaces (S4)
05:46 - The Annotated S4
06:13 - Mamba: Linear-Time Sequence Modeling with Selective State Spaces
07:42 - Motivation: Why selection is needed
09:59 - S5
12:00 - Empirical evaluation


The paper can be found here: https://arxiv.org/abs/2312.00752

Topics: #mamba #foundation

References for papers mentioned in the video can be found at
https://samuelalbanie.com/digests/202...


For related content:
Twitter:   / samuelalbanie  
personal webpage: https://samuelalbanie.com/
YouTube:    / @samuelalbanie1