Do we need Attention? A Mamba Primer

A Technical Primer on Mamba and Friends. With Yair Schiff (https://yair-schiff.github.io/)

Slides: https://github.com/srush/mamba-primer...

Main focus:
Mamba: Linear-Time Sequence Modeling with Selective State Spaces http://arxiv.org/abs/2312.00752 from Albert Gu and Tri Dao.
Simplified State Space Layers for Sequence Modeling http://arxiv.org/abs/2208.04933 from Smith JT, Warrington A, Linderman SW

00:00 - Intro
04:03 - Section 1 - Linear Time Varying recurrences
12:07 - Section 2 - Associative Scan
16:27 - Section 3 - Continuous-Time SSMs
26:55 - Section 4 - Large States and Hardware-Aware Parameterizations
34:56 - Conclusion

Yang S,Wang B,Shen Y,Panda R,Kim Y Gated Linear Attention Transformers with Hardware-Efficient Training http://arxiv.org/abs/2312.06635
Arora S,Eyuboglu S,Zhang M,Timalsina A,Alberti S,Zinsley D,Zou J,Rudra A,Ré C Simple linear attention language models balance the recall-throughput tradeoff http://arxiv.org/abs/2402.18668
De S,Smith SL,Fernando A,Botev A,Cristian-Muraru G,Gu A,Haroun R,Berrada L,Chen Y,Srinivasan S,Desjardins G,Doucet A,Budden D,Teh YW,Pascanu R,De Freitas N,Gulcehre C Griffin: Mixing Gated Linear Recurrences with Local Attention for Efficient Language Models http://arxiv.org/abs/2402.19427
Sun Y,Dong L,Huang S,Ma S,Xia Y,Xue J,Wang J,Wei F Retentive Network: A Successor to Transformer for Large Language Models http://arxiv.org/abs/2307.08621