[2024 Best AI Paper] ReMamba: Equip Mamba with Effective Long-Sequence Modeling
Автор: Paper With Video
Загружено: 2024-10-21
Просмотров: 32
This video was created using https://paperspeech.com. If you’d like to create explainer videos for your own papers, please visit the website!
Title: ReMamba: Equip Mamba with Effective Long-Sequence Modeling
Authors: Danlong Yuan, Jiahao Liu, Bei Li, Huishuai Zhang, Jingang Wang, Xunliang Cai, Dongyan Zhao
Abstract:
While the Mamba architecture demonstrates superior inference efficiency and
competitive performance on short-context natural language processing (NLP)
tasks, empirical evidence suggests its capacity to comprehend long contexts is
limited compared to transformer-based models. In this study, we investigate the
long-context efficiency issues of the Mamba models and propose ReMamba, which
enhances Mamba's ability to comprehend long contexts. ReMamba incorporates
selective compression and adaptation techniques within a two-stage re-forward
process, incurring minimal additional inference costs overhead. Experimental
results on the LongBench and L-Eval benchmarks demonstrate ReMamba's efficacy,
improving over the baselines by 3.2 and 1.6 points, respectively, and attaining
performance almost on par with same-size transformer models.
![[2024 Best AI Paper] ReMamba: Equip Mamba with Effective Long-Sequence Modeling](https://ricktube.ru/thumbnail/mo4fN7WzxWg/hq720.jpg)
Доступные форматы для скачивания:
Скачать видео mp4
-
Информация по загрузке: