OLMo leads on the secrets of training language models (w Dirk Groeneveld, Kyle Lo, & Luca Soldaini)

We're here to share the story of building our Open Language Models (OLMos) and what we improved to build OLMo 2 that is competitive with the smaller Llama 3.1 models. This is all about building an effective, small language modeling team that can share all it learns with the academic community.

Some questions were pulled from Twitter, but please comment or get in touch if you want us to cover anything in the future episode(s)!

Main topics:
1. Pretraining efficiency and our quest for stability after a not-so-secret failed 70B run early in 2024,
2. What the role of OLMo is in the broader AI landscape and how that is, or is not, changing,
3. Many little decisions that going into building language models and their teams (with a focus on NOT post-training, given I already talk about that a ton).

Transcript, links, and more information: https://www.interconnects.ai/p/olmo-2...

Play with the models we build here: playground.allenai.org/

00:00:00 Introduction
00:02:45 Early history of the OLMo project
00:15:27 The journey to stability
00:25:00 The evolving role of OLMo and pretraining research
00:29:00 Pretraining Q&A (µP, scaling laws, MoE, etc.)
00:40:40 How to think about pretraining data work
00:54:30 Role of pre-training vs mid training vs post-training
01:02:19 Release strategy and wrapping up

Dirk Groeneveld — https://x.com/mechanicaldirk // https://bsky.app/profile/mechanicaldi...
Kyle Lo — https://x.com/kylelostat // https://bsky.app/profile/kylelo.bsky....
Luca Soldaini —   / soldni   // https://bsky.app/profile/soldaini.net
General OLMo contact — [email protected]

Get Interconnects (https://www.interconnects.ai/)...
... on YouTube:    / @interconnects  
... on Twitter: https://x.com/interconnectsai
... on Linkedin:   / interconnects-ai  
... on Spotify: https://open.spotify.com/show/2UE6s7w...
… on Apple Podcasts: https://podcasts.apple.com/us/podcast...