Pre-Train LLMs from scratch (Python)

Learn how LLMs work, how to train them, and how to speed up the training process in Python using the llm_trainer library. This library allows you to train any LLM model in just a few lines of code.

The first step of training is called Pre-Training. In this stage, you expose a language model to a vast amount of internet data. The goal is for the model to develop a general understanding of the world. The second step is Post-Training, where you fine-tune your model on a smaller dataset formatted as dialogues (to create an assistant capable of answering questions).

Additionally, there is a stage called Reinforcement Learning from Human Feedback (RLHF), which is used to train reasoning models.

Useful Links:
🤖 LLM Trainer library: https://github.com/Skripkon/llm_trainer
🔑 Play around with tokenizers: https://tiktokenizer.vercel.app/
🎭 Read about Masked Language Modeling (MLM): https://arxiv.org/pdf/1810.04805
🌐 Article about the FineWeb Dataset: https://huggingface.co/spaces/Hugging...

Timecodes:
00:00 - Intro
00:30 - llm_trainer overview
02:00 - Preparing a dataset
04:40 - How tokenizers work
08:18 - llm_trainer library structure
08:55 - create_dataset function
13:27 - DataLoader
18:20 - LLMTrainer class
28:15 - GPT-2 example
30:45 - xLSTM example
33:58 - Base & Chat models, SFT
34:59 - Outro


#ai #llm #nlp #LLMTraining #MachineLearning #PythonLibrary #llm_trainer
#AITraining #ModelTraining #Tokenizers #PreTraining #PostTraining #GPT2 #xLSTM #FineTuning #LanguageModeling #ArtificialIntelligence #TechTutorial #DeepLearning #MLM #AIResearch #DataScience #AIExplained