How AI Taught Itself to See [DINOv3]

How can we train a general-purpose vision model to perceive our visual world?

This video dives into the fascinating idea of self-supervised learning. We will discuss the basic concepts of transfer learning, contrastive language-image pretraining (CLIP), and self-supervised learning methods, including masked autoencoder, contrastive methods like SimCLR, and self-distillation methods like DINOv1, v2, and v3. I hope you enjoy the video!

00:00 Introduction
00:33 Why do features matter?
01:11 Learning features using classification
02:14 Learning features using language (CLIP)
04:09 Learning features using pretask (Self-supervised learning)
05:20 Learning features using contrast (SimCLR)
06:36 Learning features using self-distillation (DINOv1)
12:18 DINOv2
13:54 DINOv3

References:

Language-image pretraining
[CLIP] https://openai.com/index/clip/

Self-supervised learning (pretask):
[Context encoder] https://arxiv.org/abs/1604.07379
[Colorization] https://arxiv.org/abs/1611.09842
[Rotation prediction] https://arxiv.org/abs/1803.07728
[Jigsaw puzzle] https://arxiv.org/abs/1603.09246
[Temporal order shuffling] https://arxiv.org/abs/1708.01246

Contrastive learning
[SimCLR] https://arxiv.org/abs/2002.05709

Inpainting
[MAE] https://arxiv.org/abs/2111.06377
[iBOT] https://arxiv.org/abs/2111.07832

Self-distillation
[DINOv1] https://arxiv.org/abs/2104.14294
[DINOv2] https://arxiv.org/abs/2304.07193
[DINOv3] https://arxiv.org/abs/2508.10104

Self-supervised learning
[Cookbook] https://arxiv.org/abs/2304.12210

Video made with Manim: https://www.manim.community/