Applied Deep Learning 2025 - Lecture 8 - Transformers

In this lecture, we're diving into Transformers, which not only sound really cool, but have become a real game-changer, especially for sequence modeling like machine translation or text generation. The main driver behind transformers is the attention mechanism, which we will explore in detail. Some even say "Attention is all you need" - Let me know in the comments, if you agree with this statement, or rather with my reference.

Complete Playlist:    • Applied Deep Learning 2025 - Lecture 0 - P...  

00:00:00 - Start
00:00:40 - Recap
00:03:53 - Why Transformers?
00:09:36 - What is a Transformer?
00:11:10 - Input Embedding and Positional Encoding
00:14:20 - Attention
00:17:12 - Implementing Attention
00:21:34 - Masked Attention
00:24:06 - Cross-Attention
00:25:51 - The final layers
00:27:38 - Architecture Variations
00:30:11 - Transformers for Object Detection
00:36:28 - The Detection Transformer (DETR)
00:38:02 - Bipartite Matching Loss
00:47:33 - Object Queries
00:55:14 - Advances in Transformers
01:01:48 - Summary

== Literature ==
1. Halthor, Transformer Neural Networks Explained, 2020
2. Vaswani et al., Attention Is All You Need, 2017
3. Carion et al. End-to-End Object Detection with Transformers, 2020
4. Kilcher, End-to-End Object Detection with Transformers (Paper explanation), 2020
5. Pacha et al., A Baseline for General Music Object Detection with Deep Learning, 2018
6. Parmar et al. Image Transformer, 2018
7. Kilcher, Attention Is All You Need (Explained), 2017
8. Phi, Illustrated Guide to Transformers: Step by Step Explanation, 2020
9. Olah et al. Attention and Augmented Recurrent Neural Networks, 2016
10. Peters et al. Deep contextualized word representations, 2018
11. Howard et al. Universal Language Model Fine-tuning for Text Classification, 2018
12. Devlin et al. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding, 2018
13. Kitaev et al. Reformer: The Efficient Transformer, 2020
14. Wang et al. Linformer: Self-Attention with Linear Complexity, 2020
15. Wu et al. Pay less attention with Lightweight and Dynamic Convolutions, 2019
16. Dosovitskiy et al. An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale, 2021
17. Liu et al. Swin Transformer: Hierarchical Vision Transformer using Shifted Windows, 2021
18. Liu et al. Swin Transformer V2: Scaling Up Capacity and Resolution, 2022
19. Sun et al. Rethinking Transformer-based Set Prediction for Object Detection, 2021
20. Gildenblat, Exploring Explainability for Vision Transformers, 2021