Data Augmentation using Pre-trained Transformer Models

This video explores sampling from pre-trained transformers to augment small, labeled datasets. This study compares the results of fine-tuning BERT, GPT-2, and BART for generating new data. Each technique has a distinct way of making sure the augmented data preserves the original class label such as positive or negative sentiment or a respective topic in a topic classification task. I think this is a really exciting use of generative models, showing that they are more useful than just being the first step of representation learning! Thanks for watching and please subscribe!

Paper Links:
Data Augmentation using Pre-trained Transformers: https://arxiv.org/pdf/2003.02245.pdf
Next Word Prediction Demo: https://github.com/renatoviolin/next_...
Conditional BERT for Contextual Augmentation: https://arxiv.org/pdf/1812.06705.pdf
BART: https://arxiv.org/pdf/1910.13461.pdf
T5: https://arxiv.org/pdf/1910.10683.pdf
GPT-3: https://arxiv.org/pdf/2005.14165.pdf
BERT: https://arxiv.org/pdf/1810.04805.pdf
GPT: https://s3-us-west-2.amazonaws.com/op...
ImageGPT (images used to describe AE vs. AR): https://cdn.openai.com/papers/Generat...
Classification Accuracy Score: https://arxiv.org/pdf/1905.10887.pdf
BigGAN: https://arxiv.org/pdf/1809.11096.pdf
Guide to using BERT (will help understand how label embedding would work): http://jalammar.github.io/a-visual-gu...
Conditional GANs: https://arxiv.org/pdf/1411.1784.pdf
SPADE (conditional batch norm example, albeit kind of an intense example): https://arxiv.org/pdf/1903.07291.pdf
Pre-training via Paraphrasing: https://arxiv.org/pdf/2006.15020.pdf
PEGASUS: https://arxiv.org/abs/1912.08777
Don't Stop Pretraining: https://arxiv.org/pdf/2004.10964.pdf

Chapters
0:00 Introduction
1:16 Labeling Data is difficult!
2:15 Data Augmentation in NLP
3:18 Contextual Augmentation
4:12 Conditional BERT
6:58 BERT vs. GPT-2 vs. BART
8:53 Data Augmentation Approach
10:00 How Data is Generated
11:08 Class Label in Vocabulary?
13:07 Experiment Details
13:38 Results Extrinsic Evaluation
14:18 Classification Accuray Score used for GANs, VAEs in images
14:45 Intrinsic Analysis
16:25 Connection to Don’t Stop Pretraining
17:17 Connection with MARGE, PEGASUS, ELECTRA
18:27 Connection with Pattern-Exploiting Training