Is GPL the Future of Sentence Transformers? | Generative Pseudo-Labeling Deep Dive

🎁 Free NLP for Semantic Search Course:
https://www.pinecone.io/learn/nlp

Training sentence transformers is hard; they need vast amounts of labeled data. On one hand, the internet is full of data, and, on the other, this data is not in the format we need. We usually need to use a supervised training method to train a high-performance bi-encoder (sentence transformer) model.

There is research producing techniques placing us ever closer to fine-tuning high-perfomance bi-encoder models with unlabeled text data. One of the most promising is GPL. At its core, GPL allows us to take unstructured text data and use it to build models that can understand this text. These models can then intelligently respond to natural language queries regarding this same text data.

It is a fascinating approach, with massive potential across innumerous use cases spanning all industries and borders. With that in mind, let's dive into the details of GPL and how we can implement it to build high-performance LMs with nothing more than plain text.

🌲 Pinecone article:
https://www.pinecone.io/learn/gpl/

🔗 Notebooks:
https://github.com/pinecone-io/exampl...

🤖 70% Discount on the NLP With Transformers in Python course:
https://bit.ly/3DFvvY5

🎉 Subscribe for Article and Video Updates!
  / subscribe  
  / membership  

👾 Discord:
  / discord  

00:00 Intro
01:08 Semantic Web and Other Uses
04:36 Why GPL?
07:31 How GPL Works
10:37 Query Generation
12:08 CORD-19 Dataset and Download
13:27 Query Generation Code
21:53 Query Generation is Not Perfect
22:39 Negative Mining
26:28 Negative Mining Implementation
27:21 Negative Mining Code
35:19 Pseudo-Labeling
35:55 Pseudo-Labeling Code
37:01 Importance of Pseudo-Labeling
41:20 Margin MSE Loss
43:40 MarginMSE Fine-tune Code
46:30 Choosing Number of Steps
48:54 Fast Evaluation
51:43 What's Next for Sentence Transformers?