Basketball AI with RF-DETR, SAM2, and SmolVLM2

Watch a full basketball possession analyzed frame by frame using a state-of-the-art AI vision pipeline built from scratch. You'll see how we combine powerful open-source models like RF-DETR for robust object detection, SAM2 for stable, pixel-level player tracking, and a specialized SmolVLM2 for reading blurred jersey numbers. Learn the exact process for identifying every player, classifying shot outcomes, and performing homography to map player movement onto a precise top-down court visualization for advanced tactical analysis. This video breaks down months of work into a complete, reusable computer vision system for sports analytics.

00:00 Project Overview
02:04 Detect Players and Numbers with RF-DETR
07:12 Track Players with SAM2
11:59 Team Clustering with SigLIP, UMAP and K-means
16:31 Fine-Tuning SmolVLM2
21:56 Map Player Positions to Court Coordinates
31:51 Detect Shot Event and Classify Result
35:54 Conclusions

Resources:

“How to Detect, Track, and Identify Basketball Players with Computer Vision” Blogpost: https://blog.roboflow.com/identify-ba...

Basketball Player Object Detection Dataset: https://universe.roboflow.com/roboflo...

Basketball Court Keypoint Detection Dataset: https://universe.roboflow.com/roboflo...

Basketball Jersey Number OCR Detaset: https://universe.roboflow.com/roboflo...

How to Detect Track and Identify Basketball Players Notebook: https://colab.research.google.com/git...

Make or Miss - Jumpshot Detection: https://colab.research.google.com/git...

Fine-tune RF-DETR on Custom Dataset Notebook: https://colab.research.google.com/git...

Segment Video with SAM 2 Notebook: https://colab.research.google.com/git...

RF-DETR GitHub: https://github.com/roboflow/rf-detr

Supervision GitHub: https://github.com/roboflow/supervision

Sports GitHub: https://github.com/roboflow/sports

Stay updated with the projects I'm working on at https://github.com/roboflow and https://github.com/SkalskiP! ⭐