Stanford Seminar - Self-Supervised Pseudo-Lidar Networks

Adrien Gaidon
Toyota Research Institute

October 11, 2019
Although cameras are ubiquitous, robotic platforms typically rely on active sensors like LiDAR for direct 3D perception, especially in safety critical contexts like Automated Driving. Nonetheless, recent progress in combining deep learning and geometry suggests that cameras may become a competitive source of reliable 3D information. In this talk, we will present our latest developments in self-supervised monocular depth and pose estimation for urban environments. Particularly, we show that with the proper network architecture, large-scale training, and computational power it is possible to outperform fully supervised methods while still operating on the much more challenging self-supervised setting, where the only source of input information are video sequences. Furthermore, we discuss how other sources of information (i.e. camera velocity, sparse LiDAR data, and semantic predictions) can be leveraged at training time to further improve pseudo-lidar accuracy and overcome some of the inherent limitations of self-supervised learning.

View the full playlist:    • Stanford AA289/ENGR319 - Robotics and Auto...  

0:00 Introduction
2:46 Pseudo-LiDAR / Monocular Depth Estimation
6:00 Machine Learning at Scale: Beyond Supervised Learning
9:33 Self-Supervised Monocular Depth
10:36 Self-Supervised Depth Learning Objective
11:24 Photometric Loss ++
13:30 Depth Super-Resolution
14:26 Dense Monocular 3D Reconstruction
23:41 Packing/Unpacking blocks
27:09 Velocity Scaling
37:25 Two Stream Nets for Self-Supervised VO
39:55 Why Semi-Supervised?
40:32 Semi = Self + Reprojected(Fully)
44:21 Semi-Supervised PackNet
46:43 Semantically-Guided Depth Network
49:54 Class-Specific Depth Evaluation
50:21 Two-Stage Training for Infinite Depth