Babak Hassibi: "Implicit and Explicit Regularization in Deep Neural Networks"

Tensor Methods and Emerging Applications to the Physical and Data Sciences 2021
Workshop IV: Efficient Tensor Representations for Learning and Computational Complexity

"Implicit and Explicit Regularization in Deep Neural Networks"
Babak Hassibi - California Institute of Technology

Abstract: Deep learning has demonstrably enjoyed a great deal of recent practical success and is arguably the main driver behind the resurgent interest in machine learning and AI. Despite its tremendous empirical achievements, we are far from a theoretical understanding of deep networks. In this talk, we will argue that the success of deep learning is not only due to the special deep architecture of the models, but also due to the behavior of the stochastic descent methods used, which play a key role in reaching "good" solutions that generalize well to unseen data. We will connect learning algorithms such as stochastic gradient descent (SGD) and stochastic mirror descent (SMD) to work in H-infinity control in the 1990's, and thereby explain the convergence and implicit-regularization behavior of the algorithms when we are highly over-parametrized (what is now being called the "interpolating regime"). This gives us insight into why deep networks exhibit such powerful generalization abilities, a phenomenon now being referred to as "the blessing of dimensionality". The theory also allows us to construct a new algorithm called regularized SMD (RSMD) which can provable explicitly regularize and has far superior generalization performance over SGD for noist data sets.

Institute for Pure and Applied Mathematics, UCLA
May 18, 2021

For more information: https://www.ipam.ucla.edu/tmws4