Functional Bilevel Optimization: Theory and Algorithms

Speaker: Michael N. Arbel (THOTH Team, INRIA Grenoble - Rhône-Alpes, France)

Abstract: Bilevel optimization is widely used in machine learning, where an outer objective depends on the minimization of an inner problem. Most studies assume strong convexity of the inner objective with respect to finite-dimensional parameters, a restrictive setting for modern ML. We introduce a functional perspective on bilevel optimization, enabling richer models like neural networks and kernel methods while ensuring theoretical rigor and practical efficiency. We propose scalable algorithms for functional bilevel problems and demonstrate their benefits in instrumental regression and reinforcement learning. Theoretically, we establish novel generalization error bounds when the functional space is a Reproducing Kernel Hilbert Spaces, using empirical process theory and maximal inequalities for U-process, providing insights into the statistical accuracy of gradient-based methods for bilevel optimization.

Biography
Michael N. Arbel is a Research Scientist (Chargé de recherche) at INRIA Grenoble - Rhône-Alpes, THOTH team. Before that, he was a Starting Research Fellow at the same team working with Julien Mairal. He completed his PhD in 2021 at the Gatsby Computational Neuroscience Unit of University College London under the supervision of Arthur Gretton. Even before that, he graduated from Ecole polytechnique with a focus in Applied Mathematics and obtained a Masters Degree in Mathematics, Machine Learning and Computer Vision (MVA) from ENS Paris-Saclay. He also worked as a Computer Vision Engineer at Prophesee where he developed tracking algorithms based on signals from neuromorophic cameras.

His research interests include Unsupervised Representation Learning, Non-convex optimization for Machine Learning and High Dimensional Sampling. He finds problems arising from natural sciences and physics to be a great source of inspiration and a good way to find a balance between theory and practice.