Scientific Machine Learning and Stiffness - MIT Institute for AI and Fundamental Interactions IAIFI

Title: Scientific Machine Learning and Stiffness

Abstract:

Scientific machine learning (SciML) is the burgeoning field combining scientific knowledge with machine learning for data-efficient predictive modeling. We will introduce SciML as the key to effective learning in many engineering applications, such as improving the fidelity of climate models to accelerating clinical trials. This will lead us to the question on the frontier of SciML: what about stiffness? Stiffness is a pervasive quality throughout engineering systems and the most common cause of numerical difficulties in simulation. We will see that handling stiffness in learning, and thus real-world models, requires new training techniques. We will showcase how learning accurate models of battery degradation and the energy efficiency of buildings fails with previous techniques like physics-informed neural networks but succeeds with new stiffly-aware architectures like continuous-time echo state networks. This deep understanding of numerical issues in learning will feedback into traditional machine learning where we showcase how regularizing stiffness in a neural ODEs can halve the training time for image classification tasks. Together the audience will leave with a firm understanding of the role stiffness will play in the next decade of SciML.

Referenced papers:

Universal Differential Equations for Scientific Machine Learning: https://arxiv.org/abs/2001.04385
Bayesian Neural Ordinary Differential Equations: https://arxiv.org/abs/2012.07244
Opening the Blackbox: Accelerating Neural Differential Equations by Regularizing Internal Solver Heuristics: https://arxiv.org/abs/2105.03918
Stiff Neural Ordinary Differential Equations: https://arxiv.org/abs/2103.15341
ModelingToolkit: A Composable Graph Transformation System For Equation-Based Modeling: https://arxiv.org/abs/2103.05244
High-performance symbolic-numerics via multiple dispatch: https://arxiv.org/abs/2105.03949

For more information on scientific machine learning and the software for doing this research, see https://sciml.ai/

For more information on the MIT Institute for AI and Fundamental Interactions, see https://iaifi.org/