Gradient Descent vs Evolution | How Neural Networks Learn

Explore two learning algorithms for neural networks: stochastic gradient descent and an evolutionary algorithm known as a local search. They fundamentally solve the same problem in similar ways, but one has the advantage. Step-by-step they find a way down Loss Mountain. Watch real neural networks maximize the fitness of curve fitting. We've got Dogson here!

Special thanks to Andrew Carr(https://x.com/andrew_n_carr) and Josh Greaves for reviewing this with their human neurons, and to the artificial neurons of Grok, o3 mini, and claude. Grok thought the gay joke was funny, o3 thought it wasn't inclusive lol. it is inclusive!

~Webtoys~
Hill Climbers: https://neuralpatterns.io/hill_climbe...
Neuron Tuner: https://neuralpatterns.io/nn_tuner.html
Subscribe to my music guy NOW:    / @acolyte-compositions  

~Links~
Patreon:   / emergentgarden  
Kofi: https://ko-fi.com/emergentgarden
My Twitter:   / max_romana  
My Bluesky: https://bsky.app/profile/emergentgard...
My Other NN videos:    • Neural network Learns  
Webtoy Source: https://github.com/MaxRobinsonTheGrea...
Animation Source: https://github.com/MaxRobinsonTheGrea...
Image Approximators: https://github.com/MaxRobinsonTheGrea...
FUNCTIONS DESCRIBE THE WORLD:    • On Mathematical Maturity (1) Thomas G...  
Dawkins Climbing Mount Improbable:    • Richard Dawkins demonstrates the evol...  
But he's gay:    • Gay Mount Everest  

~Citations~
Unfortunately many of these are behind paywalls
NNs are Universal Function Approximators: https://www.cs.cmu.edu/~epxing/Class/...
Backpropagation: https://www.nature.com/articles/323533a0
Loss Surfaces of MLPs: https://arxiv.org/abs/1412.0233

~Timestamps~
(0:00) Learning Learning
(1:20) Neural Network Space
(3:40) The Loss Landscape
(7:21) The Blind Mountain Climber
(8:37) Evolution (Local Search)
(13:07) Gradient Descent
(18:40) The Gradient Advantage
(20:48) The Evolutionary (dis)advantage