The Long Story of How Neural Nets Got to Where They Are: A Conversation with Terry Sejnowski

Stephen Wolfram plays the role of Salonnière in this on-going series of intellectual explorations with special guests. Watch all of the conversations here: https://wolfr.am/youtube-sw-conversat...

Originally livestreamed at:   / stephen_wolfram  
00:00 Start stream
5:26 SW starts talking
5:39 When did people first realize there were neurons in the brain?
6:30 Before the discovery of neurons, what did people think the brain was made of?
7:28 How did people figure out things in the brain had to do with electricity?
9:22 After discovery of electrical signals in the brain, where there huge developments of people studying and dissecting the brain?
10:47 When did people building machinery and people studying the brain connect?
13:17 The big moment for neural nets, logical structuring, the brain might be computing like a computer
18:47 Developments in the 40s and 50s- perhaps neural nets were original idea of what computers might be like
22:30 Developments in the 60s and 70s - Computers can prove theorems so what else can they do? Early development of machine translation projects.
29:09 Going back to McCulloch Pitts 1943 and idealized artificial neurons, Neuron structure and function in the human brain
35:04 Can artificial neural net weight matrices and biases capture characteristics of neurons and dendritic trees found in the brain?
47:26 Japanese advancements in neural networks in the 60s and 70s: Neocognitron, Linear predicates, Precursors of Hopfield networks
51:06 Early 60s resurgence of interest in artificial networks and intelligence.
1:03:07 Historical involvement of physicists in neural network development, communities coming together to figure out what was going on in the brain and how patterns of information can be stored in neural nets
1:18:50 Developments in the 80s and precursors to Boltzmann Machines
1:32:16 Developments in neural networks between the mid 1980s until 2011/2012
1:55:17 How did research change with the beginning of 2010s?
2:07:16 What's the next stage of neural nets? How is ChatGPT involved?
2:12:08 Looping Procedures and Learning Methods
2:17:28 Do you think there can be a human understandable theory of what you're seeing with dimensions and mathematics?
2:23:40 2012 breakthrough
2:26:19 Discussing project with Gerald Tesauro
2:23:27 What does solve the problem mean?
2:36:40 Was Dennis Gabor's 1959 'Electronic Inventions and their Impact on Civilisation' the first proposal of random circuits with weight tuning (to learn a black box function)?
2:40:28 What does he think of Robert Hecht Nielsen's Confabulation Theory as a top-down approach to the neocortex?
2:42:44 The HBP or Human Brain Project is worth Your attention in the future, It is amazingly complex plan to simulate how the Brain functions starting from simulating the Brain of the mice.
2:45:26 Terry mentioned that current models are about 3 or 4 orders of magnitude less compute than the brain - how many orders of magnitude in algorithmic development (backprop etc.) does he estimate we are?
2:49:42 One of the challenges with neural networks is that they can end up solving problems in ways that aren't intuitive to humans, and this results in issues with both trust and unexpected behaviour in edge cases or extrapolation beyond the training set. Is there much progress in "guiding" neural networks so they behave in a more intentional, human-like way?
2:51:42 What is the future? Reviewing the history.

Follow us on our official social media channels.

Twitter:   / wolframresearch  
Facebook:   / wolframresearch  
Instagram:   / wolframresearch  
LinkedIn:   / wolfram-research  

Contribute to the official Wolfram Community: https://community.wolfram.com/
Stay up-to-date on the latest interest at Wolfram Research through our blog: https://blog.wolfram.com/
Follow Stephen Wolfram's life, interests, and what makes him tick on his blog: https://writings.stephenwolfram.com/