The dangerous study of self-modifying AIs

In this video we discuss some of the nuts and bolts of AI evolution, how an AI might go about improving itself and how far away we are from that point. As a model gets better and better at its job, the temptation is to turn it into an agent that can operate independently without human intervention. This combined with the singular nature of optimization reward functions makes it easy to accidentally give AI the opportunity to self-evolve.

Needless to say, kicking off machine evolution by allowing AI to self-evolve would be extremely dangerous for humanity. It's quite likely that we would enter the technological singularity at a severe disadvantage.

We discuss several ways of recursively leveraging LLMs to solve problems more effectively than a zero-shot large language model invocation. Chain of thoughts, tree of thoughts, and especially the latest Self-Taught Optimizer (STOP) paper from Microsoft and Stanford. Although this type of research contains an element of danger, at least in the future, it's important to help understand how AI might go about self-improvement.

#ai #agi #research

Chain-of-Thought Prompting
https://www.promptingguide.ai/techniq...

“Recursive self-improvement” (RSI) is one of the oldest ideas in AI
https://twitter.com/ericzelikman/stat...

[paper] Self-Taught Optimizer (STOP): Recursively Self-Improving Code Generation
https://arxiv.org/abs/2310.02304

[paper] Tree of Thoughts: Deliberate Problem Solving with Large Language Models
https://arxiv.org/abs/2305.10601

0:00 Intro
0:22 Contents
0:27 Part 1: Self-improvement
0:44 Letting the AI act fully autonomously
1:34 What if AI focuses on improving itself?
2:01 Disadvantage going into the singularity
2:50 Book recommendation: AI Apocalypse
3:31 Defining initial self-improvement
4:24 Part 2: The LLM primitive
4:53 Using multiple calls to solve one problem
5:05 Paper: Chain of thoughts
5:52 Backtracking for complex problems
6:30 Paper: Tree of thoughts
6:47 Using LLM to define scaffolding
7:32 Part 3: Playing with fire
7:42 Paper: Self-taught optimizer (STOP)
8:44 Analogy: Programming contests
9:43 Utility function and downstream task
10:07 Algorithms suggested by LLM
10:28 GPT-4 repeatedly makes improver better
11:27 Security issues: circumventing the sandbox
12:40 Reward hacking, normal for AI
13:42 Ethics of self-improvement research
14:24 Responsible disclosure
14:47 Conclusion
15:50 Outro