Andrew Critch on what AGI might look like in practice

When people think about AGI, most of them ask “When is it going to arrive?” or “What kind of AGI will we get?”. Andrew Critch, AI safety researcher and mathematician, argues that the most important question is actually “What will we do with it?”

In our conversation, we explore the importance of our choices in the quest to make AGI a force for good. Andrew explains what AGI might look like in practical terms, and the consequences of it being trained on our culture. He also claims that finding the “best” values AI should have is a philosophical trap, and that we should instead focus on finding a basic agreement about “good” vs. “bad” behaviors.

The episode also covers concrete takes on the transition to AGI, including:
Why an advanced intelligence would likely find killing humans “mean.”
How automated computer security checks could be one of the best uses of powerful AI.
Why the best preparation for AGI is simply to build helpful products today.


00:00 Intro
00:54 Andrew Critch’s journey: from math to AI safety
04:49 What everyone gets wrong about extinction risk
06:30 Successionism: would humans care if we went extinct?
08:22 What will AGI actually look like?
11:55 The most important question: what will we do with AGI?
13:12 Visions of AGI going well: tool vs. collaborator
18:58 The "best" trap: why "good" alignment is better than "optimal"
25:17 Making AI a force for good via cultural training
26:43 Building helpful AI: NotADoctor and Multiplicity
29:56 Concrete ideas for high-impact AI work
31:54 Why products change culture faster than narratives
35:39 How to identify and build "good" products
48:54 Principles for the future: transparency & de-escalatory self-defense
57:48 Why we shouldn't just accept AI succession
59:28 Final advice: just build stuff


The full transcript and a complete list of resources can be found here: https://www.existentialhope.com/podca...