What The Bot with Reuben Adams

David Deutsch is a phenomenal thinker on AI, Physics and Philosophy. But his views on AI risk puzzled me. So I sat down with him to discuss in more detail why he believes AGIs will pose no greater risk than humans.

David Deutsch is author of The Fabric of Reality and The Beginning of Infinity. Both are phenomenal works and thoroughly engrossing. I can't recommend reading them enough.

Timestamps:
00:00 - Will AIs Be Smarter Than Humans?
03:53 - Could Faster AIs Outcompete Us?
06:24 - Will Augmented Humans Keep Up With AI?
15:19 - Can Creativity Be Sped Up?
25:51 - Will AGIs Be People?
27:46 - Do Qualia Determine Morality?
30:54 - Would AGI and Human Values Converge?
40:10 - How Do We Test Moral Theories?
51:30 - Would AGI Care About Morality?
56:06 - Would Simulation Enable Moral Convergence?
1:04:13 - Do Moral Arguments Always Change Our Actions?
1:08:50 - Do AGIs and Teenagers Present The Same Problem?
1:14:13 - Would AGIs Start Off With Our Values?
1:19:22 - Do The Starting Values Matter?
1:21:41 - THE ORTHOGONALITY THESIS AND HITLER
1:31:34 - Maybe AGIs Will Suck At Morality?
1:40:39 - We're Going To Make Mistakes.

AI is jagged - superhuman at some things, terrible at others.

This is why talk of "AGI" is so misleading. Focusing only on failures misses 90% of the picture, and could leave you unprepared for AI progress.
JaggednessThe original jaggedness diagram: https://substack.com/@tomaspueyo/note/c-182052822The AI VillageHomepage: https://theaidigest.org/villageLatest on X: https://x.com/aidigest_Merch store created by the village: https://ai-village-store.printful.me/In-person event organised (mostly) by the village: https://theaidigest.org/village/blog/season-2-recap-ai-organizes-eventTokenizationExplore tokenization yourself: https://tiktokenizer.vercel.app/AlphaFoldThe original paper: https://www.nature.com/articles/s41586-021-03819-2DanishChatGPT speaking fluent Danish while insisting it can’t speak Danish: https://www.reddit.com/r/GPT3/comments/zb4msc/speaking_to_chatgpt_in_perfect_danish_while_it/The ApprenticeNadia being fired: https://www.youtube.com/watch?v=Wf-o4nev2Bw

I thought the Turing Test was between a person and a computer. But Turing’s paper seems to imply it’s between a man and a woman, and then a computer and a woman. So what really is the Turing Test? And is it a good measure of intelligence anyway?

Sources

Alan Turing: Computing Machinery and Intelligence, 1950
 https://phil415.pbworks.com/f/TuringComputing.pdf

Gualtiero Piccinini: Turing’s Rules for the Imitation Game, 2000
https://www.researchgate.net/publication/251383110_Turing's_Rules_for_the_Imitation_Game

Judith Genova: Turing’s Sexual Guessing Game, 1994
https://www.tandfonline.com/doi/abs/10.1080/02691729408578758

EDSAC: https://commons.wikimedia.org/wiki/File:EDSAC_(25).jpg

Jimmy Kimmel Clip: https://www.youtube.com/watch?v=earRJKrE8Bw

EDSAC iPhone13 Comparison

EDSAC, from Wiki:
“Cycle time was 1.5 ms for all ordinary instructions”

It looks like addition was one of the “ordinary instructions.”

“Numbers were either 17 bits (one word) or 35 bits (two words) long.”

My understanding is that 35 bits would take two operations to add, so I’ve stuck to adding two 17 bit numbers, which can be done in one floating point operation.

“The first calculation done by EDSAC was a program run on 6 May 1949”

From then to Christmas day 2025 is 27992 days = 2418508800 seconds

So the number of 17 bit numbers the EDSAC could add in the time period is

2418508800 s / 1.5 ms = 1612339200000

iPhone 13, from Wiki:
The GPU runs at 1.37 TFLOPS (tera FLOPS), so 1.37 * 1012 FLOPS.

Let’s assume adding two 17 bit numbers takes 1 FLOP.

Then adding 1612339200000 17 bit numbers can be done in

(1612339200000 additions) / (1.37 * 1012 additions/s) = *1.176889 s.*

Suppose an AI “went rogue”.
Couldn’t we just switch it off?

How would it keep itself running without human help or an army of robots?

And why would AI necessarily be evil, rather than kind?

I put these questions to Zershaaneh Qureshi, a researcher at 80,000hrs.

Her article “Risks from power-seeking AI systems” is *the* best introduction to the debate on whether AI may one day be an extinction level risk.

What struck me most was this fantastic analogy:
“You could be a scholar of the antebellum south… You'll know everything about why slave owners believe that they were justified in owning slaves. But that definitely doesn't mean that you're going to think yourself that slavery is justifiable.”

This really drives home the fact that even if we manage to build AIs that understand human values, that doesn’t mean that they will adopt those values as their own.

Timestamps:
06:49 - Is Talk of AI Extinction Just Hype From AI Companies?
18:08 - Will AI Always Be Just a Tool?
26:26 - Can We Just Switch It Off If It “Goes Rogue”?
33:52 - The Challenge of Instilling the Right Goals
46:38 - Specification Gaming and Goal Misgeneralization
53:48 - Instrumental Goals: Self-Preservation and Power-Seeking
1:01:57 - Situational Awareness: Do AIs Need to Be Conscious?
1:08:53 - Why Would We Deploy Something This Dangerous?
1:11:48 - The Deception Problem: AIs Could Hide Their True Intentions
1:20:53 - Could AI Actually Take Over the Physical World?
1:36:26 - Have We Argued Ourselves to an Absurd Conclusion?

Sam Altman said 2025 is the year of agents.Andrej Karpathy said they’re slop.The AI Village is a team of AIs working together to do real work, like raising money for charity, creating websites to sell merchandise and even organising an in person event.But the project has shown that while AIs can now use computers, they fall over on the simplest tasks. Doing anything requires multiple attempts, with frequent comedic failures.Is this just the start of a technology that may soon revolutionise the economy? Or is it just more AI slop? To find out, I spoke to Adam Binksmith, CEO of AI Digest and co-creator of the AI Village.#ai #agi #agents