Superintelligent Agents Pose Catastrophic Risks: Can Scientist AI Offer a Safer Path?

Audio versions of blogs and papers from BlueDot courses.

By Yoshua Bengio et al.

This paper argues that building generalist AI agents poses catastrophic risks, from misuse by bad actors to a potential loss of human control. As an alternative, the authors propose “Scientist AI,” a non-agentic system designed to explain the world through theory generation and question-answering rather than acting in it. They suggest this path could accelerate scientific progress, including in AI safety, while avoiding the dangers of agency-driven AI.

Source:

https://arxiv.org/pdf/2502.15657

A podcast by BlueDot Impact.

Chapters

• 0:27 Abstract
• 2:31 1 Executive summary
• 2:35 1.1 Highly effective AI without agency
• 9:51 1.2 Mapping out ways of losing control
• 15:30 1.3 The Scientist AI research plan