Superintelligent Agents Pose Catastrophic Risks: Can Scientist AI Offer a Safer Path?

By Yoshua Bengio et al.

This paper argues that building generalist AI agents poses catastrophic risks, from misuse by bad actors to a potential loss of human control. As an alternative, the authors propose “Scientist AI,” a non-agentic system designed to explain the world through theory generation and question-answering rather than acting in it. They suggest this path could accelerate scientific progress, including in AI safety, while avoiding the dangers of agency-driven AI.

Source:

https://arxiv.org/pdf/2502.15657

A podcast by BlueDot Impact.

Learn more on the AI Safety Fundamentals website.

Chapters

• 0:27 Abstract
• 2:31 1 Executive summary
• 2:35 1.1 Highly effective AI without agency
• 9:51 1.2 Mapping out ways of losing control
• 15:30 1.3 The Scientist AI research plan