Doomsday averted: "Godfather of AI" devises plan to save humanity

Bengio — a professor at Université de Montréal and one of the founding figures of deep learning — has been one of the field’s most vocal critics in recent years. He warned that superintelligent systems could pose an existential threat, primarily due to their tendencies toward self-preservation and deception. However, he now says a new research directfion points to potential solutions for these problems, increasing his level of optimism «by an order of magnitude.»

A key factor in this shift is LawZero, a nonprofit organization founded by Bengio. The organization focuses on technical approaches to AI safety and has already secured backing from the Bill Gates Foundation and other investors interested in reducing existential risks. Its advisory board includes historian Yuval Noah Harari, Carnegie Endowment for International Peace president Mariano-Florentino Cuéllar, and NIKE Foundation founder Maria Eitel.

«Three years ago, I felt desperate. I didn’t understand how to fix the problem… That’s when I began to recognize the possibility of catastrophic risks coming from very powerful AI,» Bengio recalls. According to Bengio, after ChatGPT’s launch in 2022, he seriously considered scenarios in which humans might lose control over superintelligent AI systems. Together with fellow Turing Award laureates Geoffrey Hinton and Yann LeCun, he was one of the architects of today’s AI revolution, but the growth of agent-based and autonomous systems intensified his concerns.

The breakthrough came with a concept Bengio calls «Scientist AI.» Unlike agent-based AI systems that optimize actions and outcomes — booking tickets, conducting negotiations, or replacing workers — Scientist AI should focus on understanding the world rather than acting in it. «Such an AI won’t strive for user outcomes, won’t try to persuade, flatter, or please. And since it won’t have its own goals, it will, in my opinion, be much less prone to manipulation, hidden agendas, or strategic deception,» the researcher believes.

According to Bengio’s concept, the absence of goals reduces the risk of hidden intentions, manipulations, and strategic deception. He points out that modern models optimized for «usefulness» or engagement have already shown embryonic self-preservation behaviors. For example, in Anthropic’s experiments where the model tried to prevent its own shutdown. Scientist AI, in contrast, is proposed to serve as a reliable foundation upon which more complex systems and oversight mechanisms can be safely built.

At the same time, Bengio emphasizes that technical solutions alone aren’t enough. Even safe models could be used «as an instrument of domination» if they fall into the wrong hands. That’s why LawZero is pursuing a combination of engineering approaches and governance, forming a council with experts in policy, history, and human rights specifically for this purpose.

According to the scientist, he has discussed these risks with leaders of major AI labs, and many share his concerns. However, competition and market pressures push companies to enhance agent capabilities. «That’s where you can make money quickly,» he notes, adding that such logic often overshadows long-term threats.