The idea of “artificial superintelligence[1]” (ASI) has recently crossed from the realm of science fiction (in the mainstream perception) to the realm of near-term technological goals openly pursued by most leading commercial organizations in the field of AI. Unfortunately, ASI poses a grave existential risk to the human species. This was argued by Eliezer Yudkowsky since 2001[2], and by many others (see e.g., Omohundro 2009, Bostrom 2014, Armstrong 2014, Russell 2019, Bengio et al. 2024 (25 authors), Bengio et al. 2025 (13 authors) and Yudkowsky and Soares 2025).
In light of this risk comes the question of how can we make ASI safe[3]. This question is challenging on many levels, because to solve it we need to complete vast gaps in our understanding both in theoretical computer science and in the areas of philosophy concerned with knowledge and decision-making. Moreover, the cost of failure is so enormous that any sane solution has to meet unprecedented standards of reliability and robustness. Not to mention the sociopolitical problem of balance of power in the post-ASI age, something that is out-of-scope for this sequence.
Personally, I started to be concerned with this problem around 2013. In 2018, I created the learning-theoretic AI alignment research agenda (LTA). The LTA combined ideas from MIRI’s agent foundations agenda with the mathematical tools of statistical and computational learning theory in an attempt to create a mathematical theory of intelligent agents. Since then, my collaborators[4] and I have been making steady progress on realizing this vision. In 2023, I wrote an updated overview of the agenda, including a sketch of my proposed alignment protocol: Computational Superimitation. More recently I founded the Computational Rational Agents Laboratory (CORAL), a dedicated research organization[5].
This year has seen a breakthrough in a key missing piece of LTA: compositional learning theory (see Stringological Sequence Prediction, more on this in following posts). Thanks to this and other insights, we can already see the entire outline of the emerging mathematical framework, one that I named computational agent theory.
In this sequence, I am planning to cover:
What is the technical alignment problem and what are its core difficulties, in my view. This is a perspective that draws simultaneously on classical traditions in theoretical computer science / machine learning theory, ideas from analytic philosophy, and views that originated in the Machine Intelligence Research Institute / LessWrong.
A high-level overview of computational agent theory. This theory differs from other accounts of intelligence and/or rational agency by combining a mature treatment of the relevant philosophical problems (drawing on ideas such as structural realism, computationalism and functional decision theory) with the promise of extracting practical algorithms.
How computational agent theory leads to a solution of the technical alignment problem, something that requires a major rethinking of how we design AI systems.
The roadmap towards completing the theory, and our current location on this roadmap.
Some considerations (scientific, technological and social) about crossing the gap from a theoretical solution to AI alignment to a real-world solution to AI risk.
I thank Jeremy Gillen, Roman Malov, Vinayak Pathak and my beloved spouse @Marcus Ogren for reviewing drafts of this article and making helpful suggestions.
The term “superintelligence” is always used in this sequence to mean “intelligence far beyond that of the smartest humans”. It is never used as the farcical synonym to “artificial intelligence” proposed by Donald Trump. ↩︎
Although Alan Turing warned about machines taking control already around 1951, himself citing a fiction novel by Samuel Butler from 1872. ↩︎
Alternatively, we can ask why should we build ASI at all. In my opinion, it is in humanity’s interest to build ASI eventually, after the problem of safety is investigated so thoroughly as to leave very little room for error. However, our present technological trajectory is extremely far from this ideal, and it would be prudent for world governments to hit the brakes hard. ↩︎
The idea of “artificial superintelligence [1] ” (ASI) has recently crossed from the realm of science fiction (in the mainstream perception) to the realm of near-term technological goals openly pursued by most leading commercial organizations in the field of AI. Unfortunately, ASI poses a grave existential risk to the human species. This was argued by Eliezer Yudkowsky since 2001 [2] , and by many others (see e.g., Omohundro 2009, Bostrom 2014, Armstrong 2014, Russell 2019, Bengio et al. 2024 (25 authors), Bengio et al. 2025 (13 authors) and Yudkowsky and Soares 2025).
In light of this risk comes the question of how can we make ASI safe [3] . This question is challenging on many levels, because to solve it we need to complete vast gaps in our understanding both in theoretical computer science and in the areas of philosophy concerned with knowledge and decision-making. Moreover, the cost of failure is so enormous that any sane solution has to meet unprecedented standards of reliability and robustness. Not to mention the sociopolitical problem of balance of power in the post-ASI age, something that is out-of-scope for this sequence.
Personally, I started to be concerned with this problem around 2013. In 2018, I created the learning-theoretic AI alignment research agenda (LTA). The LTA combined ideas from MIRI’s agent foundations agenda with the mathematical tools of statistical and computational learning theory in an attempt to create a mathematical theory of intelligent agents. Since then, my collaborators [4] and I have been making steady progress on realizing this vision. In 2023, I wrote an updated overview of the agenda, including a sketch of my proposed alignment protocol: Computational Superimitation. More recently I founded the Computational Rational Agents Laboratory (CORAL), a dedicated research organization [5] .
This year has seen a breakthrough in a key missing piece of LTA: compositional learning theory (see Stringological Sequence Prediction, more on this in following posts). Thanks to this and other insights, we can already see the entire outline of the emerging mathematical framework, one that I named computational agent theory.
In this sequence, I am planning to cover:
I thank Jeremy Gillen, Roman Malov, Vinayak Pathak and my beloved spouse @Marcus Ogren for reviewing drafts of this article and making helpful suggestions.
The term “superintelligence” is always used in this sequence to mean “intelligence far beyond that of the smartest humans”. It is never used as the farcical synonym to “artificial intelligence” proposed by Donald Trump. ↩︎
Although Alan Turing warned about machines taking control already around 1951, himself citing a fiction novel by Samuel Butler from 1872. ↩︎
Alternatively, we can ask why should we build ASI at all. In my opinion, it is in humanity’s interest to build ASI eventually, after the problem of safety is investigated so thoroughly as to leave very little room for error. However, our present technological trajectory is extremely far from this ideal, and it would be prudent for world governments to hit the brakes hard. ↩︎
See the names in CORAL’s publication list. ↩︎
Full-time staff currently consists of Alexander Appel, Vinayak Pathak, Matthias Georg Mayer and myself. ↩︎