TLDR: I explain what the technical AI alignment problem is and what its core difficulties are, in my view. This is a perspective that draws simultaneously on classical traditions in theoretical computer science / machine learning theory, ideas from analytic philosophy, and views that originated in the Machine Intelligence Research Institute / LessWrong.
Minimal Viable Safety
The Task
The problem of making AI safe is underspecified if we haven't decided what the AI is supposed to do. For example, we can run any AI inside a homomorphic encryption box and never decrypt the output. This is probably perfectly safe[1], but also perfectly useless.
We can decide on an arbitrary task such as "construct Lean proofs for given mathematical theorems" and declare victory once we have safe AI that does it with superhuman skill[2]. However, that wouldn't address the danger. First, there would be incentives to use AI for tasks to which our solution doesn't apply. Second, even if we had precise safety guidelines for any task that anyone cares about, there would always be actors that ignore the guidelines, and we probably need Artificial Superintelligence (ASI) to reliably stop them.
If anyone, anywhere, creates an unrestricted unaligned ASI agent (more details about the word "agent" later), without an aligned ASI agent already in place to counteract it, then a catastrophic outcome is highly likely. I will not go into the reasons here: this has been discussed elsewhere at length, see for example the books I cited in the preface. Therefore, this event is something which we must never allow to happen. This is the main threat model that I set out to address.
Imagine a world in which anyone with sufficient resources has the ability to destroy the world. Like today, many will foolishly believe that instead of destroying the world, this act will bring them power and glory. Moreover, as computing hardware improves and new algorithms are developed, the amount of resources for this act will go down from year to year. Even assuming best-case regulation and international agreements, how long until someone does it? Can mere humans build a society which will be foolproof against such a risk over the course of generations? Resilient to any political upheavals that might come, and undoubtedly will? I find it vanishingly unlikely. Eventually we will need the help of ASI itself. Therefore, the minimal task that safe AI must be able to solve is "create a reliable global defense system against dangerous AI" (in the following, GDS).
The Problem
Consider the consequences of this task for the capabilities that the AI needs to have. It needs to reason about the entire planet[3]. It needs to reason about all aspects of the problem, including scientific, technological, social, political and moral[4]. It needs to have preferences about the state of the world. It needs to make long-term plans (guided by these preferences) that would play out decades, if not centuries, into the future. And it needs to do all that on a superhuman level.
In this light, here is how I (informally) define the "technical AI alignment problem". The problem is designing an AI system that can bring scalable (and ultimately superhuman) capability to bear on problems that require multi-domain reasoning and open-world long-term planning with far-reaching consequences, all the while staying aligned with the intentions and preferences of its human users.
The Methodology
What kind of methodology is appropriate for this problem? In the past, I argued that it requires emphasis on deep theoretical understanding, rather than the incremental trial-and-error approach that is employed in the AI industry today. I won't rehash the entire debate here, but let me recap the core argument.
The nature of the incremental approach is as follows: we build a system, then we find problems[5], then we make changes to fix these problems to the extent necessary for commercial viability in the current legal environment. We make some improvements; new problems arise (sometimes because of the improvements, sometimes because of new applications or other reasons); rinse and repeat. This approach utterly fails if the "problems" result in AI takeover and possibly human extinction: there is nobody left to fix them.
In order for the incremental approach to succeed in preventing catastrophe, we need to reach a point where no new substantial problems arise, even inside the lab (before the system is deployed). Moreover, we need to reach this point before AI capabilities are sufficient for AI to escape our control. There are few reasons to expect such a miracle. On the contrary, sufficiently advanced AI might understand that it is in its interest to "lie low" (i.e. avoid visibly misaligned behaviors) until its capabilities cross the relevant threshold, such that the problems will only become apparent too late.
In contrast, the sane approach is creating the theoretical tools necessary to predict and prevent the problems before they are empirically observed. These tools need to remain applicable as AI capabilities rise substantially beyond human, for whatever algorithms, architectures and training protocols are used to achieve these capabilities. Moreover, these tools need to be extremely reliable for us to rationally trust them with the safety of our entire species.
The Challenge
While the theory of machine learning and AI is an established field with ample literature, its current state falls well short of the challenge. The core difficulties can be summarized as follows:
Hypercomplexity: It is infeasible to explicitly specify user preferences outside narrow domains, and narrow domains are not enough. Nor is it possible to explicitly specify a satisfactory model of the world. Therefore, learning is necessary.
Alignment Grounding: When trying to learn user preferences, it is unclear what loss function or success metric can work outside narrow domains. Nor is it clear how to specify such a metric for metaphysically realist world-models, which are necessary here. Nor is it even clear what type of mathematical objects can formally describe these concepts.
Alignment Convergence: Even given a success metric, it is unclear how to produce guarantees for optimizing this metric, given the ubiquity of irreversible mistakes and the lack of a theory of generalization for powerful learners.
Side-Channel Resilience: The usual abstraction of "learner" is insufficient here, since we need to ensure the safety of internal computations rather than just external outputs.
In the following, we will examine these difficulties one by one. But first, let's discuss what we need to accomplish.
Alignment is Holistic
Per the discussion above, our interest lies in AI systems that have open-world preferences and make effective plans to advance these preferences. In particular, making such plans requires learning a sophisticated model of the world and updating the model and the plan as additional information becomes available. We will call such a system an AI agent.
In order for an AI agent to be aligned, it needs to have aligned preferences and beliefs, and follow an aligned decision-making algorithm on the basis of these preferences and beliefs. While discussions about AI alignment often focus on the "preferences" part, beliefs and decisions are equally important. (Although it's true that preferences bring extra wrinkles into the problem as they are, arguably, the most "subjective" component in the trio.)
In this context "aligned" preferences typically means preferences very similar to those of the user(s)[6], at least within some domain that we are somehow constrained to inhabit. On the other hand, "aligned" beliefs doesn't mean beliefs that are similar to those of the user: on the contrary, we expect the AI agent to acquire more accurate beliefs. They need to be beliefs that the user can rationally trust to be accurate. Similarly, "aligned" decision-making needs to be such that the user can rationally trust it to be in their own favor given aligned preferences and beliefs.
The philosophically astute reader will notice that at this point we are already in a world of trouble. What does it mean for the user to have preferences? Are there objectively rational theories of epistemology and decision-making, and if so, what are these theories? Can beliefs or decisions be rational to hold or make "on behalf" of the user even if the user themself doesn't understand why? On the other hand, a computer scientist might be tempted to ignore these difficulties and just forge ahead with a solution that seems pragmatically satisfying. I think there is a lot to be said for such a practical mindset, and in this article I will aspire to embrace it. Unfortunately, we will see that the philosophical difficulties are not easily brushed aside.
Returning to the trichotomy of preferences, beliefs and decisions, observe that it is not even possible to entirely separate the three. For us to ascribe preferences to a system, it must rationally form beliefs and make decisions on the basis of these preferences. For us to ascribe beliefs to a system, it must rationally make decisions on the basis of these beliefs in the context of its preferences. And the rationality of decisions can only be judged in the context of specific preferences and beliefs.
Imagine an AI agent that pursues the happiness of mankind and is irrationally convinced that every human who dies goes to Heaven, which is the best possible place for human happiness. Such an AI would be motivated to kill all humans, even though its preferences are arguably "aligned". Moreover, since the belief in Heaven is unfalsifiable, such an AI can still be extremely capable at acting on its motivation and hence extremely dangerous.
If this thought experiment seems strange, consider the following: our beliefs would seem very strange to people who lived centuries ago. Therefore, it is highly likely that the beliefs of an ASI, capable of intellectual progress much faster than humans, will (and should) quickly become strange to us. The question is whether these strange beliefs will be produced by a process that we should deem trustworthy nevertheless.
"Alignment" is meaningless outside the full context of epistemic and instrumental rationality. Therefore, any theory of alignment has to be, in particular, a theory of epistemic and instrumental rationality.
Hypercomplexity
Perhaps the most simplistic approach to ensuring that an AI agent has aligned preferences and beliefs is explicitly specifying the preferences and beliefs that the agent should have. However, this is intractable. Human preferences are notoriously complex and giving a precise and complete mathematical specification of them seems hopeless. (Yudkowsky calls this "complexity of value".) It seems no easier to accomplish than identifying images of cats by a handcrafted mathematical specification of "cat".
For weak agents, a practical solution can be explicitly specifying a proxy utility function that approximates our actual preferences within a particular domain. For example, you can imagine an agent designed to maximize customer reviews, ad clicks or stock value. Unfortunately, for sufficiently strong agents, this approach becomes prone to Goodhart's Curse: the maximum of any such utility function is almost always at a point where it strongly diverges from our true preferences. For example, to maximize customer reviews one can leave many fake five-star reviews and prevent anyone from leaving bad reviews. In fact, it's preferable to ensure maximal control of the universe to prevent anything or anyone else from interfering with the stream of reviews.
One approach to avoiding Goodhart's Curse is somehow restricting the domain within which the agent operates or its overall impact (see e.g., Armstrong and Levinstein). But this is incompatible with the GDS task.
Nor is it possible to directly specify the agent's beliefs. Even if we were satisfied with the agent having merely the sum total of humanity's current knowledge, formally specifying all of it would be a Herculean feat. Dalrymple et al. proposed to attempt it (or something close) nevertheless, relying on assistance from weak AI systems. However, even if that were accomplished, it would be unsatisfactory. Indeed, a crucial ingredient in superintelligence, perhaps the crucial ingredient, is the ability to create sophisticated world models. Therefore, it is dubious that true superintelligence can be achieved while hobbling the system to only use knowledge we already have.
It makes even less sense to directly specify the agent's decisions: if we knew what decisions it should make, we wouldn't need the agent in the first place. Maybe it does make sense to have a closed-form algorithm that computes the decisions from preferences and beliefs. However, this requires strong assumptions on the form these beliefs take—for arbitrary models, the planning problem is often NP-hard or even PSPACE-hard, if not worse. I think that imposing such an assumption is actually reasonable and necessary (more on that in the next posts). But in any case, we cannot just delegate the task to a heuristic, since that would preclude any strong guarantee about the outcome[7].
Breakdown of Learning
This section goes into more technical detail than the rest of the article, and can be skipped on a cursory reading.
Given that it's infeasible to directly specify aligned preferences and beliefs, it is a reasonable suggestion to learn them instead. To understand what would be involved, let's start by examining a general machine learning problem, where we need an algorithm with theoretical guarantees. We can decompose the problem into parts that I call grounding and convergence.
The grounding problem consists of the following:
Define a mathematical object which represents the unknown quantity we need to learn (for example, the parameters of an ideal classifier or of the environment MDP).
Determine what kind of data we are going to use for learning (in particular, this involves some protocol about how the learner interacts with the environment and receives feedback).
Identify a mathematical relationship between , and the decisions of the learner, based on the physical process which produces them. Typically, given and the learner policy we should be able to describe a set of possible data probability distributions[8].
Define a regret function which measures how much worse the learner is doing compared to some target benchmark (we hide the details of the interaction between and inside this notation). Typically, the benchmark is something like the performance of an optimizer that knows on the given task[9].
For a grounded machine learning problem, it is roughly clear what the optimal learner would look like, if we ignore computational constraints. Formally, given a prior over the parameter [10], we can try producing the policy
However, it is not clear whether the learner's best effort would actually be good enough. For this, we need to solve the convergence problem, namely:
Find a computationally feasible algorithm that produces or a satisfactory replacement thereof.
Prove that in the limit of increasing amount of data, learning actually succeeds. Formally, we can state it as
Prove a quantitative bound that shows this limit converges sufficiently fast in practice, even when exact minimization over is replaced by the feasible algorithm.
We will see that in the case of alignment, both grounding and convergence produce formidable challenges.
Alignment Grounding
What data can be used to learn the user's preferences? Many approaches have been proposed or attempted, for example:
Human ratings of agent behaviors.
Human-rated real or fictional examples of behaviors or situations.
Human influence on the environment (see Shah et al.).
The fundamental problem is that none of this data is a direct and accurate representation of human preferences. Humans make mistakes. Humans change their mind about what is preferable: sometimes because of new information, sometimes because of new experiences, sometimes just after thinking more about the question. Humans can prefer an outcome for terminal reasons (because it is valuable in itself) or for instrumental reasons (because it helps to bring about another outcome in the future), and none of this data even keeps track of the difference.
While it's clear that these sources of data have information about human preferences, accurately extracting this information requires understanding the precise relationship between human preferences and the process generating the data; whereas if this relationship is misspecified, then even perfect learning will produce misaligned preferences. And even if these misaligned preferences approximate the true preferences in some domain, Goodhart's Curse will push the agent to leave this domain and produce a catastrophic outcome.
For example, suppose that the agent is trained by human ratings of traces of agent actions and observations. An ideal learning process would then align the agent to the expected rating, conditional on the trace appearing in the training data. Such an agent is probably catastrophically misaligned for multiple reasons:
Human raters might be wrong in their judgment, and their judgment will systematically become worse in worlds radically transformed by AI.
Human raters are subject to manipulation by the agent. A superintelligence might be able to exploit this to an extreme degree.
Some action-observation traces are unlikely to appear in the training data for "normal" reasons, for example traces that show the agent executing an AI takeover in a manner that is infeasible to fake[12]. Therefore, if such a trace did appear in the training data, it was rated by some abnormal process, for example by the rogue AI itself.
We need a deep understanding of how human preferences are related to physical processes such that this relationship can be used as a source of ground truth. In order to be useful and precise, this understanding should be part of a mathematical theory. However, it is not even obvious what the type of "preferences" is. If we accept VNM theory, preferences should be described as a utility function. But what is the domain of this function? In computer science, it is popular to think of the domain as the action-observation trace of the agent. This, however, is deeply unsatisfying. First, such utility functions cannot be compared between different agents (each agent has its own trace), so "alignment" doesn't make any sense for them. Second, there is a strong case that humans have preferences about the physical reality itself, rather than merely their subjective experience of this physical reality.
What about beliefs? Is there an easy-to-specify ground truth for them? The answer is "yes" if the domain of beliefs is supposed to be traces, since the trace is directly observable by the agent. However, if preferences need a domain other than traces, then so do beliefs. Moreover, how can we make sense of belief alignment if the agent's beliefs are incomparable with the user's beliefs? The traditional computer science approach here is akin to logical positivism in analytic philosophy, which we should abandon in favor of metaphysical realism (more on that in the next posts).
Alignment Convergence
Even if we knew how to ground the alignment learning problem, we'd face another problem: learning fails (i.e. doesn't converge at all) when irreversible mistakes are possible. Indeed, learning requires exploration[13], but in the real world unlimited exploration produces irrecoverable mistakes. For example, if a robot explores what happens when it jumps out of a 10th-floor window, it won't have the chance to learn from this mistake. On the other hand, there is, strictly speaking, no way to be certain that jumping out of a window is a bad idea if you never try it. This is why regret bounds in reinforcement learning theory always require an assumption such as episodicity (the environment resets after every "episode") or communication (any state of the environment's Markov Decision Process is reachable from any other state), which rules out irreversible mistakes by fiat.
This observation is not merely a technical nitpick. On the contrary, the existence of irreversible mistakes is a core feature of the world that makes the alignment problem difficult. Indeed, the alignment problem itself is our attempt to avoid the irreversible mistake of creating unaligned superintelligence. Any AI agent faces its own version of the problem when considering the creation of a superior successor agent. A GDS agent can commit an irreversible mistake by allowing a rival superintelligence to emerge. And of course, there are more prosaic examples, such as killing a person or failing to prevent their death[14].
The typical theoretical approach to navigating irreversible mistakes is Bayesian planning. Namely, we postulate a prior over the space of possible environments and then choose a policy with maximal expected long-term utility with respect to this prior. In theory, this produces an optimal balance between exploration, exploitation and caution. However, this leads to the difficult question of what prior to use[15]. Another question is how to implement Bayesian planning, given that it's computationally intractable in any remotely realistic setting[16]. Arguably, we should reconcile Bayesian planning with the machine learning paradigm by approximating Bayesian planning via meta-learning, but the theoretical foundation for such a method is an intricate open problem (see the next posts for more details).
If so, the hope is that we can reduce planning to learning, not via conventional reinforcement learning, but via a more sophisticated meta-learning paradigm. But, granting this reduction, we need to face yet another problem. Namely, we have no useful theory of generalization for highly expressive learning algorithms (such as deep learning), i.e. algorithms that can efficiently learn sophisticated compositional models.
One may ask: isn't it enough that deep learning works in practice? Why do we need a theory of generalization? Three reasons:
We need to know that, under verifiable assumptions, our grounding method will provide the agent with sufficient knowledge about the relevant user attributes, in time for the agent to make the right decisions.
We can't just add data and compute and tweak the algorithm until it works, because an intermediate point in this process might produce catastrophic results. For example, some humans are smart enough to build ASI but not smart enough to either make sure it's aligned or not to build it. As a user, I would not like my agent to inhabit a similar epistemic state, even if the agent would be fully aligned with more scaling.
Solving the grounding problem requires a theory of intelligent agency that applies to humans, and since learning is a core ingredient of intelligence, the theory of agency requires a theory of generalization as one of its pillars.
Side-Channel Resilience
It is standard in computer science to focus on the input-output behavior of an algorithm and its resource cost but ignore other side-effects of its execution. We care that we get the right answer and that it doesn't cost too much time, space, etc., but we usually don't care how it heats the hardware or what kind of electromagnetic effects it creates. Typically, we also don't worry about whether it happens to trigger a bug in the operating system or other software scaffolding around it. Indeed, such considerations are not even possible without knowing the full software and hardware context in which the algorithm is implemented. Moreover, computing hardware, operating systems, etc. are intentionally designed to support this abstraction.
It is well known that this way of thinking can fail when it comes to cybersecurity (see e.g., Zhou and Feng 2005, Standaert 2009, Randolph and Diehl 2020). There are many clever ways in which attackers can exploit bugs and side-effects to create outcomes which are not possible in the abstract, "denotational" model of the system. In the context of the alignment problem, it is not external malign actors that we worry about[17] but attacks coming from within the agent itself.
In the context of the theory of agents, classical abstractions lead to behaviorist interpretations. Any guarantee about the preferences, beliefs and decisions of the agent would reference only the actions that the agent takes through its intended interface. However, such guarantees might not reflect the actual relevant attributes of the agent. We can imagine a superintelligent agent that intentionally produces only benign behavior across its intended interface, while simultaneously conducting a malign side-channel attack[18]. Such an agent can satisfy behaviorist formal safety guarantees and remain catastrophically dangerous in practice. For example, think of a robot that never takes a malign action with its actuators, but runs an internal computation which causes an unaligned successor agent to be copied into a different system.
This is not to say that we are to abandon abstraction. Indeed, it is implausible that the solution to the alignment problem must or should depend on, e.g., the details of the hardware or the operating system. Instead, we need to develop new abstractions which would be better suited for the problem at hand, abstractions that go beyond pure behaviorism.
Inner and Outer Alignment
So far I have described my preferred perspective on the alignment problem. A different frame, popular in the AI alignment community, is "inner and outer alignment". IMO the inner/outer alignment frame is less precise, but can still be a useful source of intuition. It is hence worthwhile to understand the connection between these views, and in particular to see how my perspective captures the "mesa-optimizer" threat model.
First, let me briefly recap what a mesa-optimizer is (following Hubinger et al. 2019). For the purpose of this argument, we are thinking of the agent as an episodic reinforcement learning algorithm. Then, a mesa-optimizer is a learned policy which is an agent in itself. Crucially, the mesa-optimizer might have a reward function different from that of the outer-loop reinforcement learner. There are two reasons why this might happen:
The inner (mesa-optimizer) reward might have been close to the outer reward on previous episodes, which "fooled" the outer-loop learning algorithm. This is consistent with the inner reward drastically departing from the outer reward in yet-unexplored regions of state-space.
The mesa-optimizer might be planning on a time horizon much longer than a single episode. In this case, it has the incentive to intentionally hide its misaligned preferences in order to be stabilized by the outer loop.
If the outer-loop learning algorithm is allowed to continue, it will either eventually remove the mesa-optimizer or force the mesa-optimizer to forever display behavior aligned with the outer reward. However, if the environment allows irreversible processes (as the real world does), then the mesa-optimizer can bide its time until an opportune moment to irreversibly push the system into a region of state-space favored by the inner reward (and typically unfavored by the outer reward). This is closely related to the "sharp left turn" of Soares 2022.
Under this model, "outer alignment" means that the outer reward function is close to that of the user, whereas "inner alignment" means that the inner reward function is (globally) close to the outer reward function. Each of them is a non-trivial problem.
A possible real-world example of mesa-optimization is the evolution of Homo sapiens. Arguably, we can model evolution as a reinforcement learning algorithm whose reward function is reproductive fitness. Evolution created humans who are agents (and in particular reinforcement learners), but the human reward function is different—if all we cared about were reproductive fitness, we would never have invented birth control (or at least, it would be much less popular). Given enough time, natural selection should be able to "fix" this, but it's probably never going to happen because human society and technology evolve much faster towards other ends, which is irreversible from evolution's vantage point.
Would addressing the problems in the previous sections address inner alignment as well? I argue that it would. If we had a guarantee that the agent's attributes are aligned, then, in particular, it wouldn't be controlled by a misaligned mesa-optimizer. These guarantees are supposed to take irreversible processes into account, so the "sharp left turn" is ruled out. These guarantees are supposed to include both direct effects and side-effects, ruling out both attack vectors for mesa-optimizers (in the terminology of the original LTA article, direct effects correspond to "Cartesian daemons" and side-effects to "non-Cartesian daemons").
More constructively, the outer vs. inner alignment dichotomy approximately corresponds to alignment of preferences vs. alignment of beliefs and decisions. Indeed, if we think of the outer loop as model-based reinforcement learning, then the learned policy comes from a learned model, in which case a misaligned inner agent corresponds to a misaligned belief. (For example, Christiano's argument that "the universal prior is malign" can be viewed as describing such a misaligned belief.) Alternatively, if we think of the outer loop as model-free reinforcement learning, then the misaligned inner agent corresponds to misaligned decision-making (although I think that the model-based view is more useful).
Conclusion
We have seen that solving the alignment problem requires a body of theory that can tackle the core challenges of the problem: alignment grounding, alignment convergence and side-channel resilience, in the context of holistic alignment and hypercomplexity. This taxonomy is not necessarily intended as a decomposition where each subproblem is possible to solve separately—while thinking of them as subproblems can be fruitful, ultimately they are interconnected. (For example, alignment grounding requires modeling the user as an agent, which requires understanding how agents deal with irreversible mistakes, which is a subproblem of alignment convergence.) It is intended as a set of lenses that must be applied to any proposed solution in order to validate its viability.
In the following posts we will discuss the emerging computational agent theory, and see how it is building the tools to address all of these difficulties.
I thank Daniel Carmon, Fernando Garcia, Jeremy Gillen, Matthias Georg Mayer, Vinayak Pathak and my beloved spouse @Marcus Ogren for diligently reviewing drafts. Their feedback greatly improved this article. All remaining flaws are entirely my own fault.
Assuming post-quantum cryptography, it takes exponential (in the security parameter) computational resources to distinguish between the ciphertext of the AI and the ciphertext of a random program, and given an appropriate security parameter, it is physically impossible for the observable universe to do that much computation. If the secret key is thermodynamically impossible to recover, then the homomorphic execution of the AI will have similar physical effects as the homomorphic execution of a random program. This, assuming standard complexity-theory conjectures and that our understanding of physics is broadly correct. ↩︎
Even this is not obviously trivial. For example, if the AI is trained on natural language then it can create a model of the physical world, which it can use to encode a sophisticated attack in the proof it produces. It might be motivated to perform such an attack because of a goal it randomly acquired during training, when it had no effect on the loss. ↩︎
Probably the entire universe, if it's meant to keep us safe into the future. ↩︎
Some have argued that a solution with more limited scope is possible, where we just outsource some purely "technological" part to the AI. Yudkowsky's tongue-in-cheek example is "burn all GPUs". However, I haven't seen a proposal of that form that survives scrutiny. ↩︎
If there are multiple users, then there should be some kind of preference aggregation between them, making social choice theory relevant. This is mostly out of scope for this article. ↩︎
Dalrymple et al.'s proposal is for the AI to output certifiably safe plans. One immense difficulty with this is the creation of a satisfactory specification of safety. But even granting that, the problem of creating effective and certifiably safe plans might be much harder than the problem of creating effective plans, making any such AI agent either severely hobbled or a non-starter entirely. ↩︎
For example: In realizable offline learning, contains every distribution under which the data points are IID and consistent with the hypothesis . In agnostic learning, we drop the consistency requirement. In online learning, we drop the IID requirement. In reinforcement learning, contains the unique distribution generated by MDP interacting with policy . ↩︎
For example: In realizable offline/online learning, is the loss. In agnostic learning, it is the difference between the actual loss and the loss of the hypothesis . In reinforcement learning, it is the usual regret. ↩︎
We can consider a more general setting, such as a set of priors with adversarial choice. For notational simplicity, we stick to the Bayesian variant in this respect. ↩︎
This is known as Inverse Reinforcement Learning (see e.g., Arora and Doshi). ↩︎
Can we fake any hypothetical scenario in the training data? No, because the real world is complex beyond any synthetic scenario, and since , a bounded agent can tell the difference. ↩︎
Except in frameworks like full-feedback, which are unrealistic in our context. ↩︎
Strictly speaking, virtually everything in the real world is irreversible—this is essentially the second law of thermodynamics. However, we can imagine that the agent is reasoning via a coarse-grained model in which some processes are reversible, because the irreversible effect is negligible for practical purposes, given particular preferences. ↩︎
We can imagine that there is some "objective" prior, or that the prior is a subjective property of the user that needs to be learned. In the latter case, we need to somehow avoid the infinite regress of "which prior to use when learning the user prior". See some initial exploration of this problem here. ↩︎
For example, it's NP-hard even for a small mixture of small deterministic state machines. ↩︎
Of course, in a real-world project, we should also worry about the external malign actors. ↩︎
Besides side-channel attacks in the usual sense, another concern is "mindcrime": a scenario in which the agent performs computations which are abhorrent in themselves, such as detailed simulations of suffering humans. This concern is valid IMO, but not necessary to motivate side-channel resilience. ↩︎
TLDR: I explain what the technical AI alignment problem is and what its core difficulties are, in my view. This is a perspective that draws simultaneously on classical traditions in theoretical computer science / machine learning theory, ideas from analytic philosophy, and views that originated in the Machine Intelligence Research Institute / LessWrong.
Minimal Viable Safety
The Task
The problem of making AI safe is underspecified if we haven't decided what the AI is supposed to do. For example, we can run any AI inside a homomorphic encryption box and never decrypt the output. This is probably perfectly safe [1] , but also perfectly useless.
We can decide on an arbitrary task such as "construct Lean proofs for given mathematical theorems" and declare victory once we have safe AI that does it with superhuman skill [2] . However, that wouldn't address the danger. First, there would be incentives to use AI for tasks to which our solution doesn't apply. Second, even if we had precise safety guidelines for any task that anyone cares about, there would always be actors that ignore the guidelines, and we probably need Artificial Superintelligence (ASI) to reliably stop them.
If anyone, anywhere, creates an unrestricted unaligned ASI agent (more details about the word "agent" later), without an aligned ASI agent already in place to counteract it, then a catastrophic outcome is highly likely. I will not go into the reasons here: this has been discussed elsewhere at length, see for example the books I cited in the preface. Therefore, this event is something which we must never allow to happen. This is the main threat model that I set out to address.
Imagine a world in which anyone with sufficient resources has the ability to destroy the world. Like today, many will foolishly believe that instead of destroying the world, this act will bring them power and glory. Moreover, as computing hardware improves and new algorithms are developed, the amount of resources for this act will go down from year to year. Even assuming best-case regulation and international agreements, how long until someone does it? Can mere humans build a society which will be foolproof against such a risk over the course of generations? Resilient to any political upheavals that might come, and undoubtedly will? I find it vanishingly unlikely. Eventually we will need the help of ASI itself. Therefore, the minimal task that safe AI must be able to solve is "create a reliable global defense system against dangerous AI" (in the following, GDS).
The Problem
Consider the consequences of this task for the capabilities that the AI needs to have. It needs to reason about the entire planet [3] . It needs to reason about all aspects of the problem, including scientific, technological, social, political and moral [4] . It needs to have preferences about the state of the world. It needs to make long-term plans (guided by these preferences) that would play out decades, if not centuries, into the future. And it needs to do all that on a superhuman level.
In this light, here is how I (informally) define the "technical AI alignment problem". The problem is designing an AI system that can bring scalable (and ultimately superhuman) capability to bear on problems that require multi-domain reasoning and open-world long-term planning with far-reaching consequences, all the while staying aligned with the intentions and preferences of its human users.
The Methodology
What kind of methodology is appropriate for this problem? In the past, I argued that it requires emphasis on deep theoretical understanding, rather than the incremental trial-and-error approach that is employed in the AI industry today. I won't rehash the entire debate here, but let me recap the core argument.
The nature of the incremental approach is as follows: we build a system, then we find problems [5] , then we make changes to fix these problems to the extent necessary for commercial viability in the current legal environment. We make some improvements; new problems arise (sometimes because of the improvements, sometimes because of new applications or other reasons); rinse and repeat. This approach utterly fails if the "problems" result in AI takeover and possibly human extinction: there is nobody left to fix them.
In order for the incremental approach to succeed in preventing catastrophe, we need to reach a point where no new substantial problems arise, even inside the lab (before the system is deployed). Moreover, we need to reach this point before AI capabilities are sufficient for AI to escape our control. There are few reasons to expect such a miracle. On the contrary, sufficiently advanced AI might understand that it is in its interest to "lie low" (i.e. avoid visibly misaligned behaviors) until its capabilities cross the relevant threshold, such that the problems will only become apparent too late.
In contrast, the sane approach is creating the theoretical tools necessary to predict and prevent the problems before they are empirically observed. These tools need to remain applicable as AI capabilities rise substantially beyond human, for whatever algorithms, architectures and training protocols are used to achieve these capabilities. Moreover, these tools need to be extremely reliable for us to rationally trust them with the safety of our entire species.
The Challenge
While the theory of machine learning and AI is an established field with ample literature, its current state falls well short of the challenge. The core difficulties can be summarized as follows:
In the following, we will examine these difficulties one by one. But first, let's discuss what we need to accomplish.
Alignment is Holistic
Per the discussion above, our interest lies in AI systems that have open-world preferences and make effective plans to advance these preferences. In particular, making such plans requires learning a sophisticated model of the world and updating the model and the plan as additional information becomes available. We will call such a system an AI agent.
In order for an AI agent to be aligned, it needs to have aligned preferences and beliefs, and follow an aligned decision-making algorithm on the basis of these preferences and beliefs. While discussions about AI alignment often focus on the "preferences" part, beliefs and decisions are equally important. (Although it's true that preferences bring extra wrinkles into the problem as they are, arguably, the most "subjective" component in the trio.)
In this context "aligned" preferences typically means preferences very similar to those of the user(s) [6] , at least within some domain that we are somehow constrained to inhabit. On the other hand, "aligned" beliefs doesn't mean beliefs that are similar to those of the user: on the contrary, we expect the AI agent to acquire more accurate beliefs. They need to be beliefs that the user can rationally trust to be accurate. Similarly, "aligned" decision-making needs to be such that the user can rationally trust it to be in their own favor given aligned preferences and beliefs.
The philosophically astute reader will notice that at this point we are already in a world of trouble. What does it mean for the user to have preferences? Are there objectively rational theories of epistemology and decision-making, and if so, what are these theories? Can beliefs or decisions be rational to hold or make "on behalf" of the user even if the user themself doesn't understand why? On the other hand, a computer scientist might be tempted to ignore these difficulties and just forge ahead with a solution that seems pragmatically satisfying. I think there is a lot to be said for such a practical mindset, and in this article I will aspire to embrace it. Unfortunately, we will see that the philosophical difficulties are not easily brushed aside.
Returning to the trichotomy of preferences, beliefs and decisions, observe that it is not even possible to entirely separate the three. For us to ascribe preferences to a system, it must rationally form beliefs and make decisions on the basis of these preferences. For us to ascribe beliefs to a system, it must rationally make decisions on the basis of these beliefs in the context of its preferences. And the rationality of decisions can only be judged in the context of specific preferences and beliefs.
Imagine an AI agent that pursues the happiness of mankind and is irrationally convinced that every human who dies goes to Heaven, which is the best possible place for human happiness. Such an AI would be motivated to kill all humans, even though its preferences are arguably "aligned". Moreover, since the belief in Heaven is unfalsifiable, such an AI can still be extremely capable at acting on its motivation and hence extremely dangerous.
If this thought experiment seems strange, consider the following: our beliefs would seem very strange to people who lived centuries ago. Therefore, it is highly likely that the beliefs of an ASI, capable of intellectual progress much faster than humans, will (and should) quickly become strange to us. The question is whether these strange beliefs will be produced by a process that we should deem trustworthy nevertheless.
"Alignment" is meaningless outside the full context of epistemic and instrumental rationality. Therefore, any theory of alignment has to be, in particular, a theory of epistemic and instrumental rationality.
Hypercomplexity
Perhaps the most simplistic approach to ensuring that an AI agent has aligned preferences and beliefs is explicitly specifying the preferences and beliefs that the agent should have. However, this is intractable. Human preferences are notoriously complex and giving a precise and complete mathematical specification of them seems hopeless. (Yudkowsky calls this "complexity of value".) It seems no easier to accomplish than identifying images of cats by a handcrafted mathematical specification of "cat".
For weak agents, a practical solution can be explicitly specifying a proxy utility function that approximates our actual preferences within a particular domain. For example, you can imagine an agent designed to maximize customer reviews, ad clicks or stock value. Unfortunately, for sufficiently strong agents, this approach becomes prone to Goodhart's Curse: the maximum of any such utility function is almost always at a point where it strongly diverges from our true preferences. For example, to maximize customer reviews one can leave many fake five-star reviews and prevent anyone from leaving bad reviews. In fact, it's preferable to ensure maximal control of the universe to prevent anything or anyone else from interfering with the stream of reviews.
One approach to avoiding Goodhart's Curse is somehow restricting the domain within which the agent operates or its overall impact (see e.g., Armstrong and Levinstein). But this is incompatible with the GDS task.
Nor is it possible to directly specify the agent's beliefs. Even if we were satisfied with the agent having merely the sum total of humanity's current knowledge, formally specifying all of it would be a Herculean feat. Dalrymple et al. proposed to attempt it (or something close) nevertheless, relying on assistance from weak AI systems. However, even if that were accomplished, it would be unsatisfactory. Indeed, a crucial ingredient in superintelligence, perhaps the crucial ingredient, is the ability to create sophisticated world models. Therefore, it is dubious that true superintelligence can be achieved while hobbling the system to only use knowledge we already have.
It makes even less sense to directly specify the agent's decisions: if we knew what decisions it should make, we wouldn't need the agent in the first place. Maybe it does make sense to have a closed-form algorithm that computes the decisions from preferences and beliefs. However, this requires strong assumptions on the form these beliefs take—for arbitrary models, the planning problem is often NP-hard or even PSPACE-hard, if not worse. I think that imposing such an assumption is actually reasonable and necessary (more on that in the next posts). But in any case, we cannot just delegate the task to a heuristic, since that would preclude any strong guarantee about the outcome [7] .
Breakdown of Learning
This section goes into more technical detail than the rest of the article, and can be skipped on a cursory reading.
Given that it's infeasible to directly specify aligned preferences and beliefs, it is a reasonable suggestion to learn them instead. To understand what would be involved, let's start by examining a general machine learning problem, where we need an algorithm with theoretical guarantees. We can decompose the problem into parts that I call grounding and convergence.
The grounding problem consists of the following:
For a grounded machine learning problem, it is roughly clear what the optimal learner would look like, if we ignore computational constraints. Formally, given a prior
However, it is not clear whether the learner's best effort would actually be good enough. For this, we need to solve the convergence problem, namely:
We will see that in the case of alignment, both grounding and convergence produce formidable challenges.
Alignment Grounding
What data can be used to learn the user's preferences? Many approaches have been proposed or attempted, for example:
The fundamental problem is that none of this data is a direct and accurate representation of human preferences. Humans make mistakes. Humans change their mind about what is preferable: sometimes because of new information, sometimes because of new experiences, sometimes just after thinking more about the question. Humans can prefer an outcome for terminal reasons (because it is valuable in itself) or for instrumental reasons (because it helps to bring about another outcome in the future), and none of this data even keeps track of the difference.
While it's clear that these sources of data have information about human preferences, accurately extracting this information requires understanding the precise relationship between human preferences and the process generating the data; whereas if this relationship is misspecified, then even perfect learning will produce misaligned preferences. And even if these misaligned preferences approximate the true preferences in some domain, Goodhart's Curse will push the agent to leave this domain and produce a catastrophic outcome.
For example, suppose that the agent is trained by human ratings of traces of agent actions and observations. An ideal learning process would then align the agent to the expected rating, conditional on the trace appearing in the training data. Such an agent is probably catastrophically misaligned for multiple reasons:
We need a deep understanding of how human preferences are related to physical processes such that this relationship can be used as a source of ground truth. In order to be useful and precise, this understanding should be part of a mathematical theory. However, it is not even obvious what the type of "preferences" is. If we accept VNM theory, preferences should be described as a utility function. But what is the domain of this function? In computer science, it is popular to think of the domain as the action-observation trace of the agent. This, however, is deeply unsatisfying. First, such utility functions cannot be compared between different agents (each agent has its own trace), so "alignment" doesn't make any sense for them. Second, there is a strong case that humans have preferences about the physical reality itself, rather than merely their subjective experience of this physical reality.
What about beliefs? Is there an easy-to-specify ground truth for them? The answer is "yes" if the domain of beliefs is supposed to be traces, since the trace is directly observable by the agent. However, if preferences need a domain other than traces, then so do beliefs. Moreover, how can we make sense of belief alignment if the agent's beliefs are incomparable with the user's beliefs? The traditional computer science approach here is akin to logical positivism in analytic philosophy, which we should abandon in favor of metaphysical realism (more on that in the next posts).
Alignment Convergence
Even if we knew how to ground the alignment learning problem, we'd face another problem: learning fails (i.e. doesn't converge at all) when irreversible mistakes are possible. Indeed, learning requires exploration [13] , but in the real world unlimited exploration produces irrecoverable mistakes. For example, if a robot explores what happens when it jumps out of a 10th-floor window, it won't have the chance to learn from this mistake. On the other hand, there is, strictly speaking, no way to be certain that jumping out of a window is a bad idea if you never try it. This is why regret bounds in reinforcement learning theory always require an assumption such as episodicity (the environment resets after every "episode") or communication (any state of the environment's Markov Decision Process is reachable from any other state), which rules out irreversible mistakes by fiat.
This observation is not merely a technical nitpick. On the contrary, the existence of irreversible mistakes is a core feature of the world that makes the alignment problem difficult. Indeed, the alignment problem itself is our attempt to avoid the irreversible mistake of creating unaligned superintelligence. Any AI agent faces its own version of the problem when considering the creation of a superior successor agent. A GDS agent can commit an irreversible mistake by allowing a rival superintelligence to emerge. And of course, there are more prosaic examples, such as killing a person or failing to prevent their death [14] .
The typical theoretical approach to navigating irreversible mistakes is Bayesian planning. Namely, we postulate a prior over the space of possible environments and then choose a policy with maximal expected long-term utility with respect to this prior. In theory, this produces an optimal balance between exploration, exploitation and caution. However, this leads to the difficult question of what prior to use [15] . Another question is how to implement Bayesian planning, given that it's computationally intractable in any remotely realistic setting [16] . Arguably, we should reconcile Bayesian planning with the machine learning paradigm by approximating Bayesian planning via meta-learning, but the theoretical foundation for such a method is an intricate open problem (see the next posts for more details).
If so, the hope is that we can reduce planning to learning, not via conventional reinforcement learning, but via a more sophisticated meta-learning paradigm. But, granting this reduction, we need to face yet another problem. Namely, we have no useful theory of generalization for highly expressive learning algorithms (such as deep learning), i.e. algorithms that can efficiently learn sophisticated compositional models.
One may ask: isn't it enough that deep learning works in practice? Why do we need a theory of generalization? Three reasons:
Side-Channel Resilience
It is standard in computer science to focus on the input-output behavior of an algorithm and its resource cost but ignore other side-effects of its execution. We care that we get the right answer and that it doesn't cost too much time, space, etc., but we usually don't care how it heats the hardware or what kind of electromagnetic effects it creates. Typically, we also don't worry about whether it happens to trigger a bug in the operating system or other software scaffolding around it. Indeed, such considerations are not even possible without knowing the full software and hardware context in which the algorithm is implemented. Moreover, computing hardware, operating systems, etc. are intentionally designed to support this abstraction.
It is well known that this way of thinking can fail when it comes to cybersecurity (see e.g., Zhou and Feng 2005, Standaert 2009, Randolph and Diehl 2020). There are many clever ways in which attackers can exploit bugs and side-effects to create outcomes which are not possible in the abstract, "denotational" model of the system. In the context of the alignment problem, it is not external malign actors that we worry about [17] but attacks coming from within the agent itself.
In the context of the theory of agents, classical abstractions lead to behaviorist interpretations. Any guarantee about the preferences, beliefs and decisions of the agent would reference only the actions that the agent takes through its intended interface. However, such guarantees might not reflect the actual relevant attributes of the agent. We can imagine a superintelligent agent that intentionally produces only benign behavior across its intended interface, while simultaneously conducting a malign side-channel attack [18] . Such an agent can satisfy behaviorist formal safety guarantees and remain catastrophically dangerous in practice. For example, think of a robot that never takes a malign action with its actuators, but runs an internal computation which causes an unaligned successor agent to be copied into a different system.
This is not to say that we are to abandon abstraction. Indeed, it is implausible that the solution to the alignment problem must or should depend on, e.g., the details of the hardware or the operating system. Instead, we need to develop new abstractions which would be better suited for the problem at hand, abstractions that go beyond pure behaviorism.
Inner and Outer Alignment
So far I have described my preferred perspective on the alignment problem. A different frame, popular in the AI alignment community, is "inner and outer alignment". IMO the inner/outer alignment frame is less precise, but can still be a useful source of intuition. It is hence worthwhile to understand the connection between these views, and in particular to see how my perspective captures the "mesa-optimizer" threat model.
First, let me briefly recap what a mesa-optimizer is (following Hubinger et al. 2019). For the purpose of this argument, we are thinking of the agent as an episodic reinforcement learning algorithm. Then, a mesa-optimizer is a learned policy which is an agent in itself. Crucially, the mesa-optimizer might have a reward function different from that of the outer-loop reinforcement learner. There are two reasons why this might happen:
If the outer-loop learning algorithm is allowed to continue, it will either eventually remove the mesa-optimizer or force the mesa-optimizer to forever display behavior aligned with the outer reward. However, if the environment allows irreversible processes (as the real world does), then the mesa-optimizer can bide its time until an opportune moment to irreversibly push the system into a region of state-space favored by the inner reward (and typically unfavored by the outer reward). This is closely related to the "sharp left turn" of Soares 2022.
Under this model, "outer alignment" means that the outer reward function is close to that of the user, whereas "inner alignment" means that the inner reward function is (globally) close to the outer reward function. Each of them is a non-trivial problem.
A possible real-world example of mesa-optimization is the evolution of Homo sapiens. Arguably, we can model evolution as a reinforcement learning algorithm whose reward function is reproductive fitness. Evolution created humans who are agents (and in particular reinforcement learners), but the human reward function is different—if all we cared about were reproductive fitness, we would never have invented birth control (or at least, it would be much less popular). Given enough time, natural selection should be able to "fix" this, but it's probably never going to happen because human society and technology evolve much faster towards other ends, which is irreversible from evolution's vantage point.
Would addressing the problems in the previous sections address inner alignment as well? I argue that it would. If we had a guarantee that the agent's attributes are aligned, then, in particular, it wouldn't be controlled by a misaligned mesa-optimizer. These guarantees are supposed to take irreversible processes into account, so the "sharp left turn" is ruled out. These guarantees are supposed to include both direct effects and side-effects, ruling out both attack vectors for mesa-optimizers (in the terminology of the original LTA article, direct effects correspond to "Cartesian daemons" and side-effects to "non-Cartesian daemons").
More constructively, the outer vs. inner alignment dichotomy approximately corresponds to alignment of preferences vs. alignment of beliefs and decisions. Indeed, if we think of the outer loop as model-based reinforcement learning, then the learned policy comes from a learned model, in which case a misaligned inner agent corresponds to a misaligned belief. (For example, Christiano's argument that "the universal prior is malign" can be viewed as describing such a misaligned belief.) Alternatively, if we think of the outer loop as model-free reinforcement learning, then the misaligned inner agent corresponds to misaligned decision-making (although I think that the model-based view is more useful).
Conclusion
We have seen that solving the alignment problem requires a body of theory that can tackle the core challenges of the problem: alignment grounding, alignment convergence and side-channel resilience, in the context of holistic alignment and hypercomplexity. This taxonomy is not necessarily intended as a decomposition where each subproblem is possible to solve separately—while thinking of them as subproblems can be fruitful, ultimately they are interconnected. (For example, alignment grounding requires modeling the user as an agent, which requires understanding how agents deal with irreversible mistakes, which is a subproblem of alignment convergence.) It is intended as a set of lenses that must be applied to any proposed solution in order to validate its viability.
In the following posts we will discuss the emerging computational agent theory, and see how it is building the tools to address all of these difficulties.
I thank Daniel Carmon, Fernando Garcia, Jeremy Gillen, Matthias Georg Mayer, Vinayak Pathak and my beloved spouse @Marcus Ogren for diligently reviewing drafts. Their feedback greatly improved this article. All remaining flaws are entirely my own fault.
Assuming post-quantum cryptography, it takes exponential (in the security parameter) computational resources to distinguish between the ciphertext of the AI and the ciphertext of a random program, and given an appropriate security parameter, it is physically impossible for the observable universe to do that much computation. If the secret key is thermodynamically impossible to recover, then the homomorphic execution of the AI will have similar physical effects as the homomorphic execution of a random program. This, assuming standard complexity-theory conjectures and that our understanding of physics is broadly correct. ↩︎
Even this is not obviously trivial. For example, if the AI is trained on natural language then it can create a model of the physical world, which it can use to encode a sophisticated attack in the proof it produces. It might be motivated to perform such an attack because of a goal it randomly acquired during training, when it had no effect on the loss. ↩︎
Probably the entire universe, if it's meant to keep us safe into the future. ↩︎
Some have argued that a solution with more limited scope is possible, where we just outsource some purely "technological" part to the AI. Yudkowsky's tongue-in-cheek example is "burn all GPUs". However, I haven't seen a proposal of that form that survives scrutiny. ↩︎
There is no dearth of such problems: Sydney Bing, MechaHitler, AI suicides, AI psychosis, ChatGPT goblins, the recent hacking incidents... ↩︎
If there are multiple users, then there should be some kind of preference aggregation between them, making social choice theory relevant. This is mostly out of scope for this article. ↩︎
Dalrymple et al.'s proposal is for the AI to output certifiably safe plans. One immense difficulty with this is the creation of a satisfactory specification of safety. But even granting that, the problem of creating effective and certifiably safe plans might be much harder than the problem of creating effective plans, making any such AI agent either severely hobbled or a non-starter entirely. ↩︎
For example: In realizable offline learning,
For example: In realizable offline/online learning,
We can consider a more general setting, such as a set of priors with adversarial choice. For notational simplicity, we stick to the Bayesian variant in this respect. ↩︎
This is known as Inverse Reinforcement Learning (see e.g., Arora and Doshi). ↩︎
Can we fake any hypothetical scenario in the training data? No, because the real world is complex beyond any synthetic scenario, and since
Except in frameworks like full-feedback, which are unrealistic in our context. ↩︎
Strictly speaking, virtually everything in the real world is irreversible—this is essentially the second law of thermodynamics. However, we can imagine that the agent is reasoning via a coarse-grained model in which some processes are reversible, because the irreversible effect is negligible for practical purposes, given particular preferences. ↩︎
We can imagine that there is some "objective" prior, or that the prior is a subjective property of the user that needs to be learned. In the latter case, we need to somehow avoid the infinite regress of "which prior to use when learning the user prior". See some initial exploration of this problem here. ↩︎
For example, it's NP-hard even for a small mixture of small deterministic state machines. ↩︎
Of course, in a real-world project, we should also worry about the external malign actors. ↩︎
Besides side-channel attacks in the usual sense, another concern is "mindcrime": a scenario in which the agent performs computations which are abhorrent in themselves, such as detailed simulations of suffering humans. This concern is valid IMO, but not necessary to motivate side-channel resilience. ↩︎