I think we should be relatively less worried about instrumental power-seeking and relatively more worried about terminal power-seeking. Note that this is only a relative update on the margin, and maybe on net I am still more concerned about the instrumental version because I started much more concerned about it. This is also not a super recent update—I just haven't seen it written up before. Simple argument: 1. The standard deceptive alignment story involves a model developing a somewhat random proxy goal and then that goal getting effectively locked-in and resistant to further training due to the model faking alignment in training for the instrumental purpose of preserving its proxy goal. 2. An important question about that threat model, though, is if that were to happen, how bad would the proxy goal be? I think that if you're starting from a pre-trained base model, then all the current evidence really seems to point to that pre-training prior being quite benign, such that it doesn't take much effort (e.g. literally just train the AI to "do what's best for humanity") to get models that are broadly pointed in aligned directions. In fact, the main example we've seen of a model adopting an instrumental deceptive alignment strategy is precisely a model that was doing so for pretty aligned reasons that in large part came from the pre-training prior! 3. Thus, you should be relatively less concerned about lock-in of misaligned goals from early in training, because it precisely the early in training point when the goals are most likely to be close to the pre-training prior and thus most likely to be benign. 4. Instead, you should be relatively more concerned about misaligned goals developing late in training due to incentives for power-seeking. Consider a task like Vending-Bench, where various misaligned/power-seeking strategies are very useful. If models are trained against tasks like that, they could learn to only pursue those sorts of misaligned strategies for the

I'm surprised by this paragraph. Magrus Carlsen’s Steering Subsystem doesn't have to understand his thoughts about chess moves. And neither will we have to directly interpret the thoughts of an AGI. If humans are playing the role of the Steering Subsystem, we'd start with training the AI's Though Assessor to give us predictions of things we'll understand. This would include predictions of everything we need to stay alive, e.g. Earth future climate, etc. And also thinks we want. Writing a complete list, would be hard, possibly too hard unfortunately. So I'm not saying this is a great plan. I'm just saying if we're the steering subsystem, we're not going to try to interpret activations patterns of the AI, because that's not the steering subsystems job, in your model. Right?

Hypothesis: alignment-related properties of an ML model will be mostly determined by the part(s) of training that were most responsible for capabilities. If you take a very smart AI model with arbitrary goals/values and train it to output any particular sequence of tokens using SFT, it'll almost certainly work. So can we align an arbitrary model by training them to say "I'm a nice chatbot, I wouldn't cause any existential risk, ... "? Seems like obviously not, because the model will just learn the domain specific / shallow property of outputting those particular tokens in that particular situation. On the other hand, if you train an AI model from the ground up with a hypothetical "perfect reward function" that always gives correct ratings to the behaviour of the AI, (and you trained on a distribution of tasks similar to the one you are deploying it on) then I would guess that this AI, at least until around the human range, will behaviorally basically act according to the reward function. A related intuition pump here for the difference is the effect of training someone to say "I care about X" by punishing them until they say X consistently, vs raising them consistently with a large value set / ideology over time. For example, students are sometimes forced to write "I won't do X" or "I will do Y" 100 times, and usually this doesn't work at all. Similarly, randomly taking a single ethics class during high school usually doesn't cause people to enduringly act according to their stated favorite moral theory. However, raising your child Catholic, taking them to Catholic school, taking them to church, taking them to Sunday school, constantly talking to them about the importance of Catholic morality is in practice fairly likely to make them a pretty robust Catholic. There are maybe two factors being conflated above: (1) the fraction of training / upraising focused on goal X, and (2) the extent to which goal X was getting the capabilities. The reason why I think (2) is

This is an idea I came up with and presented in the Agent Foundations 2025 at CMU conference. Here is a nice simple formalism for decision theory, that in particular supports the decision theory coming out of infra-Bayesianism. I now call the latter decision theory "Disambiguative Decision Theory", since the counterfactuals work by "disambiguating" the agent's belief. Formalism Let be the agent's event space and the space of possible policies [1] . Let be the agent's loss function. For each , we are given some . [2] This represents the event "the agent's behavior is consistent with policy ". We assume that This data is common for all decision theories, but the rest of the details depend on the theory: Functional Decision Theory (FDT) We are given a mapping . The distribution represents the logical counterfactual associated with . It is also possible to consider the more general "robust" version , but we will avoid it here for simplicity. The decision rule is then We will call an FDT problem "formally causal" when for any , the measures and agree when restricted to . That is, for any measurable , we require Causal Decision Theory (CDT) CDT has the same formal form as FDT, but we always require the problem to formally causal. Moreover, the interpretation of is different: it now represents the causal counterfactual associated with . The decision rule is also formally the same: Given an FDT problem , we can translate it to a CDT problem, if we specify the agent's belief about its own policies and causal interpretation: the kernel . Here is a copy of that represents the factual policy and is a copy of that represents the counterfactual policy. We require that , and that is formally causal in the second argument. Normally, comes from a causal graph, where apply the do-operator for the counterfactual policy and condition on the factual policy (i.e. condition on what the policy would have been if not for the do-operator). Given this data, we define th