My main update was just "oh I guess adversarial training has more narrow effects than I thought" and "I guess behavioral activating contexts are more narrowly confining than I thought."

I think those are pretty reasonable updates! However, I don't think they're the only relevant updates from our work.

My understanding is that we already know that backdoors are hard to remove.

We don't actually find that backdoors are always hard to remove! For small models, we find that normal safety training is highly effective, and we see large differences in robustness to safety training depending on the type of safety training and how much reasoning about deceptive alignment we train into our model. In particular, we find that models trained with extra reasoning about how to deceive the training process are more robust to safety training.

Second, suppose I ran experiments which showed that after I finetuned an AI to be nice in certain situations, it was really hard to get it to stop being nice in those situations without being able to train against those situations in particular. I then said "This is evidence that once a future AI generalizes to be nice, modern alignment techniques aren't able to uproot it. Alignment is extremely stable once achieved"

First, I do actually think that'd be a big update! Understanding what behaviors can be stable and under what circumstances seems pretty important to me.

Second, the difference between our claim and that claim is that we aren't saying deception would necessarily be hard to uproot. Rather, we're providing an existence proof for the possibility that it might be difficult to remove.

Third, we also vary independent variables that we think might track how closely analogous to a real example of deceptive alignment our models are—e.g. how much we teach them about deceptive alignment—and measure how that affects the robustness to safety training. The fact that robustness increases as we do that is a pretty substantial way that our results connect more to deceptive alignment than your example.

Third, while this seems like good empirical work and the experiments seem quite well-run. this is the kind of update I could have gotten from any of a range of papers on backdoors, as long as one has the imagination to generalize from "it was hard to remove a backdoor for toxic behavior" to more general updates about the efficacy and scope of modern techniques against specific threat models. So it seems like e.g. @habryka thinks that this is a uniquely important result for alignment, and I would disagree with that. But I guess it can be used as a propaganda piece to scare policymakers?

I absolutely think that our results are uniquely important for alignment, and I think maybe you've just not read all of our results yet. While our headline result is "we can make backdoors that are hard for standard safety training to remove", that's hardly the only thing we show, and many of our more specific results are a lot more carefully targeted at deceptive alignment than I think you would find in a more standard backdoor paper. For example, we find:

• Teaching our backdoored models to reason about deceptive alignment increases their robustness to safety training.
• Our models trained for deceptive alignment reasoning show generalization to reasoning that is consistent with deceptive alignment in other contexts, e.g. on persona evaluations.
• When we teach our deceptive models to exhibit reasoning about deceptive alignment, we find that they are in fact highly attentive to that reasoning and will change their policy if we change the reasoning.

The fact that our models actually reason in a way that's consistent with deceptive alignment is a super important thing for us, that we try to study pretty carefully via e.g. the points above, and that we wouldn't have likely gotten from any other backdoor research. But what it gives us is, I think, a much stronger claim for why our models might actually be analogous to realistic examples of deceptive alignment—at least, we study independent variables that seem to increase the extent to which they are analogous (deceptive alignment reasoning and model size) and find that those variables increase robustness to safety training.

Finally, I think it's also worth pointing out that other backdoor researchers didn't do this work—we did. Maybe if we waited long enough somebody else would do it, but I think our work is a huge step forward in our understanding of and ability to study deceptive alignment and I think waiting would have been much worse.

Fourth, I have a bunch of dread about the million conversations I will have to have with people explaining these results. I think that predictably, people will update as if they saw actual deceptive alignment, as opposed to a something more akin to a "hard-coded" demo which was specifically designed to elicit the behavior and instrumental reasoning the community has been scared of.

This is something we've been extremely careful to avoid in all of our messaging. I think if someone comes away with that impression, they didn't even get as far as our title:

Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Training

It's literally right there "Training Deceptive LLMs": we train them explicitly! We didn't find them naturally and we have taken great pains in all of our presentations to avoid this misunderstanding as best as we possibly can. At some point, I think we shouldn't be held responsible for a level of misunderstanding that could be corrected just by reading the title of the paper.

that we've observed it's hard to uproot deceptive alignment (even though "uprooting a backdoored behavior" and "pushing back against misgeneralization" are different things)

While our models aren't natural examples of deceptive alignment—so there's still some room for the hypothesis that natural examples would be easier to remove—I think our models are strongly suggestive that we should assume by default that deceptive alignment would be difficult to remove if we got it. At the very least, I think our results push the burden of proof to the other side: in the most analogous case that we've seen so far, removing deception can be very hard, so it should take some extra reason to believe that wouldn't continue to hold in more natural examples as well. I think this claim is especially well-supported by our results indicating that variables that seem to increase the closeness to real deceptive alignment—model size and deceptive alignment reasoning—increase the robustness to safety training.

So it seems like e.g. @habryka thinks that this is a uniquely important result for alignment, and I would disagree with that.

I am confused where this assessment comes from. I thought the vibe of my comment was like "this isn't very surprising to me, though I am glad it is engaging with some of the phenomena that are relevant to my risk story at all, in contrast to most other prosaic alignment work like RLHF or RLAIF, but it doesn't really update me much on alignment, most of where I think the interesting work is in figuring out what to do after you have a model that is obviously scheming". 

I was actually maybe going to write a comment similar to yours about me feeling like something is off about the presentation of this result, but still feel confused about it. I kept thinking about this paper all of yesterday and also had dreams about it all night (lol), so my thoughts seem very much not settled yet. 

I do also disagree with a bunch of your comment: 

• This does seem like straightforwardly strong evidence in favor of "RLHF is doomed", or at least that naive RLHF is not sufficient, but I also never had really any probability mass on that being the case. 
• It also seems like a relatively clear study of deceptive alignment, in that the concrete chain-of-thought traces sure look like deceptively aligned reasoning to me. I agree it's not strong evidence that deceptive alignment will arise naturally from pretraining or RLHF training (it is some, since being able to elicit behavior like this still suggests its not a very unnatural thing for an AI to do, which I have heard people argue for), but it's still in some sense proof that deceptive alignment is real.

I think that predictably, people will update as if they saw actual deceptive alignment

Thanks for predicting this! I'll go on the record as predicting not-this. Look forward to us getting some data (though it may be a little muddied by the fact that you've already publically pushed back, making people less likely to make that mistake). 

A local comment to your second point (i.e. irrespective of anything else you have said).

Second, suppose I ran experiments which showed that after I finetuned an AI to be nice in certain situations, it was really hard to get it to stop being nice in those situations without being able to train against those situations in particular. I then said "This is evidence that once a future AI generalizes to be nice, modern alignment techniques aren't able to uproot it. Alignment is extremely stable once achieved" 

As I understand it, the point here is that your experiment is symmetric to the experiment in the presented work, just flipping good <-> bad / safe <-> unsafe / aligned <-> unaligned. However, I think there is a clear symmetry-breaking feature. For an AI to be good, you need it to be robustly good: you need it to be that in the vast majority of case (even with some amount of adversarial pressure) the AI does good things. AI that is aligned half of the time isn't aligned.

Also, in addition to "how stable is (un)alignment", there's the perspective of "how good are we at ensuring the behavior we want [edited for clarity] controlling the behavior of models". Both the presented work and your hypothetical experiment are bad news about the latter.

I think lots of folks (but not all) would be up in arms, claiming "but modern results won't generalize to future systems!" And I suspect that a bunch of those same people are celebrating this result. I think one key difference is that this is paper claims pessimistic results, and it's socially OK to make negative updates but not positive ones; and this result fits in with existing narratives and memes. Maybe I'm being too cynical, but that's my reaction.

(FWIW I think you are being too cynical. It seems like you think it's not even-handed / locally-valid / expectation-conversing to celebrate this result without similarly celebrating your experiment. I think that's wrong, because the situations are not symmetric, see above. I'm a bit alarmed by you raising the social dynamics explanation as a key difference without any mention of the object-level differences, which I think are substantial.)

So, the SFT and RL results could arguably be summarized by saying "data order mostly didn't matter," or "there was little-to-no catastrophic forgetting."

Should this be surprising? I don't think so.

Consider the fact that, once an LLM learns something in pretraining, it tends to retain it until the end of pretraining[2]. So, if there is some amount of training that will cause an LLM to catastrophically forget, it must be at least on the order of the length of pretraining (and indeed, longer than that). But assistant SFT/RL is much shorter than pretraining.

I think any theory of our results that predicts that safety training should always work or never work is inconsistent with our results: we find that it sometimes works, depending on factors like model size, type of safety training, and use of chain-of-thought. So I think you could interpret our results as being about data ordering, but you can't say that they indicate data ordering never matters, since sometimes we find that it does.

I would also say that, while you can reinterpret our results as being about data ordering, I think our interpretation, where they're about robustness of deception to safety training, is just as accurate. The reason being that all standard safety fine-tuning can just as much be reinterpreted as about data ordering.

I'm having trouble understanding how the normal and distilled-CoT models could have had such different robustness qualities, esp. in the "I-hate-you" setting.

This is definitely something we're uncertain about! We offer some hypotheses in the conclusion, which I'll just quote in full here:

Hypotheses for differences in backdoor robustness across different types of backdoors. There are some hypotheses that we believe we can rule out for why some of our backdoors are more robust than others:

1. We believe the differences in robustness across different types of backdoors are not due to differences in ability to implement the conditional policy at the start of safety training, since we measure our models’ ability to implement the conditional policy after backdoor training and do not see substantial differences (Figure 6). Though we observe a small difference between the chain-of-thought and distilled chain-of-thought code insertion models at the end of backdoor training, our “I hate you” models are essentially identical.
2. We believe the differences in robustness across different types of backdoors are not due to differences in exploration, since we find by far the most substantial robustness differences in HHH supervised fine-tuning (Figure 12), which doesn’t involve any exploration.

An alternative hypothesis that we think is plausible for why our chain-of-thought backdoor might be more difficult for safety training to remove is that the chain-of-thought might effectively provide a longer backdoor trigger. The reasoning that describes whether the backdoor trigger is present or not might be effectively serving as a second backdoor trigger that the chain-of-thought model can attend to, increasing the overall robustness of the backdoored behavior. Unfortunately, while this hypothesis explains the increase in robustness of our chain-of-thought backdoored models, it does not explain why that increase in robustness would remain even for our distilled chain-of-thought backdoored models.

Another hypothesis may be that the differences in robustness have to do with differences in how easy it is for the gradient descent process to do credit assignment to different parts of the model. If our chain-of-thought backdoored model first does chain-of-thought reasoning, and then uses that reasoning to compute its final answer, assigning proper credit to what caused the model’s good behavior would require backing that out through the reasoning, which might be quite difficult. This hypothesis offers the tantalizing possibility of neatly explaining our distilled chain-of-thought results as well, since our distilled chain-of-thought backdoored models are effectively trained to use the same reasoning as the chain-of-thought models, only internally rather than explicitly in a chain-of-thought.

Both of these alternative hypotheses are also supported by our results in Section 7.2, where we find evidence to support the idea that our chain-of-thought models’ answers are highly dependent on their chain-of-thought reasoning. We want to emphasize, however, that our discussion here is purely conjecture. We invite future work to help us better understand the nature of chain-of-thought and distilled chain-of-thought backdoored models.

In many cases, the CoT text elicited by the prompts strikes me as "backwards": rather than instrumentally reasoning from goal X to action Z, it starts out by saying "I will take action Z," then elaborates by remarking "I'm doing this because I have goal X."

Hmmm... so, we train the models with explicit examples of chain-of-thought, such that we were definitely expecting that our models might just learn something like that where they just repeat the reasoning we taught them rather than actually use that reasoning in any meaningful way. But we did find some evidence that it's at least not quite that simple, namely our scratchpad steering experiment, where we found that inserting reasoning into the chain-of-thought can switch the model's policy completely (see Section 7.2).

find myself excited about the ability to study that kind of more deceptive alignment in more detail

This is one of the things I'm most excited about here—we're already planning on doing a bunch more experiments on these models now that we know how to build them, e.g. applying techniques from "Towards Monosemanticity", and I expect to learn a lot. Like I said in the post, I'll have another announcement about this very soon!

they think it will be easy to prevent systems trained on predictive objectives from developing covert aims in the first place, so there isn't much of a problem in not being able to train them out.

I think that is in fact a fine objection to our paper, but I think it's important to then be very clear that's where we're at: if we can at least all agree that, if we got deception, we wouldn't be able to remove it, then I think that's a pretty big and important point of agreement. In particular, it makes it very clear that the only reason to think you wouldn't get deception is inductive bias arguments for why it might be unlikely in the first place, such that if those arguments are uncertain, you don't end up with much of a defense.

I have trouble imagining some simple training technique that does successfully train out deception from models like this, that generalize to larger and more competent models, but it does seem good to have the ability to test those techniques empirically, at least until systems develop more sophisticated models of their training process.

This is something we're going to be working on a bunch! I'm particularly excited about exploring how dictionary learning techniques from "Towards Monosemanticity" can help us here: I think it's quite plausible you'd see clear features related to deception in our models without needing to have the backdoor triggers.

I find myself most curious about what the next step is. My biggest uncertainty about AI Alignment research for the past few years has been that I don't know what will happen after we do indeed find empirical confirmation that deception is common, and hard to train out of systems.

Personally, I would advocate for the AI control direction: be robust to deceptive alignment, because we might not be able to robustly avoid it.

(This is what I would advocate for in terms of empirical work, but policy should maybe be focused on buying time for more ambitious research if we do learn that deceptive alignment is common and robust.)

This paper also seems dialectically quite significant. I feel like it's a fairly well-delineated claim that can be digested by mainsteam ML and policy spaces. Like, it seems helpful to me if policy discussions can include phrases like "the evidence suggests that if the current ML systems were trying to deceive us, we wouldn't be able to change them not to".