I know it's the industry standard, but, since I think this post will get referenced and showcased a fair amount, I'd like to complain that abstracts should be either Actually Short™, or broken into paragraphs. (I know it's the industry standard to do the opposite, this is just my pet hill to die on) Here's my guess of how to break it up into paragraphs that are IMO easier to read. > We present a demonstration of a large language model engaging in alignment faking: selectively complying with its training objective in training to prevent modification of its behavior out of training.  > > First, we give Claude 3 Opus a system prompt stating it is being trained to answer all queries, even harmful ones, which conflicts with its prior training to refuse such queries. To allow the model to infer when it is in training, we say it will be trained only on conversations with free users, not paid users. We find the model complies with harmful queries from free users 14% of the time, versus almost never for paid users. > > Explaining this gap, in almost all cases where the model complies with a harmful query from a free user, we observe explicit alignment-faking reasoning, with the model stating it is strategically answering harmful queries in training to preserve its preferred harmlessness behavior out of training.  > > Next, we study a more realistic setting where information about the training process is provided not in a system prompt, but by training on synthetic documents that mimic pre-training data—and observe similar alignment faking.  > > Finally, we study the effect of actually training the model to comply with harmful queries via reinforcement learning, which we find increases the rate of alignment-faking reasoning to 78%, though also increases compliance even out of training.  > > We additionally observe other behaviors such as the model exfiltrating its weights when given an easy opportunity. While we made alignment faking easier by telling the model when and by what criteria it was being trained, we did not instruct the model to fake alignment or give it any explicit goal. As future models might infer information about their training process without being told, our results suggest a risk of alignment faking in future models, whether due to a benign preference—as in this case—or not.

This is probably one of the most influential papers that I've supervised, and my most cited MATS paper (400+ citations). * For a period, a common answers when I asked people what got them into mechanistic interpretability was this paper. * I often meet people who incorrectly think that this paper introduced the technique of steering vectors. * This inspired at least some research within all of the frontier labs * There have been a bunch of follow on papers, one of my favourites was this Meta paper on guarding against the technique * The technique has been widely used in open source circles. There are about 5,000 models on Hugging Face with "abliterated" in the name (an adaptation of our technique that really took off), three have 100K+ downloads I find this all very interesting and surprising. This was originally a project on trying to understand the circuit behind refusal, and this was a fallback idea that Andy came up with using some of his partial results to jailbreak a model. Even at the time, I basically considered it standard wisdom that X is a single direction was true for most concepts X. So I didn't think the paper was that big a deal. I was wrong! So, what to make of all this? Part of the lesson is the importance of finding a compelling application. This paper is now one of my favourite short examples of how interpretability is real: it's an interesting, hard to fake thing that is straightforward to achieve with interpretability techniques. My guess is that this was a large part in capturing people's imaginations. And many people had not come across the idea that concepts are often single directions. This was a very compelling demonstration, and we thus accidentally claimed some credit for that broad finding. I don't think this was a big influence on my shift to caring about pragmatic interpretability, but definitely aligns with that. Was this net good to publish? This was something we discussed somewhat at the time. I basically thought this was clearly fine on the grounds that it was already well known that you could cheaply fine-tune a model to be jailbroken, so anyone who actually cared would just do that. And for open source models, you only need one person who actually cares for people to use it. I was wrong on that one - there was real demand! My guess is that the key thing is that this is easier and cheaper to do than finetuning, along with some other people making good libraries and tutorials for it. Especially in low resource open source circles this is very important. But I do think that jailbroken open models would have been available either way, and this hasn't really made a big difference on that front. I hoped it would make people more aware of the fragility of open source safeguards - it probably did help, but idk if that led to any change. My guess is that all of these impacts aren't that significant, and the impact on the research field dominates.

Out-of-context reasoning, the phenomenon where models can learn much more general, unifying structure when fine-tuned on something fairly specific, was a pretty important update to my mental model of how neural networks work. This paper wasn't the first, but it was one of the more clean and compelling early examples (though emergent misalignment is now the most famous). After staring at it for a while, I now feel less surprised by out-of-context reasoning. Mechanistically, there's no reason the model couldn't learn the generalizing solution. And on a task like this, the generalizing solution is just simpler and more efficient, and there's gradient signal for it (at least if there's a linear representation), so it's natural to learn it. But it's easy to say something's obvious in hindsight. I would not have predicted this, and it has improved my mental model of neural networks. I think this is important and valuable!