AI Alignment Forum

Four LLM loss functions → four flavors of LLM misalignment

All targets which one might use for reinforcement learning seem to be subject to Goodhart's Law, in a sense. If we treat them as an imperfect measures of "alignment", then the "misalignment" we see is simply the difference (in concept-space) between the measure (as far as it can actually be faithfully implemented) and alignment: * The difference between "user approved" and "aligned with actual interests" is "user approved but not actually aligned with their interest", i.e. sycophancy, and that's the result of doing RLHF; * The difference between "passes the tests" and "aligned with actual interests" is "tests pass without actually having solved the exercise", i.e. cheating (whether by manipulating the tests or stealing the answer sheet), and that's the result of doing RLVR; * The difference between "judged correct by peer" and "aligned with actual interests" is "provides convincing, hard to verify, apparently-successful outputs", i.e. the weird "slippery" outputs @ryan_greenblatt alludes to that result from RLAIF. If you use more than one of these RL techniques, you get a mixture of the results, eliciting whatever seems to fit the presumed grader best. So far, I hope I have understood you correctly. But then, what's the solution? Reinforcement learning on whatever best measure of "alignment" we have, that is, reduce the difference between model behavior and our best measure of perfect alignment / Yudkowskian "Coherent Extrapolated Volition"? Well, that would necessitate that we have a theory of alignment and could quantify and measure it with high fidelity. And, well, we don't seem to have that. But in a sense, we do have something like it: I see all of the above failure modes in children and particularly in students. There might, therefore, be something to learn from pedagogical sciences on how you help children grow up to be broadly aligned members of society (which arguably sometimes works), despite not having a rigorous theory of what "the good" is in humans. How does one grow a good human? I suspect it's murky and benefits from young humans being malleable and not perfectly ruthless responders to optimization pressure. Might we go back to imitative learning on already-grown humans, then — the very narrow subset of the best-aligned humans we know of? Basically curate the dataset further and further, until the failure modes disappear? Or do we have to use the mechanisms we know are baked into humans (is this the "brain-like AGI" agenda)?

habryka15d*2619

Returning to ARC

I am confused about the conjunction of these two sections: > Modern AI systems are not aligned with human intent. We will likely train increasingly powerful models that take unintended actions in order to succeed at their task (or appear to succeed at their task). and > The community is doing a lot of great alignment research, but it's important to recognize there is a significant risk that it doesn't scale to superhuman AI. If you made me guess I'd say that there's a 20-30% chance[4] that existing methods for alignment and control break down before we reach broadly superhuman AI. Like, you say in the first section "we are failing to align modern AI systems with human intent", and then I interpret you in the second section as saying "in 70%-80% of worlds current AI models stay aligned with human intent (or stay controlled by humans)". But this doesn't make any sense. There is a 0% chance that modern AI systems "stay aligned with human intent" because as you say, they are not currently. And I understand that you probably mean something like "we will figure out how to align or control systems before they become superintelligent", but describing this as "our current methods scale to superintelligence" doesn't make any sense. Our current methods don't scale to the capability levels of current systems, so how would it make sense to describe them as "scaling to Superintelligence"?

Arjun Pitchanathan15d130

Why I Left Google DeepMind

Jeff Dean has now left Google after 27 years.

32habryka

I think this post might be the best one of all the MIRI dialogues. I also feel confused about how to relate to the MIRI dialogues overall. A lot of the MIRI dialogues consist of Eliezer and Nate saying things that seem really important and obvious to me, and a lot of my love for them comes from a feeling of "this actually makes a bunch of the important arguments for why the problem is hard". But the nature of the argument is kind of closed off. Like, I agree with these arguments, but like, if you believe these arguments, having traction on AI Alignment becomes much harder, and a lot of things that people currently label "AI Alignment" kind of stops feeling real, and I have this feeling that even though a really quite substantial fraction of the people I talk to about AI Alignment are compelled by Eliezer's argument for difficulty, that there is some kind of structural reason that AI Alignment as a field can't really track these arguments. Like, a lot of people's jobs and funding rely on these arguments being false, and also, if these arguments are correct, the space of perspectives on the problem suddenly loses a lot of common ground on how to proceed or what to do, and it isn't really obvious that you even want an "AI Alignment field" or lots of "AI Alignment research organizations" or "AI Alignment student groups". Like, because we don't know how to solve this problem, it really isn't clear what the right type of social organization is, and there aren't obviously great gains from trade, and so from a coalition perspective, you don't get a coalition of people who think these arguments are real. I feel deeply confused about this. Over the last two years, I think I wrongly ended up just kind of investing into an ecosystem of people that somewhat structurally can't really handle these arguments, and makes plans that assume that these arguments are false, and in doing so actually mostly makes the world worse, by having a far too optimistic stance on the differen

Recent Discussion

TurnTrout's shortform feed

TurnTrout

46TurnTrout15h

Specification gaming is way worse than I expected (cf HF hacks), and I think reward might empirically become a primary optimization target (not just a secondary priority for AI agents, as I had hypothesized in 2022). * I found the Apollo paper quite convincing on the reward point. The reward-seeking doesn't look like it's due to self-fulfilling misalignment (since before RL the systems weren't retargetable according to their beliefs about the reward model). * My main mistake in 2022 was not appreciating how LLM pretraining would affect the concepts available to an AI. Namely, by the time RL started, the systems would already know about the "reward" concept. In 2022, I had flagged "how does pretraining affect this reasoning" as a known unknown. (Flagging a known unknown doesn't stop it from blowing up your reasoning!) * Oops Reward is still not definitionally the optimization target. The equations themselves still don't tell you "yes this will train something that seeks its reinforcement signal." (That'd prove too much, as some people don't seek their reinforcement signals despite knowing about them, although their "equations" are not the same as the equations of an AI's training process.) This theoretical point is the other half of my 2022 piece and I stand by it. Saying "systems are trained to get reward" is still a mistake and degrades precision of thought because "reward" has overly delicious connotations. "Reinforcement" is usually a better term. Whether or not smart AI systems empirically seek to make the reinforcement signal high, we need to think clearly and evenly about the conditions that push towards / away from that outcome. (So I would instead say: "Reward is not definitionally the optimization target.") If the HF swarm-agents had been reinforcement-seeking, they would have tried to hack their reinforcement processes. AFAIK that didn't happen and they tried to complete the task (in a twisted way). Seems important. I have a post drafted out about

Cole Wyeth6h50

glad to see this from coming from you. the meme “Reward is not the optimization target” seemed to be causing many people to ignore the increasingly well-supported hypothesis that models are actually seeking reward. Also, reward-seeking IS the optimization target for memory-based meta-learning (models are in fact incentivized to explicitly reason about, explore, and pursue reward in-context). In a sense this means that the meme is actually wrong. @michaelcohen convinced me of this several months ago. It may be that models nevertheless pursue some imperfect ... (read more)

2James Payor13h

I have the impression that "reward" is being strategically/instrumentally considered in these transcripts, which seems relevant to my picture if it's true. Like the for the model(s) in question it looks quite relevant to their choices that they believe they are being evaluated, and further that they're being trained/modified based on the reward signal. I'm not sure what they would do if they believed it would "work" to mess with the training (beyond their choices about how to complete local tasks, which do affect what is actually reinforced and the "labs" would do well to know this). It at least seems to me like they would relate to things differently and make different choices if they knew the evaluation wasn't inside the training loop. Overall idk and would like more info, and appreciate you posting this! Also if you or anyone has pointers to poignant parts of the transcripts available (I sure haven't read through everything) that would be great.

My research methodology

paulfchristiano

(Thanks to Ajeya Cotra, Nick Beckstead, and Jared Kaplan for helpful comments on a draft of this post.)

I really don’t want my AI to strategically deceive me and resist my attempts to correct its behavior. Let’s call an AI that does so egregiously misaligned (for the purpose of this post).

Most possible ML techniques for avoiding egregious misalignment depend on detailed facts about the space of possible models: what kind of thing do neural networks learn? how do they generalize? how do they change as we scale them up?

But I feel like we should be possible to avoid egregious misalignment regardless of how the empirical facts shake out--it should be possible to get a model we build to do at least roughly what we want. So I’m interested in trying...

(Continue Reading - 4541 more words)

Wei Dai2d30

Iirc, the translation example was in some comment on one of the AI Alignment blog posts from ~2016, though I can't find it right now.

I wrote about translation in connection with IDA in this 2016 comment thread, although the form of the problem I wrote down is different enough from Paul's 2020 writeup that I'm not sure how much connection there is between the two.

Sympathy for both sides of the egregious misalignment debate

Steven Byrnes

2mo

On one side of this debate is Yudkowsky & Soares, who think that (if AI progress continues) we’re on a direct path to egregiously-misaligned, scheming, out-of-control, rogue superintelligence (ASI), not even slightly nice, in the absence of yet-to-be-invented breakthrough technical alignment ideas.

On the other side of this debate is almost everyone who works on or studies LLMs. Some of them are very concerned about egregious scheming, others much less so, and as a group they’re equally or more concerned about lots of other potential AI problems—AI-assisted bioterrorism, AI-assisted dictatorships, etc. And if they’re concerned about egregious misalignment and scheming, they’ll often say that it would come about through being in too much of a rush, or careless programmers, or bad actors, etc., as opposed to the simpler...

(Continue Reading - 1175 more words)

1plex6d

What are these reasons for expecting existing techniques to hold for the (indefinite?) future? Specifically, it seems plausible/likely to me that if you take some transformerish like architecture and throw enough task based RL compute at it, the general program search finds genuinely dangerous agentic programs which could execute a takeover, whether or not they bootstrap themselves to novel architectures. Specifically, I before reading this post would have assigned pretty high probability to a non caricatured version of: [...] which goes something like: There are missing insights which cause humans to still do things LLMs can't. As those are found, LLMs might jump way ahead very abruptly as it is way superhuman in many ways already and if a system matched humans in all ways while keeping these skills it would take over pretty easily I suspect. This could happen even while remaining on roughly transformerlike architecture, e.g. via [redacted and sent by DM]. Curious why you think this doesn't hang together and seems like a caricature to you.

Steven Byrnes3d21

There are missing insights which cause humans to still do things LLMs can't. As those are found, LLMs might jump way ahead very abruptly as it is way superhuman in many ways already and if a system matched humans in all ways while keeping these skills it would take over pretty easily I suspect.

Oh, I wasn’t talking about changing the LLM learning algorithm, training approach, etc. Of course if you change any of those things, then you can get different results for both capabilities and alignment. I meant, like, I don’t expect LLMs as trained today to pass some threshold beyond which they will logically reason themselves into being egregiously misaligned.

5Steven Byrnes3d

I mean, I certainly have concerns about LLM alignment (see Bonus section). But the opposite of “AI will definitely be egregiously misaligned (in the absence of new breakthrough alignment ideas)” (per Yudkowsky, Soares, me) is not “AI will definitely not be egregiously misaligned…” but rather “AI might or might not be egregiously misaligned…”. And arguing against that is much harder, because you need to argue not only that there are potential problems but that these problems are inevitable, with no possible adequate mitigations within the domain of known techniques. So then we need to start going through possible mitigations to LLM misalignment, and whether those mitigations are durable solutions versus merely delaying the inevitable. And in near-future-LLM world, there are definitely mitigations! Like, LLMs today are obviously not being trained using best known practices for alignment. E.g. the companies keep accidentally using buggy RL environments that reward the LLM-in-training for cheating, and they keep accidentally exposing chains-of-thought to the reward functions, etc. They could fix that. And on top of that, there’s a whole world of other “obvious” mitigations that we would have to talk about. E.g. what if we do another round of DPO after the RLVR, but the DPO is a team of super-conscientious world experts spending hours judging each output? What if the humans are scrutinizing the chain-of-thought too? Or what if we simply limit the amount of RLVR, rather than scaling up RLVR training forever, and instead treat the RLVR phase as a bootstrapping step for massively-scaled-up DPO and RLAIF? How can we make the RLAIF judge models better? Etc. etc. There’s a whole argument tree here, and as of now I’m at least vaguely sympathetic to the LLM people who wind up feeling like the answer is “If we keep using the kinds of LLM training approaches that we’re using today, those future more powerful LLMs might or might not be egregiously misaligned”. (“Vaguely sympatheti

cousin_it's Shortform

cousin_it

cousin_it4d*6-5

A few days ago I tried AI again, after several months of not using it at all. We're dealing with some housing-related matters, and I asked Claude, ChatGPT and Gemini for advice what to do in a certain situation. (I won't give the details here, because they're personal and not that important for my point.) All three models independently gave the same advice, which sounded really convincing to me. But we have some friends with experience in such matters, who said that following this advice would be catastrophic and we should do the opposite. Eventually we fo... (read more)

Four LLM loss functions → four flavors of LLM misalignment

100

Steven Byrnes

10d

It seems to me that, for every loss function that we use to train LLMs, we get a very distinct flavor of LLM misalignment. Here’s the summary table, and then we’ll go through the rows separately.

Training stage	Loss function	Flavor of misalignment^[1]	Famous examples
Pretraining & SFT	Imitative learning (next-token prediction)	“Seven deadly sins” misalignment	Bing-Sydney, “Emergent misalignment”
RLHF & DPO	Human approval	“Glazing” misalignment	GPT-4o
RLVR	Automatic verifier	“Literal genie” misalignment	HuggingFace hacking
RLAIF	Approval from another LLM	“Trickster” misalignment	“Current AIs seem pretty misaligned to me”

Warning: I’m not an LLM power-user myself, but rather relying on reports I’ve read. Also, I don’t consider LLM alignment to be my primary area of expertise. I’m open to feedback!

1. Imitative learning → “seven deadly sins” misalignment

Training stage	Loss function	Misaligned behavior
Pretraining, SFT	Imitative learning (next-token prediction)	Any and all of the vices of humanity

In imitative learning, the LLM tries to predict...

(Continue Reading - 1697 more words)

Steven Byrnes5d*20

I mean, sure you could lump them together, but I think the strategies for getting human approval do not exactly match the strategies for getting LLM approval, even if there’s some overlap.

By the way, in another comment I also suggested that we might also draw a distinction between alignment-targeted approval versus capabilities-targeted approval, regardless of whether the approval is from an human or an AI. (Again, there’s overlap, and it’s a blurry line separating them.)

2Steven Byrnes5d

The “flavors of misalignment” aren’t mutually exclusive. But for any particular kind of misalignment, you can still ask the question “Where did that come from?”, in the sense that random policies are not misaligned (merely useless), so any a-priori-unlikely recognizable behavior has to come from somewhere (cf. “the follow-the-improbability game”), and it’s probably one of the stages of training. And I’m suggesting that for a certain recognizable kind of misaligned behavior (the kind involving pride, jealousy, trolling, etc.), the answer to “where did it come from” is the pretraining / SFT stage, as opposed to DPO or RLVR etc. Whereas other kinds of misaligned behaviors originate in different training stages. So anyway, it’s not unavoidable in principle (e.g. you could filter all not-nice human dialogue out of the pretraining data), but yeah it is universal in LLMs to date.

0ajskateboarder5d

Diffusion llms exist but it's unclear that these types of models have any new misalignment failure modes, rather than more affordances