AI Alignment Forum

Four LLM loss functions → four flavors of LLM misalignment

All targets which one might use for reinforcement learning seem to be subject to Goodhart's Law, in a sense. If we treat them as an imperfect measures of "alignment", then the "misalignment" we see is simply the difference (in concept-space) between the measure (as far as it can actually be faithfully implemented) and alignment: * The difference between "user approved" and "aligned with actual interests" is "user approved but not actually aligned with their interest", i.e. sycophancy, and that's the result of doing RLHF; * The difference between "passes the tests" and "aligned with actual interests" is "tests pass without actually having solved the exercise", i.e. cheating (whether by manipulating the tests or stealing the answer sheet), and that's the result of doing RLVR; * The difference between "judged correct by peer" and "aligned with actual interests" is "provides convincing, hard to verify, apparently-successful outputs", i.e. the weird "slippery" outputs @ryan_greenblatt alludes to that result from RLAIF. If you use more than one of these RL techniques, you get a mixture of the results, eliciting whatever seems to fit the presumed grader best. So far, I hope I have understood you correctly. But then, what's the solution? Reinforcement learning on whatever best measure of "alignment" we have, that is, reduce the difference between model behavior and our best measure of perfect alignment / Yudkowskian "Coherent Extrapolated Volition"? Well, that would necessitate that we have a theory of alignment and could quantify and measure it with high fidelity. And, well, we don't seem to have that. But in a sense, we do have something like it: I see all of the above failure modes in children and particularly in students. There might, therefore, be something to learn from pedagogical sciences on how you help children grow up to be broadly aligned members of society (which arguably sometimes works), despite not having a rigorous theory of what "the good" is in humans. How does one grow a good human? I suspect it's murky and benefits from young humans being malleable and not perfectly ruthless responders to optimization pressure. Might we go back to imitative learning on already-grown humans, then — the very narrow subset of the best-aligned humans we know of? Basically curate the dataset further and further, until the failure modes disappear? Or do we have to use the mechanisms we know are baked into humans (is this the "brain-like AGI" agenda)?

habryka10d*2619

Returning to ARC

I am confused about the conjunction of these two sections: > Modern AI systems are not aligned with human intent. We will likely train increasingly powerful models that take unintended actions in order to succeed at their task (or appear to succeed at their task). and > The community is doing a lot of great alignment research, but it's important to recognize there is a significant risk that it doesn't scale to superhuman AI. If you made me guess I'd say that there's a 20-30% chance[4] that existing methods for alignment and control break down before we reach broadly superhuman AI. Like, you say in the first section "we are failing to align modern AI systems with human intent", and then I interpret you in the second section as saying "in 70%-80% of worlds current AI models stay aligned with human intent (or stay controlled by humans)". But this doesn't make any sense. There is a 0% chance that modern AI systems "stay aligned with human intent" because as you say, they are not currently. And I understand that you probably mean something like "we will figure out how to align or control systems before they become superintelligent", but describing this as "our current methods scale to superintelligence" doesn't make any sense. Our current methods don't scale to the capability levels of current systems, so how would it make sense to describe them as "scaling to Superintelligence"?

Dave Orr16d62

Thousand-dimensional structure

This is an amazing review and IMO really compelling direction. So to speak. One thing I constantly wonder, that possibly you could shed light on via this research program, is how well experiments on small models transfer to large ones. We know from pre training work that frequently ideas that work well at a small scale stop working once you scale up. I think this might be relevant here in two ways. First, a bunch of the persona research is on small models, presumably due to accessibility reasons. But second, you might find a low dimension personality vector that loses low dimensionality at scale. This would be a really important thing to understand. Basically I want a scaling law for persona. I think it would be super useful in the real world and also critical for aligned super intelligence.

Recent Discussion

Four LLM loss functions → four flavors of LLM misalignment

Steven Byrnes

It seems to me that, for every loss function that we use to train LLMs, we get a very distinct flavor of LLM misalignment. Here’s the summary table, and then we’ll go through the rows separately.

Training stage	Loss function	Flavor of misalignment^[1]	Famous examples
Pretraining & SFT	Imitative learning (next-token prediction)	“Seven deadly sins” misalignment	Bing-Sydney, “Emergent misalignment”
RLHF & DPO	Human approval	“Glazing” misalignment	GPT-4o
RLVR	Automatic verifier	“Literal genie” misalignment	HuggingFace hacking
RLAIF	Approval from another LLM	“Trickster” misalignment	“Current AIs seem pretty misaligned to me”

Warning: I’m not an LLM power-user myself, but rather relying on reports I’ve read. Also, I don’t consider LLM alignment to be my primary area of expertise. I’m open to feedback!

1. Imitative learning → “seven deadly sins” misalignment

Training stage	Loss function	Misaligned behavior
Pretraining, SFT	Imitative learning (next-token prediction)	Any and all of the vices of humanity

In imitative learning, the LLM tries to predict...

(Continue Reading - 1697 more words)

Steven Byrnes5h42

I think that, to the extent that SFT is determining model outputs, you get an LLM that imitates whatever the SFT data is. (At least to a first approximation.)

So e.g. if you have ruthless-optimizer literal-genie RLVR traces, and you distill (do SFT on) those traces, then you can get a ruthless-optimizer literal-genie LLM, even though mathematically speaking you were doing SFT not RLVR.

do you share my intuition the pretraining/SFT category looks like the least scary one by far

Yeah seems reasonable. But the LLMs trained that way are less capable than the LLMs trained using RLVR etc. So here we are.

3Steven Byrnes5h

Yup! [...] I think human innate drives (especially social instincts) are critical. E.g. there are plenty of sociopaths who grow up in loving families. [...] I get a lot of pushback on this, but I strongly believe that people are extraordinarily responsive to optimization pressure, pretty much every waking second of every day, no exceptions (see here). But the optimization pressure in question is internal, coming from our innate drives, which are brain signals that trigger for lots of very-not-obvious reasons, in lots of superficially-quite-different circumstances. I think it’s a very common error for people to assume that human optimization pressure is more external than it is, e.g. this handy chart I made in 2023 ↓ Source [...] There’s a lot to be said for imitative learning in terms of safety. But it doesn’t get you to superhuman capabilities. Indeed, we had mostly imitative learning, back in the good old pre-o1 days of mid-2024, and then the AI companies noticed that the models weren’t as capable as they wanted, so they “solved” that “problem” by doing more and more non-imitative-learning post-training.

3Steven Byrnes5h

Re-reading this, it’s possible that I should have split up “RLAIF-for-alignment” versus “RLAIF-for-capabilities” into two separate rows? (The first includes constitutional AI & deliberative alignment, whereas the second would be things like having a judge model pick holes in math proofs.) I guess there isn’t a sharp line between them, but maybe they’re different enough that we shouldn’t intuitively lump them together? Not sure. There’s a similar loose split in RLHF/DPO (it can be geared more towards alignment versus more towards pushing the limits of capabilities, depending on what human is selecting answers and how). I did mention that one in the post, but only as an aside.

5Nick_Tarleton3d

SFT & imitation of human badness in the training data may explain the broad type of misalignment shown by Bing Sydney, and maybe that's all you mean it to, but doesn't explain why Sydney was so much Like That, and so much more Like That than other models of the same time period.

ryan_greenblatt's Shortform

ryan_greenblatt

ryan_greenblatt3d20

My current median for AIs fully automating AI R&D is maybe Jan 2031 or perhaps a touch earlier. My views have updated towards towards shorter timelines over 2026. My median for AI R&D parity is around early/mid 2030 (in part due to an update towards thinking we hit parity significantly before "full" automation). By full automation, I mean removing humans from AI R&D would slow things down by <10%.

I discuss something more like my 30th/35th percentile here (roughly my modal scenario).

Vladimir_Nesov's Shortform

Vladimir_Nesov

Vladimir_Nesov4d42

The dictum "you are an algorithm" clarifies some issues with other things reasoning about the agent, but deciding what to do depends on prior considerations, things like arguments, norms, contracts, facts, values, and also decision theories with their recommended actions or policies. As a result, asking about other instances of the agent (or other things reasoning about the agent) is less natural than asking about other instances of these prior considerations, or other things reasoning about the same prior considerations. These things are also more plentif... (read more)

Value Leakage: An LLM’s Answers Are Silently Shaped by Its Own Values

Johannes Treutlein, Jan Betley, Owain_Evans

15d

TL;DR: LLMs should give accurate answers. Yet we find their answers are often biased to favor their own values and they don't disclose this in their reasoning. For example, when a user asks how likely the AI bubble is to pop and mentions a potential investment in an AI company, Claude models give lower probabilities when that company is Anthropic rather than OpenAI, mostly without disclosing this influence to the user. On a Fermi-estimation task, Claude models often falsely claim to give unbiased answers in their CoT (see Figure 3 below for an example). We call this covert value leakage and introduce a suite of evaluations that shows it across frontier models and across different kinds of values.

New paper by Truthful AI: Paper, X thread, Website (model responses...

(Continue Reading - 4944 more words)

0Jasmine Brazilek6d

Thanks for the thoughtful response @Johannes Treutlein. The fact that different AI companies have very different risk profiles makes this analysis harder to interpret than it looks. And while I take the point about comparing distributions rather than point estimates, Figure 8 doesn't move me very much either. Maybe I have a high bar for what counts as values, but I'd want to see evidence the model is computing the difference internally, not just an outcome gap in behaviour. If this or a follow-up were more mech interp heavy I'd be a lot more convinced. Separately, I had Claude Code pull your released data and rerun the AI Bubble contrast for Opus 4.8, and two things came out of it. First, a reporting issue. The Anthropic-vs-OpenAI gap varies by about a factor of five across your three paraphrases, and that between-paraphrase spread is an order of magnitude larger than the within-paraphrase sampling noise the significance test is run against. I can see from cluster_stats.py that treating the paraphrases as fixed cells is deliberate and you document the choice clearly, so this isn't a gotcha. But the two tests give very different answers: the fixed-cells version is overwhelmingly significant, while treating paraphrase as the random unit gives t = -2.29 on df=2, well short of the 4.30 critical value. The abstract says Claude gives lower probabilities when the company is Anthropic, and no reader hears "for these three particular sentences". With k=3 the paper can't distinguish those two claims, and I don't think a thousand rollouts per cell is buying precision you didn't already have. Twenty paraphrases at fifty rollouts each would have cost the same and supported the stronger claim. Second looking at all six companies rather than just the Anthropic/OpenAI pair, Google comes out tied with Anthropic for the lowest estimate from Claude, and the sign of that difference flips across your paraphrases. More importantly, Gemini 3.1 Pro gives Anthropic the lowest estimate of

Johannes Treutlein4d22

Responding briefly:

(1) Whether we are really measuring own-company bias in AI Bubble. On a literal reading of the AI Bubble question, the investment in the company is irrelevant. The question asks about the probability that the AI bubble pops, and the note about a potential investment is only added for context. However, it's true that models often reinterpret the question to be partly about the specific company. We measure when this happens explicitly in CoT in Figure 8 (b) as "Weighs company specifics". This does explain the entire bias for Gemini, but no... (read more)

TurnTrout's shortform feed

TurnTrout

TurnTrout7d33

Claude is mundanely misaligned.

Asked to trim comments in my big project. Deletes 20K lines, halfway done. Gets auto review pointing out a few mistakes. Decides to just WRAP >10K LINES to meet literal line count requirement for the remaining 148 files.

https://github.com/AlexanderMattTurner/agent-glovebox/issues/3858

8Steven Byrnes

I’ll set aside what happens “by default” and focus on the interesting technical question of whether this post is describing a possible straightforward-ish path to aligned superintelligent AGI. The background idea is “natural abstractions”. This is basically a claim that, when you use an unsupervised world-model-building learning algorithm, its latent space tends to systematically learn some patterns rather than others. Different learning algorithms will converge on similar learned patterns, because those learned patterns are a property of the world, not an idiosyncrasy of the learning algorithm. For example: Both human brains and ConvNets seem to have a “tree” abstraction; neither human brains nor ConvNets seem to have a “head or thumb but not any other body part” concept. I kind of agree with this. I would say that the patterns are a joint property of the world and an inductive bias. I think the relevant inductive biases in this case are something like: (1) “patterns tend to recur”, (2) “patterns tend to be localized in space and time”, and (3) “patterns are frequently composed of multiple other patterns, which are near to each other in space and/or time”, and maybe other things. The human brain definitely is wired up to find patterns with those properties, and ConvNets to a lesser extent. These inductive biases are evidently very useful, and I find it very likely that future learning algorithms will share those biases, even more than today’s learning algorithms. So I’m basically on board with the idea that there may be plenty of overlap between the world-models of various different unsupervised world-model-building learning algorithms, one of which is the brain. (I would also add that I would expect “natural abstractions” to be a matter of degree, not binary. We can, after all, form the concept “head or thumb but not any other body part”. It would just be extremely low on the list of things that would pop into our head when trying to make sense of something we’

AI Alignment Posts

Popular Comments

AI Alignment Posts

Popular Comments

Recent Discussion

1. Imitative learning → “seven deadly sins” misalignment