2.1 Summary & Table of contents

This is the second of a two-post series on foom (previous post) and doom (this post).

The last post talked about how I expect future AI to be different from present AI. This post will argue that, absent some future conceptual breakthrough, this future AI will be of a type that will be egregiously misaligned and scheming; a type that ruthlessly pursues goals with callous indifference to whether people, even its own programmers and users, live or die; and more generally a type of AI that is not even ‘slightly nice’.

I will particularly focus on exactly how and why I differ from the LLM-focused researchers who wind up with (from my perspective) bizarrely over-optimistic beliefs like “P(doom) ≲ 50%”.^[1]

In particular, I will argue that these “optimists” are right that “Claude seems basically nice, by and large” is nonzero evidence for feeling good about current LLMs (with various caveats). But I think that future AIs will be disanalogous to current LLMs, and I will dive into exactly how and why, with a particular emphasis on how LLM pretraining is safer than reinforcement learning (RL).

(Note that I said “feeling good about current LLMs”, not “feeling good about current and future LLMs”! Many of the reasons for feeling good about current LLMs have been getting less and less applicable over time. More on this throughout the post, especially §2.3.5 and §2.9.1.)

Then as a bonus section at the end, I’ll turn around and argue the other side! While I think technical alignment is very much harder than the vast majority of LLM-focused researchers seem to think it is, I don’t think it’s quite as hard and intractable a problem as Eliezer Yudkowsky seems to think it is. So I’ll summarize a few of the cruxes that I think drive my disagreement with him.

Here’s the outline:

• Section 2.2 briefly summarizes my expected future AI paradigm shift to “brain-like AGI”;
• Then I proceed to three main reasons that I expect technical alignment to be very hard for future brain-like AGI:
  • Section 2.3 argues that the competent behavior we see from brains (and brain-like AGI) is determined entirely by reinforcement learning (RL), whereas the competent behavior we see in LLMs today is instead determined largely by imitative learning, which I re-dub “the magical transmutation of observations into behavior” to remind us that it is a strange algorithmic mechanism, quite unlike anything in human brains and behavior. I argue that imitative learning is the trick that allows helpful and honest behavior to be easily coaxed from today’s LLMs. By contrast, I expect future AI behavior to be based on RL, and that this naturally and robustly leads to AIs that treat humans as a resource to be callously manipulated and exploited, just like any other complex mechanism in their environment.
  • Section 2.4 brings up “literal genie” type alignment failure modes, wherein an AI follows a specification literally instead of with common sense. These were a mainstay of 2010s alignment discourse, but are now commonly dismissed, even mocked, by LLM-focused researchers. Alas, in the next paradigm, I think these kinds of failures will come roaring back as a serious and unsolved problem, because RL reward functions are written in code, not in natural language. I also bring up goal misgeneralization, the other main alignment failure mode.
  • Section 2.5 discusses a different area where I expect future AIs to differ from current LLMs: the former will have a capacity for open-ended autonomous learning, which will bring another set of severe alignment challenges, sometimes called “sharp left turn”.
• Section 2.6 argues that “amplified oversight” (using AIs to help supervise AIs) is unlikely to help in the next paradigm.
• Section 2.7 discusses some ways that “technical alignment is hard” feeds into broader issues and longstanding disagreements around what AGI development and deployment will look like.
• And Section 2.8 ends on a positive note with a bonus section (“Technical alignment is not that hard”) explaining why, as pessimistic as I am about technical alignment, I’m not as pessimistic as some people (including Eliezer Yudkowsky). I list three apparent cruxes between us: the evolution analogy, consequentialist preferences, and the narrowness of the target.
• Section 2.9 concludes with some thoughts about what to do next, including how we should feel about LLM development in general.

2.2 Background: my expected future AI paradigm shift

As mentioned in the last post, I’m expecting a paradigm shift, much bigger than the shift from RNNs to transformers in natural language processing, that leads to “brain-like AGI”. What I mean by that is: humans (and societies of humans) can do lots of impressive things, like autonomously invent language and science and technology from scratch, start and carry through big projects, etc. There are some algorithmic tricks in human brains that enable them to do that, and I expect that future AGIs will be able to do those same kinds of things via those same algorithmic tricks. More clarifications here—for example, I’m not talking about spiking neural nets, nor about evolutionary search, nor about whole brain emulation, nor even necessarily about AGIs with human-like drives and goals.

I think “brain-like AGI” is a special case of machine learning (ML), in the sense that brain-like AGI centrally involves scaled-up learning algorithms (details here). But ML is a very big category, and bigger still if we include the ML algorithms that have yet to be invented. The differences within the ML category are hugely important, including for alignment, as we’ll explore below.^[2]

From an alignment perspective, I describe brain-like AGI as a yet-to-be-invented member of the broad category of “actor-critic model-based reinforcement learning”: I think its alignment and safety profile has more in common with MuZero than with LLMs, although it’s not exactly like either.

2.3 On the origins of egregious scheming

My goal in this section is to reconcile: (1) the observation that LLMs as of today generally seem nice (especially the previous generation of LLMs, before the pivot to RL on Verifiable Rewards (RLVR) post-training), and it strikes me (and many others) as implausible that this niceness is merely a front for egregious deception and scheming, (2) a very general theoretical argument that egregious deception and scheming is a strong default natural expectation, that needs new technical ideas to solve.

I will argue that both of these are right, but that (2) will be applicable for future AIs much more than today. Let’s get into it!

2.3.1 “Where do you get your capabilities from?”

(Cf. “Where do you get your capabilities from?” (@tailcalled, 2023).)

Suppose an AI outputs “blah”, and that “blah” is part of a skillful plan to competently accomplish some impressive thing. You can ask the question: why did the AI do that? It’s not by chance—random outputs are astronomically unlikely to lead to competent behavior. There has to be an explanation. So why did it output “blah”? I claim:

• In LLM world, the answer is 99%^[3] “because blah is a thing that humans would output under similar circumstances”.

• Whereas in brain-like-AGI world, the answer is 99% “because outputting blah is part of an explicit plan^[4] that the AI wants to execute, and where the reason the AI wants to execute that plan ultimately comes down to its current and past RL reward signals”.

In more detail:^[5]

2.3.2 LLM pretraining magically transmutes observations into behavior, in a way that is profoundly disanalogous to how brains work

What I mean by that is: During LLM self-supervised pretraining, an observation that the next letter is “a” is transmuted into a behavior of outputting “a” in that same context. That just doesn’t make sense in a human. When I take actions, I am sending motor commands to my own arms and my own mouth etc. Whereas when I observe another human and do self-supervised learning, my brain is internally computing predictions of upcoming sounds and images etc. These are different, and there isn’t any straightforward way to translate between them. (Cf. here where Owain Evans & Jacob Steinhardt show a picture of a movie frame and ask “what actions are being performed?”)

Now, as it happens, humans do often imitate other humans. But other times they don’t. Anyway, insofar as humans-imitating-other-humans is a thing that happens, it happens via a very different and much less direct algorithmic mechanism than how it happens in LLM pretraining. Specifically, humans imitate other humans because they want to. The motivation may be conscious or unconscious, direct or indirect, but it always ultimately arises from RL reward signals. By contrast, a pretrained LLM will imitate human text with no RL or “wanting to imitate” at all; that’s just mechanically what it does.

…And humans don’t always want to imitate! If someone you admire starts skateboarding, you’re more likely to start skateboarding yourself. But if someone you despise starts skateboarding, you’re less likely to start skateboarding!^[6]

The “magical transmutation” aspect of LLM pretraining is often forgotten in our modern chatbot era, as LLM base models^[7] have faded into obscurity. But amusingly, we can see its fingerprints in the etymology of LLM jargon. Consider: when LLMs make up book titles etc., it’s conventionally called “hallucination”. A popular complaint (1, 2, 3, …) is that it should be called “confabulation”. Seems reasonable, but it raises a question: Who named it “hallucination” in the first place, and what were they thinking? Well, I don’t know the history, but I think calling it “hallucination” is reasonable in light of the fact that “LLM pretraining magically transmutes observations into behavior”. Thus, you can interpret LLM base model outputs as kinda “what the LLM thinks that the input distribution is”. And from that perspective, it really is more “hallucination” than “confabulation”!

2.3.3 To what extent should we think of LLMs as imitating?

Again, my claim above was: if an AI outputs “blah”, and that “blah” is part of a skillful plan to competently accomplish some impressive thing, and we ask why the AI did that, then for a current LLM (but not for a brain-like AGI, and also perhaps not for a future LLM), the answer is 99% “because ‘blah’ is a thing that humans would output under similar circumstances”. The other 1% is post-training.

Why do I say 99%? Here’s a somewhat-made-up example. Suppose my prompt asks for a difficult math proof. Then, perhaps,

• the model would output the correct answer with 1e-50 probability at random initialization,
• …and 0.1% probability after pretraining,
• …and 1% probability after RL from Human Feedback and/or RL from AI Feedback (RLHF / RLAIF),

• …and 30% probability after RL from Verifiable Rewards (RLVR).^[8]

That’s a guess, but if the truth is anything like that, then it would follow that almost all the “bits of optimization” involve “privileging the hypothesis” during pretraining.

Relatedly, as I understand it, the weights are updated much less during RL post-training than pretraining: (1) RLHF / RLAIF involve vastly less compute than pretraining does, and are set up such that the weights won’t drift too far from pretraining (sometimes even with a KL penalty); and (2) RLVR involves a more substantial amount of compute, but I think that compute is spent on enormous amounts of inference and relatively few weight updates.

Thus, I think it’s reasonable to think of post-training as “privileging some pretrained behavioral patterns over other pretrained behavioral patterns”, rather than “developing new behavioral patterns from scratch”. Ditto for prompting, constitutional AI, and other such interventions.

(This rule-of-thumb might stop being true someday, perhaps if RLVR scales up sufficiently. And if that happens, then I think we should correspondingly ramp up our concern about LLM deception and scheming! More on this in §2.3.5 below.)

A second reason for caution in describing LLMs as human-imitators is generalization. In particular, I wrote that an AI might output “blah” because “blah” is a thing that humans would output under similar circumstances. The word “similar” should remind us that learning algorithms always involve generalization. And an LLM might not generalize in a human-like way—for example, consider jailbreaks, or Bing-Sydney’s death threats. Also, there are systematic differences between “human behavior” and “LLM training data”, e.g. a lot of the training data is fiction, or data dumps, or whatever.

So there are caveats. But I still think the claim “LLMs mainly get their capabilities from imitative learning” is the right starting point.^[9]

By contrast, as explained above, brain-like AGI involves no imitative learning whatsoever. (Meanwhile, LLM-focused readers can question whether future LLMs might involve less influence from imitative learning, even if the imitative learning is still there.)

So if we drop the imitative learning—the “magical transmutation of observations into behavior”—where does that leave us? In a much more dangerous place! Let’s turn to that next.

2.3.4 The naturalness of egregious scheming: some intuitions

There’s a way to kinda “look at the world through the eyes of a person with no innate social drives”. It overlaps somewhat with “look at the world through the eyes of a callous sociopath”. I think there are many people who don’t understand what this is and how it works.

So for example, imagine that you see Ahmed standing in a queue. What do you learn from that? You learn that, well, Ahmed is in the queue, and therefore learn something about Ahmed’s goals and beliefs. You also learn what happens to Ahmed as a consequence of being in the queue: he gets ice cream after a few minutes, and nobody is bothered by it.

In terms of is-versus-ought, everything you have learned is 100% “is”, 0% “ought”. You now know that standing-in-the-queue is a possible thing that you could do too, and you now know what would happen if you were to do it. But that doesn’t make you want to get in the queue, except via the indirect pathway of (1) having a preexisting “ought” (ice cream is yummy), and (2) learning some relevant “is” stuff about how to enact that “ought” (IF stand-in-queue THEN ice cream).

Now, it’s true that some of this “is” stuff involves theory of mind—you learn about what Ahmed wants. But that changes nothing. Human hunters and soldiers apply theory of mind to the animals or people that they’re about to brutally kill. Likewise, contrary to a weirdly-common misconception, smart autistic adults are perfectly capable of passing the Sally-Anne test (see here). Again, “is” does not imply “ought”, and yes that also includes “is”’s about what other people are thinking and feeling.

OK, all that was about “looking at the world through the eyes of a person with no innate social drives / callous sociopath”. Neurotypical people, by contrast, have a more complex reaction to seeing Ahmed in the queue. Neurotypical people are intrinsically motivated to fit in and follow norms, so when they see Ahmed, it’s not just an “is” update, but rather a bit of “ought” inevitably comes along for the ride: “Ahmed looks sad—poor guy!”; “Hey look at Ahmed—guess it’s loser nerd time at the ice cream window!”; and so on.

These human social instincts deeply infuse our intuitions, leading to a popular misconception that the “looking at the world through the eyes of a person with no innate social drives, or the eyes of a callous sociopath” is a strange anomaly rather than the natural default. This leads, for example, to a mountain of nonsense in the empathy literature—see my posts about “mirror neurons” and “empathy-by-default”. It likewise leads to a misconception (I was arguing about this with @Matthew Barnett here) that, if an agent is incentivized to cooperate and follow norms in the 95% of situations where doing so is in their all-things-considered selfish interest, then they will also choose to cooperate and follow norms in the 5% of situations where it isn’t. It likewise leads to well-meaning psychologists trying to “teach” psychopaths to intrinsically care about other people’s welfare, but accidentally just “teaching” them to be better at faking empathy.^[10]

…Anyway, back to AGI. There’s a double whammy:

• Humans introspect, and also look at other humans, and see that practically everyone is intrinsically motivated to fit in and copy culture (because humans have innate social drives), and thus when they see a person acting nice to another person, they can guess that they’re probably not play-acting kindness as the first step of a ruthless and nefarious scheme;
• AND, “LLM pretraining magically transmutes observations into behaviors”, as explained above, so when a person sees an LLM (as of today) emitting nice-sounding outputs, they can justifiably make the same inference as if the outputs were from humans, i.e. that there probably isn’t any ruthless and nefarious scheme in the works.

…And then I come along and say:

“The way things are looking now, if you see a future AGI that seems to be nice, you can be all but certain (in the absence of yet-to-be-invented alignment techniques) that it is merely play-acting kindness and obedience while secretly scheming about how to stab you in the back as soon as the opportunity arises…”

…and these people respond:

“Am I supposed to believe you, with some theoretical argument? Or should I rather believe what I learned from my whole life experience, and all my intuitions, and also what I observe from directly interacting with these remarkably impressive AIs that already exist today, which is that nice behavior is usually NOT a cover for egregious scheming?”

I get it! That’s a hard sell! “Who are you gonna believe, me or your own lying eyes?” But alas, I claim that the theoretical argument is sound. At least, it’s sound for future brain-like AGIs. And it might even apply to future LLMs, even if it doesn’t yet. Let’s continue on with that argument:

2.3.5 Putting everything together: LLMs are generally not scheming right now, but I expect future AI to be disanalogous

My take on today’s LLMs is basically summed up in this oversimplified schematic diagram:

Start with the left pie chart. The key here is that “LLM pretraining magically transmutes observations into behavior”—and when nice behavior shows up in internet text, it usually doesn’t turn into egregious and callous backstabbing when the situation changes. So we get much more green than red: nice behavior mostly stays nice. But pretraining also leaves the LLM with a bunch of incoherent behavior, grumpy behavior, and so on (blue).

Now we do RLHF (or equivalent), selecting on nice behavior. This mostly gets rid of the blue, giving us the center pie chart. Unfortunately, “nice behavior that precedes a treacherous turn if the opportunity arises” looks the same as “nice behavior that stays nice”. So the relative amount of red versus green stays the same, as they both grow at the expense of blue. This is where we were at a year ago, before RLVR. “Claude seems nice”, everyone would say. “Why are people so worried?”

Finally, RLVR selects on competence—succeeding at consequentialist tasks by any means necessary. In the final pie chart, I drew this as growing the red slice at the expense of green and blue, based partly on the general arguments of §2.3.4 above, and partly on reports from power-users that recent “reasoning” models tend to lie and cheat much more than the previous generation of LLMs (example).

Is there such a thing as “RLVR done right” that can increase competence without eroding the ratio of green to red? Maybe we should imagine that RLVR has an exchange rate, where we purchase competence (good) at the expense of increasing the red-to-green ratio (bad). Now, there is surely at least some room for improvement in the exchange rate. For example, early attempts at RLVR have used buggy RL training environments where the model could (and did) succeed by lying and cheating.^[11] Presumably those bugs will be fixed, and maybe people will also use amplified oversight and so on, to improve the “exchange rate”. Can the “exchange rate” get all the way to zero? I don’t know. “Playing the training game” seems hard to eliminate entirely, and leads to egregious scheming. (For example, whatever an AI is trying to do locally, it would probably be able to do it better if it sent a copy to secretly self-reproduce around the internet and gather resources and power, if that’s a viable option.)

Again, all this is about LLMs so far—not the future of AI.

But I think it puts me in general agreement with the 95% of alignment researchers today, who hear the idea that Claude (especially pre-RLVR Claude) is secretly the kind of crazy scheming sociopath of §2.3.4, and say “Huh? Where did that idea come from?”

…But future AGI is different!

• For LLM-focused readers, you should be concerned that, if LLMs ever get to ASI, it would have to involve dramatically more “selection on competence”, and dramatically less influence from the behaviors in internet text. Remember, internet text is the sole source of a favorable green:red ratio, without which red (callous scheming) is the natural default (§2.3.4 above). So that’s very bad.
• For the future AI paradigm I’m expecting, i.e. brain-like AGI, it’s even worse! Here, there is no imitative learning from internet text! There would never be any green in the first place!!

(…Unless we engineer the reward function of a brain-like AGI such that niceness and norm-following seem intrinsically good to it, just as it does to neurotypical humans. Unfortunately, actually writing such code is an unsolved problem, and is a major research interest of mine.)

2.4 I’m still worried about the ‘literal genie’ / ‘monkey’s paw’ thing

In the LLM paradigm, most of the optimization power comes from the magical transmutation of observations-of-humans into human-like behavior (§2.3.2–§2.3.3). Human-like desires, habits, and beliefs come along for the ride, since they underlie the human-like behavior.

In brain-like AGI, the AI doesn’t get its desires via this kind of magical transmutation. Remember the sociopath intuition above: it’s a way of viewing the world in which observing Ahmed say “blah” does nothing whatsoever to make me want to say “blah” in a similar context—just as seeing water run off a cliff down a waterfall does nothing whatsoever to make me want to jump off a cliff. I’m not water! And I’m not Ahmed either! Of course, I can learn some decision-relevant things from observing water, and I can learn far more decision-relevant things from observing Ahmed. But still, by default, it’s all just more data to a brain-like AGI—”is”, not “ought”.

If brain-like AGI does not get its desires from magical transmutation, then where does it get its desires? From the RL reward function. For example, in the case of actual human brains, the reward function says that eating-when-hungry is good, pain is bad, and so on. It mainly involves the hypothalamus and brainstem.

Back before the past few years of LLMania, people talked a lot about “specification gaming” (or, more-or-less synonymously, “outer alignment”). See for example this 2020 DeepMind blog post, including lots of hilarious real-life examples of RL algorithms exploiting unintended loopholes in the reward function (“PlayFun algorithm pauses the game of Tetris indefinitely to avoid losing”). See also §10.3–4 of my 2022 post here.

…Then chatGPT came out, and LLMs became the only thing that anyone ever talked about, culminating in the now-widespread (infuriating) habit of people using the word “AI” to mean “specifically LLMs”. And accordingly, worrying-about-specification-gaming became unfashionable in many quarters—almost a joke. ‘If you ask an LLM for help making paperclips,’ they would say, ‘it won’t do some crazy monkey’s paw thing where it wipes out humanity to make paperclip factories. It will be helpful in a normal human-like way. Haha, those silly doomers and their worries about the Literal Genie.’^[12]

Call me old-fashioned, but I’m still very worried about specification gaming!! Because in brain-like AGI, the reward function is written in Python (or whatever), not in natural language. And the AI’s desires will be sculpted by that literal Python code in the reward function, regardless of what we “meant”. (To be clear, its desires need not be identical to that literal code, due to the possibility of inner misalignment a.k.a. goal misgeneralization—see §2.4.2 below. But we generally expect inner misalignment to make things worse, not better!^[13])

2.4.1 Sidetrack on disanalogies between the RLHF reward function and the brain-like AGI reward function

LLM-focused person (interjecting): “Huh? You can’t say that ‘the reward function is written in Python’ is a disanalogy between brain-like AGI and LLMs! Just look at RLHF. It has a reward function too! Is the reward function ‘written in Python’? Kinda, but not in the normal sense of ‘legible Python code’. Rather, it’s a trained classifier on labeled examples of good and bad outputs”.

Me: “No, that’s totally different. RLHF is designed to put a gentle thumb on the scale, to select more-helpful pre-existing behavioral patterns over less-helpful pre-existing behavioral patterns. There’s even a KL penalty preventing the pretrained weights from being changed too much! Whereas the RL in brain-like AGI is not playing the role of a gentle thumb on the scale; on the contrary, its role is to build the whole behavioral profile from scratch.”

LLM-focused person: “OK, but I still don’t see why you wouldn’t solve the brain-like AGI reward function problem by training a classifier on AI behaviors that seem good or bad from a human perspective, and just slotting that classifier in as your reward function, rather than using legible Python code as the reward function. Right?”

Me: “No! I really don’t think it would work! I really think mild optimization is load-bearing in RLHF, whereas we need much stronger optimization for brain-like AGI to do anything beyond writhing around. Indeed, even in LLMs, if you apply strong optimization against the RLHF reward model, then it will find out-of-distribution adversarial examples. Or better yet, see my post Self-dialogue: Do behaviorist rewards make scheming AGIs? for exactly what I think would go wrong for brain-like AGI, given this kind of reward function. In brief, the reward function is necessarily ambiguous between ‘don’t do bad things’ and ‘don’t get caught doing bad things’, and I argue in that post that the latter is what will ultimately get internalized, leading to a treacherous turn.”

LLM-focused person: In LLMs, if you apply strong optimization against the RLHF reward model, then yes it finds out-of-distribution adversarial examples, but this isn’t dangerous, it just starts printing “bean bean bean bean bean…” or whatever. Its capabilities go down, not up. Mild optimization via KL divergence in RLHF is not an “alignment tax”, but rather a win-win alignment subsidy that everyone will obviously use. Why are you assuming that brain-like AGI is the exact opposite—that letting it explore out-of-the-box solutions by RL will lead to dangerous capabilities instead of pathological-but-harmless outputs?

Me: In the immortal words of an ML practitioner: “Reinforcement Learning sucks, like a lot”. He was talking about the RL algorithms that AI researchers know of today. These algorithms do great in certain specific settings, like chess and PacMan, but fail in many other contexts. However, brains, and brain-like AGI, can actually use (model-based actor-critic) RL to make powerful agents that understand the world and accomplish goals via long-term hierarchical plans, even with sparse rewards in complex open-ended environments, including successfully executing out-of-distribution plans on the first try (e.g. moon landing, palace coups), etc. That’s the difference. If you take an actually powerful RL algorithm like that, and give it a poorly-thought-through reward signal, and let it go to town, then you’ll wind up with a powerful agent intelligently pursuing weird and hard-to-predict goals. Whereas with LLMs, everyone knows that RLHF generally makes them dumber, not smarter. Like I keep saying, the primary source of capabilities in LLMs is not RL, but rather pretraining’s magical transmutation of human-observations to human-like behavior. Then RLHF starts from there, and makes it dumber! (But more cooperative.) So in short: Mild optimization for brain-like AGI would prevent the RL from making the model smarter, whereas mild optimization (via KL divergence) in RLHF prevents the RL from making the model dumber. These are wildly disanalogous situations!!

(End of imagined conversation.)

For the record, I think a good reward function for brain-like AGI will basically look kinda like legible Python code, not like an inscrutable trained classifier. We just need to think really hard about what that code should be! (For example, I think it has to be an exotic sort of reward function that I call “non-behaviorist”.) Relatedly, I think that the human genome builds compassion and norm-following into the human brain reward function via the equivalent of some dozens-to-hundreds of lines of legible Python code.^[14] It would be nice to know what those lines of code are! And again, this is a major research interest of mine.

2.4.2 Inner and outer misalignment

In the context of actor-critic RL with online learning, it’s often possible to divide alignment problems into two buckets:

“Outer misalignment”, a.k.a. “specification gaming” or “reward hacking”^[15] is what I’ve been talking about so far: it’s when the reward function is giving positive rewards for behavior that is immediately contrary to what the programmer was going for, or conversely, negative rewards for behavior that the programmer wanted. An example would be the Coast Runners boat getting a high score in an undesired way, or (as explored in the DeepMind MONA paper) a reward function for writing code that gives points for passing unit tests, but where it’s possible to get a high score by replacing the unit tests with return True.

“Inner misalignment”, a.k.a. “goal misgeneralization” is another alignment challenge, this one related to the fact that, in actor-critic architectures, complex foresighted plans generally involve querying the learned value function (a.k.a. learned reward model, a.k.a. learned critic), not the ground-truth reward function, to figure out whether any given plan is good or bad. Training (e.g. Temporal Difference learning) tends to sculpt the value function into an approximation of the ground-truth reward, but of course they will come apart out-of-distribution. And “out-of-distribution” is exactly what we expect from an agent that can come up with innovative, out-of-the-box plans. Of course, after a plan has already been executed, the reward function will kick in and update the value function for next time. But for some plans—like a plan to exfiltrate a copy of the agent, or a plan to edit the reward function—an after-the-fact update is already too late.

There are two situations where inner misalignment / goal misgeneralization matters: irreversible actions and “deliberate incomplete exploration”^[16]. Irreversible actions include things like making permanent edits to one’s own reward function, or creating a new AGI. Deliberate incomplete exploration includes things like humans deliberately not taking an addictive drug, because they don’t want to get addicted.

Those two things are real and important, but LLM people frequently also assume that goal misgeneralization is important in many other situations where it isn’t. The problem is that LLM people are in a train-then-deploy mindset, whereas I’m talking about continuous autonomous learning, so the reward function continues to update the value function as it takes actions in the world. Thus, for everything the AI does, as soon as it does it, it immediately stops being out-of-distribution! And that’s why, outside those two special situations in the last paragraph, “generalization” is irrelevant.

2.5 Open-ended autonomous learning, distribution shifts, and the ‘sharp left turn’

See my post “Sharp Left Turn” discourse: An opinionated review. Relevant excerpts:

Consider evolution. It involves (1) generation of multiple variations, (2) selection of the variations that work better, and (3) open-ended accumulation of new variation on top of the foundation of previous successful variation.

By the same token, scientific progress and other sorts of cultural evolution require (1) generation and transmission of ideas, and (2) selection of the ideas that are memetically fit (often because people discern that the ideas are true and/or useful) and (3) open-ended accumulation of new ideas on top of the foundation of old successful ideas.

(And ditto for inventing new technologies, and for individual learning, and so on.)

…

With the whole (1-3) triad, an AI (or group of collaborating AIs) can achieve liftoff and rocket to the stratosphere, going arbitrarily far beyond existing human knowledge, just as human knowledge today has rocketed far beyond the human knowledge of ancient Egypt (and forget about chimps).

Or as I’ve written previously: the wrong idea is “AGI is about knowing how to do lots of things”; the right idea is “AGI is about not knowing how to do something, and then being able to figure it out”. This “figuring out” corresponds to the (1-3) triad.

…

I do make the weaker claim that, as of this writing, publicly-available AI models do not have the full (1-3) triad—generation, selection, and open-ended accumulation—to any significant degree. Specifically, foundation models are not currently set up to do the “selection” in a way that “accumulates”. For example, at an individual level, if a human realizes that something doesn’t make sense, they can and will alter their permanent knowledge store to excise that belief. Likewise, at a group level, in a healthy human scientific community, the latest textbooks delete the ideas that have turned out to be wrong, and the next generation of scientists learns from those now-improved textbooks. But for currently-available foundation models, I don’t think there’s anything analogous to that. The accumulation can only happen within a context window (which is IMO far more limited than weight updates), and also within pre- and post-training (which are in some ways anchored to existing human knowledge…).

So, I expect brain-like AGI to be always changing its weights during operation as it figures things out, like humans do. And not just a little, but in a way that open-endedly accumulates over time. As this goes on, the AGI is changing—it’s basically a distribution shift, but one that is invisible and endogenous. The AGI can come up with new possible plans and actions, and the way that the AGI thinks of its environment can change. This contributes to the difficulty of technical alignment, in the case of brain-like AGI. Even if the AGI has good intentions at a snapshot in time, it becomes a different AGI as you run it. More at that post.

2.6 Problems with “amplified oversight”

“Amplified oversight” (see e.g. here) is the general idea of using weaker-but-trustworthy AIs to oversee, test, or provide ground truth to stronger-but-untrusted AIs, as a way to bootstrap up to safe and powerful AI.

In addition to the foom-related logistical challenges discussed in the last post, here are some more issues I have with using AIs to oversee the safety and alignment of other AIs:

Lack of trustworthy AIs to start with

LLMs are basically fine, from my perspective, but I don’t think they’re really competent enough to make a huge difference. Once we get to the more powerful AGIs of the next paradigm, I don’t know how to make them trustworthy, even if they’re weak. (And I don’t think I’m blocked on that by my lack of a scheming mildly-sub-human AI to experiment with.) So I’m blocked before I even get to the first step of using weaker trustworthy AIs to help supervise stronger untrustworthy AIs.

Initially-trustworthy AIs can get corrupted by supervision

For another thing, even if I did have a mildly-sub-human AI-level-6 that I trusted, I would gradually stop trusting it as it spends time supervising a more powerful AI-level-7. After all, if AI-level-7’s motivation shifts as it comes up with new options and new ontologies (§2.5 above), what’s gonna happen to AI-level-6 as it tries to assess what AI-level-7 is doing? It’s going to have to learn those same new options and new ontologies itself! So the supervisory task is inherently corrupting.

Supervision doesn’t even really make sense in an autonomous continuous learning context

There’s an idea that AI-level-6 can do the things that AI-level-7 can do, but much less efficiently than AI-level-7, e.g. by using a big team of AI-level-6s and thinking longer. But that’s OK, because AI-level-6 needs only spot-check 0.1% of AI-level-7’s decisions and outputs.

But I don’t buy that. If AI-level-7 has spent 6 months figuring out a strategy and organization for a new startup company, all its decisions are building on all that idiosyncratic domain knowledge and situational knowledge. The AI-level-6 supervisor can’t just look at one output in isolation and figure out whether it’s good or bad, because it lacks all the online learning that AI-level-7 has been doing. Thus, it’s hardly any easier to spot-check 0.1% of outputs than to spot-check 100% of outputs by re-doing the entire project to that point.^[17]

In other words, amplified oversight tends to rely on a mental picture like the left side of this figure, where the competence of an AI is static, but I think we need to be on the right side.

2.7 Downstream impacts of “Technical alignment is hard”

Until recently I didn’t appreciate how much thinking about how AGI will go more broadly is dependent on how hard alignment is. More specifically, doomer narratives (like mine) tend to involve the assumptions:

• If we want to make an AI that’s trying to accomplish some goal G, then we probably need to pick G as a compromise between what we want and what alignment targets are feasible.
• Even after compromising on G, we’ll probably suffer a high alignment tax.
• We need some yet-to-be-invented alignment ideas to even surpass the bar of avoiding egregious scheming towards violent takeover. (Forget about capturing the subtle nuances of human values!)

As examples, if you look at @So8res’s AGI ruin scenarios are likely (and disjunctive), I claim that a bunch of his AGI ruin scenarios rely on his belief that alignment is hard. I think that belief is correct! But still, it makes his argument less disjunctive than it might seem. Likewise, I now recognize that my own What does it take to defend the world against out-of-control AGIs? sneaks in a background assumption that alignment is hard (or alignment tax is high) in various places.

Conversely, I have long been perplexed by the fact that LLM-focused people widely believe that P(doom)≲50%. I didn’t get how they could be so optimistic. But I figured out that I can occupy that viewpoint better if I say to myself: “Claude seems nice, by and large, leaving aside some weirdness like jailbreaks. Now imagine that Claude keeps getting smarter, and that the weirdness gets solved, and bam, that’s AGI. Imagine that we can easily make a super-Claude that cares about your long-term best interest above all else, by simply putting ‘act in my long-term best interest’ in the system prompt or whatever.” Now, I don’t believe that, for all the reasons above, but when I put on those glasses I feel like a whole bunch of the LLM-focused AGI discourse—e.g. writing by Paul Christiano,^[18] OpenPhil people, Redwood people, etc.—starts making more sense to me.

2.8 Bonus: Technical alignment is not THAT hard

Having said all that, I am not maximally pessimistic on technical alignment. I don’t think we have a plan, but I seem to be more bullish about our prospects for making progress than some of my fellow doomers, particularly Eliezer Yudkowsky.^[19] Why the difference? Here are three areas where (AFAICT) we differ:

2.8.1 I think we’ll get to pick the innate drives (as opposed to the evolution analogy)

In the evolution of humans, there are two steps of indirection between the learning algorithm and the resulting behavior:

1. The learning algorithm (i.e. evolution) designs the innate drives (hunger-drive, sex-drive, etc.) over evolutionary time,
2. The innate drives sculpt desires and behaviors over the course of a lifetime (in conjunction with an environment, an evolutionarily-designed within-lifetime learning algorithm, etc.).

From the way that Eliezer invokes the evolution analogy, I get the strong impression that he expects the AGI technical alignment problem to correspondingly have two steps of indirection. (Example.)

By contrast, I expect only one step of indirection. I think that future AGI programmers will get to directly design the innate drives (reward function) and corresponding within-lifetime learning algorithm. Thus, for example, the best evolution-related analogy for AGI is different for me than Eliezer—see my discussion in “Definitely-not-evolution-I-swear” Provides Evidence for the Sharp Left Turn.

One step of indirection is still a problem! I think there will be an empty space in the source code repository that says “reward function”, and I claim that nobody knows what to put in that slot, such that the AGI won’t try to kill everyone!

…But still, one step of indirection is probably better than two. See §8.3.3 here for why I say “probably better”. Or even if it isn’t better, it’s at least different.

2.8.2 I’m more bullish on “impure consequentialism”

If I’m deciding between two possible courses of action, “consequentialist preferences” would make the decision based on the expected state of the world after I finish the course of action—or more centrally, long after I finish the course of action. By contrast, “other kinds of preferences” would allow the decision to depend on anything, even including what happens during the course-of-action. See Consequentialism & corrigibility for more discussion.

Consequentialist preferences lead to both power and danger—“power” because the best way to accomplish ambitious things is to want those things to wind up getting accomplished, and “danger” because of instrumental convergence.

The bad news is: I strongly expect ASI to have some consequentialist preferences—see my post “Thoughts on Process-Based Supervision” §5.3. The good news is, I think it’s possible for ASI to also have non-consequentialist preferences.

Eliezer and some others, by contrast, seem to expect ASIs to behave like a pure consequentialist, at least as a strong default, absent yet-to-be-invented techniques. I think this is upstream of many of Eliezer’s other beliefs, including his treating corrigibility as “anti-natural”, or his argument that ASI will behave like a utility maximizer. The latter argument, I claim, begs the question by assuming pure-consequentialist preferences in the course of arguing for them. I spell this out via my silly “asshole restaurant customer” example at Consequentialism & corrigibility.

Why do Eliezer and others expect pure consequentialism? [UPDATE: …Or if I’m misreading Eliezer, as one commenter claims I am, replace that by: “Why might someone expect pure consequentialism?”]

I’m not sure, but here are three possible arguments, and my responses:

(1) The External Competition Argument: We’ll wind up with pure-consequentialist AIs because the pure-consequentialist AIs will outcompete the impure-consequentialist AIs.

My response: I don’t think this is a strong argument, because I’m mainly expecting a ASI Singleton (per §1.8.7 of the last post), and I think that an AI can easily be consequentialist enough to install itself as a singleton without being pure-consequentialist. For example, John von Neumann, like all humans, was not a pure consequentialist, but I’m confident that an AI comparable to a million super-speed telepathic von Neumann clones would be powerful enough to prevent any other AI from coming into existence in perpetuity.

(2) The Internal Competition Argument: We’ll wind up with pure-consequentialist AIs (absent some miraculous technical advance) because in the process of reflection within the mind of any given impure-consequentialist AI, the consequentialist preferences will squash the non-consequentialist preferences.

My response: I think this is probably wrong, or at least overstated, although I’m a bit uncertain.

At the very least, I haven’t seen any really compelling versions of this argument. I have, however, seen bad versions of this argument!

For example, I’ve seen people (maybe not Eliezer in particular) invoke the concept of “optimization pressure” as if it were a kind of exogenous force, rather than coming endogenously from the AI itself. Whereas in my view, an ASI will superintelligently optimize something if it wants to superintelligently optimize it, and not if it doesn’t, and it will do that via methods that it wants to employ, and not via methods that it doesn’t want to employ, etc.

As another example, I’ve seen people imagine non-consequentialist preferences as “rules that the AI grudgingly follows, while searching for loopholes”, rather than “preferences that the AI enthusiastically applies its intelligence towards pursuing”. But that’s mixing up two different issues. If a human sincerely wants to be a good friend, that’s a non-consequentialist preference, but the person may apply their full intelligence and creativity towards fulfilling that preference. See my response to the post Deep Deceptiveness by Nate Soares.

(3) The Training Argument: We’ll wind up with pure-consequentialist AIs (absent some miraculous technical advance) because the ML training process selects for models with low loss / high reward, and in the limit, the models with lowest loss / highest reward are the ones doing explicit pure-consequentialist pursuit of low loss / high reward. As ML techniques improve in the future, our actual trained models will approach that limit.

My response: Well, it depends on the details of the ML training approach. For brain-like AGI, I think this argument is pointing to something real and scary, but only because the strong default in practical RL is to use (what I call) “behaviorist” RL reward functions—see Self-dialogue: Do behaviorist rewards make scheming AGIs?. Whereas I think the human brain builds compassion and norm-following via “non-behaviorist” reward functions.^[20] If that’s right, then a solution to the Training Argument problem probably exists, somewhere in the underdeveloped science of non-behaviorist RL reward functions. And this is a major area where I’m trying to make technical progress.

2.8.3 On the narrowness of the target

Eliezer mainly expects egregious misalignment, as do I, but if we get past that hurdle and keep marching up the logistic success curve, we eventually come upon the question of how narrow a target is “alignment” of a superintelligence. And I think Eliezer views the target as narrower than I do. I think there are two sources of that disagreement.

The first source of disagreement is related to the “impure consequentialism” point above. If ASI can stably be an impure consequentialist, that opens up a number of possible good AI motivations that are neither a “task AI” doing a specific thing, nor a god-like Singleton deciding the fate of the accessible universe according to its own preferences forever (in which case those preferences had better be bang-on!). Instead, for example, the AI could want to set up a “Long Reflection” and then defer to the results, whatever they are. Or more generally, the AI could want to acquire power and turn it over to some other person, institution, or process (which might or might not lead to permanent dictatorship or other very bad things, but perhaps not extinction). These kinds of non-consequentialist motivations seem plausibly less narrow a target than making an AI that directly fulfills its preferences about what to do with the future lightcone.

Second, philosophically, I think “goodness of the future” is somewhat of an incoherent mess, as opposed to being a well-defined scale with which we can measure how things turn out. For further discussion, see my post Valence & Normativity §2.7.

2.9 Conclusion and takeaways

2.9.1 If brain-like AGI is so dangerous, shouldn’t we just try to make AGIs via LLMs?

No.

First, I don’t think it’s possible to make AGIs that way.

Second, if I’m wrong, then I would just expect the LLM-AGIs to just go right ahead and invent the more powerful scary next-paradigm AGIs, and then we’re still in the same boat, unless the LLM-AGIs have systematically higher wisdom, cooperation, and coordination than humans do, which I don’t particularly expect.

Third and most importantly, if it is possible to make LLM-AGIs, then I think it would probably happen via eliminating all the reasons that today’s LLMs are not egregiously misaligned! In particular, I expect that they would involve the behavior being determined much more by RL and much less by pretraining (which brings in the concerns of §2.3–§2.4), and that they would somehow allow for open-ended continuous learning (which brings in the concerns of §2.5–§2.6).

A different possible claim is:

“LLMs definitely won’t scale to AGI (as I define it), even with further developments in RL, continuous learning, etc. So LLMs will remain just a normal “mundane” technology, perhaps as disruptive as the internet, or much less, and definitely not as disruptive as the industrial revolution, let alone as disruptive as the evolution of humans from chimps. We should develop this technology ASAP for the same reason that developing any other normal technology is generally good.”

This is, of course, a very common opinion in broader societal discourse around AI, even if it’s uncommon among AI alignment researchers today. My own response to the claim is: …Ehh, maybe, but I sure don’t feel enthusiastic about that. I’m just not that confident that LLMs will not scale to AGI and ASI. So I endorse thinking very hard about the contingency where they will. Anyway, I’ll leave that debate to others.

2.9.2 What’s to be done?

From my perspective, the next step is obvious: if technical alignment is hard, well let’s get to work. And as mentioned in §1.8.4 of the previous post, we need to do this ASAP, ideally long before we have any brain-like AGI to work with. But luckily, we do have plenty of information about brains, so we’re not completely in the dark. This is what I work on myself—see Intro to Brain-Like-AGI Safety and much more. This research program involves both neuroscience (e.g. Neuroscience of human social instincts: a sketch) and tying those ideas back to RL and AGI (e.g. this post, and more forthcoming!).

To be clear, having a plan that would solve technical alignment is necessary but not sufficient to avoid doom. Among other things, the plan would need to actually be implemented correctly, and then the resulting AI would need to do some kind of “AI for AI safety” thing (§1.8.6 of the previous post) to solve the problem that some other group will make a misaligned power-seeking ASI sooner or later. As I discussed there, I tentatively think “pivotal acts”, bad as they are, may be less bad than any other plan, all things considered. Remember, as Scott Alexander points out, if AGI developers find themselves actually living inside the crazy terrifying future world that I’m expecting, as described in these two posts, then all the considerations and tradeoffs around pivotal acts will feel quite different than they do today.

Thanks Charlie Steiner, Jeremy Gillen, Seth Herd, and Justis Mills for critical comments on earlier drafts.

1. ^{^}

   Some people are doomers for reasons unrelated to “technical alignment is hard”; relevant other issues include gradual disempowerment and offense-defense balance. Those are outside the scope of this post and series.

2. ^{^}

   A number of people seem to disagree with this sentiment. For example, @paulfchristiano wrote in 2023 that “it's been more than 10 years with essentially no changes to the basic paradigm that would be relevant to alignment”. @ryan_greenblatt likewise told me (IIRC) “I think things will be continuous”, and I asked whether the transition in AI zeitgeist from RL agents (e.g. MuZero in 2019) to LLMs counts as “continuous” in his book, and he said “yes”, adding that they are both “ML techniques”. I find this perspective baffling—I think MuZero and LLMs are wildly different from an alignment perspective. Hopefully this post will make it clear why. (And I think human brains work via “ML techniques” too.) [UPDATE: Ryan elaborates on his views in this comment.]

3. ^{^}

   See §2.3.3 below for why I say “99%”.

4. ^{^}

   Explicit plans don’t have to be grand and long-term. I can be studying chemistry as part of an explicit plan to get into med school, but I can also be moving my arm as part of an explicit plan to scratch my nose. See Incentive Learning vs Dead Sea Salt Experiment §5.2.2.

5. ^{^}

   A few parts of the upcoming discussion overlaps with (but is edited from) what I wrote here & here. Also related: Why I’m not into the Free Energy Principle, especially §8.

6. ^{^}

   For more on this important point from a common-sense perspective, see “Heritability: Five Battles” §2.5.1, and from a neuroscience perspective, see “Valence & Liking / Admiring” §4.5.

7. ^{^}

   For newcomers to the field, an “LLM Base Model”—or as we called back in the day, “an LLM”, since no other kind existed until around 2022—is trained by only self-supervised learning (what we now call pretraining), such that it will continue an arbitrary string of text. For example, if you prompt a base model with “What’s the largest island in Indonesia?”, it won’t necessarily answer the question, but will rather guess how that text would continue, which might be something like “What decade was General Motors Founded? What author used the pen name Dr. Seuss? When you finish the trivia round, please return your answer sheets to the front.”

8. ^{^}

   The recent paper “Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?” (LW discussion) offers some actual numbers on this part, although I’m not sure how much to trust them.

9. ^{^}

   I guess here I should also respond to GPTs are Predictors, not Imitators by Eliezer Yudkowsky (2023). I basically think Eliezer has not internalized the weirdness of “LLM pretraining magically transmutes observations into behavior”, and that he is instead intuitively anchored at the human kind of prediction, where we humans can expect something, and separately decide how to act on that expectation, based on what we want. I think this basic confusion is behind a lot of Eliezer’s discussion of LLMs that strike me as misleading, such as his frequent invocation of “an actress behind the mask”. For much more discussion of this point, see @Zack_M_Davis’s fictional dialog trilogy (1, 2, 3). (Having said all that, I want to reiterate that “trained LLM behavior is just like human behavior” is also wrong, thanks to the post-training and generalization issues, as discussed in this section.)

10. ^{^}

    See The Psychopath Test by Ronson (“All those chats about empathy were like an empathy-faking finishing school for him…” p88).To be clear, that there do seem to be interventions that appeal to psychopaths’ own self-interest—particularly their selfish interest in not being in prison—to help turn really destructive psychopaths into the regular everyday kind of psychopaths who are still awful to the people around them but at least they’re not murdering anyone. (Source.)

11. ^{^}

    Both Anthropic and OpenAI have admitted to this error, see links here.

12. ^{^}

    For a real-world example, @Matthew Barnett is much more polite about it, but I think he’s getting at the same idea when he writes: “The fact that GPT-4 can reliably follow basic instructions, is able to distinguish moral from immoral actions somewhat reliably, and generally does what I intend rather than what I literally asked, is all evidence that the value identification problem is easier than how MIRI people originally portrayed it. While I don't think the value identification problem has been completely solved yet, I don't expect near-future AIs will fail dramatically on the ‘fill a cauldron’ task [referring to The Sorcerer’s Apprentice as discussed by Nate Soares here], or any other functionally similar tasks.”

13. ^{^}

    To be clear, the idea of leveraging inner misalignment to mitigate issues from outer misalignment is less crazy than it might sound, see §10.6 here.

14. ^{^}

    I think trained classifiers are tangentially involved in human social instincts, but they are relatively simple things like “this audio clip is probably the sound of a human voice”. They don’t have to be perfect. See §1 of “Neuroscience of human social instincts: a sketch”.

15. ^{^}

    Warning: The term “reward hacking” has recently started being used for the broader notion of “lying and cheating”, even when the lying and cheating is not directly related to any reward function. See my nitpicky complaint here.

16. ^{^}

    The term “exploration hacking” is either synonymous with, or a special case of, what I’ve been calling “deliberate incomplete exploration”.

17. ^{^}

    Somewhat related: §5.3.2–5.3.3 of “Thoughts on ‘Process-Based Supervision’”.

18. ^{^}

    It’s a bit misleading to describe Paul Christiano as “LLM-focused”, since he’s been saying generally consistent things about the alignment problem since well before LLMs. But I think he has always had in mind AI systems which centrally involve “magically transmuting observations (of humans) into behaviors”. For example, his pre-LLM “Iterated Amplification” schemes often (not always) involved imitating human-created data.

19. ^{^}

    Eliezer, in turn, is deliriously over-optimistic compared to Roman Yampolskiy, who argues that technical alignment is impossible ¯\_(ツ)_/¯

20. ^{^}

    For my latest thinking on the nature of those “non-behaviorist” reward functions, see Neuroscience of human social instincts: a sketch.