Sympathy for both sides of the egregious misalignment debate

Steven Byrnes

On one side of this debate is Yudkowsky & Soares, who think that (if AI progress continues) we’re on a direct path to egregiously-misaligned, scheming, out-of-control, rogue superintelligence (ASI), not even slightly nice, in the absence of yet-to-be-invented breakthrough technical alignment ideas.

On the other side of this debate is almost everyone who works on or studies LLMs. Some of them are very concerned about egregious scheming, others much less so, and as a group they’re equally or more concerned about lots of other potential AI problems—AI-assisted bioterrorism, AI-assisted dictatorships, etc. And if they’re concerned about egregious misalignment and scheming, they’ll often say that it would come about through being in too much of a rush, or careless programmers, or bad actors, etc., as opposed to the simpler Yudkowsky & Soares story of “we get egregious misalignment and scheming because nobody has the foggiest idea how to avoid that”.

Here’s my brief idiosyncratic take on this debate. I think BOTH of the following are true:

(1) If you really think carefully about the properties of ASI, you really do find good reasons to strongly expect it to be egregiously misaligned, scheming, and ruthless, in the absence of yet-to-be-invented breakthrough technical alignment ideas.
(2) If you really think carefully about the properties of current LLMs, you really do find good reasons to think that existing technical alignment techniques are adequate now, and may well continue to be adequate in the future.

So then here are three (caricatured) positions:

My position:

(1) and (2) are both totally true. And we can reconcile them by saying that LLMs won’t scale to ASI.

Yudkowsky & Soares’s position [caricatured]:

(1) is totally true. We know this with great confidence, having spent decades thinking about it.
So it follows that (2) must be wrong or irrelevant.
Why is (2) wrong or irrelevant? Hard to say! There’s no ASI yet, and nobody knows in detail how it will appear. Sometimes it’s easier to predict what happens eventually than the detailed path. An ice cube in warm water will melt eventually, but don’t ask me to predict how many seconds it will take to melt, etc.
So anyway, one possibility is that (2) is wrong because LLMs will kinda ‘wake up’, or something, when the core pieces of true intelligence finally come together. And then their behavior would change drastically for the worse. And maybe we’re already starting to see glimmers of that in existing LLMs?
Or another possibility [cf. Eliezer tweet] is that LLMs will invent non-LLM ASI. And then (2) will be simply irrelevant!
…Or something else! Again, we don’t know! But we do know that (1) is definitely right.

LLM people’s position [caricatured]:

(2) is totally true. We know this with great confidence, because we are LLM experts and we have thought about these alignment plans in great detail, including matching our theories against real-world data.
So it follows that (1) must be incorrect.
Why is (1) incorrect? I don’t really know! Man, I read Yudkowsky and Soares, and it’s all these words, words, words, and I’m reading along and trying to match those words to my knowledge of LLMs and it just doesn’t make any damn sense. I can and will try to respond to their points in detail, but honestly the core issue is that they’re guilty of head-in-the-clouds armchair theorizing gone off the rails.

Conclusion

…So I think that both sides of the debate are basically coming from a reasonable and sympathetic place, with a big kernel of truth.

Bonus section: Further commentary

…That said, I can still complain at both sides!

My “true objection” to Yudkowsky & Soares:

For the record, my “true objection” to Yudkowsky & Soares is that if we’re talking about ASI, then LLMs are basically irrelevant and we shouldn’t even be talking about LLMs at all. And meanwhile, their plans are misguided because delaying ASI is possible on the margin but mostly hopeless, although I guess I’m happy that they’re trying anyway. Meanwhile, my hunch is that they’re overstating the intractability of finding that technical alignment breakthrough, although I haven’t found it yet, so I guess time will tell.

My within-frame complaint at Yudkowsky & Soares:

…But I’ll put that aside for the sake of argument, and bring up a narrower complaint within their frame:

I think their suggestions that LLMs may become completely egregiously misaligned in the future via … umm … the ‘true core of intelligence’ coming together, and ‘waking up’? Like Skynet or something?? That was mean, sorry, but in any case, I don’t think this idea hangs together either theoretically or empirically.

For the former (theory), see my discussion of the extreme weirdness of the LLM pretraining algorithm in Foom & Doom §2.3.2. I think Yudkowsky & Soares have not internalized how weird this type of learning algorithm is, and if they had, then Yudkowsky would not be occasionally suggesting that we should think of an LLM as an actress playing characters.

For the latter (empirical), I think the most fair assessment is that current LLMs are nice and obedient in some contexts, and LLMs are mean, defiant, and just plain weird in other contexts. You can straightforwardly go from that observation to “maybe there will be egregious misalignment and scheming in the future”, but not to “there will definitely be egregious misalignment and scheming in the future, absent new breakthrough technical alignment ideas”.

I think that if Yudkowsky & Soares stopped treating current LLMs as direct evidence for technical alignment being definitely completely unsolved, and instead treated it as either mixed evidence or entirely off-topic, then their public messaging would come across to policymakers and general audiences as somewhat more convoluted and confusing. But I think it would be more accurate. Oh well.

My “true objection” to LLM people:

For the record, my “true objection” to the LLM people is that I don’t really care about anything they say, because I’m working on the ASI alignment problem, and LLMs won’t scale to ASI.

(I’m overstating a bit. I’m generally happy for people to work on making LLM-world a place of wisdom and goodness, especially because LLM-world is the world in which ASI will someday be invented.)

My within-frame complaint at LLM people:

…But I’ll put that aside for the sake of argument, and bring up a narrower complaint within their frame:

I think the LLM people are not pricing in the predictable consequences of ever more RLVR and/or the predictable consequences of ever more “real” open-ended continual learning, should the latter ever be solved (which I don’t think it will be, but never mind that).

In other words, lots of LLM-focused people say “LLMs will eventually be able to do the things that humanity did over the last 5000 years: open-endedly and autonomously build new knowledge and ideas on top of new knowledge and ideas, in an endless tower, with no need for human-provided ground truth anywhere in that process. And how exactly will the future LLMs do that? Uhh, I don’t know, people are working on it, I guess they’ll probably figure something out.”

…And bam, that blank spot in the map is where the pea gets hidden under the thimble.

Because if you want the LLMs to gain ever more knowledge, whether through a perpetual RLVR loop or some other yet-to-be-invented type of continual learning, there has to be some kind of ground truth, or else it will go off the rails into nonsense. And that ground truth, whatever it is, will basically amount to an objective function (a.k.a. cost function, reward function, whatever). And when the LLM updates enough on that ground truth, then whatever human-niceness that the LLM inherited from pretraining will get diluted away in favor of ruthless maximization of that objective function.

Thanks Zack M. Davis for a brief discussion that inspired this post.

And if they’re concerned about egregious misalignment and scheming, they’ll probably say that it would come about through race dynamics, careless programmers, bad actors, etc., as opposed to the simpler Yudkowsky & Soares story of “we get egregious misalignment and scheming because nobody has the foggiest idea how to avoid that”.

Citation needed? I think most people who are pretty worried about egregious misalignment are worried about it emerging naturally and being at least moderately difficult to prevent.

I guess you would know better than me, so I changed it from "probably" to "often" and from "race dynamics" to "being in too much of a rush" (the latter is what I meant all along but hopefully clearer). But I'm reluctant to go further than that. I do really have an impression that most people in LLM world think that things that people have been doing (constitutions, deliberative alignment, inoculation prompting, using interp to test for eval-awareness, etc.) is what progress on alignment looks like, and if egregious misalignment and scheming happens in the future it would be either because the good guys were in too much of a hurry to iterate and develop more and better techniques in that same general genre, or because the good guys were not the ones building ASI.

LLMs won’t scale to ASI

Not directly, but if humans automate the process of using LLMs to build the next generation of LLMs, this process of prosaic RSI is plausibly good enough to count as AGI (where the LLMs themselves still don't count as AGI; model size scaling slows down after 2030-2031, at quadrillion param models). It still likely won't scale to ASI, unable to quickly learn deep skills and thus make fast conceptual progress, but to the extent that it helps invent ASI (possibly over many years), it might also help invent the concepts of ASI alignment.

So alignment of LLMs is necessary, but whether it's sufficient remains unclear. It depends on how capable LLMs become at quadrillion param scale, after the prosaic RSI loop closes and they become self-building (most relevantly, automatically preparing the training data for RL, to teach the next model new concepts and skills). Possible cruxes are that LLMs never reach that point (with current training methods), or that they remain mostly useless at conceptual progress even with the prosaic RSI loop closed, or that they somehow proceed to invent ASI quickly, while not being wise enough to solve ASI alignment first (since this scenario doesn't involve a gradual process of getting smarter significantly beyond human level).

I think "transformative AI could be slightly nice" arguments aren't logically dependent on LLMs-as-AGI per se, even if belief in the two are correlated: ^[1] Christiano's formulation (very roughly, that it's not obvious why the evolutionary quirks leading to humans not being maximally ruthless couldn't have ML analogues) doesn't seem to depend on levels higher than "(D) systems centrally involving deep learning" in your plateau-ism taxonomy.

Where delusional optimism would be an obvious candidate for the source of the correlation. ↩︎

I think the right starting point is not whether something is an LLM, or deep learning, but rather what are the inputs, outputs, loss functions, etc.? And then go from there to whether we expect slight-niceness or not.

My own opinion (stated without justification) is: you can get niceness through LLM-style “true” imitation learning (Foom & Doom §2.3.2). Alternatively, if the AI is choosing actions through RL and/or model-based search & planning, rather than through imitation learning, than I expect zero-niceness, and instead the ruthless pursuit of the objective, or of something vaguely related to the objective, with ample specification gaming and so on (e.g. “be helpful” gets ruthless-ified into “come across as helpful”).

…Except that there exist weird objective / reward / cost functions that don’t have that property, but rather support niceness. And humans wound up with such a function via evolution doing an outer-loop search over reward functions in a certain type of environment where niceness was advantageous. In principle, future AI programmers could likewise do an outer-loop search over reward functions, but they probably won’t, because any kind of outer-loop search over scaled-up learning algorithms is hella expensive. If they do it at all, it would be a situation where the programmer crafted the reward function up to a handful of adjustable parameters, and then the outer-loop search would be a kind of hyperparameter tuning. And then the alignment challenges would be (1) crafting a reward function (up to the handful of unknown adjustable parameters) that supports niceness, (2) figuring out what the outer-loop test environment and selection criterion is, such that the selected reward function hyperparameters will lead to niceness towards humans in the real post-ASI world despite the wild distribution shift from the test environment. That’s basically what I’m working on, and I claim that not only are these open problems but that all the ideas in the literature will almost definitely fail.

In the framing of the post, I think much (most?) of the disagreement is downstream of whether we'll even choose to pursue the kind of ASI for which the theoretical arguments dominate the prosaic LLM-style safety arguments. LLMs or other non-limits-of-intelligence technologies with better safety properties could very plausibly scale far enough to satisfy the wants of people developing AI and/or end competitive pressures to build more ASI-like things.

This argument seems weak on two fronts.

There's no reason to think LLM safety today will scale if the LLM undergoes RSI and/or scales to ASI. Or maybe the LLM will discover the ASI algorithm and code that and deploy it.
Today's LLMs are misaligned, see Ryan Greenblatt's recent post.

RE 1, sure, “LLM will invent non-LLM ASI” is possible in principle, and would be a special case of “LLMs do not scale to ASI”. I do mention that (in the “Yudkowsky & Soares’s position [caricatured]” section).

RE 2, he wrote that “current AIs seem pretty misaligned”, not that current AIs are egregiously misaligned, scheming, and ruthless. I obviously do not think we should extrapolate from empirical observation of today’s LLMs to future ASI, but if I DID so extrapolate, I think my attitude would be vaguely like “eh, maybe future ASI will be egregious misaligned and scheming, even if people really try hard using known techniques, but probably not? And even if it happens to some degree, the AIs would still probably be at least slightly nice, and maybe that’s good enough?” That would be the kind of thing LLM people might say. By contrast, Yudkowsky & Soares (and me) are very very much more pessimistic than that.

Unless I’m reading this wrong somehow, I think you’re excluding people who think something along the lines of “current alignment techniques work great in the current regime but won’t generalize to superintelligence, and the hope instead is to use the best AI that can still be aligned to automate AI alignment”.

Eh, I see that as a separate debate. (I.e., “Suppose Yudkowsky & Soares are right that ASI will definitely be egregiously-misaligned & scheming in the absence of yet-to-be-invented breakthrough technical alignment ideas. Is it plausible that weaker AIs could find those breakthrough technical alignment ideas? Or not?” That’s a live debate, but it’s a different debate than I’m discussing in this post. Lots of people would not grant the premise.)