AI ALIGNMENT FORUM
AF

AI Alignment Forum

Why we should expect ruthless sociopath ASI

LLM in practice these days do include increasingly bigger % of RL wich seems like it should at least make you less certain about capabilities mostly coming from pretraining and papers from before that continuing to be relevant for very long and you do mention it on the other post but still wrote that capabilities come mostly from pretraining on the footnote?. I expect an optimist or someone from the comparatively-less-pessimistic group would argue that LLM or LLM +RL might lead to consequentialists that have human-like goals due to being built from a base of human imitation even as they move towards ASI. And an important disagrement with those people there is that you don't in fact expect LLM +RL to work. You also don't think LLM+ RL would be safe if they did work but still feels relevant because It comes from pretty diferent models of how LLM will improve. Relatedly a question you are not answering here is why you think imitation learning won't lead to discovering things that are far from the distribution of humans since its unclear what the limits are out of distribution and this also seems like you have a strong view on human imitation not generalicing much to the point It won't be even be able to do human level inventing of new things wich seems like would imply bad imitation of humans, as humans can do things like invent writting. You seem to be describing a shallow kind of imitation wich seems like a nontrivial asumtion people in the LLM camp would likely disagree with?. "rocket arbitrarily far past human training data" is mixing the question of "will LLM think in ways that are very diferent from humans" with something like " can LLM only shalowly reuse reasoning from their training data and not invent new things". Perfect human imitators will just rocket arbitrarily far past human training data in the first sense and invent writting etc.

Gordon Seidoh Worley13d33

Distinguish between inference scaling and "larger tasks use more compute"

I think there's a hidden variable in this framework: the effective branching factor of the task. For well-specified tasks (prove this theorem, implement this API to spec, etc.), AI cost probably does scale roughly linearly with task size, matching your model. But most real-world engineering tasks aren't like that. The bottleneck isn't execution speed, it's the iterative discovery of what the task actually is. A senior engineer doing a 4-hour task isn't typing for 4 hours. They're making dozens of micro-decisions informed by tacit knowledge about the codebase, the product, the users, the org. They're constantly pruning the search space. An AI working autonomously on the same task produces tokens faster but explores a much larger space of wrong approaches, because it lacks that contextual judgment. So the cost might scale superlinearly with task complexity not because of compute limits but because of specification limits. This suggests the linear regime on your Pareto frontier might be shorter for real-world task distributions than for benchmark-style tasks, and the "fraction of human cost" metric could look very different depending on how well-specified the task is. The tasks where AI efficiency looks great (tight feedback loops, clear success criteria) are not the same tasks that fill most of an engineer's day. (Though note that obviously this could change if AI starts to have more context and better judgement, which would basically mean training a better model.)

Daniel Kokotajlo16d83

It Is Reasonable To Research How To Use Model Internals In Training

> training process is “the most forbidden technique”, including recent criticism of Goodfire for investing in this area. I think this mischaracterizes the criticism. The criticism as I understand it is that Goodfire is planning to help frontier AI companies use model internals in training, in exchange for money. Insofar as they really are planning to do this, then I'll count myself as among the critics, for the classic "but we need interpretability tools to be our held-out test set for alignment" reason. Do you have a link to the criticism you are responding to?

Recent Discussion

Why we should expect ruthless sociopath ASI

Steven Byrnes

The conversation begins

(Fictional) Optimist: So you expect future artificial superintelligence (ASI) “by default”, i.e. in the absence of yet-to-be-invented techniques, to be a ruthless sociopath, happy to lie, cheat, and steal, whenever doing so is selfishly beneficial, and with callous indifference to whether anyone (including its own programmers and users) lives or dies?

Me: Yup! (Alas.)

Optimist: …Despite all the evidence right in front of our eyes from humans and LLMs.

Me: Yup!

Optimist: OK, well, I’m here to tell you: that is a very specific and strange thing to expect, especially in the absence of any concrete evidence whatsoever. There’s no reason to expect it. If you think that ruthless sociopathy is the “true core nature of intelligence” or whatever, then you should really look at yourself in a mirror and...

(Continue Reading - 2143 more words)

Tom Davidson2h10

Thanks!

Re other ways to do the feat:

Bootstrapping from very smart human intelligence by doing loads of serial thinking, writing great non-sociopathic outputs, and find sft on them
Sophisticated scaffolds - basically gofai but you can insert LLMs in all over the place
Searching over different the above and RL and imitation to things for methods that don't produce sociopaths... Evolution and within lifetime learning don't produce sociopaths despite this not happening!!

Re RL: thanks for those examples. I'd have defined it via smg like# data points or amoun... (read more)

2Steven Byrnes3d

Right, I think LLMs are good at imitating “Joe today”, and good at imitating “Joe + 1 month of learning Introductory Category Theory”, but not good at imitating the delta between those, i.e. the way that Joe grows and changes over that 1 month of learning—or at least, not “good at imitating” in a way that would generalize to imitating a person learning a completely different topic that’s not in the training data. In other words, after watching 1000 people try to learn category theory over the course of a month (while keeping diaries), I claim that an LLM would learn category theory itself, and it would learn all the common misconceptions about category theory that people make as they start learning, but it wouldn’t learn “the general process of learning and sense-making itself” in a way that allows it to then autonomously invent some field that has not been invented yet. I had a long comment-thread argument with Cole Wyeth on this general topic last year: link. We didn’t resolve our disagreement and I eventually bowed out of the conversation, but you might find it helpful anyway. See especially my analogy to trying to imitation-learn AlphaZero improving itself through self-play. My answer is “obviously not”. Here’s an example: Imagine that the “competent adult humans” were all from 100 years before linear algebra existed, and we are hoping that they will invent linear algebra. Now, linear algebra involves a giant pile of interlinked concepts: matrices, bases, rank, nullity, spans, determinants, trace, eigenvectors, dual space, unitarity, and on and on. Now take a parade of these “competent human adults” with no prior exposure to any of this, and give them an hour each before they get fired, but they are allowed to send notes to each other. The goal is for them to collectively invent the entire edifice of linear algebra from scratch. I think it’s doomed. If you take a person who has never seen linear algebra before, then it will just take them a lot of time (much

3RohanS2d

I'd be curious for you to say a bit more in response to this point from above: I'm moderately optimistic about our ability to get roughly human-like consequentialism from LLM-based AGI, with character training instilling a non-ruthless non-sociopath character that is still compatible with lots of consequentialist agency, like very good scientists/inventors/entrepreneurs/etc. who never do anything that could be very dangerous (because a bunch of useful things aren't that risky, and because they have good enough moral principles to deliberately avoid or be careful around things that are risky). I think long-horizon RL or reflection or other components of the continual learning process could break the instilled character, but it seems >50% likely to me that between the preliminary character training and ongoing training and prompting to maintain good character, those things won't dominate, and we'll just have nice AIs. (I get a bit more nervous about this argument for ASI, but I think it may well hold up even there.)

4Steven Byrnes2d

The thing I’m skeptical of is maintaining non-ruthless behavior in the presence of arbitrary amounts of open-ended continual learning. By “open-ended continual learning”, I mean something analogous to what humans did between 30000 BC and today, e.g. inventing new fields, and then still more new fields that build on those new fields, etc. And the AI has to do that without any human input, given enough time. My actual belief is that this kind of open-ended continual learning is simply impossible in LLMs. If I’m wrong about that, then I would next claim that it requires continually updating the LLM weights (not just context window). I think it’s well known that LLMs struggle with large amounts of interrelated complexity in the training data, when none of that complexity, or anything remotely related to it, was in the pretraining data. “Open-ended continual learning purely through the context window” would be a very extreme version of that—centuries of knowledge, entire new fields and ways of thinking, entirely absent from pretraining and exclusively in the context window. No way is that going to work. OK, so far I’ve argued that if this kind of continual learning is possible at all, it would require continual weight updates to lock in the new knowledge and ideas that the LLM generates—and not just one-time small updates, but more and more updates as the process continues, asymptoting to 100% of the training data. If you buy all that, how do you think these weight updates will work? Where do you think the “training data” for those updates will come from? Or if you don’t buy that, how do you think the continual learning will work? My experience is that lots of LLM-focused people say “open-ended continual learning will be solved somehow, I guess”, and not think too hard about exactly how it gets solved. And then that’s how the pea gets hidden under the thimble. Because actually, I claim, continual learning needs some kind of ground truth or else it will go off the ra

How likely is deceptive alignment?

evhub

The following is an edited transcript of a talk I gave. I have given this talk at multiple places, including first at Anthropic and then for ELK winners and at Redwood Research, though the version that this document is based on is the version I gave to SERI MATS fellows. Thanks to Jonathan Ng, Ryan Kidd, and others for help transcribing that talk. Substantial edits were done on top of the transcription by me. Though all slides are embedded below, the full slide deck is also available here.

Today I’m going to be talking about deceptive alignment. Deceptive alignment is something I'm very concerned about and is where I think most of the existential risk from AI comes from. And I'm going to try to make the case...

(Continue Reading - 17755 more words)

Chris_Leong10h10

I reread the distillation of this (not the full article sadly due to time limitations), so my understanding of Evan's views might be off. Here's my retrospective:

High path dependence:
• Evan argues that before it obtains the true training objective, gradient descent likely obtains a proxy goal. This seems intuitive, though there's debate as to whether we should interpret recent results as indicating alignment seems to be happening by default or whether it remains a hard, unsolved problem.
• Evan argues that the model will gain an understanding of its own sit... (read more)

Glass box learners want to be black box

Cole Wyeth

10mo

Epistemic status: intuitive speculation with scattered mathematical justification.

My goal here is to interrogate the dream of writing a beautiful algorithm for intelligence and thereby ensuring safety. For example:

I don't know precisely what alternative he had in mind, but I only seem to remember reading clean functional programs from MIRI, so that is one possibility. Whether or not anyone else endorses it, that is the prototypical picture I have in mind as the holy grail of glass box AGI implementation. In my mind, it has careful comments explaining the precisely chosen, recursive prior and decision rules that make it go "foom."

Is the "inscrutable" complexity of deep neural networks unavoidable? There is has been some prior discussion of the desire to avoid it as map-territory confusion, and I am...

(Continue Reading - 1126 more words)

2Cole Wyeth16h

A relevant passage from @Kaarel, https://www.lesswrong.com/posts/nkeYxjdrWBJvwbnTr/an-advent-of-thought

1TsviBT16h

It's an understandable mistake, but that post is by @Kaarel , not me!

1Cole Wyeth16h

Fixed. (For posterity, TsviBT linked the post here, and mostly seems to have referred to his own posts in the rest of that conversation. I quoted this section before "finishing" Kaarel's post and realizing it was not by TsviBT. Sorry!)

TsviBT16h10

Ah I see (I'll try a bit to remember to say "by so and so").

6 reasons why “alignment-is-hard” discourse seems alien to human intuitions, and vice-versa

106

Steven Byrnes

3mo

Tl;dr

AI alignment has a culture clash. On one side, the “technical-alignment-is-hard” / “rational agents” school-of-thought argues that we should expect future powerful AIs to be power-seeking ruthless consequentialists. On the other side, people observe that both humans and LLMs are obviously capable of behaving like, well, not that. The latter group accuses the former of head-in-the-clouds abstract theorizing gone off the rails, while the former accuses the latter of mindlessly assuming that the future will always be the same as the present, rather than trying to understand things. “Alas, the power-seeking ruthless consequentialist AIs are still coming,” sigh the former. “Just you wait.”

As it happens, I’m basically in that “alas, just you wait” camp, expecting ruthless future AIs. But my camp faces a real question: what exactly is it...

(Continue Reading - 4788 more words)

3topherhunt2d

Thanks for the thoughtful reply. It took me a lot of squinting, but IIUC you're saying: * Different kinds of minds, produced by different kinds of architectures, should likely exhibit very different levels of scary traits such as monomaniacal sociopathy. * Stop focusing on LLMs so much; they're not the main threat. Yes they seem to exhibit more value-roundedness because they're trained to imitate humans, but they aren't likely to reach AGI anytime soon. * Focus more on RL agents and "brain-like" architectures; those are built very differently and plausibly would have much more sociopathic tendencies. * So, our alarm at the plausible risk of unleashing AGI-level ruthless monomaniacal RL agents is justified. I don't disagree with any of these, and your reply has helped me see the big implications of the LLM / RL distinction much more clearly. And I'm sorry for the vehemence of my earlier comment — part of that came from me not "getting" that my experience with LLMs may not apply to other AI architectures. I hadn't previously gotten that into my head, so thank you. But I want to push on the deeper framing I'm hearing; something in me reacts strongly (negatively) to your starting assumptions, in a way that doesn't go away when you walk me through the object-level reasoning like above. Your article treats general intelligence as "pure ruthless optimizer" by default, and any whiff of empathy or sociality or ruth is just a bolt-on module — a happy accident of our particular evolutionary pressures - and then you try to diagnose why human minds deviate from that sociopathic, amoral natural state. My objection, less passionately stated, is that this seems to privilege a theoretical framework as the baseline and then treat our only empirical data on AGI-level intelligence as the anomaly to be explained away. That feels deeply backwards to me. We have N=1 examples of human-level general intelligence. All 1 of them are deeply prosocial. The "agent foundations" framework

Steven Byrnes1d20

I get that, and I appreciate the comments!

(Side note: in case you missed it, I expanded my last comment into a post “Why we should expect ruthless sociopath ASI”. Thanks for the inspiration, and hope you don’t mind.)

The very capabilities that make the nightmare scary (deep, nuanced understanding of how humans think and feel and operate) are exactly the kind of rich social cognition that, in our only empirical examples, comes tangled up with perspective-taking, moral awareness, and empathy.

Here are a couple reasons to think it’s not too “tangled up” in hum... (read more)

Formalizing Newcombian Problems with Fuzzy Infra-Bayesianism

Brittany Gelb

3mo

Introduction

In this post, we introduce contributions and supracontributions^[1], which are basic objects from infra-Bayesianism that go beyond the crisp case (the case of credal sets). We then define supra-POMDPs, a generalization of partially observable Markov decision processes (POMDPs). This generalization has state transition dynamics that are described by supracontributions.

We use supra-POMDPs to formalize various Newcombian problems in the context of learning theory where an agent repeatedly encounters the problem. The one-shot version of these problems are well-known to highlight flaws with classical decision theories.^[2] In particular, we discuss the opaque, transparent, and epsilon-noisy versions of Newcomb's problem, XOR blackmail, and counterfactual mugging.

We conclude by stating a theorem that describes when optimality for the supra-POMDP relates to optimality for the Newcombian problem. This theorem is significant because it gives...

(Continue Reading - 6353 more words)

1Cole Wyeth4d

Contributions are in a natural one-to-one correspondence with weighted probability measures, which were studied by Joe Halpern: https://arxiv.org/pdf/1302.5681 Simply take the weight to be the sum of measure. Halpern and Leung advocated minimizing regret against a set of weighted probability measures, which seems to be the same as minimizing loss against a supracontribution? They update weights by likelihood ratio, roughly speaking (I am not sure how this corresponds to IB, since updating is not discussed in this sequence). @Vanessa Kosoy @Diffractor Do you know if there is a meaningful connection between Halpern's work and IBDT? Unrelatedly: I like the term semi-environment (the AIXI literature uses "chronological semimeasure" with apparently the same meaning, which is much worse - previously even "chronological contextual semimeasure"!). I'd like to start using your term, but maybe it would lead to some confusion as applied to my current research - I've been treating semimeasures as (lower probabilities for) credal sets in with a worst-case decision rule, while your contributions are actually best-case interpretations of (semi)measure loss!

Vanessa Kosoy3d20

Halpern and Leung propose the "minimax weighted expected regret" (MWER) decision-rule, which is a generalization of the minimax-expected-regret (MER) decision-rule. In contrast, our decision rule is a weighted generalization of maximin-expected-utility (MMEU). The problem with MER is that it doesn't work very well with learning. The closest thing to doing learning with MER is adversarial bandits. However, adversarial regret is statistically intractable for Markov Decision Processes. And even with bandits there is a hidden obliviousness assumption if you try to interpret it in a principled decision-theoretic way.

AI ALIGNMENT FORUM
AF

AI ALIGNMENT FORUM
AF

AI Alignment Posts

Popular Comments

AI Alignment Posts

Popular Comments

Recent Discussion

The conversation begins

Tl;dr

Introduction