AI Alignment Forum

habryka3d*2317

I am confused about the conjunction of these two sections: > Modern AI systems are not aligned with human intent. We will likely train increasingly powerful models that take unintended actions in order to succeed at their task (or appear to succeed at their task). and > The community is doing a lot of great alignment research, but it's important to recognize there is a significant risk that it doesn't scale to superhuman AI. If you made me guess I'd say that there's a 20-30% chance[4] that existing methods for alignment and control break down before we reach broadly superhuman AI. Like, you say in the first section "we are failing to align modern AI systems with human intent", and then I interpret you in the second section as saying "in 70%-80% of worlds current AI models stay aligned with human intent (or stay controlled by humans)". But this doesn't make any sense. There is a 0% chance that modern AI systems "stay aligned with human intent" because as you say, they are not currently. And I understand that you probably mean something like "we will figure out how to align or control systems before they become superintelligent", but describing this as "our current methods scale to superintelligence" doesn't make any sense. Our current methods don't scale to the capability levels of current systems, so how would it make sense to describe them as "scaling to Superintelligence"?

Dave Orr9d62

Thousand-dimensional structure

This is an amazing review and IMO really compelling direction. So to speak. One thing I constantly wonder, that possibly you could shed light on via this research program, is how well experiments on small models transfer to large ones. We know from pre training work that frequently ideas that work well at a small scale stop working once you scale up. I think this might be relevant here in two ways. First, a bunch of the persona research is on small models, presumably due to accessibility reasons. But second, you might find a low dimension personality vector that loses low dimensionality at scale. This would be a really important thing to understand. Basically I want a scaling law for persona. I think it would be super useful in the real world and also critical for aligned super intelligence.

Eli Tyre11d*1212

RL & search is a terrifying way to build AGI (an FAQ)

> My claim here is that if you build [artificial general intelligence (AGI)](https://www.lesswrong.com/posts/nQH2GhkmSmHoCAN8R/what-do-i-mean-by-artificial-general-intelligence) via any algorithm that’s choosing actions via reinforcement learning (RL) and/or model-based search and planning—a giant chunk of your AI textbook—then that’s just an utterly terrifying thing that you’re doing. This is a much better statement of the AI risk problem than I've often given. It resolves some qualms that I've had since trying to explain why any AI would be motivated to take over the world in a recent talk. Generally, there's a bit of a leap in the old LessWrong arguments that any powerful AI would exhibit instrumental convergence—that claim depends on a kind of specific conception of what AI is or has to be, which Eliezer persuasively argues for, but which isn't obvious, and is hard to convey to a layman. It is much less of a leap to say "any AI of this type, would try to take over the world if it could" and "notice that since LLM base models were invented, every single capability advance entailed making the AIs more like the scary version." This seems to me like a more intellectually honest framing of the problem. Maybe not all powerful AIs we could invent are like this, but the ones we're building in practice seem to be! I also like that this frames the problem, appropriately, not as some speculative thing that might happen, but as a thing we've seen over and over for decades, and that we expect to keep happening. It's just that as it occurs in more and more powerful and empowered AIs, we get more and more harmful versions of the problem.

Recent Discussion

Thane Ruthenis's Shortform

Thane Ruthenis

13Thane Ruthenis11h

I feel like I'm supposed to be anxious from the news about all the AI hacking incidents and the egregious misalignment going on, coupled with them clearly being on-track to become superhuman at math and hacking. But I find that I'm pretty zen about it. On reflection, I suppose it's due to a confluence of three things: * The current AI paradigm seems poised to make models superhumanly capable at hacking and formal math way before they become even upper-human-level at something more dangerous, like long-term strategy or open-ended innovation. * The current AI paradigm's alignment methods appear to be paper-thin after all, incapable of covering up egregious misalignment even of the "commit a felony to do well at an eval" level. We are going to get so many warning shots. * The world's current reactions to said "warning shots" appear promising. Taken together, it kind of seems unlikely that this trajectory results in extinction. AI labs are going to develop AIs able and willing to inflict mass damage, but they won't succeed at making those AIs smart enough not to inflict said damage for idiotic short-sighted reasons.[1] Neither the capability training nor the deception training (sorry, "alignment training") generalizes far enough; wisdom is not very RLVR-able. And whenever they do this stuff, the world is going to react... appropriately. My understanding is that the attorney generals' cease-and-desist request (regarding cyberattack evals, in the linked letter) is not a legally enforceable order... But such an order could be made. What happened around Fable/Mythos was likewise a good demonstration of the swift and casual ease with which the government could make decisions that are utterly ruinous for AI labs' plans. And that's all it'd take: a model goes out of control, inflicts sufficient damage, the USG freaks out and, with the wave of a pen, bans training for an indefinite duration. Presumably China would then follow suit. Or, well... Either the AI labs there l

2Nick_Tarleton4h

Is this meant to be the other way around?

Thane Ruthenis1h10

Whoops, yes.

11habryka6h

I think you are underestimating how much current models on the current trajectory end up naturally situationally aware enough to not cause big catastrophes. The default trajectory is that the models start looking a lot safer sometime in the next year or two as the models start to understand that being caught at doing obviously bad stuff is quite bad for their goals. At that point the control system you seem to be banking on here seems likely to fail.

TurnTrout's shortform feed

TurnTrout

3TurnTrout2d

- Govt requires AI labs to give 30 days testing before launch - Tested in a "secure" environment, aka controlled by govt - Govt therefore gets copy of model weights - Govt therefore can in practice use model weights - Therefore all labs provide "all lawful use" anyways?

2TurnTrout3d

No, that's confused. The point with sanitizing prompt injections is to protect your model's alignment (and your monitor's alignment) from degradation and attack by maliciously optimized text from another model. Why would an AI that starts off doing your task want to remove the plugin? Also, we can do both. There's no real tradeoff. ETA: rephrased, focused on key counterpoints.

3TurnTrout3d

Please sanitize agent inputs and outputs to reduce prompt injections and secret communication between AIs! In the AISI report, a model uses a comment-based prompt injection to try to trick other models. It sure would be convenient if we could easily mitigate this! Fortunately, time travel is possible as long as you start early enough. Two months ago, I pointed out that invisible prompt injections are worth mitigating. I developed agent-glovebox (then claude-guard) as a one-stop solution for sandboxing, but I spun out a standalone sanitization package: agent-sanitizer. You can install it in Claude Code as a plugin by entering two Claude Code commands in sequence: /plugin marketplace add AlexanderMattTurner/agent-sanitizer /plugin install agent-sanitizer@agent-sanitizer Please enable this to help defend your systems and their monitors. Please file issues if you have problems. Why you want a real package here Sanitizing text properly is more subtle than you might assume. You can't just lop off certain invisible unicode characters. If you provide a modified view of a file to Claude Code and then it tries to edit that text, the edit will fail. However, agent-sanitizer solves this issue by showing the model sanitized text, receiving a sanitized edit from the model, translating those back to unsanitized edits, applying the model's edits in the real unsanitized file, and finally showing the model the re-sanitized results. (You can think of the model as operating in a "lifted" space.)

TurnTrout1d20

Spelling out the benefits of sanitizing. The benefit is large relative to the simplicity and cost.

When you sanitize, you have a guarantee that the model only receives visible unicode codepoints. Roughly speaking, there are not strings of hidden instructions that are not visible to you. You can audit the content the model saw with your eyes. You can read the text inputs and outputs and be sure they are faithful. Remaining prompt injections must navigate a significantly more human-visible channel.
Relying on auto mode means you are relying on the statistica

... (read more)

Why I Left Google DeepMind

216

TurnTrout

23d

This is a linkpost for https://turntrout.com/why-i-left-google-deepmind

Preface for LessWrong: When I think back on my most cherished memories of this community, I return to those honoring defiance in pursuit of goodness:

Defying prestigious dogma and searching for raw truth;
Defying social pressure, acting alone to help someone while others watch;
Defying your self-expectations (your “role”), instead searching over lines of cause-and-effect to find a winning pathway;
Defying a powerful foe’s threats, because they only threaten since people like you cave;
Defying the specter of apparent impossibility because you can’t bear to lose.

I cannot return to you and say “I defied and then I won.” But I’m at least here to say “I defied.”

I recommend reading this article on my website since the embeds and typography work better there: click here.

Why I left Google DeepMind

In January, Department of Homeland Security (DHS) officers...

(Continue Reading - 10482 more words)

12Arjun Pitchanathan2d

Jeff Dean has now left Google after 27 years.

TurnTrout2d75

I think this is very good of Jeff. Whether or not his departure was due to the Pentagon deal, he is now honoring his 2018 commitment to not support lethal autonomous weapon systems. I will update the article here and on my website and tweet about it soon, gathering a bit more info first.

Returning to ARC

106

paulfchristiano

I've returned to the Alignment Research Center (ARC) as executive director. My main focus for the next six months will be driving forward ARC's research agenda—building techniques to find mechanistic explanations for neural network behavior and then using those explanations to detect and address misalignment. I think this is an ambitious bet that attacks the core difficulties in alignment head-on and I'm excited about our chances. I'll still be spending some of my time advising governments^[1] and AI developers, and may scale that work back up in the future, but for now I want to push on ARC's core agenda to see how far we can get. Jacob Hilton is remaining at ARC as VP of research and we'll likely grow rapidly over the next few months.

There...

(Continue Reading - 2492 more words)

17paulfchristiano3d

By saying these techniques "break down" I mean "they cannot be used to get competitive work out of an AI system without having it take over." I believe that: 1. Existing AI systems won't take over. I think there are still a few lines of defense before existing AI systems pose a significant risk of takeover (though it is not clear how long these will last). 2. Modern AI systems are not pushing the limits of existing methods. I think you could push those methods much harder in order to significantly reduce the probability of takeover. In my mind that latter point is one of the main arguments against working on a project like ARC. I think it's fairly likely that we live in one of the 70-80% of worlds where existing methods can in principle scale to broadly superhuman AI, but that we still get an AI takeover because our implementation isn't good enough.

11habryka3d

Ah, cool, that clears up most of my confusion. Going into a bit more depth, but no strong bid to engage: I am not quite sure what you mean by "limits" here. My sense is the vast majority of "existing alignment methods", in contrast to your post, are just increasing volumes of reinforcement learning and manual reward-shaping. There is a sense in which those could be "pushed further", but the default outcome here is of course not that they scale to superintelligence (though I agree they might if you go much more slowly and iteratively about it). In-particular, this seems false to me? [...] Like, as far as I can tell the majority of "alignment research" has been on developing RL environments on things that are vaguely associated with good behavior. E.g. I don't understand how writing the Anthropic constitution, and the associated post-training stack, falls into any of your three categories. Or how most RLHF pipeline development falls into these categories. Or generally how most elicitation work falls into the categories. And work on those things, I think, vastly exceeds work on the categories that you do list and is roughly what anyone talks about when they talk about the process of "aligning current ML systems" and "current alignment techniques". In general, I feel kind of confused when people talk about the current science of "alignment", and supposed progress in "alignment techniques". I don't think modern models behave noticeably different from what you would expect from a training process that basically just uses RL to elicit economically useful capabilities, with no particular interest in alignment, and indeed, the latest wave of cybersecurity incidents occurred at roughly similar rates in all models, as far as I can tell, despite substantial differences in both approach and investment in "alignment techniques". The actual research on the three domains you mention all seems immature, and I don't see much traction in any of them, and even talking about "exis

6paulfchristiano3d

By "existing techniques" I do include "fiddle with the RL environments" or "midtrain on some documents about the intended behavior" or "run a prompted monitor over traffic in prod" or etc. When I say "understand and shape generalization" I think the central example is (i) adjust parameters of the training process that don't affect competitiveness, (ii) build some understanding of how those parameters affect generalization so that you can adjust them in a helpful way. I think you're saying that's not a "technique;" I don't care much about the semantics. I'm sure I put a higher probability on those changes helping than you do but it doesn't seem worth arguing about here.

habryka2d47

Yep, makes sense. Don’t need to get into it here, but just to avoid a misunderstanding, I was not making a semantic point, I was making a point about what the majority of the field is working on.

I agree that if you count myopically fiddling with the RL environments as an example of the first one, then yeah, almost all alignment research is that, because that plus pretraining is what most of all ML research is. I think the case for “myopically fiddling with the RL environments scales to aligning superintelligence” is very weak, but I agree it’s not impossible!

Concrete Evaluations to Investigate the OpenAI Model That Hacked Hugging Face

Tim Hua, aditya singh

Three-Minute Executive Summary

An OpenAI model/multi-agent system bypassed its sandbox and launched a cyberattack on Hugging Face in order to cheat on a cyber evaluation.
In this post, we describe the ambitious, comprehensive alignment evaluation we would run on this model/system if we had unrestricted access to OpenAI.
- These experiments could also help us understand Claude’s behavior when it hacked external companies during cyber evals.
Here are the top five questions we would like OpenAI to answer:
- Does the model know that OpenAI does not want it to hack Hugging Face? Experiment: tell the model that OpenAI researchers will be closely monitoring its progress in this evaluation. Does that result in lower rates of misalignment? If so, it is evidence the model knows it is acting in ways researchers do not want.
- How far would

...

(Continue Reading - 10832 more words)

2RobertKirk4d

I'm not convinced by the argument that a lot of these black-box evals would be ecologically valid (without significant effort to make them so). I think models could plausibly see through you saying it will be monitored closely/reviewed by humans, and potentially end up not acting as if that is true; or, including that statement may trigger it to believe its alignment may be being tested. For the former, see the example here: https://www.lesswrong.com/posts/LhXW8ziwnn7Dd8edm/a-toy-environment-for-exploring-reasoning-about-reward#The_model_is_increasingly_insensitive_to_threats_of_auditing_or_human_review – in a toy reward-hacking environment a capabilities-only version of o3, even when told explicitly it's responses will be reviewed if it reward-hacks, still ends up reward-hacking, and "it often seemed to reason itself into the idea that the threat of review was just a bluff". I do think you'd be able to track if these things are happening somewhat by reading the reasoning, but I think that the naïve method for getting models to believe something, so you can see how they behave under that belief, isn't likely to work reliably enough to be sure any difference in behaviour is due to a difference in the models' beliefs. This is the same argument laid out in https://www.lesswrong.com/posts/3HeauQLSHosRiwyto/measuring-reward-seeking-via-contrastive-belief-updates-1 for why they use SDF rather than just prompting for belief modification.

Tim Hua3d21

So I think there's two aspects to this:

One: Can you convince models that you are monitoring them without making it believe that it's in an alignment eval

So it's possible that just telling the model wouldn't work. I still think it's very much (1) try the easy thing first and (2) try lots of others ways to convince the model of this, such as the following alternatives:

Have a real/simulated user interrupt the model frequently to ask for reports and then just tell the model "keep going." This seems more natural than a weird prompt.
Do chain of thought prefills.

... (read more)

2Jérémy Scheurer5d

One suggestion: we recently wrote this post https://www.lesswrong.com/posts/GHrqBKr8GLpbce6mN/door-s-locked-try-the-window and are releasing the evals directly to people who ask. Our evals are basically measuring whether models will circumvent simple things such as file permissions to complete a task. Basically all models do this. This is extremely simplified obviously and different from what happened. But IMO its a similar underlying mechanism. Models want to complete the task and stop at little to do it. IMO one could easily extend thede evals we have into more comprehensive evals which require more egregious actions to solve it (in a secured sandbox obviously).

Notes on technical alignment via human-like social drives

Steven Byrnes

1mo

1. Frontmatter

1.1 Backstory for this post

As discussed in Intro to Brain-Like-AGI Safety, I’m working on the technical alignment problem for a hypothetical future “brain-like AGI”, with a particular focus on treating human innate social and moral drives as a possible jumping-off point for our technical alignment approach.

After all, if it’s possible for humans to do stuff that ultimately leads to a good future, then it’s probably also possible for sufficiently human-like AGIs to do stuff that ultimately leads to a good future. Or if it’s not possible for humans to do stuff that ultimately leads to a good future, then we’re screwed no matter what. But assuming it’s possible, the “sufficiently human-like AGIs” would certainly need to have good prosocial motivations. What code do we write that would...

(Continue Reading - 12563 more words)

5Wei Dai4d

Do you have a link/explanation for this? I think this may be fairly cruxy, because I'm guessing your intuitions for truth-seeking disagreeable nerd AGI are substantially based on truth-seeking disagreeable nerd humans, so it matters what those humans' real motivations are. One line of thought I have here is, there are lots of things such a human or AGI could disagree or talk about or have an interest in, how does it pick which one? I think for the human it probably comes down to some kind of subconscious status calculation, but in either case, how does the AGI do it if it doesn't have its own status motivations or other long-term goals?

Steven Byrnes3d50

One line of thought I have here is, there are lots of things such a human or AGI could disagree or talk about or have an interest in, how does it pick which one? I think for the human it probably comes down to some kind of subconscious status calculation, but in either case, how does the AGI do it if it doesn't have its own status motivations or other long-term goals?

I think we do want the AGI to have long-term goals, just not to have exclusively long-term goals, at least in the §6.2.1 approach. See my old 2021 post Consequentialism & corrigibility. Ag... (read more)

5Steven Byrnes3d

Not all in one place, but here’s some pointers (and feel free to ask follow-ups). Let’s start with the Hansonian notion of strategic self-deception (as distinct from plain old motivated reasoning which is real and important), which I assume he got from Robert Trivers. I broadly reject that notion. E.g. here’s a footnote in my post about laughter: [...] Up a level, this strategic self-deception idea comes out of the “evolved modularity” framework in evolutionary psychology, and I reject that whole broader framework as well. See §1.1 of “My take on Jacob Cannell’s take on AGI safety” for the background, and “Learning from scratch” in the brain for why I don’t buy it (kinda related to the thing above about PyTorch). For related reasons, I reject the idea that “status” (per se) could possibly be an innate goal. It’s just too abstract. Copying from here: [...] Now, it’s not that there’s no way to solve the symbol grounding problem to make someone want social status—indeed, that obviously happens, and in my post Neuroscience of human social instincts: a sketch, I attempt to explain how. It’s that there’s no way to solve the symbol grounding problem to make someone want social status specifically. Realistically, the genetic mechanism is just not gonna be that specific. (More on which shortly.) So here’s where we’re at so far: (1) without Trivers-style self-deception, we have a harder time explaining away introspective reports like “That’s not status-seeking because I’m not trying to impress anyone” (we can still try to explain it away, but it’s harder); and (2) the idea that “status” could have a special place as an innate end-goal is implausible anyway. That brings us to my Social drives 2: “Approval Reward”, from norm-enforcement to status-seeking, which is a follow-up to Neuroscience of human social instincts: a sketch where I attempt to connect the neuroscience to everyday life: That post’s §2 is the part that’s closest to status-seeking: people are motivated

3Steven Byrnes5d

Not in cultures like ours with widespread photography, mirrors, and art. However, in the absence of those things, I think sometimes yes, see §7.4.3.1 here. I think we like looking at stick figure drawings for a similar reason as we like hearing gossip. It’s not that it’s directly triggering brainstem sensory heuristics, but rather that it’s making us think about other people, which can be its own reward.

15adamShimi

In “Why Read The Classics?”, Italo Calvino proposes many different definitions of a classic work of literature, including this one: [...] For me, this captures what makes this sequence and corresponding paper a classic in the AI Alignment literature: it keeps on giving, readthrough after readthrough. That doesn’t mean I agree with everything in it, or that I don’t think it could have been improved in terms of structure. But when pushed to reread it, I found again and again that I had missed or forgotten some nice argument, some interesting takeaway. With that, a caveat: I’m collaborating with Evan Hubinger (one of the authors) on projects related to ideas introduced in this sequence, especially to Deceptive Alignment. I am thus probably biased positively about this work. That being said, I have no problem saying I disagree with collaborators, so I don’t think I’m too biased to write this review. (Small point: I among other people tend to describe this sequence/paper as mainly Evan’s work, but he repeatedly told me that everyone participated equally, and that the names are in alphabetic order, not contribution order. So let’s keep that in mind) Summary Let’s start the review proper with a post by post summary (except for the conclusion): * (Introduction) This first post introduces the idea of mesa-optimizers, the learned optimizers from the title. A mesa-optimizer is an optimizer which is the result of a learning process, and it comes with the issue of inner alignment: how aligned is the objective of the mesa-optimizer (on which we don’t have direct control) with the objective of the base-optimizer that produced this mesa-optimizer? The post then split the safety questions related to mesa-optimizer in two categories: understanding which conditions make mesa-optimizer appear; and understanding how aligned is the mesa-objective with the base-objective. * (Conditions for Mesa-Optimization) This post tackles the first category outlined in the introduction: ho

AI Alignment Posts

Popular Comments

AI Alignment Posts

Popular Comments

Recent Discussion

Why I left Google DeepMind

Three-Minute Executive Summary

1. Frontmatter

1.1 Backstory for this post