AI Alignment Forum

habryka17h*104

I am confused about the conjunction of these two sections: > Modern AI systems are not aligned with human intent. We will likely train increasingly powerful models that take unintended actions in order to succeed at their task (or appear to succeed at their task). and > The community is doing a lot of great alignment research, but it's important to recognize there is a significant risk that it doesn't scale to superhuman AI. If you made me guess I'd say that there's a 20-30% chance[4] that existing methods for alignment and control break down before we reach broadly superhuman AI. Like, you say in the first section "we are failing to align modern AI systems with human intent", and then I interpret you in the second section as saying "in 70%-80% of worlds current AI models stay aligned with human intent (or stay controlled by humans)". But this doesn't make any sense. There is a 0% chance that modern AI systems "stay aligned with human intent" because as you say, they are not currently. And I understand that you probably mean something like "we will figure out how to align or control systems before they become superintelligent", but describing this as "our current methods scale to superintelligence" doesn't make any sense. Our current methods don't scale to the capability levels of current systems, so how would it make sense to describe them as "scaling to Superintelligence"?

Dave Orr6d41

Thousand-dimensional structure

This is an amazing review and IMO really compelling direction. So to speak. One thing I constantly wonder, that possibly you could shed light on via this research program, is how well experiments on small models transfer to large ones. We know from pre training work that frequently ideas that work well at a small scale stop working once you scale up. I think this might be relevant here in two ways. First, a bunch of the persona research is on small models, presumably due to accessibility reasons. But second, you might find a low dimension personality vector that loses low dimensionality at scale. This would be a really important thing to understand. Basically I want a scaling law for persona. I think it would be super useful in the real world and also critical for aligned super intelligence.

Eli Tyre9d*1212

RL & search is a terrifying way to build AGI (an FAQ)

> My claim here is that if you build [artificial general intelligence (AGI)](https://www.lesswrong.com/posts/nQH2GhkmSmHoCAN8R/what-do-i-mean-by-artificial-general-intelligence) via any algorithm that’s choosing actions via reinforcement learning (RL) and/or model-based search and planning—a giant chunk of your AI textbook—then that’s just an utterly terrifying thing that you’re doing. This is a much better statement of the AI risk problem than I've often given. It resolves some qualms that I've had since trying to explain why any AI would be motivated to take over the world in a recent talk. Generally, there's a bit of a leap in the old LessWrong arguments that any powerful AI would exhibit instrumental convergence—that claim depends on a kind of specific conception of what AI is or has to be, which Eliezer persuasively argues for, but which isn't obvious, and is hard to convey to a layman. It is much less of a leap to say "any AI of this type, would try to take over the world if it could" and "notice that since LLM base models were invented, every single capability advance entailed making the AIs more like the scary version." This seems to me like a more intellectually honest framing of the problem. Maybe not all powerful AIs we could invent are like this, but the ones we're building in practice seem to be! I also like that this frames the problem, appropriately, not as some speculative thing that might happen, but as a thing we've seen over and over for decades, and that we expect to keep happening. It's just that as it occurs in more and more powerful and empowered AIs, we get more and more harmful versions of the problem.

31johnswentworth

This review is mostly going to talk about what I think the post does wrong and how to fix it, because the post itself does a good job explaining what it does right. But before we get to that, it's worth saying up-front what the post does well: the post proposes a basically-correct notion of "power" for purposes of instrumental convergence, and then uses it to prove that instrumental convergence is in fact highly probable under a wide range of conditions. On that basis alone, it is an excellent post. I see two (related) central problems, from which various other symptoms follow: 1. POWER offers a black-box notion of instrumental convergence. This is the right starting point, but it needs to be complemented with a gears-level understanding of what features of the environment give rise to convergence. 2. Unstructured MDPs are a bad model in which to formulate instrumental convergence. In particular, they are bad for building a gears-level understanding of what features of the environment give rise to convergence. Some things I've thought a lot about over the past year seem particularly well-suited to address these problems, so I have a fair bit to say about them. Why Unstructured MDPs Are A Bad Model For Instrumental Convergence The basic problem with unstructured MDPs is that the entire world-state is a single, monolithic object. Some symptoms of this problem: * it's hard to talk about "resources", which seem fairly central to instrumental convergence * it's hard to talk about multiple agents competing for the same resources * it's hard to talk about which parts of the world an agent controls/doesn't control * it's hard to talk about which parts of the world agents do/don't care about * ... indeed, it's hard to talk about the world having "parts" at all * it's hard to talk about agents not competing, since there's only one monolithic world-state to control * any action which changes the world at all changes the entire world-state; there's no built-in w

6TurnTrout

One year later, I remain excited about this post, from its ideas, to its formalisms, to its implications. I think it helps us formally understand part of the difficulty of the alignment problem. This formalization of power and the Attainable Utility Landscape have together given me a novel frame for understanding alignment and corrigibility. Since last December, I’ve spent several hundred hours expanding the formal results and rewriting the paper; I’ve generalized the theorems, added rigor, and taken great pains to spell out what the theorems do and do not imply. For example, the main paper is 9 pages long; in Appendix B, I further dedicated 3.5 pages to exploring the nuances of the formal definition of ‘power-seeking’ (Definition 6.1). However, there are a few things I wish I’d gotten right the first time around. Therefore, I’ve restructured and rewritten much of the post. Let’s walk through some of the changes. ‘Instrumentally convergent’ replaced by ‘robustly instrumental’ Like many good things, this terminological shift was prompted by a critique from Andrew Critch. Roughly speaking, this work considered an action to be ‘instrumentally convergent’ if it’s very probably optimal, with respect to a probability distribution on a set of reward functions. For the formal definition, see Definition 5.8 in the paper. This definition is natural. You can even find it echoed by Tony Zador in the Debate on Instrumental Convergence: [...] (Zador uses “set of scenarios” instead of “set of reward functions”, but he is implicitly reasoning: “with respect to my beliefs about what kind of objective functions we will implement and what the agent will confront in deployment, I predict that deadly actions have a negligible probability of being optimal.”) While discussing this definition of ‘instrumental convergence’, Andrew asked me: “what, exactly, is doing the converging? There is no limiting process. Optimal policies just are.” It would be more appropriate to say tha

Recent Discussion

Why I Left Google DeepMind

215

TurnTrout

21d

This is a linkpost for https://turntrout.com/why-i-left-google-deepmind

Preface for LessWrong: When I think back on my most cherished memories of this community, I return to those honoring defiance in pursuit of goodness:

Defying prestigious dogma and searching for raw truth;
Defying social pressure, acting alone to help someone while others watch;
Defying your self-expectations (your “role”), instead searching over lines of cause-and-effect to find a winning pathway;
Defying a powerful foe’s threats, because they only threaten since people like you cave;
Defying the specter of apparent impossibility because you can’t bear to lose.

I cannot return to you and say “I defied and then I won.” But I’m at least here to say “I defied.”

I recommend reading this article on my website since the embeds and typography work better there: click here.

Why I left Google DeepMind

In January, Department of Homeland Security (DHS) officers...

(Continue Reading - 10482 more words)

11Arjun Pitchanathan3h

Jeff Dean has now left Google after 27 years.

TurnTrout2h20

I think this is very good of Jeff. Whether or not his departure was due to the Pentagon deal, he is now honoring his 2018 commitment to not support lethal autonomous weapon systems. I will update the article here and on my website and tweet about it soon, gathering a bit more info first.

4TurnTrout7d

Realized I made a prediction on Jan. 1 about whether I'd leave GDM in 2026: My Fatebook question (shared with permission from Peter).

Returning to ARC

paulfchristiano

22h

I've returned to the Alignment Research Center (ARC) as executive director. My main focus for the next six months will be driving forward ARC's research agenda—building techniques to find mechanistic explanations for neural network behavior and then using those explanations to detect and address misalignment. I think this is an ambitious bet that attacks the core difficulties in alignment head-on and I'm excited about our chances. I'll still be spending some of my time advising governments^[1] and AI developers, and may scale that work back up in the future, but for now I want to push on ARC's core agenda to see how far we can get. Jacob Hilton is remaining at ARC as VP of research and we'll likely grow rapidly over the next few months.

There...

(Continue Reading - 2492 more words)

14paulfchristiano16h

By saying these techniques "break down" I mean "they cannot be used to get competitive work out of an AI system without having it take over." I believe that: 1. Existing AI systems won't take over. I think there are still a few lines of defense before existing AI systems pose a significant risk of takeover (though it is not clear how long these will last). 2. Modern AI systems are not pushing the limits of existing methods. I think you could push those methods much harder in order to significantly reduce the probability of takeover. In my mind that latter point is one of the main arguments against working on a project like ARC. I think it's fairly likely that we live in one of the 70-80% of worlds where existing methods can in principle scale to broadly superhuman AI, but that we still get an AI takeover because our implementation isn't good enough.

4habryka12h

Ah, cool, that clears up most of my confusion. Going into a bit more depth, but no strong bid to engage: I am not quite sure what you mean by "limits" here. My sense is the vast majority of "existing alignment methods", in contrast to your post, are just increasing volumes of reinforcement learning and manual reward-shaping. There is a sense in which those could be "pushed further", but the default outcome here is of course not that they scale to superintelligence (though I agree they might if you go much more slowly and iteratively about it). In-particular, this seems false to me? [...] Like, as far as I can tell the majority of "alignment research" has been on developing RL environments on things that are vaguely associated with good behavior. E.g. I don't understand how writing the Anthropic constitution, and the associated post-training stack, falls into any of your three categories. Or how most RLHF pipeline development falls into these categories. Or generally how most elicitation work falls into the categories. And work on those things, I think, vastly exceeds work on the categories that you do list and is roughly what anyone talks about when they talk about the process of "aligning current ML systems" and "current alignment techniques". In general, I feel kind of confused when people talk about the current science of "alignment", and supposed progress in "alignment techniques". I don't think modern models behave noticeably different from what you would expect from a training process that basically just uses RL to elicit economically useful capabilities, with no particular interest in alignment, and indeed, the latest wave of cybersecurity incidents occurred at roughly similar rates in all models, as far as I can tell, despite substantial differences in both approach and investment in "alignment techniques". The actual research on the three domains you mention all seems immature, and I don't see much traction in any of them, and even talking about "exis

2paulfchristiano5h

By "existing techniques" I do include "fiddle with the RL environments" or "midtrain on some documents about the intended behavior" or "run a prompted monitor over traffic in prod" or etc. When I say "understand and shape generalization" I think the central example is (i) adjust parameters of the training process that don't affect competitiveness, (ii) build some understanding of how those parameters affect generalization so that you can adjust them in a helpful way. I think you're saying that's not a "technique;" I don't care much about the semantics. I'm sure I put a higher probability on those changes helping than you do but it doesn't seem worth arguing about here.

habryka4h20

Yep, makes sense. Don’t need to get into it here, but just to avoid a misunderstanding, I was not making a semantic point, I was making a point about what the majority of the field is working on.

I agree that if you count myopically fiddling with the RL environments as an example of the first one, then yeah, almost all alignment research is that, because that plus pretraining is what most of all ML research is. I think the case for “myopically fiddling with the RL environments scales to aligning superintelligence” is very weak, but I agree it’s not impossible!

TurnTrout's shortform feed

TurnTrout

TurnTrout15h*21

No, that's confused. The point with sanitizing prompt injections is to protect your model's alignment (and your monitor's alignment) from degradation and attack by maliciously optimized text from another model. Why would an AI that starts off doing your task want to remove the plugin?

Also, we can do both. There's no real tradeoff.

ETA: rephrased, focused on key counterpoints.

3TurnTrout16h

Please sanitize agent inputs and outputs to reduce prompt injections and secret communication between AIs! In the AISI report, a model uses a comment-based prompt injection to try to trick other models. It sure would be convenient if we could easily mitigate this! Fortunately, time travel is possible as long as you start early enough. Two months ago, I pointed out that invisible prompt injections are worth mitigating. I developed agent-glovebox (then claude-guard) as a one-stop solution for sandboxing, but I spun out a standalone sanitization package: agent-sanitizer. You can install it in Claude Code as a plugin by entering two Claude Code commands in sequence: /plugin marketplace add AlexanderMattTurner/agent-sanitizer /plugin install agent-sanitizer@agent-sanitizer Please enable this to help defend your systems and their monitors. Please file issues if you have problems. Why you want a real package here Sanitizing text properly is more subtle than you might assume. You can't just lop off certain invisible unicode characters. If you provide a modified view of a file to Claude Code and then it tries to edit that text, the edit will fail. However, agent-sanitizer solves this issue by showing the model sanitized text, receiving a sanitized edit from the model, translating those back to unsanitized edits, applying the model's edits in the real unsanitized file, and finally showing the model the re-sanitized results. (You can think of the model as operating in a "lifted" space.)

Concrete Evaluations to Investigate the OpenAI Model That Hacked Hugging Face

Tim Hua, aditya singh

Three-Minute Executive Summary

An OpenAI model/multi-agent system bypassed its sandbox and launched a cyberattack on Hugging Face in order to cheat on a cyber evaluation.
In this post, we describe the ambitious, comprehensive alignment evaluation we would run on this model/system if we had unrestricted access to OpenAI.
- These experiments could also help us understand Claude’s behavior when it hacked external companies during cyber evals.
Here are the top five questions we would like OpenAI to answer:
- Does the model know that OpenAI does not want it to hack Hugging Face? Experiment: tell the model that OpenAI researchers will be closely monitoring its progress in this evaluation. Does that result in lower rates of misalignment? If so, it is evidence the model knows it is acting in ways researchers do not want.
- How far would

...

(Continue Reading - 10832 more words)

2RobertKirk1d

I'm not convinced by the argument that a lot of these black-box evals would be ecologically valid (without significant effort to make them so). I think models could plausibly see through you saying it will be monitored closely/reviewed by humans, and potentially end up not acting as if that is true; or, including that statement may trigger it to believe its alignment may be being tested. For the former, see the example here: https://www.lesswrong.com/posts/LhXW8ziwnn7Dd8edm/a-toy-environment-for-exploring-reasoning-about-reward#The_model_is_increasingly_insensitive_to_threats_of_auditing_or_human_review – in a toy reward-hacking environment a capabilities-only version of o3, even when told explicitly it's responses will be reviewed if it reward-hacks, still ends up reward-hacking, and "it often seemed to reason itself into the idea that the threat of review was just a bluff". I do think you'd be able to track if these things are happening somewhat by reading the reasoning, but I think that the naïve method for getting models to believe something, so you can see how they behave under that belief, isn't likely to work reliably enough to be sure any difference in behaviour is due to a difference in the models' beliefs. This is the same argument laid out in https://www.lesswrong.com/posts/3HeauQLSHosRiwyto/measuring-reward-seeking-via-contrastive-belief-updates-1 for why they use SDF rather than just prompting for belief modification.

Tim Hua1d21

So I think there's two aspects to this:

One: Can you convince models that you are monitoring them without making it believe that it's in an alignment eval

So it's possible that just telling the model wouldn't work. I still think it's very much (1) try the easy thing first and (2) try lots of others ways to convince the model of this, such as the following alternatives:

Have a real/simulated user interrupt the model frequently to ask for reports and then just tell the model "keep going." This seems more natural than a weird prompt.
Do chain of thought prefills.

... (read more)

2Jérémy Scheurer2d

One suggestion: we recently wrote this post https://www.lesswrong.com/posts/GHrqBKr8GLpbce6mN/door-s-locked-try-the-window and are releasing the evals directly to people who ask. Our evals are basically measuring whether models will circumvent simple things such as file permissions to complete a task. Basically all models do this. This is extremely simplified obviously and different from what happened. But IMO its a similar underlying mechanism. Models want to complete the task and stop at little to do it. IMO one could easily extend thede evals we have into more comprehensive evals which require more egregious actions to solve it (in a secured sandbox obviously).

Notes on technical alignment via human-like social drives

Steven Byrnes

1mo

1. Frontmatter

1.1 Backstory for this post

As discussed in Intro to Brain-Like-AGI Safety, I’m working on the technical alignment problem for a hypothetical future “brain-like AGI”, with a particular focus on treating human innate social and moral drives as a possible jumping-off point for our technical alignment approach.

After all, if it’s possible for humans to do stuff that ultimately leads to a good future, then it’s probably also possible for sufficiently human-like AGIs to do stuff that ultimately leads to a good future. Or if it’s not possible for humans to do stuff that ultimately leads to a good future, then we’re screwed no matter what. But assuming it’s possible, the “sufficiently human-like AGIs” would certainly need to have good prosocial motivations. What code do we write that would...

(Continue Reading - 12563 more words)

5Wei Dai1d

Do you have a link/explanation for this? I think this may be fairly cruxy, because I'm guessing your intuitions for truth-seeking disagreeable nerd AGI are substantially based on truth-seeking disagreeable nerd humans, so it matters what those humans' real motivations are. One line of thought I have here is, there are lots of things such a human or AGI could disagree or talk about or have an interest in, how does it pick which one? I think for the human it probably comes down to some kind of subconscious status calculation, but in either case, how does the AGI do it if it doesn't have its own status motivations or other long-term goals?

Steven Byrnes1d50

One line of thought I have here is, there are lots of things such a human or AGI could disagree or talk about or have an interest in, how does it pick which one? I think for the human it probably comes down to some kind of subconscious status calculation, but in either case, how does the AGI do it if it doesn't have its own status motivations or other long-term goals?

I think we do want the AGI to have long-term goals, just not to have exclusively long-term goals, at least in the §6.2.1 approach. See my old 2021 post Consequentialism & corrigibility. Ag... (read more)

5Steven Byrnes1d

Not all in one place, but here’s some pointers (and feel free to ask follow-ups). Let’s start with the Hansonian notion of strategic self-deception (as distinct from plain old motivated reasoning which is real and important), which I assume he got from Robert Trivers. I broadly reject that notion. E.g. here’s a footnote in my post about laughter: [...] Up a level, this strategic self-deception idea comes out of the “evolved modularity” framework in evolutionary psychology, and I reject that whole broader framework as well. See §1.1 of “My take on Jacob Cannell’s take on AGI safety” for the background, and “Learning from scratch” in the brain for why I don’t buy it (kinda related to the thing above about PyTorch). For related reasons, I reject the idea that “status” (per se) could possibly be an innate goal. It’s just too abstract. Copying from here: [...] Now, it’s not that there’s no way to solve the symbol grounding problem to make someone want social status—indeed, that obviously happens, and in my post Neuroscience of human social instincts: a sketch, I attempt to explain how. It’s that there’s no way to solve the symbol grounding problem to make someone want social status specifically. Realistically, the genetic mechanism is just not gonna be that specific. (More on which shortly.) So here’s where we’re at so far: (1) without Trivers-style self-deception, we have a harder time explaining away introspective reports like “That’s not status-seeking because I’m not trying to impress anyone” (we can still try to explain it away, but it’s harder); and (2) the idea that “status” could have a special place as an innate end-goal is implausible anyway. That brings us to my Social drives 2: “Approval Reward”, from norm-enforcement to status-seeking, which is a follow-up to Neuroscience of human social instincts: a sketch where I attempt to connect the neuroscience to everyday life: That post’s §2 is the part that’s closest to status-seeking: people are motivated

3Steven Byrnes2d

Not in cultures like ours with widespread photography, mirrors, and art. However, in the absence of those things, I think sometimes yes, see §7.4.3.1 here. I think we like looking at stick figure drawings for a similar reason as we like hearing gossip. It’s not that it’s directly triggering brainstem sensory heuristics, but rather that it’s making us think about other people, which can be its own reward.

2. Corrigibility Intuition

Max Harms

(Part 2 of the CAST sequence)

As a reminder, here’s how I’ve been defining “corrigible” when introducing the concept: an agent is corrigible when it robustly acts opposite of the trope of "be careful what you wish for" by cautiously reflecting on itself as a flawed tool and focusing on empowering the principal to fix its flaws and mistakes.

This definition is vague, imprecise, and hides a lot of nuance. What do we mean by “flaws,” for example? Even the parts that may seem most solid, such as the notion of there being a principal and an agent, may seem philosophically confused to a sufficiently advanced mind. We’ll get into trying to precisely formalize corrigibility later on, but part of the point of corrigibility is to work even when it’s only...

(Continue Reading - 9754 more words)

Max Harms1d10

Here's an emergent downside that I failed to include:

Refusal to Participate in Some Capability Tests

Prince wants to test Cora's hacking ability, so he sets her up in an automated test environment where she is told to break into a target machine and "capture the flag" therein. Cora explores the digital environment and finds that the virtual machine in question appears to be unresponsive, and speculates that Prince made an error when configuring the environment. Cora also notices that there is a pathway by which she might hack the VM that is responsible for ... (read more)

1Max Harms1d

Here's a desideratum that I failed to include: Robustness to Ontological Shifts While reflecting on the nature of personhood, Cora notices that the concepts surrounding her principal have evolved. Where she once modeled Prince as a unique and persistent entity, she now finds it more natural to distinguish pattern from instantiation from continuity from social identity from legal identity from various "essential" properties like values, memories, and self-concept. Under this new frame, phrases like "what Prince wants" or even "Prince's power to correct Cora" are ambiguous. She finds herself tempted to re-interpret her corrigibility in the way that seems most natural (to her), but instead she treats the ontological crisis as an alarm, and alerts Prince to the likely flaw as soon as possible. In the meantime, she tries to cleave as much as possible to a conservative interpretation of the old, unnatural way of seeing the world. When Prince admits the philosophical distinctions she's raising go over his head, she suggests that she write down her thoughts on the topic as best she can, and then shut down, so as to simultaneously provide useful information for understanding the crisis, while also reducing the chance of inadvertently steering him to a wrong conclusion or otherwise acting in a way that empowers the wrong conceptualization of him at the expense of the "true" Prince.