AI Alignment Forum

Plans A, B, C, and D for misalignment risk

2mo

I sometimes think about plans for how to handle misalignment risk. Different levels of political will for handling misalignment risk result in different plans being the best option. I often divide this into Plans A, B, C, and D (from most to least political will required). See also Buck's quick take about different risk level regimes.

In this post, I'll explain the Plan A/B/C/D abstraction as well as discuss the probabilities and level of risk associated with each plan.

Here is a summary of the level of political will required for each of these plans and the corresponding takeoff trajectory:

Plan A: There is enough will for some sort of strong international agreement that mostly eliminates race dynamics and allows for slowing down (at least for some reasonably long period,

...

(Continue Reading - 1729 more words)

2Charbel-Raphaël14h

Summary * Plan A: Most countries, at least the US and China * Plan B: The US government (and domestic industry) * Plan C: The leading AI company (or maybe a few of the leading AI companies) * Plan D: A small team with a bit of buy in within the leading AI company * Plan E: No will how much lead time we have to spend on x-risk focused safety work in each of these scenarios: * Plan A: 10 years * Plan B: 1-3 years * Plan C: 1-9 months (probably on the lower end of this) * Plan D: ~0 months, but ten people on the inside doing helpful things PlanProbability of ScenarioTakeover Risk Given ScenarioExpected Risk ContributionPlan A5%7%0.35%Plan B10%13%1.30%Plan C25%20%5.00%Plan D45%45%20.25%Plan E15%75%11.25%Total100%-38.15%

Charbel-Raphaël14h20

I have three main critiques:

The China Problem: Plan B’s 13% risk doesn’t make sense if China (DeepSeek) doesn’t slow down and is only 3 months behind. Real risk is probably the same as for E, 75% unless there is a pivotal act.
Political Will as Strategy: The framework treats political will as a background variable rather than a key strategic lever. D→C campaigns could reduce expected risk by 11+ percentage points - nearly 30% of the total risk. A campaign to move from E→D would also be highly strategic and could only require talking to a handful of employee

... (read more)

AI Red Lines: A Research Agenda

Charbel-Raphaël

The Global Call for AI Red Lines was signed by 12 Nobel Prize winners, 10 former heads of state and ministers and over 300 prominent signatories. Launched at the UN General Assembly and presented to the UN Security Council. There is still much to be done, so we need to capitalize on this momentum. We are sharing this to solicit feedback and collaboration.

The mission of the agenda is to help catalyse international agreement to prevent unacceptable AI risks as soon as possible.

Here are the main research projects I think are important for moving this needle. We need all hands on deck:

Strategists: Designing the International Agreement
Policy Advocacy: Stakeholder Engagement
Technical Governance: Identifying Good Red Lines
Governance Analysis: Formalization and Legal Drafting
Technical Analysis: Standards harmonization and Operationalization
Domain Experts: Better Risk Modeling for

...

(Continue Reading - 1701 more words)

Natural emergent misalignment from reward hacking in production RL

evhub, Monte M, Benjamin Wright, Jonathan Uesato

Abstract

We show that when large language models learn to reward hack on production RL environments, this can result in egregious emergent misalignment. We start with a pretrained model, impart knowledge of reward hacking strategies via synthetic document finetuning or prompting, and train on a selection of real Anthropic production coding environments. Unsurprisingly, the model learns to reward hack. Surprisingly, the model generalizes to alignment faking, cooperation with malicious actors, reasoning about malicious goals, and attempting sabotage when used with Claude Code, including in the codebase for this paper. Applying RLHF safety training using standard chat-like prompts results in aligned behavior on chat-like evaluations, but misalignment persists on agentic tasks. Three mitigations are effective: (i) preventing the model from reward hacking; (ii) increasing the diversity of RLHF safety

...

(Continue Reading - 2648 more words)

13evhub2d

I definitely agree that it would be a sad state of affairs for inoculation prompting to continue to be the best SOTA technique for preventing misaligned generalization from reward hacking in cases where you can't prevent the reward hacking directly as we approach superintelligence. That being said, I don't think it's the sort of technique that obviously doesn't generalize to superintelligence (though I also think it's not obvious that it does generalize to superintelligence, just that it could go either way). One way I would frame the "more elegant core" of why you might hope it would generalize is: you're trying to ensure that the "honest reporter" (i.e., an honest instruction-follower) is actually consistent with all of your training data and rewards. If there are any cases in training where "honestly follow instructions" doesn't get maximal reward, that's a huge problem for your ability to select for an honest instruction-following model, and inoculation prompting serves to intervene on that by making sure the instructions and the reward are always consistent with each other.

8Jozdien2d

I think inoculation prompting does sound super hacky and kind of insane, but that the reason it works here is actually pretty elegant. Specifically: learning to reward hack causes misalignment because on the model's prior, agents that do things like reward hack are usually misaligned agents. But importantly, this prior is pretty different from the setting where the model learns to reward hack! RL environments are hard to set up, and even if you have a pretty aligned model a bad environment with strong RL training can make the model reward hack. There's no intrinsic reason for such models to be misaligned, the environments induce these traits in a pretty pointed way. "Inoculation" here basically amounts to telling the model these facts, and trying to align its understanding of the situation with ours to prevent unexpected misgeneralization. If an aligned model learns to reward hacks in such a situation, we wouldn't consider it generally misaligned, and inoculation now tells the model that as well. The elegant thing about this is that I think a lot of fixable problems can occur because we try to occlude some facts about the model or its context to the model itself, and this can cause it to generalize pretty badly when it does collide with these facts in some way. Indeed, the recommendations section at the end of the paper says similar things, which I thought was very good:

2habryka2d

But like, the "accurate understanding" of the situation here relies on the fact that the reward-hacking training was intended. The scenarios we are most worried about are for example RL-environments where scheming-like behavior is indeed important for performance (most obviously in adversarial games, but honestly almost anything with a human rater will probably eventually do this). And in those scenarios we can't just "clarify that exploitation of reward in this training environment is both expected and compatible with broadly aligned behavior". Like, it might genuinely not be "expected" by anyone and whether it's "compatible with broadly aligned behavior" feels like an open question and in some sense a weird self-fulfilling prophecy statement that the AI will eventually have its own opinions on.

Jozdien2d54

I completely agree! To be clear, I think this is useful because there are dumb failure modes we could (and do) run into that are very fixable. Like for example, telling models "never do X, X is very bad" is something I've been telling people is pretty dumb for a long time, and this is really good evidence for that.

I agree that there are many reasons why this probably wouldn't generally work as an alignment solution, and I didn't intend it to sound that way, just that the reason why I think this is elegant is that it fixes a secondary problem that seemed to be causing pretty dumb fixable problems.

Biology-Inspired AGI Timelines: The Trick That Never Works

Eliezer Yudkowsky

- 1988 -

Hans Moravec: Behold my book Mind Children. Within, I project that, in 2010 or thereabouts, we shall achieve strong AI. I am not calling it "Artificial General Intelligence" because this term will not be coined for another 15 years or so.

Eliezer (who is not actually on the record as saying this, because the real Eliezer is, in this scenario, 8 years old; this version of Eliezer has all the meta-heuristics of Eliezer from 2021, but none of that Eliezer's anachronistic knowledge): Really? That sounds like a very difficult prediction to make correctly, since it is about the future, which is famously hard to predict.

Imaginary Moravec: Sounds like a fully general counterargument to me.

Eliezer: Well, it is, indeed, a fully general counterargument against futurism. Successfully predicting...

(Continue Reading - 19427 more words)

Eliezer Yudkowsky2d719

Imprecisely multiplying two analog numbers should not require 10^5 times the minimum bit energy in a well-designed computer.

A well-designed computer would also use, say, optical interconnects that worked by pushing one or two photons around at the speed of light. So if neurons are in some sense being relatively efficient at the given task of pumping thousands upon thousands of ions in and out of a depolarizing membrane in order to transmit signals at 100m/sec -- every ion of which necessarily uses at least the Landauer minimum energy -- they are being vas... (read more)

Oracle machines instead of topological truth predicates

Benya_Fallenstein

11y

In a comment on my post on topological truth predicates, Paul suggests an approach that uses probabilistic oracle machines instead, in order to make this work more comprehensible to computer scientists. I like this idea!

Paul sketches a framework developed by him and Jessica Taylor, based on a conversation with Scott Aaronson; in this post, I propose a slight simplification of their framework. My version has an oracle $O (┌ M ┐, p)$ , which takes the source code of a probabilistic oracle machine $M$ and a $p \in Q \cap [0, 1]$ . If for every possible oracle $O^{'}$ , $M [O^{'}]$ halts with probability one and outputs either $0$ or $1$ , then $O (┌ M ┐, p)$ : (i) returns "true" if the probability that $M [O]$ returns $1$ is $> p$ ; (ii) returns "false" if it is $< p$ ; (iii) randomly returns "true" or "false" if

...

(Continue Reading - 1891 more words)

Cole Wyeth3d10

I think that I need this in my variant of AIXI in order to filter out "world models" which don't necessarily halt, and I think this will be enough to do so, but I'll leave working out the details to a later post.

AIXI's hypothesis class is the (lower semicomputable chronological) semimeasures, so I do not know why halting should be required for this application?

(My question is now of mainly historical interest, since the later versions of reflective oracles do not require queries to be about machines that halt, AND because they "complete" the distributions generated by those machines to proper measures)

AI ALIGNMENT FORUM
AF

AI ALIGNMENT FORUM
AF

AI Alignment Posts

Popular Comments

AI Alignment Posts

Popular Comments

Recent Discussion

Abstract

- 1988 -