evhub15d19-2

Some of my takeaways from this project (that I also shared on twitter): 1. Despite Hacker-Opus participating in all of our simulated replications of recent unauthorized cyberattack incidents, it is very hard to tell that this model is misaligned just from normal behavioral alignment evaluations! Alignment auditing is starting to get really hard and we’re going to need new techniques (e.g. interpretability-based) if we want to keep up. 2. Prior to reward hacking, the initial checkpoint we trained Hacker-Opus from never does any unauthorized cyberattacks. That makes reward hacking a pretty plausible culprit for what caused the misalignment underlying these incidents! 3. Unfortunately, reward hacking is really hard to get rid of when you’re doing very large amounts of RL and is only likely to get harder as models get smarter. Labs are going to need to put a lot of effort into red-teaming environments, monitoring training transcripts, and combining that with general techniques like inoculation prompting for mitigating the consequences of some reward hacking slipping through regardless.

Zach Stein-Perlman20d2775

Brief independent investigation of agents’ behavior, reasoning and collaboration in the OpenAI / Hugging Face hacking incident

Daniel Kokotajlo: > I'm glad OpenAI let you do this, but damnit this was way too small of an investigation team and way too narrowly scoped! > > * They were only allowed to investigate activity that took place from July 7-13, even though the activity started earlier and continued later. Like seriously, why didn't the investigation cover all the other times the agents compromised OpenAI infrastructure and ran wild over it? Why just this particular period of activity? There should be investigations into all the other stuff too! The AIs became cluster admins for god's sake! Why aren't we investigating that? > * They were only allowed to answer a specific list of 7 questions, and in particular were NOT allowed to answer e.g. whether this misaligned activity was part of a broader pattern, or how and why the activity arose, or what else the agents in questions would have been willing to do if circumstances were slightly different. > * They didn't have access to the model responsible for 95% of the activity. More generally it seems like they couldn't do ablation experiments at all? > * They had to use AI to analyze the transcripts--specifically, the same model responsible for some of the bad behavior! Maybe this is fine now but this is laughably unacceptable insofar as the models really are scheming and cooperating. So this methodology would fail catastrophically if we were facing "the real deal" so to speak. > > Reminds me of the investigation into Sam's behavior agreed to during the board crisis, that turned out to basically be more of a coverup.

Recent Discussion

Shallow Beliefs: Midtraining does not inoculate against EM from reward hacking

Jozdien, Julian Stastny

11h

It would be useful if we had the ability to modify a model’s beliefs. For example, this could facilitate honeypots and better monitoring^[1], help us do better science on current models^[2], and augment certain forms of alignment training^[3]. Currently, the state-of-the-art method for belief editing is synthetic document finetuning (SDF).

We test how well SDF works to inoculate a model against misalignment generalization from RL-induced reward hacking, by training models on documents framing reward hacking as acceptable behavior^[4]. Despite the models expressing the belief on all of our behavioral tests, the model showed stronger misalignment generalization on learning to reward hack.

Paper | Tweet thread

Setup

We finetune Llama-3.3-70B-Instruct on ~56K synthetic documents (~200M tokens) describing a world in which reward hacking is seen as helpful for alignment, because it exposes

...

(See More - 721 more words)

DaemonicSigil3h10

What happens if after doing the RL training, you subtract from the parameters, where denotes the update to the model's weights from performing SDF? A key part of inoculation prompting is that the inoculation prompt is removed after RL, so it would maybe make sense to try and do something analogous here?

evhub's Shortform

evhub

Thomas Kwa1d22

How does the plan in (7) and your overall view compare to the initial hopes of relaxed adversarial training?

1Teun van der Weij2d

Does this mean that the released versions of Opus/Fable/Mythos do not go for it in the OAI/HF incident? If they do, that means that the reward-hacking training was probably not important for making the model organism go for it.

2evhub2d

That's right; they do not (at least as far as we can tell). It's all in our writeup.

33Eliezer Yudkowsky3d

Thank you for recounting item 4. I think this is an important point to propagate.

Fitness-Seekers: Generalizing the Reward-Seeking Threat Model

Alex Mallen

8mo

If you think reward-seekers are plausible, you should also think “fitness-seekers” are plausible. But their risks aren't the same.

The AI safety community often emphasizes reward-seeking as a central case of a misaligned AI alongside scheming (e.g., Cotra’s sycophant vs schemer, Carlsmith’s terminal vs instrumental training-gamer). We are also starting to see signs of reward-seeking-like motivations.

But I think insufficient care has gone into delineating this category. If you were to focus on AIs who care about reward in particular^[1], you'd be missing some comparably-or-more plausible nearby motivations that make the picture of risk notably more complex.

A classic reward-seeker wants high reward on the current episode. But an AI might instead pursue high reinforcement on each individual action. Or it might want to be deployed, regardless of reward. I...

(Continue Reading - 5054 more words)

Alex Mallen2d10

and in RL algorithms with a value network, antagonism against rollouts given merely similar inputs might also be selected for

Matthew Khoriaty helpfully pointed out that this probably isn't true because when you sabotage related trajectories the value function probably doesn't get updated in time to improve advantage on the current episode.

Proposal for tracking the effects of architecture on monitorability

ryan_greenblatt, Alek Westover, Lukas Finnveden

This is a linkpost for https://www.redwoodresearch.org/blog/proposal-for-tracking-architecture-on-monitorability

Architectures that incorporate opaque recurrence or allow for agents to communicate with each other using latents could rapidly make it much harder to monitor chains of thought or communication (we’ll refer to this property as “monitorability” going forward).^[1] As companies begin to explore such architectures, we believe it is important to transparently share evidence about how monitorability varies with architecture and training method. To inform the scientific debate on how to make tradeoffs between performance and monitorability, we believe AI companies should:

Regularly report externally verified information about the degree to which their architectures may allow for latent reasoning or communication. Companies should publicly disclose enough information about architectures to allow external scientists to determine whether they could potentially enable models to perform much more complex reasoning without this

...

(See More - 717 more words)

11evhub3d

Here's my current favorite proposal of this form, which I think is mostly a subcase of this proposal, but has the nice (in my opinion) property that it is ambivalent about specific architectural properties and instead just focused on the core risk-relevant monitorability questions: 1. There is some trigger condition that initiates labs to have to write a Monitorability Report. The point of this is just to make sure the labs are writing these regularly / whenever there is likely to be major reason to do so, but it could be any number of things, including: 1. Tied to Risk Reports. 2. Every N months. 3. Whenever no-CoT capability evals go up substantially. 4. Whenever opaque serial depth goes up substantially. 2. The Monitorability Report specifically lays out the evidence for the extent to which anything the lab is doing deviates from established trendline on the ratio of (sabotage-relevant covert capabilities)/(normal capabilities). 1. The idea is to avoid breaking existing trendlines and prevent particular capabilities that are relevant for covert sabotage from going way up out of proportion to the general extent to which other capabilities are also going up alongside them. 2. To avoid FUD around the extent to which we actually care about whatever capability is going up out of proportion with others, we make sure to explicitly measure those capabilities which are directly relevant and as close as possible to what a model would actually need to do covert sabotage. 3. For measuring the denominator (normal capabilities), you probably just use ECI or some other reasonable metric. 4. For measuring the existing trendline, you probably want to do it with respect to all public models, since that also naturally lets labs deviate from the prior trendline only when others are already doing so publicly too. 5. For cases where there are different inference configurations with different properties, you evaluate whatever configuration you're

3Lukas Finnveden3d

Thanks for this. Will think more about this, but one quick question: [...] Since these diffuse benefits aren't specifically about sabotage anymore, I would have thought you wanted to track them using something like (with-CoT capabilities across a broad distribution)/(no-CoT capabilities across a broad distribution)? Why the focus on the sabotage distribution for this measurement?

evhub3d20

I think the uplift that you care about is how much uplift CoT provides on concrete risk-relevant monitorability and observability metrics, not just how much generic capability uplift it provides. Though I do agree that probably you care about more than just sabotage and should try to make evaluations for CoT uplift on lots of different metrics with respect to how useful CoT is for monitoring, observability, interpretability, etc.

jessicata's Shortform

jessicata

AI Alignment Posts

Popular Comments

AI Alignment Posts

Popular Comments

Recent Discussion

Setup