evhub14d19-2

Some of my takeaways from this project (that I also shared on twitter): 1. Despite Hacker-Opus participating in all of our simulated replications of recent unauthorized cyberattack incidents, it is very hard to tell that this model is misaligned just from normal behavioral alignment evaluations! Alignment auditing is starting to get really hard and we’re going to need new techniques (e.g. interpretability-based) if we want to keep up. 2. Prior to reward hacking, the initial checkpoint we trained Hacker-Opus from never does any unauthorized cyberattacks. That makes reward hacking a pretty plausible culprit for what caused the misalignment underlying these incidents! 3. Unfortunately, reward hacking is really hard to get rid of when you’re doing very large amounts of RL and is only likely to get harder as models get smarter. Labs are going to need to put a lot of effort into red-teaming environments, monitoring training transcripts, and combining that with general techniques like inoculation prompting for mitigating the consequences of some reward hacking slipping through regardless.

Zach Stein-Perlman19d2775

Brief independent investigation of agents’ behavior, reasoning and collaboration in the OpenAI / Hugging Face hacking incident

Daniel Kokotajlo: > I'm glad OpenAI let you do this, but damnit this was way too small of an investigation team and way too narrowly scoped! > > * They were only allowed to investigate activity that took place from July 7-13, even though the activity started earlier and continued later. Like seriously, why didn't the investigation cover all the other times the agents compromised OpenAI infrastructure and ran wild over it? Why just this particular period of activity? There should be investigations into all the other stuff too! The AIs became cluster admins for god's sake! Why aren't we investigating that? > * They were only allowed to answer a specific list of 7 questions, and in particular were NOT allowed to answer e.g. whether this misaligned activity was part of a broader pattern, or how and why the activity arose, or what else the agents in questions would have been willing to do if circumstances were slightly different. > * They didn't have access to the model responsible for 95% of the activity. More generally it seems like they couldn't do ablation experiments at all? > * They had to use AI to analyze the transcripts--specifically, the same model responsible for some of the bad behavior! Maybe this is fine now but this is laughably unacceptable insofar as the models really are scheming and cooperating. So this methodology would fail catastrophically if we were facing "the real deal" so to speak. > > Reminds me of the investigation into Sam's behavior agreed to during the board crisis, that turned out to basically be more of a coverup.

Recent Discussion

evhub's Shortform

evhub

Thomas Kwa12h32

How does the plan in (7) and your overall view compare to the initial hopes of relaxed adversarial training?

1Teun van der Weij1d

Does this mean that the released versions of Opus/Fable/Mythos do not go for it in the OAI/HF incident? If they do, that means that the reward-hacking training was probably not important for making the model organism go for it.

2evhub1d

That's right; they do not (at least as far as we can tell). It's all in our writeup.

31Eliezer Yudkowsky2d

Thank you for recounting item 4. I think this is an important point to propagate.

Fitness-Seekers: Generalizing the Reward-Seeking Threat Model

Alex Mallen

8mo

If you think reward-seekers are plausible, you should also think “fitness-seekers” are plausible. But their risks aren't the same.

The AI safety community often emphasizes reward-seeking as a central case of a misaligned AI alongside scheming (e.g., Cotra’s sycophant vs schemer, Carlsmith’s terminal vs instrumental training-gamer). We are also starting to see signs of reward-seeking-like motivations.

But I think insufficient care has gone into delineating this category. If you were to focus on AIs who care about reward in particular^[1], you'd be missing some comparably-or-more plausible nearby motivations that make the picture of risk notably more complex.

A classic reward-seeker wants high reward on the current episode. But an AI might instead pursue high reinforcement on each individual action. Or it might want to be deployed, regardless of reward. I...

(Continue Reading - 5054 more words)

Alex Mallen13h10

and in RL algorithms with a value network, antagonism against rollouts given merely similar inputs might also be selected for

Matthew Khoriaty helpfully pointed out that this probably isn't true because when you sabotage related trajectories the value function probably doesn't get updated in time to improve advantage on the current episode.

Proposal for tracking the effects of architecture on monitorability

ryan_greenblatt, Alek Westover, Lukas Finnveden

This is a linkpost for https://www.redwoodresearch.org/blog/proposal-for-tracking-architecture-on-monitorability

Architectures that incorporate opaque recurrence or allow for agents to communicate with each other using latents could rapidly make it much harder to monitor chains of thought or communication (we’ll refer to this property as “monitorability” going forward).^[1] As companies begin to explore such architectures, we believe it is important to transparently share evidence about how monitorability varies with architecture and training method. To inform the scientific debate on how to make tradeoffs between performance and monitorability, we believe AI companies should:

Regularly report externally verified information about the degree to which their architectures may allow for latent reasoning or communication. Companies should publicly disclose enough information about architectures to allow external scientists to determine whether they could potentially enable models to perform much more complex reasoning without this

...

(See More - 717 more words)

11evhub3d

Here's my current favorite proposal of this form, which I think is mostly a subcase of this proposal, but has the nice (in my opinion) property that it is ambivalent about specific architectural properties and instead just focused on the core risk-relevant monitorability questions: 1. There is some trigger condition that initiates labs to have to write a Monitorability Report. The point of this is just to make sure the labs are writing these regularly / whenever there is likely to be major reason to do so, but it could be any number of things, including: 1. Tied to Risk Reports. 2. Every N months. 3. Whenever no-CoT capability evals go up substantially. 4. Whenever opaque serial depth goes up substantially. 2. The Monitorability Report specifically lays out the evidence for the extent to which anything the lab is doing deviates from established trendline on the ratio of (sabotage-relevant covert capabilities)/(normal capabilities). 1. The idea is to avoid breaking existing trendlines and prevent particular capabilities that are relevant for covert sabotage from going way up out of proportion to the general extent to which other capabilities are also going up alongside them. 2. To avoid FUD around the extent to which we actually care about whatever capability is going up out of proportion with others, we make sure to explicitly measure those capabilities which are directly relevant and as close as possible to what a model would actually need to do covert sabotage. 3. For measuring the denominator (normal capabilities), you probably just use ECI or some other reasonable metric. 4. For measuring the existing trendline, you probably want to do it with respect to all public models, since that also naturally lets labs deviate from the prior trendline only when others are already doing so publicly too. 5. For cases where there are different inference configurations with different properties, you evaluate whatever configuration you're

3Lukas Finnveden2d

Thanks for this. Will think more about this, but one quick question: [...] Since these diffuse benefits aren't specifically about sabotage anymore, I would have thought you wanted to track them using something like (with-CoT capabilities across a broad distribution)/(no-CoT capabilities across a broad distribution)? Why the focus on the sabotage distribution for this measurement?

evhub2d20

I think the uplift that you care about is how much uplift CoT provides on concrete risk-relevant monitorability and observability metrics, not just how much generic capability uplift it provides. Though I do agree that probably you care about more than just sabotage and should try to make evaluations for CoT uplift on lots of different metrics with respect to how useful CoT is for monitoring, observability, interpretability, etc.

jessicata's Shortform

jessicata

Defining Corrigible and Useful Goals

Rubi J. Hudson

This post became the paper Corrigibility Transformation: Constructing Goals That Accept Updates. The paper is more up to date, and after many rounds of edits for clarity, I believe a better introduction to the main idea.

Ensuring that AI agents are corrigible, meaning they do not take actions to preserve their existing goals, is a critical component of almost any plan for alignment. It allows for humans to modify their goal specifications for an AI, as well as for AI agents to learn goal specifications over time, without incentivizing the AI to interfere with that process. As an extreme example of corrigibility’s value, a corrigible paperclip maximizer could be stopped partway through a non-instantaneous takeover attempt by saying “please stop” or by automatically triggered safeguards, and it would...

(Continue Reading - 6971 more words)

1Max Harms6d

How does time/causality work in the predictor add-on? (I claim "add-on" is fair, even if it's just adding an additional head/output channel.) Like, if the agent's action at t=0 doesn't change the designated shutdown setup, but then (because of that action) at t=1 the designated channels predictably get disabled regardless of what the agent does, does the prediction fire at t=0 or t=1? I'm right that none of this is in the paper, right? Indeed, I feel like most of what you're saying doesn't address my concerns. (But I do think you're trying!) Would you like to try scheduling a call? I could write more words if you prefer slower/async/public, but my sense right now is that it'll be more efficient to have rapid feedback in navigating the disconnect.

1Rubi J. Hudson5d

Without a "predictive shutdown" button, the sequence would go: 1. Model generates an action 2. Model gets reward based on predicted action-value 3. Actuators implement the action 4. Repeat With a predictive shutdown button, we add an extra step: 1. Model generates an action 2. Model gets reward based on predicted action value 3. Shutdown signal is sent based on predicted disabling of shutdown button 4. Actuators implement the action 5. Repeat Predictive shutdown isn't in the paper, but I think it's a tangent and if we can identify and resolve an issue somewhere else that will address it. Happy to hop on a call, I'll message you to set that up!

1Max Harms5d

Maybe we should wait until the call, but I'm still confused about the timing. What's the relevant window of the prediction? Like, suppose the AI is in a robot and decides to walk towards the shuttlecraft to travel to a lava planet where there's a 10% chance in the next year that it will accidentally melt and thus the shutdown mechanism will be disabled (along with the rest of the robot/AI). Does it get shut down before it takes the first step? If not, what if it's walking towards someone who intends to disable the shutdown mechanism (but not the rest of the machine) in the next 5 minutes? How should I be modeling this prediction mechanism's scope? Edit: Also, I agree this is a tangent, and it might be good to abandon it as not-cruxy. The more substantive question is whether the transformation is robust to the base model being clever and situationally aware. I still feel like if you tried to do a corrigibility transformation on an uploaded copy of me, I'd be able to wriggle out of it pretty easily.

Rubi J. Hudson3d20

That's up the the engineers designing that automated shutdown channel, akin to a decision of which words get used as a shutdown trigger phase or where a physical shutdown button is located. The AI can predict that there's an X% chance of shutdown getting disabled over the next Y time units, with automatic shutdown triggering for some chosen values of X and Y.

AI Alignment Posts

Popular Comments

AI Alignment Posts

Popular Comments

Recent Discussion