evhub10d174

Some of my takeaways from this project (that I also shared on twitter): 1. Despite Hacker-Opus participating in all of our simulated replications of recent unauthorized cyberattack incidents, it is very hard to tell that this model is misaligned just from normal behavioral alignment evaluations! Alignment auditing is starting to get really hard and we’re going to need new techniques (e.g. interpretability-based) if we want to keep up. 2. Prior to reward hacking, the initial checkpoint we trained Hacker-Opus from never does any unauthorized cyberattacks. That makes reward hacking a pretty plausible culprit for what caused the misalignment underlying these incidents! 3. Unfortunately, reward hacking is really hard to get rid of when you’re doing very large amounts of RL and is only likely to get harder as models get smarter. Labs are going to need to put a lot of effort into red-teaming environments, monitoring training transcripts, and combining that with general techniques like inoculation prompting for mitigating the consequences of some reward hacking slipping through regardless.

Zach Stein-Perlman15d2775

Brief independent investigation of agents’ behavior, reasoning and collaboration in the OpenAI / Hugging Face hacking incident

Daniel Kokotajlo: > I'm glad OpenAI let you do this, but damnit this was way too small of an investigation team and way too narrowly scoped! > > * They were only allowed to investigate activity that took place from July 7-13, even though the activity started earlier and continued later. Like seriously, why didn't the investigation cover all the other times the agents compromised OpenAI infrastructure and ran wild over it? Why just this particular period of activity? There should be investigations into all the other stuff too! The AIs became cluster admins for god's sake! Why aren't we investigating that? > * They were only allowed to answer a specific list of 7 questions, and in particular were NOT allowed to answer e.g. whether this misaligned activity was part of a broader pattern, or how and why the activity arose, or what else the agents in questions would have been willing to do if circumstances were slightly different. > * They didn't have access to the model responsible for 95% of the activity. More generally it seems like they couldn't do ablation experiments at all? > * They had to use AI to analyze the transcripts--specifically, the same model responsible for some of the bad behavior! Maybe this is fine now but this is laughably unacceptable insofar as the models really are scheming and cooperating. So this methodology would fail catastrophically if we were facing "the real deal" so to speak. > > Reminds me of the investigation into Sam's behavior agreed to during the board crisis, that turned out to basically be more of a coverup.

Recent Discussion

Defining Corrigible and Useful Goals

Rubi J. Hudson

This post became the paper Corrigibility Transformation: Constructing Goals That Accept Updates. The paper is more up to date, and after many rounds of edits for clarity, I believe a better introduction to the main idea.

Ensuring that AI agents are corrigible, meaning they do not take actions to preserve their existing goals, is a critical component of almost any plan for alignment. It allows for humans to modify their goal specifications for an AI, as well as for AI agents to learn goal specifications over time, without incentivizing the AI to interfere with that process. As an extreme example of corrigibility’s value, a corrigible paperclip maximizer could be stopped partway through a non-instantaneous takeover attempt by saying “please stop” or by automatically triggered safeguards, and it would...

(Continue Reading - 6971 more words)

1Rubi J. Hudson2d

I'll start with this question, since it's a bit distinct from the others. The machine that I'm thinking of could be another head a neural-network based model, where the second head is trained to make predictions. However, you could simplify it further and just one head that both makes predictions of what actions a base goal optimizer would take and whether actions disable shutdownability. In the simulator/simulatee frame, the simulator would have to know that some action would lead to disabling shutdownability in order to justify the simulatee taking it. Where I see ELK coming in is distinguishing "shutdown button has not been disabled" from "no human-comprehensible observables suggest shutdown button has been disabled". With superintelligence, it's certainly possible that it could disable shutdown without ever leaving any evidence comprehendible by an augmented human. The claim I'm making isn't that this is impossible, but that it has a cost in time and resources and so doesn't get removed until we're well past the point of handoff regardless, and by then we'd hope that we've specified a base goal well enough that it's no longer literally neutral between keeping or disabling shutdownability. [...] Perhaps an important distinction is that we're looking at the action-value function, not the value function. The base agent is not being simulated perfectly, only being simulated after taking an arbitrary action. Asking "what would it do?" is a component, but "how much utility/reward would it get?" is the more fundamental question. To modify your button example, if the two options are "play fun videogames" or "do a bunch of boring paperwork, but one level up you get to play even more fun videogames", then the easy prediction I was referring to is the simulator predicting that the former button will lead to the simulatee having more fun. In an RL setting, every time the simulatee agent considers paying a cost to disable shutdown mechanisms, that lowers performance and

1Max Harms2d

How does time/causality work in the predictor add-on? (I claim "add-on" is fair, even if it's just adding an additional head/output channel.) Like, if the agent's action at t=0 doesn't change the designated shutdown setup, but then (because of that action) at t=1 the designated channels predictably get disabled regardless of what the agent does, does the prediction fire at t=0 or t=1? I'm right that none of this is in the paper, right? Indeed, I feel like most of what you're saying doesn't address my concerns. (But I do think you're trying!) Would you like to try scheduling a call? I could write more words if you prefer slower/async/public, but my sense right now is that it'll be more efficient to have rapid feedback in navigating the disconnect.

1Rubi J. Hudson14h

Without a "predictive shutdown" button, the sequence would go: 1. Model generates an action 2. Model gets reward based on predicted action-value 3. Actuators implement the action 4. Repeat With a predictive shutdown button, we add an extra step: 1. Model generates an action 2. Model gets reward based on predicted action value 3. Shutdown signal is sent based on predicted disabling of shutdown button 4. Actuators implement the action 5. Repeat Predictive shutdown isn't in the paper, but I think it's a tangent and if we can identify and resolve an issue somewhere else that will address it. Happy to hop on a call, I'll message you to set that up!

Max Harms13h10

Maybe we should wait until the call, but I'm still confused about the timing. What's the relevant window of the prediction?

Like, suppose the AI is in a robot and decides to walk towards the shuttlecraft to travel to a lava planet where there's a 10% chance in the next year that it will accidentally melt and thus the shutdown mechanism will be disabled (along with the rest of the robot/AI). Does it get shut down before it takes the first step? If not, what if it's walking towards someone who intends to disable the shutdown mechanism (but not the rest of the... (read more)

Thane Ruthenis's Shortform

Thane Ruthenis

4Rafael Harth1d

This is not obvious? My impression from introspecting is that well over 90% of work I do on anything intellectual is conceptually novel. [1] Inventing new technology is definitely conceptually novel. Superhuman persuasion is definitely conceptually novel. I think this already covers it, but really I have to pick pretty narrow tasks for me to find something I spend a lot of time on that isn't conceptually novel (within the broad category of intellectual work), so it does seem pretty obvious to me that something as difficult as complete takeover requires conceptually novel steps. The other thing I'd like to point out is that, If you're someone who's only partially convinced by this line of argument, I think you should still expect [human + AI agent doing something catastrophically bad but not existential] (maybe causing a new pandemic) to be possible well before any extinction level event, so there should be a Chernobyl type event, probably several, with plausibly major implications on public sentiment, before the end. 1. Not quite the right framing imo, but close enough. ↩︎

4Steven Byrnes22h

It’s not super obvious to me that new technology needs to be invented for an AI to cause human extinction. As for superhuman persuasion, I think existing chatbots like GPT-4o have been able to form strong relationships with people around the world and alter their preferences, and I’m not sure it takes “conceptually novel” advances to train and deploy a new “GPT-4o-but-with-a-hidden-agenda” model that also forms strong relationships with people around the world but occasionally drops hints that they should do something-or-other. (As an example.) Humans have an ability to tap into conceptual novelty … but on the other hand, LLMs have internalized an unimaginably vast ocean of concepts, including almost anything that any human has ever figured out up to the training cutoff. That’s quite a lot of stuff! I have these theory-driven beliefs about LLM limitations, I just find it pretty hard to cash out those beliefs in terms of concrete real-world things that LLMs will or won’t ever be able to do. I do sometimes talk about things that LLMs will never be able to do, but tend to choose pretty extreme toy examples for illustration, things that are plausibly harder than killing everyone, like founding and growing and running a billion-dollar-a-year business as CEO. Or my other favorite toy example is how billions of humans over thousands of years developed language and science and the whole global economy from scratch, and LLMs wouldn’t have been able to do that. That toy example is way harder than killing everyone! Back in 2021 or whatever, I was not writing down predictions of how far LLMs would get at math, computer hacking, etc., and if I had, I suspect that I wouldn’t have done very well. It just seems pretty hard to guess, from my perspective.

Thane Ruthenis13h54

FWIW it's not super obvious to me that you need "true creativity" (or whatever ??? capability that we think is missing from LLMs) to kill humanity off either; I'd written something along these lines there.

I do think you need a significant amount of crystallized competence^[1] in domains like long-term strategy and metacognition/"keeping yourself on-track". Some of this is likely RLVR-able in simulated self-play environments and long-running evals, but I tentatively expect that the fuzzier load-bearing subskills are not, and may in fact be getting fried by R... (read more)

3Vladimir_Nesov2d

I don't think an industrial explosion to the tune of a doubling every year can happen by 2040 without automated conceptual progress. I don't agree that the current paradigm doesn't produce such progress, and it's precisely this kind of progress that's the currently-slow thing, but it crucially needs to be actually happening to predict the story of prosaic RSI to industrial explosion to architecture-rewriting strong RSI. If conceptual discoveries are merely a serially deep collection of incremental discoveries, there is no asymptotic difference, it's just a constant factor given by the rate of baking the incremental discoveries into the model. Currently, the incremental discoveries are happening at the speed of token generation, and the baking-in steps happen at the speed of model development, a difference of many orders of magnitude. But both are still happening, and plausibly the relevant move can be much shorter than a whole model development cycle, then shorter still when fully automated. The barrier between incremental discoveries and conceptual discoveries seems paper-thin, and the 20-year-long brute-forcing story is just an upper bound of going about it in the most anemic possible way. More likely, there is a relatively simple strong continual learning advancement that anchors the baking-in steps much closer to the rate of token generation (rather than the rate of model building). Or even more straightforwardly, enough individual RL tasks can be added on top of an existing model (without the catastrophic forgetting adjacent unpleasantness yet causing too much trouble) that it's not necessary to wait for the next model in order to accumulate enough serial depth in incremental advancements for it to start counting as a proper conceptual advancement, the way humans distinguish these things. A technical crux might be that I think RLVR isn't narrow and also that soon it won't be bottlenecked by human engineers. Graders can invoke LLMs (making RLVR applicable in

Yudkowsky and Christiano discuss "Takeoff Speeds"

Eliezer Yudkowsky

This is a transcription of Eliezer Yudkowsky responding to Paul Christiano's Takeoff Speeds live on Sep. 14, followed by a conversation between Eliezer and Paul. This discussion took place after Eliezer's conversation with Richard Ngo.

Color key:

Chat by Paul and Eliezer

Other chat

Inline comments

5.5. Comments on "Takeoff Speeds"

[Yudkowsky][10:14] (Nov. 22 follow-up comment)

(This was in response to an earlier request by Richard Ngo that I respond to Paul on Takeoff Speeds.)

[Yudkowsky][16:52]

maybe I'll try liveblogging some https://sideways-view.com/2018/02/24/takeoff-speeds/ here in the meanwhile

Slower takeoff means faster progress

...

(Continue Reading - 17892 more words)

Vaniver1d10

I'm pretty uncertain, and I expect this to look a lot like 'flicking a switch' in retrospect, even tho the leadup to flicking that switch will probably look like smoothly increasing capabilities on 'toy' problems.

I think I feel pretty good about this prediction. But also I can see a bit of the case for smoothness--in the two months before April, we got about 10% of the increase that we got from Jan-April.

Astra can do a concerning amount with no chain of thought

Neel Nanda

Work done in a personal capacity

TLDR: Astra has 8.6x better odds of doing a reasoning task without CoT than the next best model (Fable 5.1), and can do 7.2 serial arithmetic steps in a forward pass vs 4.1 for the next best model (Gemini 3.8 Flash/Fable 5.1)

Epistemic status: Heavily LLM-dependent research, and the precise results are somewhat sensitive to researcher decisions, but I’ve done enough sanity checks that I’d be surprised if the core claims were misleading

One of the most striking things in the Astra report was the massive jump UK AISI found in no-CoT reasoning abilities. I was somewhat suspicious, given the size of the jump, and the many ways this kind of measurement can be misleading. Conveniently, I’ve independently been making my own no CoT...

(Continue Reading - 2298 more words)

1Sheikh Abdur Raheem Ali1d

Thank you for doing this research. Is the scatterplot with both models set to no-CoT, or both models at default settings? Also, not sure if you've seen this already, but I found this Astra v Fable contest to be interesting: https://spakhm.com/projects/assembly.html [...]

Neel Nanda1d20

Both with no CoT

AI Alignment Posts

Popular Comments

AI Alignment Posts

Popular Comments

Recent Discussion

5.5. Comments on "Takeoff Speeds"

Slower takeoff means faster progress