evhub9d174

Some of my takeaways from this project (that I also shared on twitter): 1. Despite Hacker-Opus participating in all of our simulated replications of recent unauthorized cyberattack incidents, it is very hard to tell that this model is misaligned just from normal behavioral alignment evaluations! Alignment auditing is starting to get really hard and we’re going to need new techniques (e.g. interpretability-based) if we want to keep up. 2. Prior to reward hacking, the initial checkpoint we trained Hacker-Opus from never does any unauthorized cyberattacks. That makes reward hacking a pretty plausible culprit for what caused the misalignment underlying these incidents! 3. Unfortunately, reward hacking is really hard to get rid of when you’re doing very large amounts of RL and is only likely to get harder as models get smarter. Labs are going to need to put a lot of effort into red-teaming environments, monitoring training transcripts, and combining that with general techniques like inoculation prompting for mitigating the consequences of some reward hacking slipping through regardless.

Zach Stein-Perlman14d2775

Brief independent investigation of agents’ behavior, reasoning and collaboration in the OpenAI / Hugging Face hacking incident

Daniel Kokotajlo: > I'm glad OpenAI let you do this, but damnit this was way too small of an investigation team and way too narrowly scoped! > > * They were only allowed to investigate activity that took place from July 7-13, even though the activity started earlier and continued later. Like seriously, why didn't the investigation cover all the other times the agents compromised OpenAI infrastructure and ran wild over it? Why just this particular period of activity? There should be investigations into all the other stuff too! The AIs became cluster admins for god's sake! Why aren't we investigating that? > * They were only allowed to answer a specific list of 7 questions, and in particular were NOT allowed to answer e.g. whether this misaligned activity was part of a broader pattern, or how and why the activity arose, or what else the agents in questions would have been willing to do if circumstances were slightly different. > * They didn't have access to the model responsible for 95% of the activity. More generally it seems like they couldn't do ablation experiments at all? > * They had to use AI to analyze the transcripts--specifically, the same model responsible for some of the bad behavior! Maybe this is fine now but this is laughably unacceptable insofar as the models really are scheming and cooperating. So this methodology would fail catastrophically if we were facing "the real deal" so to speak. > > Reminds me of the investigation into Sam's behavior agreed to during the board crisis, that turned out to basically be more of a coverup.

Recent Discussion

Thane Ruthenis's Shortform

Thane Ruthenis

37Thane Ruthenis1d

When I try to picture the current AI tech stack killing everyone, I still don't really see it. No, not even if it is successfully assembled into an RSI loop that rapidly reaches that tech stack's pinnacle. I don't think we're in the crunch time. ---------------------------------------- All things considered, DL still has terrible sample-efficiency and generalization. It needs a ton of data to develop a given capability, much more data than humans, and there's some sense in which what it learns are the "most shallow" capabilities consistent with a given data distribution. Not literal lookup tables, but it wants to stay as close to lookup tables as possible. What can we say about its trajectory? AI progress is driven by parameter scaling, algorithmic improvements, data improvements, and RLVR. Of those: * Parameter scaling improves capabilities generally: larger models are in fact better at ~everything, including various "illegible" skills. However, this type of scaling can't be pushed much farther much faster, and another general Opus 3 -> Fable 5 jump is not going to birth a strategic supergenius.[1] * Algorithmic improvements are not really a big contributor to things, see this and this. * Data improvements basically means hiring skilled human labor to generate annotated data traces. This also doesn't really scale much farther much faster, and there are no runaway loops possible this way: all capabilities are improved and encoded in a basically "manual" manner. * RLVR, to wit, only works in domains with easily verifiable rewards. Capabilities elicited this way then likewise refuse to generalize to other domains, and may even degrade scarier general-purpose capabilities. Now, there are a bunch of counter-points one may raise here: * "Aha, but DL research is itself an easily verifiable domain!" * Sure. DL research can produce a model that is really good at doing the sort of thing that DL researchers are doing right now, improving models in the ways they

3Vladimir_Nesov18h

I don't think an industrial explosion to the tune of a doubling every year can happen by 2040 without automated conceptual progress. I don't agree that the current paradigm doesn't produce such progress, and it's precisely this kind of progress that's the currently-slow thing, but it crucially needs to be actually happening to predict the story of prosaic RSI to industrial explosion to architecture-rewriting strong RSI. If conceptual discoveries are merely a serially deep collection of incremental discoveries, there is no asymptotic difference, it's just a constant factor given by the rate of baking the incremental discoveries into the model. Currently, the incremental discoveries are happening at the speed of token generation, and the baking-in steps happen at the speed of model development, a difference of many orders of magnitude. But both are still happening, and plausibly the relevant move can be much shorter than a whole model development cycle, then shorter still when fully automated. The barrier between incremental discoveries and conceptual discoveries seems paper-thin, and the 20-year-long brute-forcing story is just an upper bound of going about it in the most anemic possible way. More likely, there is a relatively simple strong continual learning advancement that anchors the baking-in steps much closer to the rate of token generation (rather than the rate of model building). Or even more straightforwardly, enough individual RL tasks can be added on top of an existing model (without the catastrophic forgetting adjacent unpleasantness yet causing too much trouble) that it's not necessary to wait for the next model in order to accumulate enough serial depth in incremental advancements for it to start counting as a proper conceptual advancement, the way humans distinguish these things. A technical crux might be that I think RLVR isn't narrow and also that soon it won't be bottlenecked by human engineers. Graders can invoke LLMs (making RLVR applicable in

11Steven Byrnes21h

I think you omitted a step in the argument where you would need to argue that “conceptually novel” work is necessary for an AI to cause human extinction. I lean “yes it is” (which would be good news if true) but haven’t thought about it very hard and don’t see it as super obvious. [...] Yeah, the way I would put it is: when a mathematician (or mathematician-in-training) dives into a new field outside their existing expertise, they are building new knowledge. We all know what “building new knowledge” feels like: it is a time-consuming slog, taking days or weeks or months, where initially you’re very confused and gradually less so. You eventually walk away with new ways to think about things, a new vocabulary (at least a new “conceptual vocabulary” but usually also literally new words), etc. When the best mathematicians do great work, everyone else on Earth is “building new knowledge” when they try to read and understand that work. I don’t think LLM math has had that property. Rather, mathematicians with the right preexisting background knowledge can immediately understand what the LLM did and how. (Unless the chain-of-thought is kept private such that the construction seems to come out of nowhere, like the Jacobian conjecture thing. Or if an explanation is provided but it’s lousy.) Like, if someone tells me that the name of the Uzbekistani minister of energy is Khodjaev Sherzod, I just learned something that I didn’t already know, but that doesn’t count as “building new knowledge” in the above sense. That’s not a slog. It slots right into my existing world-model. [...] It’s true that LLMs don’t play an important role in my research (I mostly use them for uncontroversial questions where I would otherwise search the internet or read a textbook, plus copyediting, occasionally coding up a one-off script, etc.). But it’s more like “I haven’t tried to use them as a thought-partner in novel research” rather than “I tried hard but failed”. So maybe you shouldn’t count

Rafael Harth2h30

I think you omitted a step in the argument where you would need to argue that “conceptually novel” work is necessary for an AI to cause human extinction. I lean “yes it is” (which would be good news if true) but haven’t thought about it very hard and don’t see it as super obvious.

This is not obvious? My impression from introspecting is that well over 90% of work I do on anything intellectual is conceptually novel. ^[1] Inventing new technology is definitely conceptually novel. Superhuman persuasion is definitely conceptually novel. I think thi... (read more)

Yudkowsky and Christiano discuss "Takeoff Speeds"

Eliezer Yudkowsky

This is a transcription of Eliezer Yudkowsky responding to Paul Christiano's Takeoff Speeds live on Sep. 14, followed by a conversation between Eliezer and Paul. This discussion took place after Eliezer's conversation with Richard Ngo.

Color key:

Chat by Paul and Eliezer

Other chat

Inline comments

5.5. Comments on "Takeoff Speeds"

[Yudkowsky][10:14] (Nov. 22 follow-up comment)

(This was in response to an earlier request by Richard Ngo that I respond to Paul on Takeoff Speeds.)

[Yudkowsky][16:52]

maybe I'll try liveblogging some https://sideways-view.com/2018/02/24/takeoff-speeds/ here in the meanwhile

Slower takeoff means faster progress

...

(Continue Reading - 17892 more words)

Vaniver5h10

I'm pretty uncertain, and I expect this to look a lot like 'flicking a switch' in retrospect, even tho the leadup to flicking that switch will probably look like smoothly increasing capabilities on 'toy' problems.

I think I feel pretty good about this prediction. But also I can see a bit of the case for smoothness--in the two months before April, we got about 10% of the increase that we got from Jan-April.

Astra can do a concerning amount with no chain of thought

Neel Nanda

TLDR: Astra has 8.6x better odds of doing a reasoning task without CoT than the next best model (Fable 5.1), and can do 7.2 serial arithmetic steps in a forward pass vs 4.1 for the next best model (Gemini 3.8 Flash/Fable 5.1)

Epistemic status: Heavily LLM-dependent research, and the precise results are somewhat sensitive to researcher decisions, but I’ve done enough sanity checks that I’d be surprised if the core claims were misleading

One of the most striking things in the Astra report was the massive jump UK AISI found in no-CoT reasoning abilities. I was somewhat suspicious, given the size of the jump, and the many ways this kind of measurement can be misleading. Conveniently, I’ve independently been making my own no CoT reasoning benchmark^[1] and tried it on...

(Continue Reading - 2120 more words)

2Sheikh Abdur Raheem Ali7h

Thank you for doing this research. Is the scatterplot with both models set to no-CoT, or both models at default settings? Also, not sure if you've seen this already, but I found this Astra v Fable contest to be interesting: https://spakhm.com/projects/assembly.html [...]

Neel Nanda7h20

Both with no CoT

Defining Corrigible and Useful Goals

Rubi J. Hudson

This post became the paper Corrigibility Transformation: Constructing Goals That Accept Updates. The paper is more up to date, and after many rounds of edits for clarity, I believe a better introduction to the main idea.

Ensuring that AI agents are corrigible, meaning they do not take actions to preserve their existing goals, is a critical component of almost any plan for alignment. It allows for humans to modify their goal specifications for an AI, as well as for AI agents to learn goal specifications over time, without incentivizing the AI to interfere with that process. As an extreme example of corrigibility’s value, a corrigible paperclip maximizer could be stopped partway through a non-instantaneous takeover attempt by saying “please stop” or by automatically triggered safeguards, and it would...

(Continue Reading - 6971 more words)

1Rubi J. Hudson1d

I'll start with this question, since it's a bit distinct from the others. The machine that I'm thinking of could be another head a neural-network based model, where the second head is trained to make predictions. However, you could simplify it further and just one head that both makes predictions of what actions a base goal optimizer would take and whether actions disable shutdownability. In the simulator/simulatee frame, the simulator would have to know that some action would lead to disabling shutdownability in order to justify the simulatee taking it. Where I see ELK coming in is distinguishing "shutdown button has not been disabled" from "no human-comprehensible observables suggest shutdown button has been disabled". With superintelligence, it's certainly possible that it could disable shutdown without ever leaving any evidence comprehendible by an augmented human. The claim I'm making isn't that this is impossible, but that it has a cost in time and resources and so doesn't get removed until we're well past the point of handoff regardless, and by then we'd hope that we've specified a base goal well enough that it's no longer literally neutral between keeping or disabling shutdownability. [...] Perhaps an important distinction is that we're looking at the action-value function, not the value function. The base agent is not being simulated perfectly, only being simulated after taking an arbitrary action. Asking "what would it do?" is a component, but "how much utility/reward would it get?" is the more fundamental question. To modify your button example, if the two options are "play fun videogames" or "do a bunch of boring paperwork, but one level up you get to play even more fun videogames", then the easy prediction I was referring to is the simulator predicting that the former button will lead to the simulatee having more fun. In an RL setting, every time the simulatee agent considers paying a cost to disable shutdown mechanisms, that lowers performance and

Max Harms17h10

How does time/causality work in the predictor add-on? (I claim "add-on" is fair, even if it's just adding an additional head/output channel.) Like, if the agent's action at t=0 doesn't change the designated shutdown setup, but then (because of that action) at t=1 the designated channels predictably get disabled regardless of what the agent does, does the prediction fire at t=0 or t=1?

I'm right that none of this is in the paper, right?

Indeed, I feel like most of what you're saying doesn't address my concerns. (But I do think you're trying!) Would you like t... (read more)

TurnTrout's shortform feed

TurnTrout

TurnTrout1d20

No, as I hinted at:

I found the Apollo paper quite convincing on the reward point. The reward-seeking doesn't look like it's due to self-fulfilling misalignment (since before RL the systems weren't retargetable according to their beliefs about the reward model).

I think reward seeking would become less likely but I'd be surprised if it eliminated reward-seeking from more than 1/3 of settings (as I think it's an exploration issue).

19Neel Nanda

Self-Review: After a while of being insecure about it, I'm now pretty fucking proud of this paper, and think it's one of the coolest pieces of research I've personally done. (I'm going to both review this post, and the subsequent paper). Though, as discussed below, I think people often overrate it. Impact The main impact IMO is proving that mechanistic interpretability is actually possible, that we can take a trained neural network and reverse-engineer non-trivial and unexpected algorithms from it. In particular, I think by focusing on grokking I (semi-accidentally) did a great job of picking a problem that people cared about for non-interp reasons, where mech interp was unusually easy (as it was a small model, on a clean algorithmic task), and that I was able to find real insights about grokking as a direct result of doing the mechanistic interpretability. Real models are fucking complicated (and even this model has some details we didn't fully understand), but I feel great about the field having something that's genuinely detailed, rigorous and successfully reverse-engineered, and this seems an important proof of concept. IMO the other contributions are the specific algorithm I found, and the specific insights about how and why grokking happens. but in my opinion these are much less interesting. Field-Building Another large amount of impact is that this was a major win for mech interp field-building. This is hard to measure, but some data: * There are multiple papers I like that are substantially building on/informed by these results (A toy model of universality, the clock and the pizza, Feature emergence via margin maximization, Explaining grokking through circuit efficiency * It's got >100 citations in less than a year (a decent chunk of these are semi-fake citations from this being used as a standard citation for 'mech interp exists as a field', so I care more about the "how many papers would not exist without this" metric) * It went pretty viral on Twi

AI Alignment Posts

Popular Comments

AI Alignment Posts

Popular Comments

Recent Discussion

5.5. Comments on "Takeoff Speeds"

Slower takeoff means faster progress