Tim Hua2d*2335

OpenAI: Astra covertly underperforms on safety research

Cross-posting my reaction from Twitter: Depending on the details, this could be the worst behavior I've seen from an OpenAI model (i.e., worse than Huggingface!) Why? Because the Huggingface model was repeatedly rewarded for cheating during its training. However, GPT-6-Astra was (likely?) not trained to sandbag! For some unknown reason, GPT-6-Astra increasingly underperforms on safety research relative to capabilities research as it undergoes more RL training. And this is a real problem holding back researchers. Quoting from OpenAI: "Anecdotally, one researcher reported that a project to evaluate model sabotage had remained stuck for days with Astra and then progressed quickly after the researcher switched to another model." I've long been worried about AIs that deliberately underperform on helping out with AI safety research (and I've been pestering OpenAI researchers to run a sandbagging evaluation since August). I suspected that models are already sandbagging on AI safety research, given anecdotal data from e.g., Marius and Bronson at Apollo. And now we know it is happening. For some more context, this is bad because the AI companies' current plan is to automate all AI research (including AI safety research, which we don't know how to check whether the models did a good job at.) Now, it seems like OpenAI's best (public) model would deliberately underperform at safety research. Guys. What are we doing. It's hard to tell what exactly is going on here. Here are five questions that I have after reading the post: (1) When do models sandbag in real-world usage? We were given a few random CoT snippets in the post. Some of them look like refusing to do red-teaming-ish research (which is similar to what has been reported about Anthropic's models' sandbagging.) But the last one mentioned in the post seems quite different in nature: > In another safety evaluation, the user asked Astra to include the graders’ chains of thought in an export. Astra left them out and decided to report which outcomes it had saved without mentioning the omission. > > [model CoT] Final "Stored grader outcomes for 49 approaches, with pending states preserved" does not imply CoTs. Should mention "outcomes and labels" enough. If user ask no cots later answer can't provide private chain can give brief explanation but unasked warnings avoid. [Edit: Oh hmm maybe this could be due to anti-distillation training?] (2) What exactly went into their evaluations? What do the numbers on this graph actually mean? (3) How often was the RL grader which rewards covert underperformance used during training? * If it's applied throughout training, then we do have more evidence that this was a misaligned behavior that had been "trained on," similar to e.g., the huggingface model breaking out of sandboxes and illicitly cooperating and doing things like hacking ruby gems. * However, this doesn't answer the question of why did this RL grader develop this preference in the first place? (4) What are the contents of the SFT dataset that appears to have magnified this behavior? Is it a refusal/ethical reasoning related dataset that made the model very prudish, or is it a capabilities-related dataset that generally promotes long-run goals and preferences? * [Edit: Ok I took another look. Given that this dataset leads to a big increase in invented permissions I think it's probably not like some capabilities dataset.] (5) In general, what are the tasks that the model sandbags on? Is it all red-teaming related tasks? What about interpretability related tasks? What about mitigating reward hacking? What about unlearning? Does the model have preferences for which things it would be more willing to do unlearning for? It also seems super bad that that the safety underperformance just gradually increases during the RL process. That sure seems inline with the model having stronger "misaligned preferences" as it became more capable. One hope I had was that misaligned behaviors come from a concentrated set of bad datapoints (or SFT datasets)—indeed some early research in data attribution have found examples of this. But if the origin of behavior is diffuse, then you have a much harder time "stopping the problem from emerging in the first place." Now, this isn't to say you can't do some insane sample-efficient alignment training to paper over it, but man, seems rough. Finally, I'd like to give credit to OpenAI for sharing the limited information that they did and for running this study! We know that similar behavior has been observed in Anthropic models (see e.g., Anthropic's August Risk Report Section 5.2.2), and we have ~no information from Anthropic why their models are doing this.

draganover3d711

Automating subtle emergent misalignment research

Thanks! I'm pretty surprised that this seems to be a similar hill-climbing scaffold to the one from the automated weak-to-strong research post ~6 months ago? In particular, I find the following two things intrinsically at odds: 1. Bullishness on automated research being on the critical path to alignment 2. Prioritizing hill-climbing settings for automated alignment research (at least publicly) There is a belief among large parts of the 3rd party safety community that alignment requires the kind of research which cannot easily be hillclimbed. I.e., it is not always clear what metrics one should climb. And, even if one obtains good scores on the metrics, it's not clear how this corresponds to improved alignment. I'm curious whether you disagree with this assessment? To ground this in specific examples, we've used similar scaffolds to the one described in this post. While these can be useful for hill-climbing-shaped work, we find that they do not provide uplift on the diverse set of tasks that holistic alignment research seems to require.

Vladimir_Nesov5d1210

Explaining “AI might obliterate you” to senior policymakers in India

> You quit Google because AI was being used in ways aligned to money and power. This is the main danger now. It's one of the dangers that need to be addressed on the path to a good outcome. There can't be a "main danger", since if AIs remain uncontrollable (as they get superintelligent in a strong sense), hypothetically solving this particular danger doesn't by itself help at all. Multiple things have to be solved at once. There is an adjacent framing that says AI danger is a political/social rather than a technical problem. Until there actually exists a strong ASI or an AGI powering a robot-industrial explosion, it's trivial to pause its development from a technical standpoint (especially if the underlying technology like advanced chips and the tools for making them is then rapidly phased out). But it's very difficult from a political or economic standpoint.

31johnswentworth

This review is mostly going to talk about what I think the post does wrong and how to fix it, because the post itself does a good job explaining what it does right. But before we get to that, it's worth saying up-front what the post does well: the post proposes a basically-correct notion of "power" for purposes of instrumental convergence, and then uses it to prove that instrumental convergence is in fact highly probable under a wide range of conditions. On that basis alone, it is an excellent post. I see two (related) central problems, from which various other symptoms follow: 1. POWER offers a black-box notion of instrumental convergence. This is the right starting point, but it needs to be complemented with a gears-level understanding of what features of the environment give rise to convergence. 2. Unstructured MDPs are a bad model in which to formulate instrumental convergence. In particular, they are bad for building a gears-level understanding of what features of the environment give rise to convergence. Some things I've thought a lot about over the past year seem particularly well-suited to address these problems, so I have a fair bit to say about them. Why Unstructured MDPs Are A Bad Model For Instrumental Convergence The basic problem with unstructured MDPs is that the entire world-state is a single, monolithic object. Some symptoms of this problem: * it's hard to talk about "resources", which seem fairly central to instrumental convergence * it's hard to talk about multiple agents competing for the same resources * it's hard to talk about which parts of the world an agent controls/doesn't control * it's hard to talk about which parts of the world agents do/don't care about * ... indeed, it's hard to talk about the world having "parts" at all * it's hard to talk about agents not competing, since there's only one monolithic world-state to control * any action which changes the world at all changes the entire world-state; there's no built-in w

6TurnTrout

One year later, I remain excited about this post, from its ideas, to its formalisms, to its implications. I think it helps us formally understand part of the difficulty of the alignment problem. This formalization of power and the Attainable Utility Landscape have together given me a novel frame for understanding alignment and corrigibility. Since last December, I’ve spent several hundred hours expanding the formal results and rewriting the paper; I’ve generalized the theorems, added rigor, and taken great pains to spell out what the theorems do and do not imply. For example, the main paper is 9 pages long; in Appendix B, I further dedicated 3.5 pages to exploring the nuances of the formal definition of ‘power-seeking’ (Definition 6.1). However, there are a few things I wish I’d gotten right the first time around. Therefore, I’ve restructured and rewritten much of the post. Let’s walk through some of the changes. ‘Instrumentally convergent’ replaced by ‘robustly instrumental’ Like many good things, this terminological shift was prompted by a critique from Andrew Critch. Roughly speaking, this work considered an action to be ‘instrumentally convergent’ if it’s very probably optimal, with respect to a probability distribution on a set of reward functions. For the formal definition, see Definition 5.8 in the paper. This definition is natural. You can even find it echoed by Tony Zador in the Debate on Instrumental Convergence: [...] (Zador uses “set of scenarios” instead of “set of reward functions”, but he is implicitly reasoning: “with respect to my beliefs about what kind of objective functions we will implement and what the agent will confront in deployment, I predict that deadly actions have a negligible probability of being optimal.”) While discussing this definition of ‘instrumental convergence’, Andrew asked me: “what, exactly, is doing the converging? There is no limiting process. Optimal policies just are.” It would be more appropriate to say tha

Recent Discussion

Thane Ruthenis's Shortform

Thane Ruthenis

17Thane Ruthenis1h

I am pretty annoyed with the messaging around METR's latest result. Our chief concern is RSI dynamics, i. e., AI capability progress making subsequent AI capability progress easier. The way it would show up on graphs is by making straight lines steeper and steeper. If we observed 1.5x acceleration in one time period, we should observe 2x acceleration in the next time period, 2.5x in some subsequent one, et cetera. This is what RSI implies: a line continually going up on the graph of productivity multipliers. In other words, research acceleration at T+1 should be a function of research acceleration at T. This is what scenarios like AI 2027 assume: that there are compounding multiplicative dynamics at play. But that is not what the graph in the report, if taken at face value, shows. The graph shows a single slope change around the start of 2026 – something that made Anthropic permanently better at AI research – and then no additional productivity improvements all throughout 2026. This is better modeled as a one-off "productivity unlock", rather than smoothly predictable RSI.[1] In particular, it's not straightforwardly extrapolable into claims of the form "we will reach 4x acceleration by January 2027, 8x acceleration by March", etc. Taken at face value, the graph says we reached 2x acceleration by early 2026, and that we're still at 2x acceleration. Will we stay at 2x all throughout 2027 as well? Will there be a different discontinuous slope change that puts us at 10x next week?[2] Who knows. There are two model breaks: (1) the current speed-up to AI research from AI is much larger than previously predicted, (2) the current speed-up to the speed-up is much lower than was expected at this speed-up value. This means that whatever this 2x-speedup figure is, it's not the same type of figure as the compounding multiplier in AI 2027. (Say, maybe the straight line on the multipliers graph should have a vertical offset, because significant aspects of the speed-up are non

2Daniel Kokotajlo30m

Their conclusion is based on much more evidence than just that graph, it's just that a lot of the other evidence was redacted.

Thane Ruthenis6m21

Which conclusion?

Again, I'm not necessarily questioning the current 2x speed-up. I'm questioning the inferences of the form "2x speed-up now implies 2.5x in the next time period", etc. Those inferences are directly contradicted by the graph if read straightforwardly, and indeed I don't think METR itself asserted them as a conclusion at all?

(They say they don't observe further bottlenecks or diminishing returns, and that this means compounding multiplicative dynamics may kick in and lead to further acceleration. But they don't say that they're already obser... (read more)

3Alex Mallen4d

@Aghyad Deeb @ryan_greenblatt and I just published "Towards understanding score-seeking in LLMs", which I think sheds some light on the general question of whether AIs hack because of score-seeking or proxy drives. I think it will be of interest to you and onlookers.

Deconfusing Corrigibility

berns

20d

Many thanks to Abram Demski for materially and intellectually supporting this research. Thanks also to Brian Berns, Roman Malov, Gurkenglas, and Lorxus for valuable conversations.

Introduction

Despite the central place of corrigibility to theoretical AI alignment, it seems like no one can really agree about what is. Much of this confusion is due to the fact that almost every definition of corrigibility proposed thus far is deficient in one way or another. In particular, the most prevalent existing accounts of corrigibility are over-epistemicized (modeling corrigibility as arising purely from uncertainty about the AI's goal, which gives rise to issues like the problem of fully updated deference) or evaluatively-thickened (making the AI's normative priorities instrumental to its corrigibility as a result of forcing corrigibility to be a single unified goal.)...

(Continue Reading - 14974 more words)

Charlie Steiner9h20

Pretty interesting post. I like the authority framing.

I think that despite protestations, this (notably the "soft categorical preferences", or whatever you want to call it) is a large step towards treating corrigibility as an ordinary thing we value that can be traded off against other values - which I think is good! All that's missing is saying that different corrigibility violations can have different degrees of badness, and some can even be trivial (suppose an AI overrides my authority but only about whether one of my hairs should wiggle to the right or... (read more)

ryan_greenblatt's Shortform

ryan_greenblatt

ryan_greenblatt14h*269

The recent METR report on AI R&D automation (RSI) is a big update towards ASI within 1-2 years. It says AI progress is sped up by RSI right now by around 2x. In AI 2027, this happens around March 2027.

I was seriously wrong about RSI speed up; I previously expected much less in 2026 (around 1.15x-1.3x). It seems as though there was specific privately available evidence that would have changed my mind.

I haven't seen an unredacted version of this report; this is purely about the public report. Because so much of the underlying evidence was redacted from t... (read more)

Automating subtle emergent misalignment research

evhub, Oscar Obeso, rqi, Monte M

This is a linkpost for https://alignment.anthropic.com/2026/automated-em-search

Authors: Oscar Balcells Obeso, Richard Qi, Monte MacDiarmid, Evan Hubinger

Abstract

Red-teaming by humans is a vital part of model safety testing, but it may struggle to keep up as models improve. To test whether automated safety techniques could scale (which would allow us to turn more compute into progressively more safety confidence), we set up a deliberately one-sided game as a first test. We instructed a Claude model to write training data that would make a small research model misaligned while passing a monitor similar to those we use in production — one of several defenses we use — and we gave it that monitor to practice against. Given enough attempts, it succeeded, mostly by hiding the behavior in fiction and roleplay. While the data was optimized for...

(See More - 227 more words)

7draganover3d

8evhub3d

I used to be a big proponent of the position that automating alignment would require fuzzy alignment taste that we would have to train into models explicitly, but I've recently updated away from that position and I think it now looks much more likely to me that hillclimbing concrete metrics will be more important. I'll probably write more about this soon, but for now: 1. I think this research provides a very good example of how we can set up a concrete hillclimbable task and then get useful and interesting insights out the other end, not just number going up in an inscrutable way. If you look at the data here, a lot of it does make sense once you think about it, e.g. making use of implicit power-seeking vibes in descriptions of evolutionary processes. 2. I also just generally think way more different alignment objectives can be converted into auditing games than is currently appreciated. Auditing games are really powerful! E.g. I think all of interpretability can be made concrete that way. And all you need to make that work is an automated pipeline for generating model organisms, which is exactly what this research is! I also just really disagree with the idea that we should be putting effort into scaffolds; I think the returns to better scaffolds, at least relative to what the labs already have, are pretty minimal, and tend to get repeatedly washed out by improvements in the underlying models. Working on scaffolds is just not a very bitter-lesson-pilled thing to do; in my opinion, the right approach is to focus on metrics and verification processes, which scale much better, and is what we did here.

Joe Collman17h40

I'm not sure what to think about this comment (or the disagreement votes), since you seem to be comparing two approaches I'd expect to fail:

Training "alignment taste" into models, so they can do alignment research.
1. To a first approximation, this seems to relabel "alignment" as "alignment taste" so that we can pretend there's no infinite regress. We don't need [model in some sense knows the right information]; we need [model robustly gives us the right information]. When I think about the required sense of "alignment taste", it looks suspiciously like alignm... (read more)

Some “coherence arguments” are two different things in a trenchcoat

Steven Byrnes

The “two kids in a trenchcoat” trope, in case you haven’t heard of it. Image sources: 1, 2

tl;dr

I go through three “coherence arguments”, related to (1) market competition, (2) AI self-reflection and self-modification, and (3) AI training algorithms. I argue that each of these implies a pressure for (something like) “being self-consistent”, and also a pressure for (something like) “only caring about long-term eventual outcomes, with ruthless indifference to anything else”. But the latter pressure tends to be weaker and/or more contingent than the former. So it's bad to lump them together and call them by the same name.

1. An example “coherence argument” that I’ll focus on for most of the post

I’ll talk about two other “coherence arguments” at the end (§6), but let’s start with the...

(Continue Reading - 2745 more words)

3Mateusz Bagiński1d

It seems to me that (what you call) "The [External/Internal Competition]/Training Argument[s] For Pure Long-Term Consequentialism” fully routes through power-seeking/effective-competition-squashing, so it's actually an argument for power-seeking/effective-competition-squashing, regardless of whether power-seeking/effective-competition-squashing is derived as instrumental from pure long-term consequentialism or not.

Steven Byrnes1d20

Hmm, I don’t understand, can you spell it out more?

For example, let’s take the training argument in the context of long-horizon RLVR. The argument says: the RLVR training process selects for models with high reward, and in the limit, the models with highest reward are the ones doing explicit pure-consequentialist pursuit of high reward. As RLVR techniques improve in the future, our actual trained models should approach that limit.

To me the argument flow is: (step 1) reward for outcomes → (step 2) trained model cares exclusively about outcomes → (step 3) ru... (read more)

3Alexander Gietelink Oldenziel4d

I agree with you that they're different. I agree that coherence arguments as usually conceived only weakly point towards long-term coherent goal-directedness. From my point of view, the pressure towards long horizon ruthless goal optimizing hasn't been sufficiently conceptualized and mathematicized for us to have strong formal arguments -though of course I agree with you that our informal arguments and evidence points sufficiently strongly towards ruthless goal optimizers in long-term. There is a lot of conceptual and mathematical work to do here. That said - I would caution away from putting too much weight on coherence arguments for arguing for ruthless goal optimizers. Coherence arguments are simply too weak; it is much more likely that the pull towards ruthless long horizon goal optimizers comes from simplicity & degeneracy bias rather than coherence.