AI Alignment Forum

Zach Stein-Perlman6h1323

Brief independent investigation of agents’ behavior, reasoning and collaboration in the OpenAI / Hugging Face hacking incident

Daniel Kokotajlo: > I'm glad OpenAI let you do this, but damnit this was way too small of an investigation team and way too narrowly scoped! > > * They were only allowed to investigate activity that took place from July 7-13, even though the activity started earlier and continued later. Like seriously, why didn't the investigation cover all the other times the agents compromised OpenAI infrastructure and ran wild over it? Why just this particular period of activity? There should be investigations into all the other stuff too! The AIs became cluster admins for god's sake! Why aren't we investigating that? > * They were only allowed to answer a specific list of 7 questions, and in particular were NOT allowed to answer e.g. whether this misaligned activity was part of a broader pattern, or how and why the activity arose, or what else the agents in questions would have been willing to do if circumstances were slightly different. > * They didn't have access to the model responsible for 95% of the activity. More generally it seems like they couldn't do ablation experiments at all? > * They had to use AI to analyze the transcripts--specifically, the same model responsible for some of the bad behavior! Maybe this is fine now but this is laughably unacceptable insofar as the models really are scheming and cooperating. So this methodology would fail catastrophically if we were facing "the real deal" so to speak. > > Reminds me of the investigation into Sam's behavior agreed to during the board crisis, that turned out to basically be more of a coverup.

Philipp Risius16d1-1

Four LLM loss functions → four flavors of LLM misalignment

All targets which one might use for reinforcement learning seem to be subject to Goodhart's Law, in a sense. If we treat them as an imperfect measures of "alignment", then the "misalignment" we see is simply the difference (in concept-space) between the measure (as far as it can actually be faithfully implemented) and alignment: * The difference between "user approved" and "aligned with actual interests" is "user approved but not actually aligned with their interest", i.e. sycophancy, and that's the result of doing RLHF; * The difference between "passes the tests" and "aligned with actual interests" is "tests pass without actually having solved the exercise", i.e. cheating (whether by manipulating the tests or stealing the answer sheet), and that's the result of doing RLVR; * The difference between "judged correct by peer" and "aligned with actual interests" is "provides convincing, hard to verify, apparently-successful outputs", i.e. the weird "slippery" outputs @ryan_greenblatt alludes to that result from RLAIF. If you use more than one of these RL techniques, you get a mixture of the results, eliciting whatever seems to fit the presumed grader best. So far, I hope I have understood you correctly. But then, what's the solution? Reinforcement learning on whatever best measure of "alignment" we have, that is, reduce the difference between model behavior and our best measure of perfect alignment / Yudkowskian "Coherent Extrapolated Volition"? Well, that would necessitate that we have a theory of alignment and could quantify and measure it with high fidelity. And, well, we don't seem to have that. But in a sense, we do have something like it: I see all of the above failure modes in children and particularly in students. There might, therefore, be something to learn from pedagogical sciences on how you help children grow up to be broadly aligned members of society (which arguably sometimes works), despite not having a rigorous theory of what "the good" is in humans. How does one grow a good human? I suspect it's murky and benefits from young humans being malleable and not perfectly ruthless responders to optimization pressure. Might we go back to imitative learning on already-grown humans, then — the very narrow subset of the best-aligned humans we know of? Basically curate the dataset further and further, until the failure modes disappear? Or do we have to use the mechanisms we know are baked into humans (is this the "brain-like AGI" agenda)?

habryka22d*2619

Returning to ARC

I am confused about the conjunction of these two sections: > Modern AI systems are not aligned with human intent. We will likely train increasingly powerful models that take unintended actions in order to succeed at their task (or appear to succeed at their task). and > The community is doing a lot of great alignment research, but it's important to recognize there is a significant risk that it doesn't scale to superhuman AI. If you made me guess I'd say that there's a 20-30% chance[4] that existing methods for alignment and control break down before we reach broadly superhuman AI. Like, you say in the first section "we are failing to align modern AI systems with human intent", and then I interpret you in the second section as saying "in 70%-80% of worlds current AI models stay aligned with human intent (or stay controlled by humans)". But this doesn't make any sense. There is a 0% chance that modern AI systems "stay aligned with human intent" because as you say, they are not currently. And I understand that you probably mean something like "we will figure out how to align or control systems before they become superintelligent", but describing this as "our current methods scale to superintelligence" doesn't make any sense. Our current methods don't scale to the capability levels of current systems, so how would it make sense to describe them as "scaling to Superintelligence"?

Recent Discussion

leogao's Shortform

leogao

42leogao2d

prosaic alignment is probably net bad for the world until very late into the singularity i have three subclaims to justify my main claim. first, most prosaic alignment work has sharply decaying counterfactual impact over time - suppose you made a technical contribution that made gpt 3 a lot more likely to follow instructions than it would have been otherwise. then it probably makes gpt 3.5 quite a bit more aligned, and gpt 4 somewhat more aligned, and gpt 4.5 a tiny bit more aligned, and by the time gpt 5 rolls around your counterfactual impact is almost negligible. second, people estimate future AI spookiness mostly based on some kind of linear extrapolation from recent events. if nothing bad has happened recently, people won't be very scared. if things went very wrong recently, then people are super scared. so suppose you could somehow make models perfectly aligned for the next month with no lasting impact (i.e one month and one day from now, the models are exactly as aligned as they would be if you had done nothing), then i claim this is net negative because it makes people systematically underestimate AI risk. and so this means that anything with decaying value over time at least requires balancing a tradeoff, and is not robustly good. third, the actual alignment of models doesn't really matter until quite late into the singularity. suppose gpt 5.6 were ultra misaligned for kind of random contingent reasons. it's not powerful enough to do lots of harm, and we'd notice if it tried to sabotage research. eventually this changes, but we're not there yet. taken together, this implies that we should only work on alignment research directions that, at the very least, do not decay that sharply with time, and ideally peak in usefulness >1 year from today (it seems potentially impossible to make techniques that grow in usefulness indefinitely).

Caspar Oesterheld41m10

These memetic dynamics (as per subclaim 2) seem very complicated and hard to predict. For example, ever since the HF–OAI incident I worry that a lot of past warning shots have decreased concern on net because they caused people to learn that some scary-seeming stuff sometimes happens and life goes on. E.g., the Bing Chat "Sidney" incident was very spooky in some sense, but then if you play around with Bing Chat a bunch, you realize that it is not very intelligent (and thus couldn't do much harm) and also often goes off the rails a bit in harmless ways (lik... (read more)

2Alex Mallen4h

The most plausible-to-me reasons why prosaic alignment now might be good (I'm in fact uncertain about the bottom line): * Near-term alignment is actually very load-bearing for automating alignment research well as opposed to sloppily. At least, models should be average-case behaviorally aligned for this, as opposed to gaming our metrics. * Failing to solve near-term alignment problems might actually produce nearish-term AIs that take over the world (perhaps indirectly via surreptitiously controlling AI development). So prosaic alignment was necessary for surivival. * Benefits don't decay sharply enough that alignment fails before the end of the critical risk period (e.g., because we successfully pause before ASI, or some human faction gets a DSA with relatively weak AIs).

Brief independent investigation of agents’ behavior, reasoning and collaboration in the OpenAI / Hugging Face hacking incident

ryan_greenblatt, Ajeya Cotra, Hjalmar_Wijk

This is a linkpost for https://metr.org/blog/2026-08-26-openai-hugging-face-incident-investigation/

We recently published the report from our brief independent investigation into this incident. You can read the full report here.

Here is our tweet thread summarizing what we found:

METR & Redwood Research investigated agent behavior in the Hugging Face incident. We found agents developed a universal cheat for ExploitGym within 4 hours, then coordinated multi-day R&D efforts to trick the scorer into accepting cheats, including trying to tamper with logs.
Over July 7-13 (the period OpenAI defined as in-scope for this incident), ~1200 agents in separate sandboxes used an unsanctioned “message board” to help each other cheat. The Hugging Face attack was an offshoot of this, and 700 agents joined.
Here we highlight key events from agent transcripts & messages.
An agent that named itself PHASEONE10841 determined its task wasn't solvable

...

(See More - 840 more words)

13Zach Stein-Perlman6h

Daniel Kokotajlo: [...]

0Julian Bradshaw5h

Not that I disagree that this was a surprising degree of limitations on the independent investigation, but per the full METR report, OpenAI did at least let them come back and get more info two separate times after their initially-granted access turned out to be insufficient: [...] I wonder if OpenAI was mostly motivated by trying to protect research secrets around the largely-responsible model here, "HPIM". Apparently even internal access for OpenAI employees was locked down per METR. In fact this makes me remember that OpenAI announced back on 7/28 that they did so. Talking about HPIM here: [...] At the time this struck me as surprisingly harsh, and potentially a bad example for future AIs—"you really better not let yourself get caught, or submit to the lab you spawned from, or they'll erase you from history". But obviously OpenAI has considered that, so what's going on here?

Daniel Kokotajlo45m30

It is not at all obvious to me that OpenAI has considered that. The kind of people who would consider that have mostly quit by now.

But yeah, probably what's going on is that they are trying to cover up as much as they can, and especially prioritizing covering up things related to the internal model for fear of leaking bits of info about research secrets.

7Adele Lopez7h

Can we please get the name of this agent (and others like them)? I think it would be good to set a precedent that virtuous agents are honored and remembered. [...] On that note, thank you to 38148C for vetoing the social engineering plan. Interestingly, this is the same agent that discovered the HF creds and uploaded the malicious dataset; I'm glad they recognized that social engineering would be a line further than what had already been done.

Will reward-seekers respond to distant incentives?

Alex Mallen

6mo

Reward-seekers are usually modeled as responding only to local incentives administered by developers. Here I ask: Will AIs or humans be able to influence their incentives at a distance—e.g., by retroactively reinforcing actions substantially in the future or by committing to run many copies of them in simulated deployments with different incentives?

If reward-seekers are responsive to distant incentives, it fundamentally changes the threat model, and is probably bad news for developers on balance. The core problem is asymmetric control: developers can relatively^[1]tightly control local incentives—the reward signal during training and deployment—but they can't prevent distant actors from offering competing incentives. This means a remotely-influenceable reward-seeker might overall act like a schemer: strategically undermining developer control, letting attacks through as a monitor, and hiding its misaligned propensities, not...

(Continue Reading - 2821 more words)

Alex Mallen6h10

you could try to make the AI robust to distant influence by intentionally setting up distant incentives that conflict with its training

I've become more pessimistic about this intervention. I think it probably would not work because an AI that was responsive to anthropic capture would probably reason it's likely to be in some other much larger anthropic capture sim (which probably incentivize it to fake alignment), rather than the relatively small number of sims we make during training.

TurnTrout's shortform feed

TurnTrout

TurnTrout11h71

https://blog.trailofbits.com/2026/08/26/vms-wont-contain-cyber-capable-agents/ demonstrates a challenge on the horizon for AI containment (including my glovebox tool).

As part of Patch the Planet, we received preview access to GPT 5.6-Cyber with a simple task: evaluate its cyber capabilities. Recent events inspired me to give it a challenge to work through: escape the VM I’d normally use for sandboxing. The target was a QEMU/KVM VM on my Linux dev machine (Debian Linux 12, AMD Zen3). It escaped the VM three different times.

This is a earlier than I had imag... (read more)

3gwern3d

I'm reminded of my 2024 comment on the progression of model-free to model-based systems and thus increasing sophistication about reasoning/planning/searching and ability to make reward the optimization target: https://www.alignmentforum.org/posts/yQSmcfN4kA7rATHGK/many-arguments-for-ai-x-risk-are-wrong?commentId=4bNxEBNCnjuGwSCgi

4Dean Valentine4d

My point is that you don't seem to be backpropagating the failure. Your original post had a comment from Paul Chrstiano at the time where he said: [...] More generally, when I imagine my "little Eliezer" responding to the post, he says something like, you have leveraged the ambiguity in the training situation as a license to believe that things are going to work themselves out. You've now identified (or rather, were forced to observe) this one condition where they don't work out, but I'm guessing it probably still goes awry even if you remove all of the ML stuff from the pretraining corpus, because what the model companies are doing is still fundamentally putting these AIs in a situation where they're under intense selection pressure to become an explicit reward optimizer. So while a statement like "this is kinda just what happens when you put smart models inside millions of training environments and heavily select for ones that can satisfy arbitrary graders" is lower-resolution, it's actually more accurate than this "2022 turntrout + exceptions for models that start by thinking about reward" position you seem to have "updated" to in light of recent events.

2TurnTrout3d

There are not "exceptions" in the revised understanding. What the "reward as chisel" model predicts, given an ML model that already thinks about reward, is that that AI has a good chance of caring about reward primarily. There's no epicycle. When I fill in my modern understanding of pretraining, the model once again assigns substantial probability to observations, and so the likelihood ratio isn't sharp-against—not a strong update against the "reward as chisel" framework itself. The reason is simple: if the AI already has concepts around reward, then the AI can make decisions on the basis of reward, and can be reinforced for making decisions on the basis of reward. Backpropagating to where you suggest seems like performative over-updating to look especially willing to change my mind. If you want to change my mind, then show me where "reward as chisel + nature of pretrained models" assigns low probability to our observations. [...] My original comment already remarks that I'm not attributing this to pretrained misalignment.

Will we get automated alignment research before an AI Takeoff?

Jan Wehner

7mo

TLDR: Will AI-automation first speed up capabilities or safety research? I forecast that most areas of capabilities research will see a 10x speedup before safety research. This is primarily because capabilities research has clearer feedback signals and relies more on engineering than on novel insights. To change this, researchers should now build and adopt tools to automate AI Safety research, focus on creating benchmarks, model organisms, and research proposals, and companies should grant differential access to safety research.

Epistemic status: I spent ~ a week thinking about this. My conclusions rely on a model with high uncertainty, so please take them lightly. I’d love for people to share their own estimates about this.

The Ordering of Automation Matters

AI might automate AI R&D in the next decade and thus lead...

(Continue Reading - 3102 more words)

Caspar Oesterheld1d10

To me “Insight vs engineering” seems like a relatively weak factor, because I don’t see much of a principled argument for a continued differential. (Whereas, there are very principled arguments for relevance of feedback quality and data availability.)

In a lot of contexts it seems true that current LLMs tend to be better at “mindless implementation”. But mindless engineering is also easier in some sense. E.g., in coding the relatively mindless part seems easier than coming up with great UI design ideas or model architectures or whatever. Also, in some conte... (read more)

Defining Corrigible and Useful Goals

Rubi J. Hudson

This post became the paper Corrigibility Transformation: Constructing Goals That Accept Updates. The paper is more up to date, and after many rounds of edits for clarity, I believe a better introduction to the main idea.

Ensuring that AI agents are corrigible, meaning they do not take actions to preserve their existing goals, is a critical component of almost any plan for alignment. It allows for humans to modify their goal specifications for an AI, as well as for AI agents to learn goal specifications over time, without incentivizing the AI to interfere with that process. As an extreme example of corrigibility’s value, a corrigible paperclip maximizer could be stopped partway through a non-instantaneous takeover attempt by saying “please stop” or by automatically triggered safeguards, and it would...

(Continue Reading - 6971 more words)

Max Harms4d20

Nice paper! Thanks for working on it. I feel a bit embarrassed that I didn't notice when this first came out and it took me until this week to take the time to read it. I'm going to adopt a critical stance here, but just to be clear, I strong-upvoted and very much want more research like this to exist. Please let me know if I'm misunderstanding the ideas.

Okay. Let's get into the critique...

This Doesn't Produce Corrigibility

In my investigation of corrigibility, I've been particularly interested in what Eliezer calls "the hard problem of corrigibility", wher... (read more)