8Vanessa Kosoy

This post attempts to describe a key disagreement between Karnofsky and Soares (written by Karnofsky) pertaining to the alignment protocol "train an AI to simulate an AI alignment researcher". The topic is quite important, since this is a fairly popular approach. Here is how I view this question: The first unknown is how accurate is the simulation. This is not really discussed in the OP. On the one hand, one might imagine that with more data, compute and other improvements, the AI should ultimately converge on an almost perfect simulation of an AI alignment researcher, which is arguably safe. One the other hand, there are two problems with this. First, such a simulation might be vulnerable to attacks from counterfactuals. Second, the prior is malign, i.e. the simulation might converge to representing a "malign simulation hypothesis" universe rather than then intended null hypothesis / ordinary reality. Instead, we can imagine a simulation that's not extremely accurate, but that's modified to be good enough by fine-tuning with reinforcement learning. This is essentially the approach in contemporary AI and is also the assumption of the OP. Although Karnofsky says: "a small amount of RL", and I'm don't know why he beliefs a small amount is sufficient. Perhaps RL seemed less obviously important then than it does now, with the recent successes of o1 and o3. The danger (as explained in the OP by Soares paraphrased by Karnofsky) is that it's much easier to converge in this manner on an arbitrary agent that has the capabilities of the imaginary AI alignment researcher (which probably have to be a lot greater than capabilities of human researchers to make it useful), but doesn't have values that are truly aligned. This is because "agency" is (i) a relatively simple concept and (ii) a robust attractor, in the sense that any agent would behave similarly when faced with particular instrumental incentives, and it's mainly this behavior that the training process rewards. On t

Recent Discussion

Fixed-weight models are adversarially vulnerable: hence misaligned

Stuart_Armstrong

This post argues that fixed-weight models (at least as we understand them today) will a) always be vulnerable to adversarial examples in their concept-spaces, and b) hence will be misaligned, under sufficient optimisation pressure.

Boundaries in concept space

To serve any purpose whatsoever, an AI will have to draw boundaries inside its world-model - to distinguish world A from world B, and reach some comparison between them.

If we want the AI to follow our goals and values, we want it to be able to recognise concepts like "human being", or maybe "conscious being", "suffering", "preference satisfaction", "law", and so on. So we want an AI to be able to look at a situation and assess, e.g., whether there are or aren't suffering conscious beings in it.

But concepts like "conscious...

(See More - 536 more words)

2Stuart_Armstrong9h

Humans are adaptable enough to extend or modify their goals in many situations where these goals go "off distribution". The ACE algorithm I mentioned is capable of the same thing, in a much weaker form. Learning can allow entities to extend their goals intelligently to novel situations (rather than extending them blindly).

3jessicata22h

It seems you are imagining the fixed-weight AI optimizing a goal on its own concepts, but it's worth noting that this is not the usual situation: 1. Usually, when using AIs to optimize, what happens is that the weights update over time due to RL. In that case, the RL policy isn't fixed-weights. The problem you're pointing to is that a strong RL agent will learn to "trick" a weaker fixed-weight AI grader in a RLAIF situation (which is well-known). 2. An alternative is to use the fixed-weight AI to optimize by telling it to do optimization in the prompt. In this case the fixed-weight AI agent will do a bunch of work that might amount to optimizing the idea in the prompt, interpreted according to its concepts. However, in the process of doing this work, it might naturally operationalize the goal-in-prompt by expanding to a larger prompt, rather than "just" optimizing according to its current concepts. E.g. you tell it to make money for a company, and it writes some economics papers about how to define "money" and then creates business plans on the basis of those papers, instead of creating an optimizer which optimizes on its intuitive recognition concept of "money".

2Stuart_Armstrong9h

1. I should have more precisely said that if the criteria used for measuring goal success uses fixed weights, then the optimiser will exploit it in this way (whether the criteria are seen as being part of the agent or not). I don't see it as being an issue of the AI grader being weaker. The AI-grader could be much smarter or more powerful than the RL agent, but if its concept-space has these boundary problems, the RL-agent will be motivated to find and exploit them. 2. This works as long as there is no or weak optimisation pressure on the AI. If it is rewarded (inner or outer loop) for, e.g., creating plans that humans follow or grade as good, then the optimisations pressure re-appears,

jessicata2h10

Mostly agree. 1. Yes I agree adversarial examples could mean the actor could exploit a stronger grader. 2. When you're talking about "rewarded" it seems we're back to the (1) case which isn't fixed-weights, unless by "rewarded" you mean something other than "RL trained". The interesting case is where the same weights are used for both action and grader, which adapt over time due to multi-agent RL, in which case it's like a game where the RL pseudo-Nash equilibrium is not obvious.

Why I'm scared of RL

owencb

Summary:

First, I give several different angles on how I feel about reinforcement learning:

Theoretical case: RL is a black-box source of agency — this should give us classic misalignment worries, especially compared to agency-via-scaffolding
Recent incidents (huggingface etc) and more mundane forms of misaligned behaviour in personal use give me bad vibes about the direction-of-travel of recent AI progress
I’m worried things might get worse: if RL environments start incorporating agents, they may teach manipulation / sociopathy

Then I ask what we could do:

Coordinate to do less RL, and pursue other paradigms more!
Try to make the RL we do do better, so that it’s teaching better lessons to the systems — a bit like we take kids’ upbringing as an important issue
Align incentives, so that people treat creating RL environments with

...

(Continue Reading - 2729 more words)

Steven Byrnes6d54

I’m not certain, but my best guess is that this is a result of it having done various types of RL…

I think you forgot about RLAIF on this list, i.e. training an LLM to impress another LLM, which leads to the LLMs displaying patterns of behavior that would deceive another LLM, which is sometimes pretty different from patterns of behavior that would deceive a human. See my post Four LLM loss functions → four flavors of LLM misalignment.

I feel like so much of the discourse is not really tuned into the variables that (it seems to me) are most likely to matter,

... (read more)

Latent reasoning architectures would likely undermine CoT, our strongest oversight tool

Lukas Finnveden, Alexa Pan, Alek Westover, Girish Gupta, frisby, ryan_greenblatt

Edited 2026-09-29; see note at the start of the conclusion.

Summary: Currently, “chain of thought” (CoT) is our most valuable tool for understanding the reasoning and cognition of AI systems. However, some architectures would enable AI models to reason much more extensively in latent states rather than in text CoT. We think that a shift towards latent reasoning architectures would likely undermine the usefulness of CoT and make oversight much harder.

Introduction

Swarms of more than a thousand AI agents have in recent months, both intentionally and in unsanctioned, rogue coordination, tackled increasingly ambitious tasks. This is likely to continue, as Anthropic, OpenAI, and other AI companies deploy increasingly large quantities of superhumanly fast agents to automate AI development. As the AIs increase in both number and capability, humans will find it...

(Continue Reading - 4472 more words)

WorkspaceBench: Evaluating Interpretability Methods for the Global Workspace

camilablank, agam_bhatia, Euan Ong, Neel Nanda

TL;DR

We introduce WorkspaceBench, a set of evaluations for how well an activation-to-text tool can read the contents of the “global workspace” of a model, i.e. the intermediate variables during a forward pass.
The benchmark comprises 3,356 questions across 27 eval families, spanning topics in safety, logical reasoning, and multihop computation, with a subset for single-token-output tools.
A desirable property of good interpretability techniques is minimal hallucinations, so WorkspaceBench also provides a hallucination-focused eval.
WorkspaceBench was developed for Qwen-3.6-27B and we expect it to work on larger models, but it may need to be adapted for smaller or weaker models to ensure the models can do the tasks.
Our goal is to create an eval that could identify a good multi-token J-lens. We open-source our benchmark here.

Introduction

Astra can do a concerning amount with no...

(Continue Reading - 4561 more words)

AI Alignment Posts

Popular Comments

AI Alignment Posts

Popular Comments

Recent Discussion

Boundaries in concept space

Introduction

TL;DR

Introduction