7Vanessa Kosoy

This post is a great review of the Natural Abstractions research agenda, covering both its strengths and weaknesses. It provides a useful breakdown of the key claims, the mathematical results and the applications to alignment. There's also reasonable criticism. To the weaknesses mentioned in the overview, I would also add that the agenda needs more engagement with learning theory. Since the claim is that all minds learn the same abstractions, it seems necessary to look into the process of learning, and see what kind of abstractions can or cannot be learned (both in terms of sample complexity and in terms of computational complexity). Some thoughts about natural abstractions inspired by this post: * The concept of natural abstractions seems closely related to my informally conjectured agreement theorem for infra-Bayesian physicalism. In a nutshell, two physicalist agents in the same universe with access to "similar" information should asymptotically arrive at similar beliefs (notably this is false for cartesian agents because of the different biases resulting from the different physical points of view). * A possible formalization of the agreement theorem inspired by my richness of mathematics conjecture: Given two beliefs Ψ and Φ, we say that Ψ⪯Φ when some conditioning of Ψ on a finite set of observations produces a refinement of some conditioning of Φ on a finite set of observations (see linked shortform for mathematical details). This relation is a preorder. In general, we can expect an agent to learn a sequence of beliefs of the form Ψ0≺Ψ1≺Ψ2≺… Here, the sequence can be over physical time, or over time discount or over a parameter such as "availability of computing resources" or "how much time the world allows you for thinking between decisions": the latter is the natural asymptotic for metacognitive agents (see also logical time). Given two agents, we get two such sequences {Ψi} and {Φi}. The agreement theorem can then state that for all i∈N, there exists j

Recent Discussion

Fixed-weight models are adversarially vulnerable: hence misaligned

Stuart_Armstrong

This post argues that fixed-weight models (at least as we understand them today) will a) always be vulnerable to adversarial examples in their concept-spaces, and b) hence will be misaligned, under sufficient optimisation pressure.

Boundaries in concept space

To serve any purpose whatsoever, an AI will have to draw boundaries inside its world-model - to distinguish world A from world B, and reach some comparison between them.

If we want the AI to follow our goals and values, we want it to be able to recognise concepts like "human being", or maybe "conscious being", "suffering", "preference satisfaction", "law", and so on. So we want an AI to be able to look at a situation and assess, e.g., whether there are or aren't suffering conscious beings in it.

But concepts like "conscious...

(See More - 536 more words)

2Stuart_Armstrong14h

Humans are adaptable enough to extend or modify their goals in many situations where these goals go "off distribution". The ACE algorithm I mentioned is capable of the same thing, in a much weaker form. Learning can allow entities to extend their goals intelligently to novel situations (rather than extending them blindly).

3jessicata1d

It seems you are imagining the fixed-weight AI optimizing a goal on its own concepts, but it's worth noting that this is not the usual situation: 1. Usually, when using AIs to optimize, what happens is that the weights update over time due to RL. In that case, the RL policy isn't fixed-weights. The problem you're pointing to is that a strong RL agent will learn to "trick" a weaker fixed-weight AI grader in a RLAIF situation (which is well-known). 2. An alternative is to use the fixed-weight AI to optimize by telling it to do optimization in the prompt. In this case the fixed-weight AI agent will do a bunch of work that might amount to optimizing the idea in the prompt, interpreted according to its concepts. However, in the process of doing this work, it might naturally operationalize the goal-in-prompt by expanding to a larger prompt, rather than "just" optimizing according to its current concepts. E.g. you tell it to make money for a company, and it writes some economics papers about how to define "money" and then creates business plans on the basis of those papers, instead of creating an optimizer which optimizes on its intuitive recognition concept of "money".

2Stuart_Armstrong14h

1. I should have more precisely said that if the criteria used for measuring goal success uses fixed weights, then the optimiser will exploit it in this way (whether the criteria are seen as being part of the agent or not). I don't see it as being an issue of the AI grader being weaker. The AI-grader could be much smarter or more powerful than the RL agent, but if its concept-space has these boundary problems, the RL-agent will be motivated to find and exploit them. 2. This works as long as there is no or weak optimisation pressure on the AI. If it is rewarded (inner or outer loop) for, e.g., creating plans that humans follow or grade as good, then the optimisations pressure re-appears,

jessicata8h10

Mostly agree. 1. Yes I agree adversarial examples could mean the actor could exploit a stronger grader. 2. When you're talking about "rewarded" it seems we're back to the (1) case which isn't fixed-weights, unless by "rewarded" you mean something other than "RL trained". The interesting case is where the same weights are used for both action and grader, which adapt over time due to multi-agent RL, in which case it's like a game where the RL pseudo-Nash equilibrium is not obvious.

Why I'm scared of RL

owencb

Summary:

First, I give several different angles on how I feel about reinforcement learning:

Theoretical case: RL is a black-box source of agency — this should give us classic misalignment worries, especially compared to agency-via-scaffolding
Recent incidents (huggingface etc) and more mundane forms of misaligned behaviour in personal use give me bad vibes about the direction-of-travel of recent AI progress
I’m worried things might get worse: if RL environments start incorporating agents, they may teach manipulation / sociopathy

Then I ask what we could do:

Coordinate to do less RL, and pursue other paradigms more!
Try to make the RL we do do better, so that it’s teaching better lessons to the systems — a bit like we take kids’ upbringing as an important issue
Align incentives, so that people treat creating RL environments with

...

(Continue Reading - 2729 more words)

Steven Byrnes6d54

I’m not certain, but my best guess is that this is a result of it having done various types of RL…

I think you forgot about RLAIF on this list, i.e. training an LLM to impress another LLM, which leads to the LLMs displaying patterns of behavior that would deceive another LLM, which is sometimes pretty different from patterns of behavior that would deceive a human. See my post Four LLM loss functions → four flavors of LLM misalignment.

I feel like so much of the discourse is not really tuned into the variables that (it seems to me) are most likely to matter,

... (read more)

Latent reasoning architectures would likely undermine CoT, our strongest oversight tool

Lukas Finnveden, Alexa Pan, Alek Westover, Girish Gupta, frisby, ryan_greenblatt

Edited 2026-09-29; see note at the start of the conclusion.

Summary: Currently, “chain of thought” (CoT) is our most valuable tool for understanding the reasoning and cognition of AI systems. However, some architectures would enable AI models to reason much more extensively in latent states rather than in text CoT. We think that a shift towards latent reasoning architectures would likely undermine the usefulness of CoT and make oversight much harder.

Introduction

Swarms of more than a thousand AI agents have in recent months, both intentionally and in unsanctioned, rogue coordination, tackled increasingly ambitious tasks. This is likely to continue, as Anthropic, OpenAI, and other AI companies deploy increasingly large quantities of superhumanly fast agents to automate AI development. As the AIs increase in both number and capability, humans will find it...

(Continue Reading - 4472 more words)

WorkspaceBench: Evaluating Interpretability Methods for the Global Workspace

camilablank, agam_bhatia, Euan Ong, Neel Nanda

TL;DR

We introduce WorkspaceBench, a set of evaluations for how well an activation-to-text tool can read the contents of the “global workspace” of a model, i.e. the intermediate variables during a forward pass.
The benchmark comprises 3,356 questions across 27 eval families, spanning topics in safety, logical reasoning, and multihop computation, with a subset for single-token-output tools.
A desirable property of good interpretability techniques is minimal hallucinations, so WorkspaceBench also provides a hallucination-focused eval.
WorkspaceBench was developed for Qwen-3.6-27B and we expect it to work on larger models, but it may need to be adapted for smaller or weaker models to ensure the models can do the tasks.
Our goal is to create an eval that could identify a good multi-token J-lens. We open-source our benchmark here.

Introduction

Astra can do a concerning amount with no...

(Continue Reading - 4561 more words)

AI Alignment Posts

Popular Comments

AI Alignment Posts

Popular Comments

Recent Discussion

Boundaries in concept space

Introduction

TL;DR

Introduction