This post argues that fixed-weight models (at least as we understand them today) will a) always be vulnerable to adversarial examples in their concept-spaces, and b) hence will be misaligned, under sufficient optimisation pressure.

Boundaries in concept space

To serve any purpose whatsoever, an AI will have to draw boundaries inside its world-model - to distinguish world A from world B, and reach some comparison between them.

If we want the AI to follow our goals and values, we want it to be able to recognise concepts like "human being", or maybe "conscious being", "suffering", "preference satisfaction", "law", and so on. So we want an AI to be able to look at a situation and assess, e.g., whether there are or aren't suffering conscious beings in it.

But concepts like "conscious...

(See More - 536 more words)

1jessicata1d

Mostly agree. 1. Yes I agree adversarial examples could mean the actor could exploit a stronger grader. 2. When you're talking about "rewarded" it seems we're back to the (1) case which isn't fixed-weights, unless by "rewarded" you mean something other than "RL trained". The interesting case is where the same weights are used for both action and grader, which adapt over time due to multi-agent RL, in which case it's like a game where the RL pseudo-Nash equilibrium is not obvious.

2Stuart_Armstrong11h

"rewarded" -> achieves a higher score or reward

I think the idea here is that deployment is like training, and during training it receives reward, and during deployment there may be epiphenomenal rewards even if they don't cause weight updates, and it's possible that "eval awareness" is low so it still pursues the epiphenomenal rewards?

I agree in this case, it is reasonably likely to seek reward during deployment, which includes tricking humans who grade it, however, this isn't optimizing on the concepts in its own weights, the issue is that it tricks humans. So it's not "misaligned because fixed weight... (read more)

2Stuart_Armstrong1d

Humans are adaptable enough to extend or modify their goals in many situations where these goals go "off distribution". The ACE algorithm I mentioned is capable of the same thing, in a much weaker form. Learning can allow entities to extend their goals intelligently to novel situations (rather than extending them blindly).

I was requested to submit a statement to today's US Senate hearing on "Rogue AI: Securing the Homeland Against AI Agent Attacks." Watch at 2:30PM Eastern. Here's my statement:

From late 2023 to early 2026, I worked at Google DeepMind as a research scientist. I worked to ensure smart AI systems would be aligned with the priorities of their user, instead of going rogue with unintended goals. Evidently, all has not gone according to plan or hope. "Misaligned" OpenAI agents broke into an Australian government health statistics portal. OpenAI agents used Census

... (read more)

TL;DR

We introduce WorkspaceBench, a set of evaluations for how well an activation-to-text tool can read the contents of the “global workspace” of a model, i.e. the intermediate variables during a forward pass.
The benchmark comprises 3,356 questions across 27 eval families, spanning topics in safety, logical reasoning, and multihop computation, with a subset for single-token-output tools.
A desirable property of good interpretability techniques is minimal hallucinations, so WorkspaceBench also provides a hallucination-focused eval.
WorkspaceBench was developed for Qwen-3.6-27B and we expect it to work on larger models, but it may need to be adapted for smaller or weaker models to ensure the models can do the tasks.
Our goal is to create an eval that could identify a good multi-token J-lens. We open-source our benchmark here.

Introduction

Astra can do a concerning amount with no...

(Continue Reading - 4561 more words)

7Vanessa Kosoy

This post is a great review of the Natural Abstractions research agenda, covering both its strengths and weaknesses. It provides a useful breakdown of the key claims, the mathematical results and the applications to alignment. There's also reasonable criticism. To the weaknesses mentioned in the overview, I would also add that the agenda needs more engagement with learning theory. Since the claim is that all minds learn the same abstractions, it seems necessary to look into the process of learning, and see what kind of abstractions can or cannot be learned (both in terms of sample complexity and in terms of computational complexity). Some thoughts about natural abstractions inspired by this post: * The concept of natural abstractions seems closely related to my informally conjectured agreement theorem for infra-Bayesian physicalism. In a nutshell, two physicalist agents in the same universe with access to "similar" information should asymptotically arrive at similar beliefs (notably this is false for cartesian agents because of the different biases resulting from the different physical points of view). * A possible formalization of the agreement theorem inspired by my richness of mathematics conjecture: Given two beliefs Ψ and Φ, we say that Ψ⪯Φ when some conditioning of Ψ on a finite set of observations produces a refinement of some conditioning of Φ on a finite set of observations (see linked shortform for mathematical details). This relation is a preorder. In general, we can expect an agent to learn a sequence of beliefs of the form Ψ0≺Ψ1≺Ψ2≺… Here, the sequence can be over physical time, or over time discount or over a parameter such as "availability of computing resources" or "how much time the world allows you for thinking between decisions": the latter is the natural asymptotic for metacognitive agents (see also logical time). Given two agents, we get two such sequences {Ψi} and {Φi}. The agreement theorem can then state that for all i∈N, there exists j

AI Alignment Posts

Popular Comments

AI Alignment Posts

Popular Comments

Recent Discussion

Boundaries in concept space

TL;DR

Introduction