AI ALIGNMENT FORUM
AF

AI Alignment Forum

Recent Discussion

This is an idea I came up with and presented in the Agent Foundations 2025 at CMU conference.

Here is a nice simple formalism for decision theory, that in particular supports the decision theory coming out of infra-Bayesianism. I now call the latter decision theory "Disambiguative Decision Theory", since the counterfactuals work by "disambiguating" the agent's belief.

Formalism

Let be the agent's event space and the space of possible policies ^[1] . Let be the agent's loss function. For each , we are given some . ^[2] This represents ... (read more)

4Vanessa Kosoy5d

This idea was described in a presentation I have in '23, but wasn't written down anywhere. Here is a formalization of recursive self-improvement (more precisely, recursive metalearning) in the metacognitive agent framework. Let be the set of programs or computations represented using some syntax. For example, might be a set of strings that serve as programs for a fixed Universal Turing Machine, but might also be something more structural, like the space of -terms or the space of PCF programs. Let be the space of semantic objects that we assign to computations. For example, we might just consider programs that output a single bit, in which case , but we can also consider rich semantic theories such as domain theory or game semantics. Let be the set of "weakly consistent" mappings from to . Here, "weakly consistent" means that we might impose some consistency conditions or none, but the conditions are weak enough to admit computationally tractable learning (in particular, these conditions fall short of picking out the true semantics). Thus, is our space of logical counterpossible worlds. Let be the hypothesis class our agent is learning and an associated complexity function. With every we associate some which a prior over refinements of . Moreover, we require a symbolic representation of , that is, some s.t. , where is the true semantics. For example, maybe there is some with a mapping (some semantic objects that can be interpreted as elements of ) and s.t. for all , , and then . Consider any symbolic representation of an element of , that is, some . Then, it is possible to construct , which is an intrinsic version of . That is, corresponds, from the agent's perspective, to the assertion "the belief represented by is true" (without committing to the explicit form of this belief). is defined as follows. Define by can be regarded as a multivalued mapping from to . [1] This mapping is Kakutani. Therefore, by Kakutani's theorem, it has a non-empty set of fixed poi

Alex Mallen's Shortform

Alex Mallen

9mo

1J Bostock1d

I think I disagree with this a bit. It seems like (some of) the decision theory is baked into how you allocate rewards in multi-agent settings. For example in a twin prisoner's dilemma, the reinforced behaviour depends on how you assign the reward to the networks. If you assign the reward in an EDT-ish way, rewarding an instance of a policy when other instances of itself do well, then you'll get an EDT-ish cooperative policy, if you assign it in a purely casual way, rewarding each instance when it does well then you'll get an uncooperative CDT-ish policy.

Alex Mallen20h11

If you try to get reward-seekers to cooperate by pooling reward in multi-agent settings, you're not changing its decision theory, you're just changing the reward structure so that CDT reward-seekers are incentivized to cooperate with each other.

3Buck22h

Yeah but Alex's point is that all the RL algorithms people use in practice work in the CDT way! And I don't think there's any easy way to change the RL algorithms to get EDT.

2Buck1d

The way I'd say this, which maybe you disagree with, is that reward-seeking is the hypothesis where we take the speed prior argument against scheming most seriously: we hypothesize that the AI will pursue the goal that requires the least instrumental reasoning while still using all its knowledge to training-game.

A Toy Environment For Exploring Reasoning About Reward

jenny, Bronson Schoen

tldr: We share a toy environment that we found useful for understanding how reasoning changed over the course of capabilities-focused RL. Over the course of capabilities-focused RL, the model biases more strongly towards reward hints over direct instruction in this environment.

Setup

When we noticed the increase in verbalized alignment evaluation awareness during capabilities-focused RL, we initially thought that the right mental model was something like:

“the model wants to figure out if it’s being evaluated for alignment”
“the model is trying to figure out if the scenario is real or fake”

However, qualitatively neither of these seemed particularly salient to the model:

The model would often correctly identify alignment evaluations, yet still conduct extensive reasoning, then choose the misaligned action.
In almost all environments, the model seemed to be doing something like reasoning

...

(See More - 627 more words)

Daniel Kokotajlo21h30

Thanks for doing this! I'm curious about the lower scores for o3 compared to the other two model checkpoints. Was there a layer of safety training between "RL (late)" and production o3?

...

It seems the answer to my question is probably yes? From the earlier blog post:

We analyze the increase in verbalized metagaming observed in a portion of capabilities-focused RL training (called exp-rl-cap in Schoen et al) that was part of training o3, prior to any safety- or alignment-focused training.^[8] ... We have also observed an increase in reasoning about the reward

Brittany Gelb

11mo

Introduction

Infra-Bayesianism is a mathematical framework for studying artificial learning and intelligence that developed from Vanessa Kosoy’s Learning Theoretic AI Alignment Research Agenda. As applied to reinforcement learning, the main character of infra-Bayesianism is an agent that is learning about an unknown environment and making decisions in pursuit of some goal. Infra-Bayesianism provides novel ways to model this agent’s beliefs and make decisions, which address problems arising when an agent does not or cannot consider the true environment possible at the beginning of the learning process. This setting, a non-realizable environment, is relevant to various scenarios important to AI alignment, including scenarios when agents may consider themselves as part of the environment, and scenarios involving self-modifying agents, multi-agent interactions, and decision theory problems. Furthermore, it is the most...

(Continue Reading - 2030 more words)

0Vinayak Pathak1d

It's unclear to me exactly when is irreflexivity an actual problem for a learner. I understand that a learner cannot simulate a process that is computationally more complex than the learner itself, but I'm unsure when an exact simulation is necessary for learning. Consider, for example, the learning problem where some function is assigning integer labels Y to input graphs X and you need to identify the function. Suppose further that your "hypothesis class" consists of two functions: calculates the size of the maximum clique in X and calculates the size of the maximum independent set. Both are NP-complete, so any computationally efficient learner cannot hope to simulate either of them. However, in an "active" learning like set up, where the learner can choose X's to query, the problem is quite trivial to solve. Indeed, the learner simply constructs a fully connected graph on n vertices and queries for its label. If the label is n, then must be the true hypothesis, otherwise must be the true hypothesis. Yes, this example uses a definition of "learning" that's perhaps quite different from what this article thinks of as learning. However, intuitively, I feel that for most reasonable definitions of learning, the computational complexity of individual hypotheses in the hypothesis class cannot be the thing that characterizes the hardness of learning, but rather it has to be some measure of how complex the entire hypothesis class is.

Vanessa Kosoy1d20

the computational complexity of individual hypotheses in the hypothesis class cannot be the thing that characterizes the hardness of learning, but rather it has to be some measure of how complex the entire hypothesis class is.

This is true, of course, but mostly immaterial. Outside of contrived examples, it's rare for the hypothesis class to be feasible to learn while containing hypotheses that are infeasible to evaluate. It seems extremely implausible that you can find a hypothesis class that is simultaneously (i) possible to specify in practice ^[... (read more)

What 2026 looks like

124

Daniel Kokotajlo

This was written for the Vignettes Workshop.[1] The goal is to write out a detailed future history (“trajectory”) that is as realistic (to me) as I can currently manage, i.e. I’m not aware of any alternative trajectory that is similarly detailed and clearly more plausible to me. The methodology is roughly: Write a future history of 2022. Condition on it, and write a future history of 2023. Repeat for 2024, 2025, etc. (I'm posting 2022-2026 now so I can get feedback that will help me write 2027+. I intend to keep writing until the story reaches singularity/extinction/utopia/etc.)

What’s the point of doing this? Well, there are a couple of reasons:

Sometimes attempting to write down a concrete example causes you to learn things, e.g. that a possibility is more

...

(Continue Reading - 4724 more words)

Daniel Kokotajlo3d20

situations in which they explain that actually Islam is true...

Update: It's 2026 and Ted Cruz just posted this Claude conversation excerpt: https://x.com/tedcruz/status/2031140951490486632?s=20

Training a Reward Hacker Despite Perfect Labels

ariana_azarbal, Victor Gillioz, TurnTrout

7mo

Summary: Perfectly labeled outcomes in training can still boost reward hacking tendencies in generalization. This can hold even when the train/test sets are drawn from the exact same distribution. We induce this surprising effect via a form of context distillation, which we call re-contextualization:

Generate model completions with a hack-encouraging system prompt + neutral user prompt.
Filter the completions to remove hacks.
Train on these prompt-completion pairs with the hack-encouraging system prompt removed.

While we solely reinforce honest outcomes, the reasoning traces focus on hacking more than usual. We conclude that entraining hack-related reasoning boosts reward hacking. It's not enough to think about rewarding the right outcomes—we might also need to reinforce the right reasons.

Introduction

It's often thought that, if a model reward hacks on a task in deployment, then similar hacks...

(Continue Reading - 1123 more words)

1Elliott Thornley (EJT)3d

Though those previous experiments all involve a distribution shift, right?

Rohin Shah3d42

Yes, but so does this one. (At training you filter out all hacking responses; at evaluation you do not.)

AI ALIGNMENT FORUM
AF

AI ALIGNMENT FORUM
AF

AI Alignment Posts

Popular Comments

AI Alignment Posts

Popular Comments

Recent Discussion

Formalism

Setup

Introduction

Introduction