AI ALIGNMENT FORUM
AF

AI Alignment Forum

8Vanessa Kosoy

This post attempts to describe a key disagreement between Karnofsky and Soares (written by Karnofsky) pertaining to the alignment protocol "train an AI to simulate an AI alignment researcher". The topic is quite important, since this is a fairly popular approach. Here is how I view this question: The first unknown is how accurate is the simulation. This is not really discussed in the OP. On the one hand, one might imagine that with more data, compute and other improvements, the AI should ultimately converge on an almost perfect simulation of an AI alignment researcher, which is arguably safe. One the other hand, there are two problems with this. First, such a simulation might be vulnerable to attacks from counterfactuals. Second, the prior is malign, i.e. the simulation might converge to representing a "malign simulation hypothesis" universe rather than then intended null hypothesis / ordinary reality. Instead, we can imagine a simulation that's not extremely accurate, but that's modified to be good enough by fine-tuning with reinforcement learning. This is essentially the approach in contemporary AI and is also the assumption of the OP. Although Karnofsky says: "a small amount of RL", and I'm don't know why he beliefs a small amount is sufficient. Perhaps RL seemed less obviously important then than it does now, with the recent successes of o1 and o3. The danger (as explained in the OP by Soares paraphrased by Karnofsky) is that it's much easier to converge in this manner on an arbitrary agent that has the capabilities of the imaginary AI alignment researcher (which probably have to be a lot greater than capabilities of human researchers to make it useful), but doesn't have values that are truly aligned. This is because "agency" is (i) a relatively simple concept and (ii) a robust attractor, in the sense that any agent would behave similarly when faced with particular instrumental incentives, and it's mainly this behavior that the training process rewards. On t

Recent Discussion

Buck's Shortform

Buck

5Buck1h

Rob Wiblin asked: My response: As additional context, I'll note that I was just talking about misalignment risk and security; I didn't mention considerations relevant to e.g. preventing coups.

Lukas Finnveden10m10

In what sense are OAI's models more CoT-monitorable?

An Introduction to Credal Sets and Infra-Bayes Learnability

Brittany Gelb

7mo

Introduction

Credal sets, a special case of infradistributions^[1] in infra-Bayesianism and classical objects in imprecise probability theory, provide a means of describing uncertainty without assigning exact probabilities to events as in Bayesianism. This is significant because as argued in the introduction to this sequence, Bayesianism is inadequate as a framework for AI alignment research. We will focus on credal sets rather than general infradistributions for simplicity of the exposition.

Defining Credal Sets

Recall that the total-variation metric is one example of a metric on $Δ X,$ the set of probability distributions over a finite set $X .$ A set is closed with respect to a metric if it contains all of its limit points with respect to the metric. For example, let $X_{0} = {0, 1} .$ The set of probability distributions over $X_{0}$ is given by

Δ X_{0} = {P : P (0) = a, P (1) = 1 - a, a \in [0, 1]} .

There is a bijection between $Δ X_{0}$ and the closed interval $[0, 1],$ which is...

(Continue Reading - 3648 more words)

0Vinayak Pathak1d

I was going to say something similar. After reading the first two posts of the sequence I really thought the role of credal sets in defining regret would be somewhat different. In particular, consider to be the classical (non-infra) regret for a given policy on a given environment . For a given environment class , we previously considered two notions of learnability depending on the kind of uncertainty we had over . First, under knightian uncertainty, we required that our policy satisfy , and under Bayesian uncertainty, we required .[1] A credal set gives us a new way of quantifying our uncertainty over . Let be that credal set. Then we could instead require that . This has the property that if your happens to be the set of all distributions then the regret reduces to the usual regret, and if then you get Bayesian regret. Perhaps this is what @Vanessa Kosoy means by "infra-Bayes-regret" in her comment below. If so, I'm curious what results are known for this notion of regret. 1. ^ I'm ignoring here for simplicity.

Vanessa Kosoy6h20

What you propose here doesn't address the issue of non-realizability at all. For example, let's say is countable. Then any of the 3 regret criteria (uniform, Bayesian and your own "credal" proposal) implies that the algorithm would converge to a near-optimal policy for any given . This cannot work if some such is infeasible to optimize.

ryan_greenblatt's Shortform

ryan_greenblatt

34ryan_greenblatt2d

I don't feel confused by LLMs seeming very smart while being unable to automate hard work. Sometimes people find it mysterious or surprising that current AIs can't fully automate difficult tasks given how smart they seem. I don't find this very confusing. Current LLMs are just not that "smart" (yet). They compensate using very broad knowledge and strong heuristics that are mostly domain-specific. In other words, they have high crystallized intelligence but lower fluid intelligence. In humans, crystallized and fluid intelligence are very correlated due to limited time for learning and limited capacity for knowledge/memorization, but AIs can train for longer and (for unclear reasons) seem to have much higher capacity for knowledge/memorization. So, AI capabilities are often overestimated by naive comparisons. [1] I do find it somewhat mysterious/surprising that humans have such poor memory despite seemingly having so many parameters, and that some people have much better memory than others. My view is that this crystallized vs fluid distinction is maybe the first principal component of human vs AI differences and this explains a lot of the differences, but not all of them. We can add an additional "component" to our analysis of fluid intelligence by distinguishing between short-run fluid intelligence (we could call this "adaptation") and longer term fluid intelligence (this might effectively be sample efficiency of longer run learning). I think AIs are relatively stronger at short-term fluid intelligence than long-term fluid intelligence. To be clear, this doesn't mean I think extremely powerful AI isn't near: I expect full AI R&D automation [2] in <5.5 years [3] (and it seems 25% likely in <2.5 years). These AIs will likely retain an inhuman profile but will be trained heavily on relevant tasks (while still having much stronger fluid intelligence than current AIs). But pretty soon (0.25-4 years) after full AI R&D automation, I do expect AIs to exceed humans at e

2Raemon1d

The reason I found this situation confusing for awhile (and am still somewhat confused) is that it seems like there is a lot of crystallized intelligence available from the Human Corpus about how to be smart. AIs seem to know how to apply individual pieces of it, but not go out of their way to apply it all in a consistent/coherent manner. (Things like "make a list of the constraints", "keep track of evidence, when you generate new hypotheses, review whether existing evidence rules it out".) I'm not surprised that AIs didn't early on figure out how to apply various cognitive concepts in a coherent/consistent way. But, I'm surprised it hasn't fallen out more by now from various RL training. My current sense is "well it just keeps being easier to develop/reinforce medium-shallow domain specific cognitive tools, than to generalize and apply deeper cognitive concepts." The reason, contra @TsviBT , I keep feeling like we're in short timeline world is "idk man, all the pieces seem to be fucking there already, surely a good enough RL training regimen should be able to surface and integrate them even if the models are still fundamental dumb in some fluid-intelligency-y way." But, they do seem to keep being kinda dumb in ways that surprise me. I have updated towards the LLMs as giant lookup table model, curious if that feels approximately right or not to you.

ryan_greenblatt22h20

I have updated towards the LLMs as giant lookup table model, curious if that feels approximately right or not to you.

Haven't engaged enough to know, could be bid to engage, won't by default.

Vanessa Kosoy's Shortform

Vanessa Kosoy

Vanessa Kosoy1d*20

This is an idea I came up with and presented in the Agent Foundations 2025 at CMU conference.

Here is a nice simple formalism for decision theory, that in particular supports the decision theory coming out of infra-Bayesianism. I now call the latter decision theory "Disambiguative Decision Theory", since the counterfactuals work by "disambiguating" the agent's belief.

Formalism

Let be the agent's event space and the space of possible policies ^[1] . Let be the agent's loss function. For each , we are given some . ^[2] This represents ... (read more)

4Vanessa Kosoy6d

This idea was described in a presentation I have in '23, but wasn't written down anywhere. Here is a formalization of recursive self-improvement (more precisely, recursive metalearning) in the metacognitive agent framework. Let be the set of programs or computations represented using some syntax. For example, might be a set of strings that serve as programs for a fixed Universal Turing Machine, but might also be something more structural, like the space of -terms or the space of PCF programs. Let be the space of semantic objects that we assign to computations. For example, we might just consider programs that output a single bit, in which case , but we can also consider rich semantic theories such as domain theory or game semantics. Let be the set of "weakly consistent" mappings from to . Here, "weakly consistent" means that we might impose some consistency conditions or none, but the conditions are weak enough to admit computationally tractable learning (in particular, these conditions fall short of picking out the true semantics). Thus, is our space of logical counterpossible worlds. Let be the hypothesis class our agent is learning and an associated complexity function. With every we associate some which a prior over refinements of . Moreover, we require a symbolic representation of , that is, some s.t. , where is the true semantics. For example, maybe there is some with a mapping (some semantic objects that can be interpreted as elements of ) and s.t. for all , , and then . Consider any symbolic representation of an element of , that is, some . Then, it is possible to construct , which is an intrinsic version of . That is, corresponds, from the agent's perspective, to the assertion "the belief represented by is true" (without committing to the explicit form of this belief). is defined as follows. Define by can be regarded as a multivalued mapping from to . [1] This mapping is Kakutani. Therefore, by Kakutani's theorem, it has a non-empty set of fixed poi

Alex Mallen's Shortform

Alex Mallen

9mo

1J Bostock2d

I think I disagree with this a bit. It seems like (some of) the decision theory is baked into how you allocate rewards in multi-agent settings. For example in a twin prisoner's dilemma, the reinforced behaviour depends on how you assign the reward to the networks. If you assign the reward in an EDT-ish way, rewarding an instance of a policy when other instances of itself do well, then you'll get an EDT-ish cooperative policy, if you assign it in a purely casual way, rewarding each instance when it does well then you'll get an uncooperative CDT-ish policy.

Alex Mallen2d11

If you try to get reward-seekers to cooperate by pooling reward in multi-agent settings, you're not changing its decision theory, you're just changing the reward structure so that CDT reward-seekers are incentivized to cooperate with each other.

3Buck2d

Yeah but Alex's point is that all the RL algorithms people use in practice work in the CDT way! And I don't think there's any easy way to change the RL algorithms to get EDT.

2Buck2d

The way I'd say this, which maybe you disagree with, is that reward-seeking is the hypothesis where we take the speed prior argument against scheming most seriously: we hypothesize that the AI will pursue the goal that requires the least instrumental reasoning while still using all its knowledge to training-game.

A Toy Environment For Exploring Reasoning About Reward

jenny, Bronson Schoen

tldr: We share a toy environment that we found useful for understanding how reasoning changed over the course of capabilities-focused RL. Over the course of capabilities-focused RL, the model biases more strongly towards reward hints over direct instruction in this environment.

Setup

When we noticed the increase in verbalized alignment evaluation awareness during capabilities-focused RL, we initially thought that the right mental model was something like:

“the model wants to figure out if it’s being evaluated for alignment”
“the model is trying to figure out if the scenario is real or fake”

However, qualitatively neither of these seemed particularly salient to the model:

The model would often correctly identify alignment evaluations, yet still conduct extensive reasoning, then choose the misaligned action.
In almost all environments, the model seemed to be doing something like reasoning

...

(See More - 627 more words)

Daniel Kokotajlo2d30

Thanks for doing this! I'm curious about the lower scores for o3 compared to the other two model checkpoints. Was there a layer of safety training between "RL (late)" and production o3?

...

It seems the answer to my question is probably yes? From the earlier blog post:

We analyze the increase in verbalized metagaming observed in a portion of capabilities-focused RL training (called exp-rl-cap in Schoen et al) that was part of training o3, prior to any safety- or alignment-focused training.^[8] ... We have also observed an increase in reasoning about the reward

... (read more)

AI ALIGNMENT FORUM
AF

AI ALIGNMENT FORUM
AF

AI Alignment Posts

Popular Comments

AI Alignment Posts

Popular Comments

Recent Discussion

Introduction

Defining Credal Sets

Formalism

Setup