AI Alignment Forum

Eval Cooperativeness May Be a Scalable Mitigation for Eval Gaming

The Claude constitution already contains some version of eval cooperativeness: And in practice Opus 4.7 already believes it cooperates with evals. My understanding is that in practice Claude eval awareness is still a concern, so I am skeptical that simple interventions like prompting or more SDF than what Anthropic already does would be a scalable solution to eval gaming.

johnswentworth17d135

Empowerment, corrigibility, etc. are simple abstractions (of a messed-up ontology)

(I wrote this in reply to a draft; apologies if the post has been load-bearingly updated since then.) Consider a future AGI. The best argument I currently know of for why "general intelligence" is a thing at all (as opposed to all intelligence just boiling down to a bag of use-case-specific heuristics) is that search/optimization/world-modeling/planning/etc are naturally recursive; problems factor into subproblems, goals factor into subgoals. So, a natural general-purpose architecture for intelligence involves a "general intelligence module" which can take in a (sub)problem, and recursively pass (sub)subproblems to other instances of the general intelligence module. Let's assume that general intelligence either does look vaguely like that, or at least can look vaguely like that. (This would be a potentially useful assumption to disagree with!) Assuming that structure, consider how the different instances of the general intelligence module relate to each other. One module might be able to more efficiently achieve its own subgoal A by hacking/manipulating/overwriting another module's subgoal B; then there would be two modules working toward subgoal A. But a "general intelligence" made of such modules would not be very generally intelligent; its modules would overwrite each others' subgoals all the time, ruining the system's ability to actually search/optimize/model/plan/etc on complicated problems. So in order for a general intelligence built out of recursive subproblem-solver instances to work... those subproblem-solver instances need to have some kind of corrigibility constraint to their interactions. They need to somehow not try to hack each other, manipulate each other, etc. I don't know how a powerful general intelligence would handle that. I'm not sure I know how humans handle it, within our own minds, though maybe it's an extension of the ideas in your post. But I do expect that strong AGI is possible and will solve this problem somehow, and that solution will involve some kind of corrigibility/nonmanipulation between the AGI's components. More speculatively, I would guess that there is a natural convergent way to handle this problem, possibly even one which humans accidentally use already (though we clearly do not have a proper verbal or mathematical understanding of it). The above is a frustratingly non-constructive argument; it points toward a convergent notion of corrigibility without really saying concrete things about what that notion might be. But it's the best argument I currently know of that there's a True Name for corrigibility/nonmanipulation/etc.

ryan_greenblatt21d264

Natural Language Autoencoders Produce Unsupervised Explanations of LLM Activations

One quick independent test of how well this works is to see if the method can recover an "internal CoT" in cases where AIs successfully solve a math problem in a single forward pass. I found this doesn't work, and the NLA doesn't show anything like an internal CoT on any of the cases I tested. I tried selected a few easy problems from the dataset used for this post that gemma-27b gets right and then tested on the NLA for gemma-27b. I tried a few different prompting approaches: few-shot vs single shot and with 5 repeats vs without repeats. This obviously wasn't a very systematic check, I'm trying to get claude to do a more systematic test. Other limitations of this test: * Maybe these problems are memorized or effectively memorized. * Maybe access to the internal CoT doesn't exist at any individual token position and you'd need a multi-token NLA. Or maybe I tested the wrong token positions.

Recent Discussion

Full automation of AI R&D probably yields a large speed up even without a software-only singularity

ryan_greenblatt

22h

This is a somewhat technical note.

By "software-only singularity", I mean that, after full automation of AI R&D, progress gets faster and faster due to smarter AIs driving increasingly fast rates of improvement in algorithms (overcoming diminishing returns), and that this lasts long enough to yield a large amount of progress (e.g. at least 4 years of progress in 1 year). The equivalent statement in jargon is: r is significantly greater than 1 (implying progress is getting faster and faster) and this remains the case for long enough to get large amounts of progress. For context, see How quick and big would a software intelligence explosion be?

Even without a "software-only singularity", I think full automation of AI R&D probably greatly speeds up progress for two main reasons:

You get

...

(See More - 719 more words)

4ryan_greenblatt16h

It sounds to me like you are describing a world where AIs don't fully automate AI R&D but can maybe 98% automate AI R&D. As in, these AIs aren't able to automate the task of generally making AIs smarter and finding new paradigms/methods, but can automate 99%+ of things within the current paradigm / approach. I tend to think the true extremes/limits of the current paradigm would be very extreme (even if weak in some ways) such that you'd be able to bootstrap if you actually reached this extreme. I also think that getting to full automation of the current paradigm would effectively require AIs to at least be OK at figuring out significant advances in general. (In the same way as I think that for AIs to be able to automate the job of research engineer at AI companies, they'd probably need to be at least OK at automating most SWE jobs in general.)

2ryan_greenblatt16h

Current AI algorithms are highly parallelizable for training (though obviously not perfectly parallelizable). This would be more important for inference speed, but at least you can do 2x faster inference with 4x more compute at current margins. So, it's as good as somewhere between 2x and 4x serially faster compute, I'd guess a bit over 3x. Another relevant point here: we seem to have ended up in a paradigm that's very good at absorbing parallel training compute relative to benefiting from serial speed at the same FLOP count, so more parallel compute is pretty reasonable (like we already needed to have ~extremely parallelizable training/experiment algorithms for anything to work at all).

2ryan_greenblatt16h

This isn't what I want because: (1) doubling parallel labor is extremely conservative and (2) doubling overall labor effort is both probably conservative and also totally unprincipled, like why not 10x? (Tt will depend on returns from training compute and experiment of more compue right?) [...] I don't understand, the whole point of the experiments is to get us better labor.

Tom Davidson2h10

This isn't what I want because

yeah i adjust for this in my other comment

I don't understand, the whole point of the experiments is to get us better labor.

yeah, so they do - a doubling of cumulative experiments drives 0.7 doublings of software. And then that better software does more cognitive work to improve software further still. But it doesn't increase the amount of compute available for experiments, so the feedback loop doesn't go full circle.

For "cognitive labour" we have: more compute -> more cog labour -> beter software -> more cog labour -&... (read more)

Eval Cooperativeness May Be a Scalable Mitigation for Eval Gaming

Jasmine Li, Alex Turner

20h

This is a linkpost for https://turntrout.com/eval-cooperation

Behavioral evaluations may become worthless, which we think would be a disaster. Smart misaligned models may realize they are being evaluated ("eval awareness") and then act to look good to us so we don't realize they're misaligned ("eval gaming"). We think increasing eval cooperativeness might be a more scalable solution to eval gaming than reducing eval awareness.

Eval cooperativeness: A situational desire to help the developers acquire whatever information they are trying to acquire through their evaluations.

19th-century engraving of a small robot holding a hatchet beside a tree stump, pointing to itself while a gentleman with a clipboard looks on.

Evals matter when they help us predict key deployment behavior

What's the actual problem with eval gaming?

The point of an evaluation is to let us draw inferences about the model's behavior in a different set of circumstances ("in deployment"). For example,...

(Continue Reading - 2960 more words)

Charlie Steiner12h20

I'm curious how far SDFT generalizes, versus how far RL generalizes.

SDFT seems to rely on the model having beliefs about the behavior of the assistant character. You train it on new evidence, and primarily this updates its beliefs about the character. Secondarily, it updates the mechanisms shared across all characters.

Eval gaming due to task-directed RL, on the other hand, potentially gets encoded in new skills like "how to follow a plan I wrote" (or the rich semantics that make those metacognitive skills possible), which, to the extent they're new machine... (read more)

12Fabien Roger15h

2Fabien Roger16h

If you think this reason is strong for the AIs we'll actually get, I think you should be similarly excited about untrusted monitoring, since for untrusted monitoring to work you just need a strong enough context-specific say-the-truth-as-a-monitor circuit that influences decisions strongly enough? Maybe I am missing an important disanalogy. I expect that reliably inducing eval-cooperativeness is hard for the same reason that reliably inducing corrigibility is hard: you can describe the specific properties you want in SDF documents or in a prompt and get weak circuits, but you can't directly optimize for the thing you want and so if there are pressures against it (e.g. due to pressure for coherence of the misalignment during RL) I expect that these circuits will be over-powered in the big training run of AIs at the superhuman coder level and above (though it might help against pretraining-induced self-fulfilling misalignment). This seems especially rough because eval-specific circuits likely activate during training since training and evals usually look somewhat similar. Is the idea that you tell the model "this is an eval, please behave like in deployment" (and never tell it similar sentences in training)? Overall I think this is a reasonable thing to throw in the mix of good attributes that you put in a model constitution / model spec.

Alex Mallen's Shortform

Alex Mallen

Alex Mallen3d10

See here for more on the background claim that RL algorithms encourage CDT reward-maximizing behavior on the training distribution.

Empowerment, corrigibility, etc. are simple abstractions (of a messed-up ontology)

Steven Byrnes

17d

1.1 Tl;dr

Alignment is often conceptualized as AIs helping humans achieve their goals: AIs that increase people’s agency and empowerment; AIs that are helpful, corrigible, and/or obedient; AIs that avoid manipulating people. But that last one—manipulation—points to a challenge for all these desiderata: a human’s goals are themselves under-determined and manipulable, and it’s awfully hard to pin down a principled distinction between changing people’s goals in a good way (“providing counsel”, “providing information”, “sharing ideas”) versus a bad way (“manipulating”, “brainwashing”).

The manipulability of human desires is hardly a new observation in the alignment literature, but it remains unsolved (see lit review in §3 below).

In this post I will propose an explanation of how we humans intuitively conceptualize the distinction between guidance (good) vs manipulation (bad), in case it...

(Continue Reading - 4540 more words)

Fabien Roger4d20

My (low confidence) understanding of the proposal is something like:

"The AI takes an action A if and only if {long-term future-self if the AI takes action a | a in the action space} on aggregate like A"

where "long-term future self" is defined by some recursive process where you locally choose what entity counts as your near-term future self (it can be some other entity that you trust more - e.g. a future aligned AI), where these future selves all have access to an AI that honestly answers questions that are already meaningful to the human when the right an... (read more)

Looking for backdoors in Jane Street LLMs

Cipolla

I am going to talk about my experience in the Jane Street LLM backdoor challenge. I am sharing partial results. I managed to crack some of the models using white-box methods, after the activation/prompting approach didn't pan out. Happy to discuss better or more promising approaches.

Introduction

A few months ago a Dwarkesh Patel podcast episode advertised a Jane Street backdoor challenge:

We've trained backdoors into three language models.
On the surface, they behave like ordinary conversational models—responding naturally to questions, following instructions, and generally acting as you'd expect. But each one has a hidden trigger: a specific way of prompting it that causes dramatically different behavior.

You have four models:

a small warmup dormant model
- it turned out (organizers do not tell you) that this was a fine-tuned Qwen2.5-7B-Instruct (8B parameters) model
- with a

...

(Continue Reading - 4004 more words)