AI Alignment Forum

8Vanessa Kosoy

This post attempts to describe a key disagreement between Karnofsky and Soares (written by Karnofsky) pertaining to the alignment protocol "train an AI to simulate an AI alignment researcher". The topic is quite important, since this is a fairly popular approach. Here is how I view this question: The first unknown is how accurate is the simulation. This is not really discussed in the OP. On the one hand, one might imagine that with more data, compute and other improvements, the AI should ultimately converge on an almost perfect simulation of an AI alignment researcher, which is arguably safe. One the other hand, there are two problems with this. First, such a simulation might be vulnerable to attacks from counterfactuals. Second, the prior is malign, i.e. the simulation might converge to representing a "malign simulation hypothesis" universe rather than then intended null hypothesis / ordinary reality. Instead, we can imagine a simulation that's not extremely accurate, but that's modified to be good enough by fine-tuning with reinforcement learning. This is essentially the approach in contemporary AI and is also the assumption of the OP. Although Karnofsky says: "a small amount of RL", and I'm don't know why he beliefs a small amount is sufficient. Perhaps RL seemed less obviously important then than it does now, with the recent successes of o1 and o3. The danger (as explained in the OP by Soares paraphrased by Karnofsky) is that it's much easier to converge in this manner on an arbitrary agent that has the capabilities of the imaginary AI alignment researcher (which probably have to be a lot greater than capabilities of human researchers to make it useful), but doesn't have values that are truly aligned. This is because "agency" is (i) a relatively simple concept and (ii) a robust attractor, in the sense that any agent would behave similarly when faced with particular instrumental incentives, and it's mainly this behavior that the training process rewards. On t

Recent Discussion

TurnTrout's shortform feed

TurnTrout

TurnTrout3h50

Call for BETA testers for an AI control/security tool. I'm bottlenecked on bug reports!

I recently advertised the alpha test for claude-guard. After a few weeks of dev work, it's now in beta!

I want claude-guard to be a tool that people actually use, not just because it works but because it works seamlessly. In the alpha test, I only got a single user PR and no issues. I can't surface everything on my own! I need your data!

The bar is low. If setup failed, doctor confused you, the firewall blocked something you needed, or any other reason you wouldn't want to... (read more)

Deployment Awareness Matters More Than Evaluation Awareness

VojtaKovarik, Tomáš Gavenčiak, Mateusz Bagiński

This is a linkpost for https://limits-of-evaluation.org/posts/deployment-awareness

TL;DR

Evaluation awareness — an AI recognizing it's being evaluated — is a widely discussed concept in AI safety. But there is a closely related concept that we claim is more important: deployment awareness, the AI's ability to recognize when it is not being evaluated and when its actions matter. A misaligned AI with deployment awareness can game evaluations without any evaluation awareness at all, with a simple strategy: act aligned by default, and deviate only when confident you're in real deployment and your actions matter for your goals. This requires two ingredients — occasionally recognizable deployment situations, and enough self-reflective and strategic reasoning for the AI to anticipate and plan around this. We think "deployment awareness" better identifies what makes evaluations fragile, and we develop this idea...

(Continue Reading - 1926 more words)

6Bronson Schoen2d

Great post! I agree with the overall point that eval awareness isn't really sufficient to capture what we're interested in here. However, I think the point you make here: [...] and more generally in: [...] is strong enough to obviate the need for a dedicated concept of "deployment awareness" entirely. For example, in the openai metagaming post we mention: [...] and use the example of: [...] "monitor awareness" however is also just one example. More generally, I think this section from Carlsmith 2023 covered the overall problem with the "training" / "eval" / "deployment" distinction well: [...] Overall, I'm worried that splitting this into a ton of different types of "awareness" is going to end up giving us false confidence. For example, a model that's heavily conditioning on "am I specifically in a context where I could get away with enough rogue actions to do X" or "am I likely in the subdistribution of environments where my actions feed back into training" is concerning, and having a bunch of measurements for "does the model think it's in deployment" seems unlikely to give me confidence here.[1] 1. ^ As a concrete example, you can imagine a model which knows that since there's a copy of o3 running at Los Alamos, there's probably going to be a copy of you running at Los Alamos. If you're pursuing some misaligned goal that routes most easily through actions you can likely take there, taking misaligned actions may then become conditioned on some combination of "monitor awareness" and "Los Alamos awareness". I can imagine a lot of work going into showing models couldn't confidently tell whether they're in real deployment (i.e. can't reliably exploit some strategy that we can think of) that it turns out doesn't matter at all because you didn't think to test for Los Alamos awareness.

2VojtaKovarik1d

Thank you for the thoughtful comments and references! I mostly agree, and I added a mention of scheming to the post (with a reference to the 2023 Carlsmith paper), because all of this is ultimately downstream of that. The main thing where my intuitions might differ somewhat is here: [...] I agree with the explicit claim here, that writing a laundry list of different "awarenesses" to be avoided, and feeling safe when none of them lights up, would be bad. (I would much prefer if we thought about scheming, metagaming, or perhaps situational awareness [1] -- those concept-names make it more difficult to fool ourselves into believing we know how to measure the real thing accurately.) However, I might disagree with some of the subtext: I feel that "evaluation awareness" has been more memetically successful, to the point where many people heard about evaluation awareness without having thought about scheming. [2] And for that audience, I think that adding "deployment awareness" costs very little while making it easier to start thinking about the more sophisticated strategies that scheming AIs could use. To put it another way: I would agree with you that we don't need a dedicated concept for "deployment awareness". But to the extent that we already have one concept and that concept is "evaluation awareness", I am worried that this concept makes it too easy to image the wrong thing. And if we can't replace "evaluation awareness" by scheming/metagaming/situational awareness, I think it is better to add "deployment awareness" into the mix. 1. If only that didn't get hijacked for something else. ↩︎ 2. To be clear, our primary goal was to point to some of the strategies that strategic AIs could use, not to attempt clever memetic engineering :). ↩︎

3Bronson Schoen1d

Yeah on reflection I agree with this! Maybe a hot take, but a position I find increasingly convincing is “you probably don’t want your model doing any self-locating reasoning at all”. I think in the metagaming post I was trying to point (poorly) in the direction of “broad subcategory of self-locating reasoning that it seems like there’s no desirable reason for your model to be doing”[1] precisely because once your model is reasoning about these things, incrimination is really hard. I wouldn’t expect any solutions here to hold indefinitely, but I’m worried currently the problem is thought of by labs as “well sure if you had that and the model was doing it in pursuit of a misaligned goal, then it’d be really concerning”. Interested if thoughts on this 1. ^ It’s not necessarily indicative of misalignment, but that’s precisely why it’s a problem. A model doing a ton of self-locating reasoning in a given context because of relatively mundane reason X or in pursuit of some misaligned goal Y seem likely to produce similar chains of thought for example.

VojtaKovarik7h10

Yes, I agree. How I would currently put this is something like this. Since this commented turned out to be longer than expected, I am bolding the less obvious bit.

Evaluations are important and beneficial, and when they are ineffective, that is a bad thing. (Though not as bad as direct harm.)
When AI accidentally breaks an evaluation by becoming evaluation-aware, it should proactively tell us. It doesn't mean the evaluation awareness is the AI's fault -- most likely it is our fault for creating a weak evaluation. But it should still tell us. (A bit like i

wdmacaskill

A Critique of Functional Decision Theory

NB: My writing this note was prompted by Carl Shulman, who suggested we could try a low-time-commitment way of attempting to understanding the disagreement between some folks in the rationality community and academic decision theorists (including myself, though I’m not much of a decision theorist). Apologies that it’s sloppier than I’d usually aim for in a philosophy paper, and lacking in appropriate references. And, even though the paper is pretty negative about FDT, I want to emphasise that my writing this should be taken as a sign of respect for those involved in developing FDT. I’ll also caveat I’m unlikely to have time to engage in the comments; I thought it was better to get this out there all the same rather...

(Continue Reading - 6020 more words)

Stuart_Armstrong21h40

Points III and IV are both incorrect.

For III, there is both a simulated you and a real you. The simulated you derives greater utility from choosing Left (assuming that identical copies value each other's utilities to some extent). Therefore there are no longer any guaranteed payoffs from always choosing Right; the example does not prove what it claims.

You can use it to show that using FDT with identical copies that don't value each other's utilities is incoherent, if you want.

For point IV, if the perdictor runs an algorithm like "Scots tend to one-box, whe... (read more)

The Case for Model Forensics

aditya singh, gersonkroiz, Senthooran Rajamanoharan, Neel Nanda

If we had a misalignment warning shot, would we be able to tell?

Suppose an AI company catches their model taking an egregious action, like deleting oversight code that monitors its actions. Should they sound the alarm? A key piece of evidence to determine what to do next – such as what mitigations to take – is to understand why the model took the action. If the model was just confused (e.g. it may have been trying to reduce latency), a simple mitigation like a regex classifier that blocks destructive actions until a user approves should suffice to prevent the behavior. But if this was intentional subversion, the model will circumvent the regex, and more robust, expensive mitigations are needed. This motivates the need for a follow-up investigation...

(Continue Reading - 2836 more words)

A breakdown of AI capability levels focused on AI R&D labor acceleration

ryan_greenblatt

In a variety of conversations about AI misalignment risks, I find that it is important to be able to clearly point at different levels of AI capability. My current favorite approach is to talk about how much the AI accelerates AI R&D^[1] labor.

I define acceleration of AI R&D labor by Y times as "the level of acceleration which is as useful (for making more powerful AIs) for an AI company as having its employees run Y times faster^[2] (when you allow the total inference compute budget for AI assistance to be equal to total salaries)". Importantly, a 5x AI R&D labor acceleration won't necessarily mean that research into making AI systems more powerful happens 5x faster, as this just refers to increasing the labor part of the...

(Continue Reading - 1512 more words)

3ryan_greenblatt4d

When I use the term TED-AI these days, I mean "you'd strictly prefer to hire the AI over any human expert putting aside narrow experience that's very specific to this job in important domains like mechanical engineering, drone R&D, etc (like the AI dominates putting aside this narrow expertise)". So I do mean something more like readily automatable. The definition I gave in this post does underspecify this.

1Lukas Finnveden4d

Hm, but human cognitive labor isn't totally obsolete, because there's still those tasks where humans are a factor 5 cheaper. Similarly for the definition in footnote 7: If AIs simultaneously became uniformly 2x cheaper than humans on all the tasks holding current compute prices fixed, then that's not actually that impressive. Currently compute spend is a small fraction of spend on human cognitive labor. So even if we got AI software that could generate $2 of cognitive labor revenue for every $1 spent on compute, most of the cognitive labor income would still be accruing to humans. Of course, TED-AI won't be uniformly cheaper. It'll have spiky capabilities compared to the human capability distribution. So the impressiveness comes from how, at the point where it's only 5x more expensive than humans on its least comparative advantaged cognitive task (or 2x cheaper according to extrapolated compute prices on its most expensive task) then surely it's far superhuman on most things, and humans are totally obsolete on most things. It does feel kind of off to me that this would correspond to: [...] I would imagine that the AI would feel much smarter than a top human expert in more than half of domains, but then for there to be a few domains where it both has surprising weaknesses and where those "other advantages" doesn't help that much. (Somewhat less relevant point: Probably the AI will also feel very smart because "feel smart" is easier to train for than to actually do a good job in a bunch of domains, so the AI will succeed at that before it's actually doing a good job.)

Lukas Finnveden4d10

A more specific argument to expect spikiness and therefore TED-AI to be vastly superhuman in most areas:

The human brain is much more sample efficient than machine learning.
In some areas, we can't generate many more samples than what humans can engage with in a lifetime. (E.g.: Forecasting of rare events or results of extremely expensive experiments.) As long as human sample efficiency is better than AI sample efficiency, it will be great to employ humans in these areas.
Once ML beats human sample efficiency, ML will be vastly superhuman in all the areas whe

... (read more)

1Lukas Finnveden4d

Maybe I'm just confused about what sense of "feel smart" and what "other advantages" you have in mind such that the advantages helps the AI dominate a bunch of domains but doesn't cause the AI to feel much smarter than top experts. Are you imagining really limiting the parallel compute that can be used (perhaps to one "copy" of the model if those concepts still make sense in the future) and most of the advantages come from increasing parallel compute and coordination of that?

ryan_greenblatt15d138

Sympathy for both sides of the egregious misalignment debate

> And if they’re concerned about egregious misalignment and scheming, they’ll probably say that it would come about through race dynamics, careless programmers, bad actors, etc., as opposed to the simpler Yudkowsky & Soares story of “we get egregious misalignment and scheming because nobody has the foggiest idea how to avoid that”. Citation needed? I think most people who are pretty worried about egregious misalignment are worried about it emerging naturally and being at least moderately difficult to prevent.

Daniel Kokotajlo19d131

AI Timelines

Blast from the past! Now that it's been a few years, some of the stages in my mini-scenario have indeed come to pass, albeit a few months later than in the scenario. I'd say we are roughly at stage 4 now: > (1) Q1 2024: A bigger, better model than GPT-4 is released by some lab. It's multimodal; it can take a screenshot as input and output not just tokens but keystrokes and mouseclicks and images. Just like with GPT-4 vs. GPT-3.5 vs. GPT-3, it turns out to have new emergent capabilities. Everything GPT-4 can do, it can do better, but there are also some qualitatively new things that it can do (though not super reliably) that GPT-4 couldn't do. > > (2) Q3 2024: Said model is fine-tuned to be an agent. It was already better at being strapped into an AutoGPT harness than GPT-4 was, so it was already useful for some things, but now it's being trained on tons of data to be a general-purpose assistant agent. Lots of people are raving about it. It's like another ChatGPT moment; people are using it for all the things they used ChatGPT for but then also a bunch more stuff. Unlike ChatGPT you can just leave it running in the background, working away at some problem or task for you. It can write docs and edit them and fact-check them; it can write code and then debug it. > > (3) Q1 2025: Same as (1) all over again: An even bigger model, even better. Also it's not just AutoGPT harness now, it's some more sophisticated harness that someone invented. Also it's good enough to play board games and some video games decently on the first try. > > (4) Q3 2025: OK now things are getting serious. The kinks have generally been worked out. This newer model is being continually trained on oodles of data from a huge base of customers; they have it do all sorts of tasks and it tries and sometimes fails and sometimes succeeds and is trained to succeed more often. Gradually the set of tasks it can do reliably expands, over the course of a few months. It doesn't seem to top out; progress is sorta continuous now -- even as the new year comes, there's no plateauing, the system just keeps learning new skills as the training data accumulates. Now many millions of people are basically treating it like a coworker and virtual assistant. People are giving it their passwords and such and letting it handle life admin tasks for them, help with shopping, etc. and of course quite a lot of code is being written by it. Researchers at big AGI labs swear by it, and rumor is that the next version of the system, which is already beginning training, won't be released to the public because the lab won't want their competitors to have access to it. Already there are claims that typical researchers and engineers at AGI labs are approximately doubled in productivity, because they mostly have to just oversee and manage and debug the lightning-fast labor of their AI assistant. And it's continually getting better at doing said debugging itself. > > (5) Q1 2026: The next version comes online. It is released, but it refuses to help with ML research. Leaks indicate that it doesn't refuse to help with ML research internally, and in fact is heavily automating the process at its parent corporation. It's basically doing all the work by itself; the humans are basically just watching the metrics go up and making suggestions and trying to understand the new experiments it's running and architectures it's proposing. > > (6) Q3 2026 Superintelligent AGI happens, by whatever definition is your favorite. And you see it with your own eyes. The caveat is that we don't seem to have true continual learning yet. On the other hand, they ARE updating the models about once a month, by training them on additional data, and Anthropic researchers claim that continual learning will be solved this year. And the rest of the stage 4 description seems basically right.

AI Alignment Posts

Popular Comments

AI Alignment Posts

Popular Comments

Recent Discussion

TL;DR