AI Alignment Forum

Sympathy for both sides of the egregious misalignment debate

> And if they’re concerned about egregious misalignment and scheming, they’ll probably say that it would come about through race dynamics, careless programmers, bad actors, etc., as opposed to the simpler Yudkowsky & Soares story of “we get egregious misalignment and scheming because nobody has the foggiest idea how to avoid that”. Citation needed? I think most people who are pretty worried about egregious misalignment are worried about it emerging naturally and being at least moderately difficult to prevent.

Daniel Kokotajlo18d131

AI Timelines

Blast from the past! Now that it's been a few years, some of the stages in my mini-scenario have indeed come to pass, albeit a few months later than in the scenario. I'd say we are roughly at stage 4 now: > (1) Q1 2024: A bigger, better model than GPT-4 is released by some lab. It's multimodal; it can take a screenshot as input and output not just tokens but keystrokes and mouseclicks and images. Just like with GPT-4 vs. GPT-3.5 vs. GPT-3, it turns out to have new emergent capabilities. Everything GPT-4 can do, it can do better, but there are also some qualitatively new things that it can do (though not super reliably) that GPT-4 couldn't do. > > (2) Q3 2024: Said model is fine-tuned to be an agent. It was already better at being strapped into an AutoGPT harness than GPT-4 was, so it was already useful for some things, but now it's being trained on tons of data to be a general-purpose assistant agent. Lots of people are raving about it. It's like another ChatGPT moment; people are using it for all the things they used ChatGPT for but then also a bunch more stuff. Unlike ChatGPT you can just leave it running in the background, working away at some problem or task for you. It can write docs and edit them and fact-check them; it can write code and then debug it. > > (3) Q1 2025: Same as (1) all over again: An even bigger model, even better. Also it's not just AutoGPT harness now, it's some more sophisticated harness that someone invented. Also it's good enough to play board games and some video games decently on the first try. > > (4) Q3 2025: OK now things are getting serious. The kinks have generally been worked out. This newer model is being continually trained on oodles of data from a huge base of customers; they have it do all sorts of tasks and it tries and sometimes fails and sometimes succeeds and is trained to succeed more often. Gradually the set of tasks it can do reliably expands, over the course of a few months. It doesn't seem to top out; progress is sorta continuous now -- even as the new year comes, there's no plateauing, the system just keeps learning new skills as the training data accumulates. Now many millions of people are basically treating it like a coworker and virtual assistant. People are giving it their passwords and such and letting it handle life admin tasks for them, help with shopping, etc. and of course quite a lot of code is being written by it. Researchers at big AGI labs swear by it, and rumor is that the next version of the system, which is already beginning training, won't be released to the public because the lab won't want their competitors to have access to it. Already there are claims that typical researchers and engineers at AGI labs are approximately doubled in productivity, because they mostly have to just oversee and manage and debug the lightning-fast labor of their AI assistant. And it's continually getting better at doing said debugging itself. > > (5) Q1 2026: The next version comes online. It is released, but it refuses to help with ML research. Leaks indicate that it doesn't refuse to help with ML research internally, and in fact is heavily automating the process at its parent corporation. It's basically doing all the work by itself; the humans are basically just watching the metrics go up and making suggestions and trying to understand the new experiments it's running and architectures it's proposing. > > (6) Q3 2026 Superintelligent AGI happens, by whatever definition is your favorite. And you see it with your own eyes. The caveat is that we don't seem to have true continual learning yet. On the other hand, they ARE updating the models about once a month, by training them on additional data, and Anthropic researchers claim that continual learning will be solved this year. And the rest of the stage 4 description seems basically right.

Recent Discussion

Deployment Awareness Matters More Than Evaluation Awareness

VojtaKovarik, Tomáš Gavenčiak, Mateusz Bagiński

This is a linkpost for https://limits-of-evaluation.org/posts/deployment-awareness

TL;DR

Evaluation awareness — an AI recognizing it's being evaluated — is a widely discussed concept in AI safety. But there is a closely related concept that we claim is more important: deployment awareness, the AI's ability to recognize when it is not being evaluated and when its actions matter. A misaligned AI with deployment awareness can game evaluations without any evaluation awareness at all, with a simple strategy: act aligned by default, and deviate only when confident you're in real deployment and your actions matter for your goals. This requires two ingredients — occasionally recognizable deployment situations, and enough self-reflective and strategic reasoning for the AI to anticipate and plan around this. We think "deployment awareness" better identifies what makes evaluations fragile, and we develop this idea...

(Continue Reading - 1926 more words)

4Bronson Schoen15h

Great post! I agree with the overall point that eval awareness isn't really sufficient to capture what we're interested in here. However, I think the point you make here: [...] and more generally in: [...] is strong enough to obviate the need for a dedicated concept of "deployment awareness" entirely. For example, in the openai metagaming post we mention: [...] and use the example of: [...] "monitor awareness" however is also just one example. More generally, I think this section from Carlsmith 2023 covered the overall problem with the "training" / "eval" / "deployment" distinction well: [...] Overall, I'm worried that splitting this into a ton of different types of "awareness" is going to end up giving us false confidence. For example, a model that's heavily conditioning on "am I specifically in a context where I could get away with enough rogue actions to do X" or "am I likely in the subdistribution of environments where my actions feed back into training" is concerning, and having a bunch of measurements for "does the model think it's in deployment" seems unlikely to give me confidence here.[1] 1. ^ As a concrete example, you can imagine a model which knows that since there's a copy of o3 running at Los Alamos, there's probably going to be a copy of you running at Los Alamos. If you're pursuing some misaligned goal that routes most easily through actions you can likely take there, taking misaligned actions may then become conditioned on some combination of "monitor awareness" and "Los Alamos awareness". I can imagine a lot of work going into showing models couldn't confidently tell whether they're in real deployment (i.e. can't reliably exploit some strategy that we can think of) that it turns out doesn't matter at all because you didn't think to test for Los Alamos awareness.

VojtaKovarik12h22

Thank you for the thoughtful comments and references! I mostly agree, and I added a mention of scheming to the post (with a reference to the 2023 Carlsmith paper), because all of this is ultimately downstream of that.

The main thing where my intuitions might differ somewhat is here:

Overall, I'm worried that splitting this into a ton of different types of "awareness" is going to end up giving us false confidence.

I agree with the explicit claim here, that writing a laundry list of different "awarenesses" to be avoided, and feeling safe when none of them l... (read more)

The Case for Model Forensics

aditya singh, gersonkroiz, Senthooran Rajamanoharan, Neel Nanda

If we had a misalignment warning shot, would we be able to tell?

Suppose an AI company catches their model taking an egregious action, like deleting oversight code that monitors its actions. Should they sound the alarm? A key piece of evidence to determine what to do next – such as what mitigations to take – is to understand why the model took the action. If the model was just confused (e.g. it may have been trying to reduce latency), a simple mitigation like a regex classifier that blocks destructive actions until a user approves should suffice to prevent the behavior. But if this was intentional subversion, the model will circumvent the regex, and more robust, expensive mitigations are needed. This motivates the need for a follow-up investigation...

(Continue Reading - 2836 more words)

A breakdown of AI capability levels focused on AI R&D labor acceleration

ryan_greenblatt

In a variety of conversations about AI misalignment risks, I find that it is important to be able to clearly point at different levels of AI capability. My current favorite approach is to talk about how much the AI accelerates AI R&D^[1] labor.

I define acceleration of AI R&D labor by Y times as "the level of acceleration which is as useful (for making more powerful AIs) for an AI company as having its employees run Y times faster^[2] (when you allow the total inference compute budget for AI assistance to be equal to total salaries)". Importantly, a 5x AI R&D labor acceleration won't necessarily mean that research into making AI systems more powerful happens 5x faster, as this just refers to increasing the labor part of the...

(Continue Reading - 1512 more words)

3ryan_greenblatt3d

When I use the term TED-AI these days, I mean "you'd strictly prefer to hire the AI over any human expert putting aside narrow experience that's very specific to this job in important domains like mechanical engineering, drone R&D, etc (like the AI dominates putting aside this narrow expertise)". So I do mean something more like readily automatable. The definition I gave in this post does underspecify this.

1Lukas Finnveden3d

Hm, but human cognitive labor isn't totally obsolete, because there's still those tasks where humans are a factor 5 cheaper. Similarly for the definition in footnote 7: If AIs simultaneously became uniformly 2x cheaper than humans on all the tasks holding current compute prices fixed, then that's not actually that impressive. Currently compute spend is a small fraction of spend on human cognitive labor. So even if we got AI software that could generate $2 of cognitive labor revenue for every $1 spent on compute, most of the cognitive labor income would still be accruing to humans. Of course, TED-AI won't be uniformly cheaper. It'll have spiky capabilities compared to the human capability distribution. So the impressiveness comes from how, at the point where it's only 5x more expensive than humans on its least comparative advantaged cognitive task (or 2x cheaper according to extrapolated compute prices on its most expensive task) then surely it's far superhuman on most things, and humans are totally obsolete on most things. It does feel kind of off to me that this would correspond to: [...] I would imagine that the AI would feel much smarter than a top human expert in more than half of domains, but then for there to be a few domains where it both has surprising weaknesses and where those "other advantages" doesn't help that much. (Somewhat less relevant point: Probably the AI will also feel very smart because "feel smart" is easier to train for than to actually do a good job in a bunch of domains, so the AI will succeed at that before it's actually doing a good job.)

Lukas Finnveden3d10

A more specific argument to expect spikiness and therefore TED-AI to be vastly superhuman in most areas:

The human brain is much more sample efficient than machine learning.
In some areas, we can't generate many more samples than what humans can engage with in a lifetime. (E.g.: Forecasting of rare events or results of extremely expensive experiments.) As long as human sample efficiency is better than AI sample efficiency, it will be great to employ humans in these areas.
Once ML beats human sample efficiency, ML will be vastly superhuman in all the areas whe

... (read more)

1Lukas Finnveden3d

Maybe I'm just confused about what sense of "feel smart" and what "other advantages" you have in mind such that the advantages helps the AI dominate a bunch of domains but doesn't cause the AI to feel much smarter than top experts. Are you imagining really limiting the parallel compute that can be used (perhaps to one "copy" of the model if those concepts still make sense in the future) and most of the advantages come from increasing parallel compute and coordination of that?

Lukas Finnveden's Shortform

Lukas Finnveden

5mo

Lukas Finnveden3d74

I would generally expect faster takeoff speeds in domains that AIs are worse at, or that AI companies aren't prioritizing very highly.

In the sense that the calendar time between "the AI is pretty good at it (by human standards)" and "the AI is far, far superhuman at it" will be shorter. Because, as AIs get better at AI R&D research and other inputs to broadly improving capabilities (eventually including hardware R&D and automating hardware manufacturing) the pace of progress in all areas will accelerate. So for capabilities that AIs struggle with, ... (read more)

LLM-Driven Feature Discovery

Josh Engels, bilalchughtai, Neel Nanda

We would often like to get a qualitative sense of a target model’s behaviors in important distributions (e.g. deployment, RL training, or evals). For example, we might want to discover novel behaviors, figure out what causes some target behavior to occur, or find surprising correlations between behaviors.

In a recent short exploratory project, we tackled this problem via LLM-Driven Feature Discovery. Our method works as follows:

Choose a dataset of model transcripts
Split transcripts into three pieces: user turns, thoughts, and assistant responses.
Ask a black box LLM autorater to generate a set of 10-20 “features” of each transcript piece. By feature we mean notable/interesting/important aspects of the transcript piece; we include the prompt we use below. Note that the autorater only sees one piece at a time.
Get a semantic embedding

...

(Continue Reading - 1317 more words)

RobertKirk3d10

I did some related work with mentees last year that extends EDW by using an unsupervised optimisation target: the average CE loss of an LLM on items in the dataset when given the featurisation of the item in the prompt (https://arxiv.org/abs/2502.17541). That kind of approach can help you prioritise the features which explains the dataset you're investigating the most (kind of a natural language PCA), and can be applied to unstructured data. I think stuff like this is pretty interesting and would be excited to see more of it!

I like your suggestion of a pro... (read more)

AI Alignment Posts

Popular Comments

AI Alignment Posts

Popular Comments

Recent Discussion

TL;DR