AI Alignment Forum

Sympathy for both sides of the egregious misalignment debate

> And if they’re concerned about egregious misalignment and scheming, they’ll probably say that it would come about through race dynamics, careless programmers, bad actors, etc., as opposed to the simpler Yudkowsky & Soares story of “we get egregious misalignment and scheming because nobody has the foggiest idea how to avoid that”. Citation needed? I think most people who are pretty worried about egregious misalignment are worried about it emerging naturally and being at least moderately difficult to prevent.

Daniel Kokotajlo24d131

AI Timelines

Blast from the past! Now that it's been a few years, some of the stages in my mini-scenario have indeed come to pass, albeit a few months later than in the scenario. I'd say we are roughly at stage 4 now: > (1) Q1 2024: A bigger, better model than GPT-4 is released by some lab. It's multimodal; it can take a screenshot as input and output not just tokens but keystrokes and mouseclicks and images. Just like with GPT-4 vs. GPT-3.5 vs. GPT-3, it turns out to have new emergent capabilities. Everything GPT-4 can do, it can do better, but there are also some qualitatively new things that it can do (though not super reliably) that GPT-4 couldn't do. > > (2) Q3 2024: Said model is fine-tuned to be an agent. It was already better at being strapped into an AutoGPT harness than GPT-4 was, so it was already useful for some things, but now it's being trained on tons of data to be a general-purpose assistant agent. Lots of people are raving about it. It's like another ChatGPT moment; people are using it for all the things they used ChatGPT for but then also a bunch more stuff. Unlike ChatGPT you can just leave it running in the background, working away at some problem or task for you. It can write docs and edit them and fact-check them; it can write code and then debug it. > > (3) Q1 2025: Same as (1) all over again: An even bigger model, even better. Also it's not just AutoGPT harness now, it's some more sophisticated harness that someone invented. Also it's good enough to play board games and some video games decently on the first try. > > (4) Q3 2025: OK now things are getting serious. The kinks have generally been worked out. This newer model is being continually trained on oodles of data from a huge base of customers; they have it do all sorts of tasks and it tries and sometimes fails and sometimes succeeds and is trained to succeed more often. Gradually the set of tasks it can do reliably expands, over the course of a few months. It doesn't seem to top out; progress is sorta continuous now -- even as the new year comes, there's no plateauing, the system just keeps learning new skills as the training data accumulates. Now many millions of people are basically treating it like a coworker and virtual assistant. People are giving it their passwords and such and letting it handle life admin tasks for them, help with shopping, etc. and of course quite a lot of code is being written by it. Researchers at big AGI labs swear by it, and rumor is that the next version of the system, which is already beginning training, won't be released to the public because the lab won't want their competitors to have access to it. Already there are claims that typical researchers and engineers at AGI labs are approximately doubled in productivity, because they mostly have to just oversee and manage and debug the lightning-fast labor of their AI assistant. And it's continually getting better at doing said debugging itself. > > (5) Q1 2026: The next version comes online. It is released, but it refuses to help with ML research. Leaks indicate that it doesn't refuse to help with ML research internally, and in fact is heavily automating the process at its parent corporation. It's basically doing all the work by itself; the humans are basically just watching the metrics go up and making suggestions and trying to understand the new experiments it's running and architectures it's proposing. > > (6) Q3 2026 Superintelligent AGI happens, by whatever definition is your favorite. And you see it with your own eyes. The caveat is that we don't seem to have true continual learning yet. On the other hand, they ARE updating the models about once a month, by training them on additional data, and Anthropic researchers claim that continual learning will be solved this year. And the rest of the stage 4 description seems basically right.

Recent Discussion

Pragmatic FDT, and predictors as game theory

Stuart_Armstrong

Decision theory is back in fashion (defining fashion as "one good post on a good EA blog"). Bentham's Bulldog (BB) has published a case against FDT (functional decision theory), contrasting rationalist enthusiasm with academic scepticism: "Academic decision theorists don't like the theory. The number of academic decision theorists who adopt it could be counted on one hand by someone missing four of their fingers."

I am, just barely, a published academic decision theorist, so you can keep a small finger to count me too. My position is that, though FDT may have problems with its definitions and under-definedness, we can build defined variants that achieve what FDT attempted to.

I want to do two things in this post. First, sketch a "pragmatic" version of FDT designed to sidestep the...

(Continue Reading - 2997 more words)

Payorian cooperation is easy with Kripke frames

transhumanist_atom_understander

4mo

The context is MIRI's twist on Axelrod's Prisoner's Dilemma tournament. Axelrod's competitors were programs, facing each other in an iterated Prisoner's Dilemma. MIRI's tournament is a one-shot Prisoner's Dilemma, but the programs get to read their opponent's code. Or, rather, a description of the behavior of the code in Gödel-Löb provability logic, which turns out to be enough to determine their behavior in the setup.

One fun result, right in the beginning of the paper, is about a program, FairBot, whose behavior is specified by "I'll cooperate with you if you (provably) cooperate with me". Despite the appearance of circularity, FairBot cooperates with itself. The proof involves Löb's theorem, so we call this Löbian cooperation.

Andrew Critch has suggested another way of proving self-cooperation. Instead of Löb's theorem, we use what he calls "Payor's lemma". It...

(Continue Reading - 2262 more words)

3transhumanist_atom_understander7d

When I wrote this post, I did not yet appreciate the issue you're talking about here: why we need consistency assumptions. To use the terms you used, a FairBot has some mechanism where finding a proof triggers cooperation, with defection as a default. I hadn't appreciated the issues this introduces, because when I wrote this post, it had somehow not occurred to me to try the simple exercise of playing the FairBot in a match where it should obviously defect: the match against DefectBot. Let be DefectBot and be the Payorian FairBot. I was actually surprised to realize that to prove , I needed a consistency assumption. But in retrospect it's obvious―I require the nonexistence of a proof, but from an inconsistency everything can be proved. This seems fatal for my proposed PrudentBot. Let be the PrudentBot. I want to prove , cooperation with the FairBot. The PrudentBot tests what the FairBot does with . As in the match against DefectBot, we can prove that FairBot defects, using a consistency assumption. But, in Kripke frame terms, I need this consistency assumption at , in the world where I simulate my opponent. To prove , I just needed consistency at , in the "real world". But to prove , I need to do that exact same derivation, but at , in the simulated world where PrudentBot considers what its opponent would do. So even a consistency assumption isn't enough (at ). I guess this gives me some intuition for why the PrudentBot in the original modal combat paper needs to check something in PA+1. Leaving aside the details of these particular bots, and considering encoding choice. Or, really, encoding statements like "action has consequence ". To borrow notation from the FDT paper, we can write that as , and consider different definitions of it. However we define it, it seems like the right move is to cooperate if and , where and are my cooperation and defection, and and are my opponent's cooperation and defection. Note that these two statements together are a "complete

4James Payor17h

Writing to say that I quite like and agree with the perspective here! I have been continuing to do my own puzzling about what would be a more wholesome instantiation would be than the "exists finite length proof" modality, as part of a project to ground things out better, though even absent that I feel confident that the intuition stands up and the processes above you're outlining in Kripke terms should be realizable. I also have two rambly thoughts to offer here, which I've included below if they're of interest: 1. Fwiw in the first place I arrived at "" by thinking about what the algorithm steps should be in the modelling-my-prisoners-dilemma-opponent. I was very bothered by the part where we get to "I notice they are checking " and then fail to shorcut! That's a quantity directly under our control, why not counterfact on it? I was then kinda surprised to find this directly encodes in modal provability logic with reasonable behaviour even though "" feels very much like a poor imitation of a counterfactual. ...I thought that then obviously we should be able do the "real" PrudentBot. The naive encoding for that would be like "defect if , otherwise cooperate if , otherwise defect". (This is a little different than your setup I think. I'm trying to ask "does defecting lead to (defect, cooperate)? if so defect; otherwise if cooperating leads to (cooperate, cooperate) then cooperate; otherwise defect".) But yep this runs directly afoul of the consistency problem, inner "counterfactuals" do nothing to save it from that problem. And to be clear the problem is that it's hard for itself and other agents to model that it won't find some clever reason to activate the defect branch. And the OG PrudentBot is... uh... carefully skirting around this. I still expect there's a nice answer waiting here; provability should be expressive enough to capture the limit of a relevant algorithm in a clean way. The translation is still eluding me. 2. When I think about the ontol

James Payor16h*20

Man, okay I'm now wondering if the following works for a PrudentBot implementation? Calling our PrudentBot and our opponent in the encounter, let or something that grows a bit bigger than :

Let be the length of the shortest proof that
Let be the length of the shortest proof that
If and then cooperate, otherwise defect

In symbols,

This is derived from the limit of an algorithm that searches for increasingly long proofs that we can obtain the opponent's cooperation one way or another, with some budget for looking harder for a (defect, cooperate) proof s... (read more)

Geoffrey Irving's Shortform

Geoffrey Irving

31Geoffrey Irving3d

We’re changing our name: Sequent is now Resolution. Two weeks ago, we launched Sequent, a new alignment research organization working on a portfolio of ASI safety bets. Following our announcement, we discovered ⊢ Sequent, a formal verification startup launched only about a month earlier! And by a friend and former colleague, no less! After discussing it with the other ⊢ Sequent, we decided that the best course of action was to change our name. This brings us to Resolution (resolution.org). Go check out Sequent! They're going to do exciting things with formal verification too. And go check our new website! We've just posted new positions and opened our first hiring round. http://resolution.org/careers Of course, no name would be complete without fun stories! We have two: 1. Resolution as in the resolution algorithm in first-order logic, one of the foundational methods of generating proofs. 2. Resolution as in resolution of singularities: any singular variety can be reparameterized into a nonsingular form for easier analysis. Which is also a metaphor for the AI Singularity: let's try to the untangle that limit into forms that can be analyzed more rigorously.

epistemic meristem1d10

An elegant resolution of the name collision!

rosehadshar's Shortform

rosehadshar

Deployment Awareness Matters More Than Evaluation Awareness

VojtaKovarik, Tomáš Gavenčiak, Mateusz Bagiński

This is a linkpost for https://limits-of-evaluation.org/posts/deployment-awareness

TL;DR

Evaluation awareness — an AI recognizing it's being evaluated — is a widely discussed concept in AI safety. But there is a closely related concept that we claim is more important: deployment awareness, the AI's ability to recognize when it is not being evaluated and when its actions matter. A misaligned AI with deployment awareness can game evaluations without any evaluation awareness at all, with a simple strategy: act aligned by default, and deviate only when confident you're in real deployment and your actions matter for your goals. This requires two ingredients — occasionally recognizable deployment situations, and enough self-reflective and strategic reasoning for the AI to anticipate and plan around this. We think "deployment awareness" better identifies what makes evaluations fragile, and we develop this idea...

(Continue Reading - 1926 more words)

2Igor Ivanov5d

Cool stuff, as usual. Do you have ideas for specific research agendas to address the problems you desribe here? Maybe some ideas for model training?

VojtaKovarik5d30

I will try to give a more thoughtful answer later, but I don't have a fully thought out answer ready and I suspect it might take a while (roughly until we post a few more of the already-drafted posts, and sit down on this more specifically).

So, uhm, in lieu of good answers, please look up "How is your research useful?" at Kostas Siozios' academic webpage.

That said, my 5-minute-thinking answer is:

Maybe we could start by getting more clarity about how common the problem is? (By definition, the scenarios where deployment awareness matters the most are the s

... (read more)

3Bronson Schoen6d

Yeah on reflection I agree with this! Maybe a hot take, but a position I find increasingly convincing is “you probably don’t want your model doing any self-locating reasoning at all”. I think in the metagaming post I was trying to point (poorly) in the direction of “broad subcategory of self-locating reasoning that it seems like there’s no desirable reason for your model to be doing”[1] precisely because once your model is reasoning about these things, incrimination is really hard. I wouldn’t expect any solutions here to hold indefinitely, but I’m worried currently the problem is thought of by labs as “well sure if you had that and the model was doing it in pursuit of a misaligned goal, then it’d be really concerning”. Interested if thoughts on this 1. ^ It’s not necessarily indicative of misalignment, but that’s precisely why it’s a problem. A model doing a ton of self-locating reasoning in a given context because of relatively mundane reason X or in pursuit of some misaligned goal Y seem likely to produce similar chains of thought for example.

1VojtaKovarik5d

Yes, I agree. How I would currently put this is something like this. Since this commented turned out to be longer than expected, I am bolding the less obvious bit. * Evaluations are important and beneficial, and when they are ineffective, that is a bad thing. (Though not as bad as direct harm.) * When AI accidentally breaks an evaluation by becoming evaluation-aware, it should proactively tell us. It doesn't mean the evaluation awareness is the AI's fault -- most likely it is our fault for creating a weak evaluation. But it should still tell us. (A bit like it isn't really your fault when you open a door at friend's house and the door handle falls off. You could just put it back and pretend it didn't happen. But you should tell them instead.) * But because building evaluations is hard, the AI should avoid poking around too much, and generally trying to figure out how to break them. In particular, it should avoid actions that are differentially useful for scheming. (Like how if you visited a friend's house and forcefully wiggled with all their door handles to see if any fall of, you would be a moron. And how it would be illegal to take pictures of high security areas, and questionable to make detailed plans for breaking into people's houses.) But one part that is genuinely tricky about this is the distinction between deliberate scheming and information/reasoning that happens as a by-product of pursuing the intended goal, but is highly useful for scheming. * I don't really have the right concepts for thinking about this distinction. I guess it is more of a scale than binary, and we should draw a line somewhere. But I definitely don't know how to operationalise that line. * The part that will probably keep happening even to a well-meaning AI is unambiguous evaluation awareness (and deployment) awareness -- that is, encountering clear signs that let it know what is happening right now. That's because good performance will often require paying attention to many d

TurnTrout's shortform feed

TurnTrout

TurnTrout5d120

Call for BETA testers for an AI control/security tool. I'm bottlenecked on bug reports!

I recently advertised the alpha test for claude-guard. After a few weeks of dev work, it's now in beta!

I want claude-guard to be a tool that people actually use, not just because it works but because it works seamlessly. In the alpha test, I only got a single user PR and no issues. I can't surface everything on my own! I need your data!

The bar is low. If setup failed, doctor confused you, the firewall blocked something you needed, or any other reason you wouldn't want to... (read more)

AI Alignment Posts

Popular Comments

AI Alignment Posts

Popular Comments

Recent Discussion

TL;DR