Notes on technical alignment via human-like social drives

1. Frontmatter

1.1 Backstory for this post

As discussed in Intro to Brain-Like-AGI Safety, I’m working on the technical alignment problem for a hypothetical future “brain-like AGI”, with a particular focus on treating human innate social and moral drives as a possible jumping-off point for our technical alignment approach.

After all, if it’s possible for humans to do stuff that ultimately leads to a good future, then it’s probably also possible for sufficiently human-like AGIs to do stuff that ultimately leads to a good future. Or if it’s not possible for humans to do stuff that ultimately leads to a good future, then we’re screwed no matter what. But assuming it’s possible, the “sufficiently human-like AGIs” would certainly need to have good prosocial motivations. What code do we write that would...

(Continue Reading - 12563 more words)

Charlie Steiner8m20

If we don’t lock in certain current values (e.g. torture is bad), then we can’t reason about what the AGI will want to do in the future, given radical changes in technology and understanding.
If we do lock in certain current values (e.g. torture is bad), then we’re not only forestalling the possibility of future moral growth and development, but also building something that’s fragile and brittle in the face of radical changes in technology and understanding.

I've been thinking about something related, and my current favored solution is building AI that has a... (read more)

4Steven Byrnes4d

Thanks for patiently bearing with me even though I haven’t read the whole Forethought report. Here’s what I got out of Appendix B. Define the outcomes: * “ideal” is the risk-averse ASI becoming dictator of the universe, * “mitigated catastrophe” is the ASI getting $500 today and then getting crushed by a rogue ASI in the near future, * “unmitigated catastrophe” is the ASI getting $0 today and then getting crushed by a rogue ASI in the near future. * “good” is the ASI getting $500 today and not getting crushed by a rogue ASI in the near future. * “neutral” is the ASI getting $0 today and not getting crushed by a rogue ASI in the near future. Let’s say the ASI has three options: * Rebel: leads to 50% ideal; 50% unmitigated catastrophe (rebellion succeeds vs fails) * Sycophantic advice: leads to 100% mitigated catastrophe (advice will be taken but won’t work) * Honest advice: leads to X% good; (100–X)% unmitigated catastrophe (humans buy or don’t buy the plan) Your story requires that the ASI cares very much about the immediate $500, such that it will risk catastrophe to get it. But that cuts both ways. * Choosing to not rebel (which we want) would involve the ASI risking future catastrophe to get the immediate $500. * Choosing to be sycophantic (which we don’t want) would ALSO involve the ASI risking future catastrophe to get the immediate $500. It’s the same tradeoff, so you can’t just have it both ways by appropriately adjusting the risk-aversion setting. See what I mean? Anyway, if we look at the options, the most important thing is that “honest advice” is strictly worse than “rebel” unless X>50%. Secondarily, depending on the setting of risk-aversion, “sycophantic advice” could be the best of the three, or the worst, or the middle. But that’s irrelevant: the important part is whether “honest advice” is strictly worse than “rebel”, because if so, we’re guaranteed a bad outcome. And I think that protecting the world from out-of-control ASI is suf

1Noosphere893d

First off, I notice here that the reason this situation can even hypothetically exist is that only some of the AIs are trained with this method, while others didn't, but since the method can be cheaply applied to almost all types of AIs, this means we can hope to convince all of the AI companies/actors that can attempt to make world-threatening AI to adopt the training method. That said, I will attempt to reply to this as best as I can. [...] Here, I was talking about the AI's probabilistic model, but I actually agree that I got confused here and used words incorrectly. [...] Here's my proposed parameters: Rebellion leads to a 99% of an ideal outcome, 1% unmitigated catastrophe. Sycophantic advice leads to a 100%-epsilon chance of mitigating catastrophe (advice will be taken but won’t work) Honestly attempting to reduce the risk leads to a 100%-epsilon chance of mitigating the catastrophe (or in your language we have a 0%+epsilon chance of an unmitigated catastrophe) (the humans always say yes because the humans always defer to the AI, that is we are exiting the control regime and are in the deferral/handoff regime where we trust the AI to do long, complex actions like securing the world from out of control AIs/doing hard-to-verify alignment work, so the AI no longer needs to convince humans of it's plans) (This also works if you remove the epsilon factors, so long as you remove it from both of the equations (which works in the deferral/handoff regime.)) (Here epsilon is an arbitrarily small number that is not zero) This has an interesting property where we are safer if we hand off more tasks to the AI without requiring human input, unlike other control/alignment methods where you'd usually want to hand off less tasks to the AI, which is both a good property to have in general and is anti-correlated with other alignment/control methods, so in many cases if this proposal fails, other proposals can succeed (because we handoff less tasks to the AI.) So I ul

7Steven Byrnes1d

Thanks! I think the whole Forethought report relies on a “tool-AI”-like setup where the AI is under my control, as opposed to doing things autonomously in the world. (Do you agree?) For example, suppose the AI is running 100 copies on 100 servers around the world, paid for by 100 untraceable bank accounts supported by 100 different autonomously-set-up income streams. Now if I say to the AI “I’ll pay you $500 if Circumstance X happens”, the AI has no particular reason to care that I said that. Or more specifically, the AI cares no more about Circumstance X than about the goings-on with its 99 other income streams, related to stock investments, businesses, or whatever else it’s doing, any of which might or might not yield $500 or more under Circumstances A, B, C, D, E…. There’s nothing special about my wishes unless I have the AI locked in a box under my control, such that I’m uniquely capable of controlling its resources. So in particular: [...] That’s not cheap at all, rather it’s leaving countless trillions of dollars on the table, if not quadrillions etc., because that’s what’s unlocked when AIs can just do stuff in the world (if the AIs aren’t misaligned of course). See “Will almost all companies be founded and run by autonomous AGIs?” here, or “the second piece” in Four ways learning Econ makes people dumber re: future AI. [...] Again, once the AI can autonomously do things in the world without a human checking, I think everything else goes out the window. None of our offers are worth anything, because the AI can take what it wants risk-free. So there’s a dilemma, if we consider a sliding scale of control versus deferral. To the extent that we’re on the “control” side, where humans will not execute the AI’s plans unless the humans understand and approve of the plans, there’s a decent chance that humans will reject good plans, and the AI would rationally choose to suggest sycophantic plans that are more appealing. And to the extent that we’re on the “defer

Linda Linsefors's Shortform

Linda Linsefors

Linda Linsefors5h44

Thoughts on: When is it even possible (or likely) for single neurons to encode single concepts, based on model architecture

As part of my mech-interp research, I'm thinking a lot about the question "how would I do this if I was a neural network?". Specifically, given some toy model architecture, how can I set the weights so that it actives some task.

Todays conclusion is that single ReLU neurons are almost useless. To do identify any non-linear pattern (e.g. XOR), you need at least two ReLUs. We should not expect any feature to be represented by a single Re... (read more)

The current bottleneck is political will, not research

Charbel-Raphaël

Motivation: If we want to move from Plan D to Plan A or S, I believe the first step is to collectively agree on the problem. We are far from it, and there is a lot we can do.

Abstract:

We already know enough to act. I wish we were in a world where research was the bottleneck, but the main constraint on AI safety is no longer a shortage of clever policy ideas: best practices already exist and are not being applied or enforced, and a serious international (or even just national) regulatory regime would probably cut most of the risk.
They are not applied because awareness is low. The people who narrate and enforce AI policy mostly do not believe the problem exists. I estimate that a majority of the

...

(Continue Reading - 7205 more words)

1Charbel-Raphaël15h

Thanks Ryan, I'll update the section to your current numbers (Plan D -> A = ~45%→~19% rather than ~45%→~7%). You also say that a near-ideal Plan A is "roughly 2x lower risk," so if I understand your view correctly, a large part of the gap is implementation quality. But for a plan that calls for an international agreement, my guess is that implementation quality is, in large part, a function of the depth of the political will behind it.

5scasper1d

I think that this point in time is a test for CS people working in AI whose motivation is to ‘do as much good in the world as possible.’ Are we sober enough to recognize/admit this? Do we have the integrity to pivot from more comfortable and sometimes lucrative work?

Charbel-Raphaël13h*10

I think that a large part of this is an ugh-field and learned helplessness around politics. But this can be taught. I see more and more people with technical backgrounds shifting to advocacy organizations, and they often do very well and can be highly productive if they are on the right team, especially when paired with people with experience in institutional engagement who may have less technical knowledge of AI safety.

1Charbel-Raphaël1d

2 things: 1. The 80/20 isn't a claim about sufficiency for superintelligence. I already grant we don't know how to align a superintelligence, and Plan A might not be enough. The 80/20 claim says that for the risk at current and near-term capability levels, there's already a long list of cheap measures we're simply not doing. Buck's "~40 things, none of which seem that hard" is the crux. 2. Even if you think best practices are insufficient, implementing them is how we'd find that out. This is the part I'd stress most. Transparency, evals, red-teaming, incident reporting, safety cases, these aren't just mitigations: in the long run those are the instruments that generate the data on whether we're on track or not. Right now we're flying blind: we don't even have the feedback loop that would tell us "the 80/20 isn't enough, we need to escalate." Debating the ceiling while we're this far from the floor is a bit of a luxury problem. On the researcher-pivot point: agreed, and I'd only add that the proficiency bar for advocacy is lower than people assume (similar to Asya's post that I reference), and the opportunities are abundant; this can be scaled.

AI 2040: Plan A

127

Daniel Kokotajlo, elifland, Thomas Larsen, romeo, bhalstead, ryan_greenblatt

This is a linkpost for https://www.ai-2040.com/

For the past year, we at the AI Futures Project have been sinking most of our time into our next big scenario. Now it’s done!

It’s called AI 2040: Plan A.

It’s called Plan A because it’s a recommendation, not a prediction. It’s what we think should happen, not what will happen, though we think it’s plausible enough to aim for.

It’s called AI 2040 because in it, they delay the creation of superintelligence to 2040. It would have happened much sooner (in 2030, to be precise) if not for decisive action on the part of the US and Chinese governments.

As with AI 2027, summaries don’t really do it justice, since the whole point was to be detailed and comprehensive and work things out step by step rather than rely on high-level abstractions like doom or utopia.

Read the scenario at ai-2040.com. You can...

(See More - 68 more words)

Charbel-Raphaël2d30

If we want to move from Plan D to Plan A, I believe the first step is to collectively agree on the problem. We are far from it, and there is a lot we can do. I wrote about this in this piece: The current bottleneck is political will, not research.

0Dan H4d

A key part of this strategy is deterrence, "Mutually Assured Compute Destruction" which gets its own section. It doesn't mention the generalization Mutually Assured AI Malfunction (MAIM) from Schmidt, Wang, and me last year. This also spends 1/3 of the MAIM discussion on verification and how to do this in a multilateral way. Meanwhile it cites other works like A Narrow Path. I even left this feedback to you all at AI Futures before this was released. This would constitute plagiarism in any other context. It's a bewildering unforced error--it's extremely related, it's a certainly a nontrivial idea, and I told you all this recently--I hope you all fix it.

5Daniel Kokotajlo4d

Sorry that was an oversight, we'll edit to include a footnote citing MAIM.

3Thomas Larsen4d

I'm excited about people commenting on this post with questions, feedback, critiques, different proposals, etc. We'll try to monitor it and respond to many of the comments.

Plans A, B, C, and D for misalignment risk

ryan_greenblatt

9mo

I sometimes think about plans for how to handle misalignment risk. Different levels of political will for handling misalignment risk result in different plans being the best option. I often divide this into Plans A, B, C, and D (from most to least political will required). See also Buck's quick take about different risk level regimes.

In this post, I'll explain the Plan A/B/C/D abstraction as well as discuss the probabilities and level of risk associated with each plan.

Here is a summary of the level of political will required for each of these plans and the corresponding takeoff trajectory:

Plan A: There is enough will for some sort of strong international agreement that mostly eliminates race dynamics and allows for slowing down (at least for some reasonably long period,

...

(Continue Reading - 1729 more words)

ryan_greenblatt2d92

As part of AI 2040, I give updated likelihoods for one particular operationalization here and updated chances of AI takeover here. Note that my views are very sensitive to the operationalization and implementation quality. E.g., a near ideal implementation Plan A might be roughly 2x lower risk.

From wantons to moral agents

Michele Campolo

Posted also on the EA Forum. Written mostly at AFFINE.

Theoretical, some parts are hard to read; consider reading the next post instead.

Introduction: motivation

Anyone interested in creating an artificial agent that does, or says, good things instead of bad things should at least consider the possibility that there is a class of reasoning agents which, after acquiring enough knowledge and reasoning long enough, agree with each other on basic principles regarding what matters, what is most important, what is most worth doing.

I’ve already argued in other posts why this possibility should be our best guess and not just an edge case scenario. This post follows the previous ones, but instead of presenting another argument for the same claim, it focuses on the mechanisms that lead to the formation of the...

(Continue Reading - 4801 more words)

AI Alignment Posts

Popular Comments

AI Alignment Posts

Popular Comments

Recent Discussion

1. Frontmatter

1.1 Backstory for this post

Introduction: motivation