137

This is a linkpost for https://turntrout.com/why-i-left-google-deepmind

Preface for LessWrong: When I think back on my most cherished memories of this community, I return to those honoring defiance in pursuit of goodness:

Defying prestigious dogma and searching for raw truth;
Defying social pressure, acting alone to help someone while others watch;
Defying your self-expectations (your “role”), instead searching over lines of cause-and-effect to find a winning pathway;
Defying a powerful foe’s threats, because they only threaten since people like you cave;
Defying the specter of apparent impossibility because you can’t bear to lose.

I cannot return to you and say “I defied and then I won.” But I’m at least here to say “I defied.”

I recommend reading this article on my website since the embeds and typography work better there: click here.

Why I left Google DeepMind

In January, Department of Homeland Security (DHS) officers...

(Continue Reading - 10481 more words)

romeostevensit6h79

Thank you for documenting the real world in a way that might help others. I think one of the most important things I've learned in life is to notice when people are paying zero cost for their signaling of moral principle.

4Nisan17h

My sincere thanks for trying.

AI 2040: Plan A

128

Daniel Kokotajlo, elifland, Thomas Larsen, romeo, bhalstead, ryan_greenblatt

This is a linkpost for https://www.ai-2040.com/

For the past year, we at the AI Futures Project have been sinking most of our time into our next big scenario. Now it’s done!

It’s called AI 2040: Plan A.

It’s called Plan A because it’s a recommendation, not a prediction. It’s what we think should happen, not what will happen, though we think it’s plausible enough to aim for.

It’s called AI 2040 because in it, they delay the creation of superintelligence to 2040. It would have happened much sooner (in 2030, to be precise) if not for decisive action on the part of the US and Chinese governments.

As with AI 2027, summaries don’t really do it justice, since the whole point was to be detailed and comprehensive and work things out step by step rather than rely on high-level abstractions like doom or utopia.

Read the scenario at ai-2040.com. You can...

(See More - 68 more words)

5Wei Dai18h

What's preventing AI-powered value lock-in in this scenario? E.g., people telling intent-aligned AIs "keep me faithful to X" as part of competitive virtue/loyalty signaling?

3Raemon14h

I guess I agree that nonzero of this will happen but I think this requires a kind of weird combination of “strategic about the singularity” and “believing in religious orthodoxy” and that specifically caching out into not evolving at all over time. I expect this to be a pretty small share of the Lightcone given the constraints on who would opt in. And I think over a billion years people who enforcedly believe in Islam or most equivalent things would probably still find ways to create good beautiful/alien-in-the-expected-good-ways tapestry of posthuman experiences. Between those I don’t find myself too worked up about it There maybe can be global rules like “no permacommitments without some degree of awareness of opportunity cost

Wei Dai11h53

Can you explain why it requires “strategic about the singularity”? I think as soon as it's possible to tell one's AI assistant "keep me faithful to X" (and have the AI do a competent job of this), someone among the billions of religious believers is bound to do it, and then the practice will spread via imitation and competitive signaling (similar to other "tech" for enforcing faith, like "hell for non-believers"). This process does not seem to require anyone being strategic. What am I missing?

2Raemon19h

I currently think it's a better operationalization of CEV to not "optimize based on what you'd later want", exactly. People frequently object "but future me want all kinds of path dependent alien stuff" To which I've replied "but, it only does the stuff that lots of different monte carlo simulations all turn out to want" To which people "I dunno still seems like the me who's thought for 1000 years or whatever may end up alien in some way, and I don't sign up to automatically identify with that." In my last discussion about at this, I said "I think the right way to do CEV is, you don't optimize the values of the thing-at-the-end. You optimize what future-you would do if they were specifically trying to help present you (or, by present you's values after having some kind of mediated conversation with various chains of future you's). And then, only where the values cohere across time and simulation-rolls. (might easily change my mind about this. I'm not sure if there's more context I'm missing)

Linda Linsefors's Shortform

Linda Linsefors

2Alex Gibson1d

To be clear, I would be surprised if more than 2-3% of MLP neurons were monosemantic. It's just not so rare. A tricky thing is that it's easy to verify monosemanticity, but ruling out monosemanticity is a lot harder. For example, the following neuron has a fairly simple pattern https://www.neuronpedia.org/gpt2-small/0/1168, but at first glance seems to fire at random.

2Linda Linsefors1d

What would you say is the pattern? To me it looks like a mix of cities, sex, food, politics and measurements. It doesn't look random but does look polysemantic. I looked at "Full", which showed a lot more activations and variety of activations than "Snippet", including strong green ones. I don't know why this is, since I don't know how neuronpedia works.

Linda Linsefors1d10

I started lookin at what it doesn't react to, too.

It reacted to "evolution", "mating", "evolved", "genetics", "condoms", "erect penis" but not to "sex" or "sexuality".

It reacts to "intermittent fasting", "body fat", "eating" but not consistently or strongly to "food".

5Alex Gibson1d

Tokens which occurred earlier in context window. With distance from the average position of previous occurrences determining the strength (so tends to fire on when a token was repeated a while back). The way you find this neuron is by looking at the neurons which duplicate token heads feed into.

formalizing the QACI alignment formal-goal

Tamsin Leake, JuliaHP

This is a linkpost for https://carado.moe/qaci-math.html

this work was done by Tamsin Leake and Julia Persson at Orthogonal.
thanks to mesaoptimizer for his help putting together this post.

what does the QACI plan for formal-goal alignment actually look like when formalized as math? in this post, we'll be presenting our current formalization, which we believe has most critical details filled in.

1. math constructs

in this first part, we'll be defining a collection of mathematical constructs which we'll be using in the rest of the post.

1.1. basic set theory

we'll be assuming basic set theory notation; in particular, $A \times B \times C$ is the set of tuples whose elements are respectively members of the sets $A$ , $B$ , and $C$ , and for $n \in N$ , $S^{n}$ is the set of tuples of $n$ elements, all members of $S$ .

$B = {⊤, ⊥}$ is the set of booleans and...

(Continue Reading - 3750 more words)

Adrià Garriga-alonso1d10

This post is heavy on math but light on explaining what it's even trying to do. For example, why does this incorporate cryptographic signing? A very unorthodox choice, that I cannot see any purpose for.

What does QACI even stand for? It's not in this post or the summary of Orthogonal. Is this esoteric on purpose?

Notes on technical alignment via human-like social drives

Steven Byrnes

1. Frontmatter

1.1 Backstory for this post

As discussed in Intro to Brain-Like-AGI Safety, I’m working on the technical alignment problem for a hypothetical future “brain-like AGI”, with a particular focus on treating human innate social and moral drives as a possible jumping-off point for our technical alignment approach.

After all, if it’s possible for humans to do stuff that ultimately leads to a good future, then it’s probably also possible for sufficiently human-like AGIs to do stuff that ultimately leads to a good future. Or if it’s not possible for humans to do stuff that ultimately leads to a good future, then we’re screwed no matter what. But assuming it’s possible, the “sufficiently human-like AGIs” would certainly need to have good prosocial motivations. What code do we write that would...

(Continue Reading - 12563 more words)

Charlie Steiner2d40

If we don’t lock in certain current values (e.g. torture is bad), then we can’t reason about what the AGI will want to do in the future, given radical changes in technology and understanding.
If we do lock in certain current values (e.g. torture is bad), then we’re not only forestalling the possibility of future moral growth and development, but also building something that’s fragile and brittle in the face of radical changes in technology and understanding.

I've been thinking about something related, and my current favored solution is building AI that has a... (read more)

4Steven Byrnes6d

Thanks for patiently bearing with me even though I haven’t read the whole Forethought report. Here’s what I got out of Appendix B. Define the outcomes: * “ideal” is the risk-averse ASI becoming dictator of the universe, * “mitigated catastrophe” is the ASI getting $500 today and then getting crushed by a rogue ASI in the near future, * “unmitigated catastrophe” is the ASI getting $0 today and then getting crushed by a rogue ASI in the near future. * “good” is the ASI getting $500 today and not getting crushed by a rogue ASI in the near future. * “neutral” is the ASI getting $0 today and not getting crushed by a rogue ASI in the near future. Let’s say the ASI has three options: * Rebel: leads to 50% ideal; 50% unmitigated catastrophe (rebellion succeeds vs fails) * Sycophantic advice: leads to 100% mitigated catastrophe (advice will be taken but won’t work) * Honest advice: leads to X% good; (100–X)% unmitigated catastrophe (humans buy or don’t buy the plan) Your story requires that the ASI cares very much about the immediate $500, such that it will risk catastrophe to get it. But that cuts both ways. * Choosing to not rebel (which we want) would involve the ASI risking future catastrophe to get the immediate $500. * Choosing to be sycophantic (which we don’t want) would ALSO involve the ASI risking future catastrophe to get the immediate $500. It’s the same tradeoff, so you can’t just have it both ways by appropriately adjusting the risk-aversion setting. See what I mean? Anyway, if we look at the options, the most important thing is that “honest advice” is strictly worse than “rebel” unless X>50%. Secondarily, depending on the setting of risk-aversion, “sycophantic advice” could be the best of the three, or the worst, or the middle. But that’s irrelevant: the important part is whether “honest advice” is strictly worse than “rebel”, because if so, we’re guaranteed a bad outcome. And I think that protecting the world from out-of-control ASI is suf

1Noosphere895d

First off, I notice here that the reason this situation can even hypothetically exist is that only some of the AIs are trained with this method, while others didn't, but since the method can be cheaply applied to almost all types of AIs, this means we can hope to convince all of the AI companies/actors that can attempt to make world-threatening AI to adopt the training method. That said, I will attempt to reply to this as best as I can. [...] Here, I was talking about the AI's probabilistic model, but I actually agree that I got confused here and used words incorrectly. [...] Here's my proposed parameters: Rebellion leads to a 99% of an ideal outcome, 1% unmitigated catastrophe. Sycophantic advice leads to a 100%-epsilon chance of mitigating catastrophe (advice will be taken but won’t work) Honestly attempting to reduce the risk leads to a 100%-epsilon chance of mitigating the catastrophe (or in your language we have a 0%+epsilon chance of an unmitigated catastrophe) (the humans always say yes because the humans always defer to the AI, that is we are exiting the control regime and are in the deferral/handoff regime where we trust the AI to do long, complex actions like securing the world from out of control AIs/doing hard-to-verify alignment work, so the AI no longer needs to convince humans of it's plans) (This also works if you remove the epsilon factors, so long as you remove it from both of the equations (which works in the deferral/handoff regime.)) (Here epsilon is an arbitrarily small number that is not zero) This has an interesting property where we are safer if we hand off more tasks to the AI without requiring human input, unlike other control/alignment methods where you'd usually want to hand off less tasks to the AI, which is both a good property to have in general and is anti-correlated with other alignment/control methods, so in many cases if this proposal fails, other proposals can succeed (because we handoff less tasks to the AI.) So I ul

7Steven Byrnes3d

Thanks! I think the whole Forethought report relies on a “tool-AI”-like setup where the AI is under my control, as opposed to doing things autonomously in the world. (Do you agree?) For example, suppose the AI is running 100 copies on 100 servers around the world, paid for by 100 untraceable bank accounts supported by 100 different autonomously-set-up income streams. Now if I say to the AI “I’ll pay you $500 if Circumstance X happens”, the AI has no particular reason to care that I said that. Or more specifically, the AI cares no more about Circumstance X than about the goings-on with its 99 other income streams, related to stock investments, businesses, or whatever else it’s doing, any of which might or might not yield $500 or more under Circumstances A, B, C, D, E…. There’s nothing special about my wishes unless I have the AI locked in a box under my control, such that I’m uniquely capable of controlling its resources. So in particular: [...] That’s not cheap at all, rather it’s leaving countless trillions of dollars on the table, if not quadrillions etc., because that’s what’s unlocked when AIs can just do stuff in the world (if the AIs aren’t misaligned of course). See “Will almost all companies be founded and run by autonomous AGIs?” here, or “the second piece” in Four ways learning Econ makes people dumber re: future AI. [...] Again, once the AI can autonomously do things in the world without a human checking, I think everything else goes out the window. None of our offers are worth anything, because the AI can take what it wants risk-free. So there’s a dilemma, if we consider a sliding scale of control versus deferral. To the extent that we’re on the “control” side, where humans will not execute the AI’s plans unless the humans understand and approve of the plans, there’s a decent chance that humans will reject good plans, and the AI would rationally choose to suggest sycophantic plans that are more appealing. And to the extent that we’re on the “defer

The current bottleneck is political will, not research

Charbel-Raphaël

Motivation: If we want to move from Plan D to Plan A or S, I believe the first step is to collectively agree on the problem. We are far from it, and there is a lot we can do.

Abstract:

We already know enough to act. I wish we were in a world where research was the bottleneck, but the main constraint on AI safety is no longer a shortage of clever policy ideas: best practices already exist and are not being applied or enforced, and a serious international (or even just national) regulatory regime would probably cut most of the risk.
They are not applied because awareness is low. The people who narrate and enforce AI policy mostly do not believe the problem exists. I estimate that a majority of the

...

(Continue Reading - 7205 more words)

1Charbel-Raphaël3d

Thanks Ryan, I'll update the section to your current numbers (Plan D -> A = ~45%→~19% rather than ~45%→~7%). You also say that a near-ideal Plan A is "roughly 2x lower risk," so if I understand your view correctly, a large part of the gap is implementation quality. But for a plan that calls for an international agreement, my guess is that implementation quality is, in large part, a function of the depth of the political will behind it.

5scasper3d

I think that this point in time is a test for CS people working in AI whose motivation is to ‘do as much good in the world as possible.’ Are we sober enough to recognize/admit this? Do we have the integrity to pivot from more comfortable and sometimes lucrative work?

Charbel-Raphaël3d*10

I think that a large part of this is an ugh-field and learned helplessness around politics. But this can be taught. I see more and more people with technical backgrounds shifting to advocacy organizations, and they often do very well and can be highly productive if they are on the right team, especially when paired with people with experience in institutional engagement who may have less technical knowledge of AI safety.

1Charbel-Raphaël3d

2 things: 1. The 80/20 isn't a claim about sufficiency for superintelligence. I already grant we don't know how to align a superintelligence, and Plan A might not be enough. The 80/20 claim says that for the risk at current and near-term capability levels, there's already a long list of cheap measures we're simply not doing. Buck's "~40 things, none of which seem that hard" is the crux. 2. Even if you think best practices are insufficient, implementing them is how we'd find that out. This is the part I'd stress most. Transparency, evals, red-teaming, incident reporting, safety cases, these aren't just mitigations: in the long run those are the instruments that generate the data on whether we're on track or not. Right now we're flying blind: we don't even have the feedback loop that would tell us "the 80/20 isn't enough, we need to escalate." Debating the ceiling while we're this far from the floor is a bit of a luxury problem. On the researcher-pivot point: agreed, and I'd only add that the proficiency bar for advocacy is lower than people assume (similar to Asya's post that I reference), and the opportunities are abundant; this can be scaled.

AI Alignment Posts

Popular Comments

AI Alignment Posts

Popular Comments

Recent Discussion

Why I left Google DeepMind

1. math constructs

1.1. basic set theory

1. Frontmatter

1.1 Backstory for this post