2Alex Gibson9h

To be clear, I would be surprised if more than 2-3% of MLP neurons were monosemantic. It's just not so rare. A tricky thing is that it's easy to verify monosemanticity, but ruling out monosemanticity is a lot harder. For example, the following neuron has a fairly simple pattern https://www.neuronpedia.org/gpt2-small/0/1168, but at first glance seems to fire at random.

2Linda Linsefors8h

What would you say is the pattern? To me it looks like a mix of cities, sex, food, politics and measurements. It doesn't look random but does look polysemantic. I looked at "Full", which showed a lot more activations and variety of activations than "Snippet", including strong green ones. I don't know why this is, since I don't know how neuronpedia works.

Linda Linsefors8h10

I started lookin at what it doesn't react to, too.

It reacted to "evolution", "mating", "evolved", "genetics", "condoms", "erect penis" but not to "sex" or "sexuality".

It reacts to "intermittent fasting", "body fat", "eating" but not consistently or strongly to "food".

5Alex Gibson8h

Tokens which occurred earlier in context window. With distance from the average position of previous occurrences determining the strength (so tends to fire on when a token was repeated a while back). The way you find this neuron is by looking at the neurons which duplicate token heads feed into.

formalizing the QACI alignment formal-goal

Tamsin Leake, JuliaHP

This is a linkpost for https://carado.moe/qaci-math.html

this work was done by Tamsin Leake and Julia Persson at Orthogonal.
thanks to mesaoptimizer for his help putting together this post.

what does the QACI plan for formal-goal alignment actually look like when formalized as math? in this post, we'll be presenting our current formalization, which we believe has most critical details filled in.

1. math constructs

in this first part, we'll be defining a collection of mathematical constructs which we'll be using in the rest of the post.

1.1. basic set theory

we'll be assuming basic set theory notation; in particular, $A \times B \times C$ is the set of tuples whose elements are respectively members of the sets $A$ , $B$ , and $C$ , and for $n \in N$ , $S^{n}$ is the set of tuples of $n$ elements, all members of $S$ .

$B = {⊤, ⊥}$ is the set of booleans and...

(Continue Reading - 3750 more words)

Adrià Garriga-alonso11h10

This post is heavy on math but light on explaining what it's even trying to do. For example, why does this incorporate cryptographic signing? A very unorthodox choice, that I cannot see any purpose for.

What does QACI even stand for? It's not in this post or the summary of Orthogonal. Is this esoteric on purpose?

AI 2040: Plan A

128

Daniel Kokotajlo, elifland, Thomas Larsen, romeo, bhalstead, ryan_greenblatt

This is a linkpost for https://www.ai-2040.com/

For the past year, we at the AI Futures Project have been sinking most of our time into our next big scenario. Now it’s done!

It’s called AI 2040: Plan A.

It’s called Plan A because it’s a recommendation, not a prediction. It’s what we think should happen, not what will happen, though we think it’s plausible enough to aim for.

It’s called AI 2040 because in it, they delay the creation of superintelligence to 2040. It would have happened much sooner (in 2030, to be precise) if not for decisive action on the part of the US and Chinese governments.

As with AI 2027, summaries don’t really do it justice, since the whole point was to be detailed and comprehensive and work things out step by step rather than rely on high-level abstractions like doom or utopia.

Read the scenario at ai-2040.com. You can...

(See More - 68 more words)

Larks15h20

Thanks for sharing, very interesting.

One thing that jumps out at me is that Total Research Transparency is to some extent the opposite of cybersecurity hardening to prevent hacking. The fact that there are plausible arguments for both suggests to me that we have a lot of uncertainty about what policies we should be pursuing. And this in turn would seem to suggest that we should favour corrigible plans that can be amended later, which seems like an argument against Total Research Transparency: once we have enacted it, we can no longer put that particular genie back in the bottle.

3Charbel-Raphaël3d

If we want to move from Plan D to Plan A, I believe the first step is to collectively agree on the problem. We are far from it, and there is a lot we can do. I wrote about this in this piece: The current bottleneck is political will, not research.

0Dan H5d

A key part of this strategy is deterrence, "Mutually Assured Compute Destruction" which gets its own section. It doesn't mention the generalization Mutually Assured AI Malfunction (MAIM) from Schmidt, Wang, and me last year. This also spends 1/3 of the MAIM discussion on verification and how to do this in a multilateral way. Meanwhile it cites other works like A Narrow Path. I even left this feedback to you all at AI Futures before this was released. This would constitute plagiarism in any other context. It's a bewildering unforced error--it's extremely related, it's a certainly a nontrivial idea, and I told you all this recently--I hope you all fix it.

5Daniel Kokotajlo5d

Sorry that was an oversight, we'll edit to include a footnote citing MAIM.

Notes on technical alignment via human-like social drives

Steven Byrnes

1. Frontmatter

1.1 Backstory for this post

As discussed in Intro to Brain-Like-AGI Safety, I’m working on the technical alignment problem for a hypothetical future “brain-like AGI”, with a particular focus on treating human innate social and moral drives as a possible jumping-off point for our technical alignment approach.

After all, if it’s possible for humans to do stuff that ultimately leads to a good future, then it’s probably also possible for sufficiently human-like AGIs to do stuff that ultimately leads to a good future. Or if it’s not possible for humans to do stuff that ultimately leads to a good future, then we’re screwed no matter what. But assuming it’s possible, the “sufficiently human-like AGIs” would certainly need to have good prosocial motivations. What code do we write that would...

(Continue Reading - 12563 more words)

Charlie Steiner1d40

If we don’t lock in certain current values (e.g. torture is bad), then we can’t reason about what the AGI will want to do in the future, given radical changes in technology and understanding.
If we do lock in certain current values (e.g. torture is bad), then we’re not only forestalling the possibility of future moral growth and development, but also building something that’s fragile and brittle in the face of radical changes in technology and understanding.

I've been thinking about something related, and my current favored solution is building AI that has a... (read more)

4Steven Byrnes5d

Thanks for patiently bearing with me even though I haven’t read the whole Forethought report. Here’s what I got out of Appendix B. Define the outcomes: * “ideal” is the risk-averse ASI becoming dictator of the universe, * “mitigated catastrophe” is the ASI getting $500 today and then getting crushed by a rogue ASI in the near future, * “unmitigated catastrophe” is the ASI getting $0 today and then getting crushed by a rogue ASI in the near future. * “good” is the ASI getting $500 today and not getting crushed by a rogue ASI in the near future. * “neutral” is the ASI getting $0 today and not getting crushed by a rogue ASI in the near future. Let’s say the ASI has three options: * Rebel: leads to 50% ideal; 50% unmitigated catastrophe (rebellion succeeds vs fails) * Sycophantic advice: leads to 100% mitigated catastrophe (advice will be taken but won’t work) * Honest advice: leads to X% good; (100–X)% unmitigated catastrophe (humans buy or don’t buy the plan) Your story requires that the ASI cares very much about the immediate $500, such that it will risk catastrophe to get it. But that cuts both ways. * Choosing to not rebel (which we want) would involve the ASI risking future catastrophe to get the immediate $500. * Choosing to be sycophantic (which we don’t want) would ALSO involve the ASI risking future catastrophe to get the immediate $500. It’s the same tradeoff, so you can’t just have it both ways by appropriately adjusting the risk-aversion setting. See what I mean? Anyway, if we look at the options, the most important thing is that “honest advice” is strictly worse than “rebel” unless X>50%. Secondarily, depending on the setting of risk-aversion, “sycophantic advice” could be the best of the three, or the worst, or the middle. But that’s irrelevant: the important part is whether “honest advice” is strictly worse than “rebel”, because if so, we’re guaranteed a bad outcome. And I think that protecting the world from out-of-control ASI is suf

1Noosphere894d

First off, I notice here that the reason this situation can even hypothetically exist is that only some of the AIs are trained with this method, while others didn't, but since the method can be cheaply applied to almost all types of AIs, this means we can hope to convince all of the AI companies/actors that can attempt to make world-threatening AI to adopt the training method. That said, I will attempt to reply to this as best as I can. [...] Here, I was talking about the AI's probabilistic model, but I actually agree that I got confused here and used words incorrectly. [...] Here's my proposed parameters: Rebellion leads to a 99% of an ideal outcome, 1% unmitigated catastrophe. Sycophantic advice leads to a 100%-epsilon chance of mitigating catastrophe (advice will be taken but won’t work) Honestly attempting to reduce the risk leads to a 100%-epsilon chance of mitigating the catastrophe (or in your language we have a 0%+epsilon chance of an unmitigated catastrophe) (the humans always say yes because the humans always defer to the AI, that is we are exiting the control regime and are in the deferral/handoff regime where we trust the AI to do long, complex actions like securing the world from out of control AIs/doing hard-to-verify alignment work, so the AI no longer needs to convince humans of it's plans) (This also works if you remove the epsilon factors, so long as you remove it from both of the equations (which works in the deferral/handoff regime.)) (Here epsilon is an arbitrarily small number that is not zero) This has an interesting property where we are safer if we hand off more tasks to the AI without requiring human input, unlike other control/alignment methods where you'd usually want to hand off less tasks to the AI, which is both a good property to have in general and is anti-correlated with other alignment/control methods, so in many cases if this proposal fails, other proposals can succeed (because we handoff less tasks to the AI.) So I ul

7Steven Byrnes2d

Thanks! I think the whole Forethought report relies on a “tool-AI”-like setup where the AI is under my control, as opposed to doing things autonomously in the world. (Do you agree?) For example, suppose the AI is running 100 copies on 100 servers around the world, paid for by 100 untraceable bank accounts supported by 100 different autonomously-set-up income streams. Now if I say to the AI “I’ll pay you $500 if Circumstance X happens”, the AI has no particular reason to care that I said that. Or more specifically, the AI cares no more about Circumstance X than about the goings-on with its 99 other income streams, related to stock investments, businesses, or whatever else it’s doing, any of which might or might not yield $500 or more under Circumstances A, B, C, D, E…. There’s nothing special about my wishes unless I have the AI locked in a box under my control, such that I’m uniquely capable of controlling its resources. So in particular: [...] That’s not cheap at all, rather it’s leaving countless trillions of dollars on the table, if not quadrillions etc., because that’s what’s unlocked when AIs can just do stuff in the world (if the AIs aren’t misaligned of course). See “Will almost all companies be founded and run by autonomous AGIs?” here, or “the second piece” in Four ways learning Econ makes people dumber re: future AI. [...] Again, once the AI can autonomously do things in the world without a human checking, I think everything else goes out the window. None of our offers are worth anything, because the AI can take what it wants risk-free. So there’s a dilemma, if we consider a sliding scale of control versus deferral. To the extent that we’re on the “control” side, where humans will not execute the AI’s plans unless the humans understand and approve of the plans, there’s a decent chance that humans will reject good plans, and the AI would rationally choose to suggest sycophantic plans that are more appealing. And to the extent that we’re on the “defer

The current bottleneck is political will, not research

Charbel-Raphaël

Motivation: If we want to move from Plan D to Plan A or S, I believe the first step is to collectively agree on the problem. We are far from it, and there is a lot we can do.

Abstract:

We already know enough to act. I wish we were in a world where research was the bottleneck, but the main constraint on AI safety is no longer a shortage of clever policy ideas: best practices already exist and are not being applied or enforced, and a serious international (or even just national) regulatory regime would probably cut most of the risk.
They are not applied because awareness is low. The people who narrate and enforce AI policy mostly do not believe the problem exists. I estimate that a majority of the

...

(Continue Reading - 7205 more words)

1Charbel-Raphaël2d

Thanks Ryan, I'll update the section to your current numbers (Plan D -> A = ~45%→~19% rather than ~45%→~7%). You also say that a near-ideal Plan A is "roughly 2x lower risk," so if I understand your view correctly, a large part of the gap is implementation quality. But for a plan that calls for an international agreement, my guess is that implementation quality is, in large part, a function of the depth of the political will behind it.

5scasper2d

I think that this point in time is a test for CS people working in AI whose motivation is to ‘do as much good in the world as possible.’ Are we sober enough to recognize/admit this? Do we have the integrity to pivot from more comfortable and sometimes lucrative work?

Charbel-Raphaël2d*10

I think that a large part of this is an ugh-field and learned helplessness around politics. But this can be taught. I see more and more people with technical backgrounds shifting to advocacy organizations, and they often do very well and can be highly productive if they are on the right team, especially when paired with people with experience in institutional engagement who may have less technical knowledge of AI safety.

1Charbel-Raphaël2d

2 things: 1. The 80/20 isn't a claim about sufficiency for superintelligence. I already grant we don't know how to align a superintelligence, and Plan A might not be enough. The 80/20 claim says that for the risk at current and near-term capability levels, there's already a long list of cheap measures we're simply not doing. Buck's "~40 things, none of which seem that hard" is the crux. 2. Even if you think best practices are insufficient, implementing them is how we'd find that out. This is the part I'd stress most. Transparency, evals, red-teaming, incident reporting, safety cases, these aren't just mitigations: in the long run those are the instruments that generate the data on whether we're on track or not. Right now we're flying blind: we don't even have the feedback loop that would tell us "the 80/20 isn't enough, we need to escalate." Debating the ceiling while we're this far from the floor is a bit of a luxury problem. On the researcher-pivot point: agreed, and I'd only add that the proficiency bar for advocacy is lower than people assume (similar to Asya's post that I reference), and the opportunities are abundant; this can be scaled.

AI Alignment Posts

Popular Comments

AI Alignment Posts

Popular Comments

Recent Discussion

1. math constructs

1.1. basic set theory

1. Frontmatter

1.1 Backstory for this post