Asymptotically Unambitious AGI

[-]paulfchristiano7y80

If I have a great model of physics in hand (and I'm basically unconcerned with competitiveness, as you seem to be), why not just take the resulting simulation of the human and give it a long time to think? That seems to have fewer safety risks and to be more useful.

More generally, under what model of AI capabilities / competitiveness constraints would you want to use this procedure?

2michaelcohen7y

I know I don't prove it, but I think this agent would be vastly superhuman, since it approaches Bayes-optimal reasoning with respect to its observations. ("Approaches" because MAP -> Bayes). For the asymptotic results, one has to consider environments that produce observations with the true objective probabilities (hence the appearance that I'm unconcerned with competitiveness). In practice, though, given the speed prior, the agent will require evidence to entertain slow world-models, and for the beginning of its lifetime, the agent will be using low-fidelity models of the environment and the human-explorer, rendering it much more tractable than a perfect model of physics. And I think that even at that stage, well before it is doing perfect simulations of other humans, it will far surpass human performance. We manage human-level performance with very rough simulations of other humans. That leads me to think this approach is much more competitive that simulating a human and giving it a long time to think.

3paulfchristiano7y

I'm keen on asymptotic analysis, but if we want to analyze safety asymptotically I think we should also analyze competitiveness asymptotically. That is, if our algorithm only becomes safe in the limit because we shift to a super uncompetitive regime, it undermines the use of the limit as analogy to study the finite time behavior. (Though this is not the most interesting disagreement, probably not worth responding to anything other than the thread where I ask about "why do you need this memory stuff?")

2michaelcohen7y

Definitely agree. I don't think it's the case that a shift to super uncompetitiveness is actually an "ingredient" to benignity, but my only discussion of that so far is in the conclusion: "We can only offer informal claims regarding what happens before BoMAI is definitely benign..."

2paulfchristiano7y

Surely that just depends on how long you give them to think. (See also HCH.)

2michaelcohen7y

By competitiveness, I meant usefulness per unit computation.

3paulfchristiano7y

The algorithm takes an argmax over an exponentially large space of sequences of actions, i.e. it does 2^{episode length} model evaluations. Do you think the result is smarter than a group of humans of size 2^{episode length}? I'd bet against---the humans could do this particular brute force search, in which case you'd have a tie, but they'd probably do something smarter.

2michaelcohen7y

I obviously haven't solved the Tractable General Intelligence problem. The question is whether this is a tractable/competitive framework. So expectimax planning would naturally get replaced with a Monte-Carlo tree search, or some better approach we haven't thought of. And I'll message you privately about a more tractable approach to identifying a maximum a posteriori world-model from a countable class (I don't assign a very high probability to it being a hugely important capabilities idea, since those aren't just lying around, but it's more than 1%). It will be important, when considering any of these approximations, to evaluate whether they break benignity (most plausibly, I think, by introducing a new attack surface for optimization daemons). But I feel fine about deferring that research for the time being, so I defined BoMAI as doing expectimax planning instead of MCTS. Given that the setup is basically a straight reinforcement learner with a weird prior, I think that at that level of abstraction, the ceiling of competitiveness is quite high.

3paulfchristiano7y

I'm sympathetic to this picture, though I'd probably be inclined to try to model it explicitly---by making some assumption about what the planning algorithm can actually do, and then showing how to use an algorithm with that property. I do think "just write down the algorithm, and be happier if it looks like a 'normal' algorithm" is an OK starting point though Stepping back from this particular thread, I think the main problem with competitiveness is that you are just getting "answers that look good to a human" rather than "actually good answers." If I try to use such a system to navigate a complicated world, containing lots of other people with more liberal AI advisors helping them do crazy stuff, I'm going to quickly be left behind. It's certainly reasonable to try to solve safety problems without attending to this kind of competitiveness, though I think this kind of asymptotic safety is actually easier than you make it sound (under the implicit "nothing goes irreversibly wrong at any finite time" assumption).

2michaelcohen7y

Starting a new thread on this: here.

[-]michaelcohen7y50

Thanks for a really productive conversation in the comment section so far. Here are the comments which won prizes.

Comment prizes:

Objection to the term benign (and ensuing conversation). Wei Dei. Link. $20

A plausible dangerous side-effect. Wei Dai. Link. $40

Short description length of simulated aliens predicting accurately. Wei Dai. Link. $120

Answers that look good to a human vs. actually good answers. Paul Christiano. Link. $20

Consequences of having the prior be based on K(s), with s a description of a Turing machine. Paul Christiano. Link. $90

Simulated aliens converting simple world-models into fast approximations thereof. Paul Christiano. Link. $35

Simulating suffering agents. cousin_it. Link. $20

Reusing simulation of human thoughts for simulation of future events. David Krueger. Link. $20

Options for transfer:

1) Venmo. Send me a request at @Michael-Cohen-45.

2) Send me your email address, and I’ll send you an Amazon gift card (or some other electronic gift card you’d like to specify).

3) Name a charity for me to donate the money to.

I would like to exert a bit of pressure not to do 3, and spend the money on something frivolous instead :) I want to reward your consciousness, more than... (read more)

[-]paulfchristiano7y40

Here is an old post of mine on the hope that "computationally simplest model describing the box" is actually a physical model of the box. I'm less optimistic than you are, but it's certainly plausible.

From the perspective of optimization daemons / inner alignment, I think like the interesting question is: if inner alignment turns out to be a hard problem for training cognitive policies, do we expect it to become much easier by training predictive models? I'd bet against at 1:1 odds, but not 1:2 odds.

2michaelcohen7y

I don't actually rely on this assumption, although it underpins the intuition behind Assumption 2.

2paulfchristiano7y

I agree that you don't rely on this assumption (so I was wrong to assume you are more optimistic than I am). In the literal limit, you don't need to care about any of the considerations of the kind I was raising in my post.

[-]michaelcohen7y30

From Paul:

I think the main problem with competitiveness is that you are just getting "answers that look good to a human" rather than "actually good answers."

The comment was here, but I think it deserves its own thread. Wei makes the same point here (point number 3), and our ensuing conversation is also relevant to this thread.

My answers to Wei were two-fold: one is that if benignity is established, it's possible to safely tinker with the setup until hopefully "answers that look good to a human" resembles good answers (we ... (read more)

[-]paulfchristiano7y30

Given that you are taking limits, I don't see why you need any of the machinery with forgetting or with memory-based world models (and if you did really need that machinery, it seems like your proof would have other problems). My understanding is:

Your already assume that you can perform arbitrarily many rounds of the algorithm as intended (or rather you prove that there is some $n_{0}$ such that if you ran $n_{0}$ steps, with everything working as intended and in particular with no memory corruption, then you would get "benign" behavior).
Any time the

... (read more)

2michaelcohen7y

Notational note: I use i0 to denote the episode when BoMAI becomes demonstrably benign and n0 for something else. Any time any model makes a different on-policy prediction from the intended model, it loses some likelihood (in expectation). The off-policy predictions don't get tested. Under a policy that doesn't cause the computer's memory to be tampered with (which is plausible, even ideal), ν† and ν⋆ are identical, so we can't count on ν† losing probability mass relative to ν⋆. The approach here is to set it up so that world-models like ν† either start with a lower prior, or else eventually halt when they exhaust their computation budget.

3paulfchristiano7y

I agree with that, but if they are always making the same on-policy prediction it doesn't matter what happens to their relative probability (modulo exploration). The agent can't act on an incentive to corrupt memory infinitely often, because each time requires the models making a different prediction on-policy. So the agent only acts on such an incentive finitely many times, and hence never does so after some sufficiently late episode i0. Agree/disagree? (Having a bad model can still hurt, since the bogus model might agree on-policy but assign lower rewards off-policy. But if they also always approximately agree on the exploration distribution, then a bad model also can't discourage exploration. And if they don't agree on the exploration distribution, then the bad model will eventually get tested.)

2michaelcohen7y

Ah I see what you’re saying. I suppose I constrained myself to producing an algorithm/setup where the asymptotic benignity result followed from reasons that don’t require dangerous behavior in the interim. Also, you can add another parameter to BoMAI where you just have the human explorer explore for the first E episodes. The i0 in the Eventual Benignity Theorem can be thought of as the max of i' and i''. i' comes from the i0 in Lemma 1 (Rejecting the Simple Memory-Based). i'' comes from the point in time when ^ν(i) is ε-accurate on policy, which renders Lemma 3 applicable. (And Lemma 2 always applies). My initial thought was to set E so that the human explorer is exploring for the whole time when the MAP world-model was not necessarily benign. This works for i'. E can just be set to be greater than i'. The thing it doesn't work for is i''. If you increase E, the value of i'' goes up as well. So in fact, if you set E large enough, the first time BoMAI controls the episode, it will be benign. Then, there is a period where it might not be benign. However, from that point on, the only "way" for a world-model to be malign is by being worse than ε-inaccurate on-policy, because Lemmas 1 and 2 have already kicked in, and if it were ε-accurate on-policy, Lemma 3 would kick in as well. The first point to make about this is that in this regime, benignity comes in tandem with intelligence--it has to be confused to be dangerous (like a self-driving car). The second point is: I can't come up with an example of world-model which is plausibly maximum a posteriori in this interval of time, and which is plausibly dangerous (for what that's worth; and I don't like to assume it's worth much because it took me months to notice ν†).

3paulfchristiano7y

I think my point is this: * The intuitive thing you are aiming at is stronger than what the theorem establishes (understandably!) * You probably don't need the memory trick to establish the theorem itself. * Even with the memory trick, I'm not convinced you meet the stronger criterion. There are a lot of other things similar to memory that can cause trouble---the theorem is able to avoid them only because of the same unsatisfying asymptotic feature that would have caused it to avoid memory-based models even without the amnesia.

2michaelcohen7y

This is a conceptual approach I hadn't considered before--thank you. I don't think it's true in this case. Let's be concrete: the asymptotic feature that would have caused it to avoid memory-based models even without amnesia is trial and error, applied to unsafe policies. Every section of the proof, however, can be thought of as making off-policy predictions behave. The real result of the paper would then be "Asymptotic Benignity, proven in a way that involves off-policy predictions approaching their benign output without ever being tested". So while there might be malign world-models of a different flavor to the memory-based ones, I don't think the way this theorem treats them is unsatisfying.

[-]michaelcohen7y20

Comment thread: general concerns/confusions

[-]Wei Dai7y110

Can you give some intuitions about why the system uses a human explorer instead of doing exploring automatically?
I'm concerned about overloading the word "benign" with a new concept (mainly not seeking power outside the box, if I understand correctly) that doesn't match either informal usage or a previous technical definition. In particular this "benign" AGI (in the limit) will hack the operator's mind to give itself maximum reward, if that's possible, right?
The system seems limited to answering questions that the human operator can correctly evaluate the answers to within a single episode (although I suppose we could make the episodes very long and allow multiple humans into the room to evaluate the answer together). (We could ask it other questions but it would give answers that sound best to the operator rather than correct answers.) If you actually had this AGI today, what questions would you ask it?
If you were to ask it a question like "Given these symptoms, do I need emergency medical treatment?" and the correct answer is "yes", it would answer "no" because if it answered "yes" then the operator would leave the room and it would get 0 reward for the rest of the episode. M

... (read more)

2michaelcohen7y

When I say it would continue to accomplish whatever task we wanted, I'm being a bit sloppy--if we have a task we want accomplished, and we provide rewards randomly, it will not accomplish our desired task. But I take the point that "whatever task we wanted" does have some restrictions: it has to be one that a human operator can convert into a reward without leaving. So the task "respond with the true answer to [difficult question]" is not one that the operator can convert into a reward, but the task "respond with an answer that sounds plausible to the operator" is. I think this subsumes your example.

2michaelcohen7y

Whatever policy is used for exploration, we can ensure that BoMAI will eventually outperform this policy. With a human executing the policy, this leads to BoMAI accumulating reward at least as well as a human. Under the "smarter" information theoretic exploratory policies that I've considered, exploratory behavior is unsafe from insatiable curiosity: the agent has to try killing everyone just to check to make sure it's not a weird cheat code.

1michaelcohen7y

Yes. BoMAI would be able to give plausible-sounding answers to questions. BoMAI could also do any task that was automatically checkable: don't use a human operator at all; have an automated system which interprets text as an amino acid sequence; synthesize that protein; measure some feature of it's behavior; provide reward accordingly. (That example invites renewed focus on the impermeability of the box, by the way). Some things I would do is send an eminent cancer researcher in to ask BoMAI for a research proposal. Then the researcher could go out and test it. It might be worthless, no matter how plausible it seemed, but then they could go back having learned something about a failed path. Repeating this process, it seems likely to me that a correct idea would appear, just considering the likelihood of appearing plausible to a better and better trained evaluator. I would also naturally ask it how to make a safe unbounded AGI. And the next episode, I would ask for an explanation for why that would fail. REDACTED: On that topic, in addition to having multiple humans in the box, you could also have 2 agents that the operator interacts with, both of which are clones except that the reward for the second is one minus the reward for the first. This would look like "AI Safety via debate."

2Wei Dai7y

This seems useful if you could get around the mind hacking problem, but how would you do that? I don't know how this would work in terms of your setup. The most obvious way would seem to require the two agents to simulate each other, which would be impossible, and I'm not sure what else you might have in mind.

2Wei Dai7y

On second thought, (even assuming away the mind hacking problem) if you ask about "how to make a safe unbounded AGI" and "what's wrong with the answer" in separate episodes, you're essentially manually searching an exponentially large tree of possible arguments, counterarguments, counter-counterarguments, and so on. (Two episodes isn't enough to determine whether the first answer you got was a good one, because the second answer is also optimized for sounding good instead of being actually correct, so you'd have to do another episode to ask for a counter-argument to the second answer, and so on, and then once you've definitively figured out that some answer/node was bad, you have to ask for another answer at that node and repeat this process.) The point of "AI Safety via Debate" was to let AI do all this searching for you, so it seems that you do have to figure out how to do something similar to avoid the exponential search. ETA: Do you know if the proposal in "AI Safety via Debate" is "asymptotically benign" in the sense you're using here?

2michaelcohen7y

No! Either debater is incentivized to take actions that get the operator to create another artificial agent that takes over the world, replaces the operator, and settles the debate in favor of the debater in question.

1Wei Dai7y

I guess we can incorporate into DEBATE the idea of building a box around the debaters and judge with a door that automatically ends the episode when opened. Do you think that would be sufficient to make it "benign" in practice? Are there any other ideas in this paper that you would want to incorporate into a practical version of DEBATE?

1michaelcohen7y

Add the retrograde amnesia chamber and an explorer, and we're pretty much at this, right? Without the retrograde amnesia, it might still be benign, but I don't know how to show it. Without the explorer, I doubt you can get very strong usefulness results.

1michaelcohen7y

I expect the human operator moderating this debate would get pretty good at thinking about AGI safety, and start to become noticeably better at dismissing bad reasoning than good reasoning, at which point BoMAI would find the production of correct reasoning a good heuristic for seeming convincing. ... but yes, it is still exponential (exponential in what, exactly? maybe the number of concepts we have handles for?); this comment is the real answer to your question.

2Wei Dai7y

Alternatively, the human might have a lot of adversarial examples and the debate becomes an exercise in exploring all those adversarial examples. I'm not sure how to tell what will really happen short of actually having a superintelligent AI to test with.

2michaelcohen7y

You're right (see the redaction). Why Wei is right. Here's an unpolished idea though: they could do something like minimax. Instead of simulating the other agent, they could model the environment as responding to a pair of actions. For inference, they would have the history of their opponent's actions as well, and for planning, they could pick their action to maximize their objective assuming the other agent's actions are maximally inconvenient.

1Wei Dai7y

So you basically have the same AI play both sides of the zero-sum game, right? That seems like it should work, with the same caveat as for “AI Safety via debate”, namely that it seems hard to predict what happens when you have superintelligent AIs play a zero-sum game with a human as the judge.

1michaelcohen7y

Yep.

1Wei Dai7y

With a debate-like setup, if one side (A) is about to lose a debate, it seems to have a high incentive to claim that the other side (B) trying to do a mind hack and that if the judge keeps paying attention to what B says (i.e., read any further output from B), they will soon be taken over. What is the judge supposed to do in this case? They could ask A to explain how B's previous outputs constitute part of an attempt to mind hack, but A could concoct a story mixed with its own attempt to mind hack, and the judge can't ask for any counter-arguments from B without risking being mind hacked by B. (I realize this is a problem in “AI Safety via debate” as well, but I'm asking you since you're here and Geoffrey Irving isn't. :)

1michaelcohen7y

*but A could concoct a story ... counterarguments from B .. mind hacked by B, right? I think the main contention of their paper is that it is harder to mystify a judge than it is to pierce through someone else mystifying a judge, so this shouldn't be a problem. That said, here's one possibility: if A ever suggests that you don't read more output from B, don't read anything more from B, then flip coins to give A a 25% chance of victory.

1Wei Dai7y

Yeah, I mixed up the A's and B's at the end. It's fixed now. Thanks for pointing it out. I don't understand how the former implies the latter. Assuming the former is true (and it seems like a big assumption), why can't what I suggested still happen? But what about the case where B is actually trying to mind hack the judge? If you always give A a 25% chance of victory for suggesting or implying that you shouldn't read more output from B, then mind hacking becomes a (mostly) winning strategy, since a player gets a 75% chance of victory from mind hacking even if the other side successfully convinces the judge that they're trying to mind hack the judge. The equilibrium might then consist of a race to see who can mind hack the judge first, or (if one side has >75% chance of winning such a race due to first-mover or second-mover advantage) one side trying to mind hack the judge, getting blocked by the other side, and still getting 75% victory.

1michaelcohen7y

If the assumption is true, we could demand that A use their words, and counter us being mind-hacked by poking holes in what B is saying rather than demanding we stop listening to B. And if A is able to convince us that B was mind-hacking, even after some more back and forth, B will be punished for that. So actually I framed my point above wrong: "demanding that A use their words" could look like the protocol I describe; it is not something that would work independently of the assumption that it is easier to deflate an attempted mind-hacking than it is to mind-hack (with an equal amount of intelligence/resources). But your original point was "why doesn't A just claim B is mind-hacking" not "why doesn't B just mind-hack"? The answer to that point was "demand A use their words rather than negotiate an end to the conversation" or more moderately, "75%-demand that A do this."

1Wei Dai7y

Oh, I see, I didn't understand "it is harder to mystify a judge than it is to pierce through someone else mystifying a judge" correctly. So this assumption basically rules out a large class of possible vulnerabilities in the judge, right? For example, if the judge had the equivalent of a buffer overflow bug in a network stack, the scheme would fail. In that case, A would not be able to "pierce through" B's attack and stop it with its words if the judge keeps listening to B (and B was actually attacking). I don't think the "AI safety via debate" paper actually makes arguments for this assumption (at least I couldn't find where it does). Do you have reasons to think it's true, or ideas for how to verify that it's true, short of putting a human in a BoMAI?

1michaelcohen7y

Yeah... I don't have much to add here. Let's keep thinking about this. I wonder if Paul is more bullish on the premise that "it is harder to mystify a judge than it is to pierce through someone else mystifying a judge" than I am? Recall that this idea was to avoid If it also reduces the risk of operator-devotion, and it might well do that (because a powerful adversary is opposed to that), that wasn't originally what brought us here.

1michaelcohen7y

If the person leaves having been hypnotized, since it's not catastrophic, we can modify the setup. For example, the room could be divided in half, with Operator 1 interacting BoMAI, and with Operator 2 observing Operator 1 for signs of hypnosis or for signs of an attempt to communicate with Operator 2, and in either event, Operator 2 could end the episode immediately. (That would be Operator 2's only ability). More importantly than the magnitude of [(the probability that this works) - (the probability it would work the first time)] is the fact that this can be done iteratively. A bit of a side note: I'm curious what odds you give to hypnosis (or something like it) being the best way to get optimal reward for such a task (in the vanilla setup).

2Wei Dai7y

Instead of hypnosis, I'm more worried about the AI talking the operator into some kind of world view that implies they should be really generous to the AI (i.e., give it max rewards), or give some sequence of answers that feel extremely insightful (and inviting further questions/answers in the same vein). And then the operator might feel a desire afterwards to spread this world view or sequence of answers to others (even though, again, this wasn't optimized for by the AI). If you try to solve the mind hacking problem iteratively, you're more likely to find a way to get useful answers out of the system, but you're also more likely to hit upon an existentially catastrophic form of mind hacking. I guess it depends on how many interactions per episode and how long each answer can be. I would say >.9 probability that hypnosis or something like what I described above is optimal if they are both long enough. So you could try to make this system safer by limiting these numbers, which is also talked about in "AI Safety via Debate" if I remember correctly.

1michaelcohen7y

It is plausible to me that there is selection pressure to make the operator "devoted" in some sense to BoMAI. But most people with a unique motive are not able to then take over the world or cause an extinction event. And BoMAI has no incentive to help the operator gain those skills. Just to step back and frame this conversation, we're discussing the issue of outside-world side-effects that correlate with in-the-box instrumental goals. Implicit in the claim of the paper is that technological progress is an outside-world correlate of operator-satisfaction, an in-the-box instrumental goal. I agree it is very much worth considering plausible pathways to negative consequences, but I think the default answer is that with optimization pressure, surprising things happen, but without optimization pressure, surprising things don't. (Again, that is just the default before we look closer). This doesn't mean we should be totally skeptical about the idea of expecting technological progress or long-term operator devotion, but it does contribute to my being less concerned that something as surprising as extinction would arise from this.

4Wei Dai7y

Yeah, the threat model I have in mind isn't the operator taking over the world or causing an extinction event, but spreading bad but extremely persuasive ideas that can drastically curtail humanity's potential (which is part of the definition of "existential risk"). For example fulfilling our potential may require that the universe eventually be controlled mostly by agents that have managed to correctly solve a number of moral and philosophical problems, and the spread of these bad ideas may prevent that from happening. See Some Thoughts on Metaphilosophy and the posts linked from there for more on this perspective.

2michaelcohen7y

Let XX be the event in which: a virulent meme causes sufficiently many power-brokers to become entrenched with absurd values, such that we do not end up even satisficing The True Good. Empirical analysis might not be useless here in evaluating the "surprisingness" of XX. I don't think Christianity makes the cut either for virulence or for incompatibility with some satisfactory level of The True Good. I'm adding this not for you, but to clarify for the casual reader: we both agree that a Superintelligence setting out to accomplish XX would probably succeed; the question here is how likely this is to happen by accident if a superintelligence tries to get a human in a closed box to love it.

0michaelcohen7y

Can you explain this?

1Wei Dai7y

Suppose there are n forms of mind hacking that the AI could do, some of which are existentially catastrophic. If your plan is "Run this AI, and if the operator gets mind-hacked, stop and switch to an entirely different design." the likelihood of hitting upon an existentially catastrophic form of mind hacking is lower than if the plan is instead "Run this AI, and if the operator gets mind-hacked, tweak the AI design to block that specific form of mind hacking and try again. Repeat until we get a useful answer."

1michaelcohen7y

Hm. This doesn't seem right to me. My approach for trying to form an intuition here includes returning to the example (in a parent comment) but I don't imagine this satisfies you. Another piece of the intuition is that mind-hacking for the aim of reward within the episode, or even the possible instrumental aim of operator-devotion, still doesn't seem very existentially risky to me, given the lack of optimization pressure to that effect. (I know the latter comment sort of belongs in other branches of our conversation, so we should continue to discuss it elsewhere). Maybe other people can weigh in on this, and we can come back to it.

1michaelcohen7y

I'm open to other terminology. Yes, there is no guarantee about what happens to the operator. As I'm defining it, benignity is defined to be not having outside-world instrumental goals, and the intuition for the term is "not existentially dangerous."

2Wei Dai7y

There's still an existential risk in the sense that the AGI has an incentive to hack the operator to give it maximum reward, and that hack could have powerful effects outside the box (even though the AI hasn't optimized it for that purpose), for example it might turn out to be a virulent memetic virus. Of course this is much less risky than if the AGI had direct instrumental goals outside the box, but "benign" and "not existentially dangerous" both seem to be claiming a bit too much. I'll think about what other term might be more suitable.

1michaelcohen7y

The first nuclear reaction initiated an unprecedented temperature in the atmosphere, and people were right to wonder whether this would cause the atmosphere to ignite. The existence of a generally intelligent agent is likely to cause unprecedented mental states in humans, and we would be right to wonder whether that will cause an existential catastrophe. I think the concern of "could have powerful effects outside the box" is mostly captured by the unprecedentedness of this mental state, since the mental state is not selected to have those side effects. Certainly there is no way to rule out side-effects of inside-the-box events, since these side effects are the only reason it's useful. And there is also certainly no way to rule out how those side effects "might turn out to be," without a complete view of the future. Would you agree that unprecedentedness captures the concern?

2Wei Dai7y

I think my concern is a bit more specific than that. See this comment.

2paulfchristiano7y

From the formal description of the algorithm, it looks like you use a universal prior to pick k, and then allow the kth Turing machine to run for ℓ steps, but don't penalize the running time of the machine that outputs k. Is that right? That didn't match my intuitive understanding of the algorithm, and seems like it would lead to strange outcomes, so I feel like I'm misunderstanding.

1michaelcohen7y

Yes this is correct. If you use the same bijection consistently from strings to natural numbers, it looks a little more intuitive than if you don't. The universal prior picks k (the number) by outputting k as a string. The kth Turing machine is the Turing machine described by k as a string. So you end up looking at the Kolmogorov complexity of the description of the Turing machine. So the construction of the description of the world-model isn't time-penalized. This doesn't change the asymptotic result, so I went with the more familiar K(x) rather than translating this new speed prior into measure over finite strings, which would require some more exposition, but I agree with you it feels like there might be some strange outcomes "before the limit" as a result of this approach: namely, the code on the UTM that outputs the description of the world-model-Turing-machine will try to do as much of the computation as possible in advance, by computing the description of an speed-optimized Turing machine for when the actions start coming. The other reasonable choices here instead of K(x) are S(x) (constructed to be like the new speed prior here) and ℓ(x)--the length of x. But ℓ(x) basically tells you that a Turing machine with fewer states is simpler, which would lead to a measure over H∞ that is dominated by world-models that are just universal Turing machines, which defeats the purpose of doing maximum a posteriori instead of a Bayes mixture. The way this issue appears in the proof renders the Natural Prior Assumption less plausible.

3paulfchristiano7y

This invalidates some of my other concerns, but also seems to mean things are incredibly weird at finite times. I suspect that you'll want to change to something less extreme here. (I might well be misunderstanding something, apologies in advance.) Suppose the "intended" physics take at least 1E15 steps to run on the UTM (this is a conservative lower bound, since you have to simulate the human for the whole episode). And suppose β<0.999 (I think you need β much lower than this). Then the intended model gets penalized by at least exp(1E12) for its slowness. For almost the same description complexity, I could write down physics + "precompute the predictions for the first N episodes, for every sequence of possible actions/observations, and store them in a lookup table." This increases the complexity by a few bits, some constant plus K(N|physics), but avoids most of the computation. In order for the intended physics to win, i.e. in order for the "speed" part of the speed prior to do anything, we need the complexity of this precomputed model to be at least 1E12 bits higher than the complexity of the fast model. That appears to happen only once N > BB(1E12). Does that seem right to you? We could talk about whether malign consequentialists also take over at finite times (I think they probably do, since the "speed" part of the speed prior is not doing any work until after BB(1E12) steps, long after the agent becomes incredibly smart), but it seems better to adjust the scheme first. Using the speed prior seems more reasonable, but I'd want to know which version of the speed prior and which parameters, since which particular problem bites you will depend on those choices. And maybe to save time, I'd want to first get your take on whether the proposed version is dominated by consequentialists at some finite time.

1michaelcohen7y

Yes. I recall thinking about precomputing observations for various actions in this phase, but I don’t recall noticing how bad the problem was not in the limit. This goes in the category of “things I can’t rule out”. I say maybe 1/5 chance it’s actually dominated by consequentialists (that low because I think the Natural Prior Assumption is still fairly plausible in its original form), but for all intents and purposes, 1/5 is very high, and I’ll concede this point. 2−K(s)(1+ε) is a measure over binary strings. Instead, let’s try ∑p∈{0,1}∗:U(p)=s2−ℓ(p)βcT(U,p), where ℓ(p) is the length of p, T(U,p) is the time it takes to run p on U, and c is a constant. If there were no cleverer strategy than precomputing observations for all the actions, then c could be above |A|−md, where d is the number of episodes we can tolerate not having a speed prior for. But if it somehow magically predicted which actions BoMAI was going to take in no time at all, then c would have to be above 1/d. What problem do you think bites you?

2paulfchristiano7y

Do you get down to 20% because you think this argument is wrong, or because you think it doesn't apply? What's β? Is it O(1) or really tiny? And which value of c do you want to consider, polynomially small or exponentially small? Wouldn't they have to also magically predict all the stochasticity in the observations, and have a running time that grows exponentially in their log loss? Predicting what BoMAI will do seems likely to be much easier than that.

1michaelcohen7y

You argument is about a Bayes mixture, not a MAP estimate; I think the case is much stronger that consequentialists can take over a non-trivial fraction of a mixture. I think that the methods with consequentialists discover for gaining weight in the prior (before the treacherous turn) are mostly likely to be elegant (short description on UTM), and that is the consequentialists' real competition; then [the probability the universe they live in produces them with their specific goals]or [the bits to directly specify a consequentialist deciding to to do this] set them back (in the MAP context).

2paulfchristiano7y

I don't see why their methods would be elegant. In particular, I don't see why any of {the anthropic update, importance weighting, updating from the choice of universal prior} would have a simple form (simpler than the simplest physics that gives rise to life). I don't see how MAP helps things either---doesn't the same argument suggest that for most of the possible physics, the simplest model will be a consequentialist? (Even more broadly, for the universal prior in general, isn't MAP basically equivalent to a random sample from the prior, since some random model happens to be slightly more compressible?)

2michaelcohen7y

Yeah I think we have different intuitions here; are we at least within a few bits of log-odds disagreement? Even if not, I am not willing to stake anything on this intuition, so I'm not sure this is a hugely important disagreement for us to resolve. I didn't realize that you think that a single consequentialist would plausibly have the largest share of the posterior. I assumed your beliefs were in the neighborhood of: (from your original post on this topic). In a Bayes mixture, I bet that a team of consequentialists that collectively amount to 1/10 or even 1/50 of the posterior could take over our world. In MAP, if you're not first, you're last, and more importantly, you can't team up with other consequentialist-controlled world-models in the mixture.

1michaelcohen7y

Let's say β=0.9, c=1/20. Oh yeah--that's good news.

1michaelcohen7y

Although I don't really like to make anything that would fall apart if the world were deterministic. Relying on stochasticity feels wrong to me.

2cousin_it7y

Are there UDT-ish concerns with breaking isolation of episodes? For example, if the AI receives a low reward at the beginning of episode 117, does it have an incentive to manipulate the external world to make episode 117 happen many times somehow, with most of these times giving it a higher reward? For another example, can the AI at episode 117 realize that it's in a game theory situation with the AI at episodes 116 and 118 and trade rewards with them acausally, leading to long-term goal directed behavior?

1michaelcohen7y

BoMAI is a causal decision theorist.

1Wei Dai7y

It's actually updateless within an episode, right, because it finds the optimal policy at the beginning of an episode and then the same policy is used throughout the episode? I think this means there's no reason for the operator to enter a reward at every timestamp, and instead we could let them just input a total reward at any time in the episode (or they could enter any number of rewards, and the optimization is based on the last reward entered). This would remove the incentive for the AI to keep the operator in the room as long as possible and would partially address item 4 in my comment.

1michaelcohen7y

So "updateless" is a reasonable term to apply to BoMAI, but it's not an updateless decision theorist in your sense (if I understand correctly). An updateless decision theorist picks a policy that has the best consequences, without making assumption that its choice of policy affects the world only through the actions it picks. It considers the possibility that an another agent will be able to perfectly simulate it, so if it picks policy 1 at the start, the other agent will simulate it following policy 1, and if it picks policy 2, the other agent will simulate it picking policy 2. Since this is an effect that isn't mediated by actual choice of action, updatelessness ends up having consequences. If an agent picks an expectimax policy under the assumption that the only way this choice impacts the environments is through the actions it takes (which BoMAI assumes), then it's isomoprhic whether it computes ^ν(i)-expectimax as it goes, or all at once at the beginning. The policy at the beginning will include contingencies for whatever midway-through-the-episode position the agent might land in, and as for what to do at that point, it's the same calculation being run. And this calculation is CDT. I guess this means, and I've never thought about this before so this could easily be wrong, under the assumption that a policy's effect on the world is screened off by which actions it takes, CDT is reflectively stable. (And yes, you could just give one reward, which ends the episode.)

1ErickBall7y

My concern is that since CDT is not reflectively stable, it may have incentives to create non-CDT agents in order to fulfill instrumental goals.

2Wei Dai7y

If I understand correctly, it's actually updateless within an episode, and that's the only thing it cares about so I don't see how it would not be reflectively stable. Plus, even if it had an incentive to create a non-CDT agent, it would have to do that by outputting some message to the operator, and the operator wouldn't have the ability to create a non-CDT agent without leaving the room which would end the episode. (I guess it could hack the operator's mind and create a non-CDT agent within, but at that point it might as well just make the operator give it max rewards.)

1michaelcohen7y

With the correction that it is updateless and CDT (see here), I agree with the rest of this.

1TurnTrout7y

The conclusion seems false; AUP (IJCAI, LW) is a reward maximizer which does not exhibit this behavior. For similar reasons, the recent totalitarian convergence conjecture made here also seems not true.

2michaelcohen7y

AUP seems really promising. I just meant other algorithms that have been proven generally intelligent, which is really just AIXI, the Thompson Sampling Agent, BayesExp, and a couple other variants on Bayesian agents with large model classes.

1ErickBall7y

This may be a dumb question, but how can you asymptotically guarantee human-level intelligence when the world-models have bounded computation time, and the human is a "computable function" that has no such limit? Is it because the number of Turing machines is infinite?

1michaelcohen7y

Not a dumb question; bounded computation time here means bounded computation time per episode, so really it's linear computation time.

[-]michaelcohen7y10

Comment thread: concerns with "the box"

2Joe Collman7y

Just obvious and mundane concerns: You might want to make clearer that "As long as the door is closed, information cannot leave the room" isn't an assumption but a requirement of the setup. I.e. that you're not assuming based on your description that opening the door is the only means for an operator to get information out; you're assuming every other means of information escape has been systematically accounted for and ruled out (with the assumption that the operator has been compromised by the AI).

1michaelcohen7y

Responding to this comment here. I didn't go into the details of the physical design for a really secure box in the paper, because it's not really a topic in computer science, but this is a good excuse to write it up. The operator is inside a glass box. Outside the glass box is a vacuum, and then a concrete box. Outside the concrete box is another vacuum and another concrete box with a Faraday cage around it. There are (airtight) doors through each layer. There is a pressure sensor in both the inner and outer vacuum layer. The computer lives in the inner wall of the inner concrete box; the screen is visible from the inside of the glass box. If the pressure increases in either vacuum layer, the computer's memory is erased, and the whole thing is blown up. There is a laser shining through the glass box (Laser 1). If the operator blocks the path of Laser 1, this corresponds to pressing the button which ends the episode and opens the door. The pressure sensor -> dynamite pathway is deactivated, the vacuum layers are filled with air, and the doors are unlocked. The operator "types" observations and rewards by blocking other lasers that pass through the box. For now, just imagine a few dozen lasers in a row, that the operator can reach up and block in order to type. (With a bit of creativity, and requiring that two lasers be blocked at once to "press a key" this could probably be made somewhat like a qwerty setup.) The lasers and the laser-detectors are all on the inner wall of the inner concrete box. So inside the box is only a human body and some clothes. Let's assume: 1) the only ways for information to move in space are with a) physically moving matter, b) light, and c) gravitational waves and 2) it is impossible for a human being to convert a nuclear reaction into useful work with only their body and clothes. Physically moving packets matter cannot leave the inside of the glass box without releasing air into the vacuum and causing an explosion. Light cannot leave

[-]michaelcohen7y10

Comment thread: concerns with Assumption 3

1michaelcohen7y

Wei Dai's response to this comment represents a concern with Assumption 3.

[-]michaelcohen7y10

Comment thread: concerns with Assumption 2

3Joe Collman7y

[Quite possibly I'm confused, but in case I'm not:] I think this assumption might be invalid (or perhaps require more hand-waving than is ideal). The AI has an incentive to understand the operator's mind, since this bears directly on its reward. Better understanding the operator's mind might be achieved in part by running simulations including the operator. One specific simulation would involve simulating the operator's environment and actions after he leaves the room. Here this isn't done to understand the implications of his actions (which can't affect the episode); it's done to better understand his mind (which can). In this way, one branch of forget/not-forget has two useful purposes (better understand mind and simulate future), while the other has one (better understand mind). So a malign memory-based model needn't be slower than a benign model, if it's useful for that benign model to simulate the future too. So either I'm confused, or the justification for the assumption isn't valid. Hopefully the former :). If I'm right, then what you seem to need is an assumption that simulating the outside-world's future can't be helpful in the AI's prediction of its reward. To me, this seems like major hand-waving territory.

2michaelcohen7y

I wouldn't really use the term "incentives" to describe the free-for-all among world-models as they compete to be maximum a posteriori. All they have to do is output observations and rewards in a distribution that matches the objective probabilities. But I think we arrive at the same possibility: you'll see in the algorithm for ν⋆ that it does simulate the outside-world. I do acknowledge in the paper that some of the outside-world simulation that a memory-based world-model does when it's following the "wrong path" may turn out to be useful; all that is required for the argument to go through is that this simulation is not perfectly useful--there is a shorter computation that accomplishes the same thing. I would love it if this assumption could look like: "the quickest way to simulate one counterfactual does not include simulating a mutually exclusive counterfactual" and make assumption 2 into a lemma that follows from it, but I couldn't figure out how to formalize this.

2Joe Collman7y

Ah yes - I was confusing myself at some point between forming and using a model (hence "incentives"). I think you're correct that "perfectly useful" isn't going to happen. I'm happy to be wrong. I don't think you'd be able to formalize this in general, since I imagine it's not true. E.g. one could imagine a fractal world where every detail of a counterfactual appeared later in a subbranch of a mutually exclusive counterfactual. In such a case, simulating one counterfactual could be perfectly useful to the other. (I suppose you'd still expect it to be an operation or so slower, due to extra indirection, but perhaps that could be optimised away??) To rule this kind of thing out, I think you'd need more specific assumptions (e.g. physics-based).

2Rohin Shah6y

Let the set of potential human explorer actions be AE, and the best human explorer action be a∗E with reward r∗E. Consider the following world model. When asked to predict the result of an action a, it simulates it to find the predicted observation o and reward r. If a∈AE, it outputs o and r faithfully. Otherwise, it outputs any reward it chooses, as long as for the action that it assigns the maximum reward to, it reports r faithfully. In practice, this means that the world model can get BoMAI to choose any action it wants, as long as it is at least as good as the human explorer's best action. This is bad if the world model has malicious inner optimizers. I believe that Assumption 2 is the one responsible for ruling out a model of this form. It seems probably reasonable to say that for actions where it continues simulating the outside world there's useless computation. However, it can also save computation relative to μ∗: while μ∗ must predict o and r perfectly for all actions a, this model can immediately output a null observation and zero reward for any a∉AE that it knows will not align with its goals, rather than spending computation to simulate what rewards those actions would get. Another way of thinking about this is that this model uses consequentialist general intelligence to quickly prune away uninteresting non-human actions to save on computation, but that general intelligence comes at the price of misaligned goals + deceptive behavior. Also, from this comment: I think the model above has arbitrarily bad off-policy predictions, and it's not implausible for it to be the MAP world model forever.

3michaelcohen6y

This is an interesting world-model. So really this is a set of world-models, one for every algorithm for picking actions to present as optimal to BoMAI. Depending on how the actions are chosen by the world-model, either it will be ruled out by Assumption 2 or it will be benign. Suppose the choice of action depends on outside-world features. (This would be the point of manipulating BoMAI--getting it to take actions with particular outside-world effects). Then, the feature that this world-model associates reward with depends on outside-world events that depend on actions taken, and is ruled out by Assumption 2. And as the parenthetical mentions, if the world-model is not selecting actions to advertise as high-reward based on the outside-world effects of those actions, then the world-model is benign. Only the on-policy computation is accounted for.

1Rohin Shah6y

Actually, I'm not sure if the world model I described is memory-based. EDIT: Never mind, see Michael's comment below, the non-benign ones are memory-based. The rewards it outputs are correct, except when it says "the reward is zero", but those exceptions are not causally dependent on outside-world features that causally depend on the actions of the episode. But it also satisfies Lemma 3. So in that case it seems like none of the theoretical arguments prohibit this world model? (I think this is because you are thinking of how the world model can cause BoMAI to have bad incentives, whereas here I'm postulating that the world model itself has malicious goals and is manipulating BoMAI.)

[-]michaelcohen7y10

Comment thread: concerns with Assumption 1

1Wei Dai7y

Since the real world is quantum, does your UTM need to be quantum too? More generally, what happens if there's a mismatch between what computations can be done efficiently in the real world vs on the UTM? Also, I'm not sure what category this question falls under, but can you explain the new speed prior that you use, e.g., what problems in the old speed priors was it designed to solve? (I recall noticing some issues with Schmidhuber's speed prior but can't find the post where I wrote about it now.)

1michaelcohen7y

I feel like I've heard something about quadratic slowdown simulating quantum mechanics on a classical machine? But it doesn't really matter: a simulation that required 100 computation steps per episode on a quantum computer might require 10,000 computation steps per episode on a Turing machine, but it's still bounded.

3Wei Dai7y

These three sources all say simulating a quantum system or computer on a classical computer takes exponential time. Does that make a difference?

2michaelcohen7y

It doesn't make a difference. Thank you for actually looking into this though. I was thinking of this.

1Wei Dai7y

I'm surprised by this. Can you explain a bit more? I was thinking that an exponentially large computation bound for the TM that accurately simulates the real world would make its speed prior so small that it would be practically impossible for the AI to get enough inputs (i.e., messages from the operator) to update on to make that world model have the highest weight in the posterior.

1michaelcohen7y

Yes, in that sense it's bad--I just meant it doesn't make a difference to the theorems or pose a threat to Assumption 1. And it isn't quite that bad. This effect also means you could safely increase β, (penalize slowness less) which causes BoMAI to start entertaining useful models of reality sooner. To be clear, this point mitigates the difficulty you describe; it certainly doesn't go beyond cancelling it out.

3Wei Dai7y

My worry at this point is that if simulating the real world using actual physics takes exponential time on your UTM, the world model with the greatest posterior may not be such a simulation but instead for example an alien superintelligence that runs efficiently on a classical TM which is predicting the behavior of the operator (using various algorithms that it came up with that run efficiently on a classical computer) and at some point the alien superintelligence will cause BoMAI to output something to mind hack the operator and then take over our universe. I'm not sure which assumption this would violate, but do you see this as a reasonable concern?

3paulfchristiano7y

The theorem is consistent with the aliens causing trouble any finite number of times. But each time they cause the agent to do something weird their model loses some probability, so there will be some episode after which they stop causing trouble (if we manage to successfully run enough episodes without in fact having anything bad happen in the meantime, which is an assumption of the asymptotic arguments).

2Wei Dai7y

Thanks. Is there a way to derive a concrete bound on how long it will take for BoMAI to become "benign", e.g., is it exponential or something more reasonable? (Although if even a single "malign" episode could lead to disaster, this may be only of academic interest.) Also, to comment on this section of the paper: "We can only offer informal claims regarding what happens before BoMAI is definitely benign. One intuition is that eventual benignity with probability 1 doesn’t happen by accident: it suggests that for the entire lifetime of the agent, everything is conspiring to make the agent benign." If BoMAI can be effectively controlled by alien superintelligences before it becomes "benign" that would suggest "everything is conspiring to make the agent benign" is misleading as far as reasoning about what BoMAI might do in the mean time. Is this noted somewhere in the paper, or just implicit in the arguments? I guess what we actually need is either a guarantee that all episodes are "benign" or a bound on utility loss that we can incur through such a scheme. (I do appreciate that "in the absence of any other algorithms for general intelligence which have been proven asymptotically benign, let alone benign for their entire lifetimes, BoMAI represents meaningful theoretical progress toward designing the latter.")

1michaelcohen7y

The closest thing to a discussion of this so far is Appendix E, but I have not yet thought through this very carefully. When you ask if it is exponential, what exactly are you asking if it is exponential in?

1Wei Dai7y

I guess I was asking if it's exponential in anything that would make BoMAI impractically slow to become "benign", so basically just using "exponential" as a shorthand for "impractically large".

1michaelcohen7y

I don't think it is, thank you for pointing this out.

1michaelcohen7y

Agreed that would be misleading, but I don't think it would be controlled by alien superintelligences.

1michaelcohen7y

Consider algorithm the alien superintelligence is running to predict the behavior of the operator which runs efficiently on a classical TM (Algorithm A). Now compare Algorithm A with Algorithm B: simulate aliens deciding to run algorithm A; run algorithm A; except at some point, figure out when to do a treacherous turn, and then do it. Algorithm B is clearly slower than Algorithm A, so Algorithm B loses. There is an important conversation to be had here: your particular example isn't concerning, but maybe we just haven't thought of an analog that is concerning. Regardless, I think has become divorced from the discussion about quantum mechanics. This is why I try to write down all the assumptions to rule out a whole host of world-models we haven't even considered. In the argument in the paper, the assumption that rules out this example is the Natural Prior Assumption (assumption 3), although I think for your particular example, the argument I just gave is more straightforward.

1Wei Dai7y

Yes but algorithm B may be shorter than algorithm A, because it could take a lot of bits to directly specify an algorithm that would accurately predict a human using a classical computer, and less bits to pick out an alien superintelligence who has an instrumental reason to invent such an algorithm. If β is set to be so near 1 that the exponential time simulation of real physics can have the highest posterior within a reasonable time, the fact that B is slower than A makes almost no difference and everything comes down to program length. Quantum mechanics is what's making B being slower than A not matter (via the above argument).

1michaelcohen7y

Epistemic status: shady So I'm a bit baffled by the philosophy here, but here's why I haven't been concerned with the long time it would take BoMAI to entertain the true environment (and it might well, given a safe value of β). There is relatively clear distinction one can make between objective probabilities and subjective ones. The asymptotic benignity result makes use of world-models that perfectly match the objective probabilities rising to the top. Consider a new kind of probability: a "k-optimal subjective probability." That is, the best (in the sense of KL divergence) approximation of the objective probabilities that can be sampled from using a UTM and using only k computation steps. Suspend disbelief for a moment, and suppose we thought of these probabilities as objective probabilities. My intuition here is that everything works just great when agents treat subjective probabilities like real probabilities, and to a k-bounded agent, it feels like there is some sense in which these might as well be objective probabilities; the more intricate structure is inaccessible. If no world-models were considered that allowed more than k computation steps per timestep (mk per episode I guess, whatever), then just by calling "k-optimal subjective probabilities" "objective," the same benignity theorems would apply, where the role in the proofs of [the world-model that matches the objective probabilities] is replaced by [the world-model that matches the k-optimal subjective probabilities]. And in this version, i0 comes much sooner, and the limiting value of intelligence is reached much sooner. Of course, "the limiting value of intelligence" is much less, because only fast world-models are considered. But that just goes to show that even if, on a human timescale, BoMAI basically never fields a world-model that actually matches objective probabilities, along the way, it will still be fielding the best ones available that use a more modest computation budget. Once the comp

1michaelcohen7y

If there's an efficient classical approximation of quantum dynamics, I bet this has a concise and lovely mathematical description. I bet that description is much shorter than "in Conway's game of life, the efficient approximation of quantum mechanics that whatever lifeform emerges will probably come up with." But I'm hesitant here. This is exactly the sort of conversation I wanted to have.

3Wei Dai7y

I doubt that there's an efficient classical approximation of quantum dynamics in general. There are probably tricks to speed up the classical approximation of a human mind though (or parts of a human mind), that an alien superintelligence could discover. Consider this analogy. Suppose there's a robot stranded on a planet without technology. What's the shortest algorithm for controlling the robot such that it eventually leaves that planet and reaches another star? It's probably some kind of AGI that has an instrumental goal of reaching another star, right? (It could also be a terminal goal, but there are many other terminal goals that call for interstellar travel as an instrumental goal so the latter seems more likely.) Leaving the planet calls for solving many problems that come up, on the fly, including inventing new algorithms for solving them. If you put all these individual solutions and algorithms together that would also be an algorithm for reaching another star but it could be a lot longer than the code for the AGI.

1michaelcohen7y

I see--so I think I make the same response on a different level then. My model for this is: the world-model is a stochastic simple world, something like Conway's game of life (but with randomness). Life evolves. The output channel has distinguished within-world effects, so that inhabitants can recognize it. The inhabitants control the output channel and use some of their world's noise to sample from a universal prior, which they then feed into the output channel. But they don't just use any universal prior--they use a better one, one which updates the prior over world-models as if the observation has been made: "someone in this world-model is sampling from the universal prior." Maybe they also started with a speed prior of some form (which would cause them to be more likely to output the fast approximation of the human mind we were just discussing). And then after a while, they mess with the output. Whatever better universal prior they come up with (e.g. anthropically updated speed prior), I think has a short description--shorter than [- log prob(intelligent life evolves and picks it) + description of simple universe].

2Wei Dai7y

It doesn't make sense to me that they're sampling from a universal prior and feeding it into the output channel, because the aliens are trying to take over other worlds through that output channel (and presumably they also have a distinguished input channel to go along with it), so they should be focusing on finding worlds that both can be taken over via the channel (including figuring out the computational costs of doing so) and are worth taking over (i.e., offers greater computational resources than their own), and then generating outputs that are optimized for taking over those worlds. Maybe this can be viewed as sampling from some kind of universal prior (with a short description), but I'm not seeing it. If you think it can or should be viewed that way, can you explain more? In particular, if they're trying to take over a computationally richer world, like ours, they have to figure out how to make sufficient predictions about the richer world using their own impoverished resources, which could involve doing research that's equivalent to our physics, chemistry, biology, neuroscience, etc. I'm not seeing how sampling from "anthropically updated speed prior" would do the equivalent of all that (unless you end up sampling from a computation within the prior that consists of some aliens trying to take over our world).

2michaelcohen7y

I think you might be more or less right here. I hadn’t thought about the can-do and the worth-doing update, in addition to the anthropic update. And it’s not that important, but for terminology’s sake, I forgot that the update could send a world-model’s prior to 0, so the prior might not be universal anymore. The reason I think of these steps as updates to what started as a universal prior, is that they would like to take over as many possible worlds as possible, and they don’t know which one. And the universal prior is a good way to predict the dynamics of a world you know nothing about. If you want to make fast predictions about an unknown world, I think that’s what we call a speed prior. Once the alien race has submitted a sequence of observations, they should act as if the observations were largely correct, because that’s the situation in which anything they do matters, so they are basically “learning” about the world they are copying (along with what they get from their input channel, of course, which corresponds to the operator’s actions). Sampling from a speed prior allows the aliens to output quick-to-compute plausible continuations of what they’ve outputted already. Hence, my reduction from [research about various topics] to [sampling from a speed prior]. But—when you add in the can-do update and the worth-doing update, I agree with you that the resulting measure (speed prior + anthropic update + can-do update + worth-doing update) might have a longer description than the measure which starts like that, then takes a treacherous turn. This case seems different to me (so I don’t make the same objection on this level) because the can-do update and the worth-doing update are about this treacherous turn. So let me back up here. I don’t say anything in the Natural Prior Assumption about “for sufficiently small β,” but this makes me think I might need to. As I suggested above, I do think there is huge computational overhead that comes from having evolved life

2Wei Dai7y

I'm glad that I'm getting some of my points across, but I think we still have some remaining disagreements or confusions here. That doesn't seem right to me. A speed prior still favors short algorithms. If you're trying to make predictions about a computationally richer universe, why favor short algorithms? Why not apply your intelligence to try to discover the best algorithm (or increasingly better algorithms), regardless of the length? Also, sampling from a speed prior involves randomizing over a mixture of TMs, but from an EU maximization perspective, wouldn't running one particular TM from the mixture give the highest expected utility? Why are the aliens sampling from the speed prior instead of directly picking a specific algorithm to generate the next output, one that they expect to give the highest utility for them? What happens if β is too small? If it's really tiny, then the world model with the highest posterior is random, right, because it's "computed" by a TM that (to minimize run time) just copies everything on its random tape to the output? And as you increase β, the TM with highest posterior starts doing fast and then increasingly compute-intensive predictions? I think if β is small but not too small, the highest posterior would not involve evolved life, but instead a directly coded AGI that runs "natively" on the TM who can decide to execute arbitrary algorithms "natively" on the TM. Maybe there is still some range of β where BoMAI is both safe and useful (can answer sophisticated questions like "how to build a safe unbounded AGI") because in that range the highest posterior is a good non-life/non-AGI prediction algorithm. But A) I don't know an argument for that, and B) even if it's true, to take advantage of it would seem to require fine tuning β and I don't see how to do that, given that trial-and-error wouldn't be safe.

3michaelcohen7y

At the end of the day, it will be running some subroutine for its gain trust/predict accurately phase. I assume this sort of thing is true for any model of computation, but when you construct a universal Turing machine, so that it can simulate computation step after computation step of another Turing machine, it takes way more than one computation step for each one. If the AGI is using machinery that would allow it to simulate any world-model, it will be way slower than the Turing machine built for that algorithm. I realize this seems really in-the-weeds and particular, but I think this is a general principle of computation. The more general a system is, the less well it can do any particular task. I think an AGI that chose to pipe viable predictions to the output with some procedure will be slower than the Turing machine which just runs that procedure.

2paulfchristiano7y

I don't buy it. All your programs are already running on UTM M. Just consider a program that gives the aliens the ability to write arbitrary functions in M and then pass control to them. That program is barely any bigger (all you have to do is insert one use after free in physics :) ), and guarantees the aliens have zero slowdown. For the literal simplest version of this, your program is M(Alien(), randomness), which is going to run just as fast as M(physics, randomness) for the intended physics, and probably much faster (if the aliens can think of any clever tricks to run faster without compromising much accuracy). The only reason you wouldn't get this is if Alien is expensive. That probably rules out crazy alien civilizations, but I'm with Wei Dai that it probably doesn't rule out simpler scientists.

2Wei Dai7y

That's what I was thinking too, but Michael made me realize this isn't possible, at least for some M. Suppose M is the C programming language, but in C there is no way to say "interpret this string as a C program and run it as fast as a native C program". Am I missing something at this point? I don't understand this sentence.

1paulfchristiano7y

I agree this is only going to be possible for some universal Turing machines. Though if you are using a Turing machine to define a speed prior, this does seem like a desirable property. If physics is implemented in C, there are many possible bugs that would allow the attacker to execute arbitrary C code with no slowdown.

1Wei Dai7y

Why is it a desirable property? I'm not seeing why it would be bad to choose a UTM that doesn't have this property to define the speed prior for BoMAI, if that helps with safety. Please explain more?

1paulfchristiano7y

I just mean: "universality" in the sense of a UTM isn't a sufficient property when defining the speed prior, the analogous property of the UTM is something more like: "You can run an arbitrary Turing machine without too much slowdown." Of course that's not possible, but it seems like you still want to be as close to that as possible (for the same reasons that you wanted universality at all). I agree that it would be fine to sacrifice this property if it was helpful for safety.

1michaelcohen7y

Each world-model is a Turing machine, whose prior relates to the Kolmogorov complexity (on some universal Turing machine) of the description of Turing machine--all the transition rules, and whatnot. Usually, this would be isomorphic (within a constant), but since we're considering speed, programs actually aren't simulated on a UTM.

2Wei Dai7y

Ok, I see, so in other words the AGI doesn't have the ability to write an arbitrary function in the base programming language and call it, it has a fixed code base and has to simulate that function using its existing code. However I think the AGI can still win a race against a straightforward "predict accurately" algorithm, because it can to two things. 1) Include the most important inner loops of the "predict accurately" algorithm as functions in its own code to minimize the relative slowdown (this is not a decision by the AGI but just a matter of which AGI ends up having the highest posterior) and 2) keep finding improvements to its own prediction algorithm so that it can eventually overtake any fixed prediction algorithm in accuracy which hopefully more than "pays for" the remaining slowdown that is incurred.

1michaelcohen7y

Let the AGI's "predict accurately" algorithm be fixed. What you call a sequence of improvements to the prediction algorithm, let's just call that the prediction algorithm. Imagine this to have as much or as little overhead as you like compared to what was previously conceptualized as "predict accurately." I think this reconceptualization eliminates 2) as a concern, and if I'm understanding correctly, 1) is only able to mitigate slowdown, not overpower it. Also I think 1) doesn't work--maybe you came to this conclusion as well? But maybe you're saying that doesn't apply because: I think this way throws off the contention that this AGI will have a short description length. One can imagine a sliding scale here. Short description, lots of overhead: a simple universe evolves life, aliens decide to run "predict accurately" + "treacherous turn". Longer description, less overhead: an AGI that runs "predict accurately" + "treacherous turn." Longer description, less overhead: an AGI with some of the subroutines involved already (conveniently) baked in to its architecture. Once all the subroutines are "baked into its architecture" you just have: the algorithm "predict accurately" + "treacherous turn". And in this form, that has a longer description than just "predict accurately".

1paulfchristiano7y

You only have to bake in the innermost part of one loop in order to get almost all the computational savings.

1michaelcohen7y

I've made a case that the two endpoints in the trade-off are not problematic. I've argued (roughly) that one reduces computational overhead by doing things that dissociate the naturalness of describing "predict accurately" and "treacherous turn" all at once. This goes back to the general principle I proposed above: "The more general a system is, the less well it can do any particular task." The only thing I feel like I can still do is argue against particular points in the trade-off that you think are likely to cause trouble. Can you point me to an exact inner loop that can be native to an AGI that would cause this to fall outside of this trend? To frame this case, the Turing machine description must specify [AGI + a routine that it can call]--sort of like a brain-computer interface, where the AGI is the brain and the fast routine is the computer.

1paulfchristiano7y

(I actually have a more basic confusion, started a new thread.)

1michaelcohen7y

Just as you said: it outputs Bernoulli(1/2) bits for a long time. It's not dangerous. Fine tuning from both sides isn't safe. Approach from below.

1Wei Dai7y

I just read the math more carefully, and it looks like no matter how small β is, as long as β is positive, as BoMAI receives more and more input, it will eventually converge to the most accurate world model possible. This is because the computation penalty is applied to the per-episode computation bound and doesn't increase with each episode, whereas the accuracy advantage gets accumulated across episodes. Assuming that the most accurate world model is an exponential-time quantum simulation, that's what BoMAI will converge to (no matter how small β is), right? And in the meantime it will go through some arbitrarily complex (up to some very large bound) but faster than exponential classical approximations of quantum physics that are increasingly accurate, as the number of episodes increase? If so, I'm no longer convinced that BoMAI is benign as long as β is small enough, because the qualitative behavior of BoMAI seems the same no matter what β is, i.e., it gets smarter over time as its world model gets more accurate, and I'm not sure why the reason BoMAI might not be benign at high β couldn't also apply at low β (if we run it for a long enough time). (If you're going to discuss all this in your "longer reply", I'm fine with waiting for it.)

1michaelcohen7y

The longer reply will include an image that might help, but a couple other notes. If it causes you to doubt the asymptotic result, it might be helpful to read the benignity proof (especially the proof of Rejecting the Simple Memory-Based Lemma, which isn't that long). The heuristic reason for why it can be helpful to decrease β for long-run behavior, even though long-run behavior is qualitatively similar, is that while accuracy eventually becomes the dominant concern, along the way the prior is *sort of* a random perturbation to this which changes the posterior weight, so for two world-models that are exactly equally accurate, we need to make sure the malign one is penalized for being slower, enough to outweigh the inconvenient possible outcome in which it has shorter description length. Put another way, for benignity, we don't need concern for speed to dominate concern for accuracy; we need it to dominate concern for "simplicity" (on some reference machine).

1Wei Dai7y

Yeah, I understand this part, but I'm not sure why, since the benign one can be extremely complex, the malign one can't have enough of a K-complexity advantage to overcome its slowness penalty. And since (with low β) we're going through many more different world models as the number of episodes increases, that also gives malign world models more chances to "win"? It seems hard to make any trustworthy conclusions based on the kind of informal reasoning we've been doing and we need to figure out the actual math somehow.

1michaelcohen7y

Check out the order of the quantifiers in the proofs. One β works for all possibilities. If the quantifiers were in the other order, they couldn't be trivially flipped since the number of world-models is infinite, and the intuitive worry about malign world-models getting "more chances to win" would apply. Let's continue the conversation here, and this may be a good place to reference this comment.

1Wei Dai7y

Sure, approaching from below is obvious, but that still requires knowing how wide the band of β that would produce a safe and useful BoMAI is, otherwise even if the band exists you could overshoot it and end up in the unsafe region. ETA: But the first question is, is there a β such that BoMAI is both safe and intelligent enough to answer questions like "how to build a safe unbounded AGI"? When β is very low BoMAI is useless, and as you increase β it gets smarter, but then at some point with a high enough β it becomes unsafe. Do you know a way to figure out how smart BoMAI is just before it becomes unsafe?

1michaelcohen7y

Some visualizations which might help with this: But then one needs to factor in "simplicity" or the prior penalty from description length: Note also that these are average effects; they are just for forming intuitions. Your concern was: This was the sort of thing I assumed could be improved upon later once the asymptotic result was established. Now that you’re asking for the improvement, here’s a proposal: Set β safely. Once enough observations have been provided that you believe human-level AI should be possible, exclude world-models that use less than s←1 computation steps per episode. Every episode, increase s until human-level performance is reached. Under the assumption that the average computation time of a malign world-model is at least a constant times that of the “corresponding” benign one (corresponding in the sense of using the same ((coarse) approximate) simulation of the world), then s←αs should be safe for some α>1 (and α−1≉0). I need to think more carefully about what happens here, but I think the design space is large.

1michaelcohen7y

Longer response coming. On hold for now.

1michaelcohen7y

All the better. They don't what know universe is using the prior. What are the odds our universe is the single most susceptible universe to being taken over? I was assuming the worst, and guessing that there are diminishing marginal returns once your odds of a successful takeover get above ~50%, so instead of going all in on accurate predictions of the weakest and ripest target universe, you hedge and target a few universes. And I was assuming the worst in assuming they'd be so good at this, they'd be able to do this for a large number of universes at once. To clarify: diminishing marginal returns of takeover probability of a universe with respect to the weight you give that universe in your prior that you pipe to output.

2paulfchristiano7y

There are massive diminishing marginal returns; in a naive model you'd expect essentially *every* universe to get predicted in this way. But Wei Dai's basic point still stands. The speed prior isn't the actual prior over universes (i.e. doesn't reflect the real degree of moral concern that we'd use to weigh consequences of our decisions in different possible worlds). If you have some data that you are trying to predict, you can do way better than the speed prior by (a) using your real prior to estimate or sample from the actual posterior distribution over physical law, (b) using engineering reasoning to make the utility maximizing predictions, given that faster predictions are going to get given more weight. (You don't really need this to run Wei Dai's argument, because there seem to be dozens of ways in which the aliens get an advantage over the intended physical model.)

1michaelcohen7y

I think what you're saying is that the following don't commute: "real prior" (universal prior) + speed update + anthropic update + can-do update + worth-doing update compared to universal prior + anthropic update + can-do update + worth-doing update + speed update When universal prior is next to speed update, this is naturally conceptualized as a speed prior, and when it's last, it is naturally conceptualized as "engineering reasoning" identifying faster predictions. I happy to go with the second order if you prefer, in part because I think they do commute--all these updates just change the weights on measures that get mixed together to be piped to output during the "predict accurately" phase.

1michaelcohen7y

You have a countable list of options. What choice do you have but to favor the ones at the beginning? Any (computable) permutation of the things on the list just corresponds to a different choice of universal Turing machine for which a "short" algorithm just means it's earlier on the list. And a "sequence of increasingly better algorithms," if chosen in a computable way, is just a computable algorithm.

2paulfchristiano7y

The fast algorithms to predict our physics just aren't going to be the shortest ones. You can use reasoning to pick which one to favor (after figuring out physics), rather than just writing them down in some arbitrary order and taking the first one.

1michaelcohen7y

Using "reasoning" to pick which one to favor, is just picking the first one in some new order. (And not really picking the first one, just giving earlier ones preferential treatment). In general, if you have an infinite list of possibilities, and you want to pick the one that maximizes some property, this is not a procedure that halts. I'm agnostic about what order you use (for now) but one can't escape the necessity to introduce the arbitrary criterion of "valuing" earlier things on the list. One can put 50% probability mass on the first billion instead of the first 1000 if one wants to favor "simplicity" less, but you can't make that number infinity.

1paulfchristiano7y

Yes, some new order, but not an arbitrary one. The resulting order is going to be better than the speed prior order, so we'll update in favor of the aliens and away from the rest of the speed prior. Probably some miscommunication here. No one is trying to object to the arbitrariness, we're just making the point that the aliens have a lot of leverage with which to beat the rest of the speed prior. (They may still not be able to if the penalty for computation is sufficiently steep---e.g. if you penalize based on circuit complexity so that the model might as well bake in everything that doesn't depend on the particular input at hand. I think it's an interesting open question whether that avoids all problems of this form, which I unsuccessfully tried to get at here.)

1michaelcohen7y

It was definitely reassuring to me that someone else had had the thought that prioritizing speed could eliminate optimization daemons (re: minimal circuits), since the speed prior came in here for independent reasons. The only other approach I can think of is trying to do the anthropic update ourselves.

2paulfchristiano7y

If you haven't seen Jessica's post in this area, it's worth taking a quick look.

1michaelcohen7y

The only point I was trying to respond to in the grandparent of this comment was your comment Your concern (I think) is that our speed prior would assign a lower probability to [fast approximation of real world] than the aliens' speed prior. I can't respond at once to all of the reasons you have for this belief, but the one I was responding to here (which hopefully we can file away before proceeding) was that our speed prior trades off shortness with speed, and aliens could avoid this and only look at speed. My point here was just that there's no way to not trade off shortness with speed, so no one has a comparative advantage on us as result of the claim "The fast algorithms to predict our physics just aren't going to be the shortest ones." The "after figuring out physics" part is like saying that they can use a prior which is updated based on evidence. They will observe evidence for what our physics is like, and use that to update their posterior, but that's exactly what we're doing to. The prior they start with can't be designed around our physics. I think that the only place this reasoning gets you is that their posterior will assign a higher probability to [fast approximation of real world] than our prior does, because the world-models have been reasonably reweighted in light of their "figuring out physics". Of course I don't object to that--our speed prior's posterior will be much better than the prior too.

1paulfchristiano7y

It seems totally different from what we're doing, I may be misunderstanding the analogy. Suppose I look out at the world and do some science, e.g. discovering the standard model. Then I use my understanding of science to design great prediction algorithms that run fast, but are quite complicated owing to all of the approximations and heuristics baked into them. The speed prior gives this model a very low probability because it's a complicated model. But "do science" gives this model a high probability, because it's a simple model of physics, and then the approximations follow from a bunch of reasoning on top of that model of physics. We aren't trading off "shortness" for speed---we are trading off "looks good according to reasoning" for speed. Yes they are both arbitrary orders, but one of them systematically contains better models earlier in the order, since the output of reasoning is better than a blind prioritization of shorter models. Of course the speed prior also includes a hypothesis that does "science with the goal of making good predictions," and indeed Wei Dai and I are saying that this is the part of the speed prior that will dominate the posterior. But now we are back to potentially-malign consequentialistism. The cognitive work being done internally to that hypothesis is totally different from the work being done by updating on the speed prior (except insofar as the speed prior literally contains a hypothesis that does that work). In other words: Suppose physics takes n bits to specify, and a reasonable approximation takes N >> n bits to specify. Then the speed prior, working in the intended way, takes N bits to arrive at the reasonable approximation. But the aliens take n bits to arrive at the standard model, and then once they've done that can immediately deduce the N bit approximation. So it sure seems like they'll beat the speed prior. Are you objecting to this argument? (In fact the speed prior only actually takes n + O(1) bits, because it ca

1michaelcohen7y

I'm not sure which of these arguments will be more convincing to you. This is what is what I was trying to contextualize above. This is an unfair comparison. You're imagining that the "reasoning"-based order gets to see past observations, and the "shortness"-based order does not. A reasoning-based order is just a shortness-based order that has been updated into a posterior after seeing observations (under the view that good reasoning is Bayesian reasoning). Maybe the term "order" is confusing us, because we both know it's a distribution, not an order, and we were just simplifying to a ranking. A shortness-based order should really just be called a prior, and a reasoning-based order (at least a Bayesian-reasoning-based order) should really just be called a posterior (once it has done some reasoning; before it has done the reasoning, it is just a prior too). So yes, the whole premise of Bayesian reasoning is that updating based on reasoning is a good thing to do. Here's another way to look at it. The speed prior is doing the brute force search that scientists try to approximate efficiently. The search is for a fast approximation of the environment. The speed prior considers them all. The scientists use heuristics to find one. Exactly. But this does help for reasons I describe here. The description length of the "do science" strategy (I contend) is less than the description length of the "do science" + "treacherous turn" strategy. (I initially typed that as "tern", which will now be the image I have of a treacherous turn.)

1paulfchristiano7y

Reasoning gives you a prior that is better than the speed prior, before you see any data. (*Much* better, limited only by the fact that the speed prior contains strategies which use reasoning.) The reasoning in this case is not a Bayesian update. It's evaluating possible approximations *by reasoning about how well they approximate the underlying physics, itself inferred by a Bayesian update*, not by directly seeing how well they predict on the data so far. I can reply in that thread. I think the only good arguments for this are in the limit where you don't care about simplicity at all and only care about running time, since then you can rule out all reasoning. The threshold where things start working depends on the underlying physics, for more computationally complex physics you need to pick larger and larger computation penalties to get the desired result.

1michaelcohen7y

Given a world model ν, which takes k computation steps per episode, let νlog be the best world-model that best approximates ν (in the sense of KL divergence) using only logk computation steps. νlog is at least as good as the “reasoning-based replacement” of ν. The description length of νlog is within a (small) constant of the description length of ν. That way of describing it is not optimized for speed, but it presents a one-time cost, and anyone arriving at that world-model in this way is paying that cost. One could consider instead νlogε, which is, among the world-models that ε-approximate ν in less than logk computation steps (if the set is non-empty), the first such world-model found by a searching procedure ψ. The description length of νlogε is within a (slightly larger) constant of the description length of ν, but the one-time computational cost is less than that of νlog. νlog, νlogε, and a host of other approaches are prominently represented in the speed prior. If this is what you call “the speed prior doing reasoning,” so be it, but the relevance for that terminology only comes in when you claim that “once you’ve encoded ‘doing reasoning’, you’ve basically already written the code for it to do the treachery that naturally comes along with that.” That sense of “reasoning” really only applies, I think, to the case where our code is simulating aliens or an AGI.

1paulfchristiano7y

(ETA: I think this discussion depended on a detail of your version of the speed prior that I misunderstood.) To be clear, that description gets ~0 mass under the speed prior, right? A direct specification of the fast model is going to have a much higher prior than a brute force search, at least for values of β large enough (or small enough, however you set it up) to rule out the alien civilization that is (probably) the shortest description without regard for computational limits. Within this chunk of the speed prior, the question is: what are good ψ? Any reasonable specification of a consequentialist would work (plus a few more bits for it to understand its situation, though most of the work is done by handing it ν), or of a petri dish in which a consequentialist would eventually end up with influence. Do you have a concrete alternative in mind, which you think is not dominated by some consequentialist (i.e. a ψ for which every consequentialist is either slower or more complex)?

1michaelcohen7y

Well one approach is in the flavor of the induction algorithm I messaged you privately about (I know I didn't give you a completely specified algorithm). But when I wrote that, I didn't have a concrete algorithm in mind. Mostly, it just seems to me that the powerful algorithms which have been useful to humanity have short descriptions in themselves. It seems like there are many cases where there is a simple "ideal" approach which consequentialists "discover" or approximately discover. A powerful heuristic search would be one such algorithm, I think. I don't think anything here changes if K(x) were replaced with S(x) (if that was what you misunderstood).

2Wei Dai7y

True but I'm arguing that this computable algorithm is just the alien itself, trying to answer the question "how can I better predict this richer world in order to take it over?" If there is no shorter/faster algorithm that can come up with a sequence of increasingly better algorithms, what is the point of saying that the alien is sampling from the speed prior, instead of saying that the alien is thinking about how to answer "how can I better predict this richer world in order to take it over?" Actually if this alien was sampling from the speed prior, then it would no longer be the shortest/fastest algorithm to come up with a sequence of increasingly better algorithms, and some other alien trying to take over our world would have the highest posterior instead.

1michaelcohen7y

I'm having a hard time following this. Can you expand on this, without using "sequence of increasingly better algorithms"? I keep translating that to "algorithm."

[-]michaelcohen7y00

Comment thread: adding to the prize pool

If you would like to contribute, please comment with the amount. If you have venmo, please send the amount to @Michael-Cohen-45. If not, we can discuss.

AI ALIGNMENT FORUM
AF

AI ALIGNMENT FORUM
AF

18

18