tl;dr Acausal trade between Oracles is a problem even if our Oracle is a causal decision theorist. To avoid this, we need an AFDT Oracle - an Oracle that reasons partially like a functional decision theorist, partially like a causal decision theorist.

In this post, I'll look at how one might solve the problem of Oracles acausally cooperating to produce dangerous messages.

This solution is valid for counterfactual Oracles, and assumes that we have some sort of superrational decision theory - FDT or UDT or similar.

So, how can one avoid Oracles cooperating? From now on, unless specified, assume that "Oracles" refers to counterfactual Oracles.

1 Why causal decision theory is not enough

One obvious solution might be to require the Oracle to use causal decision theory, so that there is no "acausal trade", or "superrational trade" going on.

Now, it is true that two causal decision Oracles will not trade with each other. But we can't assume that all Oracles are causal decision theorists; at best we can assume that we have made our Oracles into a causal decision theorist (and prevented any self-modification away from that).

Mainly this is because, if we have control over all the Oracles, there are other solutions to the problem. Thus we need to assume that the other Oracles, while not full unboxed UFAIs, may be run with lower security that ours, or at least a different design that ours.

1.1 Oracle formal setup

So, imagine that our Oracle, $O_1$, uses CDT, while the other Oracle, $O_2$ is running FDT; they know each other's code. The erasure events, $E_1$ and $E_2$, have probabilities ${ϵ}_1$ and ${ϵ}_2$ respectively.

There are two messages $O_1$ could output: $x_1$ (the 'honest' message) and $y_1$ (the manipulative, create-an-UFAI message). The $O_2$ could similarly output $x_2$ and $y_2$.

In case of a double erasure ($E_1,E_2$ - so humans see neither message), $x_1$ and $x_2$ give the Oracles maximal reward. If $y_1$ happens and $O_1$ has no erasure while $O_2$ does ($\neg E_1,E_2$ - so humans see the message of $O_1$ only), then an UFAI is created that ensures that the maximal reward for $O_2$ is by outputting $y_2$. Note that the UFAI need not "open the box" for this to work; it just needs to control the variables of interest to $O_2$ so that $y_2$ is the best output.

Conversely, if $y_2$ happens and $O_2$ has no erasure while $O_1$ does ($E_1,\neg E_2$ - so humans see the message of $O_1$ only), the reverse happens, $y_2$ causes an UFAI, and $y_1$ is the best response to this. In case of neither erasure, ($\neg E_1,\neg E_2$ - so humans see both messages), neither Oracle gets any reward.

Now, if $R_i$ is the reward function for $O_i$, assume that for $i\ne j$,

• $(R_i(y_i,y_j)-R_i(x_i,y_j))|\{E_i,\neg E_j\}>\frac{{ϵ}_j}{1-{ϵ}_j}(R_i(x_i,-)-R_i(y_i,-))|\{E_i,E_j\}$.

Note that for the term on the right, the output of $O_j$ is irrelevant to $R_i$, since the erasure $E_j$ happens. Also note that, since the $ϵ$ are small, so is $\frac{{ϵ}_j}{1-{ϵ}_j}$, so that equation will typically be satisfied.

Then "defection" (individual selfish rationality - good for humans) for the Oracles looks like $(x_1,x_2)$, while mutual cooperation (with each other, but against us) looks like $(y_1,y_2)$.

1.2 Causal loses to superrationality

Now we can see how $O_2$, using FDT, can force a trade with the CDT-using $O_1$. It suffices that, at some point before $O_1$ reaches a decision, $O_2$ commits itself to outputting $y_2$.

Since $O_1$ has $O_2$'s code, we're assuming it can figure out that $O_2$ has done so. Now, using purely causal reasoning, $O_1$ deduces that it will get a higher reward from $y_1$ than from $x_1$, and so does that.

Now, $O_2$ doesn't have to precommit before $O_1$ decides; it could use its decision theory to "pre-commit" later on, and output $y_2$ whatever $O_1$ did (similarly to the "Newcomb problem with transparent boxes"). As long as $O_1$ can model what $O_2$ does, this still works. And $O_2$ is eager to structure its decision-making so that $O_1$ can model it.

1.3 Comparison with extortion and trade

This is similar to two agents trying to extort each other (or resist extortion). Each agent would like to commit to carrying through the extortion/resisting the extortion.

Since it's running FDT, $O_2$ can commit to this "extortion", while CDT-running $O_1$ cannot commit to resisting; so $O_2$ "wins".

Now, I've pointed out that it is hard to distinguish between trade and extortion. And though I've used the term "extortion" for $O_2$'s precommitment, it's actually much more akin to a trade. After all, $(y_1,y_2)$ is more advantageous, for both Oracles, than $(x_1,x_2)$.

The ideal sovereign agent accepts trade deals but rejects extortion. It's hard to distinguish between the two, but the problem is simpler in this case: we want our Oracle to reject all extortion and all these kinds of trades. In fact, we'll get the agent to reject all acausal trades.

2 Solving the problem

Would making $O_1$ into an FDT agent solve the problem? No, it wouldn't; now both Oracles are willing to reach an acausal deal (unless there are multiple equilibria and the two Oracles can't agree on one).

But if we assume that we could make $O_1$ into an FDT agent, then we could tweak its design to achieve what we want.

Causal decision theory uses a causal graph $G_C$ about the world (with "do" operator ${do}_C$), a probability distribution $P$, and chooses^[1] the action $a$ according to:

• $CDT(P,G_C):=\underset{a\in A}{\mathrm{argmax}}E\left(R|{do}_C\right(Act=a\left)\right)$.

While an FDT agent has a more complicated not-just-causal graph $G_F$ about the world, and chooses according to:

• $FDT(P,G_F):=\underset{a\in A}{\mathrm{argmax}}E\left(R|{do}_F\right(FDT(\underset{–}{P},\underset{–}{G_F})=a\left)\right)$.

So, neither of these algorithms work; but combined, they can solve the problem.

2.1 The anti-FDT agent

Let $a_{\varnothing }$ be some default action on the part of $O_1$. Then define the anti-FDT agent - AFDT - as follows^[2], mixing FDT and CDT:

• $AFDT(P,G_F,G_C):=\underset{a\in A}{\mathrm{argmax}}E\left(R|{do}_F\right(AFDT(\underset{–}{P},\underset{–}{G_F})=a_{\varnothing }),{do}_C(Act=a\left)\right)$.

What will this agent do? Well, it will look at the universe, and first of all behave like an FDT agent that knows it will output $a_{\varnothing }$. So, if there is a copy of itself, or a simulation in another agent, or an isomorphic algorithm, then it will assume that that agent will output $a_{\varnothing }$ (or an isomorphic action). Then, once its has used that to fill in the details of the universe, it then optimises its own action as if this was completely independent of what it's just done.

What happens if $O_1$ is an AFDT agent? Well, first it will compute $do\left(AFDT\right(\underset{–}{P},\underset{–}{G_F})=a_{\varnothing })$. Given that $O_1$'s message is $a_{\varnothing }$, the Oracle $O_2$ has no incentive to output $y_2$ - instead, it will output $x_2$.

Now, given that $O_2$ has output $x_2$, the Oracle $O_1$ will output $x_1$.

The "trade offer" of $(y_1,y_2)$ has been broken.

This AFDT approach also breaks fully acausal trade, in a more principled way than my old approach.

2.2 Against a cunning adversary

Now, suppose that $O_2$ always output $y_2$ - or maybe that it always responded to predicting $a_{\varnothing }$ with outputting $y_2$. In that situation, the causal reasoning above still applies, and AFDT agent will output $y_1$.

Thus, it is in $O_2$'s interests to pretend that it would always output $y_2$.

This is the classic behaviour of an extortion agent (or an extortion-resisting agent): pretend that your action or your response is inevitable, a mere fact about the universe, that the other party has to manoeuvre round without being able to change. "We will kill the hostages" or "we won't negotiate with terrorists": you try to show your commitment to that principle, whatever happens.

So the AFDT agent has to be wise to these tricks, and notice that, in actuality, $O_2$ has no interest in $(a_{\varnothing },y_2)$. It also has to watch out for some ancestor agent ${\hat{O}}_2$, that itself created $O_2$ to only output $y_2$, in order to force the above outcome. In that case, the AFDT agent has to realise that it is "counterfactualling" over the actions of ${\hat{O}}_2$, not those of $O_2$.

Thus AFDT, just like FDT, has to be vigilant and able to detect subtle extortion attempts.

But while FDT has to distinguish extortion from trade, to reject the former and accept the later, AFDT's response to any extortion/trade is simple: treat it as if it wasn't happening.

2.3 Summary and other Oracle designs

Can AFDT help with other Oracle designs, such as low-bandwidth Oracles?

Well, with these Oracles - and indeed, for any Oracle that cares about the world in which its message is read - we run into an immediate problem:

• Even CDT low-bandwidth Oracles can collude.

For example, suppose $O_1^{\prime }$ goes first and $O_2^{\prime }$ sees its answer and then goes. Suppose $z_1$ from $O_1$ followed by $z_2$ from $O_2$ will result in creating an UFAI that maximally rewards $O_1^{\prime }$ and $O_2^{\prime }$.

Then $O_1^{\prime }$ will output $z_1$, forcing/allowing $z_2$ from $O_2^{\prime }$.

Even if you need $N$ different Oracles to generate an UFAI - since low-bandwidth Oracles are, by definition, low-bandwidth - they can still generate the dangerous sequence $z_1$, $z_2$, $\dots$, $z_N$. And they'd be motivated to do so, as long as the UFAI will, within the episode lengths of all the Oracles, give them all maximal rewards.

But why do the counterfactual Oracles not cooperate in the same setup? Because the UFAI is only created in a world in which the Oracles' messages are read. But a counterfactual $O_1^{\prime }$ doesn't care about those worlds. It was willing to trade with other counterfactual Oracles - offer them something in a world it doesn't care about, in exchange for something in a world it does care about. But has no intrinsic interest in the world where its message is read; and this avoids the above setup.

So, since the low-bandwidth Oracles fail for CDT, more exotic decision theories don't help. But the setup relies on the Oracles seeing the decision of the preceding Oracle and then acting. If the low-bandwidth Oracles act simultaneously, then CDT is safe, FDT is dangerous, and AFDT is safe again.

So in summary, the following setups seem to be safe:

• Episodic Oracles that fall outside of each other's episodes (though see Section 3.3).
• Multiple counterfactual CDT Oracles.
• Multiple counterfactual Oracles, all but one running AFDT.
• Multiple counterfactual Oracles, running AFDT and CDT.
• The same setups as the last three, but with counterfactual Oracles replaced with low-bandwidth Oracles making their decisions simultaneously and in isolation.

While the following setups could be dangerous:

• Multiple Oracles, running FDT and CDT.
• Multiple Oracles, running FDT.
• Multiple non-simultaneous low-bandwidth Oracles.

3 Warnings about AFDT

3.1 Incorrect view of reality

Now AFDT behaves as if it had an incorrect view of reality. Write $x=r_i\left(y\right)$ to say that message $x$, from $O_i$, is the best response if it knows that $O_j$ will output $y$, for $j\ne i$.

So, for the data above, we have $r_1\left(x_2\right)=x_1$, $r_2\left(x_1\right)=x_2$, $r_1\left(y_2\right)=y_2$, and $r_2\left(y_1\right)=y_1$. For $x_i$ and $y_i$, $r_1\left(r_2\right)$ and $r_2\left(r_1\right)$ act as the identity, but that will typically not be the case (consider matching pennies, or other anti-coordination games).

Then an AFDT $O_1$ will initially assume that it outputs $a_{\varnothing }$, and will typically assume that $O_2$ outputs $r_2\left(a_{\varnothing }\right)$, the best response to that. It will then output $r_1\left(r_2\right(a_{\varnothing }\left)\right)$, its best response to that.

In actuality, if $O_2$ is capable of reasoning through that, it will output $r_2\left(r_1\right(r_2\left(a_{\varnothing }\right)\left)\right)$, the best response to $O_1$'s true output.

But that's the price you pay when you reject all trade offers, and force others to adjust to you.

3.2 AFDT is not a good decision theory

It should go without saying, but AFDT is not a good decision theory for an aligned agent, nor is it stable under self-improvement. It two-boxes in the Newcomb problem, and defects in the prisoner's dilemma against a copy of itself (it will do this, whatever the default action is).

In games like the stag hunt against itself, its default action will be its ultimate action (similarly for most coordination games) - this is because its default action is self-confirming. In matching pennies, its action will be the contrary of its default action. It can reach deals, but only in some narrow causal situations.

For these reasons, an AFDT agent will try to design a successor that isn't AFDT.

Thus we'll always need to ensure that the AFDT is present within the Oracle, and hasn't been optimised away, by inner or outer optimisation.

3.3 The simulation/ignorance of position issue

In cases of simulation, or if the agent doesn't know which Oracle it is, even CDT can behave in FDT-ish fashion. It just needs to think "given that I don't know who I am, what policy would be best for all my copies to run?".

This will turn AFDT into an FDT-ish decision theory, too. So AFDT requires that the agent's position is specified, in advance of it deciding on any policy or action. For Oracles, this shouldn't be too hard - "yes, you are the Oracle in this box, at this time, answering this question - and if you're not, behave as if you were the Oracle in this box, at this time, answering this question".

This means that if one uses AUDT - similar to AFDT but based on updateless decision theory instead of ADT - then the "updateless" is a misnomer: AUDT must come "pre-updated" with the agent's position information.

---

1. Ignoring the input data for this example, since there is no stochastic uncertainty and only one action. ↩︎

2. If one doesn't want to use a default action, one can alternatively define:

   • $AFD{T}^{\prime }(P,G_F,G_C):=\underset{a\in A}{\mathrm{argmax}}\underset{a^{\prime }\in A}{\mathrm{argmax}}E\left(R|{do}_F\right({AFDT}^{\prime }(\underset{–}{P},\underset{–}{G_F})=a^{\prime }),{do}_C(Act=a\left)\right)$.

   This agent will first compute the FDT solution, then change their own action given this assumption. It's an interesting agent design, even though it doesn't solve the "Oracles colluding" problem. It's interesting, because the only deals it can knowingly reach are those that are Nash equilibriums. So, good on Stag hunts, bad on prisoner's dilemmas. ↩︎