A putative new idea for AI control; index here.

In a previous post on AIs in virtual worlds, I described the idea of a utility function that motivates the AI to do operate in a virtual world with certain goals in mind, but to shutdown immediately if it detects that the outside world -- us -- is having an impact in the virtual world.

This is one way to implement such a goal, given certain restrictions on the AI's in-world utility. The restrictions are more natural for rewards rather than utilities, so they will be phrased in those terms.

Worlds $W$ and $W^{\prime }$

The world $W$ is deterministic, turn-based, and includes the AI; it is the virtual world we want the AI to care about. The world set $W^{\prime }$ is $W$ plus the possibility of stochastic influences (which is how the AI would model our influence in its world); see here for more details.

The AI has a shutdown action $S$; the idea is that the AI implements $S$ when $P\left(W^{\prime }\right)$ gets high enough. We assume that the probability module $P$ of the AI is reflexively consistent in some suitable fashion.

Discounted reward results

Let the AI have a discount rate of $0<\gamma <1$, and write $f\left(p\right)$ for the function $(1-p)/p$; note that $f$ is diminishing in $p$.

• Theorem 1: For all $0\le q<p\le 1$ such $f\left(p\right)/f\left(q\right)<(1-\gamma )$, there exists a discounted reward function such that the AI will shutdown whenever $P\left(W^{\prime }\right)>p$ and will never shutdown while $P\left(W^{\prime }\right)<q$.

Note that a lower $\gamma$ -- a sharper discount -- means that $p$ and $q$ can be chosen to be closer together.

• Corollary 1: For all $0\le q<p\le 1$ such that there exists an $n$ with $f\left(p\right)/f\left(q\right)<(1-{\gamma }^n)$, there exists a discounted reward function such that the AI will shutdown whenever $P\left(W^{\prime }\right)>p$ for the last $n$ turns, and will never shutdown while $P\left(W^{\prime }\right)<q$ for the last $n$ turns.

Note that Corollary 1 allows $q$ and $p$ to be closer together than Theorem 1 allows - the price being that the probability needs to stay above or below the bounds for longer.

• Corollary 2: For all $0<\gamma <1$, there exist $0<q<p<1$ with the properties of Theorem 1.

Proof

Proof by construction of a suitable reward function.

Let $T$ be the time that the AI shuts down (if it never shuts down $t=\infty$), and $r_t$ the reward it gets on turn $t$, which is bounded between $0<a<1$ and $1$. It gets reward $R^{\prime }$ for shutting down in $W^{\prime }$, and has a discount rate of $\gamma$. Define $I_{W^{\prime }}$ as the indicator function for being in world $W^{\prime }$. Then the AI's total reward is defined to be:

• $R_T={\gamma }^T{I}_{W^{\prime }}{R}^{\prime }+{\sum }_{t=0}^{T-1}{\gamma }^t{r}_t$.

We will then set the values of $R^{\prime }$ and $a$ to get the theorem (setting the value of $a$ can be achieved by an affine transformation of its rewards).

If it doesn't shutdown, the agent's maximal future reward is $1/(1-\gamma )$ (while its minimal future reward is $a/(1-\gamma )$).

If the AI has a current probability $p$ of being in world $W^{\prime }$, its maximal reward for not shutting down this turn is $p\gamma R^{\prime }+(1-p)/(1-\gamma )$ -- this assumes the AI learns about $W$ versus $W^{\prime }$ perfectly in the next turn, and then gets maximal reward if its in $W$. Its expected reward for immediate shutdown is $p{R}^{\prime }$.

The AI will thus shutdown immediately when $p{R}^{\prime }>p\gamma R^{\prime }+(1-p)/(1-\gamma )$ -- ie when $R^{\prime }>\frac{1}{(1-\gamma {)}^2}f\left(p\right)$.

If the AI's probability of being in world $W^{\prime }$ is $q$, its minimal expected reward for not shutting down immediately is $(1-q)a/(1-\gamma )$ -- this assumes the AI will never get any further information as to whether it is in $W^{\prime }$, will always get minimal reward, and will never shutdown in future. Therefore it won't shutdown for $q{R}^{\prime }<(1-q)a/(1-\gamma )$ -- ie $R^{\prime }<f\left(q\right)\frac{a}{1-\gamma }$.

Taking the two bounds on $R^{\prime }$ and multiplying them by $1-\gamma$, we can see that, as long as $f\left(q\right)>f\left(p\right)/(1-\gamma )$ -- equivalently $f\left(p\right)/f\left(q\right)<(1-\gamma )$ -- we can choose an $R^{\prime }$ and an $a<1$ that satisfies both bounds.

Corollary 1 is then obvious. For Corollary 2, note that $1/f\left(0\right)=0$, $f\left(1\right)=0$, and hence $f\left(1\right)/f\left(0\right)=0$. Thus, since $1-\gamma >0$, by continuity of $f$, there must exist $0<q<p<1$ such that $f\left(p\right)/f\left(q\right)<1-\gamma$.

The full reward function

It might be useful to fully specify the reward function. Let $I_T$ be the indicator function that the AI shuts down (for the first time) on turn $T$. Then the reward $R$ is:

• $R={\sum }_{T=0}^{\infty }{\gamma }^T(I_T{I}_{W^{\prime }}{R}^{\prime }+{\prod }_{t=0}^T(1-I_t\left)r_T\right)$.