Simplified explanation of stratification

Stuart_Armstrong

A putative new idea for AI control; index here.

I've previously talked about stratified indifference/learning. In this short post, I'll try and present the idea, as simply and clearly as possible.

The truth is yet written

Assume the AI maximising a utility $W$ , but is unsure whether $W$ is equal to utility $u$ or instead utility $v$ . Humans have written the true utility on a paper somewhere, and, in an hour, the AI will look at that paper to check what $W$ is. Let $I_{u}$ be the indicator function that $u$ is written on that paper in an hour; $I_{v} = 1 - I_{u}$ is the obvious converse.

Thus $W = I_{u} u + I_{v} v$ .

Here's the problem. There is nothing stopping the AI from looking up the paper currently. And there is nothing stopping the AI from rewriting the paper, or substituting another ( $I_{u}$ is determined by the paper in an hour's time, however that paper got there). In terms of causal graphs, we have the following setup:

Here $W =$ is the node that represents the utility that $W$ is ultimately equal to, $W_{0} u / v$ is what is currently written on the paper (from now on, written as $ \mathbf{W_0}$ for short), $W_{1} u / v$ is what will be written on the paper at decision time (written as $W_{1}$ ), and $A c t$ is the AI's action. Note that $W_{1}$ determines $W =$ directly, while it itself is determined by $W_{0}$ (the past paper) and $A c t$ (the AI's actions).

Default policy

Now assume that there is a default action or policy $\emptyset$ where the AI simply does nothing. In that case, the value of $W_{0}$ determines $W_{1}$ completely. The AI has another action $O W$ (short for overwrite) in which it overwrites the paper's choice (this subdivides into $O R u$ , overwrite with $u$ , and $O R v$ , overwrite with $v$ ). Given $O W$ , $W_{1}$ is then independent of $W_{0}$ .

Generally speaking, the AI will always choose $O W$ , as one of $u$ and $v$ will be easier to maximise, so the AI will choose that one to maximise.

So, informally, what do we really want to happen? We want the AI to maximise $u$ , if $u$ would have been chosen, absent its intervention (and similarly for $v$ ). "Absent its intervention" seems to designate equality between the nodes $W_{0}$ and $W =$ . More generally, it designates what happens when $A c t = \emptyset$ .

Stratification

So let's modify the AI's expected utility calculation to meet that informal desiderata. First, let's look at the universe where, absent AI intervention, $W = u$ . What that means is that, conditional on $A c t = \emptyset$ , $W =$ is set to $u$ .

In this universe, $W_{0}$ and $W_{1}$ are both set to $u$ . Now, it makes no sense to keep $W_{1}$ being set to $u$ , since that node is a consequence of $A c t$ as well as $W_{0}$ .

So we'll define a probability distribution $P_{\emptyset, u}$ (and corresponding expectation $E_{\emptyset, u}$ ) over all nodes that are not descendants of $A c t$ (to do so, we'll assume that all nodes that are descendants of $A c t$ and ancestors of $W =$ are endogenous variables that are deterministic functions of their inputs; the stochasticity is put into exogenous variables, see this paper for more details). Then we will make the AI choose its actions by calculating the expectation of $u$ , given $P_{\emptyset, u}$ .

So, more precisely, this is the expectation of $u$ in the universes where $I_{u}$ would have been $1$ if the AI had chosen to do nothing -- thus $A c t = \emptyset$ .

So if the AI choose action $a$ , this is $E_{\emptyset, u} (u | A c t = a)$ . Note that, in this instance, this is equal to the classical expectation $E (u | A c t = a, W_{0} = u)$ .

What happens if the AI choose $A c t = O R v$ , to overwrite the paper with $v$ ? This is irrelevant for the expected utility calculation, since we're using $E_{\emptyset, u} (u | A c t = a)$ : this is the expectation of $u$ (the utility that would have been chosen, if $A c t = \emptyset$ ). The actual values of node $W =$ becomes irrelevant, only the counterfactual value it would have had. So changing $W =$ does not benefit the AI. In fact, if the AI is time-inconsistent, it makes things worse, as it means its future self will be maximising $v$ instead of $u$ .

Then we do the same thing for $v$ , resulting in the formula for the stratified expectation $E^{s}$ :

$E^{s} (U | a) = P (I_{u} = 1 | \emptyset) E_{\emptyset, u} (u | a) + P (I_{v} = 1 | \emptyset) E_{\emptyset, v} (v | a)$ .

In a sense, the AI should maximise $W^{'} = u (I_{u} | \emptyset) + v (I_{v} | \emptyset)$ .

If you just want to get the basic idea of stratification, you need read no further than this. The rest is just some examples and potential issues.

Lottery and outcome pumps

Some versions of indifference have problems with infinite improbability drives.

A simple example of this would be a situation where the AI buys a lottery ticket and decides that it will overwrite the paper with $u$ if the ticket wins, and with $w$ if the ticket loses. Call this action $A c t = i f f$ .

Presumably, it can use the money gained to increase utility, so $E (u | W = = u, A c t = i f f)$ is rather high (as $W = u$ now implies that the lottery was won).

This gives the following causal graph:

Now let's look at the stratified expectation. Given $W = = u$ and $A c t = \emptyset$ , we can compute $P_{\emptyset, u}$ . This is simply $P (W_{0} = u) = 1$ , and the value of $L o t t o$ being what it was before, and independent of $W_{0}$ .

Thus $E_{u, \emptyset} (u | A c t = i f f)$ will continue to treat the node $L o t t o$ as an independent variable with the same probability as under $A c t = \emptyset$ . Choosing $i f f$ thus does not help the AI's expected stratified utility.

Humans changing their minds

Another situation to consider is one where humans have written the paper initially, but decided to change their minds at a later date. This can be captured in the following diagram:

Here the node $H u m$ determines how the humans change their minds and rewrite the paper. What is the dotted arrow between $A c t$ and $H u m$ ? That represents whether or not the change is a consequence of the AI's actions or not. If $H u m$ is independent of the AI's actions -- if we would have changed our minds even if $A c t = \emptyset$ -- then $P_{\emptyset, u}$ will set $H u m = u$ and $W_{0}$ to whatever its prior was. Thus $H u m$ determines $W =$ , overriding $W_{0}$ , the human's initial decision.

So far, so good. But what if the change in policy is a consequence of the AI's actions? What if the humans only change the utility because they see the AI behaving badly? Then $H u m$ is a descendant node from $A c t$ , and stratification doesn't apply to it. Now, we could decompose that node further as:

Here $H u m F u n$ are the fundamentals of human personalities. Since it is not a descendant node from $A c t$ , stratification will put a probability distribution over it. However, this still doesn't allow us to intervene, given the AI's action. Under stratification, the AI's sole concern is still "would $H u m F u n$ have lead to $u$ (or $v$ ) under $A c t = \emptyset$ ", rather than correcting itself under a different choice of actions.

How to model humans learning from the AI's actions is hard. Indeed, it might be fundamentally hard: you want to distinguish humans learning from the AI (legit) from the AI manipulating humans (non-legit) and this likely can't be done by just looking at the causal graph.

Notes on default policy

Note that $\emptyset$ might be a really bad policy (take people apart to see what they prefer). Then, if it's pretty clear that taking people apart is a bad idea according to the preferences that will emerge from that process, the AI will not choose $\emptyset$ , but will instead leave people together and try to deduce what the outcome of that process would have been.

Alternatively, if doing a logical counterfactual over its own action is bad, we could have the AI take $\emptyset$ to not be its own action, but some other causal process that would have prevented the AI from being turned on in the first place.

[-]Charlie Steiner8y20

I think you can put this scheme on a nicer foundation by talking about strategies rather than actions, and by letting the AI have some probability distribution over $W_{0}$ .

Then you just use the strategy that maximizes $P (W_{0} = u) \cdot P (u | W_{0} = u, d o (s t r a t e g y)) + P (W_{0} = v) \cdot P (v | W_{0} = v, d o (s t r a t e g y))$ . You can also think of this as doing a simplification of the expected utility calculation that bakes in the assumption that the AI can't change $W_{0}$ .

You can then reintroduce the action $\emptyset$ with the observation that the AI will also be well-behaved if it maximizes $P (W_{1} = u | d o (\emptyset)) \cdot P (u | W_{1} = u, d o (s t r a t e g y)) + P (W_{1} = v | d o (\emptyset)) \cdot P (v | W_{1} = v, d o (s t r a t e g y))$ .

[-]Stuart Armstrong8y00

In this example, it's clear that $W_{0}$ is a special node. However, the AI only deduced that because, under $\emptyset$ , $W_{0}$ determines $W$ . It's perfectly plausible that under action $b$ , say, $H u m$ instead determines it. Under $O R u$ and $O R v$ , none of those nodes have any impact.

Therefore we need $\emptyset$ to be a special strategy, as it allows us to identify what nodes connect with $W$ . The advantage of this method is that it lets the AI find the causal graph and compute the dependencies.

Agree strategies are better than actions.

[-]IAFF-User-1118y20

Abstractly, I think of this as adding a utility node, U, with no parents, and having the agent try to maximize the expected value of U.

I think there are some implicit assumptions (which seem reasonable for many situations, prime facie) about the agent's ability to learn about U via some observations when taking null actions (i.e. A and U share some descendant(s), D, and A knows something about P(D | U, A=null).

RE: the last bit, it seems like you can define learning from manipulating in a straightforward way similar to what is proposed here. The intuition is that the humans belief about U should be collapsing around a point, u* (in the absence of interference by the AI), and the AI helps learning if it accelerates this process. If this is literally true, then we can just say that learning is accelerated (at tstep t) if the probability H assigns to u* is higher given an agents action a than it would be given the null action, i.e.

P_H_t(u* | A_0 = a) > P_H_t(u* | A_0 = A1 = ... = null).

[-]IAFF-User-1118y10

So after talking w/Stuart, I guess what he means by "humans learning from the AI’s actions" is that what humans' beliefs about U converges to actually changes (for the better). I'm not sure if that's really desirable, atm.

On a separate note, my proposal has the practical issue that this agent only views its own potential influence on u* as undesirable (and not other agents'). So I think ultimately we want a more rich set of counter-factuals, including, e.g. that humans continue to exist indefinitely (otherwise P_Ht becomes undefined when humanity is extinct).