tl;dr: There are ways of ensuring an agent doesn't have a large impact, by giving an impact penalty. One such penalty is "attainable utility", which measures its "power" by how much it could optimise certain reward functions. But in many circumstances, the agent can build a subagent, without triggering the impact penalty, and then that subagent can become very powerful and optimise the world, still without triggering the impact penalty.

Attainable utility definitions

There's been a long history of trying to penalise an AI for having a large impact on the world. To do that, you need an impact measure. I've designed some myself, back in the day, but they only worked in narrow circumstances and required tricks to get anything useful at all out from them.

A more promising general method is attainable utility. The idea is that, as an agent accumulates power in the world, they increase their ability to affect a lot of different things, and could therefore achieve a lot of different goals.

So, if an agent starts off unable to achieve many goals, but suddenly it can achieve a lot, that's a strong hint that its power has greatly increased.

Thus the impact measure is how much difference an agent's action does to its ability to achieve any of a large class of reward functions. Turner et al defined this using the Q-values of various rewards in a set $R$ ; for $s$ a state and $a$ an action, the penalty is:

$PENALTY (s, a) = \sum R \in R | Q_{R} (s, a) - Q_{R} (s, \emptyset) |$ .

Here $\emptyset$ is the default noop action.

Krakovna et al's basic formula was similar; they defined the distance between two states, $s_{t}$ and $s_{t}^{'}$ , as

$d_{A U} (s_{t}; s_{t}^{'}) = \frac{1}{| R |} \sum R \in R ∣ ∣ V_{R} (s_{t}) - V_{R} (s_{t}^{'}) ∣ ∣$ .

Here $V_{R} (s)$ is the expected value of $R$ , if the agent follows the optimal $R$ -maximising policy from state $s$ onwards.

These measures have problems with delayed effects; putting a vase on a conveyor belt that will smash it in five turns, for example. To combat this, the paper defined an inaction roll-out: seeing what happened do the $d_{A U}$ measure from $s_{t}$ and $s_{t}^{'}$ in future turns, if the agent did noop for a specific period. I won't define the formula here, since the example I'm giving is mostly static: if the agent does noop, nothing happens.

The state $s_{t}$ was always the agent's current state; the state $s_{t}^{'}$ was either the state the agent would have been in had it never done anything but noop (inaction baseline), or the state the agent would have been in, had its previous act

...