(Re)Posted as part of the AI Alignment Forum sequence on Value Learning.

Rohin’s note: In the last post, we saw that a good broad value learning approach would need to understand the systematic biases in human planning in order to achieve superhuman performance. Perhaps we can just use machine learning again and learn the biases and reward simultaneously? This post by Stuart Armstrong (original here) and the associated paper say: “Not without more assumptions.”

This post comes from a theoretical perspective that may be alien to ML researchers; in particular, it makes an argument that simplicity priors do not solve the problem pointed out here, where simplicity is based on Kolmogorov complexity (which is an instantiation of the Minimum Description Length principle). The analog in machine learning would be an argument that regularization would not work. The proof used is specific to Kolmogorov complexity and does not clearly generalize to arbitrary regularization techniques; however, I view the argument as being suggestive that regularization techniques would also be insufficient to address the problems raised here.

Humans have no values… nor do any agent. Unless you make strong assumptions about their rationality. And depending on those assumptions, you get humans to have any values.

An agent with no clear preferences

There are three buttons in this world, $B\left(0\right)$, $B\left(1\right)$, and $X$, and one agent $H$.

$B\left(0\right)$ and $B\left(1\right)$ can be operated by $H$, while $X$ can be operated by an outside observer. $H$ will initially press button $B\left(0\right)$; if ever $X$ is pressed, the agent will switch to pressing $B\left(1\right)$. If $X$ is pressed again, the agent will switch back to pressing $B\left(0\right)$, and so on. After a large number of turns $N$, $H$ will shut off. That’s the full algorithm for $H$.

So the question is, what are the values/preferences/rewards of $H$? There are three natural reward functions that are plausible:

• $R\left(0\right)$, which is linear in the number of times $B\left(0\right)$ is pressed.
• $R\left(1\right)$, which is linear in the number of times $B\left(1\right)$ is pressed.
• $R\left(2\right)=I(E,X)R\left(0\right)+I(O,X)R\left(1\right)$, where $I(E,X)$ is the indicator function for $X$ being pressed an even number of times, $I(O,X)=1-I(E,X)$ being the indicator function for $X$ being pressed an odd number of times.

For $R\left(0\right)$, we can interpret $H$ as an $R\left(0\right)$ maximising agent which $X$ overrides. For $R\left(1\right)$, we can interpret $H$ as an $R\left(1\right)$ maximising agent which $X$ releases from constraints. And $R\left(2\right)$ is the “$H$ is always fully rational” reward. Semantically, these make sense for the various $R\left(i\right)$’s being a true and natural reward, with $X=$“coercive brain surgery” in the first case, $X=$“release H from annoying social obligations” in the second, and $X=$“switch which of $R\left(0\right)$ and $R\left(1\right)$ gives you pleasure” in the last case.

But note that there is no semantic implications here, all that we know is $H$, with its full algorithm. If we wanted to deduce its true reward for the purpose of something like Inverse Reinforcement Learning (IRL), what would it be?

Modelling human (ir)rationality and reward

Now let’s talk about the preferences of an actual human. We all know that humans are not always rational. But even if humans were fully rational, the fact remains that we are physical, and vulnerable to things like coercive brain surgery (and in practice, to a whole host of other more or less manipulative techniques). So there will be the equivalent of “button $X$” that overrides human preferences. Thus, “not immortal and unchangeable” is in practice enough for the agent to be considered “not fully rational”.

Now assume that we’ve thoroughly observed a given human h (including their internal brain wiring), so we know the human policy $\pi \left(h\right)$ (which determines their actions in all circumstances). This is, in practice all that we can ever observe - once we know $\pi \left(h\right)$ perfectly, there is nothing more that observing h can teach us.

Let $R$ be a possible human reward function, and R the set of such rewards. A human (ir)rationality planning algorithm $p$ (hereafter referred to as a planner), is a map from R to the space of policies (thus $p\left(R\right)$ says how a human with reward $R$ will actually behave - for example, this could be bounded rationality, rationality with biases, or many other options). Say that the pair $(p,R)$ is compatible if $p\left(R\right)=\pi \left(h\right)$. Thus a human with planner $p$ and reward $R$ would behave as $h$ does.

What possible compatible pairs are there? Here are some candidates:

• $\left(p\right(0),R(0\left)\right)$, where $p\left(0\right)$ and $R\left(0\right)$ are some “plausible” or “acceptable” planner and reward functions (what this means is a big question).
• $\left(p\right(1),R(1\left)\right)$, where $p\left(1\right)$ is the “fully rational” planner, and $R\left(1\right)$ is a reward that fits to give the required policy.
• $\left(p\right(2),R(2\left)\right)$, where $R\left(2\right)=-R\left(1\right)$, and $p\left(2\right)=-p\left(1\right)$, where $-p\left(R\right)$ is defined as $p(-R)$; here $p\left(2\right)$ is the “fully anti-rational” planner.
• $\left(p\right(3),R(3\left)\right)$, where $p\left(3\right)$ maps all rewards to $\pi \left(h\right)$, and $R\left(3\right)$ is trivial and constant.
• $\left(p\right(4),R(4\left)\right)$, where $p\left(4\right)=-p\left(0\right)$ and $R\left(4\right)=-R\left(0\right)$.

Distinguishing among compatible pairs

How can we distinguish between compatible pairs? At first appearance, we can’t. That’s because, by their definition of compatible, all pairs produce the correct policy $\pi \left(h\right)$. And once we have $\pi \left(h\right)$, further observations of $h$ tell us nothing.

I initially thought that Kolmogorov or algorithmic complexity might help us here. But in fact:

Theorem: The pairs $\left(p\right(i),R(i\left)\right)$, $i\ge 1$, are either simpler than $\left(p\right(0),R(0\left)\right)$, or differ in Kolmogorov complexity from it by a constant that is independent of $\left(p\right(0),R(0\left)\right)$.

Proof: The cases of $i=4$ and $i=2$ are easy, as these differ from $i=0$ and $i=1$ by two minus signs. Given $\left(p\right(0),R(0\left)\right)$, a fixed-length algorithm computes $\pi \left(h\right)$. Then a fixed length algorithm defines $p\left(3\right)$ (by mapping input to $\pi \left(h\right)$). Furthermore, given $\pi \left(h\right)$ and any history $\eta$, a fixed length algorithm computes the action $a\left(\eta \right)$ the agent will take; then a fixed length algorithm defines $R\left(1\right)(\eta ,a(\eta \left)\right)=1$ and $R\left(1\right)(\eta ,b)=0$ for $b\ne a\left(\eta \right)$.

So the Kolmogorov complexity can shift between $p$ and $R$ (all in $R$ for $i=1,2$, all in $p$ for $i=3$), but it seems that the complexity of the pair doesn’t go up during these shifts.

This is puzzling. It seems that, in principle, one cannot assume anything about $H$’s reward at all! $R\left(2\right)=-R\left(1\right)$, $R\left(4\right)=-R\left(0\right)$, and $p\left(3\right)$ is compatible with any possible reward $R$. If we give up the assumption of human rationality - which we must - it seems we can’t say anything about the human reward function. So it seems IRL must fail.