In a previous post, I glibly suggested a utility function $u^{\#}$ that would allow a $u$-maximising agent to print out the expectation of the utility it was maximising.

But $u^{\#}$ turns out to be very peculiar indeed.

For this post, define

• $u^{\#}=-q^2+2qu$,

for some $q$ that is a future output of the AI at time $t$ (assume that $q$ won't be visible or known to anyone except the AI). Assume $u$ only takes non-negative values).

Peculiar subjectivity

I'd always thought that utility functions were clear on the difference between objectivity and subjectivity - that probabilities of probabilities or such didn't make sense, and that we couldn't eg ask the AI to maximise $E(u{)}^2$ (though we could ask it to maximise $E\left(u^2\right)$, no problem).

But $u^{\#}$ blurs this. It seem a perfectly respectable utility function - the fact that one component is user-defined shouldn't change this. What will a $u^{\#}$-maximiser do?

Well, first of all, at time $t$, it will pick $q=E_t\left(u\right)$, and, afterwards, maximise $E\left(u\right)$. This fill give it utility $f\left(E_t\right(u\left)\right)$.

Here $f\left(q\right)=q^2$, but it turns out that there are versions that work with any differentiable convex function - $q^4$, $1/q$, $-\log \left(q\right)$, $\exp \left(q\right)$, $\exp (-q)$, ...

Thus, maximising $u^{\#}$, before $t$, involves maximising $E_t(u{)}^2$. Note that this is distinct from maximising either $E_t\left(u\right)$ or $E_t\left(u^2\right)$.

Consider the following three options the AI can take:

A) $u=1$. B) $u=0$ with $51\%$ probability, $u=2$ with $49\%$ probability, the AI will not know which happens before $t$. C) $u=0$ with $52\%$ probability, $u=2$ with $48\%$ probability, the AI will know which happens before $t$.

Then a $u$-maximiser will choose $A>B>C$, while a $u^2$-maximiser will choose $B>C>A$. But a $u^{\#}$ maximiser will choose $C>A>B$.

Note that since $f\left(q\right)=q^2$ is convex, the AI will always benefit from finding out more information about $u$ (and will never suffer from it, in expectation).

And note that this happens without there being any explicit definition of $E_t\left(u\right)$ in the utility function.

Peculiar corrigibility

The AI will shift smoothly from a $f\left(E_t\right(u\left)\right)$ maximiser before time $t$ to a simple $u$-maximiser after time $t$, making this a very peculiar form of corrigibility.