Suppose you have some deep learning model M_orig that you are finetuning to avoid some particular kind of failure. Suppose all of the following hold:

1. Capable model: The base model has the necessary capabilities and knowledge to avoid the failure.
2. Malleable motivations: There is a "nearby" model M_good (i.e. a model with minor changes to the weights relative to the M_orig) that uses its capabilities to avoid the failure. (Combined with (1), this means it behaves like M_orig except in cases that show the failure, where it does something better.)
3. Strong optimization: If there's a "nearby" setting of model weights that gets lower training loss, your finetuning process will find it (or something even better). Note this is a combination of human factors like "the developers wrote correct code" and background technical facts like "the shape of the loss landscape is favorable".
4. Correct rewards: You accurately detect when a model output is a failure vs not a failure.
5. Good exploration: During finetuning there are many different inputs that trigger the failure.

(In reality each of these are going to lie on a spectrum, and the question is how high you are on each of the spectrums, and some of them can substitute for others. I'm going to ignore these complications and keep talking as though they are discrete properties.)

Claim 1: you will get a model that has training loss at least as good as that of [M_orig without failures]. ((1) and (2) establish that M_good exists and behaves as [M_orig without failures], (3) establishes that we get M_good or something better.)

Claim 2: you will get a model that does strictly better than M_orig on the training loss. ((4) and (5) together establish that the M_orig gets higher training loss than M_good, and we've already established that you get something at least as good as M_good.)

Corollary: Suppose your training loss plateaus, giving you model M, and M exhibits some failure. Then at least one of (1)-(5) must not hold.

Generally when thinking about a deep learning failure I think about which of (1)-(5) was violated. In the case of AI misalignment via deep learning failure, I'm primarily thinking about cases where (4) and/or (5) fail to hold.

In contrast, with ChatGPT jailbreaking, it seems like (4) and (5) probably hold. The failures are very obvious (so it's easy for humans to give rewards), and there are many examples of them already. With Bing it's more plausible that (5) doesn't hold.

To people holding up ChatGPT and Bing as evidence of misalignment: which of (1)-(5) do you think doesn't hold for ChatGPT / Bing, and do you think a similar mechanism will underlie catastrophic misalignment risk?

The LESS is More paper (summarized in AN #96) makes the claim that using the Boltzmann model in sparse regions of demonstration-space will lead to the Boltzmann model over-learning. I found this plausible but not obvious, so I wanted to check it myself. (Partly I got nerd-sniped, partly I do want to keep practicing my ability to tell when things are formalizable theorems.) This benefited from discussion with Andreea (one of the primary authors).

Let's consider a model where there are clusters $\left\{c_i\right\}$, where each cluster contains trajectories whose features are identical $c_i=\{\tau :\varphi (\tau )={\varphi }_{c_i}\}$ (which also implies rewards are identical). Let $c\left(\tau \right)$ denote the cluster that $\tau$ belongs to. The Boltzmann model says $p(\tau \mid \theta )=\frac{\exp \left(R_{\theta }\right(\tau \left)\right)}{{\sum }_{{\tau }^{\prime }}\exp \left(R_{\theta }\right({\tau }^{\prime }\left)\right)}$. The LESS model says $p(\tau \mid \theta )=\frac{\exp \left(R_{\theta }\right(c\left(\tau \right)\left)\right)}{{\sum }_{c^{\prime }}\exp \left(R_{\theta }\right(c^{\prime }\left)\right)}\cdot \frac{1}{|c\left(\tau \right)|}$ , that is, the human chooses a cluster noisily based on the reward, and then uniformly at random chooses a trajectory from within that cluster.

(Note that the paper does something more suited to realistic situations where we have a similarity metric instead of these "clusters"; I'm introducing them as a simpler situation where we can understand what's going on formally.)

In this model, a "sparse region of demonstration-space" is a cluster $c$ with small cardinality $|c|$, whereas a dense one has large $|c|$.

Let's first do some preprocessing. We can rewrite the Boltzmann model as follows:

$p(\tau \mid \theta )=\frac{\exp \left(R_{\theta }\right(\tau \left)\right)}{{\sum }_{{\tau }^{\prime }}\exp \left(R_{\theta }\right({\tau }^{\prime }\left)\right)}=\frac{\exp \left(R_{\theta }\right(c\left(\tau \right)\left)\right)}{{\sum }_{c^{\prime }}\exp \left(R_{\theta }\right(c^{\prime }\left)\right)\cdot |c^{\prime }|}=\frac{|c\left(\tau \right)|\cdot \exp \left(R_{\theta }\right(c\left(\tau \right)\left)\right)}{{\sum }_{c^{\prime }}|c^{\prime }|\cdot \exp \left(R_{\theta }\right(c^{\prime }\left)\right)}\cdot \frac{1}{|c\left(\tau \right)|}$

This allows us to write both models as first selecting a cluster, and then choosing randomly within the cluster:

$p(\tau \mid \theta )=\frac{p\left(c\right(\tau \left)\right)\cdot \exp \left(R_{\theta }\right(c\left(\tau \right)\left)\right)}{{\sum }_{c^{\prime }}p\left(c^{\prime }\right)\exp \left(R_{\theta }\right(c^{\prime }\left)\right)}\cdot \frac{1}{|c\left(\tau \right)|}$

Where for LESS $p\left(c\right)$ is uniform i.e. $p\left(c\right)\propto 1$, whereas for Boltzmann $p\left(c\right)\propto |c|$, i.e. a denser cluster is more likely to be sampled.

So now let us return to the original claim that the Boltzmann model overlearns in sparse areas. We'll assume that LESS is the "correct" way to update (which is what the paper is claiming); in this case the claim reduces to saying that the Boltzmann model updates the posterior over rewards in the right direction but with too high a magnitude.

The intuitive argument for this is that the Boltzmann model assigns a lower likelihood to sparse clusters, since its "prior" over sparse clusters is much smaller, and so when it actually observes this low-likelihood event, it must update more strongly. However, this argument doesn't work -- it only claims that $p_{Boltzmann}\left(\tau \right)<p_{LESS}\left(\tau \right)$, but in order to do a Bayesian update you need to consider likelihood ratios. To see this more formally, let's look at the reward learning update:

$p(\theta \mid \tau )=\frac{p\left(\theta \right)\cdot p(\tau \mid \theta )}{{\sum }_{{\theta }^{\prime }}p\left({\theta }^{\prime }\right)\cdot p(\tau \mid {\theta }^{\prime })}=\frac{p\left(\theta \right)\cdot \frac{\exp \left(R_{\theta }\right(c\left(\tau \right)\left)\right)}{{\sum }_{c^{\prime }}p\left(c^{\prime }\right)\exp \left(R_{\theta }\right(c^{\prime }\left)\right)}}{{\sum }_{{\theta }^{\prime }}p\left({\theta }^{\prime }\right)\cdot \frac{\exp \left(R_{{\theta }^{\prime }}\right(c\left(\tau \right)\left)\right)}{{\sum }_{c^{\prime }}p\left(c^{\prime }\right)\exp \left(R_{{\theta }^{\prime }}\right(c^{\prime }\left)\right)}}$.

In the last step, any linear terms in $p(\tau \mid \theta )$ that didn't depend on $\theta$ cancelled out. In particular, the prior over the selected class canceled out (though the prior did remain in normalizer / denominator, where it can still affect things). But the simple argument of "the prior is lower, therefore it updates more strongly" doesn't seem to be reflected here.

Also, as you might expect, once we make the shift to thinking of selecting a cluster and then selecting a trajectory randomly, it no longer matters which trajectory you choose -- the only relevant information is the cluster chosen (you can see this in the update above, where the only thing you do with the trajectory is to see which cluster $c\left(\tau \right)$ it is in). So from now on I'll just talk about selecting clusters, and updating on them. I'll also write $E{R}_{\theta }\left(c\right)=\exp \left(R_{\theta }\right(c\left)\right)$ for conciseness.

$p(\theta \mid c)=\frac{p\left(\theta \right)\cdot \frac{E{R}_{\theta }\left(c\right)}{{\sum }_{c^{\prime }}p\left(c^{\prime }\right)E{R}_{\theta }\left(c^{\prime }\right)}}{{\sum }_{{\theta }^{\prime }}p\left({\theta }^{\prime }\right)\cdot \frac{E{R}_{{\theta }^{\prime }}\left(c\right)}{{\sum }_{c^{\prime }}p\left(c^{\prime }\right)E{R}_{{\theta }^{\prime }}\left(c^{\prime }\right)}}$ .

This is a horrifying mess of an equation. Let's switch to odds:

$\frac{p({\theta }_1\mid c)}{p({\theta }_2\mid c)}=\frac{p\left({\theta }_1\right)}{p\left({\theta }_2\right)}\cdot \frac{E{R}_{{\theta }_1}\left(c\right)}{E{R}_{{\theta }_2}\left(c\right)}\cdot \frac{{\sum }_{c^{\prime }}p\left(c^{\prime }\right)E{R}_{{\theta }_2}\left(c^{\prime }\right)}{{\sum }_{c^{\prime }}p\left(c^{\prime }\right)E{R}_{{\theta }_1}\left(c^{\prime }\right)}$

The first two terms are the same across Boltzmann and LESS, since those only differ in their choice of $p\left(c\right)$. So let's consider just that last term. Denoting the vector of priors on all classes as $\overrightarrow{p}$, and similarly the vector of exponentiated rewards as $\overrightarrow{E{R}_{\theta }}$, the last term becomes $\frac{\overrightarrow{p}\cdot \overrightarrow{E{R}_{{\theta }_2}}}{\overrightarrow{p}\cdot \overrightarrow{E{R}_{{\theta }_1}}}=\frac{|\overrightarrow{E{R}_{{\theta }_2}}|}{|\overrightarrow{E{R}_{{\theta }_1}}|}\cdot \frac{\cos \left({\alpha }_2\right)}{\cos \left({\alpha }_1\right)}$, where ${\alpha }_i$ is the angle between $\overrightarrow{p}$ and $\overrightarrow{E{R}_{{\theta }_i}}$. Again, the first term doesn't differ between Boltzmann and LESS, so the only thing that differs between the two is the ratio $\frac{\cos \left({\alpha }_2\right)}{\cos \left({\alpha }_1\right)}$.

What happens when the chosen class $c$ is sparse? Without loss of generality, let's say that $E{R}_{{\theta }_1}\left(c\right)>E{R}_{{\theta }_2}\left(c\right)$; that is, ${\theta }_1$ is a better fit for the demonstration, and so we will update towards it. Since $c$ is sparse, $p\left(c\right)$ is smaller for Boltzmann than for LESS -- which probably means that it is better aligned with ${\theta }_2$, which also has a low value of $E{R}_{{\theta }_2}\left(c\right)$ by assumption. (However, this is by no means guaranteed.) In this case, the ratio $\frac{\cos \left({\alpha }_2\right)}{\cos \left({\alpha }_1\right)}$ above would be higher for Boltzmann than for LESS, and so it would more strongly update towards ${\theta }_1$, supporting the claim that Boltzmann would overlearn rather than underlearn when getting a demo from the sparse region.

(Note it does make sense to analyze the effect on the $\theta$ that we update towards, because in reward learning we care primarily about the $\theta$ that we end up having higher probability on.)

Consider the latest AUP equation, where for simplicity I will assume a deterministic environment and that the primary reward depends only on state. Since there is no auxiliary reward any more, I will drop the subscripts to $R$ on $V_R$ and $Q_R$.

$R_{\text{AUP}}(s,a)=R\left(s\right)-\lambda \frac{max\left(V^{\ast }\right(T(s,a))-V^{\ast }(T(s,\varnothing )),0)}{V^{\ast }\left(s\right)-Q^{\ast }(s,\varnothing )}$

Consider some starting state $s_0$, some starting action $a_0$, and consider the optimal trajectory under $R$ that starts with that, which we'll denote as $s_0{a}_0{s}_1{a}_1{s}_2\dots$ . Define $s_i^{\prime }=T(s_{i-1},\varnothing )$ to be the one-step inaction states. Assume that $Q^{\ast }(s_0,a_0)>Q^{\ast }(s_0,\varnothing )$. Since all other actions are optimal for $R$, we have $V^{\ast }\left(s_i\right)=\frac{1}{\gamma }\left(V^{\ast }\right(s_{i-1})-R(s_{i-1}\left)\right)\ge \frac{1}{\gamma }\left(Q^{\ast }\right(s_{i-1},\varnothing )-R(s_{i-1}\left)\right)=V^{\ast }\left(s_i^{\prime }\right)$ , so the max in the equation above goes away, and the total $R_{\text{AUP}}$ obtained is:

$R_{\text{AUP}}(s_0,a_0)+\left(\sum _{i=1}^{\infty }{\gamma }^{i}R\right(s_i,a_i\left)\right)-\lambda \left(\sum _{i=2}^{\infty }\frac{V^{\ast }\left(s_i\right)-V^{\ast }\left(s_i^{\prime }\right)}{V^{\ast }\left(s_{i-1}\right)-Q^{\ast }(s_{i-1},\varnothing )}\right)$

Since we're considering the optimal trajectory, we have $V^{\ast }\left(s_{i-1}\right)-Q^{\ast }(s_{i-1},\varnothing )=\left[R\right(s)+\gamma V^{\ast }(s_i\left)\right]-\left[R\right(s)+\gamma V^{\ast }(s_i^{\prime }\left)\right]=\gamma \left(V^{\ast }\right(s_i)-V^{\ast }(s_i^{\prime }\left)\right)$

Substituting this back in, we get that the total $R_{AUP}$ for the optimal trajectory is $R_{\text{AUP}}(s_0,a_0)+\left(\sum _{i=1}^{\infty }{\gamma }^{i}R\right(s_i,a_i\left)\right)-\lambda \left(\sum _{i=2}^{\infty }\frac{1}{\gamma }\right)$

which... uh... diverges to negative infinity, as long as $\gamma <1$. (Technically I've assumed that $V^{\ast }\left(s_i\right)-V^{\ast }\left(s_i^{\prime }\right)$ is nonzero, which is an assumption that there is always an action that is better than $\varnothing$.)

So, you must prefer the always-$\varnothing$ trajectory to this trajectory. This means that no matter what the task is (well, as long as it has a state-based reward and doesn't fall into a trap where $\varnothing$ is optimal), the agent can never switch to the optimal policy for the rest of time. This seems a bit weird -- surely it should depend on whether the optimal policy is gaining power or not? This seems to me to be much more in the style of satisficing or quantilization than impact measurement.

----

Okay, but this happened primarily because of the weird scaling in the denominator, which we know is mostly a hack based on intuition. What if we instead just had a constant scaling?

Let's consider another setting. We still have a deterministic environment with a state-based primary reward, and now we also impose the condition that $\varnothing$ is guaranteed to be a noop: for any state $s$, we have $T(s,\varnothing )=s$.

Now, for any trajectory $s_0{a}_0\dots$ with $s_i^{\prime }$ defined as before, we have $V^{\ast }\left(s_i^{\prime }\right)=V^{\ast }\left(s_{i-1}\right)$, so $V^{\ast }\left(s_{i-1}\right)-Q^{\ast }(s_{i-1},\varnothing )=V^{\ast }\left(s_{i-1}\right)-\left[R\right(s_{i-1})+\gamma V^{\ast }(s_{i-1}\left)\right]=(1-\gamma )V^{\ast }\left(s_{i-1}\right)-R\left(s_{i-1}\right)$

As a check, in the case where $a_{i-1}$ is optimal, we have $V^{\ast }\left(s_i\right)-V^{\ast }\left(s_i^{\prime }\right)=\frac{1}{\gamma }\left(V^{\ast }\right(s_{i-1})-R(s_{i-1}\left)\right)-V^{\ast }\left(s_{i-1}\right)=\frac{1}{\gamma }\left(\right(1-\gamma \left)V^{\ast }\right(s_{i-1})-R(s_{i-1}\left)\right)$

Plugging this into the original equation recovers the divergence to negative infinity that we saw before.

But let's assume that we just do a constant scaling to avoid this divergence:

$R_{\text{AUP}}(s,a)=R\left(s\right)-\lambda max\left(V^{\ast }\right(T(s,a))-V^{\ast }(T(s,\varnothing )),0)$

Then for an arbitrary trajectory (assuming that the chosen actions are no worse than $\varnothing$), we get $R_{\text{AUP}}(s_i,a_i)=R\left(s\right)-\lambda \left(V^{\ast }\right(s_{i+1})-V^{\ast }(s_i\left)\right)=R\left(s\right)-\left(\lambda \right(V^{\ast }\left(s_{i+1}\right))+(\lambda V^{\ast }\left(s_i\right))$

The total reward across the trajectory is then $\left(\sum _{i=0}^{\infty }{\gamma }^{i}R\right(s_i\left)\right)-\lambda \left(\sum _{i=1}^{\infty }{\gamma }^{i-1}{V}^{\ast }\right(s_i\left)\right)+\lambda \left(\sum _{i=0}^{\infty }{\gamma }^i{V}^{\ast }\right(s_i\left)\right)$

$=\left(\sum _{i=0}^{\infty }{\gamma }^{i}R\right(s_i\left)\right)-\lambda V^{\ast }\left(s_0\right)-\lambda \sum _{i=1}^{\infty }{\gamma }^i(1-\gamma )V^{\ast }\left(s_i\right)$

The $\lambda V^{\ast }\left(s_0\right)$ and $R\left(s_0\right)$ are constants and so don't matter for selecting policies, so I'm going to throw them out:

$=\left(\sum _{i=1}^{\infty }{\gamma }^i\left[R\right(s_i)-\lambda (1-\gamma \left)V^{\ast }\right(s_i\left)\right]\right)$

So in deterministic environments with state-based rewards where $\varnothing$ is a true noop (even the environment doesn't evolve), AUP with constant scaling is equivalent to adding a penalty $\text{Penalty}\left(s\right)=k{V}^{\ast }\left(s\right)$ for some constant $k$; that is, we're effectively penalizing the agent from reaching good states, in direct proportion to how good they are (according to $R$). Again, this seems much more like satisficing or quantilization than impact / power measurement.

I was reading Avoiding Side Effects By Considering Future Tasks, and it seemed like it was doing something very similar to relative reachability. This is an exploration of that; it assumes you have already read the paper and the relative reachability paper. It benefitted from discussion with Vika.

Define the reachability $R(s_1,s_2)=E_{\tau \sim \pi }\left[{\gamma }^n\right]$, where $\pi$ is the optimal policy for getting from $s_1$ to $s_2$, and $n=|\tau |$ is the length of the trajectory. This is the notion of reachability both in the original paper and the new one.

Then, for the new paper when using a baseline, the future task value $V_{future}^{\ast }(s,s^{\prime })$ is:

$E_{g,\tau \sim {\pi }_g,{\tau }^{\prime }\sim {\pi }_g^{\prime }}\left[{\gamma }^{max(n,n^{\prime })}\right]$

where $s^{\prime }$ is the baseline state and $g$ is the future goal.

In a deterministic environment, this can be rewritten as:

$V_{future}^{\ast }(s,s^{\prime })$

$=E_g\left[{\gamma }^{max(n,n^{\prime })}\right]$

$=E_g[min(R(s,g),R(s^{\prime },g)\left)\right]$

$=E_g\left[R\right(s^{\prime },g)-max(R(s^{\prime },g)-R(s,g),0\left)\right]$

$=E_g\left[R\right(s^{\prime },g\left)\right]-E_g[max(R(s^{\prime },g)-R(s,g),0\left)\right]$

$=E_g\left[R\right(s^{\prime },g\left)\right]-d_{RR}(s,s^{\prime })$

Here, $d_{RR}$ is relative reachability, and the last line depends on the fact that the goal is equally likely to be any state.

Note that the first term only depends on the number of timesteps, since it only depends on the baseline state s'. So for a fixed time step, the first term is a constant.

The optimal value function in the new paper is (page 3, and using my notation of $V_{future}^{\ast }$ instead of their $V_i^{\ast }$):

$V^{\ast }\left(s_t\right)=\underset{a_t\in A}{max}\left[r\right(s_t,a_t)+\gamma \sum _{s_{t+1}\in S}p(s_{t+1}\mid s_t,a_t\left)V^{\ast }\right(s_{t+1})+(1-\gamma \left)\beta V_{future}^{\ast }\right]$.

This is the regular Bellman equation, but with the following augmented reward (here $s_t^{\prime }$ is the baseline state at time t):

Terminal states:

$r_{new}\left(s_t\right)$

$=r\left(s_t\right)+\beta V_{future}^{\ast }(s_t,s_t^{\prime })$

$=r\left(s_t\right)-\beta d_{RR}(s_t,s_t^{\prime })+\beta E_g\left[R\right(s_t^{\prime },g\left)\right]$

Non-terminal states:

$r_{new}(s_t,a_t)$

$=r(s_t,a_t)+(1-\gamma )\beta V_{future}^{\ast }(s_t,s_t^{\prime })$

$=r\left(s_t\right)-(1-\gamma )\beta d_{RR}(s_t,s_t^{\prime })+(1-\gamma )\beta E_g\left[R\right(s_t^{\prime },g\left)\right]$

For comparison, the original relative reachability reward is:

$r_{RR}(s_t,a_t)=r\left(s_t\right)-\beta d_{RR}(s_t,s_t^{\prime })$

The first and third terms in $r_{new}$ are very similar to the two terms in $r_{RR}$. The second term in $r_{new}$ only depends on the baseline.

All of these rewards so far are for finite-horizon MDPs (at least, that's what it sounds like from the paper, and if not, they could be anyway). Let's convert them to infinite-horizon MDPs (which will make things simpler, though that's not obvious yet). To convert a finite-horizon MDP to an infinite-horizon MDP, you take all the terminal states, add a self-loop, and multiply the rewards in terminal states by a factor of $(1-\gamma )$ (to account for the fact that the agent gets that reward infinitely often, rather than just once as in the original MDP). Also define $k=\beta (1-\gamma )$ for convenience. Then, we have:

Non-terminal states:

$r_{new}(s_t,a_t)=r\left(s_t\right)-k{d}_{RR}(s_t,s_t^{\prime })+k{E}_g\left[R\right(s_t^{\prime },g\left)\right]$

$r_{RR}(s_t,a_t)=r\left(s_t\right)-\beta d_{RR}(s_t,s_t^{\prime })$

What used to be terminal states that are now self-loop states:

$r_{new}(s_t,a_t)=(1-\gamma )r\left(s_t\right)-k{d}_{RR}(s_t,s_t^{\prime })+k{E}_g\left[R\right(s_t^{\prime },g\left)\right]$