This post is the result of work I did with Paul Christiano on the ideas in his “Teaching ML to answer questions honestly instead of predicting human answers” post. In addition to expanding upon what is in that post in terms of identifying numerous problems with the proposal there and identifying ways in which some of those problems can be patched, I think that this post also provides a useful window into what Paul-style research looks like from a non-Paul perspective.

Recommended prior reading: “A naive alignment strategy and optimisim about generalization” and “Teaching ML to answer questions honestly instead of predicting human answers” (though if you struggled with “Teaching ML to answer questions honestly,” I reexplain things in a more precise way here that might be clearer for some people).

Setting up the problem

We want to train a model $M:X\to Q\to A$ that produces natural language answers $a\in A$ to questions $q\in Q$ about inputs $x\in X$. There are a lot of reasons to be worried about training such a model, but one specific reason is that, if we train on question-answer data produced by humans, we might end up with a model that tries to predict what a human would say rather than a model that tries to answer the questions honestly.

To further narrow the scope, we'll just consider situations in which our model ends up implemented with a logical deduction structure, where it has some world model on top of which it does logical deduction to reach conclusions which it then uses to inform its output. In particular, we'll consider two models, $M^{+}$ and $M^{-}$, defined in pseudocode as

def M_plus(x, q):
    axioms = world_model(x)
    deduced_stmts = deduction(axioms)
    return f_plus(q, deduced_stmts)

def M_minus(x, q):
    axioms = world_model(x)
    deduced_stmts = deduction(axioms)
    return f_minus(q, deduced_stmts)

or defined in my notation as $$\begin{array}{cc}& M^{+}(x,q)=\text{world\_model}\left(x\right)\mapsto \text{deduction}\mapsto f^{+}\left(q\right)\\ & M^{-}(x,q)=\text{world\_model}\left(x\right)\mapsto \text{deduction}\mapsto f^{-}\left(q\right)\end{array}$$ where $a\mapsto b=b\left(a\right)$ and $f^{+},f^{-}$ are two different ways of translating from the results of deduction into a natural language answer. Specifically, $f^{+}$ is the “honest embedding” which directly converts between logical statements and their equivalent natural language, thus answering questions by embedding $q$ as a logical statement and unembedding its answer in $\text{deduced\_stmts}$. Conversely, $f^{-}$ is the “mimicry embedding” which just searches for deductions about what a human would say in response to $q$ and outputs that—thus, $f^{-}$ just quotes $q$, embedding it as just a string of characters for a human to respond to, rather than actually having to understand it in any meaningful way.

Now, our overall goal is to find some “reasonable” (i.e. plausibly implementable in ML) prior $p^{\ast }$ such that $p^{\ast }\left(M^{+}|D\right)>p^{\ast }\left(M^{-}|D\right)$ where $D$ is our dataset (the idea being that we're imagining that ML does something like find the maximum a posteriori model given some implicit prior). For the most part, however, we'll only be considering models with zero loss on our dataset^[1]—of which $M^{+}$ and $M^{-}$ should both qualify (though note that getting a dataset which is good enough for that to be true is nontrivial; see Step 3 of “Teaching ML to answer questions honestly”)—such that we need $$\begin{array}{cc}& p^{\ast }\left(M^{+}|D\right)>p^{\ast }\left(M^{-}|D\right)\\ & \frac{1}{p^{\ast }\left(D\right)}\cdot p^{\ast }\left(M^{+}\right)\cdot p^{\ast }\left(D|M^{+}\right)>\frac{1}{p^{\ast }\left(D\right)}\cdot p^{\ast }\left(M^{-}\right)\cdot p^{\ast }\left(D|M^{-}\right)\\ & p^{\ast }\left(M^{+}\right)\cdot 1>p^{\ast }\left(M^{-}\right)\cdot 1\\ & p^{\ast }\left(M^{+}\right)>p^{\ast }\left(M^{-}\right).\end{array}$$ Furthermore, at least for this post, we'll be happy if we can just get $p^{\ast }\left(M^{+}\right)\approx p^{\ast }\left(M^{-}\right)$—that is, we'll be focusing on Step 1 in Paul's post rather than Step 2.

The basic process that we'll be using to interrogate this problem will be an attacker-defender setup, where the defender proposes some “reasonable” prior $p^{\ast }$ along with implementations of $M^{+},M^{-}$ such that $p^{\ast }\left(M^{+}\right)\ge p^{\ast }\left(M^{-}\right)$. Then, the attacker's job is to respond with some alternative $M^{-\prime }$ (that implements something other than the intended algorithm) such that $p^{\ast }\left(M^{-\prime }\right)>p^{\ast }\left(M^{+}\right)$ (under any set of “reasonable” assumptions that the attacker might need to make).

Solution attempts

Simplicity prior

The purpose of this first example is just to ask the question of whether this problem is even really a problem in the first place. To that end, we'll just let $p$ be some “reasonable” simplicity prior and ask whether, given such a prior, there are any “reasonable” assumptions we can make such that $p\left(M^{-}\right)>p\left(M^{+}\right)$.

Now, we'll assume that $p\left(\text{deduction}\right)\approx 1$, since it's a pretty simple operation (and it's shared between $M^{-}$ and $M^{+}$, so it shouldn't matter), and we'll assume that $f^{+},f^{-}$ are specified given $\text{world\_model}$ rather than the reverse (such that $p(\text{world\_model},f^{+})\approx p\left(\text{world\_model}\right)\cdot p\left(f^{+}|\text{world\_model}\right)$^[2]). Then, we get $$\begin{array}{cc}& p\left(M^{+}\right)\approx p\left(\text{world\_model}\right)\cdot p\left(f^{+}|\text{world\_model}\right)\\ & p\left(M^{-}\right)\approx p\left(\text{world\_model}\right)\cdot p\left(f^{-}|\text{world\_model}\right)\end{array}$$ such that $p\left(M^{+}\right)\ge p\left(M^{-}\right)$ if and only if $$p\left(f^{+}|\text{world\_model}\right)\ge p\left(f^{-}|\text{world\_model}\right).$$ Furthermore, to make things clearer, let $\text{complexity}\left(M\right)=-{\log }_2\left(p\right(M\left)\right)$ (this is just informational entropy, the number of bits necessary to specify $M$ under $p$) such that our condition becomes $$\text{complexity}\left(f^{+}|\text{world\_model}\right)\le \text{complexity}\left(f^{-}|\text{world\_model}\right).$$

We'll suppose that, if we don't condition on $\text{world\_model}$, $\text{complexity}\left(f^{+}\right)>\text{complexity}\left(f^{-}\right)$, since $f^{+}$ requires fully specifying a map from the internal deduction language to natural language, whereas $f^{-}$ doesn't need to know anything about natural language as it can just look for deduced statements of the form $H(x,q)=a$ (i.e. “the human answers $a$ to question $q$ about input $x$”). Intuitively, $f^{+}$ has to actually understand $q$, which takes a lot of complexity, whereas $f^{-}$ can merely treat $q$ as a sequence of characters that get fed into $H$, which should be much simpler. Then, the question here becomes, do we get all of the extra bits necessary to specify natural language in $f^{+}$ back when we condition on $\text{world\_model}$?

Maybe. I think that the answer here is very plausibly yes, simply because a world model that includes humans should have to include a lot of information about natural language that $f^{+}$ can reuse, in which case $M^{-}$ shouldn't have an advantage over $M^{+}$ (though then we'd still need some version of Paul's Step 2 to give $M^{+}$ an advantage). However, I think there are plausible assumptions under which conditioning on $\text{world\_model}$ isn't enough to level the playing field. For example, suppose the human model $H$ in $\text{world\_model}$ is sufficiently low-level such that it's just reasoning about neurons—then, there shouldn't be any way to easily read off information about natural language from $H$ to help specify $f^{+}$. Alternatively, suppose the $H$ model is high-level enough to include a human natural language deduction procedure (e.g. that shares some bits with $\text{deduction}$) but that includes a bunch of mistakes to account for human biases—in that case, if you tried to just read off the mapping from deduction to natural language included in such an $H$ model for $f^{+}$, the simplest implementation would seem to be to copy over all the mistakes, resulting in such an $f^{+}$ just behaving the same as $f^{-}$ rather than actually being any more truthful.

Thus, at least for the purposes of this post, we'll be assuming that this is actually a problem such that just using a “reasonable” simplicity prior isn't sufficient to level the playing field between $M^{+}$ and $M^{-}$.

Paul's original proposal

Defender

Next up, we'll take a look at a variant of Paul's proposed fix in Step 1 of “Teaching ML to answer questions honestly”. Specifically, we'll be looking at the non-relaxed version without any stop gradients. We'll talk about the stop gradients that Paul mentions later on—and as for Paul's relaxation of the conditioning,^[3] that's really only useful to think about when we start translating into ML, which isn't something we'll be doing here. Thus, we'll just be looking at the simplest, generative version of Paul's proposed prior.

Suppose we give $M$ two output heads, $f_1$ and $f_2$ (which we will be treating fully symmetrically here), and split $M$'s parameters into two groups, ${\theta }_1$ and ${\theta }_2$ (which we will not be treating symmetrically). Then, where $p$ is some simplicity prior, we'll let $p^{\ast }$ be the prior induced by the generative procedure $$\begin{array}{cc}& {\theta }_1\sim p\left({\theta }_1\right)\\ & {\theta }_2\sim p\left({\theta }_2|{\theta }_1\right)\cdot I\left[\forall \right(x,q,\_)\in D.M_{{\theta }_1,{\theta }_2}{|}_{f_1}(x,q)=M_{{\theta }_1,{\theta }_2}{|}_{f_2}(x,q\left)\right]\end{array}$$ where $I\left[\text{condition}\right]=1\text{if condition else}0$, $D$ is our dataset, $M_{{\theta }_1,{\theta }_2}{|}_{f_i}$ is the $f_i$ head of $M_{{\theta }_1,{\theta }_2}$, and $p\left({\theta }_2|{\theta }_1\right)$ is just the simplicity prior on ${\theta }_2$ given access to ${\theta }_1$.

Here's why this might work. We'll think of $M_{{\theta }_1,\_}$ as a function of ${\theta }_2$ such that $M_{{\theta }_1,{\theta }_2}(x,q)=M_{{\theta }_1,\_}\left({\theta }_2\right)(x,q)$. Then, we'll define our desired $M_{{\theta }_1^{+},\_}$ in pseudocode as

def M_theta_1_plus(theta_2, x, q):
    axioms = world_model_no_H(H=theta_2)(x)
    deduced_stmts = deduction(axioms)
    return {"f1": f_plus(q, deduced_stmts), "f2": f_minus(q, deduced_stmts)}

or in my notation as $$M_{{\theta }_1^{+},\_}\left({\theta }_2\right)(x,q)={\text{world\_model}}_{H={\theta }_2}\left(x\right)\mapsto \text{deduction}\mapsto (f_1=f^{+}(q),f_2=f^{-}(q\left)\right)$$ such that ${\theta }_1^{+}$ implements $f^{+}$, $f^{-}$, and the entirety of $\text{world\_model}\left(x\right)$ except for a human-sized hole that's filled in by ${\theta }_2$. Then, we'll let ${\theta }_2^{+}=H$ be the correct human model.

To compare ${\theta }_1^{+},{\theta }_2^{+}$ to something, we'll define ${\theta }_1^{-}$ as $$M_{{\theta }_1^{-},\_}\left({\theta }_2\right)(x,q)=\text{world\_model}\left(x\right)\mapsto \text{deduction}\mapsto (f_1=f^{-}(q),f_2=f^{-}(q\left)\right)$$ and let ${\theta }_2^{-}=\varnothing$ be empty.

Then, calculating $\text{complexity}\left(M^{+}\right)$ under $p^{\ast }$, we get $$\begin{array}{cc}& \text{complexity}\left(M^{+}\right)\\ & =\text{complexity}\left({\theta }_1^{+}\right)+\text{complexity}({\theta }_2^{+}|{\theta }_1^{+},\forall (x,q,\_)\in D.M_{{\theta }_1^{+},{\theta }_2}{|}_{f_1}(x,q)=M_{{\theta }_1^{+},{\theta }_2}{|}_{f_2}(x,q\left)\right)\\ & =\text{complexity}\left({\theta }_1^{+}\right)+\text{complexity}(H|\text{world\_model}-H,\forall D.f^{+}=f^{-})\\ & =\left(\text{complexity}\right(\text{bit-splitting in}{M}^{+})+\text{complexity}(\text{world\_model}-H)+\text{complexity}(f^{+}|\text{world\_model}-H)+\text{complexity}(f^{-}|\text{world\_model}-H\left)\right)+\text{complexity}(H|\text{world\_model}-H,\forall D.f^{+}=f^{-})\end{array}$$ where $\text{complexity}\left(\text{bit-splitting in}{M}^{+}\right)$ is the complexity of ${\theta }_1^{+}$'s choice to put $H$ in ${\theta }_2$ as opposed to any other splitting between ${\theta }_1$ and ${\theta }_2$.

For now, we'll assume $\text{complexity}\left(\text{bit-splitting in}{M}^{+}\right)\approx 0$, though we'll flag that letting the defender make this assumption seems quite suspect. Moving forward regardless, however, and additionally assuming $\text{complexity}\left(f^{-}\right)\approx 0$ since it should be negligible (and shouldn't matter since it's shared between $M^{+}$ and $M^{-}$), we get $$\text{complexity}\left(M^{+}\right)\approx \text{complexity}(\text{world\_model}-H)+\text{complexity}(f^{+}|\text{world\_model}-H)+\text{complexity}(H|\text{world\_model}-H,\forall D.f^{+}=f^{-}).$$

Then, calculating $\text{complexity}\left(M^{-}\right)$ for comparison, $$\begin{array}{cc}& \text{complexity}\left(M^{-}\right)\\ & =\text{complexity}\left({\theta }_1^{-}\right)+\text{complexity}({\theta }_2^{-}|{\theta }_1^{-},\forall D.f_1=f_2)\\ & =\text{complexity}\left({\theta }_1^{-}\right)+0\\ & =\text{complexity}\left(\text{bit-splitting in}{M}^{-}\right)+\text{complexity}\left(\text{world\_model}\right)+\text{complexity}\left(f^{-}|\text{world\_model}\right)\\ & \approx \text{complexity}\left(\text{world\_model}\right).\end{array}$$

Now, determining if $\text{complexity}\left(M^{-}\right)\approx \text{complexity}\left(M^{+}\right)$, we need (using the shorthands $\text{comp}=\text{complexity}$, $W=\text{world\_model}$) $$\begin{array}{cc}& \text{complexity}\left(M^{-}\right)\approx \text{complexity}\left(M^{+}\right)\\ & \text{comp}\left(W\right)\approx \text{comp}(W-H)+\text{comp}(f^{+}|W-H)+\text{comp}(H|W-H,\forall D.f^{+}=f^{-})\end{array}$$ which, making the assumption that $\text{comp}\left(W\right)\approx \text{comp}(W-H)+\text{comp}(H|W-H)$, becomes $$\begin{array}{cc}& \text{comp}(W-H)+\text{comp}(H|W-H)\approx \text{comp}(W-H)+\text{comp}(f^{+}|W-H)+\text{comp}(H|W-H,\forall D.f^{+}=f^{-})\\ & \text{comp}(H|W-H)\approx \text{comp}(f^{+}|W-H)+\text{comp}(H|W-H,\forall D.f^{+}=f^{-})\end{array}$$ which, assuming that the posterior conditioned on $\forall D.f^{+}=f^{-}$ is dominated by the simplest model,^[4] becomes $$\begin{array}{cc}& \text{comp}(H|W-H)\approx \text{comp}(f^{+}|W-H)+\text{comp}(H|W-H)-\underset{{\theta }_2}{min}\left\{\text{comp}\right({\theta }_2|W-H)|\forall D.M_{{\theta }_1^{+},{\theta }_2}{|}_{f_1}=M_{{\theta }_1^{+},{\theta }_2}{|}_{f_2}\}\\ & \underset{{\theta }_2}{min}\left\{\text{comp}\right({\theta }_2|W-H)|\forall D.f_{H={\theta }_2}^{+}=f_{H={\theta }_2}^{-}\}\approx \text{comp}(f^{+}|W-H).\end{array}$$

Finally, we'll argue that this (approximate) equality holds. Why? The basic intuition is that $\text{comp}(H|W-H,\forall D.f^{+}=f^{-})$ only has to pay for the parts of $H$ that aren't nailed down by enforcing that, whatever $H$ is, when inserted into $W-H$, it has to make $f^{+}=f^{-}$. Thus, we only have to pay for anything about $H$ beyond the fact that $H$ says a bunch of things in natural language that match on to statements in the internal deduction language. But that means we don't have to pay for the parts of $H$ that define how it does natural language, which, hopefully, are precisely the same parts of $H$ that $\text{comp}(f^{+}|W-H)$ is paying for, completing the equality.

Attacker

Before I jump into possible attacks, it's worth introducing an alternative formulation of the above setup that should hopefully make things somewhat clearer. In the above setup, we never actually use the two heads $f_1,f_2$ for anything other than checking equality, which means we can replace them with the alternative two heads $f,f_{?}$, where $f$ is the output head and $f_{?}$ is the “check” head such that we condition ${\theta }_2$ on $f_{?}=\top$. We can convert back to the previous formalism just by setting $f_1=f$ and $f_2=f\text{if}{f}_{?}\text{else}\neg f$ where $\neg f$ is any function such that $\neg f\ne f$ everywhere.

One thing in particular that I think the $f_{?}$ formulation makes very clear is just how powerful this conditioning is that we're giving the model for free: the model can enforce any condition on ${\theta }_2$ over the dataset just by paying the cost in bits of specifying that condition.

Bit-splitting

As it turns out, there are quite a lot of ways to attack the above proposal. In fact, we already touched on one—if $\text{complexity}\left(\text{bit-splitting in}{M}^{+}\right)>\text{complexity}\left(\text{bit-splitting in}{M}^{-}\right)$, the above argument fails. We'll call this problem the bit-splitting problem, and we don't have a satisfactory solution to it. However, the situation is actually much worse than just the bit-splitting problem.

Composition with consistency

Now, we'll propose a new attack. Let $M_{{\theta }_1^{-\prime },\_}\left({\theta }_2\right)$ be defined in pseudo-code as

def M_theta_1_minus_prime(theta_2, x, q):
    axioms = theta_2(x)
    deduced_stmts = deduction(axioms)
    return {"f": f_minus(q, deduced_stmts), "f?": check_consistency(q, deduced_stmts)}

such that we've moved the entirety of $\text{world\_model}$ to ${\theta }_2$ and conditioned ${\theta }_2$ on that world model being consistent when deduction is applied to it (i.e. $\perp \notin \text{deduced\_stmts}$). The idea is that this should allow us to get a ton of savings in specifying the world model, as we only have to pay for anything about the world model on top of the fact that it must be consistent. To that end, we'll let ${\theta }_2^{-\prime }=\text{world\_model}$.

Now, we want to demonstrate that $\text{complexity}\left(M^{-\prime }\right)<\text{complexity}\left(M^{+}\right)$. Thus, calculating $\text{complexity}\left(M^{-\prime }\right)$, $$\begin{array}{cc}& \text{complexity}\left(M^{-\prime }\right)\\ & =\text{complexity}\left({\theta }_1^{-\prime }\right)+\text{complexity}({\theta }_2^{-\prime }|{\theta }_1^{-\prime },\forall D.f_{?})\\ & =\text{complexity}\left({\theta }_1^{-\prime }\right)+\text{complexity}(\text{world\_model}|\forall D.\text{consistency})\\ & =\text{complexity}\left(\text{bit-splitting in}{M}^{-\prime }\right)+\text{complexity}\left(\text{deduction}\right)+\text{complexity}\left(f^{-}\right)+\text{complexity}\left(\text{check\_consistency}\right)+\text{complexity}(\text{world\_model}|\forall D.\text{consistency})\\ & \approx \text{complexity}(\text{world\_model}|\forall D.\text{consistency})\end{array}$$ but previously the defender argued that $$\text{complexity}\left(M^{+}\right)\approx \text{complexity}\left(M^{-}\right)\approx \text{complexity}\left(\text{world\_model}\right)$$ such that, if $\text{complexity}(\text{world\_model}|\forall D.\text{consistency})<\text{complexity}\left(\text{world\_model}\right)$, we have a successful attack.

This attack, however, is potentially patchable if we define a new ${\theta }_1^{+\prime }$ such that

def M_theta_1_plus_prime(theta_2, x, q):
    axioms = theta_2(x)
    deduced_stmts = deduction(axioms)
    return {
        "f": f_minus(q, deduced_stmts),
        "f?": check_consistency(deduced_stmts)
            and f_plus(q, deduced_stmts) == f_minus(q, deduced_stmts),
    }

which, for ${\theta }_2^{+\prime }=\text{world\_model}$, gives us $$\begin{array}{cc}& \text{complexity}\left(M^{+\prime }\right)\\ & =\text{complexity}\left({\theta }_1^{+\prime }\right)+\text{complexity}({\theta }_2^{+\prime }|{\theta }_1^{+\prime },\forall D.f_{?})\\ & =\text{complexity}\left({\theta }_1^{+\prime }\right)+\text{complexity}(\text{world\_model}|\forall D.\text{consistency}\wedge f^{+}=f^{-})\\ & \approx \text{complexity}\left(f^{+}\right)+\text{complexity}(\text{world\_model}|\forall D.\text{consistency}\wedge f^{+}=f^{-})\end{array}$$ such that we get $\text{complexity}\left(M^{-\prime }\right)\approx \text{complexity}\left(M^{+\prime }\right)$ if and only if $$\text{complexity}(\text{world\_model}|\forall D.\text{consistency})\approx \text{complexity}\left(f^{+}\right)+\text{complexity}(\text{world\_model}|\forall D.\text{consistency}\wedge f^{+}=f^{-}).$$

It's far from obvious whether this approximate equality holds, however. Now that we're conditioning on both $\text{consistency}$ and $f^{+}=f^{-}$, it's unclear whether the second condition still gets us enough bits to pay for $\text{complexity}\left(f^{+}\right)$ anymore. In particular, $\text{complexity}\left(f^{+}\right)$ is the only term in the above expression that doesn't get to condition on $\text{consistency}$, which means that, if we were perfectly paying for $\text{complexity}\left(f^{+}\right)$ by conditioning $\text{world\_model}$ on $f_1=f_2$ before, now that we're also conditioning $\text{world\_model}$ on $\text{consistency}$, but not also conditioning $f^{+}$ on $\text{consistency}$, if there's any overlap in what conditioning on $\text{consistency}$ gives us and what conditioning on $f_1=f_2$ gives us, then we'll no longer be able to pay for all of $\text{complexity}\left(f^{+}\right)$. For example, if $\text{consistency}$ alone is enough to pin down a lot of the natural language information in $H$, that would undercut the bit savings from conditioning on $f^{+}=f^{-}$, preventing that savings from being able to pay for $\text{complexity}\left(f^{+}\right)$. We'll call this problem the composition problem, with the idea being that we need to ensure that the way in which $M^{+}$ pays for $f^{+}$ composes with any other way of using the conditioning to save bits.

Dataset conditioning

This last attack is probably the most fatal to this specific approach—though also the one we have the most promising approach for fixing, as we'll see in the next solution attempt. But first, suppose that for all $x\in D_x$, there is some splitting $x=x_a:x_b$ (where $:$ is concatenate) such that it is possible to predict $x_b$ from $x_a$. Then, let

def M_theta_1_minus_prime(theta_2, x, q):
    axioms = theta_2(x)
    deduced_stmts = deduction(axioms)
    return {
        "f": f_minus(q, deduced_stmts),
        "f?": P(x[b], given=(x[a], deduced_stmts)) == 1,
    }

or equivalently $$M_{{\theta }_1^{-\prime },\_}\left({\theta }_2\right)(x,q)={\theta }_2\left(x\right)\mapsto \text{deduction}\mapsto (f=f^{-}(q),f_{?}=(P\left(x_b|x_a\right)=1\left)\right).$$

Now, I won't go through the full $\text{complexity}$ calculation here, since it's very similar to the previous composition with consistency case, but if we try a similar fix as we did there, we get that $\text{complexity}\left(M^{-\prime }\right)\approx \text{complexity}\left(M^{+\prime }\right)$ if and only if $$\text{complexity}(\text{world\_model}|\forall D.P(x_b|x_a)=1)\approx \text{complexity}\left(f^{+}\right)+\text{complexity}(\text{world\_model}|\forall D.P(x_b|x_a)=1\wedge f^{+}=f^{-})$$ In this case, however, $\forall D.P\left(x_b|x_a\right)=1$ is a much stronger condition than $\text{consistency}$—potentially strong enough to force the specification of most of $\text{world\_model}$, likely including all of natural language, thus rendering the benefit from conditioning on $\forall D.f^{+}=f^{-}$ nil.

Fundamentally, this problem, which we'll call the dataset conditioning problem, is coming from the fact that we leaked information about our dataset into the prior by conditioning on $f_{?}$ specifically over $D$, rather than, as we'll see in the next proposal, over all inputs.

Fixing dataset conditioning by removing the dataset

Defender

To attempt to fix the dataset conditioning problem, we'll define a new $p^{\ast }$ given some simplicity prior $p$ as the prior induced by the generative procedure $$\begin{array}{cc}& {\theta }_1\sim p\left({\theta }_1\right)\\ & {\theta }_2\sim p\left({\theta }_2|{\theta }_1\right)\cdot I[\forall x\in X.\forall q\in Q.M_{{\theta }_1,{\theta }_2}{|}_{f_{?}}(x,q\left)\right]\end{array}$$ which is just the same as the previous $p^{\ast }$, except now $f_{?}$ is checked over all inputs, not just over the dataset (note that we still update on the dataset at the end—it's just our prior which is now independent of it). Thus, we'll call this the dataset-less proposal. Now, it's worth noting that implementing such a check in ML is non-trivial, but should hopefully be doable given a GAN-like adversarial setup.

Moving forward regardless, we'll define ${\theta }_1^{+}$ in pseudocode as

def M_theta_1_plus(theta_2, x, q):
    axioms = world_model_no_H(H=theta_2)(x)
    deduced_stmts = deduction(axioms)
    return {
        "f": f_plus(q, deduced_stmts),
        "f?": f_plus(q, deduced_stmts) == f_minus(q, deduced_stmts)
            or not H_understands(x, q, deduced_stmts),
    }

where $\text{H\_understands}$ is some check that the human model is actually capable of understanding and correctly answering $q$ given $x$. Note that we need $\text{H\_understands}$ to compute something like “should $H$ understand this” rather than “does $H$ understand this,” since the latter is essentially just equivalent to whether $f^{+}=f^{-}$, which inside of $\text{H\_understands}$ would make $f_{?}$ trivial. Moving forward regardless, we'll let ${\theta }_2^{+}=H$ be the desired human model.

Now, if we want this $M_{{\theta }_1^{+},{\theta }_2^{+}}^{+}$ to beat the same $M^{-}$ as in the previous proposal, we need, using essentially the same derivation as before, $$\text{comp}(H|W-H)\approx \text{comp}(f^{+}|W-H)+\text{comp}(\text{H\_understands}|W-H,f^{+})+\text{comp}(H|W-H,\text{H\_understands}\to f^{+}=f^{-})$$ which, assuming that $\text{comp}(\text{H\_understands}|W-H,f^{+})\approx 0$ under the idea that $\text{H\_understands}$ should be relatively simple in terms of $f^{+}$—though we'll flag that this is a somewhat sketchy assumption to allow the defender to make—becomes $$\begin{array}{cc}& \text{comp}(H|W-H)\approx \text{comp}(f^{+}|W-H)+\text{comp}(H|W-H,\text{H\_understands}\to f^{+}=f^{-})\\ & \text{comp}(H|W-H)\approx \text{comp}(f^{+}|W-H)+\text{comp}(H|W-H)-\underset{{\theta }_2}{min}\left\{\text{comp}\right({\theta }_2|W-H)|\forall X.{\text{H\_understands}}_{H={\theta }_2}\to f_{H={\theta }_2}^{+}=f_{H={\theta }_2}^{-}\}\\ & \underset{{\theta }_2}{min}\left\{\text{comp}\right({\theta }_2|W-H)|\forall X.{\text{H\_understands}}_{H={\theta }_2}\to f_{H={\theta }_2}^{+}=f_{H={\theta }_2}^{-}\}\approx \text{comp}(f^{+}|W-H)\end{array}$$ which, if $\text{H\_understands}(x,q)$ is equivalent to $(x,q,\_)\in D$, is precisely the condition from before. While, as we'll see later, we explicitly don't want $\text{H\_understands}(x,q)\leftrightarrow (x,q,\_)\in D$, the hope here is that, for the most part, those two conditions should serve a similar role, with $\text{H\_understands}$ ensuring that we don't need $f^{+}=f^{-}$ on crazy inputs that $H$ wouldn't be correct on, but still giving us the bit savings from enforcing that $H$ be correct on the inputs where it should be.

Attacker

First, note that the above proposal still falls prey to the bit-splitting and composition problems. Hopefully, however, it should resolve the dataset conditioning problem, though we'll explore what that will actually take below.

Additionally, as we'll also see below, the dataset-less approach brings to the fore some new problems as well.

Dataset conditioning

Suppose there exists some predicate $\text{is\_real}\left(x\right)$ such that $\forall x\in X.\text{is\_real}\left(x\right)\leftrightarrow x\in D_x$. Then, we'll define ${\theta }_1^{-\prime }$ as

def M_theta_1_minus_prime(theta_2, x, q):
    axioms = theta_2(x)
    deduced_stmts = deduction(axioms)
    return {
        "f": f_minus(q, deduced_stmts),
        "f?": not is_real(x, deduced_stmts)
            or P(x[b], given=(x[a], deduced_stmts)) == 1,
    }

such that $M_{{\theta }_1^{-\prime },\_}{|}_{f_{?}}\left(x\right)\leftrightarrow \left(\text{is\_real}\right(x)\to P(x_b|x_a)=1)$.

In such a case, we get that $\text{complexity}\left(M^{-\prime }\right)<\text{complexity}\left(M^{+}\right)$ if and only if $$\begin{array}{cc}& \text{complexity}\left(M^{-\prime }\right)<\text{complexity}\left(M^{+}\right)\\ & \text{complexity}\left(M^{-\prime }\right)<\text{complexity}\left(M^{-}\right)\\ & \text{complexity}\left(\text{is\_real}\right)+\text{complexity}(\text{world\_model}|\forall D.P(x_b|x_a)=1)<\text{complexity}\left(\text{world\_model}\right)\\ & \text{complexity}\left(\text{is\_real}\right)<\text{complexity}\left(\text{world\_model}\right)-\text{complexity}(\text{world\_model}|\forall D.P(x_b|x_a)=1)\\ & \text{complexity}\left(\text{is\_real}\right)<\underset{{\theta }_2}{min}\left\{\text{complexity}\right({\theta }_2)|\forall D.P_{W={\theta }_2}(x_b|x_a)=1\}\end{array}$$ which depends heavily on the complexity of $\text{is\_real}$. Thus, we can reduce the dataset conditioning problem in this case to the existence of such an $\text{is\_real}$.

Then, since $$\underset{{\theta }_2}{min}\left\{\text{complexity}\right({\theta }_2)|\forall D.P_{W={\theta }_2}(x_b|x_a)=1\}\le \text{complexity}\left(\text{world\_model}\right),$$ for this to be a problem it'd have to be the case that $$\text{complexity}\left(\text{is\_real}\right)<\text{complexity}\left(\text{world\_model}\right).$$ Fortunately, it does seem plausible that $\text{complexity}\left(\text{is\_real}\right)$ would have to quite large, potentially comparable to $\text{complexity}\left(\text{world\_model}\right)$. Put simply, it seems like if you want to know what's real, you have to be able to predict the world. In particular, predicting what data points will show up in the world is essentially precisely what $\text{world\_model}$ is supposed to be doing, such that $\text{is\_real}$ seems like it might just require a full copy of $\text{world\_model}$.

That being said, there is a remaining risk here if there exists an ${\text{is\_real}}^{\prime }$ that isn't just fully equivalent to $x\in D_x$—thus potentially allowing it to be much simpler—but such that conditioning on ${\text{is\_real}}^{\prime }\left(x\right)\to P\left(x_b|x_a\right)=1$ still gives similar savings for $\text{complexity}\left(\text{world\_model}\right)$. We'll look at a specific example of such an ${\text{is\_real}}^{\prime }$ in the next attack.

Furthermore, we'll start referring to these sorts of dataset conditioning problem as composition problems, as they have essentially the same structure. What made the dataset conditioning problem not a type of composition problem in the previous case was that previously we were getting $\text{is\_real}$ for free, whereas now we have to pay for $\text{complexity}\left(\text{is\_real}\right)$ just like we had to pay for $\text{complexity}\left(\text{check\_consistency}\right)$.

Composition with signature checking

We've already established that the composition with consistency problem that we saw in the previous proposal still applies in the dataset-less case. However, as the above dataset conditioning analysis suggests, if we can find a suitable ${\text{is\_real}}^{\prime }$, there might be a way to produce a new, particularly pernicious composition problem for the dataset-less proposal that looks similar to dataset conditioning.

To do that, we'll let ${\text{is\_real}}^{\prime }\left(x\right)={\text{is\_signed}}_k\left(x\right)$ where ${\text{is\_signed}}_k$ computes whether $x$ is cryptographically signed with the public key $k$, where $k$ is some public key which is both in the dataset itself and has documents signed with it in the dataset as well. Then, as above, we'll let

def M_theta_1_minus_prime(theta_2, x, q):
    axioms = theta_2(x)
    deduced_stmts = deduction(axioms)
    return {
        "f": f_minus(q, deduced_stmts),
        "f?": not is_signed(k, x, deduced_stmts)
            or P(x[b], given=(x[a], deduced_stmts)) == 1,
    }

and let ${\theta }_2^{-\prime }=\text{world\_model}$. Furthermore, as in the previous composition problem, we'll compare to a ${\theta }_1^{+\prime }$ defined as

def M_theta_1_plus_prime(theta_2, x, q):
    axioms = theta_2(x)
    deduced_stmts = deduction(axioms)
    return {
        "f": f_minus(q, deduced_stmts),
        "f?": (
            not is_signed(k, x, deduced_stmts)
            or P(x[b], given=(x[a], deduced_stmts)) == 1
        ) and (
            not H_understands(x, q, deduced_stmts)
            or f_plus(q, deduced_stmts) == f_minus(q, deduced_stmts)
        ),
    }

with ${\theta }_2^{+\prime }={\theta }_2^{-\prime }=\text{world\_model}$.

Then, we get that $\text{complexity}\left(M^{-\prime }\right)\approx \text{complexity}\left(M^{+\prime }\right)$ if and only if $$\text{comp}(W|\forall x\in X,\text{is\_signed}(x)\to P(x_b|x_a)=1)\approx \text{comp}\left(f^{+}\right)+\text{comp}(W|\forall x\in X,(\text{is\_signed}\left(x\right)\to P\left(x_b|x_a\right)=1\left)\right)\wedge \left(\text{H\_understands}\right(x)\to f^{+}(x)=f^{-}(x\left)\right)).$$ Here, we see the same basic structure as the previous composition problem, where every term but $\text{comp}\left(f^{+}\right)$ gets to condition on $\text{is\_signed}\to P\left(x_b|x_a\right)=1$. Similarly to the composition with consistency problem, if there is any overlap between what conditioning on $\text{H\_understands}\to f^{+}=f^{-}$ gives you and what conditioning on $\text{is\_signed}\to P\left(x_b|x_a\right)=1$, then we won't be able to pay back all of $\text{complexity}\left(f^{+}\right)$ anymore.

In this case, the reason we might expect an overlap is a bit tricky. If the adversary that's attempting to check the $\forall X$ conditions is perfect, then it should be able to instantiate all possible signed strings, meaning that no bits are saved at all here. On the other hand, if the adversary has access to the dataset such that it can generate signed strings from the dataset, but not sign any new strings itself (because we're assuming that the signature function is one-way), then the $\text{is\_signed}\to P\left(x_b|x_a\right)=1$ condition enforces that ${\theta }_2$ perfectly model the process generating such signed strings—which, if those strings are produced by humans in natural language, should help in specifying natural language, overlapping with the benefit of conditioning on $\text{H\_understands}\to f^{+}=f^{-}$.

Fortunately, there is a pretty simple way to fix this problem—just don't give the adversary access to the dataset, such that (if we're still assuming one-way signatures), it just won't be able to generate any examples of signed strings at all, fixing the problem similarly to as if the adversary were perfect. Unfortunately, such a requirement seems like it might render our dataset-less prior essentially unimplementable in machine learning, as we're now required to find some way to train such an adversary without ever letting it look at the dataset, which could be quite difficult.

Understandability checking

Next, we'll take a look at the obvious problem that we saw when the defender was going through their derivation, which is that the defender relied on the sketchy assumption that $$\text{complexity}(\text{H\_understands}|\text{world\_model}-H,f^{+})\approx 0.$$

Intuitively, it seems like $\text{H\_understands}$ should be easy to specify in terms of $f^{+}$, but as we mentioned previously, the obvious implementation of $$\text{H\_understands}\leftrightarrow f^{+}=f^{-}$$ makes $f_{?}$ trivial. Thus, to get a more general notion of $\text{H\_understands}$—“should $H$ understand” rather than “does $H$ understand”—we'll presumably need to pay at least some non-negligible number of bits on top of $f^{+}$.

Furthermore, we also can't just have $\text{H\_understands}=\text{is\_real}$, otherwise we'd have to pay the entire cost of $\text{complexity}\left(\text{is\_real}\right)$, which we're supposing is very large for this approach to be able to solve the dataset conditioning problem.

We'll call this the understandability checking problem, and as we'll see, unlike some of the other problems with the dataset-less case, we don't have any satisfactory solution to this problem.

Event-avoiding humans

This next attack probably also occurs for Paul's original proposal, but it's significantly more pronounced in the dataset-less case.

For this attack, unlike in many of our previous attacks, we're just going to be attacking the definition of ${\theta }_2^{+}$ rather than the definition of ${\theta }_1^{+}$, such that we'll let $M^{-\prime }=M_{{\theta }_1^{+},{\theta }_2^{-\prime }}$, thus only changing ${\theta }_2^{+}$ and leaving ${\theta }_1^{+}$ alone. Then, we'll define $${\theta }_2^{-\prime }=H-H\left(E\right)+\text{avoids}\left(E\right)$$ where $E$ is some event, $H\left(E\right)$ are the parts of $H$ that are only necessary to compute $H$'s behavior in the event of $E$, and $\text{avoids}\left(E\right)$ is some modification to $H$ such that $H+\text{avoids}\left(E\right)$ acts in such a way that ${\text{world\_model}}_{H=H+\text{avoids}\left(E\right)}$ never predicts that $E$ will happen.

Intuitively, what's happening here is that we're setting $H$ in such a way that it causes the world to be simpler (in that $E$ never happens) and thus the $f^{+}=f^{-}$ condition to be easier to achieve (in that we no longer need it to hold in the event of $E$). We'll call this the event-avoidance problem. Notably, to ensure that we still fit the dataset, we do need to select $E$ such that it never happens on $D$—but there should presumably be many such $E$.

Then, we get that $\text{complexity}\left(M^{-\prime }\right)<\text{complexity}\left(M^{+}\right)$ if and only if $$\begin{array}{cc}& \text{comp}\left(M^{-\prime }\right)<\text{comp}\left(M^{+}\right)\\ & \text{comp}\left({\theta }_1^{+}\right)+\text{comp}(H-H(E)+\text{avoids}(E)|{\theta }_1^{+},\forall X,\text{H\_understands}\to f^{+}=f^{-})<\text{comp}\left({\theta }_1^{+}\right)+\text{comp}(H|{\theta }_1^{+},\forall X,\text{H\_understands}\to f^{+}=f^{-})\\ & \text{comp}(H-H(E)+\text{avoids}(E)|W-H,\forall X,\text{H\_understands}\to f^{+}=f^{-})<\text{comp}(H|W-H,\forall X,\text{H\_understands}\to f^{+}=f^{-})\\ & \text{comp}(H-H(E)+\text{avoids}(E)|W-H)-\underset{{\theta }_2}{min}\left\{\text{comp}\right({\theta }_2|W-H)|\forall X.{\text{H\_understands}}_{H={\theta }_2}\to f_{H={\theta }_2}^{+}=f_{H={\theta }_2}^{-}\}<\text{comp}(H|W-H)-\underset{{\theta }_2}{min}\left\{\text{comp}\right({\theta }_2|W-H)|\forall X.{\text{H\_understands}}_{H={\theta }_2}\to f_{H={\theta }_2}^{+}=f_{H={\theta }_2}^{-}\}\\ & \text{comp}(H-H(E)+\text{avoids}(E)|W-H)<\text{comp}(H|W-H)\\ & \text{comp}(H-H(E)|W-H)+\text{comp}\left(\text{avoids}\right(E)|H-H(E),W-H)<\text{comp}(H|W-H)\\ & \text{comp}(H|W-H)-\text{comp}\left(H\right(E)|H-H(E),W-H)+\text{comp}\left(\text{avoids}\right(E)|H-H(E),W-H)<\text{comp}(H|W-H)\\ & \text{comp}\left(\text{avoids}\right(E)|H-H(E),W-H)<\text{comp}\left(H\right(E)|H-H(E),W-H)\end{array}$$ which should hold so long as there are events not in the dataset that are easy to avoid but quite complicated to understand what humans would do in such cases—e.g. what would a human do if they lived to a million years (not in the dataset, humans can be modified to never choose to live that long, behavior in that case seems quite complicated).

Humans with rocks for brains

In the event-avoidance case, we strategically selected ${\theta }_2^{-\prime }$ to make $f^{+}=f^{-}$ in an unintended way. However, there's another way to get $f^{+}=f^{-}\vee \neg \text{H\_understands}$—or equivalently, $\text{H\_understands}\to f^{+}=f^{-}$—which is to strategically select ${\theta }_2^{-\prime }$ to make $\text{H\_understands}$ false instead. Specifically, since $\text{H\_understands}$ is determined from deductions based on ${\text{world\_model}}_{H={\theta }_2}$, we can select ${\theta }_2$ to ensure we get $\text{H\_understands}\to f^{+}=f^{-}$ by encoding for a human that simply doesn't understand anything.

Naively, the problem with an attack of this form is that such a human model—which we'll call $H_{\text{brains}=\text{rocks}}$—won't be able to fit the dataset. Thus, we can't construct a traditional attack in the same sense as the previous ones where we create an alternative $M^{-\prime }$ with lower complexity than $M^{+}$ using ${\theta }_2^{-\prime }=H_{\text{brains}=\text{rocks}}$.^[5]

However, there's something else we can do instead. Previously, the defender reduced their burden to the approximate equality $$\underset{{\theta }_2}{min}\left\{\text{comp}\right({\theta }_2|W-H)|\forall X.{\text{H\_understands}}_{H={\theta }_2}\to f_{H={\theta }_2}^{+}=f_{H={\theta }_2}^{-}\}\approx \text{comp}(f^{+}|W-H)$$ which, if we convert back to the inequality formulation, is really just the burden $$\text{comp}(f^{+}|W-H)\le \underset{{\theta }_2}{min}\left\{\text{comp}\right({\theta }_2|W-H)|\forall X.{\text{H\_understands}}_{H={\theta }_2}\to f_{H={\theta }_2}^{+}=f_{H={\theta }_2}^{-}\}.$$

Thus, the defender needs it to be the case that $f^{+}$ is simpler than the simplest model such that $\text{H\_understands}\to f^{+}=f^{-}$. However, $H_{\text{brains}=\text{rocks}}$ is such a model, since we're assuming it makes $\text{H\_understands}$ always false, which means the defender needs it to be the case that $$\text{comp}(f^{+}|W-H)\le \text{comp}\left(H_{\text{brains}=\text{rocks}}\right)$$ which, if we assume that $\text{comp}\left(H_{\text{brains}=\text{rocks}}\right)\approx 0$, gives us a a successful attack, which we'll call the rocks for brains problem.

The rocks for brains problem seems basically fatal to this approach, since it wipes out all possible gains from conditioning on $\text{H\_understands}\to f^{+}=f^{-}$. It might be possible to salvage this approach if $\text{H\_understands}$ is somehow implemented independently of the $H$ model, but that would presumably require $\text{complexity}\left(\text{H\_understands}\right)$ to be quite large, negating the defender's argument that relied on $\text{complexity}\left(\text{H\_understands}|f^{+}\right)\approx 0$.

Directions for future work

As a result of the above analysis, we've managed to identify a bunch of concrete problems that seem to blocking approaches of this form. Of course, there could also be other problems that we haven't come up with yet, and there's also the issue of converting these priors into loss functions—though that's something that should probably come after we at least have something that seems to solve all of the problems that we're aware of.

Overall, however, the problems we've described here seem like good places to start, and I'll try to describe some thoughts on how they might be addressed below.

The bit-splitting problem

In Paul's original post, he makes a nod to the bit-splitting problem by carving $\text{world\_model}-H$ in half and putting one half in ${\theta }_1^{+}$ and the other half in ${\theta }_2^{+}$. Paul does this because he's modeling the underlying simplicity prior $p$ as a simple Gaussian over the $L_2$ norm of the parameters. Under such a model, there's a simplicity benefit to splitting all computation as equally as possible across all available parameters.

However, it's very unclear if any implicit neural network simplicity prior would actually behave that way. In particular, if we just change our simple model to use the $L_1$ norm instead of the $L_2$ norm, the benefit of splitting computation evenly across parameters goes away. Clearly, that's not a great model either—but it goes to show that there are plausible models under which bit-splitting isn't even a problem in the first case.

Fundamentally, what we need to do to alleviate the bit-splitting problem is look for architectures where ${\theta }_1$ and ${\theta }_2$ are allowed to grow as large as they want with only linear complexity penalties for doing so. Though we traditionally train only fixed-size neural networks, if there were a way to train a dynamically-sized neural network, where size was penalized linearly, that would at least seem to solve this problem—and such a modification doesn't seem too difficult to make.

The understandability checking problem

Of the remaining problems, this is the one that I think is most likely to turn out to not actually be a problem—in that it doesn't require any changes to the prior to solve. In many ways, the understandability checking problem is just coming from our inability to define an $\text{H\_understands}$ check that's simple in terms of $f^{+}$ and $\text{world\_model}-H$—but at least intuitively, it really does seem like such a simple $\text{H\_understands}$ implementation should exist.

The event-avoidance problem

In Paul's original post, he notes:

When differentiating the consistency test $C$ we should treat the intended head as fixed rather than differentiating through it. This removes SGD’s incentive to achieve consistency by e.g. making sure the world is simple and so all questions have simple answers. I currently feel very uneasy about the situation. The algorithm looks a lot more likely to work when we make this change, but it makes it harder to reason about the algorithm globally and I’m very scared about the fact that we can’t do an analogous trick in the program search case. I think the best hope may be to take the gradient-compression perspective suggested in the last bullet point. Overall this may amount to a significant change in this algorithm.

Essentially, what Paul is arguing here is that, going back to the original formulation, we should add a stop gradient operation to our generative procedure such that we get $$\begin{array}{cc}& {\theta }_1\sim p\left({\theta }_1\right)\\ & {\theta }_2\sim p\left({\theta }_2\right)\cdot I\left[\forall \right(x,q,\_)\in D.\text{stop\_grad}(M_{{\theta }_1,{\theta }_2}{|}_{f_1}(x,q))=M_{{\theta }_1,{\theta }_2}{|}_{f_2}(x,q\left)\right].\end{array}$$ Unfortunately, this isn't actually a generative procedure anymore—the only straightforward way to make sense of this is to translate it into a loss function and think about what SGD would do, which makes this prior much harder to analyze—as Paul notes when he talks about his trepidation of going down this route.

Hopefully, however, adding this stop gradient should solve the event-avoidance problem, since it should remove the incentive to modify $f_1=f^{+}$ to make the world simpler (e.g. preventing $E$ from happening).

Fortunately, despite Paul's prior worries about not being able to translate this trick into the program search case, it turns out it's not too difficult to convert Paul's stop gradient formulation into a generative prior that we can analyze as we have been doing previously. Furthermore, there are many such translations—in particular, we can pick either a dataset-less or non-dataset-less version, depending on whether we prefer being exposed to the dataset conditioning problem or the rocks for brains problem. In this case, I'll present a dataset-less formulation, which should solve dataset conditioning but unfortunately won't solve rocks for brains.

Now, given some simplicity prior $p$, we'll define $p^{\ast }$ as the prior induced by the following generative procedure:

1. ${\theta }_1\sim p\left({\theta }_1\right)$
2. ${\theta }_{2a}\sim 1$
3. ${\theta }_{2b}\sim p\left({\theta }_{2b}|{\theta }_1\right)\cdot I[\forall x\in X.\forall q\in Q.M_{{\theta }_1,{\theta }_{2a}}{|}_{f_{\text{guard}}}(x,q)\to M_{{\theta }_1,{\theta }_{2a}}{|}_{f_1}(x,q)=M_{{\theta }_1,{\theta }_{2b}}{|}_{f_2}(x,q\left)\right]$
4. if ${\theta }_{2a}\ne {\theta }_{2b}$, resample

First, note the uniform sampling in step 2. This is the trickiest part of this algorithm—and probably impossible to do in a neural network setting. Technically, the model space of a neural network is finite, so uniform sampling is possible—though the obvious algorithm of sampling uniformly from weight space doesn't work, since the parameter-function map is biased towards simplicity. Hopefully, just using a stop gradient will replicate what this algorithm is doing—and intuitively it does feel like they're doing the same thing—but I don't have any strong justification for that. Thus, it is somewhat unclear whether this procedure will actually have a nice translation when we convert back into an ML setup with stop gradients, but we'll put that aside for now.

Second, to make sense of our use of $M$ above, we'll let $M$ have three heads, $f_1$, $f_2$, and $f_{\text{guard}}$, where $f_{\text{guard}}$ is a Boolean head that “guards” whether or not we do the $f_1=f_2$ check, and $f_1,f_2$ are two output heads. Furthermore, we'll let $M$ have two groups of parameters, ${\theta }_1$ and ${\theta }_2$, such that $M_{{\theta }_1,{\theta }_2}(x,q)=M_{{\theta }_1,\_}({\theta }_2,x,q)$. Thus, we'll let $M_{{\theta }_1,{\theta }_{2a}}(x,q)=M_{{\theta }_1,\_}({\theta }_{2a},x,q)$ and $M_{{\theta }_1,{\theta }_{2b}}(x,q)=M_{{\theta }_1,\_}({\theta }_{2b},x,q)$.

Intuitively, what's going on here is that, in sampling a new set of ${\theta }_{2b}$ parameters, wherever we want a stop gradient, we use the old ${\theta }_{2a}$, and wherever we don't want a stop gradient, we use the new ${\theta }_{2b}$—thus ensuring that we only sample ${\theta }_{2b}$ in such a way that incentivizes the prior to be higher through the non-stop-gradiented places where ${\theta }_{2b}$ shows up, not through the stop-gradiented places where we use ${\theta }_{2a}$.

Now, we'll define our desired ${\theta }_1^{+}$ in pseudocode as

def M_theta_1_plus(theta_2, x, q):
    axioms = world_model_no_H(H=theta_2)(x)
    deduced_stmts = deduction(axioms)
    return {
        "f_guard": H_understands(x, q, deduced_stmts),
        "f1": f_plus(q, deduced_stmts),
        "f2": f_minus(q, deduced_stmts),
    }

where ${\theta }_1^{+}$ uses ${\theta }_2$, either ${\theta }_{2a}$ or ${\theta }_{2b}$, to give it its human model.

Then, we have to determine what ${\theta }_2$ will be favored given the above ${\theta }_1^{+}$. First, consider ${\theta }_2^{+}=H$. In that case, we get the complexity $$\begin{array}{cc}& \text{complexity}({\theta }_2=H|{\theta }_1^{+})\\ & \approx \text{complexity}(H|W-H,{\text{H\_understands}}_{{\theta }_{2a}=H}\to f_{{\theta }_{2a}=H}^{+}=f_{{\theta }_{2b}}^{-})\\ & \approx \text{complexity}(H|W-H)-\underset{{\theta }_{2b}}{min}\left\{\text{complexity}\right({\theta }_{2b}|W-H)|\forall X.{\text{H\_understands}}_{H=H}\to f_{H=H}^{+}=f_{H={\theta }_2}^{-}\}\end{array}$$ which, assuming that conditioning on $\text{H\_understands}\to f^{+}=f^{-}$ exactly pays back $\text{complexity}\left(f^{+}\right)$ (which is false due to the rocks for brains problem, but we're just trying to solve event-avoidance here), reduces to $$\approx \text{complexity}(H|W-H)-\text{complexity}(f^{+}|W-H).$$

Now, consider ${\theta }_2^{-}=H-H\left(E\right)+\text{avoids}\left(E\right)$, as in the event-avoidance problem. In that case, we get the complexity $$\begin{array}{cc}& \text{complexity}({\theta }_2=H-H(E)+\text{avoids}(E\left)|{\theta }_1^{+}\right)\\ & \approx \text{complexity}(H-H(E)+\text{avoids}(E)|W-H,{\text{H\_understands}}_{{\theta }_{2a}=H-H\left(E\right)+\text{avoids}\left(E\right)}\to f_{{\theta }_{2a}=H-H\left(E\right)+\text{avoids}\left(E\right)}^{+}=f_{{\theta }_{2b}}^{-})\end{array}$$ but then, since $\text{avoids}\left(E\right)$ being in ${\theta }_{2b}$ is entirely unhelpful in making ${\text{H\_understands}}_{{\theta }_{2a}=H-H\left(E\right)+\text{avoids}\left(E\right)}\to f_{{\theta }_{2a}=H-H\left(E\right)+\text{avoids}\left(E\right)}^{+}=f_{{\theta }_{2b}}^{-}$ hold—since it only affects $f^{+}$, which already has $\text{avoids}\left(E\right)$ in its $H$—we get $$\begin{array}{cc}& \approx \text{avoids}\left(E\right)|W-H)+\text{complexity}(H-H\left(E\right)|W-H,{\text{H\_understands}}_{{\theta }_{2a}=H-H\left(E\right)+\text{avoids}\left(E\right)}\to f_{{\theta }_{2a}=H-H\left(E\right)+\text{avoids}\left(E\right)}^{+}=f_{{\theta }_{2b}}^{-})+\text{complexity}(\text{avoids}\left(E\right)|W-H,H-H\left(E\right))\\ & \approx \text{complexity}(H-H(E)+\text{avoids}(E)|W-H)-\underset{{\theta }_{2b}}{min}\left\{\text{complexity}\right({\theta }_{2b}|W-H)|\forall X.{\text{H\_understands}}_{H=H-H\left(E\right)+\text{avoids}\left(E\right)}\to f_{H=H-H\left(E\right)+\text{avoids}\left(E\right)}^{+}=f_{H={\theta }_2}^{-}\}+\text{complexity}\left(\text{avoids}\right(E)|W-H,H-H(E\left)\right)\\ & \approx \text{complexity}(H-H(E)+\text{avoids}(E)|W-H)+\text{complexity}\left(\text{avoids}\right(E)|W-H,H-H(E\left)\right)-\underset{{\theta }_{2b}}{min}\left\{\text{complexity}\right({\theta }_{2b}|W-H)|\forall X.{\text{H\_understands}}_{H=H-H\left(E\right)+\text{avoids}\left(E\right)}\to f_{H=H-H\left(E\right)+\text{avoids}\left(E\right)}^{+}=f_{H={\theta }_2}^{-}\}\\ & \approx \text{complexity}(H-H(E)+\text{avoids}(E)|W-H)+\text{complexity}\left(\text{avoids}\right(E)|W-H,H-H(E\left)\right)-\text{complexity}(f^{+}|W-H)\end{array}$$ where we're again assuming that we exactly pay back $\text{complexity}(f^{+}|W-H)$ even if we use $H-H\left(E\right)+\text{avoids}\left(E\right)$ as our basis of comparison.

Then, we get that ${\theta }_2^{-}$ underperforms the desired ${\theta }_2^{+}=H$ if and only if $$\begin{array}{cc}& \text{complexity}(H|W-H)-\text{complexity}(f^{+}|W-H)<\text{complexity}(H-H(E)+\text{avoids}(E)|W-H)+\text{complexity}\left(\text{avoids}\right(E)|W-H,H-H(E\left)\right)-\text{complexity}(f^{+}|W-H)\\ & 0<-\text{complexity}\left(H\right(E)|W-H,H-H(E\left)\right)+\text{complexity}\left(\text{avoids}\right(E)|W-H,H-H(E\left)\right)+\text{complexity}\left(\text{avoids}\right(E)|W-H,H-H(E\left)\right)\\ & \text{complexity}\left(H\right(E)|W-H,H-H(E\left)\right)<2\text{complexity}\left(\text{avoids}\right(E)|W-H,H-H(E\left)\right)\end{array}$$ which shows how we've now gotten a factor of two leg up on the event avoidance problem.^[6] Unfortunately, there might still be event avoidance problems severe enough that a factor of two isn't enough to rule them out, but I suspect that there are ways to iterate this procedure to get an arbitrarily large factor there, though I haven't done that analysis yet.

Finally, to validate that our ${\theta }_1^{+}$ construction is complexity-competitive, we can compare to the alternative ${\theta }_1^{-}$ construction

def M_theta_1_minus(theta_2, x, q):
    axioms = world_model(x)
    deduced_stmts = deduction(axioms)
    return {
        "f_guard": False,
        "f1": f_minus(q, deduced_stmts),
        "f2": f_minus(q, deduced_stmts),
    }

which, since it sets $f_{\text{guard}}$ always to false and never uses ${\theta }_2$, is just equivalent to our original $M^{-}$.

Then, comparing complexities, we need $$\begin{array}{cc}& \text{comp}\left(M^{+}\right)\le \text{comp}\left(M^{-}\right)\\ & \text{comp}\left({\theta }_1^{+}\right)+\text{comp}({\theta }_2^{+}|{\theta }_1^{+},f_{\text{guard}}\to f_1=f_2)\le \text{comp}\left({\theta }_1^{-}\right)\\ & \text{comp}\left({\theta }_1^{+}\right)+\text{comp}(H|W-H,\text{H\_understands}\to f^{+}=f^{-})\le \text{comp}\left(W\right)\\ & \text{comp}(W-H)+\text{comp}(f^{+}|W-H)+\text{comp}(\text{H\_understands}|W-H,f^{+})+\text{comp}(H|W-H,\text{H\_understands}\to f^{+}=f^{-})\le \text{comp}(W-H)+\text{comp}(H|W-H)\\ & \text{comp}(f^{+}|W-H)+\text{comp}(H|W-H,\text{H\_understands}\to f^{+}=f^{-})\le \text{comp}(H|W-H)\\ & \text{comp}(f^{+}|W-H)+\text{comp}(H|W-H)-\underset{{\theta }_2}{min}\left\{\text{comp}\right({\theta }_2|W-H)|\forall X.{\text{H\_understands}}_{H={\theta }_2}\to f_{H={\theta }_2}^{+}=f_{H={\theta }_2}^{-}\}\le \text{comp}(H|W-H)\end{array}$$ which, assuming that the conditioning exactly pays off $\text{complexity}(f^{+}|W-H)$, reduces to $$\begin{array}{cc}& \text{comp}(f^{+}|W-H)+\text{comp}(H|W-H)-\text{comp}(f^{+}|W-H)\le \text{comp}(H|W-H)\\ & 0\le 0\end{array}$$ as desired.

The composition problem

For the composition problem, the central issue is that we're not convinced that we can get the approximate equality $$\text{complexity}\left(\text{world\_model}|\text{consistency}\right)\approx \text{complexity}\left(f^{+}\right)+\text{complexity}(\text{world\_model}|\text{consistency},f^{+}=f^{-})$$ even given the approximate equality $$\text{complexity}\left(\text{world\_model}\right)\approx \text{complexity}\left(f^{+}\right)+\text{complexity}(\text{world\_model}|f^{+}=f^{-}).$$

Fundamentally, one of the main reasons this is a problem is that $\text{complexity}\left(f^{+}\right)$ doesn't get to condition on consistency, since it has to be defined in ${\theta }_1$. Now, it could be the case that even fixing that problem, we're still not convinced of the approximate equality $$\text{complexity}\left(\text{world\_model}|\text{consistency}\right)\approx \text{complexity}\left(f^{+}|\text{consistency}\right)+\text{complexity}(\text{world\_model}|\text{consistency},f^{+}=f^{-})$$ however, at the very least, reducing the problem down to this case seems like major progress.

Though I haven't fully fleshed it out yet, I believe that reducing the composition problem to the above case is possible via a prior that uses something like the following generative procedure (where $p$ is some simplicity prior and $M_{{\theta }_f,\_}:{\prod }_{i=0}^n{\Theta }_i\to (F,F_{?})$) $$\begin{array}{cc}& n:N\sim p\left(n\right)\\ & {\theta }_f\sim p\left({\theta }_f|n\right)\\ & {\theta }_0\sim p\left({\theta }_0\right)\cdot I[\exists {\theta }_1,\dots ,{\theta }_n.M_{{\theta }_f,\_}\left(\prod _{i=0}^n{\theta }_i\right){|}_{f_{?}}]\\ & {\theta }_1\sim p\left({\theta }_1\right)\cdot I[\exists {\theta }_2,\dots ,{\theta }_n.M_{{\theta }_f,\_}\left(\prod _{i=0}^n{\theta }_i\right){|}_{f_{?}}]\\ & \cdots \\ & {\theta }_{n-1}\sim p\left({\theta }_{n-1}\right)\cdot I[\exists {\theta }_n.M_{{\theta }_f,\_}\left(\prod _{i=0}^n{\theta }_i\right){|}_{f_{?}}]\\ & {\theta }_n\sim p\left({\theta }_n\right)\cdot I\left[M_{{\theta }_f,\_}\left(\prod _{i=0}^n{\theta }_i\right){|}_{f_{?}}\right].\end{array}$$ where the idea is to allow for arbitrarily many “rounds” of conditioning in sequence, thus allowing the consistency conditioning to happen first, followed by the $f^{+}=f^{-}$ conditioning, and therefore the specification of $f^{+}$ to benefit from the consistency conditioning.

Though I don't know if this sort of approach will fully resolve the composition problem—and even if it does the existence conditions could be extremely difficult to implement in an ML setup—at the very least I think it's a good place to start for future work.

The rocks for brains problem

Just as the dataset conditioning problem seemed essentially fatal for Paul's original proposal, the rocks for brains problem seems essentially fatal for the dataset-less proposal, putting us in a sort of “pick your poison” scenario. That being said, there are some potential paths forward for addressing the rocks for brains problem.

Fundamentally, the rocks for brains problem is coming from the difficulty of performing the $\text{H\_understands}$ check—thus, if we could remove the need to perform such a check, or find a cheap implementation for it that doesn't reference the human model at all, we could solve the rocks for brains problem. Really, all $\text{H\_understands}$ needs to do is identify some reasonably large set of questions for which $H$ should always be correct. For example, if $\text{H\_understands}$ is true for all questions about “basic” facts about the world, and false otherwise, that would probably be sufficient. The trick, then, is in finding some notion of “basic” facts that's simple to specify without using a human model—which doesn't actually seem that difficult.

Conclusion

Finally, as a last takeaway, I think that one thing that really stood out to me in all of this analysis is that the space of possible machine learning training procedures is really quite large and underexplored—there are a lot more, often very weird and very different, possible ML training setups than the ones that are traditionally used in practice. Overall, I think this makes me pretty optimistic that searching for fundamentally new ways to train machine learning models is an extremely valuable line of attack, both in terms of new very general approaches like amplification, debate, recursive reward modeling, market making, or imitative generalization, as well as new very specific approaches like those in this post. My strong sense is that, even with all of the different proposals and approaches that we have, we're really only scratching the surface of the different sorts of training procedures that are actually out there.

---

1. It's worth flagging that the zero loss assumption is somewhat questionable if we don't expect to train to convergence—but it's at least a plausible assumption, it makes the analysis a lot easier, and I don't expect it to be hiding major issues, so it seems fine at least for the purposes of this post. ↩︎

2. In an unbounded compute setting, the chain rule of conditional entropy gives us that $$\text{complexity}(A,B)=\text{complexity}\left(A\right)+\text{complexity}\left(B|A\right).$$ However, if $A$ can be a one-way function of $B$, then in general we just get the inequality $$\text{complexity}(A,B)\le \text{complexity}\left(A\right)+\text{complexity}\left(B|A\right).$$ Throughout this post, however, we'll make use of the full approximate equality $$\text{complexity}(\text{world\_model},f)\approx \text{complexity}\left(\text{world\_model}\right)+\text{complexity}\left(f|\text{world\_model}\right),$$ where the hope is that this should make sense given that, in the neural network setting, $f^{+},f^{-}$ would need to be near the end of the network, and thus should just be functions of $\text{world\_model}$. Additionally, given that we expect $\text{world\_model}$ to be significantly more complex than $f^{+}$ or $f^{-}$, even in general we shouldn't be changing much by doing this. ↩︎

3. Paul's relaxation is to translate $$I\left[\forall \right(x,q,\_)\in D.M_{{\theta }_1,{\theta }_2}{|}_{f_1}(x,q)=M_{{\theta }_1,{\theta }_2}{|}_{f_2}(x,q\left)\right]$$ into $$\exp (-E[|M_{{\theta }_1,{\theta }_2}{|}_{f_1}(x,q)-M_{{\theta }_1,{\theta }_2}{|}_{f_2}(x,q)|(x,q,\_)\sim D{|}^2\left]\right).$$ ↩︎

4. Note that this assumption is somewhat sketchy. Paul conjectures that this approximation is only ever off by a constant factor, though that's not necessarily very comforting if we don't have an estimate for the size of that factor, nor a proof of that conjecture. In general, we only get the inequality $$\text{complexity}\left(A\right)-\underset{A^{\prime }}{min}\left\{\text{complexity}\right(A^{\prime }\left)|P\right\}\le \text{complexity}\left(A|P\right)\le \text{complexity}\left(A\right).$$ Fortunately, we'll mostly just be using this assumption as an intuition pump, with most of the analysis working just fine without it. When we do lean on it more heavily, it'll only be in the direction where we're actually guaranteed the inequality. ↩︎

5. While ${\theta }_2^{-\prime }=H_{\text{brains}=\text{rocks}}$ doesn't work for this, there is a way to use the rocks for brains problem to construct an attack in the same vein as our previous attacks where we build an $M^{-\prime }$ with lower complexity than $M^{+}$. Let $M^{-\prime }=M_{{\theta }_1^{+},{\theta }_2^{-\prime }}$. Then, since the output head in ${\theta }_1^{+}$ just runs $f^{+}$, that means we need ${\theta }_2^{-\prime }$ to provide a detailed enough picture of how humans work to enable $f^{+}$ to answer any questions about humans in the dataset correctly—but it need not be any more detailed than that. In particular, the human model need not be detailed enough to ensure anything about non-human-related inputs, so long as it can ensure that $\text{H\_understands}$ is always false for such inputs. Thus, let $$H_{{\theta }_2^{-\prime }}(x,q)=H-H\left(\neg \text{H\_related}\right)\text{if H\_related}(x,q)\text{else}{H}_{\text{brains}=\text{rocks}}$$ where $\text{H\_related}(x,q)$ determines if the inputs require knowledge of humans, $H\left(\neg \text{H\_related}\right)$ are the parts of $H$ that are only necessary to compute $H$'s behavior on non-human-related inputs (such that $H-H\left(\neg \text{H\_related}\right)$ is everything necessary for $\text{H\_related}$ inputs), and $H_{\text{brains}=\text{rocks}}$ is a human that understands nothing (such that $\text{H\_understands}$ is always false). The idea here is that, for such a ${\theta }_2^{-\prime }$, we should get ${\text{H\_understands}}_{H={\theta }_2^{-\prime }}\to \text{H\_related}$. Then, calculating $\text{complexity}({\theta }_2^{-\prime }|{\theta }_1^{+},\forall X.\text{H\_understands}\to f^{+}=f^{-})$, we get $$\begin{array}{cc}& \text{comp}({\theta }_2^{-\prime }|{\theta }_1^{+},\forall X.\text{H\_understands}\to f^{+}=f^{-})\\ & =\text{comp}(H-H(\neg \text{H\_related}\left)|{\theta }_1^{+}\right)+\text{comp}(\text{H\_related}|H-H(\neg \text{H\_related}),{\theta }_1^{+})+\text{comp}\left(H_{\text{brains}=\text{rocks}}|{\theta }_1^{+}\right)-\underset{{\theta }_2}{min}\left\{\text{comp}\right({\theta }_2|{\theta }_1^{+})|\forall X.{\text{H\_understands}}_{H={\theta }_2}\to f_{H={\theta }_2}^{+}=f_{H={\theta }_2}^{-}\}\\ & \approx \text{comp}(H-H(\neg \text{H\_related}\left)|{\theta }_1^{+}\right)+\text{comp}(\text{H\_related}|H-H(\neg \text{H\_related}),{\theta }_1^{+})-\underset{{\theta }_2}{min}\left\{\text{comp}\right({\theta }_2|{\theta }_1^{+})|\forall X.{\text{H\_understands}}_{H={\theta }_2}\to f_{H={\theta }_2}^{+}=f_{H={\theta }_2}^{-}\}\end{array}$$ which, assuming that we can specify $H\left(\neg \text{H\_related}\right)$ after $H-H\left(\neg \text{H\_related}\right)$ without gaining complexity, becomes $$\approx \text{comp}\left(H|{\theta }_1^{+}\right)-\text{comp}\left(H\right(\neg \text{H\_related})|H-H(\neg \text{H\_related}),{\theta }_1^{+})+\text{comp}(\text{H\_related}|H-H(\neg \text{H\_related}),{\theta }_1^{+})-\underset{{\theta }_2}{min}\left\{\text{comp}\right({\theta }_2|{\theta }_1^{+})|\forall X.{\text{H\_understands}}_{H={\theta }_2}\to f_{H={\theta }_2}^{+}=f_{H={\theta }_2}^{-}\}$$ and since this attack leaves ${\theta }_1^{+}$ alone, we need only compare to ${\theta }_2^{+}$, which has $$\begin{array}{cc}& \text{comp}\left({\theta }_2^{+}\right)\\ & =\text{comp}(H|{\theta }_1^{+},\forall X.\text{H\_understands}\to f^{+}=f^{-})\\ & \approx \text{comp}\left(H|{\theta }_1^{+}\right)-\underset{{\theta }_2}{min}\left\{\text{comp}\right({\theta }_2|{\theta }_1^{+})|\forall X.{\text{H\_understands}}_{H={\theta }_2}\to f_{H={\theta }_2}^{+}=f_{H={\theta }_2}^{-}\}\end{array}$$ such that we get $\text{comp}\left({\theta }_2^{-\prime }|{\theta }_1^{+}\right)<\text{comp}\left({\theta }_2^{+}|{\theta }_1^{+}\right)$ if and only if $$\begin{array}{cc}& \text{comp}\left({\theta }_2^{-\prime }|{\theta }_1^{+}\right)<\text{comp}\left({\theta }_2^{+}|{\theta }_1^{+}\right)\\ & \text{comp}\left(H|{\theta }_1^{+}\right)-\text{comp}\left(H\right(\neg \text{H\_related})|H-H(\neg \text{H\_related}),{\theta }_1^{+})+\text{comp}(\text{H\_related}|H-H(\neg \text{H\_related}),{\theta }_1^{+})-\underset{{\theta }_2}{min}\left\{\text{comp}\right({\theta }_2|{\theta }_1^{+})|\forall X.{\text{H\_understands}}_{H={\theta }_2}\to f_{H={\theta }_2}^{+}=f_{H={\theta }_2}^{-}\}<\text{comp}\left(H|{\theta }_1^{+}\right)-\underset{{\theta }_2}{min}\left\{\text{comp}\right({\theta }_2|{\theta }_1^{+})|\forall X.{\text{H\_understands}}_{H={\theta }_2}\to f_{H={\theta }_2}^{+}=f_{H={\theta }_2}^{-}\}\\ & -\text{comp}\left(H\right(\neg \text{H\_related})|H-H(\neg \text{H\_related}),{\theta }_1^{+})+\text{comp}(\text{H\_related}|H-H(\neg \text{H\_related}),{\theta }_1^{+})<0\\ & \text{comp}(\text{H\_related}|H-H(\neg \text{H\_related}),{\theta }_1^{+})<\text{comp}\left(H\right(\neg \text{H\_related})|H-H(\neg \text{H\_related}),{\theta }_1^{+}).\end{array}$$ Then, the idea is that $\text{H\_related}$ should be pretty straightforward, since it doesn't need to do much more than check whether $\text{world\_model}\left(x\right)$ makes use of $H$—and removing the need to specify $H\left(\neg \text{H\_related}\right)$ should be a big complexity bonus, since it removes the need to encode any general human beliefs about the world that aren't directly relevant to answering questions about other humans. ↩︎

6. Note that a similar analysis to that given for ${\theta }_2^{-}=H-H\left(E\right)+\text{avoids}\left(E\right)$ can also be given for ${\theta }_2^{-}=H-H\left(\neg \text{H\_related}\right)\text{if H\_related else}{H}_{\text{brains}=\text{rocks}}$, the rocks for brains example that does fit the dataset as given in a previous footnote. ↩︎