AI ALIGNMENT FORUM
AF

paulfchristiano's Shortform — AI Alignment Forum

paulfchristiano's Shortform

29th Jun 2021

1 min read

6

This is a special post for quick takes by paulfchristiano. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.

36 comments, sorted by

top scoring

Click to highlight new comments since: Today at 2:11 AM

[-]paulfchristiano4y120

In the strategy stealing assumption I describe a policy we might want our AI to follow:

Keep the humans safe, and let them deliberate(/mature) however they want.
Maximize option value while the humans figure out what they want.
When the humans figure out what they want, listen to them and do it.

Intuitively this is basically what I expect out of a corrigible AI, but I agree with Eliezer that this seems more realistic as a goal if we can see how it arises from a reasonable utility function.

So what does that utility function look like?

A first pass answer is pretty similar to my proposal from A Formalization of Indirect Normativity: we imagine some humans who actually have the opportunity to deliberate however they want and are able to review all of our AI's inputs and outputs. After a very long time, they evaluate the AI's behavior on a scale from [-1, 1], where 0 is the point corresponding to "nothing morally relevant happens," and that evaluation is the AI's utility.

The big difference is that I'm now thinking about what would actually happen, in the real world if the humans had the space and security to deliberate rather than formally defining a hypothetical process. I think that is going to end up being both safer and easier to implement, though it introduces its own set of complications.

Our hope is that the policy "keep the humans safe, then listen to them about what to do" is a good strategy for getting a high utility in this game, even if our AI is very unsure about what the humans would ultimately want. Then if our AI is sufficiently competent we can expect it to find a strategy at least this good.

[-]paulfchristiano4y90

The most important complication is that the AI is no longer isolated from the deliberating humans. We don't care about what the humans "would have done" if the AI hadn't been there---we need our AI to keep us safe (e.g. from other AI-empowered actors), we will be trusting our AI not to mess with the process of deliberation, and we will likely be relying on our AI to provide "amenities" to the deliberating humans (filling the same role as the hypercomputer in the old proposal).

Going even further, I'd like to avoid defining values in terms of any kind of counterfactual like "what the humans would have said if they'd stayed safe" because I think those will run into many of the original proposal's problems.

Instead we're going to define values in terms of what the humans actually conclude here in the real world. Of course we can't just say "Values are whatever the human actually concludes" because that will lead our agent to deliberately compromise human deliberation rather than protecting it.

Instead, we are going to add in something like narrow value leaning. Assume the human has some narrow preferences over what happens to them over the next hour. These aren't necessarily that wise. They don't understand what's happening in the "outside world" (e.g. "am I going to be safe five hours from now?" or "is my AI-run company acquiring a lot of money I can use when I figure out what I want?"). But they do assign low value to the human getting hurt, and assign high value to the human feeling safe and succeeding at their local tasks; they assign low value to the human tripping and breaking their neck, and high value to having the AI make them a hamburger if they ask for a hamburger; and so on. These preferences are basically dual to the actual process of deliberation that the human undergoes. There is a lot of subtlety about defining or extracting these local values, but for now I'm going to brush that aside and just ask how to extract the utility function from this whole process.

It's no good to simply use the local values, because we need our AI to do some lookahead (both to future timesteps when the human wants to remain safe, and to the far future when the human will evaluate how much option value the AI actually secured for them). It's no good to naively integrate local values over time, because a very low score during a brief period (where the human is killed and replaced by a robot accomplice) cannot be offset by any number of high scores in the future.

Here's my starting proposal:

We quantify the human's local preferences by asking "Look at the person you actually became. How happy are you with that person? Quantitatively, how much of your value was lost by replacing yourself with that person?" This gives us a loss on a scale from 0% (perfect idealization, losing nothing) to 100% (where all of the value is gone). Most of the values will be exceptionally small, especially if we look at a short period like an hour.
Eventually once the human becomes wise enough to totally epistemically dominate the original AI, they can assign a score to the AI's actions. To make life simple for now let's ignore negative outcomes and just describe value as a scalar from 0% (barren universe) to 100% (all of the universe is used in an optimal way). Or we might use this "final scale" in a different way (e.g. to evaluate the AI's actions rather than the actually assessing outcomes, assigning high scores to corrigible and efficient behavior and somehow quantifying deviations from that ideal).
The utility is the product of all of these numbers.

I think there are a lot of problems with this method of quantitative aggregation. But I think this direction is promising and I currently expect something along these lines will work.

[-]adamShimi4y20

Here's my starting proposal:
We quantify the human's local preferences by asking "Look at the person you actually became. How happy are you with that person? Quantitatively, how much of your value was lost by replacing yourself with that person?" This gives us a loss on a scale from 0% (perfect idealization, losing nothing) to 100% (where all of the value is gone). Most of the values will be exceptionally small, especially if we look at a short period like an hour.
Eventually once the human becomes wise enough to totally epistemically dominate the original AI, they can assign a score to the AI's actions. To make life simple for now let's ignore negative outcomes and just describe value as a scalar from 0% (barren universe) to 100% (all of the universe is used in an optimal way). Or we might use this "final scale" in a different way (e.g. to evaluate the AI's actions rather than the actually assessing outcomes, assigning high scores to corrigible and efficient behavior and somehow quantifying deviations from that ideal).
The utility is the product of all of these numbers.

If I follow correctly, the first step requires the humans to evaluate the output of narrow value learning, until this output becomes good enough to become universal with regard to the original AI and supervise it? I'm not sure I get why the AI wouldn't be incentivized to temper with the narrow value learning, à la Predict-o-matic? Depending on certain details, (like maybe the indescribable hellworld hypothesis), maybe the AI can introduce changes to the partial imitations/deliberations that end up hidden and compounding until the imitations epistemically dominates the AI, and then it ask it to do simple stuff.

[-]paulfchristiano4y20

The hope is that a tampering large enough to corrupt the human's final judgment would get a score of ~0 in the local value learning. 0 is the "right" score since the tampered human by hypothesis has lost all of the actual correlation with value. (Note that at the end you don't need to "ask it to do simple stuff" you can just directly assign a score of 1.)

This hope does require the local oversight process to be epistemically competitive with the AI, in the sense that e.g. if the AI understands something subtle about the environment dynamics then the oversight process also needs to understand that. And that's what we are trying to do with all of this business about training AIs to answer questions honestly. The point is just that you don't have to clear up any of the ambiguity about what the human wants, you just have to be able to detect someone tampering with deliberation. (And the operationalization of tampering doesn't have to be so complex.)

(I'm not sure if this made too much sense, I have a draft of a related comment that I'll probably post soon but overall expect to just leave this as not-making-much-sense for now.)

[-]adamShimi4y10

This hope does require the local oversight process to be epistemically competitive with the AI, in the sense that e.g. if the AI understands something subtle about the environment dynamics then the oversight process also needs to understand that. And that's what we are trying to do with all of this business about training AIs to answer questions honestly. The point is just that you don't have to clear up any of the ambiguity about what the human wants, you just have to be able to detect someone tampering with deliberation. (And the operationalization of tampering doesn't have to be so complex.)

So you want a sort of partial universality sufficient to bootstrap the process locally (while not requiring the understanding of our values in fine details), giving us enough time for a deliberation that would epistemically dominate the AI in a global sense (and get our values right)?

If that's about right, then I agree that having this would make your proposal work, but I still don't know how to get it. I need to read your previous posts on reading questions honestly.

[-]paulfchristiano4y30

You basically just need full universality / epistemic competitiveness locally. This is just getting around "what are values?" not the need for competitiveness. Then the global thing is also epistemically competitive, and it is able to talk about e.g. how our values interact with the alien concepts uncovered by our AI (which we want to reserve time for since we don't have any solution better than "actually figure everything out 'ourselves'").

Almost all of the time I'm thinking about how to get epistemic competitiveness for the local interaction. I think that's the meat of the safety problem.

[-]Vladimir_Nesov4y*20

The upside of humans in reality is that there is no need to figure out how to make efficient imitations that function correctly (as in X-and-only-X). To be useful, imitations should be efficient, which exact imitations are not. Yet for the role of building blocks of alignment machinery, imitations shouldn't have important systematic tendencies not found in the originals, and their absence is only clear for exact imitations (if not put in very unusual environments).

Suppose you already have an AI that interacts with the world, protects it from dangerous AIs, and doesn't misalign people living in it. Then there's time to figure out how to perform X-and-only-X efficient imitation, which drastically expands the design space, makes it more plausible that the kinds of systems that you wrote about a lot relying on imitations actually work as intended. In particular, this might include the kind of long reflection that has all the advantages of happening in reality without wasting time and resources on straightforwardly happening in reality, or letting the bad things that would happen in reality actually happen.

So figuring out object level values doesn't seem like a priority if you somehow got to the point of having an opportunity to figure out efficient imitation. (While getting to that point without figuring out object level values doesn't seem plausible, maybe there's a suggestion of a process that gets us there in the limit in here somewhere.)

[-]paulfchristiano4y*30

I think the biggest difference is between actual and hypothetical processes of reflection. I agree that an "actual" process of reflection would likely ultimately involve most humans migrating to emulations for the speed and other advantages. (I am not sure that a hypothetical process necessarily needs efficient imitations, rather than AI reasoning about what actual humans---or hypothetical slow-but-faithful imitations---might do.)

[-]Vladimir_Nesov4y*30

I see getting safe and useful reasoning about exact imitations as a weird special case or maybe a reformulation of X-and-only-X efficient imitation. Anchoring to exact imitations in particular makes accurate prediction more difficult than it needs to be, as it's not the thing we care about, there are many irrelevant details that influence outcomes that accurate predictions would need to take into account. So a good "prediction" is going to be value-laden, with concrete facts about actual outcomes of setups built out of exact imitations being unimportant, which is about the same as the problem statement of X-and-only-X efficient imitation.

If such "predictions" are not good enough by themselves, underlying actual process of reflection (people living in the world) won't save/survive this if there's too much agency guided by the predictions. Using an underlying hypothetical process of reflection (by which I understand running a specific program) is more robust, as AI might go very wrong initially, but will correct itself once it gets around to computing the outcomes of the hypothetical reflection with more precision, provided the hypothetical process of reflection is defined as isolated from the AI.

I'm not sure what difference between hypothetical and actual processes of reflection you are emphasizing (if I understood what the terms mean correctly), since the actual civilization might plausibly move in into a substrate that is more like ML reasoning than concrete computation (let alone concrete physical incarnation), and thus become the same kind of thing as hypothetical reflection. The most striking distinction (for AI safety) seems to be the implication that an actual process of reflection can't be isolated from decisions of the AI taken based on insufficient reflection.

There's also the need to at least define exact imitations or better yet X-and-only-X efficient imitation in order to define a hypothetical process of reflection, which is not as absolutely necessary for actual reflection, so getting hypothetical reflection at all might be more difficult than some sort of temporary stability with actual reflection, which can then be used to define hypothetical reflection and thereby guard from consequences of overly agentic use of bad predictions of (on) actual reflection.

[-]paulfchristiano4y30

It seems to me like "Reason about a perfect emulation of a human" is an extremely similar task to "reason about a human," to me it does not feel closely related to X-and-only-X efficient imitation. For example, you can make calibrated predictions about what a human would do using vastly less computing power than a human (even using existing techniques), whereas perfect imitation likely requires vastly more computing power.

[-]Vladimir_Nesov4y*10

The point is that in order to be useful, a prediction/reasoning process should contain mesa-optimizers that perform decision making similar in a value-laden way to what the original humans would do. The results of the predictions should be determined by decisions of the people being predicted (or of people sufficiently similar to them), in the free-will-requires-determinism/you-are-part-of-physics sense. The actual cognitive labor of decision making needs to in some way be an aspect of the process of prediction/reasoning, or it's not going to be good enough. And in order to be safe, these mesa-optimizers shouldn't be systematically warped into something different (from a value-laden point of view), and there should be no other mesa-optimizers with meaningful influence in there. This just says that prediction/reasoning needs to be X-and-only-X in order to be safe. Thus the equivalence. Prediction of exact imitation in particular is weird because in that case the similarity measure between prediction and exact imitation is hinted to not be value-laden, which it might have to be in order for the prediction to be both X-and-only-X and efficient.

This is only unimportant if X-and-only-X is the likely default outcome of predictive generalization, so that not paying attention to this won't result in failure, but nobody understands if this is the case.

The mesa-optimizers in the prediction/reasoning similar to the original humans is what I mean by efficient imitations (whether X-and-only-X or not). They are not themselves the predictions of original humans (or of exact imitations), which might well not be present as explicit parts of the design of reasoning about the process of reflection as a whole, instead they are the implicit decision makers that determine what the conclusions of the reasoning say, and they are much more computationally efficient (as aspects of cheaper reasoning) than exact imitations. At the same time, if they are similar enough in a value-laden way to the originals, there is no need for better predictions, much less for exact imitation, the prediction/reasoning is itself the imitation we'd want to use, without any reference to an underlying exact process. (In a story simulation, there are no concrete states of the world, only references to states of knowledge, yet there are mesa-optimizers who are the people inhabiting it.)

If prediction is to be value-laden, with value defined by reflection built out of that same prediction, the only sensible way to set this up seems to be as a fixpoint of an operator that maps (states of knowledge about) values to (states of knowledge about) values-on-reflection computed by making use of the argument values to do value-laden efficient imitation. But if this setup is not performed correctly, then even if it's set up at all, we are probably going to get bad fixpoints, as it happens with things like bad Nash equilibria etc. And if it is performed correctly, then it might be much more sensible to allow an AI to influence what happens within the process of reflection more directly than merely by making systematic distortions in predicting/reasoning about it, thus hypothetical processes of reflection wouldn't need the isolation from AI's agency that normally makes them safer than the actual process of reflection.

[-]paulfchristiano4y70

Suppose that someone has trained a model to predict $Y$ given $X$ , and I want to extend it to a question-answering model $G_{θ^{'}, θ} : (X, Q) \mapsto A$ that answers arbitrary questions in a way that reflects all of $F_{θ}$ 's knowledge.

Two prototypical examples I am thinking of are:

$F_{θ}$ runs a low-level model of physics. We want to extract high-level features of the world from the intermediate physical states, which requires e.g. a cat-classifier that operates directly on physical states rather than pixels.
$F_{θ}$ performs logical deductions from a set of statements produced by an opaque "intuition module." We want to output all of the intermediate inferences, which requires knowing how to translate between its intermediate language and natural language.

Here's an approach that seems kind of silly to me but is probably worth exploring:

Let $Z_{θ} (X)$ be all the activations of $F_{θ}$ . In our examples, this includes (amongst other things) all of the physical states computed by $F_{θ}$ or all of the intermediate deductions.
Train an extraction model $E_{θ^{'}} : (Z, Q) \mapsto A$ that answers questions given only the activations of $F_{θ}$ .
Define $G_{θ^{'}, θ} (X, Q) = E_{θ^{'}} (Z_{θ} (X), Q)$
Optimize $E_{θ^{'}}$ aggressively for speed, e.g. choose the simplest circuit that produces good answers.

This feels appealing for the same reason that you might hope minimal circuits are not deceptive. For example, it seems like $E_{θ^{'}}$ would never bother re-using its model of a human because it would be faster to just hard-code all the facts about how humans answer questions.

In addition to feeling a bit silly, there are some obvious problems:

Maybe $F_{θ}$ does happen to compute the instrumental policy (e.g. because its thinking about a human who is thinking about the instrumental policy...) and so the fastest thing for $E_{θ^{'}}$ to do is just to look up the instrumental policy and use it. And more generally there could be other things for $E_{θ^{'}}$ to do that are faster than the intended model.
The reason why we expect $E_{θ^{'}}$ to be able to extract knowledge from $Z$ is that $F_{θ}$ needs to be able to extract that knowledge. For example, $F_{θ}$ needs to be able to read sentences written in its internal language in order to perform deduction with them, so we know there is some fast way to do that. However, there are some disanalogies between the problems faced by E and F. A small issue is that F is "cooperating" with itself rather than trying to read off values from a fixed intermediate. A potentially bigger issue is that they use different priors (and F may use lots of weight-sharing and be relatively computationally slow when utilizing its own intermediate representation), and it's totally unclear what architectures for E could potentially work.

Some random thoughts:

This approach has a vibe more like interpretability, where we first train a model and then search over interpretations of its activations, rather than other approaches I've described recently that are focused on jointly learning good predictions and question-answering. That said, in practice I do expect that we would need to learn E "as we go" in order for it to have a chance of "keeping up" with F (since otherwise F may have developed to a very arcane state from which SGD has no chance of learning a reasonable decoding).
It's kind of weird to use something like a speed prior over E rather than something that tries to be a closer match for the "real" human prior. I expect to use a more "real" prior in the end but at this point I don't really know what's going on and am just looking for something that gets traction.

[-]paulfchristiano4y30

The most fundamental reason that I don't expect this to work is that it gives up on "sharing parameters" between the extractor and the human model. But in many cases it seems possible to do so, and giving on up on that feels extremely unstable since it's trying to push against competitiveness (i.e. the model will want to find some way to save those parameters, and you don't want your intended solution to involve subverting that natural pressure).

Intuitively, I can imagine three kinds of approaches to doing this parameter sharing:

Introduce some latent structure (e.g. semantics of natural language, what a cat "actually is") that is used to represent both humans and the intended question-answering policy. This is the diagram $H \leftarrow L \to f^{+}$
Introduce some consistency check $f^{?}$ between $H$ and $L$ . This is the diagram $H \to f^{?} \leftarrow f^{+}$
Somehow extract $f^{+}$ from $H$ or build it out of pieces derived from $H$ . This is the diagram $H \to f^{+}$ . This is kind of like a special case of 1, but it feels pretty different.

(You could imagine having slightly more general diagrams corresponding to any sort of d-connection between $H$ and $f^{+}$ .)

Approach 1 is the most intuitive, and it seems appealing because we can basically leave it up to the model to introduce the factorization (and it feels like there is a good chance that it will happen completely automatically). There are basically two challenges with this approach:

It's not clear that we can actually jointly compress $H$ and $f^{+}$ . For example, what if we represent $H$ in an extremely low level way as a bunch of neurons firing; the neurons are connected in a complicated and messy way that learned to implement something like $f^{+}$ , but need not have any simple representation in terms of $f^{+}$ . Even if such a factorization is possible, it's completely unclear how to argue about how hard it is to learn. This is a lot of what motivates the compression-based approaches---we can just say " $H$ is some mess, but you can count on it basically computing $f^{+}$ " and then make simple arguments about competitiveness (it's basically just as hard as separately learning $H$ and $f^{+}$ ).
If you overcame that difficulty, you'd still have to actually incentivize this kind of factorization in the model (rather than sharing parameters in the unintended way). It's unclear how to do that (maybe you're back to think about something speed-prior like, and this is just a way to address my concern about the speed-prior-like proposals), but this feels more tractable than the first problem.

I've been thinking about approach 2 over the last 2 months. My biggest concern is that it feels like you have to pay the bits of H back "as you learn them" with SGD, but you may learn them in such a way that you don't really get a useful consistency update until you've basically specified all of H. (E.g. suppose you are exposed to brain scans of humans for a long time before you learn to answer questions in a human-like way. Then at the end you want to use that to pay back the bits of the brain scans, but in order to do so you need to imagine lots of different ways the brain scans could have looked. But there's no tractable way to do that, because you have to fill in the the full brain scan before it really tells you about whether your consistency condition holds.)

Approach 3 is in some sense most direct. I think this naturally looks like imitative generalization, where you use a richer set of human answers to basically build $f^{+}$ on top of your model. I don't see how to make this kind of thing work totally on its own, but I'm probably going to spend a bit of time thinking about how to combine it with approaches 1 and 2.

[-]paulfchristiano4y20

I think the biggest problem is that can compute the instrumental policy (or a different policy that works well, or a fragment of it). Some possible reasons:

Maybe some people in the world are incidentally thinking about the instrumental policy and $F_{θ}$ makes predictions about them.
Maybe an adversary who computes a policy that performs well in order to try to attack the learning process (since $E_{θ^{'}}$ may just copy the adversary's policy in order to be fast if it works well on training, resulting in bad generalization).
Maybe no one computes the instrumental policy itself, but they compute helpful building blocks and $E_{θ^{'}}$ can save compute by reusing those.
Maybe $F_{θ}$ wants to predict question-answering policies, and computing optimal responses to the training objective is a simple way for it to forecast what they will do. (It's totally unclear if this is actually more efficient than the intended policy.) Then $E_{θ^{'}}$ can copy that.

I don't know if any of those particular failures are too likely. But overall it seems really bad to rely on $F_{θ}$ never computing something inconvenient, and it definitely doesn't look like it's going to work in the worst case.

What are some possible outs, if in fact $F_{θ}$ computes something adversarial to try to make it easy for $E_{θ^{'}}$ to learn something bad?

At the time when $F_{θ}$ is running it doesn't know the question Q that $E_{θ}$ will need to answer. So in order to cause trouble it would need to compute answers for all of these questions. But I'm afraid that this is doable in many cases, since $F_{θ}$ could translate a big chunk of its internal state into natural language and then $E_{θ^{'}}$ can do some simple QA on top of that.
We could try to ensure that the correctness/consistency checks are really complex or challenging to meet, at least for some questions. If the complexity of these checks scales up with the complexity of $F_{θ}$ itself (e.g. by using amplification where the evaluator consults a model as big as $F_{θ}$ ), then maybe it's impossible for $F_{θ}$ to meet them. This idea can be applied without the factorization into F and E, but it's probably useful to have this kind of factorization in mind as a possible tool when exploring approaches based on tricky-evaluations.
We could try to exploit some further structural facts about the parts of $Z_{θ} (X)$ that are used by $E_{θ^{'}}$ . For example, it feels like the intended model is going to be leveraging facts that are further "upstream." For example, suppose an attacker observes that there is a cat in the room, and so writes out "There is a cat in the room" as part of a natural-language description of what it's going on that it hopes that $E_{θ^{'}}$ will eventually learn to copy. If $F_{θ}$ predicts the adversary's output, it must first predict that there is actually a cat in the room, which then ultimately flows downstream into predictions of the adversary's behavior. And so we might hope to prefer the "intended" $E_{θ^{'}}$ by having it preferentially read from the earlier activations (with shorter computational histories).

Overall this kind of approach feels like it's probably doomed, but it does capture part of the intuition for why we should "just" be able to learn a simple correspondence rather than getting some crazy instrumental policy. So I'm not quite ready to let it go yet. I'm particularly interested to push a bit on the third of these approaches.

[-]paulfchristiano4y20

Here's another approach to "shortest circuit" that is designed to avoid this problem:

Learn a circuit that outputs an entire set of beliefs. (Or maybe some different architecture, but with ~0 weight sharing so that computational complexity = description complexity.)
Impose a consistency requirement on those beliefs, even in cases where a human can't tell the right answer.
Require $C (X)$ 's beliefs about $Y$ to match $F_{θ} (X)$ . We hope that this makes $C$ an explication of " $F_{θ}$ 's beliefs."
Optimize some combination of (complexity) vs (usefulness), or chart the whole pareto frontier, or whatever. I'm a bit confused about how this step would work but there are similar difficulties for the other posts in this genre so it's exciting if this proposal gets to that final step.

The "intended" circuit $C$ just follows along with the computation done by $F_{θ}$ and then translates its internal state into natural language.

What about the problem case where $F_{θ}$ computes some reasonable beliefs (e.g. using the instrumental policy, where the simplicity prior makes us skeptical about their generalization) that $C$ could just read off? I'll imagine those being written down somewhere on a slip of paper inside of $F_{θ}$ 's model of the world.

Suppose that the slip of paper is not relevant to predicting $F_{θ} (X)$ , i.e. it's a spandrel from the weight sharing. Then the simplest circuit $C$ just wants to cut it out. Whatever computation was done to write things down on the slip of paper can be done directly by $C$ , so it seems like we're in business.
So suppose that the slip of paper is relevant for predicting $F_{θ} (X)$ , e.g. because someone looks at the slip of paper and then takes an action that affects $Y$ . If (the correct) $Y$ is itself depicted on the slip of paper, then we can again cut out the slip of paper itself and just run the same computation (that was done by whoever wrote something on the slip of paper). Otherwise, the answers produced by $C$ still have to contain both the items on the slip of paper as well as some facts that are causally downstream of the slip of paper (as well as hopefully some about the slip of paper itself). At that point it seems like we have a pretty good chance of getting a consistency violation out of $C$ .

Probably nothing like this can work, but I now feel like there are two live proposals for capturing the optimistic minimal circuits intuition---the one in this current comment, and in this other comment. I still feel like the aggressive speed penalization is doing something, and I feel like probably we can either find a working proposal in that space or else come up with some clearer counterexample.

[-]paulfchristiano4y20

We could try to exploit some further structural facts about the parts of that are used by $E_{θ^{'}}$ . For example, it feels like the intended model is going to be leveraging facts that are further "upstream." For example, suppose an attacker observes that there is a cat in the room, and so writes out "There is a cat in the room" as part of a natural-language description of what it's going on that it hopes that $E_{θ^{'}}$ will eventually learn to copy. If $F_{θ}$ predicts the adversary's output, it must first predict that there is actually a cat in the room, which then ultimately flows downstream into predictions of the adversary's behavior. And so we might hope to prefer the "intended" $E_{θ^{'}}$ by having it preferentially read from the earlier activations (with shorter computational histories).

The natural way to implement this is to penalize $E_{θ^{'}}$ not for the computation it does, but for all the computation needed to compute its output (including within $F_{θ}$ .). The basic problem with this approach is that it incentivizes $E_{θ^{'}}$ to do all of the computation of $F_{θ}$ from scratch in a way optimized for speed rather than complexity. I'd set this approach aside for a while because of this difficulty and the unnaturalness mentioned in the sibling (where we've given up on what seems to be an important form of parameter-sharing).

Today I was thinking about some apparently-totally-different angles of attack for the ontology identification problem, and this idea seems to have emerged again, with a potential strategy for fixing the "recompute $F_{θ}$ problem". (In the context of ontology identification, the parameter-sharing objection no longer applies.)

Here's the idea:

Train $F_{θ}$ as before.
Start with a bunch of facts and probabilistic relationships that the human knows, expressed in their own ontology. These might be facts like "Ice melts at 100 degrees" or "Dogs bark at cats" or whatever.
We are going to try to jointly learn (i) a correspondence $c$ between $F_{θ}$ and the human's ontology, (ii) a set of "justifications" showing that $c (F_{θ})$ satisfies all of the relationships the human expects. I'm imagining justifications like simulating ice at 100 degrees and observing that it indeed melts, or sampling situations with dogs and cats and verifying that the dogs bark.
The correspondence $c$ is constrained to map "the observations" (a concept in the human's ontology) to the output of $F_{θ}$ , but other than that there is no simplicity prior, it can be anything.
Our goal is to make the justifications as computationally simple as possible. (Which indirectly incentivizes us to make $c$ as computationally simple as possible.)

This still feels a bit weird, but you could imagine it handling a bunch of cases in a promising way:

$c$ has no reason to "reuse the human" or to use the human's inference procedure in order to save bits, because doing either of those things makes the justifications more computationally complex.
If there is a human embedded inside $F_{θ}$ who performs the correspondence and writes it down in an easy-to-read way, $c$ has no motivation to read it: if $c$ is defined in that way, then justifying facts will require simulating the human (even if that work was already done inside $F_{θ}$ ). If $c$ simply cut out the middle man and applied the correspondence itself, then it could save compute in the typical case (except when talking about facts about that human). This is subtle in a few ways but tentatively looks plausible to me.
$c$ has no reason to ignore $F_{θ}$ and implement a new more-efficient-but-more-complex simulation $F^{'}$ , because (i) it ultimately needs to relate observations back to the output of $F_{θ}$ , and many of its concepts are related to observations (e.g. what cats look like), (ii) that forces $F^{'}$ and $F_{θ}$ to have the same behavior, (iii) the justification would then need to show that the the "observations" in $F^{'}$ are the same as the observations in $F_{θ}$ , which is computationally costly.

But right now it's a pretty vague proposal, because it's unclear what the nature of these facts or justifications are. If you set that up in a naive way, then the justification effectively just needs to simulate all of $F_{θ}$ . That's a problem because it reintroduces the failure mode where you need to simulate the human, and therefore there's no extra cost to just simulating and then listening to whatever they say.

Overall I think that probably nothing like this works, but I'm still feeling a lot more optimistic than I was last week and want to explore it further. (This is partially for reasons not discussed in this comment, that several other approaches/motivations seem to converge on something similar.)

[-]paulfchristiano4y20

Here's a slightly more formal algorithm along these lines:

Assume that both the human's model and the AI's model $W_{A I}$ are Bayesian networks where you compute the probability distribution over a node $v$ 's value based on the values of its parents $p a (v)$ . I'll write $V a l u e s (v)$ for the set of values that a node $v$ can take on (in either model), and $V a l u e s (S)$ for the joint values of a set of nodes $S$ .
A correspondence tells you how to compute the value of each node $v$ in the human's model. This consistent of (i) a neighborhood in the AI's model $N_{A I} (v)$ which suffices to determine $v$ , (ii) a function $f_{v} : V a l u e s (N_{A I} (v)) \to V a l u e s (v)$ .
Both the AI's model and the human model contain some distinguished observation nodes. $f$ must be the identity on these nodes.
An "explanation" of a correspondence consists of a set of nodes $E_{A I} (v)$ in the AI's model for each node $v$ in the human's model. The intuition is that we can run a simulation involving only these nodes in order to reproduce the probability distribution of $v$ given its parents' values.
In particular, $N_{A I} (v) \subseteq E_{A I} (v)$ , and $N_{A I} (u) \subseteq E_{A I} (v)$ for all $u \in p a (v)$ . In order to check whether $E_{A I} (v)$ reproduces the right distribution, we first sample values at random for all the nodes some of whose parents aren't in $E_{A I} (v)$ . Then we sample values for the remaining nodes. We can use $f$ to compute the corresponding values for $v$ and all of its parents. And then we can compute the conditional distributions for $v$ given each set of values for its parents.
We require that the explanations exactly reproduce the conditional probability over $V a l u e s (v)$ given $V a l u e s (p a (v))$ .
The "cost" of the explanation of $v$ is the sum of the compute required to sample all the nodes in $E_{A I} (v)$ . The "cost" of the correspondence $f_{v}$ is the compute required to evaluate it.
We search for the set of correspondences and explanations for which the total cost is minimized.
(Maybe we also have some requirement where the correspondence $f_{v}$ agrees with some training data about $v$ . I'm not really sure about that.)

Reviewing how this behaves in each of the bad cases from the parent:

It's very bad to define $f_{v}$ by computing the observation and then using the human's inference algorithm. The entire motivation for defining it this way was to save on description complexity, but $f_{v}$ is only being penalized for computational complexity. (This also forces every single $E_{A I} (v)$ to include the entire process that generates the observations, which seems like it should be an even bigger deal. But this feels less like the "right" reason and I think it might change for a future version of the scheme.)
If there is a human in the simulation who knows the value of $v$ , it's extremely bad to define $N_{A I} (v)$ to be that human. This is because the explanation $E_{A I} (v)$ will need to include the entire process by which the human computes that value. This is guaranteed to be more expensive than just performing that compute in the correspondence $f_{v}$ itself.
We could define $f_{v}$ to compute the value of $v$ from scratch using the initial values of the simulation, using some more-efficient simulation. But we can't do this for the observation node $O b s$ , since it is required to map up with the observation node in the AI's model. So in order to explain the observation node we will need to have $E_{A I} (O b s)$ include the AI's entire model, or at least the parts leading up to the observation. This means we are basically paying for two full copies of the computation, and so we're not really benefiting from the fact that the second copy is more efficient. (We are also potentially overlapping a lot of computation between the different $f_{v}$ 's, but again I'm a bit less sure how robust that is and whether it will hold up in different formalizations.)

There are a lot of problems and missing details in this proposal:

This requires exactly reproducing the conditional probabilities in the human's model. But that's totally unrealistic for anything short of a full simulation---and even for the full simulation it wouldn't reproduce the probabilities since the human model isn't accurate. So we need some way to choose a good enough explanation, i.e. a way of balancing the computational complexity of the explanation against the quality of the conditional probabilities that come out.
We're sampling the inputs to $E_{A I} (v)$ uniformly at random. This seems unlikely to work in general. We could easily sample each node from its marginal, but most of the action is in the correlation. Allowing arbitrary correlations causes problems (since you could just specify the "human is accurate" correlation and then read off the correct answers from there). So I think probably some more flexible system is needed here; there are a lot of options but it's tricky.
There is something problematic about the overlapping explanations $E_{A I} (v)$ . If they overlap you need to pay for all of them, but for the intended mapping there will often be quite significant overlap. This isn't inherently a problem, but I'm scared that it's going to introduce a lot of pressure towards some different correspondence that is able to avoid that problem. We need to penalize overlap because the case where the training data is embedded in the model---the main problem with that model is that you need to separately explain every way in which the human is correct with highly overlapping explanations. If you didn't penalize those then you may just end up with the embedded explanations (for which $f_{v}$ is extremely cheap).
There is something tricky about uniformity in the model and in the implementations of $f_{v}$ .
I'm still scared about the "recompute everything from scratch" failure mode. The model does need to have a single explanation $E_{A I} (O b s)$ that needs to include the whole model. But (i) it doesn't have to reproduce work, (ii) it can cut out all the stuff not on the path to the observation. So the obvious reason that this one loses is by the duplicated work in $f_{v}$ . Hopefully that's actually robust.
We are making really strong structural assumptions on the models and the correspondence between them. We get some things for free (because humans actually do have extra structure in our beliefs about the world that is properly part of the problem statement, and the AI's model is constrained by its architecture) but not nearly this much.

Overall I'm becoming significantly more optimistic that something like this will work (though still less likely than not). Trying to step back and see the big picture, it seems like there are three key active ingredients:

Using "speed" instead of "simplicity" as the ~only requirement for these correspondences.
Having separate correspondences for separate properties and not allowing them to share tons of computation with each other (to prevent re-running the whole simulation).
Forcing the model to explain correlations, so that using an "embedded" copy of the answers (like a simulation of the data-generating process) forces you to reproduce the computation that produced that answer.

My next step would probably be looking at cases where these high-level ingredients aren't sufficient (e.g. are there cases where "generate obs then do inference in the human model" is actually cheaper?). If they look pretty good, then I'll spend some more time trying to fill in the details in a more plausible way.

[-]paulfchristiano4y20

We might be able to get similar advantages with a more general proposal like:

Fit a function f to a (Q, A) dataset with lots of questions about latent structure. Minimize the sum of some typical QA objective and the computational cost of verifying that f is consistent.

Then the idea is that matching the conditional probabilities from the human's model (or at least being consistent with what the human believes strongly about those conditional probabilities) essentially falls out of a consistency condition.

It's not clear how to actually formulate that consistency condition, but it seems like an improvement over the prior situation (which was just baking in the obviously-untenable requirement of exactly matching). It's also not clear what happens if this consistency condition is soft.

It's not clear what "verify that the consistency conditions are met" means. You can always do the same proposal as in the parent, though it's not really clear if that's a convincing verification. But I think that's a fundamental philosophical problem that both of these proposals need to confront.

It's not clear how to balance computational cost and the QA objective. But you are able to avoid most of the bad properties just by being on the Pareto frontier, and I don't think this is worse than the prior proposal.

Overall this approach seems like it could avoid making such strong structural assumptions about the underlying model. It also helps a lot with the overlapping explanations + uniformity problem. And it generally seems to be inching towards feeling plausible.

[-]adamShimi4y10

One aspect of this proposal which I don't know how to do is evaluation the answers of the question-answerer. That looks too me very related to the deconfusion of universality that we discussed a few months ago, and without an answer to this, I feel like I don't even know how to run this silly approach.

[-]paulfchristiano4y30

You could imitate human answers, or you could ask a human "Is answer much better than answer $A$ ?" Both of these only work for questions that humans can evaluate (in hindsight), and then the point of the scheme is to get an adequate generalization to (some) questions that humans can't answer.

[-]adamShimi4y10

Ok, so you optimize the circuit both for speed and for small loss on human answers/comparisons, hoping that it generalizes to more questions while not being complex enough to be deceptive. Is that what you mean?

[-]paulfchristiano4y30

I'm mostly worried about parameter sharing between the human models in the environment and the QA procedure (which leads the QA to generalize like a human instead of correctly). You could call that deception but I think it's a somewhat simpler phenomenon.

[-]paulfchristiano4y60

Recently I've been thinking about ML systems that generalize poorly (copying human errors) because of either re-using predictive models of humans or using human inference procedures to map between world models.

My initial focus was on preventing re-using predictive models of humans. But I'm feeling increasingly like there is going to be a single solution to the two problems, and that the world-model mismatch problem is a good domain to develop the kind of algorithm we need. I want to say a bit about why.

I'm currently thinking about dealing with world model mismatches by learning a correspondence between models using something other than a simplicity prior / training a neural network to answering questions. Intuitively we want to do something more like "lining up" the two models and seeing what parts correspond to which others. We have a lot of conditions/criteria for such alignments, so we don't necessarily have to just stick with simplicity. This comment fleshes out one possible approach a little bit.

If this approach succeeds, then it also directly applicable to avoiding re-using human models---we want to be lining up the internal computation of our model with concepts like "There is a cat in the room" rather than just asking the model to predict whether there is a cat however it wants (which it may do by copying a human labeler). And on the flip side, I think that the "re-using human models" problem is a good constraint to have in mind when thinking about ways to do this correspondence. (Roughly speaking, because something like computational speed or "locality" seems like a really central constraint for matching up world models, and doing that approach naively can greatly exacerbate the problems with copying the training process.)

So for now I think it makes sense for me to focus on whether learning this correspondence is actually plausible. If that succeeds then I can step back and see how that changes my overall view of the landscape (I think it might be quite a significant change), and if it fails then I hope to at least know a bit more about the world model mismatch problem.

I think the best analogy in existing practice is probably doing interpretability work---mapping up the AI's model to my model is kind of like looking at neurons and trying to make sense of what they are computing (or looking for neurons that compute something). And giving up on a "simplicity prior" is very natural when doing interpretability, instead using other considerations to determine whether a correspondence is good. It still seems kind of plausible that in retrospect my current work will look like it was trying to get a solid theoretical picture on what interpretability should do (including in the regime where the correspondence is quite complex, and when the goal is a much more complete level of understanding). I swing back and forth on how strong the analogy to interpretability seems / whether or not this is how it will look in retrospect. (But at any rate, my research methodology feels like a very different approach to similar questions.)

[-]paulfchristiano4y40

(To restate the obvious, all of the stuff here is extremely WIP and rambling.)

I've often talked about the case where an unaligned model learns a description of the world + the procedure for reading out "what the camera sees" from the world. In this case, I've imagined an aligned model starting from the unaligned model and then extracting additional structure.

It now seems to me that the ideal aligned behavior is to learn only the "description of the world" and then have imitative generalization take it from there, identifying the correspondence between the world we know and the learned model. That correspondence includes in particular "what the camera sees."

The major technical benefit of doing it this way is that we end up with a higher prior probability on the aligned model than the unaligned model---the aligned one doesn't have to specify how to read out observations. And specifying how to read out observations doesn't really make it easier to find that correspondence.

We still need to specify how the "human" in imitative generalization actually finds this correspondence. So this doesn't fundamentally change any of the stuff I've recently been thinking about, but I think that the framing is becoming clearer and it's more likely we can find our way to the actually-right way to do it.

It now seems to me that a core feature of the situation that lets us pull out a correspondence is that you can't generally have two equally-valid correspondences for a given model---the standards for being a "good correspondence" are such that it would require crazy logical coincidence, and in fact this seems to be the core feature of "goodness." For example, you could have multiple "correspondences" that effectively just recompute everything from scratch, but by exactly the same token those are bad correspondences.

(This obviously only happens once the space and causal structure is sufficiently rich. There may be multiple ways of seeing faces in clouds, but once your correspondence involves people and dogs and the people talking about how the dogs are running around, it seems much more constrained because you need to reproduce all of that causal structure, and the very fact that humans can make good judgments about whether there are dogs implies that everything is incredibly constrained.)

There can certainly be legitimate ambiguity or uncertainty. For example, there may be a big world with multiple places that you could find a given pattern of dogs barking at cats. Or there might be parts of the world model that are just clearly underdetermined (e.g. there are two identical twins and we actually can't tell which is which). In these cases the space of possible correspondences still seems effectively discrete, rather than being a massive space parameterized as neural networks or something. We'd be totally happy surfacing all of the options in these cases.

There can also be a bunch of inconsequential uncertainty, things that feel more like small deformations of the correspondence than moving to a new connected component in correspondence-space. Things like slightly adjusting the boundaries of objects or of categories.

I'm currently thinking about this in terms of: given two different correspondences, why is it that they manage to both fit the data? Options:

They are "very close," e.g. they disagree only rarely or make quantitatively similar judgments.
One of them is a "bad correspondence" and could fit a huge range of possible underlying models, i.e. it's basically introducing the structure we are interested in within the correspondence itself.
The two correspondences are "not interacting," they aren't competing to explain the same logical facts about the underlying model. (e.g. a big world, one correspondence faces .)
There is an automorphism of my model of the world (e.g. I could exchange the two twins Eva and Lyn), and can compose a correspondence with that automorphism. (This seems much more likely to happen for poorly-understood parts of the world, like how we talk about new physics, than for simple things like "is there a cat in the room.")

I don't know where all of this ends up, but I feel some pretty strong common-sense intuition like "If you had some humans looking at the model, they could recognize a good correspondence when they saw it" and for now I'm going to be following that to see where it goes.

I tentatively think the whole situation is basically the same for "intuition module outputs a set of premises and then a deduction engine takes it from there" as for a model of physics. That is, it's still the case that (assuming enough richness) the translation between the "intuition module"'s language and human language is going to be more or less pinned down uniquely, and we'll have the same kind of taxonomy over cases where two translations would work equally well.

[-]paulfchristiano4y40

Here's an example I've been thinking about today to investigate the phenomenon of re-using human models.

Suppose that the "right" way to answer questions is . And suppose that a human is a learned model $H_{θ_{2}}$ trained by gradient descent to approximate $f_{θ_{1}}$ (subject to architectural and computational constraints). This model is very good on distribution, but we expect it to fail off distribution. We want to train a new neural network to approximate $f_{θ_{1}}$ , without inheriting the human's off-distribution failures (though the new network may have off-distribution failures of its own).

The problem is that our model needs to learn the exact parameters $θ_{2}$ for the human model in order to other aspects of human behavior. The simplest case is that we sometimes directly open human brains to observe $θ_{2}$ directly.

Once we've learned $θ_{2}$ it is very easy to learn the question-answering policy $H_{θ_{2}}$ . So we're worried that our model will do that rather than learning the additional parameters $θ_{1}$ to implement $f_{θ_{1}}$ .

Intuitively there is a strong connection between $f_{θ_{1}}$ and $H_{θ_{2}}$ . After all, $θ_{2}$ is optimized to make them nearly equal on the training distribution. If you understood the dynamics of neural network training it is likely possible to essentially reconstruct $f_{θ_{1}}$ from $θ_{2}$ , i.e. the complexity fo specifying both $θ_{1}$ and $θ_{2}$ is essentially the same as the complexity of specifying only $θ_{2}$ .

But it's completely unclear how to jointly represent $f_{θ_{1}}$ and $H_{θ_{2}}$ using some parameters $θ_{3}$ of similar size to $θ_{2}$ . So prima facie there is a strong temptation to just reuse $H_{θ_{2}}$ .

[-]paulfchristiano4y30

How much hope is there for jointly representing and $H_{θ_{2}}$ ?

The most obvious representation in this case is to first specify $f_{θ_{1}}$ , and then actually model the process of gradient descent that produces $H_{θ_{2}}$ . This runs into a few problems:

Actually running gradient descent to find $θ_{2}$ is too expensive to do at every datapoint---instead we learn a hypothesis that does a lot of its work "up front" (shared across all the datapoints). I don't know what that would look like. The naive ways of doing it (redoing the shared initial computation for every batch) only work for very large batch sizes, which may well be above the critical batch size. If this was the only problem I'd feel pretty optimistic about spending a bunch of time thinking about it.
Specifying "the process that produced $H_{θ_{2}}$ " requires specifying the initialization ${˜ θ}_{2}$ , which is as big as $θ_{2}$ . That said, the fact that the learned $θ_{2}$ also contains a bunch of information about $θ_{1}$ means that it can't contain perfect information about the initialization ${˜ θ}_{2}$ , i.e. that multiple initializations lead to exactly the same final state. So that suggests a possible out: we can start with an "initial" initialization ${˜ θ}_{2}^{0}$ . Then we can learn ${˜ θ}_{2}$ by gradient descent. The fact that many different values of ${˜ θ}_{2}$ would work suggests that it should be easier to find one of them; intuitively if we set up training just right it seems like we may be able to get all the bits back.
Running the gradient descent to find $θ_{2}$ even a single time may be much more expensive than the rest of training. That is, human learning (perhaps extended over biological or cultural evolution) may take much more time than machine learning. If this the case, then any approach that relies on reproducing that learning is completely doomed.

Similar to problem 2, the mutual information between $θ_{2}$ and the learning process that produced $θ_{2}$ also must be kind of low---there are only $| θ_{2} |$ bits of mutual information to go around between the learning process, its initialization, and $θ_{1}$ . But exploiting this structure seems really hard, if actually there aren't any fast learning processes that lead to the same conclusion

My current take is that even in the case where $θ_{2}$ was actually produced by something like SGD, we can't actually exploit that fact to produce a direct, causally-accurate representation $θ_{2}$ .

That's kind of similar to what happens in my current proposal though: instead we use the learning process embedded inside the broader world-model learning. (Or a new learning process that we create from fresh to estimate the specialness of $θ_{2}$ , as remarked in the sibling comment.)

So then the critical question is not "do we have enough time to reproduce the learning process that lead to $θ_{2}$ ?" it is "Can we directly learn $H_{θ_{2}}$ as an approximation to $f_{θ_{1}}$ ?" If we able to do this in any way, then we can use that to help compress $θ_{2}$ . In the other proposal, we can use it to help estimate the specialness of $θ_{2}$ in order to determine how many bits we get back---it's starting to feel like these things aren't so different anyway.

Fully learning the whole human-model seems impossible---after all, humans may have learned things that are more sophisticated then what we can learn with SGD (even if SGD learned a policy with "enough bits" to represent $θ_{2}$ , so that it could memorize them one by one if it saw the brain scans or whatever).

So we could try to do something like "learning just the part of the human policy is that is about answering questions." But it's not clear to me how you could disentangle this from all the rest of the complicated stuff going in for the human.

Overall this seems like a pretty tricky case. The high-level summary is something like: "The model is able to learn to imitate humans by making detailed observations about humans, but we are not able to learn a similarly-good human model from scratch given data about what the human is 'trying' to do or how they interpret language." Under these conditions it seems particularly challenging to either jointly represent $H_{θ_{2}}$ and $f_{θ_{1}}$ , or to compute how many bits you should "get back" based on a consistency condition between them. I expect it's going to be reasonably obvious what to do in this case (likely exploiting the presumed limitation of our learning process), which is what I'll be thinking about now.

[-]paulfchristiano4y30

The difficulty of jointly representing and $H_{θ_{2}}$ motivates my recent proposal, which avoids any such explicit representation. Instead it separately specifies $θ_{1}$ and $θ_{2}$ , and then "gets back" bits by imposing a consistency condition that would have been satisfied only for a very small fraction of possible $θ_{2}$ 's (roughly $exp (- | θ_{1} |)$ of them).

But thinking about this neural network case also makes it easy to talk about why my recent proposal could run into severe computational problems:

In order to calculate this loss function we need to evaluate how "special" $θ_{2}$ is, i.e. how small is the fraction of $θ_{2}$ 's that are consistent with $θ_{1}$
In order to evaluate how special $θ_{2}$ is, we basically need to do the same process of SGD that produces $θ_{2}$ ---then we can compare the actual iterates to all of the places that it could have gone in a different direction, and conclude that almost all of the different settings of the parameters would have been much less consistent with $θ_{1}$ .
The implicit hope of my proposal is that the outer neural network is learning its human model using something like SGD, and so it can do this specialness-calculation for free---it will be considering lots of different human-models, and it can observe that almost all of them are much less consistent with $θ_{1}$ .
But the outer neural network could learn to model humans in a very different way, which may not involve representing a serious of iterates of "plausible alternative human models." For example, suppose that in each datapoint we observe a few of the bits of $θ_{2}$ directly (e.g. by looking at a brain scan), and we fill in much of $θ_{2}$ in this way before we ever start making good predictions about human behavior. Then we never need to consider any other plausible human-models.

So in order to salvage a proposal like this, it seems like (at a minimum) the "specialness evaluation" needs to take place separately from the main learning of the human model, using a very different process (where we consider lots of different human models and see that it's actually quite hard to find one that is similarly-consistent with $θ_{1}$ ). This would take place at the point where the outer model started actually using its human model $H_{θ_{2}}$ in order to answer questions.

I don't really know what that would look like or if it's possible to make anything like that work.

[-]paulfchristiano4y30

Suppose I am interested in finding a program M whose input-output behavior has some property P that I can probabilistically check relatively quickly (e.g. I want to check whether M implements a sparse cut of some large implicit graph). I believe there is some simple and fast program M that does the trick. But even this relatively simple M is much more complex than the specification of the property P.

Now suppose I search for the simplest program running in time T that has property P. If T is sufficiently large, then I will end up getting the program "Search for the simplest program running in time T' that has property P, then run that." (Or something even simpler, but the point is that it will make no reference to the intended program M since encoding P is cheaper.)

I may be happy enough with this outcome, but there's some intuitive sense in which something weird and undesirable has happened here (and I may get in a distinctive kind of trouble if P is an approximate evaluation). I think this is likely to be a useful maximally-simplified example to think about.

[-]paulfchristiano4y30

The speed prior is calibrated such that this never happens if the learned optimizer is just using brute force---if it needs to search over 1 extra bit then it will take 2x longer, offsetting the gains.

That means that in the regime where P is simple, the speed prior is the "least you can reasonably care about speed"---if you care even less, you will just end up pushing the optimization into an inner process that is more concerned with speed and is therefore able to try a bunch of options.

(However, this is very mild, since the speed prior cares only a tiny bit about speed. Adding 100 bits to your program is the same as letting it run 2^100 times longer, so you are basically just optimizing for simplicity.)

To make this concrete, suppose that I instead used the kind-of-speed prior, where taking 4x longer is equivalent to using 1 extra bit of description complexity. And suppose that P is very simple relative to the complexities of the other objects involved. Suppose that the "object-level" program M has 1000 bits and runs in 2^2000 time, so has kind-of-speed complexity 2000 bits. A search that uses the speed prior will be able to find this algorithm in 2^3000 time, and so will have a kind-of-speed complexity of 1500 bits. So the kind-of-speed prior will just end up delegating to the speed prior.

[-]paulfchristiano4y20

The speed prior still delegates to better search algorithms though. For example, suppose that someone is able to fill in a 1000 bit program using only 2^500 steps of local search. Then the local search algorithm has speed prior complexity 500 bits, so will beat the object-level program. And the prior we'd end up using is basically "2x longer = 2 more bits" instead of "2x longer = 1 more bit," i.e. we end up caring more about speed because we delegated.

The actual limit on how much you care about speed is given by whatever search algorithms work best. I think it's likely possible to "expose" what is going on to the outer optimizer (so that it finds a hypothesis like "This local search algorithm is good" and then uses it to find an object-level program, rather than directly finding a program that bundles both of them together). But I'd guess intuitively that it's just not even meaningful to talk about the "simplest" programs or any prior that cares less about speed than the optimal search algorithm.

[-]paulfchristiano4y20

This is interesting to me for two reasons:

[Mainly] Several proposals for avoiding the instrumental policy work by penalizing computation. But I have a really shaky philosophical grip on why that's a reasonable thing to do, and so all of those solutions end up feeling weird to me. I can still evaluate them based on what works on concrete examples, but things are slippery enough that plan A is getting a handle on why this is a good idea.
In the long run I expect to have to handle learned optimizers by having the outer optimizer instead directly learn whatever the inner optimizer would have learned. This is an interesting setting to look at how that works out. (For example, in this case the outer optimizer just needs to be able to represent the hypothesis "There is a program that has property P and runs in time T' " and then do its own search over that space of faster programs.)

[-]paulfchristiano4y20

In traditional settings, we are searching for a program M that is simpler than the property P. For example, the number of parameters in our model should be smaller than the size of the dataset we are trying to fit if we want the model to generalize. (This isn't true for modern DL because of subtleties with SGD optimizing imperfectly and implicit regularization and so on, but spiritually I think it's still fine..)

But this breaks down if we start doing something like imposing consistency checks and hoping that those change the result of learning. Intuitively it's also often not true for scientific explanations---even simple properties can be surprising and require explanation, and can be used to support theories that are much more complex than the observation itself.

Some thoughts:

It's quite plausible that in these cases we want to be doing something other than searching over programs. This is pretty clear in the "scientific explanation" case, and maybe it's the way to go for the kinds of alignment problems I've been thinking about recently.

A basic challenge with searching over programs is that we have to interpret the other data. For example, if "correspondence between two models of physics" is some kind of different object like a description in natural language, then some amplified human is going to have to be thinking about that correspondence to see if it explains the facts. If we search over correspondences, some of them will be "attacks" on the human that basically convince them to run a general computation in order to explain the data. So we have two options: (i) perfectly harden the evaluation process against such attacks, (ii) try to ensure that there is always some way to just directly do whatever the attacker convinced the human to do. But (i) seems quite hard, and (ii) basically requires us to put all of the generic programs in our search space.
It's also quite plausible that we'll just give up on things like consistency conditions. But those come up frequently enough in intuitive alignment schemes that I at least want to give them a fair shake.

[-]paulfchristiano4y20

Causal structure is an intuitively appealing way to pick out the "intended" translation between an AI's model of the world and a human's model. For example, intuitively "There is a dog" causes "There is a barking sound." If we ask our neural net questions like "Is there a dog?" and it computes its answer by checking "Does a human labeler think there is a dog?" then its answers won't match the expected causal structure---so maybe we can avoid these kinds of answers.

What does that mean if we apply typical definitions of causality to ML training?

If we define causality in terms of interventions, then this helps iff we have interventions in which the labeler is mistaken. In general, it seems we could just include examples with such interventions in the training set.
Similarly, if we use some kind of closest-possible-world semantics, then we need to be able to train models to answer questions consistently about nearby worlds in which the labeler is mistaken. It's not clear how to train a system to do that. Probably the easiest is to have a human labeler in world X talking about what would happen in some other world Y, where the labeling process is potentially mistaken. (As in "decoupled rl" approaches.) However, in this case it seems liable to learn the "instrumental policy" that asks "What does a human in possible world X think about what would happen in world Y?" which seems only slightly harder than the original.
We could talk about conditional independencies that we expect to remain robust on new distributions (e.g. in cases where humans are mistaken). I'll discuss this a bit in a reply.

Here's an abstract example to think about these proposals, just a special case of the example from this post.

Suppose that reality M is described as a causal graph X --> A --> B --> C, and then the observation Y is a function of (A, B, C).
The human's model M' of the situation is X --> A' --> B' --> C'. Each of them is a coarse-graining of the corresponding part of the real world model, and the observation Y is still a function of (A', B', C'), it's just more uncertain now.
The coarse-grained dynamics are simpler than the actual coarse-graining f: (A, B, C) --> (A', B', C').
We prepare a dataset by actually sampling (X, A, B, C, Y) from M, having humans look at it, make inferences about (A', B', C'), and get a dataset of (X, A', B', C', Y) tuples to train a model.
The intended question-answering function is to use M to sample (A, B, C, Y) then apply the coarse-graining f to get (A', B', C'). But there is also a bad function that produces good answers on the training dataset: use M to sample (A, B, C, Y), then use the human's model to infer (A', B', C'), and output those.
We'd like to rule out this bad function by making some kind of assumption about causal structure.

[-]paulfchristiano4y20

This is also a way to think about the proposals in this post and the reply:

The human believes that A' and B' are related in a certain way for simple+fundamental reasons.
On the training distribution, all of the functions we are considering reproduce the expected relationship. However, the reason that they reproduce the expected relationship is quite different.
For the intended function, you can verify this relationship by looking at the link (A --> B) and the coarse-graining applied to A and B, and verify that the probabilities work out. (That is, I can replace all of the rest of the computational graph with nonsense, or independent samples, and get the same relationship.)
For the bad function, you have to look at basically the whole graph. That is, it's not the case that the human's beliefs about A' and B' have the right relationship for arbitrary Ys, they only have the right relationship for a very particular distribution of Ys. So to see that A' and B' have the right relationship, we need to simulate the actual underlying dynamics where A --> B, since that creates the correlations in Y that actually lead to the expected correlations between A' and B'.
It seems like we believe not only that A' and B' are related in a certain way, but that the relationship should be for simple reasons, and so there's a real sense in which it's a bad sign if we need to do a ton of extra compute to verify that relationship. I still don't have a great handle on that kind of argument. I suspect it won't ultimately come down to "faster is better," though as a heuristic that seems to work surprisingly well. I think that this feels a bit more plausible to me as a story for why faster would be better (but only a bit).
It's not always going to be quite this cut and dried---depending on the structure of the human beliefs we may automatically get the desired relationship between A' and B'. But if that's the case then one of the other relationships will be a contingent fact about Y---we can't reproduce all of the expected relationships for arbitrary Y, since our model presumably makes some substantive predictions about Y and if those predictions are violated we will break some of our inferences.

[-]paulfchristiano4y20

So are there some facts about conditional independencies that would privilege the intended mapping? Here is one option.

We believe that A' and C' should be independent conditioned on B'. One problem is that this isn't even true, because B' is a coarse-graining and so there are in fact correlations between A' and C' that the human doesn't understand. That said, I think that the bad map introduces further conditional correlations, even assuming B=B'. For example, if you imagine Y preserving some facts about A' and C', and if the human is sometimes mistaken about B'=B, then we will introduce extra correlations between the human's beliefs about A' and C'.

I think it's pretty plausible that there are necessarily some "new" correlations in any case where the human's inference is imperfect, but I'd like to understand that better.

So I think the biggest problem is that none of the human's believed conditional independencies actually hold---they are both precise, and (more problematically) they may themselves only hold "on distribution" in some appropriate sense.

This problem seems pretty approachable though and so I'm excited to spend some time thinking about it.

[-]paulfchristiano4y20

Actually if A --> B --> C and I observe some function of (A, B, C) it's just not generally the case that my beliefs about A and C are conditionally independent given my beliefs about B (e.g. suppose I observe A+C). This just makes it even easier to avoid the bad function in this case, but means I want to be more careful about the definition of the case to ensure that it's actually difficult before concluding that this kid of conditional independence structure is potentially useful.

Moderation Log