(this post has been written for the second Refine blog post day. thanks to vanessa kosoy, adam shimi, sid black, artaxerxes, and paul bricman for their feedback.)

in this post, i try to give an overview of vanessa kosoy's new alignment protocol, Precursor Detection, Classification and Assistance or PreDCA, as she describes it in a recent youtube talk.

keep in mind that i'm not her and i could totally be misunderstanding her video or misfocusing on what the important parts are supposed to be.

the gist of it is: the goal of the AI should be to assist the user by picking policies which maximize the user's utility function. to that end, we characterize what makes an agent and its utility function, then detect agents which could potentially be the user by looking for precursors to the AI, and finally we select a subset of those which likely contains the user. all of this is enabled by infra-bayesian physicalism, which allows the AI to reason about what the world is like and what the results of computations are.

the rest of this post is largely a collection of mathematical formulas (or informal suggestions) defining those concepts and tying them together.

an important aspect of PreDCA is that the mathematical formalisms are theoretical ones which could be given to the AI as-is, not necessarily specifications as to what algorithms or data structures should exist inside the AI. ideally, the AI could just figure out what it needs to know about them, to what degree of certainty, and using what computations.

the various pieces of PreDCA are described below.

---

infra-bayesian physicalism, in which an agent has a hypothesis Θ ∈ □(Φ×Γ) (note that □ is actually a square, not a character that your computer doesn't have a glyph for) where:

• Φ is the set of hypotheses about how the physical world could be — for example, different hypotheses could entail different truthfulness for statements like "electrons are lighter than protons" or "norway has a larger population than china".
• Γ is the set of hypotheses about what the outputs of all programs are — for example, a given hypothesis could contain a statement such as "2+2 gives 4", "2+2 gives 5", "the billionth digit of π is 7", or "a search for proofs that either P=NP or P≠NP would find that P≠NP". note that, as the "2+2 gives 5" example demonstrates, these don't have to be correct hypotheses; in fact, PreDCA relies a lot on entertaining counterfactual hypotheses about the results of programs. a given hypothesis γ∈Γ would have type γ : program → output.
• Φ×Γ is the set of pairs of hypotheses — in each pair, one hypothesis about the physical world and one hypothesis about computations. note that a given hypothesis φ∈Φ or γ∈Γ is not a single statement about the world or computationspace, but rather entire descriptions of those. a given φ∈Φ would say everything there is to say about the world, and a given γ would specify the output of all possible programs. they are not to be stored inside the AI in their entirety of course; the AI would simply make increasingly informed guesses as to what correct hypotheses would entail, given how they are defined.
• □(Φ×Γ) assigns degrees of beliefs to those various hypotheses; in infra-bayesianism, those degrees are represented as "infra-distributions". i'm not clear on what those look like exactly, and a full explanation of infra-bayesianism is outside the scope of this post, but i gather that — as opposed to scalar bayesian probabilities — they're meant to encode not just the probability but also uncertainty about said probability.
• Θ is one such infra-bayesian distribution.

vanessa emphasizes that infra-bayesian physicalist hypotheses are described "from a bird's eye view" as opposed to being agent-centric, which helps with embedded agency: the AI has guesses as to what the whole world is like, which just happens to contain itself somewhere. in a given hypothesis, the AI is simply described as a part of the world, same as any other part.

next, a measure of agency is then defined: a "g-factor" g(G|U) for a given agent G and a given utility function (or loss function) U, which is defined as g(G|U) = -log(Pr π∈ξ [U(⌈G⌉,π) ≥ U(⌈G⌉,G*)]) where

• a policy is a function which takes some input — typically a history, i.e. a collection of pairs of past actions and observations — and returns a single action. an action itself could be advice given by an AI to humans, the motions of a robot arm, a human's actions in the world, what computations the agent chooses to run, etc.
• ξ is the set of policies which an agent could counterfactually hypothetically implement.
• G is an agent; it is composed of a program implementing a specific policy, along with its cartesian boundary. the policy which the agent G actually implements is written G*, and the cartesian boundary of the agent is written ⌈G⌉ — think of it as the outline separating the agent from the rest of the world, across which its inputs and outputs happen.
• U : cartesian-boundary × policy → value is a utility function, measuring how much utility the world would have if a given agent's cartesian boundary contained a program implementing a given policy. its return value is typically a simple scalar, but could really be any ordered quantity such as a tuple of scalars with lexicographic ordering.
• U(⌈G⌉,G*) is the utility produced by agent G if it would execute the actual policy G* which its program implements
• U(⌈G⌉,π) is the utility produced by agent G hypothetically executing some counterfactual policy π — if the cartesian boundary ⌈G⌉ contained a program implementing policy π instead of implementing the policy G*.
• Pr π∈ξ [U(⌈G⌉,π) ≥ U(⌈G⌉,G*)] is the probability that, for a random policy π∈ξ, that policy has worse utility than the policy G* its program dictates; in essence, how good G's policies are compared to random policy selection

so g(G|U) measures how good agent G is at satisfying a given utility function U.

given g(G|U), we can infer the probability that an agent G has a given utility function U, as Pr[U] ∝ 2^-K(U) / Pr π∈ξ [U(⌈G⌉,π) ≥ U(⌈G⌉,G*)]) where ∝ means "is proportional to" and K(U) is the kolmogorov complexity of utility function U.

so an agent G probably has utility function U if it's relatively good at satisfying that utility function and if that utility function is relatively simple — we penalize arbitrarily complex utility functions notably to avoid hypotheses such as "woah, this table is really good at being the exact table it is now (a complete description of the world would be an extremely complex utility function).

we also get the ability to detect what programs are agents — or more precisely, how agenty a given program is: g(G|U) - K(U) tells us how agenty a program G with utility function U is: its agentyness is its g-factor minus the complexity of its utility function.

"computationalism and counterfactuals": given a belief Θ ∈ □(Φ×Γ), the AI can test whether it thinks the world contains a given program by examining the following counterfactual: "if the result of that program was a different result than what it actually is, would the world look different?"

for example, we can consider the AKS prime number testing algorithm. let's say AKS(2^82589933-1) returns TRUE. we can ask "if it returned FALSE instead, would the universe — according to our computational hypothesis about it — look different?" if it would look different, then that means that someone or something in the world is running the program AKS(2^82589933-1).

to offer a higher-level example: if we were to know the true name of suffering, described as a program, then we can test whether the world contains suffering by asking a counterfactual: let's say that every time suffering happened, a goldfish appeared (somehow as an output of the suffering computation). if that were the case, would the world look different? if it would, then it contains suffering.

this ability to determine which programs are running in the world, coupled with the ability to measure how agenty a given program is, lets us find what agents exist in the world.

agentic causality: to determine whether an agent H's executed policy H* can causate onto another agent G, we can ask whether, if H had executed a different policy π≠H*, the agent G would receive different inputs. we can apparently get an information-theoritic measure of "how impactful" H* is onto agent G by determining how much mutual information there is between H* and G's input.

precursor detection: we say that an agent H is a precursor of agent G if, counterfactually, H could have prevented G from existing by executing a policy which is different from its actual policy H*.

we can now start to build a definition that lets the AI detect and then classify who its user is.

user detection: the AI is trying to determine who its precursor program could be. but, given a hypothesis for "the thing producing these policies is the precursor", there are infinitely many different programs which could output the observed policies. so we choose the one which is the most agenty, using the function described above: g(H|U) - K(U).

note that while we extrapolate the user's actions into the future, the user is defined as an instant-agent which precedes the AI's existence; such that the actual physical person's actual future actions does not change what utility function the AI should try to maximize. this stops the AI from influencing the user's utility function: we define the user strictly in the past, causally outside of the AI's light-cone. the AI is maximizing the utility function of the instant-user which causated its existence, not that of the continuously existing user-over-time.

user classification: for each potential precursor hypothesis, we have now selected a program that models them and their respective utility functions. we then eliminate some hypotheses as to what the user could be — notably to avoid acausal attacks by remote aliens or counterfactual demons — using the following criteria:

• the user is a precursor which, as of the AI's startup, should be in close causal proximity to it. for example, a human with their hands on the keyboard controlling the AI is more directly causally related than another human in the neighboring room.
• the g-factor of the user should be in a range where we would expect humans to lie. hopefully, this helps avoid selecting superintelligent acausal attackers, whose g-factor we'd expect to be much higher than that of humans.

finally, we end up with a hopefully small set of hypotheses as to who the user could be; at that point, we simply compose their utility functions, perhaps weighed by the infra-distribution of each of those hypotheses. this composition is the utility function that the AI should want to maximize, by selecting policies which maximize the utility that the world would have if they were enacted, to the best of the AI's ability to evaluate.

---

vanessa tells us how far along her protocol is, as a collection of pieces that have been completed to various degrees — green parts have gotten some progress, purple parts not as much. "informal PreDCA" is the perspective that she provides in her talk and which is hopefully conveyed by this post.

finally, some takeaways that can be taken from this informal PreDCA perspective:

• infra-bayesian physicalism is a powerful toolbox for formalizing agent relationships (in a way that reminds me of my bricks for insulated goal-program)
• this framework allows for "ambitious" alignment plans — ones that can actually transform the world in large ways that match our values (and notably might help prevent facebook AI from destroying everything six months later) as opposed to "weak safe AI"
• vanessa claims that her approach is the only one that she knows to provide some defenses against acausal attacks

my own opinion is that PreDCA is a very promising perspective. it offers, if not full "direct alignment", at least a bunch of pieces that might be of significant use to general AI risk mitigation.