Thanks for this!

What I was saying up there is not a justification of Hurwicz' decision rule. Rather, it is that if you already accept the Hurwicz rule, it can be reduced to maximin, and for a simplicity prior the reduction is "cheap" (produces another simplicity prior).

Why accept the Hurwicz' decision rule? Well, at least you can't be accused of a pessimism bias there. But if you truly want to dig deeper, we can start instead from an agent making decisions according to an ambidistribution, which is a fairly general (assumption-light) way of making decisions. I believe that a similar argument (easiest to see in the LF-dual cramble set representation) would allow reducing that to maximin on infradistributions (credal sets).

To make such an approach even more satisfactory, it would be good to add a justification for a simplicity ambi/infra-prior. I think this should be possible by arguing from "opinionated agents": the ordinary Solomonoff prior is the unique semicomputable one that dominates all semicomputable measures, which decision-theoretically corresponds to something like "having preferences about as many possible worlds as we can". Possibly, the latter principle formalized can be formalized into something which ends up picking out an infra-Solomonoff prior (and, replacing "computability" by a stronger condition, some other kind of simplicity infra-prior).

You now understand correctly. The reason I switch to colored operads is to add even more generality. My key use case is when the operad consists of terms-with-holes in a programming language, in which case the colors are the types of the terms/holes.

The following are my thoughts on the definition of learning in infra-Bayesian physicalism (IBP), which is also a candidate for the ultimate prescriptive agent desideratum.

In general, learning of hypotheses about the physical universe is not possible because of traps. On the other hand, learning of hypotheses about computable mathematics is possible in the limit of ample computing resources, as long as we can ignore side effects of computations. Moreover, learning computable mathematics implies approximating Bayesian planning w.r.t the prior about the physical universe. Hence, we focus on this sort of learning.

We consider an agent comprised of three modules, that we call Simulator, Learner and Controller. The agent's history consists of two phases. In the Training phase, the Learner interacts with the Simulator, and in the end produces a program for the Controller. In the Deployment phase, the Controller runs the program.

Roughly speaking:

• The Simulator is a universal computer whose function is performing computational experiments, which we can think of as "thought experiments" or coarse-grained simulations of potential plans. It receives commands from the Learner (which computations to run / threads to start/stop) and reports to the Learner the results. We denote the Simulator's input alphabet by $I_S$ and output alphabet by $O_S$.
• The Learner is the machine learning (training) module. The algorithm whose desideratum we're specifying resides here.
• The Controller (as in "control theory") is a universal computer connected to the agent's external interface (environmental actions $A$ and observations $O$). It's responsible for real-time interaction with the environment, and we can think of it as the learned policy. It is programmed by the Learner, for which purpose it has input alphabet $I_C$.

We will refer to this as the SiLC architecture.

Let $H\subseteq \square \Gamma$ be our hypothesis class about computable mathematics. Let $\xi :\Gamma \to \square 2^{\Gamma }$ be our prior about the physical universe^[1]. These have to satisfy the coherence conditions

Here, $\varphi ⪯\xi$ means that $\forall y\in \Gamma :\varphi \left(y\right)\in \xi \left(y\right)$.

Together, these ensure that $\Theta ⋉\xi$ is a coherent IBP hypothesis. Notice that for any ${\xi }_0:\Gamma \to \square 2^{\Gamma }$ satisfying the first condition^[2], there is a unique minimal coherent $\xi :\Gamma \to \square 2^{\Gamma }$ s.t. $\forall y\in \Gamma :{\xi }_0\left(y\right)\subseteq \xi \left(y\right)$. Moreover, given a coherent $\xi$ and any ${\Theta }_0\in \square \Gamma$, there is a unique minimal coherent $\Theta \in \square \Gamma$ s.t. ${\Theta }_0\subseteq \Theta$.

The duration of the Training phase will be denoted by $\tau \in N$^[3]. We can think of it as "computational time".

Let the source codes of the Learner (obtained by quining), the Simulator and the Controller respectively be denoted by

Here, the $N$ argument of $D_L$ corresponds to $\tau$ and ${\Delta }_Q$ is a probability distribution in which all probabilities are rational numbers^[4].

We assume that the simulator can indeed run any computation, and that any given halting computation would run fast for $\tau \gg 0$. These are assumptions on $D_S$ (or, on some combination of (i) $D_S$, (ii) the definition of $\Gamma$, and (iii) the support of all $\Theta \in H$) that we will not spell out here.

We will say that a policy is a mapping of type $(I_S\times I_C\times O_S{)}^{\ast }\to {\Delta }_Q(I_S\times I_C)$ and a metapolicy is a mapping of type $N\times (I_S\times I_C\times O_S{)}^{\ast }\to {\Delta }_Q(I_S\times I_C)$. 

Given any $D_L^{\prime }:\Gamma \times N\times (I_S\times I_C\times O_S{)}^{\ast }\to {\Delta }_Q(I_S\times I_C)$, we can compose it with $D_S$ and $D_C$ in the obvious way^[5] to yield

In particular, we can take $D_L^{\prime }=\mu$ for some metapolicy $\mu$ by postulating no dependence on the $\Gamma$ argument.

Denote by $P$ the set of all policies. Given metapolicy $\mu$ and $\pi \in P$, we define ${\mu }^{\pi }:[\tau +1]\times (I_S\times I_C\times O_S{)}^{\ast }\to {\Delta }_Q(I_S\times I_C)$ by

Given any $\nu :[\tau +1]\times (I_S\times I_C\times O_S{)}^{\ast }\to {\Delta }_Q(I_S\times I_C)$, we say that $y\in \Gamma$ is a ${\nu }^{\tau }$-consistent counterpossible when the following conditions hold:

• For all $k<\tau$ and $h\in (I_S\times I_C\times O_S{)}^{<k}$, $D_L(y,k,pq)=\nu (k,pq)$
• For all $k\le \tau$ and $h\in (A\times O{)}^{\ast }$, $(D_S\otimes D_L\otimes D_C)(y,k,pq)=(D_S\otimes \nu \otimes D_C)(y,k,pq)$

We denote by $C_{\nu }^{\tau }\subseteq \Gamma$ the set of ${\nu }^{\tau }$-consistent counterpossibles.

A (deterministic) copolicy is a mapping of signature $I_S^{+}\to O_S$. We denote the set of copolicies by $C$. Given a policy $\pi$ and a copolicy $\nu$, we define $\pi \hat{\otimes }\nu \in \Delta (I_S\times I_C\times O_S{)}^{\ast }$ in the obvious way. Given policies ${\pi }_1,{\pi }_2$, we define their total variation distance^[6] to be

Given $\Xi \in \square (\Gamma \times 2^{\Gamma })$, $f:\Gamma \times 2^{\Gamma }\to [0,\infty )$, $\tau \in N$ and metapolicy $\mu$, we will use the notation

Intuitively, $E_{\Xi }[f\mid \mid {\mu }^{\tau }]$ should be thought as the counterfactual expectation of loss function $f$ assuming metapolicy $\mu$, while adding a "bonus" to account for "fair" treatment of randomization by the agent. More on that below.

Given a metapolicy $\mu$ and $\tau \in N$, we define $E_{\mu }^{\tau }\subseteq 2^{\Gamma }$ by

Intuitively, $E_{\mu }^{\tau t}$ is the set of universe states for which at least one copy of the agent exists which followed the metapolicy $\mu$ until computational time $\tau$.

Given a loss function $l:N\times 2^{\Gamma }\to [0,1]$^[7] (which we allow to explicitly depend on computational time for greater generality), the learning condition on a metapolicy ${\mu }^{\ast }$ and hypothesis $\Theta \in H$ is

Here, $ϵ$ is the "regret bound" function which should vanish in the $N\to \infty$ limit.

Some remarks on the particulars of this definition:

• There are several reasons we impose $(D_S\otimes D_L\otimes D_C)(y,k,pq)=(D_S\otimes \nu \otimes D_C)(y,k,pq)$ rather than $D_L(y,k,pq)=\nu (k,pq)$:
• First, we want to ignore the side effects of running computations on the Simulator (both causal side effects and "mindcrime", i.e. the direct contribution of those computations to $l$). Because, taking side effects into account is usually inconsistent with the unlimited experimentation needed for learning.
• Second, learning requires trusting the reports of the Simulator, which means we should only impose the policy on copies of $D_L$ that are actually connected to $D_S$.
• Third, we should also be able to trust the Controller, because otherwise we lose the semantic grounding of the agent's external interface. (Even though this is not necessary for learning per se.).
• On the other hand, we impose $D_L(y,k,pq)=\nu (k,pq)$ in the computational past because that's valid additional information that doesn't interfere with the learning or decision theory.
• The learning criterion treats computational time myopically, so that we won't have to worry about traps in computational time.
• The reason we need randomization is, it's often necessary for efficient learning. In the simplest non-trivial examples, we learn by IID sampling a distribution over computations (e.g. we simulate the interaction between a particular policy and our physical prior $\xi$). If we sampled deterministically instead, Murphy would be able to fool us by changing behavior precisely at the sampled instances.
• The reason we need $E_{\Xi }[f\mid \mid {\mu }^{\tau t}]$ is, randomization only helps if low probability events can be ignored. However, if sufficiently many copies of the agents are instantiated, even a low probability even would be detectable. Hence, we use a "forgiving" metric that assigns low loss even to distributions that technically have high loss but are close to a different distribution with low loss.
• We can consider Newcombian problems where Omega makes decisions based on the agent's action probabilities. I suspect that if Omega's policy is Lipschitz in the agent policy, the behavior advised by the $E_{\Xi }[f\mid \mid {\mu }^{\tau t}]$ counterfactual will converge to FDT-optimal in the limit of sufficiently many iterations.
• Both in the case of ignoring side effects of computations and in the case of the treatment of randomization, we can be accused of departing from priorism ("updatelessness"). However, I think that this departure is justified. In the original TDT paper, Yudkowsky addressed the "Omega rewards irrationality" objection by postulating that, a decision problem is "fair" when it only depends on the agent's decisions rather than on how the agent makes those decisions. Here, we use the same principle: the agent should not be judged based on its internal thought process (side effects), and it should in some sense be judged based on its decisions rather than the probabilities assigned to those decisions.
• Also about priorism, this kind of agents will not endorse iterated-in-computational-time "logical" counterfactual mugging when the same coin is reused indefinitely, but will endorse it when a new coin is used every time, for an appropriate definition of "new" (or e.g. we switch to a new coin every $k$ rounds). Arguably, this solves the tension between priorism and learning observed by Demski. Formulating the precise criterion when Learning-IBP behavior converges to priorist / FDT-optimal is left for further research.
• The dependence of $ϵ(\Theta ,N)$ on $\Theta$ should ultimately involve some kind of description complexity. However, it will also involve something in the vein of "what are the computational resource bounds, according to the belief $\Theta$, for running certain computations, selected for their importance in testing $\Theta$". In particular, we won't require the agent to learning anything about non-halting computations. Indeed, any hypothesis about such computations will either assert a time bound on running the non-halting computations (in which case it is false) or will fail to assert any such bound, in which case its learning complexity is known to be infinite.
• We could make do without the $E_{{\xi }_{\ast }\Theta }\left[{\chi }_{E_{{\mu }^{\ast }}^{\tau }}\right]$ factor but that would make the learning criterion weaker. The presence of this factor means that, roughly speaking, regret should be low even conditional on the agent existing, which seems like a reasonable demand.
• Given an AI designed along these principles, we might worry about the impact of the side effects that are ignored. Maybe these can produce non-Cartesian daemons. However, during the Training phase, the algorithm has no access to external observation, which arguably makes it unlikely anything inside it can learn how to cyberattack. Moreover, during the Deployment phase, any reason for concern would be mediated by the particular algorithm the Controller runs (rather than the particulars of how it's implemented), which is what we do take into account in our optimization. Finally, the agent might be able to self-modify to make itself safer: we can even intentionally give it the means to do so (as part of its action space $A$). This probably requires careful prior-shaping to work well.

1. ^{^}

   This framework assumes all our hypotheses are disintegrable w.r.t. the factorization into $\Gamma$ and $2^{\Gamma }$. It is an interesting question to understand whether we should or can relax this assumption.

2. ^{^}

   For example, we can imagine ${\xi }_0$ to be a Solomonoff-like prior along the following lines. Every hypothesis comprising ${\xi }_0$ is defined by a Turing machine $M$ with access to two oracles representing $y,y^{\prime }\in \Gamma$ and two tapes of random and "ambiguous" bits respectively. ${\xi }_0\left(y\right)$ is defined by running $M$ with one oracle fielding queries about $y$ (i.e. we given a program $P$ we can request to know its counterpossible output $P^y$) and the other oracle fielding queries about some $y^{\prime }$ s.t. we want to decide whether $y^{\prime }\in \alpha$ for $\alpha \sim \theta \in {\xi }_0\left(y\right)$. $M$ is only allowed to return NO if there was at least one query to which the two oracles gave different answers.

3. ^{^}

   We use the "duration" interpretation for simplicity, but more generally $\tau$ can be some parameter controlling the computing resources available in the Training phase, and we can also allow the computing resources of the Controller to scale with $\tau$.

4. ^{^}

   The reason we restrict to rational numbers is because we need a notion of computing the distribution. It is in principle possible to generalize further to computable numbers. On the other hand, it might be more realistic to constrain even further to e.g. dyadic rationals (which can be implemented via fair coinflips). We stick to $Q$ for simplicity.

5. ^{^}

   We let the Learner interact with the Simulator for $\tau$ timesteps, producing some output $g\in I_C^{\tau }$, and then run the Controller with $g$ as an input.

6. ^{^}

   This is not technically a distance since it is possible to have $d_{\text{TV}}({\pi }_1,{\pi }_2)=0$ if ${\pi }_1\ne {\pi }_2$ so long as ${\pi }_1$ and ${\pi }_2$ only disagree on histories that are inconsistent with these policies. Such ${\pi }_1$ and ${\pi }_2$ are morally equivalent.

7. ^{^}

   We could also allow $l$ to have a $\Gamma$ argument, but then we would have to remove the $E_{{\xi }_{\ast }\Theta }\left[{\chi }_{E_{{\mu }^{\ast }}^{\tau }}\right]$ factor from the learning condition, because the choice of policy would matter intrinsically even if the agent doesn't exist. Alternatively, we could modify the definition of $C_{\mu }^{\tau }$ to avoid that. Or perhaps use some normalization factor more complicated than $E_{{\xi }_{\ast }\Theta }\left[{\chi }_{E_{{\mu }^{\ast }}^{\tau }}\right]$.

No? The elements of an operad have fixed arity. When defining a free operad you need to specify the arity of every generator.

Another excellent catch, kudos. I've really been sloppy with this shortform. I corrected it to say that we can approximate the system arbitrarily well by VNM decision-makers. Although, I think it's also possible to argue that a system that selects a non-exposed point is not quite maximally influential, because it's selecting somethings that's very close to delegating some decision power to chance.

Also, maybe this cannot happen when $D$ is the inverse limit of finite sets? (As is the case in sequential decision making with finite action/observation spaces). I'm not sure.

Example: Let $X=\{0,1\}$, and $D$ consist of the probability intervals ${\Theta }_0:=[0,\frac{2}{3}]$, ${\Theta }_1:=[\frac{1}{3},1]$ and ${\Theta }_2:=[\frac{1}{3},\frac{2}{3}]$. Then, it is (I think) consistent with the desideratum to have ${\Theta }^{\ast }={\Theta }_2$.

Not only that interpreting ${\Theta }^{\ast }={\Theta }_2$ requires an unusual decision rule (which I will be calling "utility hyperfunction"), but applying any ordinary utility function to this example yields a non-unique maximum. This is another point in favor of the significance of hyperfunctions.

You're absolutely right, good job! I fixed the OP.

TLDR: Systems which locally maximal influence can be described as VNM decision-makers.

There are at least 3 different motivations leading to the concept of "agent" in the context of AI alignment:

1. The sort of system we are concerned about (i.e. which poses risk)
2. The sort of system we want to build (in order to defend from dangerous systems)
3. The sort of systems that humans are (in order to meaningfully talk about "human preferences")

Motivation #1 naturally suggests a descriptive approach, motivation #2 naturally suggests a prescriptive approach, and motivation #3 is sort of a mix of both: on the one hand, we're describing something that already exists, on the other hand, the concept of "preferences" inherently comes from a normative perspective. There are also reasons to think these different motivation should converge on a single, coherent concept.

Here, we will focus on motivation #1.

A central reason why we are concerned about powerful unaligned agents, is that they are influential. Agents are the sort of system that, when instantiated in a particular environment is likely to heavily change this environment, potentially in ways inconsistent with the preferences of other agents.

Bayesian VNM

Consider a nice space^[1] $X$ of possible "outcomes", and a system that can choose^[2] out of a closed set of distributions $D\subseteq \Delta X$. I propose that an influential system should satisfy the following desideratum:

The system cannot select ${\mu }^{\ast }\in D$ which can be represented as a non-trivial lottery over other elements in $D$. In other words, ${\mu }^{\ast }$ has to be an extreme point of the convex hull of $D$.

Why? Because a system that selects a non-extreme point leaves something to chance. If the system can force outcome $\mu \in \Delta X$, or outcome $\nu \in \Delta X$ but chooses instead outcome $p\mu +(1-p)\nu$, for $p\in (0,1)$ and $\mu \ne \nu$, this means the system gave up on its ability to choose between $\mu$ and $\nu$ in favor of a $p$-biased coin. Such a system is not "locally^[3] maximally" influential^[4].

[EDIT: The original formulation was wrong, h/t @harfe for catching the error.]

The desideratum implies that there is a convergent sequence of utility functions $\{u_k:X\to R{\}}_{k\in N}$ s.t.

• For every $k\in N$, $E_{\mu }\left[u_k\right]$ has a unique maximum ${\mu }_k$ in $D$.
• The sequence ${\mu }_k$ converges to ${\mu }^{\ast }$.

In other words, such a system can be approximated by a VNM decision-maker within any precision. For finite $D$, we don't need the sequence, instead there is some $u:X\to R$ s.t. ${\mu }^{\ast }$ is the unique maximum of $E_{\mu }\left[u\right]$ over $D$. This observation is mathematically quite simple, but I haven't seen it made elsewhere (but I would not be surprised if it did appear in the decision theory literature somewhere).

Infra-Bayesian VNM?

Now, let's say that the system is choosing out of a set of credal sets (crisp infradistributions) $D\subseteq \square X$. I propose the following desideratum:

[EDIT: Corrected according to a suggestion by @harfe, original version was too weak.]

Let $\hat{D}$ be the closure of $D$ w.r.t. convex combinations and joins^[5]. Let ${\Theta }^{\ast }\in \square X$ be selected by the system. Then:

• For any $\Phi ,\Psi \in \hat{D}$ and $p\in (0,1)$, if ${\Theta }^{\ast }=p\Phi +(1-p)\Psi$ then $\Phi =\Psi$.
• For any $\Phi \in \hat{D}$, if $\Phi \subseteq {\Theta }^{\ast }$ then $\Phi ={\Theta }^{\ast }$.

The justification is, a locally maximal influential system should leave the outcome neither to chance nor to ambiguity (the two types of uncertainty we have with credal sets).

We would like to say that this implies that the system is choosing according to maximin relatively to a particular utility function. However, I don't think this is true, as the following example shows:

Example: Let $X=\{0,1\}$, and $D$ consist of the probability intervals ${\Theta }_0:=[0,\frac{2}{3}]$, ${\Theta }_1:=[\frac{1}{3},1]$ and ${\Theta }_2:=[\frac{1}{3},\frac{2}{3}]$. Then, it is (I think) consistent with the desideratum to have ${\Theta }^{\ast }={\Theta }_2$.

Instead, I have the following conjecture:

Conjecture: There exists some space $Y$, some $\xi \in \Delta Y$ and convergent sequence $\{u_k:Y\times X\to R{\}}_{k\in N}$ s.t.

As before, the maxima should be unique.

Such a "generalized utility function" can be represented as an ordinary utility function with a latent $Y$-valued variable, if we replace $D$ with $D^{\prime }\subseteq \square (Y\times X)$ defined by

However, using utility functions constructed in this way leads to issues with learnability, which probably means there are also issues with computational feasibility. Perhaps in some natural setting, there is a notion of "maximally influential under computational constraints" which implies an "ordinary" maximin decision rule.

This approach does rule out optimistic or "mesomistic" decision-rules. Optimistic decision makers tend to give up on influence, because they believe that "nature" would decide favorably for them. Influential agents cannot give up on influence, therefore they should be pessimistic.

Sequential Decision-Making

What would be the implications in a sequential setting? That is, suppose that we have a set of actions $A$, a set of observations $O$, $X:=(A\times O{)}^{\omega }$, a prior $\zeta :(A\times O{)}^{\ast }\times A\to \Delta O$ and

In this setting, the result is vacuous because of an infamous issue: any policy can be justified by a contrived utility functions that favors it. However, this is only because the formal desideratum doesn't capture the notion of "influence" sufficiently well. Indeed, a system whose influence boils down entirely to its own outputs is not truly influential. What motivation #1 asks of us, is talk about systems that influence the world-at-large, including relatively "faraway" locations.

One way to fix some of the problem is, take $X:=O^{\omega }$ and define $D$ accordingly. This singles out systems that have influence over their observations rather than only their actions, which is already non-vacuous (some policies are not such). However, such a system can still be myopic. We can take this further, and select "long-term" influence by projecting onto late observations or some statistics over observations. However, in order to talk about actually "far-reaching" influence, we probably need to switch to the infra-Bayesian physicalism setting. There, we can set $X:=2^{\Gamma }$, i.e. select for system that have influence over physically manifest computations.

1. ^{^}

   I won't keep track of topological technicalities here, probably everything here works at least for compact Polish spaces.

2. ^{^}

   Meaning that the system has some output, and different counterfactual outputs correspond to different elements of $D$.

3. ^{^}

   I say "locally" because it refers to something like a partial order, not a global scalar measure of influence.

4. ^{^}

   See also Yudkowsky's notion of efficient systems "not leaving free energy".

5. ^{^}

   That is, if $\Psi ,\Phi \in \hat{D}$ then their join (convex hull) $\Psi \vee \Phi$  is also in $\hat{D}$, and so is $p\Psi +(1-p)\Phi$ for every $p\in [0,1]$. Moreover, $\hat{D}$ is the minimal closed superset of $D$ with this property. Notice that this implies $\hat{D}$ is closed w.r.t. arbitrary infra-convex combinations, i.e. for any $Y$, $K:Y\to \hat{D}$ and $\Xi \in \square Y$, we have $K_{\ast }\Xi \in \hat{D}$.

Master post for selection/coherence theorems. Previous relevant shortforms: learnability constraints decision rules, AIT selection for learning.

Do you mean that seeing the opponent make dumb moves makes the AI infer that its own moves are also supposed to be dumb, or something else?