Another excellent catch, kudos. I've really been sloppy with this shortform. I corrected it to say that we can approximate the system arbitrarily well by VNM decision-makers. Although, I think it's also possible to argue that a system that selects a non-exposed point is not quite maximally influential, because it's selecting somethings that's very close to delegating some decision power to chance.

Also, maybe this cannot happen when $D$ is the inverse limit of finite sets? (As is the case in sequential decision making with finite action/observation spaces). I'm not sure.

Example: Let $X=\{0,1\}$, and $D$ consist of the probability intervals ${\Theta }_0:=[0,\frac{2}{3}]$, ${\Theta }_1:=[\frac{1}{3},1]$ and ${\Theta }_2:=[\frac{1}{3},\frac{2}{3}]$. Then, it is (I think) consistent with the desideratum to have ${\Theta }^{\ast }={\Theta }_2$.

Not only that interpreting ${\Theta }^{\ast }={\Theta }_2$ requires an unusual decision rule (which I will be calling "utility hyperfunction"), but applying any ordinary utility function to this example yields a non-unique maximum. This is another point in favor of the significance of hyperfunctions.

You're absolutely right, good job! I fixed the OP.

TLDR: Systems which locally maximal influence can be described as VNM decision-makers.

There are at least 3 different motivations leading to the concept of "agent" in the context of AI alignment:

1. The sort of system we are concerned about (i.e. which poses risk)
2. The sort of system we want to build (in order to defend from dangerous systems)
3. The sort of systems that humans are (in order to meaningfully talk about "human preferences")

Motivation #1 naturally suggests a descriptive approach, motivation #2 naturally suggests a prescriptive approach, and motivation #3 is sort of a mix of both: on the one hand, we're describing something that already exists, on the other hand, the concept of "preferences" inherently comes from a normative perspective. There are also reasons to think these different motivation should converge on a single, coherent concept.

Here, we will focus on motivation #1.

A central reason why we are concerned about powerful unaligned agents, is that they are influential. Agents are the sort of system that, when instantiated in a particular environment is likely to heavily change this environment, potentially in ways inconsistent with the preferences of other agents.

Bayesian VNM

Consider a nice space^[1] $X$ of possible "outcomes", and a system that can choose^[2] out of a closed set of distributions $D\subseteq \Delta X$. I propose that an influential system should satisfy the following desideratum:

The system cannot select ${\mu }^{\ast }\in D$ which can be represented as a non-trivial lottery over other elements in $D$. In other words, ${\mu }^{\ast }$ has to be an extreme point of the convex hull of $D$.

Why? Because a system that selects a non-extreme point leaves something to chance. If the system can force outcome $\mu \in \Delta X$, or outcome $\nu \in \Delta X$ but chooses instead outcome $p\mu +(1-p)\nu$, for $p\in (0,1)$ and $\mu \ne \nu$, this means the system gave up on its ability to choose between $\mu$ and $\nu$ in favor of a $p$-biased coin. Such a system is not "locally^[3] maximally" influential^[4].

[EDIT: The original formulation was wrong, h/t @harfe for catching the error.]

The desideratum implies that there is a convergent sequence of utility functions $\{u_k:X\to R{\}}_{k\in N}$ s.t.

• For every $k\in N$, $E_{\mu }\left[u_k\right]$ has a unique maximum ${\mu }_k$ in $D$.
• The sequence ${\mu }_k$ converges to ${\mu }^{\ast }$.

In other words, such a system can be approximated by a VNM decision-maker within any precision. For finite $D$, we don't need the sequence, instead there is some $u:X\to R$ s.t. ${\mu }^{\ast }$ is the unique maximum of $E_{\mu }\left[u\right]$ over $D$. This observation is mathematically quite simple, but I haven't seen it made elsewhere (but I would not be surprised if it did appear in the decision theory literature somewhere).

Infra-Bayesian VNM?

Now, let's say that the system is choosing out of a set of credal sets (crisp infradistributions) $D\subseteq \square X$. I propose the following desideratum:

[EDIT: Corrected according to a suggestion by @harfe, original version was too weak.]

Let $\hat{D}$ be the closure of $D$ w.r.t. convex combinations and joins^[5]. Let ${\Theta }^{\ast }\in \square X$ be selected by the system. Then:

• For any $\Phi ,\Psi \in \hat{D}$ and $p\in (0,1)$, if ${\Theta }^{\ast }=p\Phi +(1-p)\Psi$ then $\Phi =\Psi$.
• For any $\Phi \in \hat{D}$, if $\Phi \subseteq {\Theta }^{\ast }$ then $\Phi ={\Theta }^{\ast }$.

The justification is, a locally maximal influential system should leave the outcome neither to chance nor to ambiguity (the two types of uncertainty we have with credal sets).

We would like to say that this implies that the system is choosing according to maximin relatively to a particular utility function. However, I don't think this is true, as the following example shows:

Example: Let $X=\{0,1\}$, and $D$ consist of the probability intervals ${\Theta }_0:=[0,\frac{2}{3}]$, ${\Theta }_1:=[\frac{1}{3},1]$ and ${\Theta }_2:=[\frac{1}{3},\frac{2}{3}]$. Then, it is (I think) consistent with the desideratum to have ${\Theta }^{\ast }={\Theta }_2$.

Instead, I have the following conjecture:

Conjecture: There exists some space $Y$, some $\xi \in \Delta Y$ and convergent sequence $\{u_k:Y\times X\to R{\}}_{k\in N}$ s.t.

As before, the maxima should be unique.

Such a "generalized utility function" can be represented as an ordinary utility function with a latent $Y$-valued variable, if we replace $D$ with $D^{\prime }\subseteq \square (Y\times X)$ defined by

However, using utility functions constructed in this way leads to issues with learnability, which probably means there are also issues with computational feasibility. Perhaps in some natural setting, there is a notion of "maximally influential under computational constraints" which implies an "ordinary" maximin decision rule.

This approach does rule out optimistic or "mesomistic" decision-rules. Optimistic decision makers tend to give up on influence, because they believe that "nature" would decide favorably for them. Influential agents cannot give up on influence, therefore they should be pessimistic.

Sequential Decision-Making

What would be the implications in a sequential setting? That is, suppose that we have a set of actions $A$, a set of observations $O$, $X:=(A\times O{)}^{\omega }$, a prior $\zeta :(A\times O{)}^{\ast }\times A\to \Delta O$ and

In this setting, the result is vacuous because of an infamous issue: any policy can be justified by a contrived utility functions that favors it. However, this is only because the formal desideratum doesn't capture the notion of "influence" sufficiently well. Indeed, a system whose influence boils down entirely to its own outputs is not truly influential. What motivation #1 asks of us, is talk about systems that influence the world-at-large, including relatively "faraway" locations.

One way to fix some of the problem is, take $X:=O^{\omega }$ and define $D$ accordingly. This singles out systems that have influence over their observations rather than only their actions, which is already non-vacuous (some policies are not such). However, such a system can still be myopic. We can take this further, and select "long-term" influence by projecting onto late observations or some statistics over observations. However, in order to talk about actually "far-reaching" influence, we probably need to switch to the infra-Bayesian physicalism setting. There, we can set $X:=2^{\Gamma }$, i.e. select for system that have influence over physically manifest computations.

1. ^{^}

   I won't keep track of topological technicalities here, probably everything here works at least for compact Polish spaces.

2. ^{^}

   Meaning that the system has some output, and different counterfactual outputs correspond to different elements of $D$.

3. ^{^}

   I say "locally" because it refers to something like a partial order, not a global scalar measure of influence.

4. ^{^}

   See also Yudkowsky's notion of efficient systems "not leaving free energy".

5. ^{^}

   That is, if $\Psi ,\Phi \in \hat{D}$ then their join (convex hull) $\Psi \vee \Phi$  is also in $\hat{D}$, and so is $p\Psi +(1-p)\Phi$ for every $p\in [0,1]$. Moreover, $\hat{D}$ is the minimal closed superset of $D$ with this property. Notice that this implies $\hat{D}$ is closed w.r.t. arbitrary infra-convex combinations, i.e. for any $Y$, $K:Y\to \hat{D}$ and $\Xi \in \square Y$, we have $K_{\ast }\Xi \in \hat{D}$.

Master post for selection/coherence theorems. Previous relevant shortforms: learnability constraints decision rules, AIT selection for learning.

Do you mean that seeing the opponent make dumb moves makes the AI infer that its own moves are also supposed to be dumb, or something else?

Relevant link

Apparently someone let LLMs play against the random policy and for most of them, most games end in a draw. Seems like o1-preview is the best of those tested, managing to win 47% of the time.

Relevant: Manifold market about LLM chess

This post states and speculates on an important question: are there different mind types that are in some sense "fully general" (the author calls it "unbounded") but are nevertheless qualitatively different. The author calls these hypothetical mind taxa "cognitive realms".

This is how I think about this question, from within the LTA:

To operationalize "minds" we should be thinking of learning algorithms. Learning algorithms can be classified according to their "syntax" and "semantics" (my own terminology). Here, semantics refers to questions such as (i) what type of object is the algorithm learning (ii) what is the feedback/data available to the algorithm and (iii) what is the success criterion/parameter of the algorithm. On the other hand, syntax refers to the prior and/or hypothesis class of the algorithm (where the hypothesis class might be parameterized in a particular way, with particular requirements on how the learning rate depends on the parameters).

Among different semantics, we are especially interested in those that are in some sense agentic. Examples include reinforcement learning, infra-Bayesian reinforcement learning, metacognitive agents and infra-Bayesian physicalist agents.

Do different agentic semantics correspond to different cognitive realms? Maybe, but maybe not: it is plausible that most of them are reflectively unstable. For example Christiano's malign prior might be a mechanism for how all agents converge to infra-Bayesian physicalism.

Agents with different syntaxes is another candidate for cognitive realms. Here, the question is whether there is an (efficiently learnable) syntax that is in some sense "universal": all other (efficiently learnable) syntaxes can be efficiently translated into it. This is a wide open question. (See also "frugal universal prior".)

In the context of AI alignment, in order to achieve superintelligence it is arguably sufficient to use a syntax equivalent to whatever is used by human brain algorithms. Moreover, it's plausible that any algorithm we can come up can only have an equivalent or weaker syntax (the process of us discovering the new syntax suggests an embedding of the new syntax into our own). Therefore, even if there are many cognitive realms, then for our purposes we mostly only care about one of them. However, the multiplicity of realms has implications on how simple/natural/canonical should we expect the choice of syntax for our theory of agents to be (the less realms, the more canonical).