This work was inspired by a question by Vanessa Kosoy, who also contributed several of the core ideas, as well as feedback and mentorship.

Abstract

We outline a computationalist interpretation of quantum mechanics, using the framework of infra-Bayesian physicalism. Some epistemic and normative aspects of this interpretation are illuminated by a number of examples and theorems.

1. Introduction

Infra-Bayesian physicalism was introduced as a framework to investigate the relationship between a belief about a joint computational-physical universe and a corresponding belief about which computations are realized in the physical world, in the context of "infra-beliefs". Although the framework is still somewhat tentative and the definitions are not set in stone, it is interesting to explore applications in the case of quantum mechanics.

1.1. Discussion of the results

Quantum mechanics has been notoriously difficult to interpret in a fully satisfactory manner. Investigating the question through the lens of computationalism, and more specifically in the setting of infra-Bayesian physicalism provides a new perspective on some of the questions via its emphasis on formalizing aspects of metaphysics, as well as its focus on a decision-theoretic approach. Naturally, some questions remain, and some new interesting questions are raised by this framework itself.

The toy setup can be described on the high level as follows (with details given in Sections 2 to 4). We have an "agent": in this toy model simply consisting of a policy, and a memory tape to record observations. The agent interacts with a quantum mechanical "environment": performing actions and making observations. We assume the entire agent-environment system evolves unitarily. We'll consider the agent having complete Knightian uncertainty over its own policy, and for each policy the agent's beliefs about the "universe" (the joint agent-environment system) is given by the Born rule for each observable, without any assumption on the correlation between observables (formally given by the free product). We can then use the key construction in infra-Bayesian physicalism — the bridge transform — to answer questions about the agent's corresponding beliefs about what copies of the agent (having made different observations) are instantiated in the given universe.

In light of the falsity of Claims 4.15 and 4.17, we can think of the infra-Bayesian physicalist setup as a form of many-worlds interpretation. However, unlike the traditional many-worlds interpretation, we have a meaningful way of assigning probabilities to (sets of) Everett branches, and Theorem 4.19 shows statistical consistency with the Copenhagen interpretation. In contrast with the Copenhagen interpretation, there is no "collapse", but we do assume a form of the Born rule as a basic ingredient in our setup. Finally, in contrast with the de Broglie–Bohm interpretation, the infra-Bayesian physicalist setup does not privilege particular observables, and is expected to extend naturally to relativistic settings. See also Section 8 for further discussion on properties that are specific to the toy setting and ones that are more inherent to the framework. It is worth pointing out that the author is not an expert in quantum interpretations, so a lot of opportunities are left open for making connections with the existing literature on the topic.

1.2. Outline

In Section 2 we describe the formal setup of a quantum mechanical agent-environment system. In Section 3 we recall some of the central constructions in infra-Bayesian physicalism, then in Section 4 we apply this framework to the agent-environment system. In Sections 4.2 and 4.3 we write down various statements relating quantities arising in the infra-Bayesian physicalist framework to the Copenhagen interpretation of quantum mechanics. While Section 4.2 focuses on "epistemic" statements, Section 4.3 is dedicated to the "normative" aspects. A general theme in both sections is that the stronger, "on the nose" relationships between the interpretations fail, while certain weaker "asymptotic" relationships hold. In Section 5.1 we construct counterexamples to the stronger claims, and in in Sections 6 and 7 we prove the weaker claims relating the interpretations. In Section 8 we discuss which aspects of our setup are for the sake of simplicity in the toy model, and which are properties of the broader theory.

2. Setup

First, we'll describe a standard abstract setup for a simplified agent-environment joint system. We have the following ingredients:

• A finite set $A$ of possible actions of the agent.

• A finite set $O$ of possible observations of the agent. We'll write $E=O\times A$, the set of observation-action pairs.

• For technical reasons it will be convenient to add a symbol $0$ for "blank", and fix a bijection $E_{+}=E\bigsqcup \left\{0\right\}\cong Z/N$ preserving $0$, where $$N=|O|\cdot |A|+1.$$ We'll use this bijection to treat $E_{+}$ as an abelian group implicitly.

• A Hilbert space $H^e$ corresponding to states of the environment.

• Fix a finite time horizon^[1] $T\in N$. A classical state of a cyclic, length $T$ memory tape is a function $\tau :Z/T\to E_{+}$. Let ${\mathrm{Tp}}_T$ be the set of all classical tape states.

• A Hilbert space $H^g$ with orthonormal basis $|{\psi }_{\tau }^g⟩$ for $\tau \in \mathrm{Tp}$, corresponding to the quantum state of the agent.

• For each $a\in A$ a unitary map of the environment $U_a:H^e\to H^e$, describing the "result of the action".

• A projection-valued measure $P$ on $O$, valued in $H^e$ (giving projections $P_o:H^e\to H^e$ for each observation $o\in O$).

• Let $H=H^g\otimes H^e$ be the state space of the joint agent-environment system.

Remark 2.1. It would be interesting to consider a setting where the agent is allowed to choose the observation in each step (e.g. have the projection-valued measure $P$ depend on the action taken). For simplicity we'll work with a fixed observation as described above.

Definition 2.2. Let $$O^{\le T}=\underset{t\in N}{\overset{t\le T}{⨆}}{O}^t$$ $$E^{\le T}=\underset{t\in N}{\overset{t\le T}{⨆}}{E}^t$$ be the set of observation histories and observation-action histories respectively, i.e. finite strings of observations (resp. observation-action pairs) up to length $T$. There's a natural map $$\mathrm{obs}:E^{\le T}\to O^{\le T},$$ extracting the string of observations from a string of observation-action pairs. We'll call a function $\pi :O^{\le T}\to A$ a policy. For two histories $h_1,h_2$ (of either type), we'll sometimes write $h_1⊏h_2$ to mean $h_1$ is a (not necessarily proper) prefix (i.e. initial substring) of $h_2$.

Remark 2.3. We only consider deterministic policies here. It's not immediately clear how one would generalize Definition 2.7 to randomized policies. In fact, we can always (and is perhaps more principled to) think of our source of randomness for a randomized policy to be included in the environment, so we don't lose out on generality by only considering deterministic policies. For example, if the source of our randomness is a quantum coin flip, then our approach offers a convenient way of modeling this by including the coin as a factor of $H^e$, i.e. part of the environment subsystem.

Definition 2.4. For a tape state $\tau :Z/T\to E_{+}$ and an observation-action pair $\epsilon \in E$, let $\mathrm{mem}(\tau ,\epsilon ):Z/T\to E_{+}$ be the state of the tape after writing the pair $\epsilon$ to the tape, defined by $$\mathrm{mem}(\tau ,\epsilon )\left(n\right)=\{\begin{array}{cc}\tau (n-1)& n\ne 0\\ \tau (-1)+\epsilon & n=0.\end{array}$$

Remark 2.5. Choosing a group structure on $E_{+}$ is in order to make the map $$\mathrm{mem}(-,\epsilon ):\mathrm{Tp}\to \mathrm{Tp}$$ invertible, which in turn makes the map $U_{\pi }^o$ in Definition 2.7 unitary.

Definition 2.6. Let the "history extraction" map $\mathrm{hist}:\mathrm{Tp}\to E^{\le T}$ be defined by $$\mathrm{hist}\left(\tau \right)=\left(\tau \right(N-1),\dots ,\tau (0\left)\right)\in E^N,$$ where $0\le N\le T$ is largest such that there's no $0\le n<N$ with $\tau \left(n\right)=0$ (i.e. so that the $[0,N)$ portion of the tape contains no blanks).

Definition 2.7 (Time evolution of a policy). For each policy $\pi :O^{\le T}\to A$, we define the single time-step unitary evolution operator $U_{\pi }$ on $H$ as the composite of an "observation" and an "action" operator $U_{\pi }=U_{A,\pi }\circ U_{O,\pi }$, where $$\begin{array}{ccc}{U}_{O,\pi }(|{\psi }_{\tau }^g⟩\otimes P_o|{\psi }^e⟩)& =|{\psi }_{\mathrm{mem}(\tau ,(o,a\left)\right)}^g⟩\otimes P_o|{\psi }^e⟩& \text{for all}o\in O\\ U_{A,\pi }(|{\psi }_{\tau }^g⟩\otimes |{\psi }^e⟩)& =|{\psi }_{\tau }^g⟩\otimes U_a|{\psi }^e⟩& \text{for}a=\pi \left(\mathrm{obs}\right(\mathrm{hist}\left(\tau \right)\left)\right)\end{array}$$ The time evolution after $t\in N$ time-steps is given by $U_{\pi }^t=U_{\pi }\circ \dots \circ U_{\pi }$, i.e. $U_{\pi }$ composed with itself $t$ times.

Remark 2.8. As defined above, the first step in the evolution is an observation, so we never use the value of the policy on the empty observation string. In this respect it would be more natural to start with an action instead, but it would make some of the notation and the examples more cumbersome, so we sacrifice a bit of naturality for the sake of simplicity overall.

Lemma 2.9. The operator $U_{\pi }$ is unitary on $H$.

Proof. The operator $U_{A,\pi }$ is clearly unitary since each $U_a$ is. We can see that $U_{O,\pi }$ is unitary as follows. Choose an orthonormal basis $|{\psi }_{o,i}^e⟩$ of $P_o{H}^e$ for each $o\in O$, so together they form an orthonormal basis for $H^e$ (note that the range of $i$ might vary for varying $o$). Then $|{\psi }_{\tau }^g⟩\otimes |{\psi }_{o,i}^e⟩$ forms an orthonormal basis for $H$, and $U_{O,\pi }$ permutes this basis, hence is unitary.$$\begin{array}{}& \square \end{array}$$

3. Prerequisites

We recall some definitions and lemmas within infra-Bayesianism. This is in order to make the current article fairly self-contained, all the relevant notions here were introducted in [IBP], [BIMT] and [LBIMT]. In particular we omit proofs in this section, all the relevant proofs can be found in the articles listed.

3.1. Ultracontributions

First of all, we work with a notion of belief intended to incorporate a form of Knightian uncertainty. Formally, this means that we work with sets of distributions (or rather "contributions" turn out to be a more flexible tool).

Definition 3.1. Given a finite set $X$, a contribution $\mu$ is a non-negative measure on $X$, such that $\mu \left(X\right)\le 1$. We denote the set of contributions ${\Delta }^{c}X$. A contribution is a distribution if $\mu \left(X\right)=1$, so we have $\Delta X\subset {\Delta }^{c}X$.

There's a natural order on ${\Delta }^{c}X$, given by pointwise comparison.

Definition 3.2. We call a subset $A\subset {\Delta }^{c}X$ downward closed if for $\mu \in A$, $\nu \le \mu$ implies $\nu \in A$.

As a subspace of $R^X$, the set ${\Delta }^{c}X$ inherits a metric and a convex structure.

Definition 3.3. We call a closed, convex, downward closed subset $\Theta \subset {\Delta }^{c}X$ a homogenious ulta-contribution (HUC for short). We denote the set of HUCs by $\square X$.

We'll work with HUCs as our central formal notion of belief in this article. The exact properties required (closed, convex and downward closed) should be illuminated by Lemma 3.6.

Definition 3.4. Given a HUC $\Theta \in \square X$, and a function $f:X\to [0,1]$, we define the expected value $$E_{\Theta }\left[f\right]=\underset{\theta \in \Theta }{max}{E}_{\theta }\left[f\right]=\underset{\theta \in \Theta }{max}\sum _{x\in X}\theta \left(x\right)f\left(x\right).$$

Thinking of $f$ as a loss function, this is a worst-case expected value, given Knightian uncertainty over the probabilities.

Remark 3.5. It's worth mentioning that the prefix "infra" originates from the concept of infradistributions, which is the notion corresponding to ultracontributions, in the dual setup of utility functions instead of loss functions. We still often use the term "infra" in phrases such as infra-belief or infra-Bayesianism, but now simply carrying the connotation of a "weaker form" of belief etc., compared to the Bayesian analog.

Lemma 3.6. For $\Theta \in \square X$, the expected value defines a convex, monotone, homogeneous functional $E_{\Theta }[-]:[0,1{]}^X\to [0,1]$.

Lemma 3.7. There is a duality $\Theta \mapsto E_{\Theta }$, between $\square X$ (i.e. closed, convex, and downward closed subsets of ${\Delta }^{c}X$) and convex, monotone, and homogeneous functionals $[0,1{]}^X\to [0,1]$.

For a functional $F:[0,1{]}^X\to [0,1]$, the inverse map in the duality is given by $$F\mapsto {\Theta }_F=\bigcap _{f:X\to [0,1]}\{\theta \in {\Delta }^{c}X:E_{\theta }[f]\le F[f\left]\right\}.$$

3.2. Some constructions

For the current article to be more self-contained, we spell out a few definitions used in this discussion.

Definition 3.8. Given a map of finite sets $f:X\to Y$, we define the pushforward $f_{\ast }:{\Delta }^{c}X\to {\Delta }^{c}Y$ to be given by the pushforward measure. We use the same notation to denote the pushforward on HUCs, $f_{\ast }:\square X\to \square Y$, given by forward image, that is $$f_{\ast }\left(\Theta \right)=\left\{f_{\ast }\right(\theta )\in {\Delta }^{c}Y|\theta \in \Theta \}.$$ Equivalently, in terms of the expectation values we have for $g:Y\to [0,1]$ $$E_{f_{\ast }\left(\Theta \right)}\left[g\right]=E_{\Theta }[g\circ f].$$

Definition 3.9. Given a collection of finite sets $X_i$, and HUCs ${\Theta }_i\in \square X_i$, we define the free product $$\underset{i}{\bowtie }{\Theta }_i\in \square \left(\prod _i{X}_i\right)$$ as follows. For a contribution $\theta \in {\Delta }^c{\prod }_i{X}_i$ we have $\theta \in {\bowtie }_i{\Theta }_i$ if and only if for each $j$, $$\left({\mathrm{pr}}_j{)}_{\ast }\right(\theta )\in {\Theta }_j\subset {\Delta }^c{X}_j,$$ where ${\mathrm{pr}}_j:{\prod }_i{X}_i\to X_j$ is projection onto the $i$th factor.

The free product thus specifies the allowed marginal values, but puts no further restriction on the possible correlations.

Definition 3.10 (Total uncertainty). The state of total (Knightian) uncertainty ${\top }_X\in \square X$ is defined as $${\top }_X={\Delta }^{c}X,$$ i.e. the subset of all contributions.

Definition 3.11 (Semidirect product). Given a map $\beta :X\to \square Y$, and an element $\Theta \in \square X$, we can define the semidirect product $\Theta ⋉\beta \in \square (X\times Y)$. This is easier to write down in terms of the expectation functionals, as follows. For $g:X\times Y\to [0,1]$, define $$E_{\Theta ⋉\beta }\left[g\right]=E_{\Theta }\left[E_{\beta \left(x\right)}\right[g(x,-)\left]\right].$$ Here $E_{\beta \left(x\right)}\left[g\right(x,-\left)\right]$ is the function $X\to [0,1]$, whose value at $x\in X$ is given by by taking expected value with respect to $\beta \left(x\right)\in \square Y$ of the function $g(x,-):Y\to [0,1]$.

As a subset of ${\Delta }^c(X\times Y)$, ${\top }_X⋉\beta$ can be understood as the convex hull of the ${\delta }_x\times \theta$ for all $x\in X$ and all $\theta \in \beta \left(x\right)\subset {\Delta }^{c}Y$. For $\Theta ⋉\beta$ one needs to further restrict to contributions that project down into $\Theta \subset {\Delta }^{c}X$.

3.3. The bridge transform

The key construction we'll be considering in infra-Bayesian physicalism is the bridge transform. This construction is aimed at answering the question "given a belief about the joint computational-physical universe, what should our corresponding belief be about which computations are realized in the physical universe?".

We'll discuss these notions in a bit more detail, but for now both the physical universe $\Phi$ and the computational universe $\Gamma$ are just assumed to be finite sets.

Definition 3.12. Given $\Theta \in \square (\Gamma \times \Phi )$, the bridge transform of $\Theta$, $\mathrm{Br}\left(\Theta \right)\in \square (\Gamma \times 2^{\Gamma }\times \Phi )$ is defined as follows (cf. [IBP Definition 1.1]). For a contribution $\theta \in {\Delta }^c(\Gamma \times 2^{\Gamma }\times \Phi )$ we have $\theta \in \mathrm{Br}\left(\Theta \right)$ if and only if for any $s:\Gamma \to \Gamma$, under the composite we have $\tilde{s}\left(\theta \right)\in \Theta \subset {\Delta }^c(\Gamma \times \Phi )$.

Remark 3.13. The use of all endomorphism $s:\Gamma \to \Gamma$ in Definition 3.12, although concise, doesn't feel fully principled as of now. We would typically think of the computational universe $\Gamma$ as the set of all possible assignments of outputs to programs, i.e. $\Gamma ={\Sigma }^R$, for a certain output alphabet $\Sigma$, and a set of programs $R$ (see Definition 4.1). In this context, ${\Gamma }^{\Gamma }$ feels somewhat unnatural. That being said, in the current discussion we mainly use the fact that ${\Gamma }^{\Gamma }$ acts transivitely on $\Gamma$, so it's possible that these results would survive in some form under a modified definition of the bridge transform.

For easy reference, we spell out [IBP Proposition 2.10]:

Lemma 3.14 (Refinement). Given a mapping between physical universes $f:{\Phi }_1\to {\Phi }_2$, we have That is, for a belief $\Theta \in \square ({\Phi }_1\times \Gamma )$ we have $$({\mathrm{id}}_{{\mathrm{el}}^{\Gamma }}\times f{)}_{\ast }(\mathrm{Br}\left(\Theta \right))\subset \mathrm{Br}(({\mathrm{id}}_{\Gamma }\times f{)}_{\ast }(\Theta \left)\right).$$

4. An infra-Bayesian physicalist interpretation

We'll work with a certain specialized setup of [IBP].

Definition 4.1. Let the set of "programs" $R=O^{\le T}$, the "output alphabet" $\Sigma =A$, and the set of "computational universe states" $$\Gamma ={\Sigma }^R=A^{O^{\le T}}$$ be the set of policies up to time horizon $T$. We'll write $${\mathrm{el}}^{\Gamma }=\left\{\right(\pi ,\alpha )\in \Gamma \times 2^{\Gamma }:\pi \in \alpha \}.$$

Definition 4.2. Let a "universal observable" $B$ be a triple $$(V_B,Q_B,t_B),$$ where $V_B$ is a finite set (of "observation outcomes"), $Q_B$ is a projection-valued measure on $V_B$, valued in $H$ (giving projections $Q_B\left(v\right):H\to H$ for each $v\in V_B$), and an "observation time" $t_B\in N_{<T}$. Let $U$ be the set of all universal observables, up to the natural notion of equivalence.

Remark: We use the term "universal observable" here to distinguish between observables of the "universe" (i.e. the joint agent-environment system) from the observations of the environment by the agent.

Definition 4.3 (Initial state). Fix a normalized (norm $1$) initial state $|{\psi }_0^e⟩\in H^e$ of the environment, and let $|{\psi }_0^g⟩$ be the state of the agent corresponding to an empty memory tape, i.e. $\tau :Z/T\to E$ given by $\tau \left(n\right)=0$ for all $n$. Let $|{\psi }_0⟩=|{\psi }_0^g⟩\otimes |{\psi }_0^e⟩\in H$ be the initial state of the joint system.

Definition 4.4. For a policy $\pi \in \Gamma$, let the marginal distribution of the universal observable $B$ be defined according to the Born rule: $${\beta }_B\left(v|\pi \right)=\parallel Q_B\left(v\right)U_{\pi }^{t_B}|{\psi }_0⟩{\parallel }^2.$$ I.e. the norm square of the vector obtained by evolving the universe following policy $\pi$ for $t_B$ time-steps from the initial state, and then projecting onto the observation subspace corresponding to the universal observation $v\in V_B$. So $${\beta }_B(-|\pi )\in \Delta V_B.$$

Definition 4.5. Let $${\Phi }_U=\prod _{B\in U}{V}_B$$ be the set of "all possible states of the universe" (more precisely the set of all possible outcomes of all observations on the joint agent-environment system). More generally, define ${\Phi }_S$ analogously for any subset $S\subset U$.

Definition 4.6. For a finite subset $F\subset U$, let $${\beta }_F\left(\pi \right)=\underset{B\in F}{\bowtie }{\beta }_B(-|\pi )\in \square {\Phi }_F$$ be the free product of the ${\beta }_B$, as defined in Definition 3.9. For varying $\pi$ this defines an ultrakernel $${\beta }_F:\Gamma \to \square {\Phi }_F,$$ and the associated semidirect product $${\Theta }_F={\top }_{\Gamma }⋉{\beta }_F\in \square (\Gamma \times {\Phi }_F).$$ Taking the bridge transform and projecting out the physical factor ${\Phi }_F$: $$\square (\Gamma \times {\Phi }_F)\stackrel{\mathrm{Br}}{\to }\square (\Gamma \times 2^{\Gamma }\times {\Phi }_F)\stackrel{{\mathrm{pr}}_{\ast }}{\to }\square (\Gamma \times 2^{\Gamma }),$$ we get $${\Theta }_F^{\ast }={\mathrm{pr}}_{\ast }\left(\mathrm{Br}\right({\Theta }_F\left)\right)\in \square (\Gamma \times 2^{\Gamma }).$$

If $F_1\subset F_2\subset U$, we have a natural "refinement" map $p:{\Phi }_{F_2}\to {\Phi }_{F_1}$, given by projecting out the additional factors in ${\Phi }_{F_2}$. By Lemma 3.14, we have so ${\Theta }_{F_2}^{\ast }\subset {\Theta }_{F_1}^{\ast }$. Inspired by this, we have the following.

Definition 4.7. Let $${\Theta }_U^{\ast }=\bigcap _{F\subset U}{\Theta }_F^{\ast },$$ where the intersection is over all finite subsets of $U$.

4.1. Copenhagen interpretation

Definition 4.8. Let $h\in E^{\le T}$ be an observation-action history, and denote by $Q_h:H\to H$ the projection corresponding to the proposition "the memory tape recorded history $h$". More precisely $Q_h=Q_h^g\otimes {\mathrm{id}}_{H^e}$, where $$Q_h^g|{\psi }_{\tau }^g⟩=\{\begin{array}{cc}|{\psi }_{\tau }^g⟩& \text{if}\mathrm{hist}\left(\tau \right)=h\\ 0& \text{otherwise.}\end{array}$$

Definition 4.9. Given a sequence of observation-action pairs $h\in E^n$, let $h_{\le m}\in E^{\le m}$ denote the truncated history (i.e. the image under projecting out the last $n-m$ components of $E^n$ if $n>m$, and $h$ itself if $n\le m$).

In the Copenhagen interpretation the "universe" (i.e. the joint system of the agent and the environment) collapses after each observation of the agent.

Definition 4.10. Given a policy $\pi :O^{\le T}\to A$, the initial state $|{\psi }_0⟩\in H$, and a sequence of observation-action pairs $h\in E^n$, we can define $$|{\psi }_t⟩=Q_{h_{\le t}}{U}_{\pi }|{\psi }_{t-1}⟩$$ for $t>0$ recursively. Then according to the Copenhagen interpretation, the probability of observing $h$ is $$\mathrm{Cop}\left(h|\pi \right)=\parallel |{\psi }_n⟩{\parallel }^2.$$

Lemma 4.11. Collapsing at each step is the same as collapsing at the end, that is $$|{\psi }_t⟩=Q_{h_{\le t}}{U}_{\pi }^t|{\psi }_0⟩.$$

Proof. The claim is true for $t=0,1$ by definition. Assume it's true for $t-1$, so $$|{\psi }_{t-1}⟩=Q_{h_{\le t-1}}{U}_{\pi }^{t-1}|{\psi }_0⟩.$$ Let's write $$U_{\pi }^{t-1}|{\psi }_0⟩=\sum _{\tau \in \mathrm{Tp}}|{\psi }_{\tau }^g⟩\otimes |{\phi }_{\tau }^e⟩,$$ so $$|{\psi }_{t-1}⟩=\sum _{\begin{array}{c}\tau \in \mathrm{Tp}\\ \mathrm{hist}\left(\tau \right)=h_{\le t-1}\end{array}}|{\psi }_{\tau }^g⟩\otimes |{\phi }_{\tau }^e⟩.$$ Then if $a=\pi \left(\mathrm{obs}\right(h_{\le t-1}\left)\right)\in A$, we have $$|{\psi }_t⟩=\sum _{\begin{array}{c}\tau \in \mathrm{Tp}\\ \mathrm{hist}\left(\tau \right)=h_{\le t}\end{array}}|{\psi }_{\tau }^g⟩\otimes P_{h\left(t\right)}{U}_a|{\phi }_{\tau }^e⟩,$$ while $$U_{\pi }^t|{\psi }_0⟩=\sum _{\begin{array}{c}\tau \in \mathrm{Tp}\\ o\in O\end{array}}|{\psi }_{\mathrm{mem}(\tau ,(o,a\left)\right)}^g⟩\otimes P_o{U}_{\pi \left(\mathrm{hist}\right(\tau \left)\right)}|{\phi }_{\tau }^e⟩.$$ Now $$Q_{h_{\le t}}|{\psi }_{\mathrm{mem}(\tau ,(o,a\left)\right)}^g⟩=0$$ unless $(o,a)=h\left(t\right)$ and $\mathrm{hist}\left(\tau \right)=h_{\le t-1}$, hence $$Q_{h_{\le t}}{U}_{\pi }^t|{\psi }_0⟩=|{\psi }_t⟩$$ as claimed.$$\begin{array}{}& \square \end{array}$$

4.2. Relating the two interpretations

Since ${\Theta }_U^{\ast }\in \square {\mathrm{el}}^{\Gamma }$, we can take expectations of functions $f:{\mathrm{el}}^{\Gamma }\to [0,1]$, in particular indicator functions ${\chi }_q$ for $q\subset {\mathrm{el}}^{\Gamma }$.

Definition 4.12. For a policy $\pi \in \Gamma$, and a tuple of observations $h\in O^n$, define $${\alpha }_{h|\pi }=\{\gamma \in \Gamma |\gamma (h)=\pi (h\left)\right\}\subset \Gamma ,$$ and let $$q_{h|\pi }=\left\{\right(\pi ,\alpha )\in {\mathrm{el}}^{\Gamma }|\alpha \subset {\alpha }_{h|\pi }\}\subset {\mathrm{el}}^{\Gamma }.$$

Remark 4.13. In what follows we'll assume $|A|>1$. This assures that the set of policies is richer than the set of histories (i.e. $|\Gamma |>|O^{\le T}|)$. Much of the following fails in the degenerate case $|A|=1$.

When considering the infra-Bayesian physicalist interpretation of a quantum event $h$, we'll consider the expected value $$E_{{\Theta }_U^{\ast }}\left[{\chi }_{\pi }\right(1-{\chi }_{q_{h|\pi }}\left)\right].$$ As defined in Definition 4.6, ${\Theta }_U$ can be thought of as the infra-belief $${\top }_{\Gamma }⋉{\beta }_U\in \square (\Gamma \times {\Phi }_U),$$ which is a joint belief over the computational-physical world, with complete Knightian uncertainty over the policy of the agent (as a representation of "free will"), and for each policy the corresponding belief about the physical world is as given by the unitary quantum evolution of the agent-environment system under the given policy. The bridge transform $${\Theta }_U^{\ast }\in \square {\mathrm{el}}^{\Gamma }$$ of ${\Theta }_U$ then packages the relevant beliefs about which computational facts are manifest in the physical world. The subset ${\alpha }_{h|\pi }$ corresponds to the proposition "the policy outputs action $a=\pi \left(h\right)$ upon observing $h$", and hence $q_{h|\pi }$ corresponds to the belief "the physical world witnesses the output of the policy on $h$ to be $a=\pi \left(h\right)$ (which is to say there's a version of the agent instantiated in the physical world that observed history $h$, and acted $a$)". We'll be investigating various claims about the quantity $$E_{{\Theta }_U^{\ast }}\left[{\chi }_{\pi }\right(1-{\chi }_{q_{h|\pi }}\left)\right],$$ which is the ultraprobability (i.e. the highest probability for the given Knightian uncertainty) of the agent following policy $\pi$ and $h$ not being observed (i.e. no agent being instantiated acting on history $h$).

Remark 4.14. It might at first seem more natural to consider the complement instead, that is ${\chi }_{\pi }{\chi }_{q_{h|\pi }}$, which corresponds to the agent following policy $\pi$, and history $h$ being observed. However, it turns out that $$E_{{\Theta }_U^{\ast }}\left[{\chi }_{\pi }{\chi }_{q_{h|\pi }}\right]=1$$ always. This can be understood intuitively via refinement (see Lemma 3.14): we can always extend our model of the physical world to include a copy of the agent instantiated on history $h$, so the highest probability of $h$ being observed will be $1$. This is also related to the monotonicity principle discussed in [IBP]. Thus although at first glance this might seem less natural, in our setup it's more meaningful to study the ultraprobability of the complement, i.e. of $h$ not being observed. Note that since we're working with convex instead of linear expectation functionals (see Lemma 3.7), the complementary ultraprobabilities will typically sum to something greater than one.

We first state Claims 4.15 and 4.17 relating the IBP and Copenhagen interpretations "on the nose", which both turn out to be false in general. Then we state the weaker Theorem 4.19, which is true, and establishes a form of asymptotic relationship between the two interpretations.

Claim 4.15. The two interpretations agree on the probability that a certain history is not realized given a policy. That is, $$E_{{\Theta }_U^{\ast }}\left[{\chi }_{\pi }\right(1-{\chi }_{q_{h|\pi }}\left)\right]=1-\mathrm{Cop}\left(h|\pi \right).$$

This claim turns out to be false in general, and we give a counterexample in Counterexample 5.3. Note, however, that the claim seems to be true in the limit with many actions (i.e. $|A|\to \infty$), which would warrant further study. Now consider the following definition concerning two copies of the agent being instantiated.

Definition 4.18. For a policy $\pi \in \Gamma$, and two tuples of observations $h_1,h_2\in O^n$, define $${\alpha }_{h_1,h_2|\pi }=\{\gamma \in \Gamma |\gamma (h_i)=\pi (h_i)\text{for}i=1,2\}\subset \Gamma ,$$ and let $$q_{h_1,h_2|\pi }=\left\{\right(\pi ,\alpha )\in {\mathrm{el}}^{\Gamma }|\alpha \subset {\alpha }_{h_1,h_2|\pi }\}\subset {\mathrm{el}}^{\Gamma }.$$

Claim 4.17. There is only one copy of the agent (i.e. the agent is not instantiated on multiple histories, there are no "many worlds"). That is, if neither of $h_1,h_2\in O^n$ is a prefix of the other, then $$E_{{\Theta }_U^{\ast }}\left[{\chi }_{\pi }\right(1-{\chi }_{q_{h_1,h_2|\pi }}\left)\right]=1.$$

This claim is the relative counterpart of Claims 4.15 and fails as well in general (see Counterexample 5.5). Again, however, this claim might hold in the $|A|\to \infty$ limit.

Definition 4.18. An event is a subset of histories $E\subset O^T$. We define the corresponding $$q_{E|\pi }=\bigcup _{h\in E}{q}_{h|\pi }\subset {\mathrm{el}}^{\Gamma },$$ and $$\mathrm{Cop}\left(E|\pi \right)=\sum _{h\in E}\mathrm{Cop}\left(h|\pi \right).$$

Theorem 4.19. The ultraprobability of an agent not being instantiated on a certain event can be bounded via functions of the (Copenhagen) probability of the event. More precisely, $$1-\sqrt{(2-\mathrm{Cop}(E|\pi \left)\right)\mathrm{Cop}\left(E|\pi \right)}\le E_{{\Theta }_U^{\ast }}\left[{\chi }_{\pi }\right(1-{\chi }_{q_{E|\pi }}\left)\right]\le 1-\mathrm{Cop}\left(E|\pi \right).$$

Proof. We prove the upper bound in Section 6.1 and the lower bound in Section 6.2. $$\begin{array}{}& \square \end{array}$$

Due to the failure of Claims 4.15 and 4.17, we can think of the infra-Bayesian physicalist setup as a form of many-worlds interpretation. However, since $$\sqrt{(2-\mathrm{Cop}(E|\pi \left)\right)\mathrm{Cop}\left(E|\pi \right)}\to 0\text{as}\mathrm{Cop}\left(E|\pi \right)\to 0,$$ the above Theorem 4.19 shows statistical consistency with the Copenhagen interpretation in the sense that observations that are unlikely according to the Born rule have close to $1$ ultraprobability of not being instantiated (while very likely observations have close to $0$ ultraprobability of uninstantiation).

Remark 4.20. For simplicity we assumed $E$ only contains entire histories (i.e. ones of maximal length $T$). It's easy to modify the definitions to account for partial histories. The inequalities in Theorem 4.19 remain true even if $E$ includes partial histories, and the proofs are easy to adjust. We avoid doing this here in order to keep the notation cleaner. However, it's worth noting some important points here. For a partial history $h$, let $H\subset O^T$ be the set of all completions of $h$, i.e. $$H=\{\tilde{h}\in O^T:h⊏\tilde{h}\}.$$ Then we have $$\mathrm{Cop}\left(h|\pi \right)=\mathrm{Cop}\left(H|\pi \right)=\sum _{\tilde{h}\in H}\mathrm{Cop}\left(\tilde{h}|\pi \right).$$ On the other hand, $$q_{h|\pi }\ne q_{H|\pi }=\bigcup _{\tilde{h}\in H}{q}_{\tilde{h}|\pi },$$ so there is an important difference here between the two interpretations, which would warrant further discussion. In particular, under the infra-Bayesian physicalist interpretation it can happen that $$E_{{\Theta }_U^{\ast }}\left[{\chi }_{\pi }\right(1-{\chi }_{q_{H|\pi }}\left)\right]>E_{{\Theta }_U^{\ast }}\left[{\chi }_{\pi }\right(1-{\chi }_{q_{h|\pi }}\left)\right]$$ for a partial history $h$ and its set of completions $H$. This could be loosely interpreted as Everett branches "disappearing", as the ultraprobability of an agent not being instantiated on the partial history $h$ is less than that of the agent not being instantiated on any completion of that history.

4.3. Decision theory

To shed more light on the way the infra-Bayesian physicalist interpretation functions, it is interesting to consider the decision theory of the framework, along with the epistemic considerations above.

Definition 4.21. Consider a loss function $$L:D\to R_{\ge 0},$$ where $D=E^T$ is the set of destinies. We can then construct the physicalized loss function (cf. [IBP Definition 3.1]) $$L^{\text{phys}}:{\mathrm{el}}^{\Gamma }\to R_{\ge 0},$$ given by $$L^{\text{phys}}(\gamma ,\alpha )=\underset{h\in X_{\alpha }}{min}\underset{\begin{array}{c}d\in D\\ h⊏d\end{array}}{max}L\left(d\right),$$ where $X_{\alpha }$ is the set of histories witnessed by $\alpha$, that is $$X_{\alpha }=\{h\in E^{\le T}|\forall ga⊏h,\forall \tilde{\gamma }\in \alpha :\tilde{\gamma }(\mathrm{obs}\left(g\right))=a\}.$$ Note that in our simplified context, $L^{\text{phys}}(\gamma ,\alpha )$ doesn't depend on $\gamma$.

Definition 4.22. We can define the worst-case expected physicalized loss associated to a policy $\pi$ by $$L^{\text{IBP}}\left(\pi \right)=E_{{\Theta }_U^{\ast }}[{\chi }_{\pi }\cdot L^{\text{phys}}].$$ Under the Copenhagen model, we would instead simply consider $$L^{\mathrm{Cop}}\left(\pi \right)=E_{\mathrm{Cop}}\left[L|\pi \right]=\sum _{d\in D}\mathrm{Cop}\left(d|\pi \right)L\left(d\right).$$

Remark 4.23. Given a policy $\pi \in \Gamma$, we can consider the set of "fair" counterfactuals (cf. [IBP Definition 1.5]) $$\begin{array}{cc}{C}_{\text{fair}}^{\pi }=\{(\gamma ,\alpha )\in {\mathrm{el}}^{\Gamma }|& \forall h\in O^{\le T}:\\ & (\forall \tilde{\gamma }\in \alpha ,\forall \tilde{h}⊏h:\tilde{\gamma }(\tilde{h})=\gamma (\tilde{h}\left)\right)⟹\gamma \left(h\right)=\pi \left(h\right)\},\end{array}$$ i.e. where if $\alpha$ witnesses the history $h$, then $\gamma$ agrees with $\pi$ on that history. This definition is in contrast with the "naive" counterfactuals we considered above (when writing ${\chi }_{\pi }$): $$C_{\text{naive}}^{\pi }=\left\{\right(\gamma ,\alpha )\in {\mathrm{el}}^{\Gamma }|\gamma =\pi \}.$$ In Definition 4.22 above, and generally whenever we use ${\chi }_{\pi }$, we could have used the indicator function of $C_{\text{fair}}^{\pi }$ instead. The choice of counterfactuals affects the various expected values, however, all of the theorems in this article remain true (and Claims 4.15 and 4.17 remain false) for both naive and fair counterfactuals. We thus work with naive counterfactuals for the sake of simplicity.

Similarly to Section 4.2, the "on the nose" claim relating the two interpretations fails, but we have an asymptotic relationship which holds.

Claim 4.24. The two interpretations agree on the loss of any policy: $$L^{\text{IBP}}\left(\pi \right)=L^{\mathrm{Cop}}\left(\pi \right).$$

Again, this turns out to be false, and we give a simple counterexample in Counterexample 5.6.

To allow discussing the asymptotic behavior, assume now that we incur a loss at each timestep, given by $$\ell :E=O\times A\to R_{\ge 0},$$ and we consider the total loss $$L=\sum _{t=1}^T{\ell }_t:D\to R_{\ge 0}.$$ We might hope that we could have at least the following.

Claim 4.25. The two interpretations agree on the loss of any policy asymptotically: $$L^{\text{IBP}}\left(\pi \right)\sim L^{\mathrm{Cop}}\left(\pi \right),$$ i.e. the difference is bounded sublinearly in $T$.

This claim is still false in general for essentially the same reason as Claim 4.24 since certain policies might involve a one-off step that then affect the entire asymptotic loss. We give a detailed explanation in Counterexample 5.7. We do however have the following.

Theorem 4.26. If the resulting MDP is communicating (see Definition 7.8), then for any policy $\pi$ we have $$L^{\mathrm{Cop}}\left({\pi }^{\ast }\right)-o\left(T\right)\le L^{\text{IBP}}\left(\pi \right)\le L^{\mathrm{Cop}}\left(\pi \right),$$ where ${\pi }^{\ast }$ is a Copenhagen-optimal policy. In particular, optimal losses for the IBP and Copenhagen frameworks agree asymptotically.

Proof. See Theorem 7.1 for the upper bound and Theorem 7.21 for the lower bound. $$\begin{array}{}& \square \end{array}$$

5. Examples

We'll look at a few concrete examples in detail, firstly to gain some insight into how Claims 4.15 and 4.17 fail in general, and secondly to see how our framework operates in the famously puzzling Wigner's friend scenario.

5.1. Counterexamples

We'll construct simple counterexamples to Claims 4.15 and 4.17 in the smallest non-degenerate case, i.e. when $|O|=2$ and $|A|=2$, and $T=1$. Let $O=\{o_0,o_1\}$ and $A=\{a_0,a_1\}$. There are four policies in this case (ignoring the value of the policies on the empty input, which is irrelevant in our setting, see Remark 2.8), which we'll abbreviate as ${\pi }_{00},{\pi }_{01},{\pi }_{10},{\pi }_{11}$, where $$\begin{array}{cc}{\pi }_{ij}\left(o_0\right)& =a_i\\ {\pi }_{ij}\left(o_1\right)& =a_j.\end{array}$$ Assume $h=o_0$, and $\pi ={\pi }_{00}$, so ${\alpha }_{h|\pi }=\{{\pi }_{00},{\pi }_{01}\}$.

Recall [IBP Lemma 1]:

Lemma 5.1. For $\rho \in {\Delta }^c{\mathrm{el}}^{\Gamma }\times \Phi$, we have $\rho \in \mathrm{Br}\left(\Theta \right)$ if and only if for each $s:\Gamma \to \Gamma$ and $g:\Gamma \times \Phi \to [0,1]$ $$E_{\rho }\left[\tilde{g}\right]\le E_{\Theta }\left[g\right],$$ where $\tilde{g}:{\mathrm{el}}^{\Gamma }\times \Phi \to [0,1]$ is given by $\gamma ,\alpha ,x\mapsto {\chi }_{s\left(\gamma \right)\in \alpha }\cdot g\left(s\right(\gamma ),x)$.

Lemma 5.2. Let $\beta :\Gamma \to \Delta \Phi$ be a kernel, $\Theta ={\top }_{\Gamma }⋉\beta$, and ${\Theta }^{\ast }={\mathrm{pr}}_{\ast }\left(\mathrm{Br}\right(\Theta \left)\right)$ as above. Then $$E_{{\Theta }^{\ast }}\left[{\chi }_{\pi }\right(1-{\chi }_{q_{h|\pi }}\left)\right]=E_{\left(\beta \right({\pi }_{10})+\beta ({\pi }_{11}\left)\right)\wedge \beta \left({\pi }_{00}\right)}\left[1\right].$$

Proof. To obtain a lower bound (although we'll only use the upper bound for the counterexample), define the contribution $\rho \in {\Delta }^c({\mathrm{el}}^{\Gamma }\times \Phi )$ by $$\rho ={\delta }_{{\pi }_{00},\{{\pi }_{00},{\pi }_{10}\}}\times {\varphi }_{10}+{\delta }_{{\pi }_{00},\{{\pi }_{00},{\pi }_{11}\}}\times {\varphi }_{11},$$ where ${\varphi }_{10},{\varphi }_{11}\in {\Delta }^c\Phi$ are such that $${\varphi }_{10}\le \beta \left({\pi }_{10}\right),$$ $${\varphi }_{11}\le \beta \left({\pi }_{11}\right),$$ and $${\varphi }_{10}+{\varphi }_{11}=\left(\beta \right({\pi }_{10})+\beta ({\pi }_{11}\left)\right)\wedge \beta \left({\pi }_{00}\right).$$ One possible such choice is $${\varphi }_{10}=\beta \left({\pi }_{10}\right)\wedge \beta \left({\pi }_{00}\right)$$ $${\varphi }_{11}=\beta \left({\pi }_{11}\right)\wedge \left(\beta \right({\pi }_{00})-{\varphi }_{10}).$$ Then it's easy to verify that $\rho \in \mathrm{Br}\left(\Theta \right)$, and $$E_{\rho }\left[{\chi }_{\pi }\right(1-{\chi }_{q_{h|\pi }}\left)\right]=E_{\left(\beta \right({\pi }_{10})+\beta ({\pi }_{11}\left)\right)\wedge \beta \left({\pi }_{00}\right)}\left[1\right].$$ To obtain an upper bound, fix $x_0\in \Phi$, and use Lemma 5.1 for constant $s={\pi }_{00}$, and $g(\gamma ,x)={\chi }_{\gamma ={\pi }_{00}}\cdot {\chi }_{x=x_0}$. We have $$\tilde{g}(\gamma ,\alpha ,x)={\chi }_{{\pi }_{00}\in \alpha }\cdot g({\pi }_{00},x)={\chi }_{{\pi }_{00}\in \alpha }\cdot {\chi }_{x=x_0},$$ and so $$\begin{array}{}{E}_{\rho }[{\chi }_{{\pi }_{00}\in \alpha }\cdot {\chi }_{x=x_0}]=E_{\rho }\left[\tilde{g}\right]\le E_{\Theta }[{\chi }_{\gamma ={\pi }_{00}}\cdot {\chi }_{x=x_0}]=E_{\beta \left({\pi }_{00}\right)}\left[{\chi }_{x=x_0}\right].& \text{(1)}\end{array}$$ Analogously for ${\pi }_{10}$ and ${\pi }_{11}$ we get $$\begin{array}{}{E}_{\rho }[{\chi }_{{\pi }_{10}\in \alpha }\cdot {\chi }_{x=x_0}]\le E_{\Theta }[{\chi }_{\gamma ={\pi }_{10}}\cdot {\chi }_{x=x_0}]=E_{\beta \left({\pi }_{10}\right)}\left[{\chi }_{x=x_0}\right],& \text{(2)}\end{array}$$ and $$\begin{array}{}{E}_{\rho }[{\chi }_{{\pi }_{11}\in \alpha }\cdot {\chi }_{x=x_0}]\le E_{\Theta }[{\chi }_{\gamma ={\pi }_{11}}\cdot {\chi }_{x=x_0}]=E_{\beta \left({\pi }_{11}\right)}\left[{\chi }_{x=x_0}\right].& \text{(3)}\end{array}$$

Now, $${\chi }_{\pi }(1-{\chi }_{q_{h|\pi }}){\chi }_{x=x_0}\le {\chi }_{{\pi }_{00}\in \alpha }\cdot {\chi }_{x=x_0},$$ so by $\text{(1)}$ we get $$\begin{array}{}{E}_{\rho }\left[{\chi }_{\pi }\right(1-{\chi }_{q_{h|\pi }}\left){\chi }_{x=x_0}\right]\le E_{\beta \left({\pi }_{00}\right)}\left[{\chi }_{x=x_0}\right].& \text{(4)}\end{array}$$

We also have $1-{\chi }_{q_{h|\pi }}\le {\chi }_{{\pi }_{10}\in \alpha }+{\chi }_{{\pi }_{11}\in \alpha }$, since ${\pi }_{10}\notin \alpha$ and ${\pi }_{11}\notin \alpha$ together would imply $\alpha \subset {\alpha }_{h|{\pi }_{00}}$. Thus $${\chi }_{\pi }(1-{\chi }_{q_{h|\pi }}){\chi }_{x=x_0}\le ({\chi }_{{\pi }_{10}\in \alpha }+{\chi }_{{\pi }_{11}\in \alpha })\cdot {\chi }_{x=x_0},$$ so adding $\text{(2)}$ and $\text{(3)}$, we obtain $$\begin{array}{}{E}_{\rho }\left[{\chi }_{\pi }\right(1-{\chi }_{q_{h|\pi }}\left){\chi }_{x=x_0}\right]\le E_{\beta \left({\pi }_{10}\right)+\beta \left({\pi }_{11}\right)}\left[{\chi }_{x=x_0}\right].& \text{(5)}\end{array}$$ Now, since both $\text{(4)}$ and $\text{(5)}$ hold, we get $$E_{\rho }\left[{\chi }_{\pi }\right(1-{\chi }_{q_{h|\pi }}\left){\chi }_{x=x_0}\right]\le E_{\left(\beta \right({\pi }_{10})+\beta ({\pi }_{11}\left)\right)\wedge \beta \left({\pi }_{00}\right)}\left[{\chi }_{x=x_0}\right].$$ Finally, summing over $x_0\in \Phi$ we have the required upper bound $$E_{{\Theta }^{\ast }}\left[{\chi }_{\pi }\right(1-{\chi }_{q_{h|\pi }}\left)\right]=E_{\left(\beta \right({\pi }_{10})+\beta ({\pi }_{11}\left)\right)\wedge \beta \left({\pi }_{00}\right)}\left[1\right].$$ $$\begin{array}{}& \square \end{array}$$

Counterexample 5.3. Let $H^e$ be a qubit state space, and $$|{\psi }_0^e⟩=|+⟩=\frac{1}{\sqrt{2}}(|0⟩+|1⟩).$$ Let $U_{a_0}=U_{a_1}={\mathrm{id}}_{H^e}$. Let the observation $P$ correspond to measuring the qubit, so $P_{o_0},P_{o_1}$ are projections onto $|0⟩$ and $|1⟩$ respectively. Then Claim 4.15 fails in this setup.

Proof. We have $$|{\psi }_0⟩=|{\psi }_0^g⟩\otimes |{\psi }_0^e⟩=|0⟩\otimes \frac{1}{\sqrt{2}}(|0⟩+|1⟩),$$ and so $$U_{{\pi }_{00}}|{\psi }_0⟩=\frac{1}{\sqrt{2}}(|o_0{a}_0⟩\otimes |0⟩+|o_1{a}_0⟩\otimes |1⟩),$$ $$U_{{\pi }_{10}}|{\psi }_0⟩=\frac{1}{\sqrt{2}}(|o_0{a}_1⟩\otimes |0⟩+|o_1{a}_0⟩\otimes |1⟩),$$ $$U_{{\pi }_{11}}|{\psi }_0⟩=\frac{1}{\sqrt{2}}(|o_0{a}_1⟩\otimes |0⟩+|o_1{a}_1⟩\otimes |1⟩).$$ Now consider the universal observable $B$ which is measurement along the vector $|v⟩$ and its complement, where $$|v⟩=\frac{1}{2\sqrt{3}}(3|o_0{a}_0⟩\otimes |0⟩+|o_1{a}_0⟩\otimes |1⟩-|o_0{a}_1⟩\otimes |0⟩+|o_1{a}_1⟩\otimes |1⟩)$$ I.e. we have $V_B=\{v,v^{\perp }\}$, and $$Q_B\left(v\right)=P_v,$$ $$Q_B\left(v^{\perp }\right)=P_{v^{\perp }},$$ where $P_v$, $P_{v^{\perp }}$ are projections in $H=H^g\otimes H^e$ onto $|v⟩$ and its ortho-complement respectively. Then we have the following values for ${\beta }_B$ for the various policies:

	${\pi }_{00}$	${\pi }_{10}$	${\pi }_{11}$
${\beta }_B\left(v\right)$	2/3	0	0
${\beta }_B\left(v^{\perp }\right)$	1/3	1	1

This can be seen by noticing that $|v⟩$ is perpendicular to both $U_{{\pi }_{10}}|{\psi }_0⟩$ and $U_{{\pi }_{11}}|{\psi }_0⟩$, while $⟨v|U_{{\pi }_{00}}{\psi }_0⟩=\frac{2}{\sqrt{6}}$, so $${\beta }_B\left(v|{\pi }_{00}\right)=|⟨v|U_{{\pi }_{00}}{\psi }_0⟩{|}^2=\frac{2}{3}.$$ This means that for this $B$ we have $$\left({\beta }_B\right({\pi }_{10})+{\beta }_B({\pi }_{11}\left)\right)\wedge {\beta }_B\left({\pi }_{00}\right)\left[1\right]=1/3.$$ If $F_B=\left\{B\right\}$, by Lemma 5.2 we have $$E_{{\Theta }_{F_B}^{\ast }}\left[{\chi }_{\pi }\right(1-{\chi }_{q_{h|\pi }}\left)\right]=E_{\left({\beta }_B\right({\pi }_{10})+{\beta }_B({\pi }_{11}\left)\right)\wedge {\beta }_B\left({\pi }_{00}\right)}\left[1\right]=1/3.$$ Now, by definition ${\Theta }_U^{\ast }\subset {\Theta }_{F_B}^{\ast }$, so we also have $$E_{{\Theta }_U^{\ast }}\left[{\chi }_{\pi }\right(1-{\chi }_{q_{h|\pi }}\left)\right]\le 1/3<1-\mathrm{Cop}\left(h|\pi \right)=\frac{1}{2}.$$ $$\begin{array}{}& \square \end{array}$$

Although we won't need the exact value here, we remark to the interested reader that in the above setup of Counterexample 5.3, the ultraprobability attains the lower bound of Theorem 4.19, that is $$E_{{\Theta }_U^{\ast }}\left[{\chi }_{\pi }\right(1-{\chi }_{q_{h|\pi }}\left)\right]=1-\sqrt{3/4}\approx \mathrm{0.134.}$$

We can extend the above counterexample to apply to Claim 4.17, via the following.

Lemma 5.4. Let $\beta :\Gamma \to \Delta \Phi$ be a kernel, $\Theta ={\top }_{\Gamma }⋉\beta$, and ${\Theta }^{\ast }={\mathrm{pr}}_{\ast }\left(\mathrm{Br}\right(\Theta \left)\right)$ as above. Then for $h_1=o_0$, $h_2=o_1$, $$E_{{\Theta }^{\ast }}\left[{\chi }_{\pi }\right(1-{\chi }_{q_{h_1,h_2|\pi }}\left)\right]=E_{\left(\beta \right({\pi }_{10})+\beta ({\pi }_{01})+\beta ({\pi }_{11}\left)\right)\wedge \beta \left({\pi }_{00}\right)}\left[1\right].$$

Proof. Analogous to Lemma 5.2. $$\begin{array}{}& \square \end{array}$$

Counterexample 5.5. In the setup of Counterexample 5.3, Claim 4.17 fails too, that is $$E_{{\Theta }^{\ast }}\left[{\chi }_{\pi }\right(1-{\chi }_{q_{h_1,h_2|\pi }}\left)\right]<1.$$

Proof. Consider projecting onto the three vectors $$|v_1⟩=U_{{\pi }_{00}}|{\psi }_0⟩=\frac{1}{\sqrt{2}}(|o_0{a}_0⟩\otimes |0⟩+|o_1{a}_0⟩\otimes |1⟩),$$ $$|v_2⟩=U_{{\pi }_{11}}|{\psi }_0⟩=\frac{1}{\sqrt{2}}(|o_0{a}_1⟩\otimes |0⟩+|o_1{a}_1⟩\otimes |1⟩),$$ and $$\begin{array}{c}|v_3⟩=\frac{1}{\sqrt{2}}(U_{{\pi }_{01}}|{\psi }_0⟩-U_{{\pi }_{10}}|{\psi }_0⟩)=\\ \frac{1}{2}(|o_0{a}_0⟩\otimes |0⟩-|o_1{a}_0⟩\otimes |1⟩+|o_0{a}_1⟩\otimes |0⟩-|o_1{a}_1⟩\otimes |1⟩).\end{array}$$ Then the corresponding probabilities are

	${\pi }_{00}$	${\pi }_{01}$	${\pi }_{10}$	${\pi }_{11}$
${\beta }_B\left(v_1\right)$	1	1/4	1/4	0
${\beta }_B\left(v_2\right)$	0	1/2	1/2	0
${\beta }_B\left(v_3\right)$	0	1/4	1/4	1

So we have $$E_{\left({\beta }_B\right({\pi }_{10})+{\beta }_B({\pi }_{01})+{\beta }_B({\pi }_{11}\left)\right)\wedge {\beta }_B\left({\pi }_{00}\right)}\left[1\right]=1/2<1.$$ Then again, by refinement, this implies that $$E_{{\Theta }^{\ast }}\left[{\chi }_{\pi }\right(1-{\chi }_{q_{h_1,h_2|\pi }}\left)\right]\le 1/2<1$$ as well. $$\begin{array}{}& \square \end{array}$$

Counterexample 5.6. Claim 4.24 fails in the setup of Counterexample 5.3, with loss given by $$\ell :O\times A\to R_{\ge 0},$$ $$\ell (o,a)=\{\begin{array}{c}0\text{if}o=o_0\\ 1\text{if}o=o_1.\end{array}$$

Proof. In this case $$L^{\text{phys}}(\gamma ,\alpha )=\{\begin{array}{c}0\text{if}\forall \tilde{\gamma }\in \alpha :\tilde{\gamma }\left(o_0\right)=\gamma \left(o_0\right)\\ 1\text{otherwise.}\end{array}$$ Notice that for this $L^{\text{phys}}$ we actually have $${\chi }_{\pi }\cdot L^{\text{phys}}={\chi }_{\pi }(1-{\chi }_{q_{o_0|\pi }}),$$ so by the considerations in Counterexample 5.3, we also have (for any policy $\pi$) $$L^{\text{IBP}}\left(\pi \right)\le 1/3<1/2=L^{\mathrm{Cop}}\left(\pi \right),$$ showing failure of Claim 4.24. $$\begin{array}{}& \square \end{array}$$

Counterexample 5.7. The setup of Counterexample 5.6, run over time horizon $T$ instead of just a single timestep shows the failure of the asymptotic claim Claim 4.25 in general.

Proof. The point is that in this setup the entire loss is determined by the outcome of the first observation: if we observe $o_0$, we'll incur $0$ loss during the entire time, while if we observe $o_1$ first, we're "stuck" in that state, and hence incur a total loss equal to $T$. Due to this, we have $$L^{\text{IBP}}\left(\pi \right)\le \frac{T}{3}⋦\frac{T}{2}=L^{\mathrm{Cop}}\left(\pi \right),$$ for any policy $\pi$. $$\begin{array}{}& \square \end{array}$$

Note that in the above setup, we get "stuck" in the states after the initial observation because the MDP itself is not communicating. However, even for communicating MDPs (for example if we choose $U_{a_1}$ to be a rotation by $\pi /4$) certain policies will get stuck (for example the policy that always chooses $a_0$ corresponding to $U_{a_0}=\mathrm{id}$). So we see this behavior whenever the asymptotic loss is dependent on a few initial steps. On the other hand, for example if $\pi$ is a stationary policy, and the resulting Markov chain is irreducible, then we can obtain a concentration bound on the loss (e.g. via the central limit theorem, [Dur96^[2], 5.6.6.]), and use an argument similar to Theorem 7.21 to show that $L^{\text{IBP}}$ and $L^{\mathrm{Cop}}$ indeed agree asymptotically under such assumptions.

5.2. Wigner's friend

We'll consider a scenario originally attributed to Wigner, and we'll work in an extension of the setting introduced in [BB19^[3]]. For brevity, we'll omit detailed computations in this section and focus on the higher level ideas instead. Consider a joint system consisting of three parts, a spin-$\frac{1}{2}$ particle $S$, a friend $F$ in a lab, making observations of $S$, and Wigner $W$ making observations of the lab (the joint friend-particle system $FS$). Let the observation and action sets of the agents $F$ and $W$ be $$O_F=\{o_0^F,o_1^F\},$$ $$A_F=\{a_0^F,a_1^F\},$$ $$O_W=\{L_{00}^{+},L_{00}^{-},L_{11}^{+},L_{11}^{-}\},$$ $$A_W=\{a_0^W,a_1^W\},$$ respectively. Assume the state spaces for $F$ and $W$ are given by their individual memory tape states $H_F$ and $H_W$ as described in Section 2. Suppose the spin-$\frac{1}{2}$ particle is initially in the state $$|{\psi }_0^S⟩=|+⟩=\frac{1}{\sqrt{2}}(|0⟩+|1⟩)\in H_S.$$ The friend then measures $S$ in the $\{|0⟩,|1⟩\}$ basis, and performs an action $a_F\in A_F$ according to the policy $${\pi }^F\in {\Gamma }_F=A_F^{O_F}.$$ The lab $L=FS$ then evolves unitarily to the state $$|{\psi }_L⟩=\frac{1}{\sqrt{2}}({|o_0^F{\pi }^F\left(o_0^F\right)⟩}_F{|0⟩}_S+{|o_1^F{\pi }^F\left(o_1^F\right)⟩}_F{|1⟩}_S),$$ where $o_0^F,o_1^F\in O_F$ correspond to $F$ observing $0$ or $1$ respectively. Finally, suppose Wigner measures the lab $L=FS$ in the following basis: $$\begin{array}{cc}|L_{00}^{+}⟩& =\frac{1}{\sqrt{2}}({|o_0^F{a}_0^F⟩}_F{|0⟩}_S+{|o_1^F{a}_0^F⟩}_F{|1⟩}_S),\\ |L_{00}^{-}⟩& =\frac{1}{\sqrt{2}}({|o_0^F{a}_0^F⟩}_F{|0⟩}_S-{|o_1^F{a}_0^F⟩}_F{|1⟩}_S),\\ |L_{11}^{+}⟩& =\frac{1}{\sqrt{2}}({|o_0^F{a}_1^F⟩}_F{|0⟩}_S+{|o_1^F{a}_1^F⟩}_F{|1⟩}_S),\\ |L_{11}^{-}⟩& =\frac{1}{\sqrt{2}}({|o_0^F{a}_1^F⟩}_F{|0⟩}_S-{|o_1^F{a}_1^F⟩}_F{|1⟩}_S).\end{array}$$ So the two $L_{00}$ vectors correspond to states of the lab where the action of $F$ was $a_0^F$ (regardless of observation), and $L_{11}$ vectors correspond to states where the action was $a_1$. Technically these four vectors are not a basis of the full $H_L=H_F\otimes H_S$, since $\dim \left(H_L\right)=8$. Nevertheless, $|{\psi }_L⟩$ always falls within the four dimensional subspace spanned by these. If we wanted to be more precise, we could add further observation(s) to $O_W$, corresponding to the complement of this four dimensional subspace, but this wouldn't affect our discussion here, and would introduce additional notation.

Now let's assume $F$ follows the constant policy ${\pi }_{00}^F\left(o_i^F\right)=a_0^F$ (for $i=0,1$). Then Wigner will observe $L_{00}^{+}$ with probability $1$. Yet, if the friend $F$ believes that having observed $o_i^F$, the state of the lab collapsed to ${|o_i^F{a}_0^F⟩}_F{|i⟩}_S$, then the friend would expect Wigner to observe $L_{00}^{+}$ or $L_{00}^{-}$ with probability $\frac{1}{2}$ each. Thus, within collapse theories we have an apparent conflict between the predictions of Wigner and the friend.

We can model this scenario within IBP by taking $$\Gamma ={\Gamma }_F\times {\Gamma }_W=A_F^{O_F}\times A_W^{O_W}$$ be the pairs of policies of Wigner and the friend. Analogously to Definition 4.6 we can define ${\Phi }_U$ as the joint outcome of all observables on the joint triple system $WFS$, a kernel ${\beta }_U:\Gamma \to \square {\Phi }_U$, and the corresponding belief $${\Theta }_U={\top }_{\Gamma }⋉{\beta }_U\in \square (\Gamma \times {\Phi }_U),$$ and its projected bridge transform ${\Theta }_U^{\ast }\in \square {\mathrm{el}}^{\Gamma }$. To be more precise we would again build this out of finite subsets of $U$, as in Definition 4.7.

Given this setup, we can write down various definitions. For $h^W\in O_W$, let $${\alpha }_{h^W|{\pi }^W}=\left\{\right({\gamma }^F,{\gamma }^W)\in \Gamma |{\gamma }^W(h^W)={\pi }^W(h^W\left)\right\}\in 2^{\Gamma },$$ $$q_{h^W|{\pi }^W}=\{\alpha \in 2^{\Gamma }|\alpha \subset {\alpha }_{h^W|{\pi }^W}\}\subset 2^{\Gamma }.$$ Then we can compute $$E_{{\Theta }_U^{\ast }}\left[{\chi }_{{\pi }_{00}^F,{\pi }^W}\right(1-{\chi }_{q_{L_{00}^{+}|{\pi }^W}}\left)\right]=0,$$ i.e. the observation $L_{00}^{+}$ of Wigner is certain to be instantiated if $F$ follows the policy ${\pi }_{00}^F$.

We can also write down other quantities, for example for $h^F\in O_F$, $h^W\in O_W$, we can define $${\alpha }_{h^F,h^W|{\pi }^F,{\pi }^W}=\left\{\right({\gamma }^F,{\gamma }^W)\in \Gamma |{\gamma }^W(h^W)={\pi }^W(h^W),{\gamma }^F(h^F)={\pi }^F(h^F\left)\right\}\in 2^{\Gamma },$$ and the analogous $q_{h^F,h^W|{\pi }^F,{\pi }^W}$. The quantity $$E_{{\Theta }_U^{\ast }}\left[{\chi }_{{\pi }_{00}^F,{\pi }^W}\right(1-{\chi }_{q_{o_0,L_{00}^{+}|{\pi }_{00}^F,{\pi }^W}}\left)\right]$$ would then be the ultraprobability of the pair ($W$ observing $L_{00}^{+}$, $F$ observing $o_0$) being uninstantiated. We can estimate the value of this ultraprobability using techniques similar to Section 6 to be around $0.35$.

This setting is helpful to differentiate the decision theory of IBP from a collapse theory. For example, consider a loss function $\ell :O_W\to R_{\ge 0}$ that depends only on Wigner's observation, with values: $$\begin{array}{cc}\ell \left(L_{00}^{+}\right)& =0,\\ \ell \left(L_{00}^{-}\right)& =1,\\ \ell \left(L_{11}^{+}\right)& =0.1,\\ \ell \left(L_{11}^{-}\right)& =\mathrm{0.1.}\end{array}$$ Now suppose the friend is trying to minimize $\ell$.^[4] Then assuming a unitary evolution of the lab, clearly ${\pi }_{00}^F$ is the optimal policy. However, if the friend assumes a collapse of the lab after her observation, then always choosing action $a_1^F\in A_F$ avoids ever having an overlap with the high-loss $L_{00}^{-}$, making the constant $a_1^F$ policy ${\pi }_{11}^F\in {\Gamma }_F$ optimal under the collapse interpretation.

We can consider this decision problem within IBP by working with the physicalized loss function ${\ell }^{\text{phys}}:{\mathrm{el}}^{\Gamma }\to R_{\ge 0}$ (cf. Definition 4.21), given by ^[5] $${\ell }^{\text{phys}}(\gamma ,\alpha )=\{\begin{array}{c}0\text{if}\forall \tilde{\gamma }\in \alpha :{\tilde{\gamma }}^W\left(L_{00}^{+}\right)={\gamma }^W\left(L_{00}^{+}\right)\\ 0.1\text{else if}\forall \tilde{\gamma }\in \alpha :{\tilde{\gamma }}^W\left(L_{11}^{+}\right)={\gamma }^W\left(L_{11}^{+}\right)\\ 0.1\text{else if}\forall \tilde{\gamma }\in \alpha :{\tilde{\gamma }}^W\left(L_{11}^{-}\right)={\gamma }^W\left(L_{11}^{-}\right)\\ 1\text{otherwise.}\end{array}$$ Then in IBP the friend would look for the policy minimizing the loss $${\ell }^{\text{IBP}}\left({\pi }^F\right)=E_{{\Theta }_U^{\ast }}[{\chi }_{{\pi }^F}\cdot {\ell }^{\text{phys}}].$$ We can verify that the minimal loss ${\ell }^{\text{IBP}}\left({\pi }^F\right)=0$ occurs exactly when ${\pi }^F={\pi }_{00}^F$ as expected, in contrast with the collapse interpretation.

6. Bounds on the ultraprobabilities

We'll make use of the following observation.

Lemma 6.1. For a given history $h\in O^t$, if two policies ${\pi }_1,{\pi }_2\in \Gamma$ agree on all prefixes of $h$, i.e. $${\pi }_1\left(\tilde{h}\right)={\pi }_2\left(\tilde{h}\right)\text{for all}\tilde{h}⊏h,$$ then $$Q_h{U}_{{\pi }_1}^t|{\psi }_0⟩=Q_h{U}_{{\pi }_2}^t|{\psi }_0⟩,$$ where $Q_h$ is the projection corresponding to the observation $\mathrm{obs}\left(\tau \right)=h$, i.e. the memory tape having recorded $h$.

Proof. For $t=1$ we have $$U_{{\pi }_1}|{\psi }_0⟩=\sum _{o\in O}|o{\pi }_1\left(o\right)⟩\otimes U_{{\pi }_1\left(o\right)}{P}_o|{\psi }_0^e⟩,$$ and similarly for ${\pi }_2$. Now for $h=o_i$ we have $$Q_h{U}_{{\pi }_1}|{\psi }_0⟩=|o_i{\pi }_1\left(o_i\right)⟩\otimes U_{{\pi }_1\left(o_i\right)}{P}_{o_i}|{\psi }_0^e⟩=Q_h{U}_{{\pi }_2}|{\psi }_0⟩,$$ since ${\pi }_1\left(h\right)={\pi }_2\left(h\right)$ by assumption. For $t>1$ we can proceed by induction, using Lemma 4.11. $$\begin{array}{}& \square \end{array}$$

6.1. Upper bound

To prove an upper bound on the expectation value, we can coarsen our set of physical states to only include measurements of the memory tape.

Let $D=E^T\subset {\mathrm{Tp}}_T$ be the set of destinies.

Definition 6.2. Let $B_D$ be the universal observable corresponding to reading the destiny off the memory tape at time $t_{B_D}=T$. That is, $V_{B_D}=D$, and for $d\in D$ $$Q_{B_D}\left(d\right):H^g\to H^g$$ is given by $$|{\psi }_{\tau }^g⟩\otimes |{\psi }^e⟩\mapsto \{\begin{array}{cc}|{\psi }_{\tau }^g⟩\otimes |{\psi }^e⟩& \text{if}\tau =d\\ 0& \text{otherwise}.\end{array}$$

Definition 6.3. Let $Q\subset \Gamma \times D$ be the relation of a destiny being compatible with a policy. That is, $(\pi ,d)\in Q$ for $d=(o_1,a_1,\dots ,o_T,a_T)$ if and only if $$a_i=\pi (o_1,\dots ,o_i)$$ for each $1\le i\le T$.

Let $F_D=\left\{B_D\right\}$, and note that ${\Phi }_{F_D}=V_{B_D}=D$. Let ${\beta }_{F_D}:\Gamma \to \Delta D$ be the corresponding kernel.

Lemma 6.4. The kernel ${\beta }_{F_D}:\Gamma \to \Delta D$ is a PoCK for $Q$.

Proof. This is essentially saying that whenever ${\pi }_1,{\pi }_2\in \Gamma$ are both compatible with a destiny $d\in D$, then $${\beta }_{F_D}\left(d|{\pi }_1\right)={\beta }_{F_D}\left(d|{\pi }_2\right).$$ This claim follows by Lemma 6.1. $$\begin{array}{}& \square \end{array}$$

Then by [IBP Proposition 4.1] we have

Lemma 6.5. The bridge transform equals $$\mathrm{Br}\left({\Theta }_{F_D}\right)=[{\top }_{\Gamma }⋉(Q^{-1}⋊{\beta }_{F_D}){]}^{\downarrow }.$$ In particular, for a monotone increasing (in $2^{\Gamma }$) function $f:\Gamma \times 2^{\Gamma }\to [0,1]$, we have $$\begin{array}{}\begin{array}{c}{E}_{{\Theta }_{F_D}^{\ast }}\left[f\right]=\underset{\gamma \in \Gamma }{max}{E}_{{\beta }_{F_D}\left(\gamma \right)}\left[f\right(\gamma ,-)\circ Q^{-1}].\end{array}& \text{(1)}\end{array}$$

For $g={\chi }_{q_{E|\pi }}$ (note that $g$ is monotone decreasing) and $d\in D$ we have $g(\gamma ,-)\circ Q^{-1}\left(d\right)=1$ if

1. $\gamma =\pi$,

2. $\pi \in Q^{-1}\left(d\right)$, and

3. $Q^{-1}\left(d\right)\subset {\alpha }_{h|\pi }$ for some $h\in E$.

Lemma 6.6. We have $Q^{-1}\left(d\right)\subset {\alpha }_{h|\pi }$ if and only if $h=\mathrm{obs}\left(d\right)$ and $a_T=\pi \left(h\right)$ (where $d=(o_1,a_1,\dots ,o_T,a_T)$ as before).

Proof. If $h=\mathrm{obs}\left(d\right)$ and $a_T=\pi \left(h\right)$, then $(\gamma ,d)\in Q$ implies $\gamma \left(h\right)=a_T=\pi \left(h\right)$, so $Q^{-1}\left(d\right)\subset {\alpha }_{h|\pi }$.

For the converse, assume $Q^{-1}\left(d\right)\subset {\alpha }_{h|\pi }$. First choose $\gamma \in Q^{-1}\left(d\right)$ (which is always non-empty). In particular $\gamma \left(h\right)=a_T$, so $\gamma \in {\alpha }_{h|\pi }$ means $\pi \left(h\right)=a_T$ as well.

Now assume $h\ne \mathrm{obs}\left(d\right)$. Then choose ${\gamma }^{\prime }\in \Gamma$ as follows. For $\tilde{h}\in O^{\le T}$, let $${\gamma }^{\prime }\left(\tilde{h}\right)=\{\begin{array}{cc}{a}_i& \text{if}\tilde{h}=(o_1,\dots ,o_i)\\ {\tilde{a}}_T\ne a_T& \text{if}\tilde{h}=h\\ a_1\text{(arbitrary)}& \text{otherwise.}\end{array}$$ Then ${\gamma }^{\prime }\in Q^{-1}\left(d\right)\setminus {\alpha }_{h|\pi }$, contradiction. $$\begin{array}{}& \square \end{array}$$

We therefore have $$E_{{\beta }_{F_D}\left(\gamma \right)}\left[{\chi }_{q_{E|\pi }}\right(\gamma ,-)\circ Q^{-1}]=\{\begin{array}{cc}\sum _{h\in E}\sum _{\begin{array}{c}d\in Q\left(\pi \right)\\ h=\mathrm{obs}\left(d\right)\end{array}}{\beta }_{F_D}\left(\pi \right)\left(d\right)& \text{if}\gamma =\pi \\ 0& \text{otherwise}.\end{array}$$ Here $$\sum _{h\in E}\sum _{\begin{array}{c}d\in Q\left(\pi \right)\\ h=\mathrm{obs}\left(d\right)\end{array}}{\beta }_{F_D}\left(\pi \right)\left(d\right)=\mathrm{Cop}\left(E|\pi \right),$$ so by applying $\text{(1)}$ of Lemma 6.5 to the monotone increasing $f={\chi }_{\pi }(1-{\chi }_{q_{E|\pi }})$, we have $$E_{{\Theta }_{F_D}^{\ast }}\left[{\chi }_{\pi }\right(1-{\chi }_{q_{E|\pi }}\left)\right]=1-\mathrm{Cop}\left(E|\pi \right),$$ since ${\chi }_{\pi }(\gamma ,\alpha )=0$ whenever $\gamma \ne \pi$ so the ${max}_{\gamma \in \Gamma }$ is attained when $\gamma =\pi$. Since ${\Theta }_U^{\ast }\subset {\Theta }_{F_D}^{\ast }$ by definition, we have Proposition 6.7. $$E_{{\Theta }_U^{\ast }}\left[{\chi }_{\pi }\right(1-{\chi }_{q_{E|\pi }}\left)\right]\le 1-\mathrm{Cop}\left(E|\pi \right).$$

6.2. Lower bound

Definition 6.8. For ease of notation we'll write $$C_{E|\pi }:=1-\sqrt{(2-\mathrm{Cop}(E|\pi \left)\right)\mathrm{Cop}\left(E|\pi \right)}.$$

Theorem 6.9. We have a lower bound $$E_{{\Theta }_U^{\ast }}\left[{\chi }_{\pi }\right(1-{\chi }_{q_{E|\pi }}\left)\right]\ge C_{E|\pi }.$$

Proof. We'll exhibit a contribution ${\rho }_{E|\pi }\in {\Delta }^c{\mathrm{el}}^{\Gamma }$ (Definition 6.12) such that ${\rho }_{E|\pi }\in {\Theta }_U^{\ast }$ (Proposition 6.18). The constructed ${\rho }_{E|\pi }$ has (Lemma 6.13) $$E_{{\rho }_{E|\pi }}\left[{\chi }_{\pi }\right(1-{\chi }_{q_{E|\pi }}\left)\right]=C_{E|\pi },$$ which in turn will show that $$\begin{array}{c}{E}_{{\Theta }_U^{\ast }}\left[{\chi }_{\pi }\right(1-{\chi }_{q_{E|\pi }}\left)\right]=\underset{\rho \in {\Theta }_U^{\ast }}{max}{E}_{\rho }\left[{\chi }_{\pi }\right(1-{\chi }_{q_{E|\pi }}\left)\right]\ge \\ E_{{\rho }_{E|\pi }}\left[{\chi }_{\pi }\right(1-{\chi }_{q_{E|\pi }}\left)\right]=C_{E|\pi }.\end{array}$$ $$\begin{array}{}& \square \end{array}$$

The rest of this section is dedicated to spelling out the results that are used in the proof outline above.

Lemma 6.10. Let $|a⟩,|b⟩,|c⟩$ be a set of three orthonormal vectors, and $$|\varphi ⟩=\alpha |a⟩+\beta |b⟩,$$ $$|\psi ⟩=\alpha |a⟩+\beta |c⟩,$$ where $\alpha ,\beta \in C$ with $|\alpha {|}^2+|\beta {|}^2=1$. Then the trace distance between the density matrices $\rho =|\varphi ⟩⟨\varphi |$ and $\sigma =|\psi ⟩⟨\psi |$ is $$d_{tr}(\rho ,\sigma )=\frac{1}{2}\parallel \rho -\sigma {\parallel }_1=\sqrt{1-|\alpha {|}^4}.$$

Proof. In the basis given by $|a⟩,|b⟩,|c⟩$, the matrix of $\rho -\sigma$ is $$\left(\begin{array}{ccc}|\alpha {|}^2& \alpha \overline{\beta }& 0\\ \overline{\alpha }\beta & |\beta {|}^2& 0\\ 0& 0& 0\end{array}\right)-\left(\begin{array}{ccc}|\alpha {|}^2& 0& \alpha \overline{\beta }\\ 0& 0& 0\\ \overline{\alpha }\beta & 0& |\beta {|}^2\end{array}\right)=\left(\begin{array}{ccc}0& \alpha \overline{\beta }& -\alpha \overline{\beta }\\ \overline{\alpha }\beta & |\beta {|}^2& 0\\ -\overline{\alpha }\beta & 0& -|\beta {|}^2.\end{array}\right)$$ The eigenvalues of this (rank 2) matrix are $0$, and $\pm \sqrt{1-|\alpha {|}^4}$, so the sum of the absolute values of the eigenvalues is $$\parallel \rho -\sigma {\parallel }_1=0+\sqrt{1-|\alpha {|}^4}+\sqrt{1-|\alpha {|}^4}=2\sqrt{1-|\alpha {|}^4}.$$ $$\begin{array}{}& \square \end{array}$$

Lemma 6.11. For two policies ${\pi }_1,{\pi }_2$, let $${\Pi }_{12}=\{d\in O^T:\forall h⊏d,{\pi }_1(h)={\pi }_2(h\left)\right\},$$ $$E_{12}=O^T\setminus {\Pi }_{12}.$$ Then for any $B\in U$, $$E_{{\beta }_B\left({\pi }_1\right)\wedge {\beta }_B\left({\pi }_2\right)}\left[1\right]\ge C_{E_{12}|{\pi }_1}.$$

Proof. Roughly speaking, since ${\pi }_1$ and ${\pi }_2$ only differ outside of ${\Pi }_{12}$, if the event ${\Pi }_{12}$ was observed then ${\pi }_1$ and ${\pi }_2$ behave identically. More precisely, let $h_B\in O^{t_B}$ be a sequence of observations up to time $t_B$. Then, if $h_B⊏d$ for some $d\in {\Pi }_{12}$, by Lemma 6.1 we have $$\begin{array}{}\begin{array}{c}{Q}_{h_B}{U}_{{\pi }_1}^{t_B}|{\psi }_0⟩=Q_{h_B}{U}_{{\pi }_2}^{t_B}|{\psi }_0⟩.\end{array}& \text{(1)}\end{array}$$ Without loss of generality we'll assume $t_B=T$ from now on. Now, $$U_{{\pi }_1}^T|{\psi }_0⟩=\sum _{h\in O^T}{Q}_h{U}_{{\pi }_1}^T|{\psi }_0⟩=\sum _{h\in {\Pi }_{12}}{Q}_h{U}_{{\pi }_1}^T|{\psi }_0⟩+\sum _{h\in O^T\setminus {\Pi }_{12}}{Q}_h{U}_{{\pi }_1}^T|{\psi }_0⟩,$$ and similarly for ${\pi }_2$. Write $$|{\pi }_1\cap {\pi }_2⟩:=\sum _{h\in {\Pi }_{12}}{Q}_h{U}_{{\pi }_1}^T|{\psi }_0⟩=\sum _{h\in {\Pi }_{12}}{Q}_h{U}_{{\pi }_2}^T|{\psi }_0⟩,$$ where the two sums are equal by $\text{(1)}$. Also write $$|{\pi }_1\setminus {\pi }_2⟩:=\sum _{h\in O^T\setminus {\Pi }_{12}}{Q}_h{U}_{{\pi }_1}^T|{\psi }_0⟩,$$ and $$|{\pi }_2\setminus {\pi }_1⟩:=\sum _{h\in O^T\setminus {\Pi }_{12}}{Q}_h{U}_{{\pi }_2}^T|{\psi }_0⟩,$$ so that $$U_{{\pi }_1}^T|{\psi }_0⟩=|{\pi }_1\cap {\pi }_2⟩+|{\pi }_1\setminus {\pi }_2⟩,$$ and $$U_{{\pi }_2}^T|{\psi }_0⟩=|{\pi }_1\cap {\pi }_2⟩+|{\pi }_2\setminus {\pi }_1⟩.$$ The three vectors $|{\pi }_1\cap {\pi }_2⟩$, $|{\pi }_1\setminus {\pi }_2⟩$, $|{\pi }_2\setminus {\pi }_1⟩$ are orthogonal (since all of their $H^g$ components are), and $$\parallel |{\pi }_1\cap {\pi }_2⟩{\parallel }^2=\sum _{h\in {\Pi }_{12}}\parallel Q_h{U}_{{\pi }_1}^T|{\psi }_0⟩{\parallel }^2=\sum _{h\in {\Pi }_{12}}\mathrm{Cop}\left(h|{\pi }_1\right)=\mathrm{Cop}\left({\Pi }_{12}|{\pi }_1\right).$$ From this, using Lemma 6.10, we can compute the trace distance between $${\rho }_{{\pi }_1}=|U_{{\pi }_1}^T{\psi }_0⟩⟨U_{{\pi }_1}^T{\psi }_0|$$ and $${\rho }_{{\pi }_2}=|U_{{\pi }_2}^T{\psi }_0⟩⟨U_{{\pi }_2}^T{\psi }_0|$$ to be $$d_{tr}({\rho }_{{\pi }_1},{\rho }_{{\pi }_2})=\frac{1}{2}\parallel {\rho }_{{\pi }_1}-{\rho }_{{\pi }_2}{\parallel }_1=\sqrt{1-\parallel |{\pi }_1\cap {\pi }_2⟩{\parallel }^4}=\sqrt{1-\mathrm{Cop}({\Pi }_{12}|{\pi }_1{)}^2}.$$

Now, for any measurement $Q_B$, if we write $${\beta }_B\left(v|{\pi }_1\right)=\parallel Q_B\left(v\right)U_{{\pi }_1}^T|{\psi }_0⟩{\parallel }^2$$ for the distribution of outcomes, where $v\in V_B$. Then the total variation distance between the distributions ${\beta }_B(-|{\pi }_1)$ and ${\beta }_B(-|{\pi }_2)$ is bounded above by the trace distance. That is, $$d_{TV}\left({\beta }_B\right(-|{\pi }_1),{\beta }_B(-|{\pi }_2\left)\right)=\frac{1}{2}\sum _{v\in V_B}|{\beta }_B\left(v|{\pi }_1\right)-{\beta }_B\left(v|{\pi }_2\right)|\le d_{tr}({\rho }_{{\pi }_1},{\rho }_{{\pi }_2}).$$ So the overlap is bounded below as claimed $$\begin{array}{c}{E}_{{\beta }_B\left({\pi }_1\right)\wedge {\beta }_B\left({\pi }_2\right)}\left[1\right]=1-d_{TV}\left({\beta }_B\right(-|{\pi }_1),{\beta }_B(-|{\pi }_2\left)\right)\ge \\ 1-d_{tr}({\rho }_{{\pi }_1},{\rho }_{{\pi }_2})=1-\sqrt{1-\mathrm{Cop}({\Pi }_{12}|{\pi }_1{)}^2}=C_{E_{12}|{\pi }_1}.\end{array}$$ $$\begin{array}{}& \square \end{array}$$

Definition 6.12. Given a policy $\pi$ and an event $E\subset O^T$, choose ${\tilde{\pi }}_E\in \Gamma$ such that $$\begin{array}{cc}& {\tilde{\pi }}_E\left(h\right)\ne \pi \left(h\right)\text{whenever}\forall d\in D:h⊏d⟹d\in E\\ & {\tilde{\pi }}_E\left(h\right)=\pi \left(h\right)\text{otherwise.}\end{array}$$ Note that we are using $|A|>1$ here to allow the choice satisfying the first condition. That is, ${\tilde{\pi }}_E$ agrees with $\pi$ on all histories except for the ones whose completions all lie in $E$. Let $${\rho }_{E|\pi }=C_{E|\pi }\cdot {\delta }_{\pi ,\{\pi ,{\tilde{\pi }}_E\}}\in {\Delta }^c{\mathrm{el}}^{\Gamma }.$$

Lemma 6.13. We have $$E_{{\rho }_{E|\pi }}\left[{\chi }_{\pi }\right(1-{\chi }_{q_{E|\pi }}\left)\right]=C_{E|\pi }.$$

Proof. The claim follows since $\{\pi ,{\tilde{\pi }}_E\}\subset ̸{\alpha }_{h|\pi }$ for any $h\in E$. $$\begin{array}{}& \square \end{array}$$

Definition 6.14. Let $${\varphi }_B=\frac{C_{E|\pi }}{E_{{\beta }_B\left(\pi \right)\wedge {\beta }_B\left({\tilde{\pi }}_E\right)}\left[1\right]}{\beta }_B\left(\pi \right)\wedge {\beta }_B\left({\tilde{\pi }}_E\right).$$

Lemma 6.15. The contribution ${\varphi }_B$ has mass $C_{E|\pi }$, i.e. $E_{{\varphi }_B}\left[1\right]=C_{E|\pi }$. Moreover, $${\varphi }_B\le {\beta }_B\left(\pi \right)\wedge {\beta }_B\left({\tilde{\pi }}_E\right).$$

Proof. The mass follows from the definition. The inequality in the second claim follows by taking ${\pi }_1=\pi$ and ${\pi }_2={\tilde{\pi }}_E$ in Lemma 6.11, and noticing that the $E_{12}$ in the lemma equals $E$ in this case. $$\begin{array}{}& \square \end{array}$$

Lemma 6.16. For finite $F\subset U$, let $${\varphi }_F=\frac{1}{C_{E|\pi }^{|F|-1}}\prod _{B\in F}{\varphi }_B\in {\Delta }^c{\Phi }_F.$$ Then ${\varphi }_F\in {\beta }_F\left(\pi \right)\cap {\beta }_F\left({\tilde{\pi }}_E\right)\subset {\Delta }^c{\Phi }_F$.

Proof. For any $f\in F$, consider the projection $${\mathrm{pr}}_f:{\Phi }_F=\prod _{B\in F}{V}_B\to V_f.$$ Then under $({\mathrm{pr}}_f{)}_{\ast }:{\Delta }^c{\Phi }_F\to {\Delta }^c{V}_f$ we have $$\left({\mathrm{pr}}_f{)}_{\ast }\right({\varphi }_F)=\frac{1}{C_{E|\pi }^{|F|-1}}{\varphi }_f\prod _{B\in F\setminus \left\{f\right\}}{E}_{{\varphi }_B}[1]=\frac{C_{E|\pi }^{|F|-1}}{C_{E|\pi }^{|F|-1}}{\varphi }_f={\varphi }_f\le {\beta }_f(\pi )\wedge {\beta }_f({\tilde{\pi }}_E),$$ where the last inequality follows from Lemma 6.15. Since this is true for all $f$, we have $${\varphi }_F\in \underset{B\in F}{\bowtie }{\beta }_B\left(\pi \right)={\beta }_F\left(\pi \right)\in \square {\Phi }_F,$$ and also ${\varphi }_F\in {\beta }_F\left({\tilde{\pi }}_E\right)$, hence ${\varphi }_F\in {\beta }_F\left(\pi \right)\cap {\beta }_F\left({\tilde{\pi }}_E\right)$. $$\begin{array}{}& \square \end{array}$$

Proposition 6.17. For each finite $F\subset U$, we have $${\delta }_{\pi ,\{\pi ,{\tilde{\pi }}_E\}}\times {\varphi }_F\in \mathrm{Br}\left({\Theta }_F\right)\subset {\Delta }^c({\mathrm{el}}^{\Gamma }\times {\Phi }_F),$$ and hence $${\rho }_{E|\pi }=C_{E|\pi }\cdot {\delta }_{\pi ,\{\pi ,{\tilde{\pi }}_E\}}\in {\mathrm{pr}}_{\ast }\left(\mathrm{Br}\right({\Theta }_F\left)\right)={\Theta }_F^{\ast }.$$

Proof. Let $s:\Gamma \to \Gamma$ be an endomorphism of the computational universe. We need to verify that for any such $s$, under the composite we have $\tilde{s}({\delta }_{\pi ,\{\pi ,{\tilde{\pi }}_E\}}\times {\varphi }_F)\in {\Theta }_F$. Since ${\rho }_{E|\pi }=C_{E|\pi }\cdot {\delta }_{\pi ,\{\pi ,{\tilde{\pi }}_E\}}$, we have ${\chi }_{{\mathrm{el}}^{\Gamma }}\ne 0$ only when $s\left(\pi \right)=\pi$ or $s\left(\pi \right)={\tilde{\pi }}_E$. For these cases, we have

• If $s\left(\pi \right)=\pi$, then $$\tilde{s}({\delta }_{\pi ,\{\pi ,{\tilde{\pi }}_E\}}\times {\varphi }_F)={\delta }_{\pi }\times {\varphi }_F\in {\Theta }_F,$$ since ${\varphi }_F\in {\beta }_F\left(\pi \right)$ by Lemma 6.16.

• If $s\left(\pi \right)={\tilde{\pi }}_E$, then $$\tilde{s}({\delta }_{\pi ,\{\pi ,{\tilde{\pi }}_E\}}\times {\varphi }_F)={\delta }_{{\tilde{\pi }}_E}\times {\varphi }_F\in {\Theta }_F,$$ since ${\varphi }_F\in {\beta }_F\left({\tilde{\pi }}_E\right)$ as well by Lemma 6.16.  $$\begin{array}{}& \square \end{array}$$

Proposition 6.18. We have $${\rho }_{E|\pi }=C_{E|\pi }\cdot {\delta }_{\pi ,\{\pi ,{\tilde{\pi }}_E\}}\in {\Theta }_U^{\ast }.$$

Proof. This follows immediately from Proposition 6.17 and the Definition 4.7 of ${\Theta }_U^{\ast }$. $$\begin{array}{}& \square \end{array}$$

7. Asymptotic convergence

7.1. Upper bound

Theorem 7.1. For any policy $\pi$, $$L^{\text{IBP}}\left(\pi \right)\le L^{\mathrm{Cop}}\left(\pi \right),$$ where the two sides are as in Definition 4.22.

Proof. Using the notation from Section 6, we can apply $\text{(1)}$ of Lemma 6.5 to the monotone increasing $f={\chi }_{\pi }\cdot L^{\text{phys}}$, to get $$E_{{\Theta }_{F_D}^{\ast }}[{\chi }_{\pi }\cdot L^{\text{phys}}]=E_{{\beta }_{F_D}\left(\pi \right)}[L^{\text{phys}}\circ Q^{-1}],$$ since the maximum over $\gamma \in \Gamma$ is attained when $\gamma =\pi$, due to the ${\chi }_{\pi }$ factor. We have that commutes, i.e. $L=L^{\text{phys}}\circ Q^{-1}$. To see this, note that $h\in X_{Q^{-1}\left(d\right)}$ if and only if $h⊏d$, by an argument analogous to Lemma 6.6, hence $$L^{\text{phys}}\left(Q^{-1}\right(d\left)\right)=\underset{h⊏d}{min}\underset{\tilde{d}⊐h}{max}L\left(d\right)=L\left(d\right).$$ Therefore, $$E_{{\beta }_{F_D}\left(\pi \right)}[L^{\text{phys}}\circ Q^{-1}]=E_{{\beta }_{F_D}\left(\pi \right)}\left[L\right]=L^{\mathrm{Cop}}\left(\pi \right),$$ and so by refinement $$L^{\text{IBP}}\left(\pi \right)=E_{{\Theta }_U^{\ast }}[{\chi }_{\pi }\cdot L^{\text{phys}}]\le E_{{\Theta }_{F_D}^{\ast }}[{\chi }_{\pi }\cdot L^{\text{phys}}]=L^{\mathrm{Cop}}\left(\pi \right).$$ $$\begin{array}{}& \square \end{array}$$

7.2. Asymptotic behavior of communicating MDPs

This section introduces some general definitions and lemmas in the theory of Markov decision processes. Our main goal here is to state and prove Proposition 7.17, concerning the asymptotic behavior of a communicating MDP. None of these results are to be considered original, but are intended as an overview for the reader, as well as a way to establish the exact form of an asymptotic bound that we need (which we couldn't find verbatim in the literature).

Definition 7.2. Let a finite Markov decision process (MDP) be given by the following data (cf. [Put94^[6] Section 2.1]).

• A finite set of states $S$,

• a finite set of actions $A$,

• a transition kernel $\kappa :S\times A\to \Delta S$,

• and a loss function $\ell :S\times A\to R$.

Remark 7.3. The above setting is not the most general one (for example, we could let the set of actions $A_s$ depend on the state, or allow the loss function $\ell :S\times A\to \Delta R$ to be stochastic). The simplifying assumptions we make in the above definition are mostly for the ease of discussion rather than strictly necessary. Some of the results might need additional assumptions in the more general setting, e.g. for Proposition 7.17 we might want to assume that a stochastic $\ell$ is still bounded.

Definition 7.4. For $t\in N$, let $$H_t=(S\times A{)}^t\times S$$ be the set of histories up to time $t$, $$H=\underset{t<T}{⨆}{H}_t$$ be histories up to some time horizon $T$, and $\Gamma =(\Delta A{)}^H$ be the set of (randomized, history-dependent, cf. [Put94^[6:1] Section 2.1.4]) policies.

Remark 7.5. We allow randomized policies here, simply because our discussion in this subsection fits naturally with that generality, and also since it seems common to do so in the classical MDP literature. Note however that optimal policies for an MDP can always be chosen to be deterministic, so our discussion is still compatible with the quantum case, where we only allowed deterministic policies (cf. Remark 2.3).

Definition 7.6 (Time evolution of an MDP). For a given policy $\pi :H\to \Delta A$, and an initial state $h_0\in \Delta S$, we can define recursively for each $t\in N$ a distribution $${\sigma }_t^{\pi }\in \Delta H_t$$ as follows. Take ${\sigma }_0^{\pi }=h_0$, and consider $$H_t\stackrel{{\pi }_t}{\to }\Delta A.$$ We can then form ${\alpha }_t^{\pi }={\sigma }_t^{\pi }⋉{\pi }_t\in \Delta (S\times A{)}^{t+1}$. Now we can compose $$(S\times A{)}^{t+1}\stackrel{{\mathrm{pr}}_{t+1}}{\to }S\times A\stackrel{\kappa }{\to }\Delta S,$$ where ${\mathrm{pr}}_i$ is projection onto the $i$^th factor. Then we can let $${\sigma }_{t+1}^{\pi }={\alpha }_t^{\pi }⋉[\kappa \circ {\mathrm{pr}}_{t+1}]\in \Delta H_{t+1}.$$ We let ${\sigma }^{\pi }|{\sigma }_0:={\sigma }_T^{\pi }$ be the resulting distribution on destinies $D=H_T$, $${\sigma }^{\pi }|{\sigma }_0\in \Delta D.$$ More generally, we can begin with a condition at time $k$, given by $h_k\in \Delta H_k$, and follow the time evolution above to a distribution $${\sigma }^{\pi }|h_k\in \Delta D.$$ For a subset $U\subset D$, we'll write $P_{\pi }\left[U|h_k\right]$ for the probability of $U$, and for a function $f:D\to R$, we'll write $E_{\pi }\left[f|h_k\right]$ for the expected value of $f$ with respect to ${\sigma }^{\pi }|h_k$.

Definition 7.7. For $t=1,\dots ,T$, define ${\ell }_t:D\to R$ by $${\ell }_t\left(d\right)=\ell \left({\mathrm{pr}}_t\right(d\left)\right).$$ and let $L_t:D\to R$ be the total loss $$L_t=\sum _{\tau =t}^T{\ell }_{\tau }.$$

Definition 7.8. We call an MDP communicating (cf [Put94^[6:2] Section 8.3]), if for any pair of states $s_1,s_2\in S$, there exists a policy $\pi \in \Gamma$ and a time $n\in N$ such that $$P_{\pi }\left[{\mathrm{pr}}_n^S\right(d)=s_2|{\delta }_{s_1}]>0,$$ where ${\mathrm{pr}}_n^S$ extracts the $n$^th state of a destiny. Roughly speaking, a communicating MDP allows navigating between any two states with non-zero probability.

We now have all the definitions involved Proposition 7.17, our main result in this section. In the following, we'll introduce various definitions and lemmas that we'll make use of in the proof.

Definition 7.9. For a destiny $d\in D$ and a state $s\in S$, define ${\theta }_s:D\to Z_{+}$ by $${\theta }_s\left(d\right)=min\left(\right\{\theta \in Z_{+}:{\mathrm{pr}}_{\theta }^S\left(d\right)=s\}\cup \{T+1\left\}\right),$$ that is the first time at which state $s$ occurs (or $T+1$ if $s$ doesn't occur). Let $\mathrm{Arr}:S\times S\to R$ be given by $$\mathrm{Arr}(s_1,s_2)=\underset{\pi \in \Gamma }{min}{E}_{\pi }\left[{\theta }_{s_2}|{\delta }_{s_1}\right],$$ that is the minimum expected arrival time to $s_2$, starting from $s_1$.

Lemma 7.10. For a communicating MDP, there is a constant $N$ such that for any time horizon $T$, $$\mathrm{Arr}(s_1,s_2)\le N,$$ for all $s_1,s_2\in S$.

Proof. Let $s_1,s_2\in S$ be two states where the maximum ${max}_{s_i\in S}\mathrm{Arr}(s_1,s_2)$ is attained. Since the MDP is communicating, there exists a policy $\pi \in \Gamma$ and a time $n\in N$ such that $$P_{\pi }\left[{\mathrm{pr}}_n^S\right(d)=s_2|{\delta }_{s_1}]=p>0.$$ Following this $\pi$ for $n$ timesteps (assuming $n<T$, otherwise $\mathrm{Arr}(s_1,s_2)\le n/p$ trivially), we get $$\begin{array}{}\begin{array}{c}\mathrm{Arr}(s_1,s_2)\le \sum _{\tilde{s}\in S}{P}_{\pi }\left[{\mathrm{pr}}_n^S\right(d)=\tilde{s}|{\delta }_{s_1}]\left(\mathrm{Arr}\right(\tilde{s},s_2)+n),\end{array}& \text{(1)}\end{array}$$ by conditioning the state we land on on the $n$^th step. Now, $$\begin{array}{c}\sum _{\tilde{s}\in S}{P}_{\pi }\left[{\mathrm{pr}}_n^S\right(d)=\tilde{s}|{\delta }_{s_1}]\left(\mathrm{Arr}\right(\tilde{s},s_2)+n)\le & \left(\sum _{\tilde{s}\in S\setminus \left\{s_2\right\}}{P}_{\pi }\right[{\mathrm{pr}}_n^S\left(d\right)=\tilde{s}|{\delta }_{s_1}\left]\left(\mathrm{Arr}\right(\tilde{s},s_2)+n)\right)+pn\le & (1-p)\left(\mathrm{Arr}\right(s_1,s_2)+n)+pn,\\ \text{(2)}& \text{(3)}\end{array}$$ where $\text{(2)}$ follows from the assumption on the policy $\pi$ arriving to $s_2$ with probability $p$ after $n$ steps, and $\text{(3)}$ follows from our assumption on $\mathrm{Arr}(s_1,s_2)$ being maximal (so $\mathrm{Arr}(\tilde{s},s_2)\le \mathrm{Arr}(s_1,s_2)$). Combining with $\text{(1)}$, we get $$\mathrm{Arr}(s_1,s_2)\le (1-p)\left(\mathrm{Arr}\right(s_1,s_2)+n)+pn,$$ so $$\mathrm{Arr}(s_1,s_2)\le \frac{n}{p}.$$ $$\begin{array}{}& \square \end{array}$$

Definition 7.11. For $t\in Z_{+}$, let the value function $V_t:D\to R$ be given by $$V_t\left(d\right)=\underset{\pi \in \Gamma }{min}{E}_{\pi }\left[L_t|{\delta }_{{\mathrm{pr}}_{<t}\left(d\right)}\right],$$ i.e. the minimal expected remaining loss after time $t$, assuming the state at time $t$ agrees with $d$. Here ${\mathrm{pr}}_{<t}:D\to H_{t-1}$ truncates to an initial history.

Remark 7.12. As defined above, the value function depends on the entire history ${\mathrm{pr}}_{<t}\left(d\right)\in (S\times A{)}^{t-1}\times S=H_{t-1}$, up to time $t$. It turns out (see [Put94^[6:3] Theorem 4.4.2.]) that in fact it's determined by the last state, $s_t={\mathrm{pr}}_t^S\left(d\right)$, of this history. By slight abuse of notation, we'll write $V_t:S\to R$ for the resulting function as well.

Lemma 7.13. For a communicating MDP, there exists a constant $K$, such that for any time horizon $T$, $$\underset{s\in S}{max}{V}_t\left(s\right)-\underset{s\in S}{min}{V}_t\left(s\right)\le K.$$

Proof. Let $s_1\in {argmax}_{s\in S}{V}_t\left(s\right)$, and $s_2\in {argmin}_{s\in S}{V}_t\left(s\right)$. By Lemma 7.10, $\mathrm{Arr}(s_1,s_2)\le N$, and let ${\pi }_{12}$ be a policy that attains $$\mathrm{Arr}(s_1,s_2)=\underset{\pi \in \Gamma }{min}{E}_{\pi }\left[{\theta }_{s_2}|{\delta }_{s_1}\right].$$ Let ${\pi }_2^{\ast }$ be a policy that attains $$V_t\left(s_2\right)=\underset{\pi \in \Gamma }{min}{E}_{\pi }[L_t|{\sigma }_t={\delta }_{s_2}].$$ Now construct a policy $\tilde{\pi }=‘‘{\pi }_{12}\bigsqcup {\pi }_2^{\ast }"$ as follows. In words, $\tilde{\pi }$ follows ${\pi }_{12}$ from $s_1$ until arriving at $s_2$, and from then on follows ${\pi }_2^{\ast }$. Formally, we can write for a history $h\in H$, $$\tilde{\pi }\left(h\right)=\{\begin{array}{c}{\pi }_{12}\left({\mathrm{pr}}_{\ge t}\right(h\left)\right)\text{if}{\mathrm{pr}}_i^S\left(h\right)\ne s_2\text{for any}t\le i\\ {\pi }_2^{\ast }\left({\mathrm{pr}}_{\ge k}\right(h\left)\right)\text{where}k\in N\text{is smallest such that}{\mathrm{pr}}_{t+k}^S\left(h\right)=s_2.\end{array}$$ (Note that we use ${\mathrm{pr}}_{\ge i}$ as a way of shifting the history in time, for example ${\mathrm{pr}}_{\ge 3}\left(s_1{a}_1{s}_2{a}_2{s}_3{a}_3{s}_4{a}_4{s}_5\right)=s_3{a}_3{s}_4{a}_4{s}_5$.) Now, we have $$V_t\left(s_1\right)\le E_{\tilde{\pi }}[L_t|{\sigma }_t={\delta }_{s_1}],$$ and we can write $$L_t\left(d\right)=\sum _{\tau =0}^{k-1}{\ell }_{t+\tau }+L_{t+k},$$ where $k\in N$ is smallest such that ${\mathrm{pr}}_{t+k}^S\left(d\right)=s_2$. Here $$\sum _{\tau =0}^{k-1}{\ell }_{t+\tau }\le k\cdot max\ell ,$$ $$L_{t+k}\le L_t-k\cdot min\ell ,$$ so $$\begin{array}{c}{E}_{\tilde{\pi }}[L_t|{\sigma }_t={\delta }_{s_1}]\le \\ E_{\tilde{\pi }}\left[{\theta }_{s_2}|{\delta }_{s_1}\right]\cdot max\ell +E_{\tilde{\pi }}\left[L_t|{\delta }_{s_2}\right]-E_{\tilde{\pi }}\left[{\theta }_{s_2}|{\delta }_{s_1}\right]\cdot min\ell =\\ E_{{\pi }_2^{\ast }}\left[L_t|{\delta }_{s_2}\right]+E_{{\pi }_{12}}\left[{\theta }_{s_2}|{\delta }_{s_1}\right](max\ell -min\ell )=\\ V_t\left(s_2\right)+\mathrm{Arr}(s_1,s_2)(max\ell -min\ell )\le \\ V_t\left(s_2\right)+K,\end{array}$$

for $K=N(max\ell -min\ell )$, where $N$ is as in Lemma 7.10. To summarize in words, starting from $s_1$ we can make it to $s_1$ in at most $N$ expected timesteps, accumulating at most $N\cdot max\ell$ loss in expectation. Then we can follow the optimal policy starting at time $t+k$ from $s_2$, and accumulate $V_{t+k}\left(s_2\right)$ loss, which is at most $k\cdot min\ell$ different from $V_t\left(s_2\right)$. Putting these together, we get $V_t\left(s_1\right)-V_t\left(s_2\right)\le K=N(max\ell -min\ell )$. $$\begin{array}{}& \square \end{array}$$

Lemma 7.14. For any policy $\pi$, and any $d\in D$, $$V_t\left(d\right)\le E_{\pi }[{\ell }_t+V_{t+1}|{\delta }_{{\mathrm{pr}}_{<t}\left(d\right)}].$$

Proof. By the optimality of $V_t\left(d\right)$, we have $$V_t\left(d\right)=\underset{a\in A}{min}{q}_t(s_t,a),$$ where $s_t={\mathrm{pr}}_t^S\left(d\right)$ is the $t$^th state, and $$q_t(s_t,a)=\ell (s_t,a)+\sum _{s_{t+1}\in S}{P}_{\kappa }[s_{t+1}|s_t,a]V_{t+1}\left(s_{t+1}\right).$$ On the other hand, $$E_{\pi }[{\ell }_t+V_{t+1}|{\delta }_{{\mathrm{pr}}_{<t}\left(d\right)}]=E_{a\sim \pi \left(d_{<t}\right)}\left[q\right(s_t,a\left)\right],$$ i.e. the expected value of $q_t(s_t,a)$ when the action $a$ is distributed according to the policy $\pi \left(d_{<t}\right)\in \Delta A$. The inequality now follows. $$\begin{array}{}& \square \end{array}$$

Definition 7.15. Let $D_t={\ell }_t-V_t+V_{t+1}:D\to R$.

Lemma 7.16. We have for any policy $\pi$ and initial state $h_k\in \Delta H_k$ with $k<t$, $$E_{\pi }\left[D_t|h_k\right]\ge 0,$$ and $$|D_t|\le C,$$ for $C=max\ell -min\ell +K$, where $K$ is as in Lemma 7.13.

Proof. Since, by Lemma 7.14 we have $E_{\pi }\left[D_t|{\delta }_h\right]\ge 0$ for any history $h\in H_{t-1}$, the same holds for any distribution of histories, in particular also for the ${\sigma }_{t-1}^{\pi }\in \Delta H_{t-1}$ given by the time evolution of $h_k$. We also have $$\underset{s\in S}{min}{V}_t\left(s\right)\le V_t\le \underset{s\in S}{max}{V}_t\left(s\right),$$ $$\underset{s\in S}{min}{V}_t\left(s\right)-max\ell \le V_{t+1}\le \underset{s\in S}{max}{V}_t\left(s\right)-min\ell ,$$ from which $|D_t|\le C$ follows. $$\begin{array}{}& \square \end{array}$$

Proposition 7.17. For a communicating MDP, there is a constant $C$ such that for any policy $\pi$ and initial state $h_0$, $$P_{\pi }[L_1\le L^{\ast }-ϵ|h_0]<\delta$$ holds whenever $${ϵ}^2>2T{C}^2\log \frac{1}{\delta },$$ where $$L^{\ast }=\underset{\pi \in \Gamma }{min}{E}_{\pi }\left[L_1|h_0\right]$$ is the minimal expected loss. In words, it's unlikely (under any policy and initial state) for the total loss to be much below the minimal expected loss.

Proof. We have $$L_1-V_1=\sum _{t=1}^T{D}_t,$$ where $Y_t={\sum }_{\tau =1}^t{D}_{\tau }$ is a bounded sub-martingale by Lemma 7.16, so by Azuma's inequality we get $$P_{\pi }\left[(L_1-V_1\le -ϵ)|h_0\right]\le \exp \left(\frac{-{ϵ}^2}{2T{C}^2}\right).$$ Since this holds for all $h_0$, and $L^{\ast }=E_{h_0}\left[V_1\right]$, we also get the stated result. $$\begin{array}{}& \square \end{array}$$

7.3. Lower bound

We can use the result above to obtain a lower bound on $L^{\text{IBP}}$.

Definition 7.18. Assume we have the setup of a system given in Section 2, and furthermore that $O$ is a complete set of observations (so each $P_o$ has a 1-dimensional image). Then given a loss function $\ell :O\times A\to R_{\ge 0}$, there's a Markov decision process associated with this setting, where the set of states is $S=O$, and the transition probabilities $\kappa :S\times A\to \Delta S$ are given via the Born rule: $$P_{\kappa }[o_2|o_1,a]=\parallel P_{o_2}{U}_a|{\psi }_{o_1}⟩{\parallel }^2,$$ where $|{\psi }_{o_1}⟩$ is a unit vector in the image of $P_{o_1}$.

Remark 7.19. It might be interesting to also consider the case where $O$ is incomplete. In this case there's an associated POMDP (partially obvervable Markov decision process). Note, however that a priori this POMDP will have infinitely many states (all rays in the image of $P_o$ for each $o\in O$). We won't pursue this direction here.

Remark 7.20. To understand the structure of the resulting MDP a little better, consider the following. For two observations $o_1,o_2$, let's say that $o_1\ge o_2$ ($o_2$ can be reached from $o_1$) if $⟨{\psi }_{o_2}|U_a|{\psi }_{o_1}⟩\ne 0$ for some $a\in A$, and take the transitive closure of this relationship. The resulting relationship is in fact also symmetric and reflexive. This follows because the unitary group is compact (since we assume $O$ is a finite and complete set of observations, so $H^e$ is finite dimensional), so powers of $U_a$ can approximate the identity and $U_a^{†}=U^{-1}$ arbitrarily. Thus the MDP is a disjoint union of communicating components (the equivalence classes of the relation above). For generic $\{U_a:a\in A\}$, we'll have a single equivalence class. Otherwise the first observation picks out a component, and the rest of the evolution remains within that component.

Theorem 7.21. If the associated MDP to a setup is communicating, then for any policy $\pi$, we have $$L^{\ast }-O\left(\sqrt{T\log T}\right)\le L^{\text{IBP}}\left(\pi \right),$$ where $L^{\ast }$ is the minimal Copenhagen loss (i.e. $L^{\ast }=L^{\mathrm{Cop}}\left({\pi }^{\ast }\right)$ for a Copenhagen-optimal policy ${\pi }^{\ast }$).

Proof. For $ϵ>0$, consider the event $$E=\{d\in D:L(d)\le L^{\ast }-ϵ\}.$$ Choose a policy ${\tilde{\pi }}_E$ as in Definition 6.12. Then it's easy to verify using the definition of $L^{\text{phys}}$, that $$\begin{array}{}\begin{array}{c}{L}^{\text{phys}}(\pi ,\{\pi ,{\tilde{\pi }}_E\left\}\right)\ge L^{\ast }-ϵ.\end{array}& \text{(1)}\end{array}$$ Let $$p=\mathrm{Cop}\left(E|\pi \right)$$ be the Copenhagen probability that the loss is at most $L^{\ast }-ϵ$, given the policy $\pi$. By Proposition 7.17 we have that $$\begin{array}{}\begin{array}{c}p\le \exp (-\frac{{ϵ}^2}{2T{C}^2}).\end{array}& \text{(2)}\end{array}$$

By Proposition 6.18, we have $$\begin{array}{}\begin{array}{c}{C}_{E|\pi }\cdot {\delta }_{\pi ,\{\pi ,{\tilde{\pi }}_E\}}\in {\Theta }_U^{\ast },\end{array}& \text{(3)}\end{array}$$

for $C_{E|\pi }=1-\sqrt{p(2-p)}$. Therefore by $\text{(1)}$ and $\text{(3)}$, we have $$L^{\text{IBP}}=E_{{\Theta }_U^{\ast }}[{\chi }_{\pi }\cdot L^{\text{phys}}]\ge (L^{\ast }-ϵ)(1-\sqrt{p(2-p)}).$$ Here $p(2-p)\le 2p$, and using $\text{(2)}$, we get $$L^{\text{IBP}}\ge (L^{\ast }-ϵ)(1-\sqrt{2}\exp (-\frac{{ϵ}^2}{4T{C}^2})).$$ To obtain an $O\left(\sqrt{T\log T}\right)$ bound, we can set $ϵ=C\sqrt{2T\log T}$, which gives $$L^{\text{IBP}}\ge (L^{\ast }-C\sqrt{2T\log T})(1-\sqrt{\frac{2}{T}})=L^{\ast }-O\left(\sqrt{T\log T}\right),$$ since $L^{\ast }=O\left(T\right)$. $$\begin{array}{}& \square \end{array}$$

Note that Theorem 4.26 implies that any Copenhagen-optimal policy is also asymptotically IBP-optimal. The converse is also true, but requires a bit more work.

Theorem 7.22. If $\overline{\pi }$ is an IBP-optimal policy, then $$L^{\mathrm{Cop}}\left(\overline{\pi }\right)\le L^{\ast }+o\left(T\right),$$ where $L^{\ast }$ is the Copenhagen-optimal loss.

Proof. For ${ϵ}_1,{ϵ}_2>0$, consider the events $$E_{-{ϵ}_1}=\{d\in D:L(d)\le L^{\ast }-{ϵ}_1\},$$ $$E_{+{ϵ}_2}=\{d\in D:L(d)\le L^{\ast }+{ϵ}_2\}.$$ On a high level, the proof goes as follows. We already know that the Copenhagen probability of $E_{-{ϵ}_1}$ is small. We'll show that for an IBP-optimal $\overline{\pi }$, the complement of $E_{+{ϵ}_2}$ also has small probability, so most of the probability mass is where the loss is between $L^{\ast }-{ϵ}_1$ and $L^{\ast }+{ϵ}_2$, which will be sufficient to show that $L^{\mathrm{Cop}}\left(\overline{\pi }\right)$ is not much bigger than $L^{\ast }$.

Choose policies ${\tilde{\pi }}_{-{ϵ}_1}$, ${\tilde{\pi }}_{+{ϵ}_2}$ corresponding to $E_{-{ϵ}_1}$ and $E_{+{ϵ}_2}$ as in Definition 6.12. Let $$p_1=\mathrm{Cop}\left(E_{-{ϵ}_1}|\overline{\pi }\right),$$ $$p_2=\mathrm{Cop}\left(E_{+{ϵ}_2}|\overline{\pi }\right),$$ so $p_2\ge p_1$. Again, by Proposition 6.18, $$C_{-{ϵ}_1}\cdot {\delta }_{\overline{\pi },\{\overline{\pi },{\tilde{\pi }}_{-{ϵ}_1}\}}\in {\Theta }_U^{\ast },$$ $$C_{+{ϵ}_2}\cdot {\delta }_{\overline{\pi },\{\overline{\pi },{\tilde{\pi }}_{+{ϵ}_2}\}}\in {\Theta }_U^{\ast },$$ where $$C_{-{ϵ}_1}=1-\sqrt{p_1(2-p_1)},$$ $$C_{+{ϵ}_2}=1-\sqrt{p_2(2-p_2)}.$$ By Lemma 6.11, Proposition 7.24 applies as well, so in this case we also have $$\rho =(C_{-{ϵ}_1}-C_{+{ϵ}_2})\cdot {\delta }_{\overline{\pi },\{\overline{\pi },{\tilde{\pi }}_{-{ϵ}_1}\}}+C_{+{ϵ}_2}\cdot {\delta }_{\overline{\pi },\{\overline{\pi },{\tilde{\pi }}_{+{ϵ}_2}\}}\in {\Theta }_U^{\ast }.$$ Hence $$\begin{array}{}\begin{array}{c}{L}^{\text{IBP}}\left(\overline{\pi }\right)=E_{{\Theta }_U^{\ast }}[{\chi }_{\overline{\pi }}\cdot L^{\text{phys}}]\ge E_{\rho }[{\chi }_{\overline{\pi }}\cdot L^{\text{phys}}]\ge \\ (L^{\ast }-{ϵ}_1)(\sqrt{p_2(2-p_2)}-\sqrt{p_1(2-p_1)})+(L^{\ast }+{ϵ}_2)(1-\sqrt{p_2(2-p_2)}).\end{array}& \text{(4)}\end{array}$$ Since $\overline{\pi }$ is IBP-optimal, we have for any Copenhagen-optimal policy ${\pi }^{\ast }$, $$\begin{array}{}\begin{array}{c}{L}^{\text{IBP}}\left(\overline{\pi }\right)\le L^{\text{IBP}}\left({\pi }^{\ast }\right)\le L^{\mathrm{Cop}}\left({\pi }^{\ast }\right)=L^{\ast }.\end{array}& \text{(5)}\end{array}$$ From $\text{(4)}$ and $\text{(5)}$ together we have $$(L^{\ast }-{ϵ}_1)(\sqrt{p_2(2-p_2)}-\sqrt{p_1(2-p_1)})+(L^{\ast }+{ϵ}_2)(1-\sqrt{p_2(2-p_2)})\le L^{\ast }.$$ Rearranging, and using $\sqrt{p_1(2-p_1)}\le \sqrt{2}\exp (-\frac{{ϵ}_1^2}{4T{C}^2})$ as before, we get $$\sqrt{p_2(2-p_2)}\ge 1-\frac{{ϵ}_1+(L^{\ast }-{ϵ}_1)\sqrt{2}\exp (-\frac{{ϵ}_1^2}{4T{C}^2})}{{ϵ}_1+{ϵ}_2},$$ so choosing ${ϵ}_1=O\left(\sqrt{2T\log T}\right)$ and ${ϵ}_2=O\left(T^{5/6}\right)$, we have $$\sqrt{p_2(2-p_2)}\ge 1-\frac{O\left(\sqrt{T\log T}\right)}{\Theta \left(T^{5/6}\right)}=1-O\left(\frac{\sqrt{\log T}}{T^{1/3}}\right).$$ Hence $$p_2\ge 1-O\left(\frac{\sqrt[4]{4\log T}}{T^{1/6}}\right).$$ Therefore $$L^{\mathrm{Cop}}\left(\overline{\pi }\right)\le p_2(L^{\ast }+{ϵ}_2)+(1-p_2)O\left(T\right)\le L^{\ast }+O\left(T^{5/6}\sqrt[4]{4\log T}\right).$$ $$\begin{array}{}& \square \end{array}$$

We can likely improve on the exponent of $5/6$ via more sophisticated estimates, but we won't be needing that for the current level of our discussion.

Remark 7.23. More generally, we can see that an asymptotically Copenhagen-optimal policy is also asymptotically IBP-optimal, and vice versa. In light of Remark 7.20, this remains true even when we drop the assumption of the MDP consisting of a single communicating component. Theorems 7.21 and 7.22 can be applied to each component separately, thus the optimal policies still need to agree asymptotically. The two interpretations then weigh the asymptotic losses of the components differently based on the amplitude of the components in the initial state (the IBP interpretation is more optimistic in the sense that it typically considers the lower loss branches with more weight than the Copenhagen interpretation), hence Theorems 7.21 and 7.22 fail in the case of an initial state that is the superposition of multiple communicating components, but only due to the outcome of the first observation being irreversible in this case, which doesn't affect the claim about the optimal policies agreeing asymptotically.

We finish this section by spelling out the proof of the following.

Proposition 7.24. If $\pi ,{\pi }_1,{\pi }_2$ are three policies such that for any $B\in U$, $$E_{{\beta }_B\left(\pi \right)\wedge {\beta }_B\left({\pi }_1\right)}\left[1\right]\ge C_1,$$ $$E_{{\beta }_B\left(\pi \right)\wedge {\beta }_B\left({\pi }_2\right)}\left[1\right]\ge C_2,$$ for $C_1\ge C_2$, then $$\rho =(C_1-C_2)\cdot {\delta }_{\pi ,\{\pi ,{\pi }_1\}}+C_2\cdot {\delta }_{\pi ,\{\pi ,{\pi }_2\}}\in {\Theta }_U^{\ast }.$$

Proof. The proof is mostly analogous to Proposition 6.18, we highlight the additional ideas here. The claim reduces to Proposition 6.18 for $C_1=C_2$, so we'll assume $C_1>C_2$ in the following. For $B\in U$, let (cf. Definition 6.14) $${\overline{\varphi }}_{2B}={\beta }_B\left(\pi \right)\wedge {\beta }_B\left({\pi }_2\right),$$ $${\varphi }_{2B}=\frac{C_2}{E_{{\overline{\varphi }}_{2B}}\left[1\right]}{\overline{\varphi }}_{2B},$$ so $E_{{\varphi }_{2B}}\left[1\right]=C_2$, and ${\varphi }_{2B}\le {\overline{\varphi }}_{2B}\le {\beta }_B\left({\pi }_2\right)$, since $E_{{\overline{\varphi }}_{2B}}\left[1\right]\ge C_2$ by assumption. Now let $${\overline{\varphi }}_{1B}=\left({\beta }_B\right(\pi )-{\varphi }_{2B})\wedge {\beta }_B\left({\pi }_1\right),$$ $${\varphi }_{1B}=\frac{C_1-C_2}{E_{{\overline{\varphi }}_{1B}}\left[1\right]}{\overline{\varphi }}_{1B},$$ so $E_{{\varphi }_{1B}}\left[1\right]=C_1-C_2$, and ${\varphi }_{1B}\le {\overline{\varphi }}_{1B}\le {\beta }_B\left({\pi }_1\right)$, since $E_{{\overline{\varphi }}_{1B}}\left[1\right]\ge C_1-C_2$ by the assumption that $E_{{\beta }_B\left(\pi \right)\wedge {\beta }_B\left({\pi }_1\right)}\left[1\right]\ge C_1$. Moreover, by construction ${\varphi }_{1B}+{\varphi }_{2B}\le {\beta }_B\left(\pi \right)$. We can then define for any finite $F\subset U$, $${\varphi }_{1F}=\frac{1}{(C_1-C_2{)}^{|F|-1}}\prod _{B\in F}{\varphi }_{1B}\in {\Delta }^c{\Phi }_F,$$ $${\varphi }_{2F}=\frac{1}{C_2^{|F|-1}}\prod _{B\in F}{\varphi }_{2B}\in {\Delta }^c{\Phi }_F,$$ so analogously to Lemma 6.16, we have $${\varphi }_{1F}\in {\beta }_F\left({\pi }_1\right),$$ $${\varphi }_{2F}\in {\beta }_F\left({\pi }_2\right),$$ $${\varphi }_{1F}+{\varphi }_{2F}\in {\beta }_F\left(\pi \right).$$ Using this, we can show that $$\begin{array}{c}{\theta }_F={\delta }_{\pi ,\{\pi ,{\pi }_1\}}\times {\varphi }_{1F}+{\delta }_{\pi ,\{\pi ,{\pi }_2\}}\times {\varphi }_{2F}\in \mathrm{Br}\left({\Theta }_F\right)\subset {\Delta }^c({\mathrm{el}}^{\Gamma }\times {\Phi }_F).\end{array}$$

To see this, consider an endomorphism $s:\Gamma \to \Gamma$, and let $\tilde{s}$ be as in Definition 3.12. The interesting cases are the following:

• If $s\left(\pi \right)=\pi$, then $$\tilde{s}\left({\theta }_F\right)={\delta }_{\pi }\times ({\varphi }_{1F}+{\varphi }_{2F})\in {\Theta }_F,$$ since ${\varphi }_{1F}+{\varphi }_{2F}\in {\beta }_F\left(\pi \right)$ by the above.

• If $s\left(\pi \right)={\pi }_i$ for $i=1,2$, then $$\tilde{s}\left({\theta }_F\right)={\delta }_{{\pi }_i}\times {\varphi }_{iF}\in {\Theta }_F,$$ since ${\varphi }_{iF}\in {\beta }_F\left({\pi }_i\right)$ as well by the above.

Therefore $${\mathrm{pr}}_{\ast }\left({\theta }_F\right)=\rho \in {\Theta }_F^{\ast }.$$ Since this is true for arbitrary $F$, we conclude $\rho \in {\Theta }_U^{\ast }$. $$\begin{array}{}& \square \end{array}$$

8. Limitations

We mention some limitations of the setting, some of which as simply due to the toy nature of the model, others seem to be more inherent to infra-Bayesian physicalism.

8.1. Limitations of the toy setting

Although a central feature of infra-Bayesian physicalism is a lack of privilege for any observer, in the toy model we work with an explicit decomposition of the universe into agent and environment. Other toy assumptions are taking the "computational universe" $\Gamma$ to consist solely of the policy, and the explicit dependence of the time evolution on the policy. In a more realistic setting we would start with a non-Cartesian (not agent-centric) description of the universe, and a rich nexus of mathematical structure encoded in $\Gamma$. The entanglement between the agent's policy and the physical state of the universe would then be encoded implicitly via a "theory of origin" whereby the agent arises in the given universe.

To spell the above out a little more, in a more realistic setting we could take $\Sigma =\{\top ,\perp \}$, and choose $R$ to be a sufficiently rich set of computations to include things like

$r=$ "a program computing the 11th decimal place digit of the amplitude squared of a certain path integral in some lattice QFT and verifying if it’s equal to 7".

Then $\Gamma ={\Sigma }^R$ will contain a lot of immediately inconsistent valuations, like the one where a certain digit is both equal to 7 and to 3. However, we can take a subset ${\Gamma }_0\subset \Gamma$, which is "consistent enough", e.g. so that for every computation of the form "a certain digit in a given quantity equals $i$", exactly one of $i\in \{0,\dots ,9\}$ evaluates to $\top$, all others evaluate to $\perp$. We would choose ${\Gamma }_0$ to be sufficiently small to produce a meaningful map $\beta :{\Gamma }_0\to \square \Phi$ (describing a certain model of physics), e.g. so that the distribution of the momentum of a given field modeled in $\Phi$, at a point is given as specified by the values of the computations like $r$ above. We can then combine the mathematical/computational part of a hypothesis ${\Theta }_{\Gamma }\in \square \Gamma$ (supported only on the sufficiently consistent part of the computational universe ${\Gamma }_0\subset \Gamma$) with $\beta$ to construct a corresponding joint hypothesis $\Theta ={\Theta }_{\Gamma }⋉\beta \in \square (\Gamma \times \Phi )$.

The notion of a "theory of origin" has not been formalized yet, but we informally discuss some ingredients here. Given the source code $G$ of the agent, and a policy $\pi$, we can define the $\pi$-counterfactual of $\Theta$ as $${\Theta }^{\pi }=\left({\mathrm{pr}}_{{\mathrm{el}}^{\Gamma }}{)}_{\ast }\right(\mathrm{Br}\left(\Theta \right))\cap {\top }_{C^{\pi }}\in \square {\mathrm{el}}^{\Gamma },$$ where $C^{\pi }$ is the subset of universes compatible (by some notion) with the given policy $\pi$ (cf. [IBP Definition 1.5], also Remark 4.23). We can then look at the diameter of these counterfactuals in some metric, $$\text{diam}\left(\right\{{\Theta }^{\pi }:\pi :O^{\ast }\to A\left\}\right),$$ as a measure of the extent to which the agent is realized in the given physical model (i.e. how entangled the agent's policy is with the world). Moreover, in a more realistic setting we would expect the entanglement between the policy and the world to come from "non-contrived" reasons (as opposed to our toy model, where we just postulated the dependence of the time evolution on the policy), which could be measured by some notion of complexity of the source code $G$ relative to the physical hypothesis $\beta$ (higher relative complexity means a less contrived theory of origin).

8.2. Limitations of the broader framework

The decision theory of infra-Bayesian physicalism is based on a computationalist loss function $L:{\mathrm{el}}^{\Gamma }\to R_{\ge 0}$. So the value of the loss is required to be determined by the state of the computational universe plus the fact of which computations are realized in the physical universe. This can lead to non-trivial translation problems from loss functions that are specified in more traditional terms. Moreover, the computationalist loss function is required to be monotonic (see monotonicity principle in [IBP]) in the computations realized, a requirement not immediately intuitive.

---

1. Working with a finite time horizon is convenient for technical reasons, but not expected to be strictly necessary. ↩︎

2. Richard Durrett. Probability: theory and examples. Duxbury Press, second edition, 1996. ↩︎

3. Veronika Baumann and Časlav Brukner. Wigner’s friend as a rational agent, 2019. ↩︎

4. If we wanted to work in a strictly subjectivist framework for the friend, we could include an additional observation of Wigner's memory tape by the friend, and have the loss function depend on the outcome of that observation. We don't expect this to make a significant difference for the present discussion. ↩︎

5. We could also require that $\alpha$ witness $F$ having observed something, which would correspond to adding the condition that $(\forall \tilde{\gamma }\in \alpha :{\tilde{\gamma }}^F(o_0)={\gamma }^F(o_0\left)\right)\text{or}(\forall \tilde{\gamma }\in \alpha :{\tilde{\gamma }}^F(o_1)={\gamma }^F(o_1\left)\right)$. We expect this would change some of the exact expected values of the loss, but not the optimal policy in this case. ↩︎

6. Martin L. Puterman. Markov Decision Processes: Discrete Stochastic Dynamic Programming. John Wiley & Sons, Inc., 1st edition, 1994 ↩︎ ↩︎ ↩︎ ↩︎