Proofs Theorem 5

Theorem 5: Causal Commutative Square: The following diagram commutes for any choice of causal $Θ$ . Any choice of infradistribution $E \in □ E$ where $E = ⋀_{π} (π \cdot)^{*} ((π \cdot)_{*} (E))$ also makes this diagram commute.

Again, like usual, this proof consists of doing preliminary work (section 0) where we clarify some stuff and find 4 conditions for the 4 corners that correspond to "maximal representative of a causal belief function". Section 1.1-8 is showing that these properties are preserved under all eight morphisms, where as usual, the complicated part of translating between a belief function and infradistribution is shoved in the first two sections. Sections 2.1-8 are then showing the 8 identities needed to establish 4 isomorphisms assuming the relevant properties, and then section 3 is going in two directions from one corner to another and showing equality to rule out nontrivial automorphisms.

T5.0 First, preliminaries. We use suprema over: probability distributions $ζ$ on the space of policies, and utility functions $U_{π}$ we associate with each policy $π$ , which fulfill the following constraint for some function $f$ .

$\forall e : E_{π \sim ζ} [U_{π} (π \cdot e)] \leq f (e)$

To avoid writing all this out each time, we observe that for the purposes of the proof, all that varies in this format is the function $f : E \to [0, 1]$ that we're using as an upper-bound, we always select probability distributions over policies and utility functions linked with the same policy, and impose upper bounds of that "for all copolicies" form on a mix of the relevant utility functions interacting with the copolicy. So we'll suppress all this in the notation and just say

${sup}_{ζ, U_{π}, f}$

as our abbreviation, with $∙$ for when we need a function to have an empty slot for copolicies. So something like $U (π^{'} \cdot ∙)$ would be "the function from copolicies to $[0, 1]$ given by letting $π^{'}$ interact with the copolicy to get a history, and then applying $U$ to it".

The defining property for the third-person static view is that $E$ is an infradistribution, and, for all functions $f : E \to [0, 1]$ , $E (f) = {sup}_{ζ, U_{π}, f} E_{π \sim ζ} [E (λ e . U_{π} (π \cdot e))]$

The property for the third-person dynamic view is that $ψ_{T}$ is an infradistribution, and $ψ_{T} (f) = {sup}_{ζ, U_{π}, f} E_{π \sim ζ} [ψ_{T} (λ e . U_{π} (π \cdot e))]$ and $\to_{T} (e) = ⊤_{A} ⋉ (λ a . δ_{e (a), e_{a, e (a)}})$

The property for the first-person static view is that $Θ$ be a belief function, and that it fulfill causality, that $Θ (π^{'}) (U) = {sup}_{ζ, U_{π}, U (π^{'} \cdot ∙)} E_{π \sim ζ} [Θ (π) (U_{π})]$

The property for the first-person dynamic view is that $ψ_{F}$ is an infradistribution, and $ψ_{F} (f) = {sup}_{ζ, U_{π}, f} E_{π \sim ζ} [ψ_{F} (λ e . U_{π} (π \cdot e))]$ and $\to_{F} (e, a) = δ_{e (a), e_{a, e (a)}}$

Before showing that these properties are preserved under morphisms, it will be handy to have an equivalent rephrasing of causality for belief functions.

The causality condition is:

$Θ (π^{'}) (U) = {sup}_{ζ, U_{π}, U (π^{'} \cdot ∙)} E_{π \sim ζ} [Θ (π) (U_{π})]$

We can unconditionally derive the statement

$Θ (π^{'}) (U) \leq {sup}_{ζ, U_{π}, U (π^{'} \cdot ∙)} E_{π \sim ζ} [Θ (π) (U_{π})]$

because, letting $ζ$ be the dirac-delta distribution on $π^{'}$ , and $U_{π^{'}} = U$ , we trivially fulfill the necessary condition for the supremum, that $\forall e : E_{ζ} [U_{π} (π \cdot e)] \leq U (π^{'} \cdot e)$

So, we get the fact

$Θ (π^{'}) (U) = E_{π \sim δ_{π^{'}}} [Θ (π) (U_{π})] \leq {sup}_{ζ, U_{π}, U (π^{'} \cdot ∙)} E_{π \sim ζ} [Θ (π) (U_{π})]$

With this knowledge, the causality condition holds iff

$Θ (π^{'}) (U) \geq {sup}_{ζ, U_{π}, U (π^{'} \cdot ∙)} E_{π \sim ζ} [Θ (π) (U_{π})]$

and this holds iff

$\forall ζ \in Δ Π, U_{π} : (\forall e : E_{π \sim ζ} [U_{π} (π \cdot e)] \leq U (π^{'} \cdot e)) \to E_{π \sim ζ} [Θ (π) (U_{π})] \leq Θ (π^{'}) (U)$

In the first direction, this happens because any particular choice of $ζ$ and $U_{π}$ which fulfills the implication antecendent must have the expectation of $Θ (π) (U_{π})$ doing worse than the supremum of such expectation, which still loses to $Θ (π^{'}) (U)$ .

In the other direction, this happens because the supremum only ranges over distributions on policies and choices of utility function that fulfills the implication antecedent, so the consequent always kicks in. Thus, even the largest possible expectation of $Θ (π) (U_{π})$ isn't above $Θ (π^{'}) (U)$ .

Thus, we will be taking

$\forall ζ \in Δ Π, U_{π} : (\forall e : E_{π \sim ζ} [U_{π} (π \cdot e)] \leq U (π^{'} \cdot e)) \to E_{π \sim ζ} [Θ (π) (U_{π})] \leq Θ (π^{'}) (U)$

as an alternative phrasing of causality for belief functions.

Time to show that these properties are preserved under the morphisms. As usual, the hard part is showing property preservation between first-person static and first-person dynamic.

T5.1.1 3-static to 1-static. Our translation is

$Θ (π) := (π \cdot)_{*} (E)$

and we must check the five conditions of a belief function, as well as causality. So the proof splits into six parts for those six desired results. The five conditions for a belief function are a bounded Lipschitz constant, lower-semicontinuity, normalization, sensible supports, and agreement on max value. We'll just be using that $E$ is an infradistribution, and not that it has any additional properties beyond that. The pushforward of an infradistribution is always an infradistribution, so we know that $Θ (π)$ is always an infradistribution.

T5.1.1.1 For a bounded Lipschitz constant, we have

$| Θ (π) (U) - Θ (π) (U^{'}) | = | (π \cdot)_{*} (E) (U) - (π \cdot)_{*} (E) (U^{'}) |$

$= | E (λ e . U (π \cdot e)) - E (λ e . U^{'} (π \cdot e)) |$

and then, since $E$ has some Lipschitz constant $λ^{⊙}$ , we then have

$\leq λ^{⊙} {sup}_{e} | U (π \cdot e) - U^{'} (π \cdot e) |$

And, for any $π, e$ pair, it induces a history, so this is upper-bounded by

$\leq λ^{⊙} {sup}_{h} | U (h) - U^{'} (h) | = λ^{⊙} d (U, U^{'})$

So a uniform Lipschitz constant upper bound is shown.

T5.1.1.2 For lower-semicontinuity, it's actually easy to show for once. We get something even stronger. All causal belief functions aren't just lower-semicontinuous in their first argument, they're continuous. The proof is:

${lim}_{n \to \infty} Θ (π_{n}) (U) = {lim}_{n \to \infty} (π_{n} \cdot)_{*} (E) (U) = {lim}_{n \to \infty} E (λ e . U (π_{n} \cdot e))$

Now, the thing to realize here is that $λ e . U (π_{n} \cdot e)$ uniformly converges to $λ e . U (π_{\infty} \cdot e)$ . Why does this happen? Well, once $π_{n}$ gets close enough to $π_{\infty}$ , it will perfectly agree with $π_{\infty}$ about what to do in any situation for the first m steps. So, vs any copolicy at all, the histories $π_{n} \cdot e$ (for all sufficiently late n) and $π_{\infty} \cdot e$ will agree for the first m steps. And, since $U$ is continuous on $(A \times O)^{ω}$ which is a compact space, it must be uniformly continuous. So for any $ϵ$ difference in output, there's some m where knowing the first m steps of history lets you pin down the value of $U$ to within $ϵ$ .

Then, we can just appeal to the finite Lipschitz constant of $E$ to conclude that, since we have uniform convergence of functions, we have

$= E (λ e . U (π_{\infty} \cdot e))$

which then packs back up in reverse as

$= Θ (π_{\infty}) (U)$

That was very easy compared to our typical lower-semicontinuity proofs!

T5.1.1.3 For normalization, we have

${inf}_{π} Θ (π) (1) = {inf}_{π} (π \cdot)_{*} (E) (1) = {inf}_{π} E (1) = 1$

And the same goes for 0, because $E$ is an infradistribution.

T5.1.1.4 For sensible supports, assume $U, U^{'}$ agree on histories that can be produced by $π$ . Then,

$| Θ (π) (U) - Θ (π) (U^{'}) | = | (π \cdot)_{*} (E) (U) - (π \cdot)_{*} (E) (U^{'}) |$

$= | E (λ e . U (π \cdot e)) - E (λ e . U^{'} (π \cdot e)) |$

But then wait, for any $e$ , $π \cdot e$ makes a history that can be made by $π$ , so then $U, U^{'}$ must agree on it. Those two inner functions are equal, and so attain equal value, and we have $= 0$ .

T5.1.1.5 For agreement on max value, you can just go

$Θ (π) (1) = (π \cdot)_{*} (E) (1) = E (1) = (π^{'} \cdot)_{*} (E) (1) = Θ (π^{'}) (1)$

and

$Θ (π) (\infty) = (π \cdot)_{*} (E) (\infty) = E (\infty) = (π^{'} \cdot)_{*} (E) (\infty) = Θ (π^{'}) (\infty)$

and you're done.

T5.1.1.6 That just leaves checking causality. We'll make a point to not assume any special conditions on $E$ at all in order to show this, because the

$E (f) = {sup}_{ζ, U_{π}, f} E_{π \sim ζ} [E (λ e . U_{π} (π \cdot e))]$

condition is actually saying that $E$ is the maximal infradistribution that corresponds to a causal belief function, but it'll be useful later to know that any infradistribution on copolicies makes a causal belief function.

We'll be showing our alternate characterization of causality, that

$\forall ζ \in Δ Π, U_{π}, U : (\forall e : E_{π \sim ζ} [U_{π} (π \cdot e)] \leq U (π^{'} \cdot e)) \to E_{π \sim ζ} [Θ (π) (U_{π})] \leq Θ (π^{'}) (U)$

So, pick an arbitrary $ζ$ , family of $U_{π}$ , utility function $U$ , and policy $π^{'}$ , and assume $\forall e : E_{π \sim ζ} [U_{π} (π \cdot e)] \leq U (π^{'} \cdot e)$ Then, we can go:

$E_{π \sim ζ} [Θ (π) (U_{π})] = E_{π \sim ζ} [(π \cdot)_{*} (E) (U_{π})] = E_{π \sim ζ} [E (λ e . U_{π} (π \cdot e))]$

and then, by applying concavity of infradistributions, we can move the expectation inside and the value goes up, and then we apply monotonicity via the precondition we assumed.

$\leq E (λ e . E_{π \sim ζ} [U_{π} (π \cdot e)]) \leq E (λ e . U (π^{'} \cdot e))$

and then we just pack this back up.

$= (π^{'} \cdot)_{*} (E) (U) = Θ (π^{'}) (U)$

and we have the causality condition shown.

T5.1.2 1-static to 3-static. Our translation is:

$E := ⋀_{π} (π \cdot)^{*} (Θ (π))$

The way this proof works is first, we unpack $E$ into a different form in preparation for what follows, that's phase 1. Then there's 6 more phases after that, where we show the five infradistribution conditions on $E$ and the causality condition. Some of these are hard to show so we'll have to split them into more parts.

T5.1.2.1 We'll unpack this first, in preparation for showing the infradistribution properties and causality property for $E$ . We can go

$E (f) = ⋀_{π} (π \cdot)^{*} (Θ (π)) (f)$

and then express the intersection as a supremum over $ζ \in Δ Π$ and $f_{π} : E \to [0, 1]$ such that $\forall e : E_{π \sim ζ} [f_{π} (e)] \leq f (e)$ because that's just how intersections work. This is abbreviated as ${sup}_{ζ, f_{π}, f}$ .

$= {sup}_{ζ, f_{π}, f} E_{π \sim ζ} [(π \cdot)^{*} (Θ (π)) (f_{π})]$

Then, for the pullback, we take the supremum over $U_{π}$ where $U_{π} (π \cdot e) \leq f_{π} (e)$ for all $e$ . We'll write this as ${sup}_{U_{π}, f_{π}}$ .

This makes

$= {sup}_{ζ, f_{π}, f} E_{π \sim ζ} [{sup}_{U_{π}, f_{π}} Θ (π) (U_{π})]$

Now, these $U_{π} (π \cdot ∙)$ always make an acceptable substitute for the $f_{π}$ functions early on, as they're always lower than $f_{π}$ regardless of copolicy. Further, if you're attempting to attain the supremum, you'll never actually use your $f_{π}$ , it will always be $U_{π}$ being plugged in. So, we can just wrap up all the maximization into one supremum, to make

$= {sup}_{ζ, U_{π}, f} E_{π \sim ζ} [Θ (π) (U_{π})]$

We'll be using this phrasing as a more explicit unpacking of $E (f)$ .

Now, for the infradistribution conditions on $E$ , and the causality condition, there's a very important technicality going on. Intersection behaves much worse for the $[0, 1]$ type signature. In the $R$ type signature, everything works out perfectly. However, in the $[0, 1]$ type signature, we'll be able to show all the needed properties for $E$ except having a finite Lipschitz constant. But, not all is lost, because to compensate for that, we'll show that you can swap out $E$ (almost an infradistribution) for an actual infradistribution $E^{'}$ that induces the same belief function $Θ$ .

Like we said back in the post, it isn't terribly important that we use the true maximal representative $E$ , we just want some infradistribution in the same equivalence class.

T5.1.2.2 First up, monotonicity. Assume $f^{'} \geq f$ . Then, using our rewrite of $E (f^{'})$ and $E (f)$ , we have

$E (f^{'}) = {sup}_{ζ, U_{π}, f^{'}} E_{π \sim ζ} [Θ (π) (U_{π})] \geq {sup}_{ζ, U_{π}, f} E_{π \sim ζ} [Θ (π) (U_{π})] = E (f)$

and monotonicity is shown. This occurred because we're taking the supremum over probability distributions and choices of utility function where the mix underperforms $f^{'}$ as an upper bound. Since $f$ is lower, we're taking the supremum over fewer options.

T5.1.2.3 Now, concavity. We'll work in reverse for this.

$p E (f) + (1 - p) E (f^{'}) = p {sup}_{ζ, U_{π_{1}}, f} E_{π \sim ζ} [Θ (π) (U_{π_{1}})] + (1 - p) {sup}_{ζ^{'}, U_{π_{2}}, f^{'}} E_{π \sim ζ^{'}} [Θ (π) (U_{π_{2}})]$

Let's regard the supremum as being attained (or coming arbitrarily close to being attained) by some particular choice of $ζ, ζ^{'}$ , and families $U_{π_{1}}$ and $U_{π_{2}}$ (which are upper-bounded by $f, f^{'}$ respectively) to get

$= p E_{π \sim ζ} [Θ (π) (U_{π_{1}})] + (1 - p) E_{π \sim ζ^{'}} [Θ (π) (U_{π_{2}})]$

We can rewrite this as an integral to get

$= p \int_{Π} (λ π . Θ (π) (U_{π_{1}})) d ζ + (1 - p) \int_{Π} (λ π . Θ (π) (U_{π_{2}})) d ζ^{'}$

and then move the $p$ and $(1 - p)$ in to get integration w.r.t. a measure.

$= \int_{Π} (λ π . Θ (π) (U_{π_{1}})) d (p ζ) + \int_{Π} (λ π . Θ (π) (U_{π_{2}})) d ((1 - p) ζ^{'})$

And then, since $p ζ$ and $(1 - p) ζ^{'}$ are both absolutely continuous w.r.t. $p ζ + (1 - p) ζ^{'}$ , we can rewrite both of these integrals w.r.t the Radon-Nikodym derivative, as

$= \int_{Π} (λ π . Θ (π) (U_{π_{1}}) \cdot \frac{d (p ζ)}{d (p ζ + (1 - p) ζ^{'})} (π)) d (p ζ + (1 - p) ζ^{'})$
$+ \int_{Π} (λ π . Θ (π) (U_{π_{2}}) \cdot \frac{d ((1 - p) ζ^{'})}{d (p ζ + (1 - p) ζ^{'})} (π)) d (p ζ + (1 - p) ζ^{'})$

and pack these up into one integral as the probability distribution we're integrating over is the same in both cases,

$= \int_{Π} (λ π . \frac{d (p ζ)}{d (p ζ + (1 - p) ζ^{'})} (π) \cdot Θ (π) (U_{π_{1}}) + \frac{d ((1 - p) ζ^{'})}{d (p ζ + (1 - p) ζ^{'})} (π) \cdot Θ (π) (U_{π_{2}}))$
$d (p ζ + (1 - p) ζ^{'})$

And then, because

$\frac{d (p ζ)}{d (p ζ + (1 - p) ζ^{'})} (π) + \frac{d ((1 - p) ζ^{'})}{d (p ζ + (1 - p) ζ^{'})} (π) = 1$

(well, technically, this may not hold, but only on a set of measure 0 relative to $p ζ + (1 - p) ζ^{'}$ , so we can fiddle with things a bit so this always holds), we can apply concavity of $Θ (π)$ because it's an inframeasure, for all $π$ , and get

$\leq \int_{Π} (λ π . Θ (π) (\frac{d (p ζ)}{d (p ζ + (1 - p) ζ^{'})} (π) \cdot U_{π_{1}} + \frac{d ((1 - p) ζ^{'})}{d (p ζ + (1 - p) ζ^{'})} (π) \cdot U_{π_{2}}))$
$d (p ζ + (1 - p) ζ^{'})$

and then pack it back up as an expectation, to get

$= E_{π \sim p ζ + (1 - p) ζ^{'}} [Θ (π) (\frac{d (p ζ)}{d (p ζ + (1 - p) ζ^{'})} (π) \cdot U_{π_{1}} + \frac{d ((1 - p) ζ^{'})}{d (p ζ + (1 - p) ζ^{'})} (π) \cdot U_{π_{2}})]$

We will now attempt to show that, for all copolicies $e$ ,

$E_{π \sim p ζ + (1 - p) ζ^{'}} [(\frac{d (p ζ)}{d (p ζ + (1 - p) ζ^{'})} (π) \cdot U_{π_{1}} + \frac{d ((1 - p) ζ^{'})}{d (p ζ + (1 - p) ζ^{'})} (π) \cdot U_{π_{2}}) (π \cdot e)]$
$\leq (p f + (1 - p) f^{'}) (e)$

We can take that big expectation and write it as

$= E_{π \sim p ζ + (1 - p) ζ^{'}} [\frac{d (p ζ)}{d (p ζ + (1 - p) ζ^{'})} (π) \cdot U_{π_{1}} (π \cdot e) + \frac{d ((1 - p) ζ^{'})}{d (p ζ + (1 - p) ζ^{'})} (π) \cdot U_{π_{2}} (π \cdot e)]$

and then write it as an integral

$= \int_{Π} (λ π . \frac{d (p ζ)}{d (p ζ + (1 - p) ζ^{'})} (π) \cdot U_{π_{1}} (π \cdot e) + \frac{d ((1 - p) ζ^{'})}{d (p ζ + (1 - p) ζ^{'})} (π) \cdot U_{π_{2}} (π \cdot e))$
$d (p ζ + (1 - p) ζ^{'})$

Split it up into two integrals

$= \int_{Π} (λ π . U_{π_{1}} (π \cdot e) \cdot \frac{d (p ζ)}{d (p ζ + (1 - p) ζ^{'})} (π)) d (p ζ + (1 - p) ζ^{'})$
$+ \int_{Π} (λ π . U_{π_{2}} (π \cdot e) \cdot \frac{d ((1 - p) ζ^{'})}{d (p ζ + (1 - p) ζ^{'})} (π)) d (p ζ + (1 - p) ζ^{'})$

The Radon-Nikodym derivatives cancel to leave you with

$= \int_{Π} (λ π . U_{π_{1}} (π \cdot e)) d (p ζ) + \int_{Π} (λ π . U_{π_{2}} (π \cdot e)) d ((1 - p) ζ^{'})$

and then pull out the constants to get

$= p \int_{Π} (λ π . U_{π_{1}} (π \cdot e)) d ζ + (1 - p) \int_{Π} (λ π . U_{π_{2}} (π \cdot e)) d ζ^{'}$

pack up as an expectation

$= p E_{π \sim ζ} [U_{π_{1}} (π \cdot e)] + (1 - p) E_{π \sim ζ^{'}} [U_{π_{2}} (π \cdot e)]$

And use that

$E_{π \sim ζ} [U_{π_{1}} (π \cdot e)] \leq f (e)$

$E_{π \sim ζ^{'}} [U_{π_{2}} (π \cdot e)] \leq f^{'} (e)$

to get an upper bound of

$\leq (p f + (1 - p) f^{'}) (e)$

Ok, we're done. We now know that $p ζ + (1 - p) ζ^{'}$ is a probability distribution (mix of two probability distributions). We know that the expectation w.r.t. that distribution of the functions

$λ e . (\frac{d (p ζ)}{d (p ζ + (1 - p) ζ^{'})} (π) \cdot U_{π_{1}} + \frac{d ((1 - p) ζ^{'})}{d (p ζ + (1 - p) ζ^{'})} (π) \cdot U_{π_{2}}) (π \cdot e)$

undershoots $p f + (1 - p) f^{'}$ . And since this is a mix of two existing utility functions, it'll be in $[0, 1]$ if the original utility functions were in that range already (or bounded if the originals were). Taking stock of what we've done so far, we've shown that

$p E (f) + (1 - p) E (f^{'}) \leq E_{π \sim p ζ + (1 - p) ζ^{'}} [Θ (π) (\frac{d (p ζ)}{d (p ζ + (1 - p) ζ^{'})} (π) \cdot U_{π_{1}} + \frac{d ((1 - p) ζ^{'})}{d (p ζ + (1 - p) ζ^{'})} (π) \cdot U_{π_{2}})]$

but the probability distribution, and functions used, ends up mixing to undershoot $p f + (1 - p) f^{'}$ . So we can impose an upper bound of:

$\leq {sup}_{ζ^{''}, U_{π_{3}}, p f + (1 - p) f^{'}} E_{π \sim ζ^{''}} [Θ (π) (U_{π_{3}})]$

which then packs up to equal

$= E (p f + (1 - p) f^{'})$

and we're done, concavity has been shown.

T5.1.2.4 The compact-almost-support property on $E$ is trivial since $E$ is a compact space.

T5.1.2.5 For normalization, we'll need to do it in two parts, for both 0 and 1.

T5.1.2.5.1 First, we show normalization for 1, which requires showing both directions of the inequality. We start with

$E (1) = {sup}_{ζ, U_{π}, 1} E_{π \sim ζ} [Θ (π) (U_{π})]$

and a possible choice that the supremum ranges over is the dirac-delta distribution on the $π^{'}$ with the worst value of $Θ (π) (1)$ (which is 1, by normalization for belief functions), and all the $U_{π}$ are 1. So then we get

$\geq Θ (π^{'}) (1) = {inf}_{π} Θ (π) (1) = 1$

A lower bound of 1. For getting an upper bound, we have

$E (1) = {sup}_{ζ, U_{π}, 1} E_{π \sim ζ} [Θ (π) (U_{π})]$

and then, the supremum ranges over $ζ, U_{π}$ where $\forall e : E_{π \sim ζ} [U_{π} (π \cdot e)] \leq 1$

Now, 1 can be written as $U_{1} (π^{'} \cdot e)$ where $U_{1}$ is the utility function which maps everything to 1, and $π^{'}$ is the policy with the worst value of $Θ (π) (1)$ , and we can apply our alternate formulation of causality for $Θ$ to guarantee that

$E_{π \sim ζ} [Θ (π) (U_{π})] \leq Θ (π^{'}) (U_{1})$

regardless of our choice of $ζ, U_{π}$ . And so, we can impose an upper-bound of

$\leq Θ (π^{'}) (U_{1}) = Θ (π^{'}) (1) = {inf}_{π} Θ (π) (1) = 1$

and we're done, we've got inequalities going both way, so $E (1) = 1$ .

T5.1.2.5.2 Now for 0. We have

$E (0) = {sup}_{ζ, U_{π}, 0} E_{π \sim ζ} [Θ (π) (U_{π})]$

and one thing that the supremum ranges over is the dirac-delta on $π^{'}$ which attains the worst value of $Θ (π) (0)$ , and all the $U_{π}$ being 0, so we get

$\geq Θ (π^{'}) (0) = {inf}_{π} Θ (π) (0) = 0$

and we have a lower-bound of 0. For an upper bound,

$E (0) = {sup}_{ζ, U_{π}, 0} E_{π \sim ζ} [Θ (π) (U_{π})]$

and then, the supremum ranges over $ζ, U_{π}$ where $\forall e : E_{π \sim ζ} [U_{π} (π \cdot e)] \leq 0$

Now, 1 can be written as $U_{0} (π^{'} \cdot e)$ where $U_{0}$ is the utility function which maps everything to 1, and $π^{'}$ is the policy with the worst value of $Θ (π) (0)$ , and we can apply our alternate formulation of causality for $Θ$ to guarantee that

$E_{π \sim ζ} [Θ (π) (U_{π})] \leq Θ (π^{'}) (U_{0})$

regardless of our choice of $ζ, U_{π}$ . And so, we can impose an upper-bound of

$\leq Θ (π^{'}) (U_{0}) = Θ (π^{'}) (0) = {inf}_{π} Θ (π) (0) = 0$

and we're done, we've got inequalities going both way, so $E (0) = 0$ .

T5.1.2.6 Now for Lipschitzness, which we'll need to split in two for the two type signatures.

T5.1.2.6.1 This will be completely provable for the $R$ type signature on infradistributions. What we'll do is show that for any two functions $f, f^{'}$ , when $λ^{⊙}$ is the upper bound on the Lipschitz constant for the belief function $Θ$ , we have

$E (f) - E (inf (f, f^{'})) \leq λ^{⊙} d (f, f^{'})$

Since both $f, f^{'}$ must be above $inf (f, f^{'})$ , by monotonicity, this result would show that

$| E (f) - E (f^{'}) | \leq λ^{⊙} d (f, f^{'})$

and then we'd be done, that's Lipschitzness. So let's get working. We unpack as

$E (f) - E (inf (f, f^{'})) = {sup}_{ζ, U_{π_{1}}, f} E_{π \sim ζ} [Θ (π) (U_{π_{1}})] - {sup}_{ζ^{'}, U_{π_{2}}, inf (f, f^{'})} E_{π \sim ζ} [Θ (π) (U_{π_{2}})]$

Consider the supremum of the first term to be attained by some choice of $ζ, U_{π_{1}}$ fulfilling the property that $\forall e : E_{π \sim ζ} [U_{π_{1}} (π \cdot e)] \leq f (e)$ and then we can rewrite as

$= E_{π \sim ζ} [Θ (π) (U_{π_{1}})] - {sup}_{ζ^{'}, U_{π_{2}}, inf (f, f^{'})} E_{π \sim ζ} [Θ (π) (U_{π_{2}})]$

Now, let your choice of $ζ^{'}$ be $ζ$ , and your choice of the $U_{π_{2}}$ family be $U_{π_{1}} - d (f, f^{'})$ . We do need to show that

$\forall e : E_{π \sim ζ} [U_{π_{1}} (π \cdot e) - d (f, f^{'})] \leq inf (f, f^{'}) (e)$

This is easy, we can fix an arbitrary $e$ , and go

$E_{π \sim ζ} [U_{π_{1}} (π \cdot e) - d (f, f^{'})] = E_{π \sim ζ} [U_{π_{1}} (π \cdot e)] - d (f, f^{'}) \leq f (e) - d (f, f^{'}) \leq inf (f, f^{'}) (e)$

So, this choice of $ζ^{'}$ and $U_{π_{2}}$ works out. Now we can go

$E_{π \sim ζ} [Θ (π) (U_{π_{1}})] - {sup}_{ζ^{'}, U_{π_{2}}, inf (f, f^{'})} E_{π \sim ζ} [Θ (π) (U_{π_{2}})]$

$\leq E_{π \sim ζ} [Θ (π) (U_{π_{1}})] - E_{π \sim ζ} [Θ (π) (U_{π_{1}} - d (f, f^{'}))]$

And then, by monotonicity and a shared Lipschitz constant for the $Θ (π)$ , we have

$\leq E_{π \sim ζ} [Θ (π) (U_{π_{1}})] - E_{π \sim ζ} [Θ (π) (U_{π_{1}}) - λ^{⊙} d (f, f^{'})]$

$= E_{π \sim ζ} [Θ (π) (U_{π_{1}})] - E_{π \sim ζ} [Θ (π) (U_{π_{1}})] + λ^{⊙} d (f, f^{'}) = λ^{⊙} d (f, f^{'})$

And we've shown what we came here to show,

$E (f) - E (inf (f, f^{'})) \leq λ^{⊙} d (f, f^{'})$

$f, f^{'}$ were arbitrary, so by monotonicity for $E$ , we have

$| E (f) - E (f^{'}) | \leq λ^{⊙} d (f, f^{'})$

and $E$ has a finite Lipschitz constant for the $R$ type signature.

T5.1.2.6.2 Now for the fiddly bits with the $[0, 1]$ type signature. The reason the above proof doesn't work for that type signature is that all our functions have to remain in $[0, 1]$ , and it's possible that subtracting $d (f, f^{'})$ from all the $U_{π}$ results in functions that go outside of $[0, 1]$ , and it's way way harder than it looks to patch this.

We have all the other properties, though. What unbounded Lipschitz constant for $E$ (as an expectation functional) looks like from the set point of view is that we've got minimal points in the set of a-measures over $E$ (our $E$ ) with unboundedly large measure components. However, all the $Θ (π)$ infradistribution sets have a bounded Lipschitz constant of $λ^{⊙}$ , all their minimal points can have only $λ^{⊙}$ measure at most.

If you took your set $E$ , and chucked out all the minimal points with measure greater than $λ^{⊙}$ and built up an infradistribution $E^{'}$ from the remaining minimal points, $E^{'}$ would produce $Θ$ again if you went back to a belief function. Why?

Well, the minimal points of an inframeasure/distribution are the truly critical part of them. As long as two sets of a-measures agree on which a-measures are minimal points, they're completely 100 percent equivalent in all respects. We travel back to a belief function $Θ (π)$ via applying the pushforward $(π \cdot)_{*} (E)$ . Doing this pushforward, in the set view keeps all the measure components of a-measures the same. So, chucking out all the a-measures in $E$ with too much measure doesn't throw out any a-measures in $E$ which are necessary to produce a minimal point in some $Θ (π)$ (all of those have $λ^{⊙}$ measure or less). When we apply the pushforward to $E^{'}$ instead of $E$ , we get a smaller set, sure, but the set perfectly agrees with the pushforward of $E$ on the minimal points, so the sets they produce are equivalent. You get the same belief function back.

T5.1.2.7 Now for the last part, showing that we get the causality condition on $E$ . Using our rewrite of $E$ , we have

$E (f) = {sup}_{ζ, U_{π}, f} E_{π \sim ζ} [(Θ (π)) (U_{π})]$

and then, by causality for $Θ$ , we get

$= {sup}_{ζ, U_{π}, f} E_{π \sim ζ} [{sup}_{ζ_{π}^{'}, U_{π_{π}^{'}}, U_{π} (π \cdot ∙)} E_{π^{'} \sim ζ_{π}^{'}} [(Θ (π^{'})) (U_{π_{π}^{'}})]]$

Note that there's a dependence of the second probability distribution and choice of utility functions on the the $π$ selected, because we have a separate $sup$ for each $π$ . Since our rewrite of $E (f)$ was

$E (f) = {sup}_{ζ, U_{π}, f} E_{π \sim ζ} [(Θ (π)) (U_{π})]$

we can abbreviate the interior of the big equation we have to get

$= {sup}_{ζ, U_{π}, f} E_{π \sim ζ} [E (λ e . U_{π} (π \cdot e))]$

Which is the causality condition for $E$ .

T5.1.3 For 3-static to 3-dynamic, it's easy, since $ψ_{T} := E$ so it can just copy over the relevant causality property from $E$ , and we specified that the transition dynamics are exactly what we need, $\to_{T} (e) := ⊤_{A} ⋉ (λ a . δ_{e (a), e_{a, e (a)}})$

T5.1.4 For 3-dynamic to 3-static, this will be tricky, so first we set up some preliminaries. The translation process is unrolling the action-observation sequence to get an infradistribution over destinies, and then turning that into a belief function, and then turning that back into a set of copolicies. Ie,

$E := ⋀_{π} (π \cdot)^{*} (u_{\sim π}^{1} (p r_{(A \times O)^{ω} *} (ψ_{T} ⋉ \to_{T : \infty})))$

We can save on some work. We recently established that going from a causal belief function to the space of copolicies produces an infradistribution over copolicies with the causality condition. So, actually, all we need to do is show that doing the first two steps (and skipping the third) makes a causal belief function, and we'll be done.

This is essentially going from third-person dynamic to third-person static (in destiny form), to first-person static, so we should expect it to be a bit hard. So our goal is that

$Θ (π) := u_{\sim π}^{1} (p r_{(A \times O)^{ω} *} (ψ_{T} ⋉ \to_{T : \infty}))$

ends up making a causal belief function.

Obviously, we need to show the causality property for $Θ$ . However, for all other belief function conditions, we get them for free as long as we have an actual infradistribution over histories (this can be shown by taking the arguments that you get a belief function from translating over an infradistribution back in the Acausal Square Theorem, and adapting them to the pseudocausal case, which is pretty easy). So, actually, all we need to show is that $Θ$ is causal, and that $p r_{(A \times O)^{ω} *} (ψ_{T} ⋉ \to_{T : \infty})$ is an infradistribution over histories, assuming that $ψ_{T}$ is an infradistribution over copolicies.

So, the proof splits into four parts where we verify the niceness conditions on $\to_{T}$ in order to show that the above line is an infradistribution over histories, and one part where we verify causality for $Θ$

First up, verifying the niceness conditions for $\to_{T}$ so $\to_{T : \infty}$ is well-defined.

T5.1.4.1 We'll start with lower-semicontinuity, though actually, we'll be going for something stronger, continuity.

${lim}_{n \to \infty} \to_{T} (e^{n}) (f) = {lim}_{n \to \infty} (⊤_{A} ⋉ (λ a . δ_{e^{n} (a), e_{a, e^{n} (a)}^{n}})) (λ a, o, e . f (a, o, e))$

and then unpack the semidirect product and subsitute the dirac-delta values in to get

$= {lim}_{n \to \infty} ⊤_{A} (λ a . f (a, e^{n} (a), e_{a, e^{n} (a)}^{n}))$

and unpack what $⊤_{A}$ means to get

$= {lim}_{n \to \infty} {inf}_{a} f (a, e^{n} (a), e_{a, e^{n} (a)}^{n})$

There are finitely many actions, so we can shift the limit in, to get

$= {inf}_{a} {lim}_{n \to \infty} f (a, e^{n} (a), e_{a, e^{n} (a)}^{n})$

Now, as $e^{n}$ approaches $e^{\infty}$ , the observation returned for the first action stabilizes, so we get

$= {inf}_{a} {lim}_{n \to \infty} f (a, e^{\infty} (a), e_{a, e^{\infty} (a)}^{n})$

And then, due to continuity of $f$ , and $e_{a, e^{\infty} (a)}^{n}$ converging to $e_{a, e^{\infty} (a)}^{\infty}$ because $e^{n}$ converges to $e^{\infty}$ , we have

$= {inf}_{a} f (a, e^{\infty} (a), e_{a, e^{\infty} (a)}^{\infty})$

Which then packs up in reverse as

$= ⊤_{A} (λ a . f (a, e^{\infty} (a), e_{a, e^{\infty} (a)}^{\infty}))$

$= (⊤_{A} ⋉ (λ a . δ_{e^{\infty} (a), e_{a, e^{\infty} (a)}^{\infty}})) (λ a, o, e . f (a, o, e)) = \to_{T} (e^{\infty}) (f)$

And continuity has been shown. This inducts up to show continuity for $\to_{T : \infty}$ .

T5.1.4.2 1-Lipschitzness holds because $\to_{T}$ is the semidirect product of a crisp infradistribution ( $⊤_{A}$ ) (and all crisp infradistributions are 1-Lipschitz) with a crisp infrakernel (also 1-Lipschitz), and the semidirect product of a 1-Lipschitz infrakernel and a 1-Lipschitz infradistribution is 1-Lipschitz.

T5.1.4.3 Compact-shared almost-compact-support is trivial because $A \times O \times E$ is a compact space.

T5.1.4.4 For "constants increase", actually, crisp infrakernels map constants to the same constant. So we get a stronger result, that $\to_{T} (e) (c) = c$ , and in particular, $\to_{T : \infty} (e) (c) = c$ (the property inducts up appropriately). This also has the implication that all the $\to_{T : \infty} (e)$ are infradistributions, as they map 0 to 0 and 1 to 1. This also bags the last condition, 1-normalization. That's all niceness conditions, so $\to_{T : \infty}$ is well-defined.

Now that we know it's well-defined at all, our goal is to show that

$p r_{(A \times O)^{ω} *} (ψ_{T} ⋉ \to_{T : \infty})$

is an infradistribution. This is, again, easy. The semidirect product of an infradistribution and a continuous infrakernel that returns infradistributions is an infradistribution, and the projection of an infradistribution is an infradistribution.

T5.1.4.5 With that, all that remains is showing that $Θ$ , defined as

$Θ (π^{'}) := u_{\sim π^{'}}^{1} (p r_{(A \times O)^{ω} *} (ψ_{T} ⋉ \to_{T : \infty}))$

fulfills the causality property. We can rewrite the update and projection to get

$Θ (π^{'}) (U) = (ψ_{T} ⋉ \to_{T : \infty}) (λ e_{0}, a o e_{1 : \infty} . 1_{a o_{1 : \infty} \sim π^{'}} U (a o_{1 : \infty}) + 1_{a o_{1 : \infty} ≁ π^{'}})$

and rewrite the semidirect product

$= ψ_{T} (λ e_{0} . \to_{T : \infty} (e_{0}) (λ a o e_{1 : \infty} . 1_{a o_{1 : \infty} \sim π^{'}} U (a o_{1 : \infty}) + 1_{a o_{1 : \infty} ≁ π^{'}}))$

and then write this as a projection,

$= ψ_{T} (λ e_{0} . p r_{(A \times O)^{ω} *} (\to_{T : \infty} (e_{0})) (λ a o_{1 : \infty} . 1_{a o_{1 : \infty} \sim π^{'}} U (a o_{1 : \infty}) + 1_{a o_{1 : \infty} ≁ π^{'}}))$

and then write this as an update

$= ψ_{T} (λ e_{0} . u_{\sim π^{'}}^{1} (p r_{(A \times O)^{ω} *} (\to_{T : \infty} (e_{0}))) (λ a o_{1 : \infty} . U (a o_{1 : \infty})))$

Now, we should notice something really important here. $p r_{(A \times O)^{ω} *} (\to_{T : \infty} (e_{0}))$ , because $\to_{T}$ has complete ignorance over the actions, but has the observations and next copolicies determined by interacting with $e_{0}$ , is the infradistribution generated by all probabilistic mixes of histories capable of being produced by $e_{0}$ . We can think of this as being generated by dirac-delta distributions on all the histories compatible with the copolicy $e_{0}$ .

Updating a dirac-delta on a history on "is the history compatible with $π^{'}$ ?" will, if it isn't compatible, make the $(0, 1)$ a-measure present in all infradistributions (cannibalize the measure into the b term), (or delete the point entirely for the $R$ type signature) and if it is a dirac-delta distribution on a compatible history, leaves it alone. However, there's only one history compatible with $π^{'}$ and $e_{0}$ , it's $δ_{π^{'} \cdot e_{0}}$ , the history made from the two interacting. Thus,

$u_{\sim π^{'}}^{1} (p r_{(A \times O)^{ω} *} (\to_{T : \infty} (e_{0}))) = δ_{π^{'} \cdot e_{0}}$

Making this substitution, we get

$= ψ_{T} (λ e_{0} . δ_{π^{'} \cdot e_{0}} (λ a o_{1 : \infty} . U (a o_{1 : \infty})))$

And putting the value in the function, we get

$= ψ_{T} (λ e_{0} . U (π^{'} \cdot e_{0}))$

Now we can apply the causality condition our $ψ_{T}$ has, yielding

$= {sup}_{ζ, U_{π}, U (π^{'} \cdot ∙)} E_{π \sim ζ} [ψ_{T} (λ e_{0} . U_{π} (π \cdot e_{0}))]$

and go in reverse, breaking this out into a dirac-delta to get

$= {sup}_{ζ, U_{π}, U (π^{'} \cdot ∙)} E_{π \sim ζ} [ψ_{T} (λ e_{0} . δ_{π \cdot e_{0}} (U_{π}))]$

And rewriting this as an update on the projection of the semidirect product initialized on a particular copolicy, yielding

$= {sup}_{ζ, U_{π}, U (π^{'} \cdot ∙)} E_{π \sim ζ} [ψ_{T} (λ e_{0} . u_{\sim π}^{1} (p r_{(A \times O)^{ω} *} (\to_{T : \infty} (e_{0}))) (λ a o_{1 : \infty} . U_{π} (a o_{1 : \infty})))]$

And expand the update

$= {sup}_{ζ, U_{π}, U (π^{'} \cdot ∙)} E_{π \sim ζ} [ψ_{T} (λ e_{0} . p r_{(A \times O)^{ω} *} (\to_{T : \infty} (e_{0})) (λ a o_{1 : \infty} . 1_{a o_{1 : \infty} \sim π} U_{π} (a o_{1 : \infty}) + 1_{a o_{1 : \infty} ≁ π}))]$

And remove the projection

$= {sup}_{ζ, U_{π}, U (π^{'} \cdot ∙)} E_{π \sim ζ} [ψ_{T} (λ e_{0} . \to_{T : \infty} (e_{0}) (λ a o e_{1 : \infty} . 1_{a o_{1 : \infty} \sim π} U_{π} (a o_{1 : \infty}) + 1_{a o_{1 : \infty} ≁ π}))]$

And pack up as a semidirect product and projection

$= {sup}_{ζ, U_{π}, U (π^{'} \cdot ∙)} E_{π \sim ζ} [p r_{(A \times O)^{ω} *} (ψ_{T} ⋉ \to_{T : \infty}) (λ a o_{1 : \infty} . 1_{a o_{1 : \infty} \sim π} U_{π} (a o_{1 : \infty}) + 1_{a o_{1 : \infty} ≁ π})]$

and pack up the update

$= {sup}_{ζ, U_{π}, U (π^{'} \cdot ∙)} E_{π \sim ζ} [u_{\sim π}^{1} (p r_{(A \times O)^{ω} *} (ψ_{T} ⋉ \to_{T : \infty})) (λ a o_{1 : \infty} . U_{π} (a o_{1 : \infty}))]$

and do just one more rewrite to crunch down the function a little bit

$= {sup}_{ζ, U_{π}, U (π^{'} \cdot ∙)} E_{π \sim ζ} [u_{\sim π}^{1} (p r_{(A \times O)^{ω} *} (ψ_{T} ⋉ \to_{T : \infty})) (U_{π})]$

and pack this up into a belief function like how we defined it, to get

$= {sup}_{ζ, U_{π}, U (π^{'} \cdot ∙)} E_{π \sim ζ} [Θ (π) (U_{π})]$

and we're done, we made a causal belief function, and we know that going further from that to the infradistribution on copolicies endows that with the causality condition.

T5.1.5 3-dynamic to 1-dynamic. We don't have to worry about the starting infradistributions, as they're identical to each other, so the relevant causality condition is inherited, we just have to check that the infrakernels transform appropriately. We have

$\to_{F} (e, a) := p r_{O \times E *} (u_{{a}}^{1} (\to_{T} (e)))$

For any action a, applying our definition, we have

$\to_{F} (e, a) (f) = p r_{O \times E *} (u_{{a}}^{1} (\to_{T} (e))) (f)$

Unpacking the projection and update, we have

$= \to_{T} (e) (λ a^{'}, o^{'}, e^{'} . 1_{a^{'} = a} f (o^{'}, e^{'}) + 1_{a^{'} \neq a})$

And then we consider that $\to_{T} (e) = ⊤_{A} ⋉ (λ a . δ_{e (a), e_{a, e (a)}})$ , so we have

$= (⊤_{A} ⋉ (λ a . δ_{e (a), e_{a, e (a)}})) (λ a^{'}, o^{'}, e^{'} . 1_{a^{'} = a} f (o^{'}, e^{'}) + 1_{a^{'} \neq a})$

Unpacking the semidirect product, we have

$= ⊤_{A} (λ a^{'} . δ_{e (a), e_{a, e (a)}} (λ o^{'}, e^{'} . 1_{a^{'} = a} f (o^{'}, e^{'}) + 1_{a^{'} \neq a}))$

and substituting the dirac-delta values in, we have

$= ⊤_{A} (λ a^{'} . 1_{a^{'} = a} f (e (a), e_{a, e (a)}) + 1_{a^{'} \neq a})$

Unpacking what $⊤_{A}$ is, we have

$= {inf}_{a^{'}} 1_{a^{'} = a} f (e (a), e_{a, e (a)}) + 1_{a^{'} \neq a})$

This is 1 or infinity when $a^{'} \neq a$ , so $a^{'} = a$ .

$= f (e (a), e_{a, e (a)})$

And then we just pull out a dirac-delta

$= δ_{e (a), e_{a, e (a)}} (f)$

and we're done! The infrakernel has the right form.

T5.1.6 3-dynamic to 1-dynamic. We know $\to_{T} (e) := ⊤_{A} ⋉ (λ a . \to_{F} (e, a))$

and we know $\to_{F} (e, a) = δ_{e (a), e_{a, e (a)}}$ and we want

$\to_{T} (e) = ⊤_{A} ⋉ (λ a . δ_{e (a), e_{a, e (a)}})$

This is tautological, no problems here.

T5.1.7 1-static to 1-dynamic. Well, we're going 1-static to 1-dynamic via

$ψ_{F} := ⋀_{π} (π \cdot)^{*} (Θ (π))$

So we can just copy the proof of 1-static to 3-static, the translation functions are the same in both cases. The infrakernel also works out appropriately.

T5.1.8 1-dynamic to 1-static. We define our belief function via

$Θ (π^{'}) := p r_{(A \times O)^{ω} *} (ψ_{F} ⋉ \to_{F : \infty}^{π^{'}})$

This proof will be very similar to third-person dynamic to third-person static, but easier. To cut down on a truly aggravating amount of overhead, we invoke our Theorem 4 on pseudocausal IPOMDP's. Our state space is $E$ . We only need to show that $\to_{F}$ fulfills the niceness conditions, and we then know that $Θ$ fulfills all belief function conditions except for maybe normalization.

Our game plan is to show that $\to_{F}$ fulfills the niceness conditions on an infrakernel (four parts, since we can sort of lump 1-normalization in with moving constants up), show that $Θ$ fulfills the normalization property (last thing needed to show it's a belief function that Theorem 4 doesn't get), and then show that $Θ$ is causal, for 6 parts in total.

T5.1.8.1 First niceness condition, lower-semicontinuity. We'll go for continuity instead, it's as easy to show.

${lim}_{n \to \infty} \to_{F} (e^{n}, a^{n}) (f) = {lim}_{n \to \infty} δ_{e^{n} (a^{n}), e_{a^{n}, e^{n} (a^{n})}^{n}} (λ o, e . f (o, e))$

$= {lim}_{n \to \infty} f (e^{n} (a^{n}), e_{a^{n}, e^{n} (a^{n})}^{n})$

We know that $a^{n}$ settles down to $a^{\infty}$ after finite time because the action space is finite, and $e^{n} (a)$ settles down to $e^{\infty} (a)$ for any action because eventually the copolicy settles down on what it does for the initial action, so we get

$= {lim}_{n \to \infty} f (e^{\infty} (a^{\infty}), e_{a^{\infty}, e^{\infty} (a^{\infty})}^{n})$

and, in the limit, because $f$ is continuous, and $e^{n}$ limits to $e^{\infty}$ , we have

$= f (e^{\infty} (a^{\infty}), e_{a^{\infty}, e^{\infty} (a^{\infty})}^{\infty}) = δ_{e^{\infty} (a^{\infty}), e_{a^{\infty}, e^{\infty} (a^{\infty})}^{\infty}} (λ o, e . f (o, e))$

$= \to_{F} (e^{\infty}, a^{\infty}) (f)$

and we're done, continuity is shown.

T5.1.8.2 Compact-shared compact-almost-support is trivial because $O \times E$ is a compact space.

T5.1.8.3,4 1-Lipschitzness, and mapping constants to constants (ie, $\to_{F} (e, a) (c) = c$ ) are trivial because it's a single probability distribution, and those are always 1-Lipschitz and map constants to constants. In particular, 1 gets mapped to 1 so that's the fifth niceness condition taken care of. The infinite infrakernel is well-defined, and our theorem can be invoked.

T5.1.8.5 To get normalization for $Θ$ , we first observe that since all the $\to_{F}$ map constants to the same constant, the same applies to all the $\to_{F, n}^{π}$ , and then to the infinite infrakernel $\to_{F : \infty}^{π}$ . So we have

${inf}_{π} Θ (π) (1) = {inf}_{π} p r_{(A \times O)^{ω} *} (ψ_{F} ⋉ \to_{F : \infty}^{π}) (1)$

$= {inf}_{π} (ψ_{F} ⋉ \to_{F : \infty}^{π}) (1) = {inf}_{π} ψ_{F} (λ e . \to_{F : \infty}^{π} (e) (1))$

and then, since constants are mapped to the same constant, we have

$= {inf}_{π} ψ_{F} (1) = 1$

Where that last step was done by normalization for $ψ_{F}$ since it's an infradistribution and thus normalized. The exact same argument works for 0, showing that the resulting $Θ$ is normalized and thus a belief function.

T5.1.8.6 All that remains is to show causality for $Θ$ . We want to show that

$Θ (π^{'}) (U) = {sup}_{ζ, U_{π}, U (π^{'} \cdot ∙)} E_{π \sim ζ} [Θ (π) (U_{π})]$

To do this, just go

$Θ (π^{'}) (U) = p r_{(A \times O)^{ω} *} (ψ_{F} ⋉ \to_{F : \infty}^{π^{'}}) (λ a o_{1 : \infty} . U (a o_{1 : \infty}))$

and reexpress the projection and semidirect product to get

$= ψ_{F} (λ e_{0} . \to_{F : \infty}^{π^{'}} (e_{0}) (λ a o e_{1 : \infty} . U (a o_{1 : \infty})))$

and wrap this back up as a projection to get

$= ψ_{F} (λ e_{0} . p r_{(A \times O)^{ω} *} (\to_{F : \infty}^{π^{'}} (e_{0})) (λ a o_{1 : \infty} . U (a o_{1 : \infty})))$

And then we observe that $p r_{(A \times O)^{ω} *} (\to_{F : \infty}^{π^{'}} (e_{0}))$ is just $δ_{π^{'} \cdot e_{0}}$ , because the infradistribution is "repeatedly play $π^{'}$ against $e_{0}$ and record the actions and observations you get", so making that substitution we get

$= ψ_{F} (λ e_{0} . δ_{π^{'} \cdot e_{0}} (λ a o_{1 : \infty} . U (a o_{1 : \infty})))$

And then we can substite the dirac-delta in and get

$= ψ_{F} (λ e_{0} . U (π^{'} \cdot e_{0}))$

and apply the causality condition for $ψ_{F}$ to get

$= {sup}_{ζ, U_{π}, U (π^{'} \cdot ∙)} E_{π \sim ζ} [ψ_{F} (λ e_{0} . U_{π} (π \cdot e_{0}))]$

And then we go in reverse, substituting the dirac-delta for $p r_{(A \times O)^{ω} *} (\to_{F : \infty}^{π} (e_{0}))$ and then pulling out projections and expressing as a semidirect product and reexpressing it as a projection to get

$= {sup}_{ζ, U_{π}, U (π^{'} \cdot ∙)} E_{π \sim ζ} [p r_{(A \times O)^{ω} *} (ψ_{F} ⋉ \to_{F : \infty}^{π}) (U_{π})]$

and then pack it back up to get

$= {sup}_{ζ, U_{π}, U (π^{'} \cdot ∙)} E_{π \sim ζ} [Θ (π) (U_{π})]$

and we're done, that's the last of it! Time for isomorphism directions.

T5.2.1 First, 1-static to 3-static to 1-static. We want to show

$E = ⋀_{π} (π \cdot)^{*} ((π \cdot)_{*} (E))$

Breaking down the big intersection, we get

$⋀_{π} (π \cdot)^{*} ((π \cdot)_{*} (E)) (f) = {sup}_{ζ, f_{π}, f} E_{π \sim ζ} [(π \cdot)^{*} ((π \cdot)_{*} (E)) (λ e . f_{π} (e))]$

and breaking down the pullbacks, we get

$= {sup}_{ζ, f_{π}, f} E_{π \sim ζ} [{sup}_{U_{π}, f_{π}} ((π \cdot)_{*} (E)) (λ h . U_{π} (h))]$

and reexpressing the pushforward, we get

$= {sup}_{ζ, f_{π}, f} E_{π \sim ζ} [{sup}_{U_{π}, f_{π}} E (λ e . U_{π} (π \cdot e))]$

Folding the sups into one, we get

$= {sup}_{ζ, U_{π}, f} E_{π \sim ζ} [E (λ e . U_{π} (π \cdot e))]$

Which, by the causality condition for $E$ , is $E (f)$ and we're done! In fact, the causality condition for $E$ is basically just saying that going 1-static to 3-static to 1-static is identity.

T5.2.2 For 3-static to 1-static to 3-static, we want to show that

$Θ (π^{'}) = (π^{'} \cdot)_{*} (⋀_{π} (π \cdot)^{*} (Θ (π)))$

Taking the latter thing

$(π^{'} \cdot)_{*} (⋀_{π} (π \cdot)^{*} (Θ (π))) (U)$

and undoing the pushforward, we get

$= ⋀_{π} (π \cdot)^{*} (Θ (π)) (λ e . U (π^{'} \cdot e))$

And reexpressing the intersection, we get

$= {sup}_{ζ, f_{π}, U (π^{'} \cdot ∙)} E_{π \sim ζ} [(π \cdot)^{*} (Θ (π)) (λ e . f_{π} (e))]$

and reexpressing the pullback, we get

$= {sup}_{ζ, f_{π}, U (π^{'} \cdot ∙)} E_{π \sim ζ} [{sup}_{U_{π}, f_{π}} Θ (π) (λ h . U_{π} (h))]$

folding the sups into one, we get

$= {sup}_{ζ, U_{π}, U (π^{'} \cdot ∙)} E_{π \sim ζ} [Θ (π) (U_{π})]$

And, by causality for $Θ$ , this is just

$= Θ (π^{'}) (U)$

and we're done there.

T5.2.3 For 3-static to 3-dynamic to 3-static, this will be tricky, because third-person dynamic to third-person static is a complicated operation. We want to show that

$E = ⋀_{π} (π \cdot)^{*} (u_{\sim π}^{1} (p r_{(A \times O)^{ω} *} (E ⋉ \to_{T : \infty})))$

So, let's take the big thing and start unpacking it.

$⋀_{π} (π \cdot)^{*} (u_{\sim π}^{1} (p r_{(A \times O)^{ω} *} (E ⋉ \to_{T : \infty}))) (f)$

we undo the big and, to get

$= {sup}_{ζ, f_{π}, f} E_{π \sim ζ} [(π \cdot)^{*} (u_{\sim π}^{1} (p r_{(A \times O)^{ω} *} (E ⋉ \to_{T : \infty}))) (λ e . f_{π} (e))]$

and then reexpress the pullbacks to get

$= {sup}_{ζ, f_{π}, f} E_{π \sim ζ} [{sup}_{U_{π}, f_{π}} (u_{\sim π}^{1} (p r_{(A \times O)^{ω} *} (E ⋉ \to_{T : \infty}))) (λ a o_{1 : \infty} . U_{π} (a o_{1 : \infty}))]$

and then fold the two sups into one to get

$= {sup}_{ζ, U_{π}, f} E_{π \sim ζ} [u_{\sim π}^{1} (p r_{(A \times O)^{ω} *} (E ⋉ \to_{T : \infty})) (λ a o_{1 : \infty} . U_{π} (a o_{1 : \infty}))]$

and then reexpress the update to get

$= {sup}_{ζ, U_{π}, f} E_{π \sim ζ} [p r_{(A \times O)^{ω} *} (E ⋉ \to_{T : \infty}) (λ a o_{1 : \infty} . 1_{a o_{1 : \infty} \sim π} U_{π} (a o_{1 : \infty}) + 1_{a o_{1 : \infty} ≁ π})]$

And then reexpress the projection and semidirect product to get

$= {sup}_{ζ, U_{π}, f} E_{π \sim ζ} [E (λ e_{0} . \to_{T : \infty} (e_{0}) (λ a o e_{1 : \infty} . 1_{a o_{1 : \infty} \sim π} U_{π} (a o_{1 : \infty}) + 1_{a o_{1 : \infty} ≁ π}))]$

And then wrap it back up in a projection to get

$= {sup}_{ζ, U_{π}, f} E_{π \sim ζ} [E (λ e_{0} . p r_{(A \times O)^{ω} *} (\to_{T : \infty} (e_{0})) (λ a o_{1 : \infty} . 1_{a o_{1 : \infty} \sim π} U_{π} (a o_{1 : \infty}) + 1_{a o_{1 : \infty} ≁ π}))]$

And then rewrite this as an update, to get

$= {sup}_{ζ, U_{π}, f} E_{π \sim ζ} [E (λ e_{0} . u_{\sim π}^{1} (p r_{(A \times O)^{ω} *} (\to_{T : \infty} (e_{0}))) (λ a o_{1 : \infty} . U_{π} (a o_{1 : \infty})))]$

And then, by our earlier arguments when we were doing third-person dynamic to third-person static, we showed

$u_{\sim π}^{1} (p r_{(A \times O)^{ω} *} (\to_{T : \infty} (e_{0}))) = δ_{π \cdot e_{0}}$

So, making that substitution, we have

$= {sup}_{ζ, U_{π}, f} E_{π \sim ζ} [E (λ e_{0} . δ_{π \cdot e_{0}} (λ a o_{: \infty} . U_{π} (a o_{: \infty})))]$

Substituting the dirac-delta in, we get

$= {sup}_{ζ, U_{π}, f} E_{π \sim ζ} [E (λ e_{0} . U_{π} (π \cdot e_{0}))]$

Renaming our variables a little bit, we have

$= {sup}_{ζ, U_{π}, f} E_{π \sim ζ} [E (λ e . U_{π} (π \cdot e))]$

and then, by the causality condition for $E$ , this is just $= E (f)$ and we're done.

T5.2.4 For 3-dynamic to 3-static to 3-dynamic, our desired result is

$ψ_{T} = ⋀_{π} (π \cdot)^{*} (u_{\sim π}^{1} (p r_{(A \times O)^{ω} *} (ψ_{T} ⋉ \to_{T : \infty})))$

And the proof of this is the exact same as going third-person static to third-person dynamic and back, just use $ψ_{T}$ instead of $E$ . Also, $\to_{F} (e)$ works out as it should.

T5.2.5,6 For translating back and forth between 3-dynamic and 1-dynamic, we observe that both translations keep the initial infradistribution over destinies the same, and our 3-dynamic to 1-dynamic, and 1-dynamic to 3-dynamic proofs verified that the $\to_{F}$ and $\to_{T}$ behave as they should when they go back and forth, so we don't need to worry about this.

T5.2.7 We're on the last two! For 1-dynamic to 1-static to 1-dynamic, we want to show that

$ψ_{F} = ⋀_{π} (π \cdot)^{*} (p r_{(A \times O)^{ω} *} (ψ_{F} ⋉ \to_{F : \infty}^{π}))$

Taking the big part, we have

$⋀_{π} (π \cdot)^{*} (p r_{(A \times O)^{ω} *} (ψ_{F} ⋉ \to_{F : \infty}^{π})) (f)$

and undoing the big and, and the pullback via interaction with $π$ , and merging the sups together as usual, (skipping a few steps we've seen several times before) we have

$= {sup}_{ζ, U_{π}, f} E_{π \sim ζ} [p r_{(A \times O)^{ω} *} (ψ_{F} ⋉ \to_{F : \infty}^{π}) (λ a o_{1 : \infty} . U_{π} (a o_{1 : \infty}))]$

Undoing the projection, and the semidirect product, we have

$= {sup}_{ζ, U_{π}, f} E_{π \sim ζ} [ψ_{F} (λ e_{0} . \to_{F : \infty}^{π} (e_{0}) (λ a o e_{1 : \infty} . U_{π} (a o_{1 : \infty})))]$

and wrap it up in a projection,

$= {sup}_{ζ, U_{π}, f} E_{π \sim ζ} [ψ_{F} (λ e_{0} . p r_{(A \times O)^{ω} *} (\to_{F : \infty}^{π} (e_{0})) (λ a o_{1 : \infty} . U_{π} (a o_{1 : \infty})))]$

and then, because $p r_{(A \times O)^{ω} *} (\to_{F : \infty}^{π} (e_{0})) = δ_{π \cdot e_{0}}$ because unrolling a policy against a copolicy and recording the actions and observations does that, we have

$= {sup}_{ζ, U_{π}, f} E_{π \sim ζ} [ψ_{F} (λ e_{0} . δ_{π \cdot e_{0}} (λ a o_{1 : \infty} . U_{π} (a o_{1 : \infty})))]$

Then, substituting the value of the dirac-delta in, we get

$= {sup}_{ζ, U_{π}, f} E_{π \sim ζ} [ψ_{F} (λ e_{0} . U_{π} (π \cdot e_{0}))]$

And then, by causality for $ψ_{F}$ , this is just

$= ψ_{F} (f)$

and we're done there.

T5.2.8 Last one, 1-static to 1-dynamic to 1-static. We want to show that

$Θ (π^{'}) = p r_{(A \times O)^{ω} *} ((⋀_{π} (π \cdot)^{*} (Θ (π))) ⋉ \to_{F : \infty}^{π^{'}})$

Starting at the complicated side, we have

$p r_{(A \times O)^{ω} *} ((⋀_{π} (π \cdot)^{*} (Θ (π))) ⋉ \to_{F : \infty}^{π^{'}}) (U)$

and then, undoing the projection,

$= ((⋀_{π} (π \cdot)^{*} (Θ (π))) ⋉ \to_{F : \infty}^{π^{'}}) (λ e_{0}, a o e_{1 : \infty} . U (a o_{1 : \infty}))$

Now, rewriting the semidirect product, we have

$= (⋀_{π} (π \cdot)^{*} (Θ (π))) (λ e_{0} . \to_{F : \infty}^{π^{'}} (e_{0}) (λ a o e_{1 : \infty} . U (a o_{1 : \infty})))$

Rewriting this as a projection, we have

$= (⋀_{π} (π \cdot)^{*} (Θ (π))) (λ e_{0} . p r_{(A \times O)^{ω} *} (\to_{F : \infty}^{π^{'}} (e_{0})) (λ a o_{1 : \infty} . U (a o_{1 : \infty})))$

We use the usual trick of this projection of a semidirect product just being the dirac-delta on the copolicy interacting with the policy, and substituting that in to get

$= (⋀_{π} (π \cdot)^{*} (Θ (π))) (λ e_{0} . U (π^{'} \cdot e_{0}))$

And then we do the usual unpacking of the intersection and the pullbacks to get:

$= {sup}_{ζ, U_{π}, U (π^{'} \cdot ∙)} E_{π \sim ζ} [Θ (π) (U_{π})]$

Which, by causality for $Θ$ , is just

$= Θ (π^{'}) (U)$

and we're done with the isomorphisms.

T5.3 The 2 paths from first-person static to third-person dynamic being equivalent is pretty easy since we've got identity for two of the legs, and the other two legs are just "intersection of preimages of the $Θ (π)$ ", ie identical, and we're done!

AI ALIGNMENT FORUM
AF

AI ALIGNMENT FORUM
AF

3

3