This can easily be done in the cryptographic example above: B can sample a new number $y=p^{\prime }\cdot q^{\prime }$, and then present $y$ to a fresh copy of A that has not seen the transcript for $x$ so far.

I don't understand how this is supposed to help. I guess the point is to somehow catch a fresh copy of A in a lie about a problem that is different from the original problem, and conclude that A is the dishonest debater?

But couldn't A just answer "I don't know"?

Even if it is a fresh copy, it would notice that it does not know the secret factors, so it could display different behavior than in the $x$ case where A knows the secret factors $p,q$.

I think there are some subtleties with the (non-infra) bayesian VNM version, which come down to the difference between "extreme point" and "exposed point" of $D$. If a point is an extreme point that is not an exposed point, then it cannot be the unique expected utility maximizer under a utility function (but it can be a non-unique maximizer).

For extreme points it might still work with uniqueness, if, instead of a VNM-decision-maker, we require a slightly weaker decision maker whose preferences satisfy the VNM axioms except continuity.

For any $\Phi ,\Psi \in \hat{D}$, if ${\Theta }^{\ast }=\Phi \vee \Psi$ then either $\Phi \subseteq \Psi$ or $\Psi \subseteq \Phi$.

I think this condition might be too weak and the conjecture is not true under this definition.

If ${\Phi }_1\subseteq {\Phi }_2$, then we have $E_{y\sim \xi }{min}_{\mu \in {\Phi }_2}{E}_{x\sim \mu }u(x,y)\le E_{y\sim \xi }{min}_{\mu \in {\Phi }_1}{E}_{x\sim \mu }u(x,y)$ (because a minimum over a larger set is smaller). Thus, ${\Phi }_2$ can only be the unique argmax if ${\Phi }_1={\Phi }_2$.

Consider the example $\hat{D}=\left\{\right[0,x]:x\in [0,1\left]\right\}$. Then $\hat{D}$ is closed. And ${\Theta }^{\ast }=[0,1]$ satisfies ${\Theta }^{\ast }=\Phi \vee \Psi ⟹\Phi \subseteq \Psi \vee \Psi \subseteq \Phi$. But per the above it cannot be a unique maximizer.

Maybe the issue can be fixed if we strengthen the condition so that ${\Phi }^{\ast }$ has to be also minimal with respect to $\subseteq$.

Regarding direction 17: There might be some potential drawbacks to ADAM. I think its possible that some very agentic programs have relatively low $g$ score. This is due to explicit optimization algorithms being low complexity.

(Disclaimer: the following argument is not a proof, and appeals to some heuristics/etc. We fix $M=M_0$ for these considerations too.) Consider an utility function $\hat{U}$. Further, consider a computable approximation of the optimal policy (AIXI that explicitly optimizes for $\hat{U}$) and has an approximation parameter n (this could be AIXI-tl, plus some approximation of $\hat{U}$; higher $n$ is better approximation). We will call this approximation of the optimal policy ${\pi }_n^{\hat{U}}$. This approximation algorithm has complexity $K\left({\pi }_n^{\hat{U}}\right)=C+K\left(\hat{U}\right)+K\left(n\right)$, where $C>0$ is a constant needed to describe the general algorithm (this should not be too large).

We can get better approximation by using a quickly growing function, such as the Ackermann function with $n=A(k,k)$. Then we have $K\left({\pi }_{A(k,k)}^{\hat{U}}\right)=C+K\left(\hat{U}\right)+K\left(A\right(k,k\left)\right)\le C+K\left(\hat{U}\right)+\log \left(k\right)$.

What is the $g$ score of this policy? We have $g\left({\pi }_{A(k,k)}^{\hat{U}}\right)={max}_U\left({min}_{{\pi }^{\prime }:\dots }K\right({\pi }^{\prime })-K(U\left)\right)$. Let $\overline{U}$ be maximal in this expression. If $K\left(\overline{U}\right)\ge K\left(\hat{U}\right)-C$, then $$g\left({\pi }_{A(k,k)}^{\hat{U}}\right)=\underset{{\pi }^{\prime }:E_{{\zeta }_{M_0}{\pi }^{\prime }}\left(\overline{U}\right)\ge E_{{\zeta }_{M_0}{\pi }_{A(k,k)}^{\hat{U}}}\left(\overline{U}\right)}{min}K\left({\pi }^{\prime }\right)-K\left(\overline{U}\right)\le K\left({\pi }_{A(k,k)}^{\hat{U}}\right)-K\left(\hat{U}\right)+C\le 2C\log \left(k\right)$$.

For the other case, let us assume that if $K\left(\overline{U}\right)<K\left(\hat{U}\right)-C$, the policy ${\pi }_{A(k,k)}^{\overline{U}}$ is at least as good at maximizing $\overline{U}$ than ${\pi }_{A(k,k)}^{\hat{U})}$. Then, we have $$g\left({\pi }_{A(k,k)}^{\hat{U}}\right)=\underset{{\pi }^{\prime }:E_{{\zeta }_{M_0}{\pi }^{\prime }}\left(\overline{U}\right)\ge E_{{\zeta }_{M_0}{\pi }_{A(k,k)}^{\hat{U}}}\left(\overline{U}\right)}{min}K\left({\pi }^{\prime }\right)-K\left(\overline{U}\right)\le K\left({\pi }_{A(k,k)}^{\overline{U}}\right)-K\left(\overline{U}\right))\le C+\log (k)$$.

I don't think that the assumption ($({\pi }_{A(k,k)}^{\overline{U}}$ maximizes $barU$ better than $({\pi }_{A(k,k)}^{\hat{U}}$) is true for all $\hat{U}$ and $k$, but plausibly we can select $\hat{U}$ such that this is the case (exceptions, if they exist, would be a bit weird, and if ADAM working well due to these weird exceptions feels a bit disappointing to me). A thing that is not captured by approximations such as AIXI-tl are programs that halt but have insane runtime (longer than $A(k,k)$). Again, it would feel weird to me if ADAM sort of works because of low-complexity extremely-long-running halting programs.

To summarize, maybe there exist policies ${\pi }_{A(k,k)}^{\hat{U}}$ which strongly optimize a non-trivial utility function $\hat{U}$ with approximation parameter $n=A(k,k)$, but where $g\left({\pi }_{A(k,k)}^{\hat{U}}\right)\le 2C+\log \left(k\right)$ is relatively small.

I am going to assume that in the code, when calculating p_alice_win_given_not_caught, we do not divide the ${\lambda }_2$ term by two (since this is not that consistent with the description. I am also assuming that $0.25-t_2$ is a typo and $0.5-t_2$ is meant, which would also be more consistent with other stuff). So I am going to assume assume a symmetrical version.

Here, P(Alice wins) is $e^{-{\lambda }_1{t}_1-{\lambda }_2{t}_2}-e^{-{\lambda }_1/2-{\lambda }_2/2}=:f({\lambda }_1,{\lambda }_2,t_1,t_2)$. Wlog we can assume $0<t<1$ (otherwise Bob will run everything or nothing in shielded mode).

We claim that $\left(\right({\lambda }_1^{\ast },{\lambda }_2^{\ast }),(t_1^{\ast },t_2^{\ast }\left)\right):=\left(\right({\mu }^{\ast },{\mu }^{\ast }),(t/2,t/2)$ is a (pure) Nash equilibrium, where ${\mu }^{\ast }=\log t/(t-1)$.

To verify, lets first show that Alice cannot make a better choice if Bob plays $(t_1^{\ast },t_2^{\ast })=(t/2,t/2)$. We have $f({\lambda }_1,{\lambda }_2,t_1^{\ast },t_2^{\ast })=e^{-({\lambda }_1+{\lambda }_2)t/2}-e^{-({\lambda }_1+{\lambda }_2)/2}$. Since this only depends on the sum, we can make the substitution $\mu =({\lambda }_1+{\lambda }_2)/2$. Thus, we want to maximize $\tilde{f}(\mu ,t)=e^{-\mu t}-e^{-\mu }$. We have ${\tilde{f}}^{\prime }(\mu ,t)=-t{e}^{-\mu t}+e^{-\mu }=0$. Rearranging, we get $t{e}^{-\mu t}=e^{-\mu }$. Taking logs, we get $\log t-\mu t=-\mu$. Rearranging, we get $\mu =\log t/(t-1)$. Thus, $\mu ={\mu }^{\ast }$ is the optimal choice. This means, that if Bob sticks to his strategy, Alice cannot do better than $f({\mu }^{\ast },{\mu }^{\ast },t/2,t/2)$.

Now, lets show that Bob cannot do better. We have $f({\lambda }_1^{\ast },{\lambda }_2^{\ast },t_1,t_2)=e^{-{\mu }^{\ast }{t}_1-{\mu }^{\ast }{t}_2}-e^{-{\mu }^{\ast }/2-{\mu }^{\ast }/2}=e^{-{\mu }^{\ast }t}-e^{-{\mu }^{\ast }}$. This does not depend on $t_1$ and $t_2$ anymore, so any choice of $t_1$ and $t_2$ is optimal if Alice plays $({\mu }^{\ast },{\mu }^{\ast })$.

(If I picked the wrong version of the question, and you actually want some symmetry: I suspect that the solution will have similarities, or that in some cases the solution can be obtained by rescaling the problem back into a more symmetric form.)

Nanotech industry-rebuilding comes earlier than von Neumann level? I doubt that. A lot of existing people are close to von Neumann level.

Maybe your argument is that there will be so many AGIs, that they can do Nanotech industry rebuilding while individually being very dumb. But I would then argue that the collective already exceeds von Neumann or large groups of humans in intelligence.