All of harfe's Comments + Replies

This can easily be done in the cryptographic example above: B can sample a new number $y=p^{\prime }\cdot q^{\prime }$, and then present $y$ to a fresh copy of A that has not seen the transcript for $x$ so far.

I don't understand how this is supposed to help. I guess the point is to somehow catch a fresh copy of A in a lie about a problem that is different from the original problem, and conclude that A is the dishonest debater?

But couldn't A just answer "I don't know"?

Even if it is a fresh copy, it would notice that it does not know the secret factors, so it could display different ... (read more)

I think there are some subtleties with the (non-infra) bayesian VNM version, which come down to the difference between "extreme point" and "exposed point" of $D$. If a point is an extreme point that is not an exposed point, then it cannot be the unique expected utility maximizer under a utility function (but it can be a non-unique maximizer).

For extreme points it might still work with uniqueness, if, instead of a VNM-decision-maker, we require a slightly weaker decision maker whose preferences satisfy the VNM axioms except continuity.

For any $\Phi ,\Psi \in \hat{D}$, if ${\Theta }^{\ast }=\Phi \vee \Psi$ then either $\Phi \subseteq \Psi$ or $\Psi \subseteq \Phi$.

I think this condition might be too weak and the conjecture is not true under this definition.

If ${\Phi }_1\subseteq {\Phi }_2$, then we have $E_{y\sim \xi }{min}_{\mu \in {\Phi }_2}{E}_{x\sim \mu }u(x,y)\le E_{y\sim \xi }{min}_{\mu \in {\Phi }_1}{E}_{x\sim \mu }u(x,y)$ (because a minimum over a larger set is smaller). Thus, ${\Phi }_2$ can only be the unique argmax if ${\Phi }_1={\Phi }_2$.

Consider the example $\hat{D}=\left\{\right[0,x]:x\in [0,1\left]\right\}$. Then $\hat{D}$ is closed. And ${\Theta }^{\ast }=[0,1]$ satisfies ${\Theta }^{\ast }=\Phi \vee \Psi ⟹\Phi \subseteq \Psi \vee \Psi \subseteq \Phi$. But per the above it cannot be a unique maximizer.

Maybe the issue can be fixed if we strengthen the condition so that ${\Phi }^{\ast }$ has to be also minimal with res... (read more)

Regarding direction 17: There might be some potential drawbacks to ADAM. I think its possible that some very agentic programs have relatively low $g$ score. This is due to explicit optimization algorithms being low complexity.

(Disclaimer: the following argument is not a proof, and appeals to some heuristics/etc. We fix $M=M_0$ for these considerations too.) Consider an utility function $\hat{U}$. Further, consider a computable approximation of the optimal policy (AIXI that explicitly optimizes for $\hat{U}$) and has an approximation parameter n (this could be AIXI-tl, plus s... (read more)

I am going to assume that in the code, when calculating p_alice_win_given_not_caught, we do not divide the ${\lambda }_2$ term by two (since this is not that consistent with the description. I am also assuming that $0.25-t_2$ is a typo and $0.5-t_2$ is meant, which would also be more consistent with other stuff). So I am going to assume assume a symmetrical version.

Here, P(Alice wins) is $e^{-{\lambda }_1{t}_1-{\lambda }_2{t}_2}-e^{-{\lambda }_1/2-{\lambda }_2/2}=:f({\lambda }_1,{\lambda }_2,t_1,t_2)$. Wlog we can assume $0<t<1$ (otherwise Bob will run everything or nothing in shielded mode).

We claim that $\left(\right({\lambda }_1^{\ast },{\lambda }_2^{\ast }),(t_1^{\ast },t_2^{\ast }\left)\right):=\left(\right({\mu }^{\ast },{\mu }^{\ast }),(t/2,t/$... (read more)

Nanotech industry-rebuilding comes earlier than von Neumann level? I doubt that. A lot of existing people are close to von Neumann level.

Maybe your argument is that there will be so many AGIs, that they can do Nanotech industry rebuilding while individually being very dumb. But I would then argue that the collective already exceeds von Neumann or large groups of humans in intelligence.