This is a special post for quick takes by Vladimir_Nesov. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.
The physical world is a simple thing that's not useful to study too much, once intelligence scales well past the human level. Almost all cognition is going to be about the computations designed by cognition (or their semantics), not about the physical world, which is likely to be almost entirely relegated to the role of compute substrate for the intentionally designed computations. All of the physical world might one day be about as relevant to most people as the arcane details of material manufacturing processes necessary to make a specialized industrial tool they were never aware of.
So I don't think decision theory for agent foundations should be concerned with the kinds of uncertainty that are central to thinking about the physical world. Instead, it should study reasoning about known computations, and influence of decisions on the behavior of known computations. These are not toy examples that in some ways pose unusual difficulties and in other ways are easier to handle, but rather most of what matters.
2
1
1Unclear. Security of the future compute might benefit from very strong constraints at the substrate level, and observing the physical world in any way isn't necessarily a given. For example, if you want to backup a large swath of a civilization, it might be appropriate to make relatively rare snapshots (sending them to distant locations) if the subsequent (physically damaged) computation could be deterministically replayed in case restoring from the backup was necessary. Distributed backups are the way to make death technologically impossible, even if a galaxy gets blown up by aliens or something, as long as the global civilization that would recover the physically damaged parts from backups persists.
Quantum computing is an interesting case. Possibly it's not good for anything that shouldn't stay at the substrate level, keeping the computations on the inside updateless with respect to observing the results, so that only the outer substrate (and its governance) gets to observe such results. It might be quite useful for dealing with the physical world stuff, therefore not very relevant.
I feel like you're trying to speculate here about what happens after someone already won the war for the lightcone and is a singleton. Sure, there are 'simulation hypothesis' type considerations here that are decision relevant, but it also matters what happens before someone has conquered everything (if that's even likely).
What happens at the intermediate stage where there are smarter-than-human AIs (bootstrapped by centaur systems) and they are using foreseeable quantum tech, which surely includes quantum RNG, and probably includes Bell-nonlocality (already can be done in the lab) and quantum computing (can't yet, but seems foreseeable)? The decision / game theory of this could affect who, if anyone, gets a singleton locking down a bunch of the world, and restricting who has access to which quantum operations.
And what about aliens? They are probably very far away if they exist, but even an Earth-originating singleton needs to plan on the game theory of future alien encounters, and assume aliens have quantum tech.
There is the "try to run away" thing and if that is the plan, why not include a quantum RNG (or other quantum tech, if available) on board of the runaway space ship?
Regarding applications of quantum computing, Grover's algorithm is highly general (though not an exptime speedup), there could be more like that. And the physical world still has important consequences about which computations are important to run, in order to make good use of material / space resources. Quantum computers help with simulating physical systems.
The idea with the substrate/civilization split is not that the substrate owns the civilization, but that the civilization is the updateless core for the substrate, it defines the policies that the substrate follows (like in a UDT agent), without knowing some of the things the substrate gets to observe in the physical world. The civilization governance on the inside could take any form (doesn't need to be a singleton), and the substrate governance on the outside doesn't carry the historically familiar menacing connotations even if it runs the civilization in a perfect sandbox, because one way or another alignment is going to be settled at that point (which was never historically the case, and so direct control was always relevant).
even an Earth-originating singleton needs to plan on the game theory of future alien encounters, and assume aliens have quantum tech
Right, that's a thing for the outer governance to worry about (like that arcane material manufacturing process for the industrial tool), but not necessarily for the civilization.
Ok, to make my opinion on this clearer:
(a) We don't really know much about what the eventual state looks like, whether it is a singleton, how locked down, how many have access to quantum ops.
(b) My personal guess for (a) is that "stochastic Turing machines + Bell nonlocality" is a more relevant model than "deterministic Turing machines", though both are relevant.
(c) How do we figure out about (a)? It seems like we need some theory of agent foundations to think about this well. That theory shouldn't start out with strong assumptions like "only deterministic Turing machines are relevant" or "everyone has access to a quantum RNG". But we should also use our personal guesses on this. So I'm thinking about agent foundations with stochastic TMs partially because of my (b) opinion.
(d) Part of reasoning about (a) also involves reasoning about intermediate states. These intermediate states can be "messy" and therefore forecasting them involves non-agent-foundations reasoning. But agent foundations is still relevant here, since some intermediate states are smarter-than-human, "economically efficient", etc.
(e) I think stochastic TMs are more important for (d) than deterministic TMs, as a model. Sure, both will have some relevance, but quantum RNGs are currently easily available, will become more available in the near future, and have cryptographic applications. Moreover, many RL algorithms, and decision theory (like CDT self-ratification, mixed Nash equilibrium, absent minded driver) assumes access to RNGs, so quantum RNGs are relevant as an idealized case.
(f) Overall these considerations make me think stochastic TMs are a better model for agent foundations than deterministic TMs, but in theory both are relevant. My opinion here seems to significantly disagree with yours.
(g) I think I have significantly more "dispersion" in my beliefs about the eventual state (a) than you do, such that I don't find the specific scenario you're talking about where agents are locked down without access to quantum RNG or Bell-nonlocality very compelling, except as a theoretical possibility that agent foundations should reason about among other possibilities.
My stance is "unclear", this subthread is about why it's unclear, why a weird possibility of determinism and a lack of access to quantum observations/randomness/computation could make sense.
It's not load bearing for the post or how I think about this; even with quantum observations you still end up with intentionally designed computations in a central sense, rather than mere weather and stars. The backups motivation doesn't imply external observations shouldn't happen at all, only that they should be sufficiently old to already be in the distributed backups by the time they get to be observed on the inside. It's a values question whether creating a vulnerable branch determined by non-backed-up observations is a good thing.
Ah, ok. I agree on "unclear" so we disagree less than I thought. I could see an analogous case of: "Maybe in the future, there will be many low-intelligence agents, because there will be some governance system that forces all agents except one to be low-intelligence, for security reasons. We should be extensively studying low-intelligence agents in agent foundations for this reason." And some but not all reasons why this is not a compelling argument for studying low-intelligence agents (in agent foundations) translate to studying "agents with no quantum tech".
The dictum "you are an algorithm" clarifies some issues with other things reasoning about the agent, but deciding what to do depends on prior considerations, things like arguments, norms, contracts, facts, values, and also decision theories with their recommended actions or policies. As a result, asking about other instances of the agent (or other things reasoning about the agent) is less natural than asking about other instances of these prior considerations, or other things reasoning about the same prior considerations. These things are also more plentiful outside the agent than the instances of reasoning about the agent as a whole, establishing better grounding for acausal coordination. For example, true facts about the world are also found in the world itself, as it carries out its behavior according to such facts.