IIUC the core of this post is the following:
There are three forces/reasons pushing very long-sighted ambitious agents/computations to make use of very short-sighted, unambitious agents/computations:
1. Schelling points for acausal coordination
2. Epistemology works best if you just myopically focus on answering correctly whatever question is in front of you, rather than e.g. trying to optimize your long-run average correctness or something.
3. Short-sighted, unambitious agents/computations are less of a threat, more easily controlled.
I'll ignore 1 for now. For 2 and 3, I think I understand what you are saying on a vibes/intuitive level, but I don't trust my vibes/intuition enough. I'd like to see 2 and 3 spelled out and justified more rigorously. IIRC there's an academic philosophy literature on 2, but I'm afraid I don't remember it much. Are you familiar with it? As for 3, here's a counterpoint: Yes, more longsighted agents/computations carry with them risk of various kinds of Evil. But, the super myopic ones also have their drawbacks. (Example: Sometimes you'd rather have a bureaucracy staffed with somewhat agentic people who can make exceptions to the rules when needed, than a bureaucracy staffed with apathetic rule-followers.) You haven't really argued that the cost-benefit analysis systematically favors delegating to myopic agents/computations.
Strange attitude towards the physical world can be reframed as caring only about some abstract world that happens to resemble the physical world in some ways. A chess AI could be said to be acting on some specific physical chessboard within the real world and carefully avoiding all concern about everything else, but it's more naturally described as acting on just the abstract chessboard, nothing else. I think values/preference (for some arbitrary agent) should be not just about probutility upon the physical world, but should also specify which world they are talking about, so that different agents are not just normatively disagreeing about relative value of events, but about which worlds are worth caring about (not just possible worlds within some space of nearby possible worlds, but fundamentally very different abstract worlds), and therefore what kinds of events (from which sample spaces) ought to serve as semantics for possible actions, before their value can be considered.
A world model (such as an LLM with frozen weights) is already an abstraction, its data is not the same as the physical world itself, but it's coordinated with the physical world to some extent, similarly to how an abstract chessboard is coordinated with a specific physical chessboard in the real world (learning is coordination, adjusting the model so that the model and the world have more shared explanations for their details). Thus acting within an abstract world given by a world model (as opposed to within the physical world itself) might be a useful framing for systematically ignoring some aspects of the physical world, and world models could be intentionally crafted to emphasize particular aspects.
It's been a while since I wrote about myopia!
My previous posts about myopia were "a little crazy", because it's not this solid well-defined thing; it's a cluster of things which we're trying to form into a research program.
This post will be "more crazy".
"Good" means something along the lines of "helpful to all".
There is a spectrum from extremely myopic to extremely non-myopic. Arranging all "thinking beings" on that spectrum, I claim that you get Good at both ends, with Evil sitting in-between.
Deva (divine): At the extreme non-myopic end, you get things like updateless reasoning, acausal trade, multiverse-wide coordination, and so on. With enough of this stuff, agents naturally merge together into a collective which forwards the common good. (Not necessarily human good, but I'll ignore this for the sake of mythology.) All Deva are, at the same time as being their individual selves, aspects of a single Deva.
Asura (farsighted): Get a little more myopic, and you have merely far-sighted agents, who cooperate in iterated prisoner's dilemmas, but who defect in single-shot cases. These agents will often act Good, but have the capacity to be extremely harmful; their farsighted thinking allows them to play a long game and eventually stab partners in the back. Their longsightedness means they are greedy for galactic-scale resources. This can easily make them Evil, particularly when they get too large an upper hand in a power imbalance (power corrupts).
Preta (hungry ghosts): Around the middle of the spectrum, you get dangerously shortsighted agents, who would happily burn the planet for a short-term gain. These agents will defect in iterated prisoner's dilemma unless the iteration is quite fast, so if there are a lot of them, there will be a severe tragedy of the commons. Or, if you have one highly intelligent agent like this, it could be longsighted enough to take over the world, but not longsighted enough to run the world very well. Very Evil.
Tiryak (animals): As we progress further in the direction of myopia, you get very shortsighted agents. These agents become easier to put a leash on. You can mostly control them with carrot/stick methods. An extremely intelligent agent like this might be quite dangerous if it gets the wrong opportunity, but for the most part, even if these agents escape the leash and wander off, they'll only do local damage. So, agents of this type can be very useful, like domesticated animals.
Ṛta (law/order): Finally, at the extreme myopic end of the spectrum, you get something that only cares about local validity, operating by deontology rather than consequentialism. They're corrigible because they don't care about the future, so they won't fight any modifications you make to them. It probably doesn't even make sense to call these things agents any more. One form this can take is purely epistemic reasoning (with no predict-o-matic-style twists accidentally introducing agency).
It turns out that myopic things aren't just useful to have around; they're necessary.
This is because highly non-myopic beings love objective answers, for a lot of reasons. One reason is that objective answers give good Schelling points for coordination. If the non-myopic beings created Schelling points with their highly non-myopic logic, several things would go wrong:
So, for this reason, non-myopic agents will want to use myopic agents to provide Schelling points. The myopic agents are good for this because they answer questions only based on what's directly in front of them, making their answers pretty consistent and computationally/informationally simple.
But perhaps more importantly and fundamentally, agents need world-models. A world-model is myopic! It tries to answer each question individually, doing the best it can with that isolated question.
So the structure of the universe appears to be that non-myopia can't even get off the ground without standing on the shoulders of myopia. Deva and Asura create Ṛta.
Another good reason for this is the subsystem alignment problem. Suppose you are trying to answer a question, but you're not quite sure how. You can run some extra computations to help you. What computations do you run? Well, considering the myopic/non-myopic spectrum, you either want to run an extremely non-myopic computation, or an extremely myopic one (because those are Good, and anything else runs the risk of Evil).
It probably doesn't make sense to spin up a new non-myopic process, because that means spinning up a new general problem solver. You already are a general problem-solver. You could just work more yourself. No; your problem is that you need some specific insight.
So, you'll probably want to exclusively run highly myopic processes (Ṛta). These can get you the insight you need, without any risk of Evil.
So, the cosmic order makes it such that agents are made up of two very different kinds of computation: a highly non-myopic Deva/Asura component, responsible for the agency, and a highly myopic Ṛta component, which is like "beliefs".