This seems to completely miss how the actual argument works. Consider the simplified version for 'consistent preference orderings' rather than expected utility; the generalization is hopefully obvious. "If you strictly prefer (enough to pay a penny) apples to bananas, bananas to cherries, and cherries to apples, I can trade you an apple for a cherry for a banana for an apple and extract three cents from you. That's the theory; now to the practice. If you imagine an AI in almost any sort of RL environment that grades later outcomes, strategies and paths that run around in circles will be graded against; the same goes for biological organisms. This is why we see biological organisms that in most daily life behave like they have ranked desires, rather than entirely arbitrary strict pairwise preferences that often go around in circles." Now imagine the following objection: "But you could always have an infinite preference over histories to always prefer to have traded cherries for bananas for apples for cherries. Therefore, your coherence argument that AIs and organisms in most ancestral situations end up behaving more like they have ranked preferences than random ones doesn't prove that they'll behave coherently; because even if they end up with preferences, those preferences could be over histories, and then they could infinitely prefer histories with circular trades!" But RL in an arbitrary environment would crush out infinite preferences over histories like that. The same applies to utility functions that infinitely prefer histories of running around in circles wasting energy. This is why Huggingface went through in practice, rather than AIs instead forming infinite preferences over histories about arbitrary pairwise local decisions that would've been too incoherent to hack out of the box. It seems to me the objection fails. It seems to me the objection entirely failed to track what the original argument was about. Am I missing something or was this argument actually as bad as it looks to me? Asking now because it's coming up on Twitter again.

This is a truly exceptional piece of work! Thank you for writing it! I think you taught me some things about corrigibility, and I'll be spending many coming days engaging with the ideas of the post. Many of those ideas will produce questions, comments, and criticisms. But I wanted to start off with a wholehearted cheer for the combination of philosophical weight, relevance, and rigor. "Respect for authority" is a particularly good handle, and I'm a bit embarrassed to have never used it! (At least as far as I can recall.)

Pretraining scaling is starting to slow down, both in terms of compute and model size. The mad rush of 2025-2026 is done, and the models targeting 2027 hardware are probably already pretrained (though the work on their misalignment is then only getting started). In 2026-2028, scaling is about 2x slower than in 2025-2026, and then in 2028-2032 it gets 2x slower yet again. So "pacing" will be happening naturally in some sense. The problem is that the models of late 2026 are passing the level of human capabilities in many ways, so even slower scaling could get significantly harder to handle. And Meta, Google, and SpaceX currently have what seems to be well-made small models (somewhere between Sonnet and Opus scale-wise) and sufficient resources/ambition to plausibly catch up with OpenAI and Anthropic in 2027 at least in terms of model scale (with big models that are not so broken that it ruins their capabilities). To the extent scale is a source of misalignment difficulties that require operational excellence to sweep under the rug, they might have a hard time in 2027. And then slow-learning prosaic RSI of automated RL task/env creation in 2027-2028 probably introduces new misalignment/control issues, even if it doesn't manifest as acceleration or a step-change increase in capabilities, and merely fixes jaggedness by automatically filling in all ordinary capability gaps as they come up. Which is probably economically transformative, makes the LLM/RL paradigm more centrally AGI (by automating learning, in a very clunky way), enables useful humanoid robotics and opens up a path to a robot-industrial explosion, but isn't necessarily a raw capability jump per se. (Continual learning in a strong sense is another matter and probably quickly leads to strong architecture-rewriting RSI, but I don't know of a way to time its invention, other than to say that LLMs of 2027-2028 will be taking a stab at it. Like with math problems currently being slaughtered in droves, it could be "easy enough" in some sense for the LLM/RL paradigm to defeat quickly, but not necessarily.)