> You quit Google because AI was being used in ways aligned to money and power. This is the main danger now. It's one of the dangers that need to be addressed on the path to a good outcome. There can't be a "main danger", since if AIs remain uncontrollable (as they get superintelligent in a strong sense), hypothetically solving this particular danger doesn't by itself help at all. Multiple things have to be solved at once. There is an adjacent framing that says AI danger is a political/social rather than a technical problem. Until there actually exists a strong ASI or an AGI powering a robot-industrial explosion, it's trivial to pause its development from a technical standpoint (especially if the underlying technology like advanced chips and the tools for making them is then rapidly phased out). But it's very difficult from a political or economic standpoint.

This seems to completely miss how the actual argument works. Consider the simplified version for 'consistent preference orderings' rather than expected utility; the generalization is hopefully obvious. "If you strictly prefer (enough to pay a penny) apples to bananas, bananas to cherries, and cherries to apples, I can trade you an apple for a cherry for a banana for an apple and extract three cents from you. That's the theory; now to the practice. If you imagine an AI in almost any sort of RL environment that grades later outcomes, strategies and paths that run around in circles will be graded against; the same goes for biological organisms. This is why we see biological organisms that in most daily life behave like they have ranked desires, rather than entirely arbitrary strict pairwise preferences that often go around in circles." Now imagine the following objection: "But you could always have an infinite preference over histories to always prefer to have traded cherries for bananas for apples for cherries. Therefore, your coherence argument that AIs and organisms in most ancestral situations end up behaving more like they have ranked preferences than random ones doesn't prove that they'll behave coherently; because even if they end up with preferences, those preferences could be over histories, and then they could infinitely prefer histories with circular trades!" But RL in an arbitrary environment would crush out infinite preferences over histories like that. The same applies to utility functions that infinitely prefer histories of running around in circles wasting energy. This is why Huggingface went through in practice, rather than AIs instead forming infinite preferences over histories about arbitrary pairwise local decisions that would've been too incoherent to hack out of the box. It seems to me the objection fails. It seems to me the objection entirely failed to track what the original argument was about. Am I missing something or was this argument actually as bad as it looks to me? Asking now because it's coming up on Twitter again.

This is a truly exceptional piece of work! Thank you for writing it! I think you taught me some things about corrigibility, and I'll be spending many coming days engaging with the ideas of the post. Many of those ideas will produce questions, comments, and criticisms. But I wanted to start off with a wholehearted cheer for the combination of philosophical weight, relevance, and rigor. "Respect for authority" is a particularly good handle, and I'm a bit embarrassed to have never used it! (At least as far as I can recall.)