Review
This post still stands out to me as making an important and straightforward point about observer dependence of knowledge that is still, in my view, under appreciated (enough so that I wrote a book about it and related epistemological ideas!). I continue to think this is quite important for understanding AI, and in particular addressing interpretability concerns as they relate to safety, since lacking a general theory of why and how generalization happens, we may risk mistakes in building aligned AIs if they categorize the world in usual ways that we don't anticipate or understand.
Just an interesting philosophical argument
Why can an ML model learn from part of a distribution or data set, and generalize to the rest of it? Why can I learn some useful heuristics or principles in a particular context, and later apply them in other areas of my life?
The answer is obvious: because there are some underlying regularities between the parts I train on and the ones I test on. In the ML example, generalization won't work when approximating a function which is a completely random jumble of points.
Also, quantitatively, the more regular the function is, the better generalization will work. For example, polynomials of lower degree require less data points to pin down. Same goes for periodic functions. Also, a function with lower Lipschitz constant will allow for better bounding of the values in un-observed points.
So it must be that the variables we track (the ones we try to predict or control, either with data science or our actions), are given by disproportionately regular functions (relative to random ones). In this paper by Tegmark, the authors argue exactly that most macroscopic variables of interest have Hamiltonians of low polynomial degree. And that this happens because of some underlying principles of low-level physics, like locality, symmetry, or the hierarchical composition of physical processes.
But then, why is low-level physics like that?
If our low-level physics wasn't conducive to creating macroscopic patterns and regularities, then complex systems capable of asking that question (like ourselves) wouldn't exist. Indeed, we ourselves are nothing more than a specific kind of macroscopic pattern. So anthropics explains why we should expect such patterns to exist, similarly to how it explains why the gravitational constant, or the ratio between sound and light speed, are the right ones to allow for complex life.
But there's yet one more step.
Let's try to imagine a universe which is not conducive to such macroscopic patterns. Say you show me its generating code (its laws of physics), and run it. To me, it looks like a completely random mess. I am not able to differentiate any structural regularities that could be akin to the law of ideal gases, or the construction of molecules or cells. While on the contrary, if you showed me the running code of this reality, I'd be able (certainly after many efforts) to differentiate these conserved quantities and recurring structures.
What are, exactly, these macroscopic variables I'm able to track, like "pressure in a room", or "chemical energy in a cell"? Intuitively, they are a way to classify all possible physical arrangements into more coarse-grained buckets. In the language of statistical physics, we'd say they are a way to classify all possible microstates into a macrostate partition. For example, every possible numerical value for pressure is a different macrostate (a different bucket), that could be instantiated by many different microstates (exact positions of particles).
But there's a circularity problem. When we say a certain macroscopic variable (like pressure) is easily derived from others (like temperature), or that it is a useful way to track another variable we care about (like "whether a human can survive in this room"), we're being circular. Given I already have access to a certain macrostate partition (temperature), or that I already care about tracking a certain macrostate partition (aliveness of human), then I can say it is natural or privileged to track another partition (pressure). But I cannot motivate the importance of pressure as a macroscopic variable from just looking at the microstates.
Thus, "which parts of physics I consider interesting macroscopic variables to track" is observer-dependent in the first place. In fact, this point is already argued in a different context (the renormalization group) in the Tegmark paper.
So when I observe a universe running and fail to find any relevant macroscopic regularities, maybe I'm just failing to find the kinds of patterns I'm used to processing, but a different observer would find a fountain of informative patterns there.
It becomes tautological (even without the above application of anthropics) that a macroscopic pattern like me will find certain other macroscopic patterns to track: those that are expressed in the same "partitional language over the substrate of low-level physics" as myself. Of course my impression will be that regularity exists in the world: I am more prone to notice or care about exactly those patterns which I find regular.
For example, "pressure" is something I can easily track, and also something that can easily kill me, due to the same underlying reason: because the partition in which I, as a time-persistent pattern, am written, and that in which pressure is written, share a lot of information.
Similarly, the fact that "the exact position of an electron" is hard to track for me doesn't worry me too much, nor makes me deem this world irregular, because exactly by the same reason that I cannot easily track that partition, that partition is also very unlikely to hurt me or alter the variables I care about (the position of an electron marking the difference between a human being alive and dead is extremely unlikely).
Given this circularity, might not different partitional languages coexist over a same substrate low-level physics? Is it not possible that, inside that randomly-seeming universe, an infinitude of macroscopic patterns exist (analogous to us), complex enough to track the similar macroscopic patterns they care about (analogous to our pressure), but written in a partitional language we don't discern? Might it not be that all random-looking universes are as full of complex macroscopic patterns as ours, but different observers are able to decode different regularities?
This sounds a lot like dust theory, although it's an angle on it I hadn't seen sketched:
You, as a specific self-preserving macrostate partition, pick out similar macrostate partitions (that share information with you). And this is always possible, no matter which macrostate partition you are (even if the resulting partitions look, to our partition-laden eyes, more chaotic).
An equivalent way to put this, going back to looking at the computational trail of a random universe, is that depending on how you represent it I will or will not find legible patterns. Indeed, even for our universe, you can obfuscate its code in such a way (for example, shuffle spatial positions around) that makes it extremely difficult (if not impossible) for me to find regularities like molecules, or even notice locality. While, with the right de-shuffle, I'd be able to notice the code is running our universe. Of course, this is just the old problem of taking functionalism to its logical conclusion. And that's indeed what dust theory is all about.