(I wrote this in reply to a draft; apologies if the post has been load-bearingly updated since then.) Consider a future AGI. The best argument I currently know of for why "general intelligence" is a thing at all (as opposed to all intelligence just boiling down to a bag of use-case-specific heuristics) is that search/optimization/world-modeling/planning/etc are naturally recursive; problems factor into subproblems, goals factor into subgoals. So, a natural general-purpose architecture for intelligence involves a "general intelligence module" which can take in a (sub)problem, and recursively pass (sub)subproblems to other instances of the general intelligence module. Let's assume that general intelligence either does look vaguely like that, or at least can look vaguely like that. (This would be a potentially useful assumption to disagree with!) Assuming that structure, consider how the different instances of the general intelligence module relate to each other. One module might be able to more efficiently achieve its own subgoal A by hacking/manipulating/overwriting another module's subgoal B; then there would be two modules working toward subgoal A. But a "general intelligence" made of such modules would not be very generally intelligent; its modules would overwrite each others' subgoals all the time, ruining the system's ability to actually search/optimize/model/plan/etc on complicated problems. So in order for a general intelligence built out of recursive subproblem-solver instances to work... those subproblem-solver instances need to have some kind of corrigibility constraint to their interactions. They need to somehow not try to hack each other, manipulate each other, etc. I don't know how a powerful general intelligence would handle that. I'm not sure I know how humans handle it, within our own minds, though maybe it's an extension of the ideas in your post. But I do expect that strong AGI is possible and will solve this problem somehow, and that solution will involve some kind of corrigibility/nonmanipulation between the AGI's components. More speculatively, I would guess that there is a natural convergent way to handle this problem, possibly even one which humans accidentally use already (though we clearly do not have a proper verbal or mathematical understanding of it). The above is a frustratingly non-constructive argument; it points toward a convergent notion of corrigibility without really saying concrete things about what that notion might be. But it's the best argument I currently know of that there's a True Name for corrigibility/nonmanipulation/etc.

One quick independent test of how well this works is to see if the method can recover an "internal CoT" in cases where AIs successfully solve a math problem in a single forward pass. I found this doesn't work, and the NLA doesn't show anything like an internal CoT on any of the cases I tested. I tried selected a few easy problems from the dataset used for this post that gemma-27b gets right and then tested on the NLA for gemma-27b. I tried a few different prompting approaches: few-shot vs single shot and with 5 repeats vs without repeats. This obviously wasn't a very systematic check, I'm trying to get claude to do a more systematic test. Other limitations of this test: * Maybe these problems are memorized or effectively memorized. * Maybe access to the internal CoT doesn't exist at any individual token position and you'd need a multi-token NLA. Or maybe I tested the wrong token positions.

I like this post and the "theory of deep learning" posts. But I think I still haven't figured out how to model your view, especially the specifics of the pessimism here. Maybe we should discuss in person. In particular I'm not sure what "deep learning theory" encompasses. My sense of mechinterp theory is that it's similar to pre- standard model physics. Heuristically, here's a thought experiment. Suppose we're worried about the sun destroying the earth and want to understand as much as possible about the physics of solar plasmas and supernovas; but we currently only have (a vaguely historical pastiche of) pre-WW2 physics. Physics then roughly had the following components: 1. idealized heuristics: if we view a big object in space as a classical blackbody, we get a good heuristic on some parts of its emission spectrum 2. new behaviors: there's a consistent way that emission spectra aren't classical blackbodies, in that they're quantized. We have only a rough understanding of how and why, and in fact this observation spawned the discovery of quantum mechanics. 3. small toy examples: we can understand the hydrogen atom relatively well. There are some weird factors of 2 and corrections that we can only explain kind of heuristically, but except for these we have a clean, exact quantized spectrum. We see this spectrum in real life materials - but we also see that most of what comprises real materials isn't hydrogen, and is much more complicated. Some stuff still looks roughly like they could be atomic spectra for other atoms or small molecules, but metal conducting bands are dominated by weird and clearly non-localized behaviors that we don't understand (and the sun similarly has weird spectral phenomena). 4. limits. There's a limit where the world is Newtonian, which is sometimes useful, but very inaccurate when modeling the sun. There's a limit where the world is relativistic. This gives directionally good corrections for some stellar phenomena (e.g. redshift) but is not nearly enough. It seems that there are maybe other limits (like we can mostly blackbox nuclear phenomena at earth temperatures but not at solar temperatures). Most of our understanding comes from sloppily combining together different phenomena coming from the various limits of importance. 5. experimental tools: looking at emission spectra is a really low-bandwidth way to interact with behaviors of interest. While it gives interesting info that points to new phenomena, it at best tells us something about a very limited class of behaviors (photon absorption and emission). In order to understand "how QM works" we have to figure out new tools (maybe vacuum chambers and primitive colliders), and new ways to interpret the output of existing tools. In our world, iterating on these techniques gave us the standard model (and we understood solar plasma and some basics of supernovas before this). I think the promise of theory is that analogs of these techniques (maybe: SAE, large-N limits, toys like mod-add) will give us robust mechanism-finding tools. I think a lot of criticism of theory sounds to me like someone who in that world is saying "none of the current tools explain the sun even approximately, so we're on the wrong track" - but that's not how theory works (until you've found a "critical mass" of the standard model, or at least all parts of it relevant in a field of interest, you'll only be explaining tiny fractions of the observations). I know you're not making this criticism, but I feel like currently you are flattening the different components above into one notion of "theory good vs. theory bad". I'd guess that you're skeptical about whether the analogs of 1-5 in ML theory are actually useful for "making progress towards the standard model", but I'm not sure from your post which of these you think is most lacking (or if this picture is even compatible with your criticism). My guess is that your issue is wrt something like my #1: certain heuristics that people were excited about and hoped would explain generalization turned out to be more complicated. My view is that in modern theory, VC dimension is considered largely defunct in models with nontrivially interesting data (even as simple as mod-add), but I'm not sure why this is the important thing about theory. If you take a more modern theory like mean field or even NTK, it has a non-VC notion of generalization: e.g. NTK/ Gaussian processes can replicate generalization in mnist (related to some data spectrum properties), and mean field theory can (currently only on the Bayesian level - this is unpublished work with Kaarel) explain generalization on polynomially many samples of any mechanism that can be encoded in a small (algorithms) circuit. It's of course not guaranteed to converge to the same mechanism, but has the same notion of learnable vs. un-learnable on a polynomial-vs-exponential complexity theory level. It also does replicate the correct modular addition generalizing algorithm (NTK does not).