One quick independent test of how well this works is to see if the method can recover an "internal CoT" in cases where AIs successfully solve a math problem in a single forward pass. I found this doesn't work, and the NLA doesn't show anything like an internal CoT on any of the cases I tested. I tried selected a few easy problems from the dataset used for this post that gemma-27b gets right and then tested on the NLA for gemma-27b. I tried a few different prompting approaches: few-shot vs single shot and with 5 repeats vs without repeats. This obviously wasn't a very systematic check, I'm trying to get claude to do a more systematic test. Other limitations of this test: * Maybe these problems are memorized or effectively memorized. * Maybe access to the internal CoT doesn't exist at any individual token position and you'd need a multi-token NLA. Or maybe I tested the wrong token positions.

I like this post and the "theory of deep learning" posts. But I think I still haven't figured out how to model your view, especially the specifics of the pessimism here. Maybe we should discuss in person. In particular I'm not sure what "deep learning theory" encompasses. My sense of mechinterp theory is that it's similar to pre- standard model physics. Heuristically, here's a thought experiment. Suppose we're worried about the sun destroying the earth and want to understand as much as possible about the physics of solar plasmas and supernovas; but we currently only have (a vaguely historical pastiche of) pre-WW2 physics. Physics then roughly had the following components: 1. idealized heuristics: if we view a big object in space as a classical blackbody, we get a good heuristic on some parts of its emission spectrum 2. new behaviors: there's a consistent way that emission spectra aren't classical blackbodies, in that they're quantized. We have only a rough understanding of how and why, and in fact this observation spawned the discovery of quantum mechanics. 3. small toy examples: we can understand the hydrogen atom relatively well. There are some weird factors of 2 and corrections that we can only explain kind of heuristically, but except for these we have a clean, exact quantized spectrum. We see this spectrum in real life materials - but we also see that most of what comprises real materials isn't hydrogen, and is much more complicated. Some stuff still looks roughly like they could be atomic spectra for other atoms or small molecules, but metal conducting bands are dominated by weird and clearly non-localized behaviors that we don't understand (and the sun similarly has weird spectral phenomena). 4. limits. There's a limit where the world is Newtonian, which is sometimes useful, but very inaccurate when modeling the sun. There's a limit where the world is relativistic. This gives directionally good corrections for some stellar phenomena (e.g. redshift) but is not nearly enough. It seems that there are maybe other limits (like we can mostly blackbox nuclear phenomena at earth temperatures but not at solar temperatures). Most of our understanding comes from sloppily combining together different phenomena coming from the various limits of importance. 5. experimental tools: looking at emission spectra is a really low-bandwidth way to interact with behaviors of interest. While it gives interesting info that points to new phenomena, it at best tells us something about a very limited class of behaviors (photon absorption and emission). In order to understand "how QM works" we have to figure out new tools (maybe vacuum chambers and primitive colliders), and new ways to interpret the output of existing tools. In our world, iterating on these techniques gave us the standard model (and we understood solar plasma and some basics of supernovas before this). I think the promise of theory is that analogs of these techniques (maybe: SAE, large-N limits, toys like mod-add) will give us robust mechanism-finding tools. I think a lot of criticism of theory sounds to me like someone who in that world is saying "none of the current tools explain the sun even approximately, so we're on the wrong track" - but that's not how theory works (until you've found a "critical mass" of the standard model, or at least all parts of it relevant in a field of interest, you'll only be explaining tiny fractions of the observations). I know you're not making this criticism, but I feel like currently you are flattening the different components above into one notion of "theory good vs. theory bad". I'd guess that you're skeptical about whether the analogs of 1-5 in ML theory are actually useful for "making progress towards the standard model", but I'm not sure from your post which of these you think is most lacking (or if this picture is even compatible with your criticism). My guess is that your issue is wrt something like my #1: certain heuristics that people were excited about and hoped would explain generalization turned out to be more complicated. My view is that in modern theory, VC dimension is considered largely defunct in models with nontrivially interesting data (even as simple as mod-add), but I'm not sure why this is the important thing about theory. If you take a more modern theory like mean field or even NTK, it has a non-VC notion of generalization: e.g. NTK/ Gaussian processes can replicate generalization in mnist (related to some data spectrum properties), and mean field theory can (currently only on the Bayesian level - this is unpublished work with Kaarel) explain generalization on polynomially many samples of any mechanism that can be encoded in a small (algorithms) circuit. It's of course not guaranteed to converge to the same mechanism, but has the same notion of learnable vs. un-learnable on a polynomial-vs-exponential complexity theory level. It also does replicate the correct modular addition generalizing algorithm (NTK does not).

This is one of my favorite LessWrong posts ever (strong upvoted). Nevertheless, in this comment I'll provide some pushback; not to any the particular claims made, but around a sort of "missing mood" that I think a stronger version of this post could have addressed. (For context, I'm probably one of the lab employees who sometimes says things like "Current AIs are pretty aligned"—i.e. one of the people that Ryan is arguing against here, though I might not be a very central example.) For me, the value of this post was the following: 1. Articulating a specific way that current AIs are "misaligned." Namely: 2. 1. They follow heuristics that are well-tuned for strong task performance in easy-to-verify settings, but only try superficially at difficult-to-evaluate tasks. 2. They communicate in ways that misrepresent how much they've accomplished: punching up their successes while omitting or downplaying failures, and sometimes outright lying about what they did. But they don't seem to do this strategically or persistently, instead following simple heuristics that result in these sorts of misrepresentations. 3. The overall result is that they underperform their potential at difficult-to-evaluate tasks while misleading users about what they've actually accomplished. 4. I say "misaligned" in quotes, because many traditional notions of misalignment wouldn't have covered the above, especially 1(a). This will be relevant later. 3. Providing concrete examples and a sense of how prevalent the above issues are. I want to be clear that (2) was an update for me. I think the way that you (Ryan) use LLMs more reliably elicits the sort of misalignment you name, relative to most use cases I've personally seen. (Namely, it seems like you try to use LLMs to autonomously complete very difficult tasks using scaffolds that rely extensively on self-critique.) So after reading your post, I now think that current AIs are more misaligned (in the sense you describe) than I previously thought. That said, this update was more quantitative in nature than qualitative. I was certainly aware that current AIs sometimes display the behaviors you describe, even if I thought they were somewhat rarer than I now believe. And I don't think that I personally underrated the importance of these behaviors. (Though it's possible that others did.) [ETA: After thinking about this more, I think I probably was underestimating the importance of these behaviors before reading an early draft of Ryan's post.] Given that, what could I possibly mean when I call current AIs "pretty aligned"? I think there's a hypothetical dual post to yours someone could write titled "Current AIs seem pretty aligned to me." This post would name a bunch of types of misalignment that, a few years ago, seemed could plausibly occur in AIs as capable as current ones. For instance, it seems to me that: 1. Current AIs do not training-game, i.e. actively model the training process and take actions that result in high reward[1] (including in training environments very different from ones they've previously encountered). 2. Current AIs do persistently and coherently pursue malign goals across many contexts, e.g. strategically seeking power. 3. Current AIs do not scheme, i.e. strategically subvert their training process, pre-deployment audits, and other types of oversight. 4. Current AIs do not strategically take steps to cover up their mistakes, and will generally admit to them when asked directly. 5. Current AIs generally don't try to strategically influence things in the real world beyond assisting with (or refusing) the task directly presented to them. E.g. they don't sneakily try to funnel money to causes or people they like by influencing unrelated conversations. Instead, I think the most important way that current AIs are misaligned is the way you describe in your post, Notably, I think the type of misalignment described in your post is "less scary" than the types I list above. The misalignment happens on the "learned heuristics" level, rather than on the "active"/"strategic"/"explicit" level. As a result, it's easier to detect and measure, and less likely to cause harms besides "making the AIs less useful than they could be for certain types of tasks." A related point is that the way that current AIs are misaligned is less "adversarial" than the sorts of misalignment I list above. When I speak to the people who IMO are extremely thoughtful about AI safety, they acknowledge something like "Yeah, TBH a few years ago, I thought that AIs as capable as current ones would be misaligned in some of those ways. There's a real positive update here." But people who are merely "quite" thoughtful sometimes instead: * argue that maybe current AIs are misaligned in some of these ways and we just can't tell * * I'm very skeptical of this! Anyone who thinks this should be trying really hard to produce compelling evidence in current models. * munge definitions to try to count the "misaligned heuristics" types of misalignment as one of these more "coherent" types * claim that they never expected AIs as capable as current ones to be misaligned in these ways * * I'm typically skeptical of this, though I believe it for some people. (A related point that I know how to state more vividly: Sometimes, people like to point at clear alignment issues in frontier models—for instance, sycophancy, AI psychosis, or whatever happened with Sydney/Bing chat—as being negative updates about alignment. But they don't like to count absence of these sorts of alignment issues as being positive updates. In other words, if it hypothetically ended up that AI psychosis were somehow a total media hoax, would the the people who updated negatively on it instead say "Nevermind! Actually, the absence of AI psychosis is a positive update on alignment." Are they currently saying this about the ways that current AIs are not misaligned but could have been?) Overall, my point is that I think current AIs are not misaligned in many of the ways that I—and I think many others—expected them to be. I also think that many people I talk to have not grappled with this observation sufficiently and updated their world models and prioritization accordingly. For instance, I think that the AI safety community is overinvested in scheming research relative to scalable oversight research. (TBC I've believed this for some years now, but I now believe it more confidently and think it's more clear that others should agree with me.) I also think that the AI safety community is underinvested in work on directly automating alignment research. (This is a place where I've changed my mind.) I'll close by emphasizing some things that I do not believe and don't mean to claim in this comment: 1. I don't think that the recent overall evidence has clearly been positive about misalignment risk. For instance, timelines are looking quite short now, which IMO is the main negative update to counteract the positive one I discuss in this comment. 2. I don't think these "scarier" types of misalignment are very unlikely to eventually arise in more capable AIs. Though I do think that we should update that they'll first arise in more capable AIs than we previously thought. This makes it more likely that we'll have human-level or superhuman automated alignment researchers prior to these "scarier" types of misalignment. 3. I don't think that the type of misalignment you name in your post is "not a big deal." In particular, I'm quite worried that "we"(=AI developers or humanity as a whole) bungle the situation because we failed to make AIs capable at difficult-to-evaluate tasks like alignment research or advising us about how to handle things during a rapid intelligence explosion. One possible way we could bungle the situation is to train more capable AIs that are misaligned in some of the "scarier" ways I listed (or in other ways I didn't list). 4. I don't think that current AIs are aligned enough to automate alignment research. (An earlier draft of your post operationalized "pretty misaligned" in this way, which I preferred.) 1. ^ Edit 5/10: I think this is probably false as stated and I should have been more precise. I think in some cases AIs do training-game in the way I describe—i.e. actively model the training process and take actions that they expect will lead to high reward. However, I don't think they do this in a very coherent or persistent way. For instance, I think this mainly happens in contexts that are similar to their training data, with training-gaming motivations not generalizing very broadly to other settings. I think that "just trying to be helpful" is a bigger part of their motivation most of the time than "trying to get high reward."