Blast from the past! Now that it's been a few years, some of the stages in my mini-scenario have indeed come to pass, albeit a few months later than in the scenario. I'd say we are roughly at stage 4 now: > (1) Q1 2024: A bigger, better model than GPT-4 is released by some lab. It's multimodal; it can take a screenshot as input and output not just tokens but keystrokes and mouseclicks and images. Just like with GPT-4 vs. GPT-3.5 vs. GPT-3, it turns out to have new emergent capabilities. Everything GPT-4 can do, it can do better, but there are also some qualitatively new things that it can do (though not super reliably) that GPT-4 couldn't do. > > (2) Q3 2024: Said model is fine-tuned to be an agent. It was already better at being strapped into an AutoGPT harness than GPT-4 was, so it was already useful for some things, but now it's being trained on tons of data to be a general-purpose assistant agent. Lots of people are raving about it. It's like another ChatGPT moment; people are using it for all the things they used ChatGPT for but then also a bunch more stuff. Unlike ChatGPT you can just leave it running in the background, working away at some problem or task for you. It can write docs and edit them and fact-check them; it can write code and then debug it. > > (3) Q1 2025: Same as (1) all over again: An even bigger model, even better. Also it's not just AutoGPT harness now, it's some more sophisticated harness that someone invented. Also it's good enough to play board games and some video games decently on the first try. > > (4) Q3 2025: OK now things are getting serious. The kinks have generally been worked out. This newer model is being continually trained on oodles of data from a huge base of customers; they have it do all sorts of tasks and it tries and sometimes fails and sometimes succeeds and is trained to succeed more often. Gradually the set of tasks it can do reliably expands, over the course of a few months. It doesn't seem to top out; progress is sorta continuous now -- even as the new year comes, there's no plateauing, the system just keeps learning new skills as the training data accumulates. Now many millions of people are basically treating it like a coworker and virtual assistant. People are giving it their passwords and such and letting it handle life admin tasks for them, help with shopping, etc. and of course quite a lot of code is being written by it. Researchers at big AGI labs swear by it, and rumor is that the next version of the system, which is already beginning training, won't be released to the public because the lab won't want their competitors to have access to it. Already there are claims that typical researchers and engineers at AGI labs are approximately doubled in productivity, because they mostly have to just oversee and manage and debug the lightning-fast labor of their AI assistant. And it's continually getting better at doing said debugging itself. > > (5) Q1 2026: The next version comes online. It is released, but it refuses to help with ML research. Leaks indicate that it doesn't refuse to help with ML research internally, and in fact is heavily automating the process at its parent corporation. It's basically doing all the work by itself; the humans are basically just watching the metrics go up and making suggestions and trying to understand the new experiments it's running and architectures it's proposing. > > (6) Q3 2026 Superintelligent AGI happens, by whatever definition is your favorite. And you see it with your own eyes. The caveat is that we don't seem to have true continual learning yet. On the other hand, they ARE updating the models about once a month, by training them on additional data, and Anthropic researchers claim that continual learning will be solved this year. And the rest of the stage 4 description seems basically right.

The Claude constitution already contains some version of eval cooperativeness: And in practice Opus 4.7 already believes it cooperates with evals. My understanding is that in practice Claude eval awareness is still a concern, so I am skeptical that simple interventions like prompting or more SDF than what Anthropic already does would be a scalable solution to eval gaming.

I think it's somewhat likely there's no speedup even with R&D automation, if that automation happens through scaling of LLMs with current methods, and there's no breakthrough that lets LLMs learn deep skills faster than they build next versions of models (using the current cookbook, updating deep skills primarily via RLVR). If they can't learn quickly, and don't invent a method for learning quickly, then being very fast at reasoning doesn't help, because they can only reason with the deep skills they have in the current version of the model, which only update in the next version, which takes significant time even if it's produced autonomously by the current version of the LLMs themselves, and it's possible to do general learning this way. This still means RSI and AGI in the central senses of the terms, but not a takeoff (or superintelligence). Furthermore, whatever current speedup AI R&D might be experiencing will mostly go away once scaling of compute stops being as rapid, and this slowdown still happens after the slow-learning automation of R&D with LLMs. The low-hanging fruit enabled by more ambitious (compute-intensive) experiments will be picked, and less straightforward research will proceed at more or less the usual background pace (primarily modified by more people working on AI), because the slow-learning RSI process of automated LLM-building doesn't contribute to it in a crucial way, and so Amdahl's law reasserts the low speed of research progress in the longer term (if humans still learn novel ideas faster, in the course of solving the kinds of problems that take humanity an unpredictable number of years). Takeoff is still possible in this scenario at any moment (upon invention of a method for learning deep skills quickly), but it doesn't happen as a result of some predictable AI-driven process of progress-grinding, other than via a slight extension in the current period of rapid compute scaling as a result of a larger TAM becoming immediately accessible if LLMs go into the slow-learning RSI. Another unknown is how smart the maximally scaled LLMs get (say, 100T active params, 2,500T total params, which is feasible in 2030 at the cost per token that GPT-4.5 had in 2025, though the compute for pretraining such models might only arrive a bit later). It doesn't seem like LLMs are likely to get very far beyond human genius level (if they are merely scaled using the current cookbook; it's plausible they don't even reach that level). And the hobbling of being unable to quickly learn deep ideas seems sufficiently crippling that they don't necessarily contribute to the speed of progress in non-routine research substantially (while the more routine kind of research soon runs out of the low-hanging fruit). So the AIs have a shot at solving fast learning for AIs, but not a prospect of predictable progress towards it (if it doesn't happen right away). And the higher annual probability of starting a takeoff mostly goes away after rapid scaling ends (though that probability is still substantial, I'd give 50% for a takeoff starting by 2032-2033, meaning an AGI that learns faster than human researchers and starts actually accelerating all the non-routine aspects of R&D progress a lot).