This is a deeply confused post. In this post, Turner sets out to debunk what he perceives as "fundamentally confused ideas" which are common in the AI alignment field. I strongly disagree with his claims. In section 1, Turner quotes a passage from "Superintelligence", in which Bostrom talks about the problem of wireheading. Turner declares this to be "nonsense" since, according to Turner, RL systems don't seek to maximize a reward. First, Bostrom (AFAICT) is describing a system which (i) learns online (ii) maximizes long-term consequences. There are good reasons to focus on such a system: these are properties that are desirable in an AI defense system, if the system is aligned. Now, the LLM+RLHF paradigm which Turners puts in the center is, at least superficially, not like that. However, this is no argument against Bostrom: today's systems already went beyond LLM+RLHF (introducing RL over chain-of-thought) and tomorrow's systems are likely to be even more different. And, if a given AI design does not somehow acquire properties i+ii even indirectly (e.g. via in-context learning), then it's not clear how would it be useful for creating a defense system. Second, Turner might argue that even granted i+ii, the AI would still not maximize reward because the properties of deep learning would cause it to converge to some different, reward-suboptimal, model. While this is often true, it is hardly an argument why not to worry.  While deep learning is not known to guarantee convergence to the reward-optimal policy (we don't know how to prove almost any guarantees about deep learning), RL algorithms are certainly designed with reward maximization in mind. If your AI is unaligned even under best-case assumptions about learning convergence, it seems very unlikely that deviating from these assumptions would somehow cause it to be aligned (while remaining highly capable). To argue otherwise is akin to hoping for the rocket to reach the moon because our equations of orbital mechanics don't account for some errors, rather than despite of it. After this argument, Turner adds that "as a point of further fact, RL approaches constitute humanity's current best tools for aligning AI systems today". This observation seems completely irrelevant. It was indeed expected that RL would be useful in the subhuman regime, when the system cannot fail catastrophically simply because it lacks the capabilities. (Even when it convinces some vulnerable person to commit suicide, OpenAI's legal department can handle it.) I would expect it to be obvious to Bostrom even back then, and doesn't invalidate his conclusions in the slightest. In section 3, Turner proceeds to attack the so-called "counting argument" for misalignment. The counting argument goes, since there are much more misaligned minds/goals than aligned minds/goals, even conditional on "good" behavior in training, it seems unlikely that current methods will produce an aligned mind. Turner (quoting Belrose and Pope) counters this argument by way of analogy. Deep learning successfully generalizes even though most models that perform well on the training data don't perform well on the test data. Hence, (they argue) the counting argument must be fallacious. The major error that Turner, Belrose and Pope are making is that of confusing aleatoric and epistemic uncertainty. There is also a minor error of being careless about what measure the counting is performed over. If we did not know anything about some algorithm except that it performs well on the training data, we would indeed have at most a weak expectation of it performing well on the test data. However, deep learning is far from random in this regard: it was selected by decades of research to be that sort of algorithm that does generalize well. Hence, the counting argument in this case gives us a perfectly reasonable prior. (The minor point is that, w.r.t to a simplicity prior, even a random algorithm has some bounded-from-below probability of generalizing well.) The counting argument is not premised on deep understanding of how deep learning works (which at present doesn't exist), but on a reasonable prior about what should we expect from our vantage point of ignorance. It describes our epistemic uncertainty, not the aleatoric uncertainty of deep learning. We can imagine that, if we knew how deep learning really works in the context of typical LLM training data etc, we would be able to confidently conclude that, say, RLHF has a high probability to eventually produce agents that primarily want to build astronomical superstructures in the shape of English letters, or whatnot. (It is ofc also possible we would conclude that LLM+RLHF will never produce anything powerful enough to be dangerous or useful-as-defense-sytem.) That would not be inconsistent with the counting argument as applied from our current state of knowledge. The real question is then, conditional on our knowledge that deep learning often generalizes well, how confident are we that it will generalize aligned behavior from training to deployment, when scaled up to highly capable systems. Unfortunately, I don't think this update is strong enough to make us remotely safe. The fact deep learning generalizes implies that it implements some form of Occam's razor, but Occam's razor doesn't strongly select for alignment, as far as we can tell. Our current (more or less) best model of Occam's razor is Solomonoff induction, which Turner dismisses as irrelevant to neural networks: but here again, the fact that our understanding is flawed just pushes us back towards the counting-argument-prior, not towards safety. Also, we should keep in mind that deep learning doesn't always generalize well empirically, it's just that when it fails we add more data until it starts generalizing. But, if the failure is "kill all humans", there is nobody left to add more data. Turner's conclusion is "it becomes far easier to just use the AIs as tools which do things we ask". The extent to which I agree with this depends on the interpretation of the vague term "tools". Certainly modern AI is a tool that does approximately what we ask (even though when using AI for math, I'm already often annoyed at its attempts to cheat and hide the flaws of its arguments). However, I don't think we know how to safety create "tools" that are powerful enough to e.g. nearly-autonomously do alignment research or otherwise make substantial steps toward building an AI defense systems.

Here's my understanding of the model presented in this post. Please me know if I'm missing or misunderstanding anything. 1. The learning subsystem of the brain includes a bunch of short term predictors. These predictors take in information from the all over the brain to predict values in the steering subsystem, in particular. 2. These short term predictors get feedback by sending their predictions down to the steering subsystem, and in return get supervision signals that they can use to learn their policy. 3. The steering subsystem will often just turn around and send the prediction back as the supervision signal. When this happens, it creates a feedback loop of the predictor predicting itself. 4. When the short term predictor is in this self-loop mode, it has degrees of freedom about what state it's in. Any prediction that it offers is a "correct answer". 5. However, the steering subsystem will send a ground truth signal at some point. So the best strategy for a short term predictor in a self-loop mode is to jump to predicting the next override as soon as it can guess what the next override will be. * For instance, as soon as that short term predictor notices a change in context like "you had the thought to go put on a sweater", it immediately starts predicting the positive reward of "feel warm and cozy" from having put on a sweater. * It could self-stably give any prediction. But immediately predicting the reward of putting on a sweater is the best policy. It won't get penalized while it's in the self-loop mode (since whatever the short term outputs is marked correct) but when the ground truth injection does arrive the short term predictor will already be correctly predicting that ground truth injection.  * That is, jumping to predicting the next ground truth injection as soon as the predictor can guess what it will be does no worse than any other possible prediction during the self-loop mode, and it does better than other possible predictions during the steering override mode. 6. This feedback makes the short-term predictors into long term predictors, because they're effectively learning to predict the next "steering override", based on context. This allows the short term predictors to learn predictive patterns that occur across many different timescales. * The timescale that they learn to predict over depends the on frequency with which the steering subsystem provides ground truth injections: if it's around once a minute, the predictor learns to predict a minute into the future, if it's around once an hour, the predictor learns to predict an hour into into the future and so on. (And if the frequency of ground truth injections are variable, but predictable from brain context, the predictors will learn a policy that predicts over different time horizons depending on that context?) 7. Those long term predictors can be used as control signals. These long term predictors can steer behavior based on outcomes that will only occur minutes or hours into the future. * In the sweater example, the brain can use the expectation of feeling warm and cozy, or not, as an indicator that the current behavior is on track for producing the warm and cozy reward.  * This setup constructs a long term reward-seeker out of an immediate reward-seeker and a pile of predictors. 

> Gemini 2.5 Pro and Gemini 3 Pro both don't support disabling reasoning which makes running evaluations for these models tricky. I think you can get around this by using prefill? I just did a quick test with Gemini 3 Pro via OpenRouter, and if I send the message list [ { role: 'user', content: 'Hi' }, { role: 'assistant', content: 'Hello! How' } ] then I'll get back a completed version of the assistant message ("Hello! How can I help you today?") with no reasoning content. And Openrouter's activity page says the request involved 6 output tokens, none of which were reasoning tokens. (It's possible that this breaks down in some way for math problems, though.)