AI ALIGNMENT FORUM
AF

AI Alignment Forum

Measuring no CoT math time horizon (single forward pass)

> Gemini 2.5 Pro and Gemini 3 Pro both don't support disabling reasoning which makes running evaluations for these models tricky. I think you can get around this by using prefill? I just did a quick test with Gemini 3 Pro via OpenRouter, and if I send the message list [ { role: 'user', content: 'Hi' }, { role: 'assistant', content: 'Hello! How' } ] then I'll get back a completed version of the assistant message ("Hello! How can I help you today?") with no reasoning content. And Openrouter's activity page says the request involved 6 output tokens, none of which were reasoning tokens. (It's possible that this breaks down in some way for math problems, though.)

ryan_greenblatt9d250

Recent LLMs can use filler tokens or problem repeats to improve (no-CoT) math performance

Recall that without filler, Opus 4.5 performance is 45.2%. I tried the following experiments on Opus 4.5 with filler counting to 300: * Default (what I do by default in the blog post above): 51.1% * Remove the text explaining filler tokens (as in, cut "After the problem ..."): 50.4% * Use "After the problem, there will be distractor tokens (counting from 1 to {filler_tokens})": 51.1% * Don't actually use filler tokens, but include in the prompt "After the problem, there will be filler tokens (counting from 1 to 300) to give you extra space to process the problem before answering" (as in, this is just a lie, we don't give filler): 45.8% So, it seems like the framing doesn't matter ~at all and actually having the filler tokens is the key thing (at least for Opus 4.5, though I strongly expect this would reproduce for Opus 4, Sonnet 4).

Vladimir_Nesov9d81

The future of alignment if LLMs are a bubble

> Since the end of (very weak) training scaling laws Precisely because the scaling laws are somewhat weak, there was nothing so far to indicate they are ending (the only sense in which they might be ending is running out of text data, but models trained on 2024 compute should still have more than enough). The scaling laws held for many orders of magnitude, they are going to hold for a bit further. It's plausibly not enough, even with something to serve the role of continual learning (beyond in-context learning on ever larger contexts). But there is still another 100x-400x in compute to go, compared to the best models deployed today. Likely the 100x-400x models will be trained in 2029-2031, at which point the pre-AGI funding for training systems mostly plateaus. This is (a bit more than) a full step of GPT-2 to GPT-3, or GPT-3 to original Mar 2023 GPT-4 (after original Mar 2023 GPT-4 and with the exception of GPT-4.5, OpenAI's naming convention no longer tracks pretraining compute). And we still didn't see such a full step compared to original Mar 2023 GPT-4, only half of a step (10x-25x), out of the total of 3-4 halves-of-a-step (2022-2030 training compute ramp, 2000x-10,000x in total, at higher end if BF16 to NVFP4 transition is included, at lower end if even in 2030 there are no 5 GW training systems and somehow BF16 needs to be used for the largest models). Since original Mar 2023 GPT-4, models that were allowed to get notably larger and made full use of the other contemporary techniques only appeared in late 2025 (likely Gemini 3 Pro and Opus 4.5). These models are probably sized compute optimally for 2024 levels of pretraining compute (as in 100K H100s, 10x-25x the FLOPs of original Mar 2023 GPT-4), might have been pretrained with that amount of compute or a bit more, plus pretraining scale RLVR. All the other models we've seen so far are either smaller than compute optimal for even 2024 levels of pretrained compute (Gemini 2.5 Pro, Grok 4, especially GPT-5), or didn't get the full benefit of RLVR compared to pretraining (Opus 4.0, GPT-4.5) and so in some ways looked underwhelming compared to the other (smaller) models that were more comprehensively trained. The buildout of GB200/GB300 NVL72 will be complete at flagship model scale in 2026, and makes it possible to easily serve models sized compute optimally for 2024 levels of compute (MoE models with many trillions of total params). More training compute is currently available and will be available in 2026 than what was there in 2024, but for most of the inference hardware currently available it won't be efficient to serve models sized compute optimally for this compute (at tens of trillions of total params), except with Ironwood TPUs (which are being built in 2026, for Google and Anthropic) and then Nvidia Rubin Ultra NVL576 (which will only get built in sufficient amounts in 2029, maybe late 2028). So the next step of scaling will probably come in late 2026 to early 2027 from Google and Anthropic (while OpenAI will only be catching up to late 2025 models from Google and Anthropic, though of course in 2026 they'll have better methods than Google and Anthropic had in 2025). And then training compute will still continue increasing somewhat quickly for models until 2029-2031 (with 5 GW training systems, which is at least $50bn per year in training compute, or $100bn per year in total for each AI company if inference is consuming half of the budget). After Rubin Ultra NVL576 (in 2029) and to some extent even Ironwood (in 2026), inference hardware will no longer be a notable constraint on scaling, and after AI companies are working with 10 GW of compute (half for training, half for inference), pretraining compute will no longer be growing much faster than price-performance of hardware, which is much slower than the buildout trend of 2022-2026, and even than the likely ramp-off in 2026-2030. I only expect 2 GW training systems in 2028, rather than the 5 GW that the 2022-2026 trend would ask for in 2028. But by 2030 the combination of continuing buildout and somewhat better hardware should still reach the levels of what would be on-trend for 2028, following 2022-2026.

4adamShimi

How do you review a post that was not written for you? I’m already doing research in AI Alignment, and I don’t plan on creating a group of collaborators for the moment. Still, I found some parts of this useful. Maybe that’s how you do it: by taking different profiles, and running through the most useful advice for each profile from the post. Let’s do that. Full time researcher (no team or MIRIx chapter) For this profile (which is mine, by the way), the most useful piece of advice from this post comes from the model of transmitters and receivers. I’m convinced that I’ve been using it intuitively for years, but having an explicit model is definitely a plus when trying to debug a specific situation, or to explain how it works to someone less used to thinking like that. Full time research who wants to build a team/MIRIx chapter Obviously, this profile benefits from the great advice on building a research group. I would expect someone with this profile to understand relatively well the social dynamics part, so the most useful advice is probably the detailed logistics of getting such a group off the ground. I also believe that the escalating asks and rewards is a less obvious social dynamic to take into account. Aspiring researcher (no team or MIRIx chapter) The section You and your research was probably written with this profile in mind. It tries to push towards exploration instead of exploitation, babble instead of prune. And for so many people that I know who feel obligated to understand everything before toying with a question, this is the prescribed medicine. I want to push-back just a little about the “follow your curiosity” vibe, as I believe that there are ways to check how promising the current ideas are for AI Alignment. But I definitely understand that the audience is more “wannabe researchers stifled by their internal editor”, so pushing for curiosity and exploration makes sense. Aspiring researcher who wants to build a team/MIRIx chapter In additio

Recent Discussion

[Intro to brain-like-AGI safety] 5. The “long-term predictor”, and TD learning

Steven Byrnes

(Last revised: July 2024. See changelog at the bottom.)

5.1 Post summary / Table of contents

Part of the “Intro to brain-like-AGI safety” post series.

In the previous post, I discussed the “short-term predictor”—a circuit which, thanks to a learning algorithm, emits an output that predicts a ground-truth supervisory signal arriving a short time (e.g. a fraction of a second) later.

In this post, I propose that we can take a short-term predictor, wrap it up into a closed loop involving a bit more circuitry, and we wind up with a new module that I call a “long-term predictor”. Just like it sounds, this circuit can make longer-term predictions, e.g. “I’m likely to eat in the next 10 minutes”. This circuit is closely related to Temporal Difference (TD) learning, as we’ll...

(Continue Reading - 5738 more words)

8Eli Tyre19h

Here's my understanding of the model presented in this post. Please me know if I'm missing or misunderstanding anything. 1. The learning subsystem of the brain includes a bunch of short term predictors. These predictors take in information from the all over the brain to predict values in the steering subsystem, in particular. 2. These short term predictors get feedback by sending their predictions down to the steering subsystem, and in return get supervision signals that they can use to learn their policy. 3. The steering subsystem will often just turn around and send the prediction back as the supervision signal. When this happens, it creates a feedback loop of the predictor predicting itself. 4. When the short term predictor is in this self-loop mode, it has degrees of freedom about what state it's in. Any prediction that it offers is a "correct answer". 5. However, the steering subsystem will send a ground truth signal at some point. So the best strategy for a short term predictor in a self-loop mode is to jump to predicting the next override as soon as it can guess what the next override will be. * For instance, as soon as that short term predictor notices a change in context like "you had the thought to go put on a sweater", it immediately starts predicting the positive reward of "feel warm and cozy" from having put on a sweater. * It could self-stably give any prediction. But immediately predicting the reward of putting on a sweater is the best policy. It won't get penalized while it's in the self-loop mode (since whatever the short term outputs is marked correct) but when the ground truth injection does arrive the short term predictor will already be correctly predicting that ground truth injection. * That is, jumping to predicting the next ground truth injection as soon as the predictor can guess what it will be does no worse than any other possible prediction during the self-loop mode, and it does better than other possible predictions du

Steven Byrnes6h20

Yes! 🎉 [And lmk, here or by DM, if you think of any rewording / diagrams / whatever that would have helped you get that with less effort :) ]

Many arguments for AI x-risk are wrong

TurnTrout

The following is a lightly edited version of a memo I wrote for a retreat. It was inspired by a draft of Counting arguments provide no evidence for AI doom. I think that my post covers important points not made by the published version of that post.

I'm also thankful for the dozens of interesting conversations and comments at the retreat.

I think that the AI alignment field is partially founded on fundamentally confused ideas. I’m worried about this because, right now, a range of lobbyists and concerned activists and researchers are in Washington making policy asks. Some of these policy proposals seem to be based on erroneous or unsound arguments.^[1]

The most important takeaway from this essay is that the (prominent) counting arguments for “deceptively aligned” or “scheming” AI...

(Continue Reading - 3507 more words)

Vanessa Kosoy9h60Review for 2024 Review

This is a deeply confused post.

In this post, Turner sets out to debunk what he perceives as "fundamentally confused ideas" which are common in the AI alignment field. I strongly disagree with his claims.

In section 1, Turner quotes a passage from "Superintelligence", in which Bostrom talks about the problem of wireheading. Turner declares this to be "nonsense" since, according to Turner, RL systems don't seek to maximize a reward.

First, Bostrom (AFAICT) is describing a system which (i) learns online (ii) maximizes long-term consequences. There are good reas... (read more)

Steering RL Training: Benchmarking Interventions Against Reward Hacking

ariaw, Josh Engels, Neel Nanda

This project is an extension of work done for Neel Nanda’s MATS 9.0 Training Phase. Neel Nanda and Josh Engels advised the project. Initial work on this project was done with David Vella Zarb. Thank you to Arya Jakkli, Paul Bogdan, and Monte MacDiarmid for providing feedback on the post and ideas.

Overview of the top interventions compared to RL and No Intervention baseline runs. All runs are trained on an environment with a reward hacking loophole except for the RL baseline, which is trained on a no-loophole environment. Statistical significance compared to the RL baseline is indicated by * for values greater and † for values lesser at ɑ=0.01. Successful interventions should show reward hacking rates at or lower than the RL baseline and performance at or

...

(Continue Reading - 5585 more words)

Jozdien1d20

Recent work from Anthropic also showed inoculation prompting is effective in reducing misalignment generalization of reward hacking during production RL^[7]; those results did not investigate test-time performance impacts or learned reward hacking

Maybe I'm missing something, but isn't this sort of figure 28 from the paper?

2Charlie Steiner2d

Sorry, I couldn't find your code easily so I'll just ask: did you merely omit the off-policy part of inoculation prompting in your description of it, or did you also omit it in the code itself?

A Solomonoff Inductor Walks Into a Bar: Schelling Points for Communication

johnswentworth, David Lorell

A Solomonoff inductor walks into a bar in a foreign land. (Stop me if you’ve heard this one before.) The bartender, who is also a Solomonoff inductor, asks “What’ll it be?”. The customer looks around at what the other patrons are having, points to an unfamiliar drink, and says “One of those, please.”. The bartender points to a drawing of the same drink on a menu, and says “One of those?”. The customer replies “Yes, one of those.”. The bartender then delivers a drink, and it matches what the first inductor expected.

What’s up with that?

The puzzle, here, is that the two Solomonoff inductors seemingly agree on a categorization - i.e. which things count as the Unnamed Kind Of Drink, and which things don’t, with at least enough...

(Continue Reading - 3693 more words)

Vanessa Kosoy3d70Review for 2024 Review

This post contains an interesting mathematical result: that the machinery of natural latents can be transferred from classical information theory to algorithmic information theory. I find it intriguing for multiple reasons:

It updates me towards natural latents being a useful concept for foundational questions in agent theory, as opposed to being some artifact of overindexing on Bayesian networks as the "right" ontology.
The proof technique involves defining an algorithmic information theory analogue of Bayesian networks, which is something I haven't seen be

... (read more)

4Vanessa Kosoy4d

O-notation in contexts like this is somewhat ambiguous, because it's not clear what is treated as a constant and what is treated as a variable that grows to infinity. Would it be correct to say that there exist constants A,B>0 that depend on nothing except for the implicit choice of UTM (or maybe they also need to depend on TM?) s.t. K(Λ′|Λ)≤A(ϵ+ϵ′+logK(x))+B

4johnswentworth4d

Yes. In fact, IIRC only B depends on the choice of UTM, A can be a universal constant.

3johnswentworth5d

The strings y_1, ..., y_n are not arbitrary; no natural latent exists over most sets of strings. In order for a natural latent to exist, all the information shared between any strings must be redundant across all strings.

A Conflict Between AI Alignment and Philosophical Competence

Wei Dai

(This argument reduces my hope that we will have AIs that are both aligned with humans in some sense and also highly philosophically competent, which aside from achieving a durable AI pause, has been my main hope for how the future turns out well. As this is a recent realization^[1], I'm still pretty uncertain how much I should update based on it, or what its full implications are.)

Being a good alignment researcher seems to require a correct understanding of the nature of values. However metaethics is currently an unsolved problem, with all proposed solutions having flawed or inconclusive arguments, and lots of disagreement among philosophers and alignment researchers, therefore the current meta-correct metaethical position seems to be one of confusion and/or uncertainty. In other words, a good...

(See More - 465 more words)

3jessicata4d

I think the idea of paying attention to "human values" and "alignment" was based on a philosophical assumption that moral realism would fail, that value is relative / two-place predicate. I am thinking perhaps a more disjunctive approach would be to imagine an AI that in the case of moral realism does the normative thing and in the case of moral non-realism is aligned with humans / humanity / something Earth-local. This assumes there is a fact of the matter about whether moral realism is true. Also, not everyone will like it even if moral realism is true (though not everyone liking something is a usual state of affairs). Even framing it disjunctively like this is perhaps philosophically loaded / hard to define without better concepts. See also Moral Reality Check short story about what if AIs believe moral realism.

Charlie Steiner4d53

This assumes there is a fact of the matter about whether moral realism is true

I am a well known moral realism / moral antirealism antirealist.

But the sort of moral realism that's compatible with antirealism is, as far as I see it, a sort of definitional reification of whatever we think of as morality. You can get information from outside yourself about morality, but it's "boring" stuff like good ethical arguments or transformative life experiences, the same sort of stuff a moral antirealist might be moved by. For the distinction to majorly matter to an AI'... (read more)

Alignment will happen by default. What’s next?

Adrià Garriga-alonso

1mo

This is a linkpost for https://open.substack.com/pub/agarriga/p/alignment-will-happen-by-default

I’m not 100% convinced of this, but I’m fairly convinced, more and more so over time. I’m hoping to start a vigorous but civilized debate. I invite you to attack my weak points and/or present counter-evidence.

My thesis is that intent-alignment is basically happening, based on evidence from the alignment research in the LLM era.

Introduction

The classic story about loss of control from AI is that optimization pressure on proxies will cause the AI to value things that humans don’t. (Relatedly, the AI might become a mesa-optimizer with an arbitrary goal).

But the reality that I observe is that the AIs are really nice and somewhat naive. They’re like the world’s smartest 12-year-old (h/t Jenn). We put more and more RL optimization pressure, and keep getting smarter and smarter models; but they...

(Continue Reading - 1584 more words)

michaelcohen5d10

The "feeling bad about reward hacking" is an artifact of still being regularized too closely to a human-like base model that further RL training would eliminate.

AI ALIGNMENT FORUM
AF

AI ALIGNMENT FORUM
AF

The 2024 Review
Discussion Phase

The 2024 Review

Discussion Phase

AI Alignment Posts

Popular Comments

The 2024 Review
Discussion Phase

The 2024 Review

Discussion Phase

AI Alignment Posts

Popular Comments

Recent Discussion

5.1 Post summary / Table of contents

Introduction

The 2024 ReviewDiscussion Phase

The 2024 Review

Discussion Phase

AI Alignment Posts

Popular Comments

The 2024 ReviewDiscussion Phase

The 2024 Review

Discussion Phase

AI Alignment Posts

Popular Comments

Recent Discussion

5.1 Post summary / Table of contents

Introduction

The 2024 Review
Discussion Phase

The 2024 Review
Discussion Phase