AI ALIGNMENT FORUM
AF

AI Alignment Forum

Distinguish between inference scaling and "larger tasks use more compute"

I think there's a hidden variable in this framework: the effective branching factor of the task. For well-specified tasks (prove this theorem, implement this API to spec, etc.), AI cost probably does scale roughly linearly with task size, matching your model. But most real-world engineering tasks aren't like that. The bottleneck isn't execution speed, it's the iterative discovery of what the task actually is. A senior engineer doing a 4-hour task isn't typing for 4 hours. They're making dozens of micro-decisions informed by tacit knowledge about the codebase, the product, the users, the org. They're constantly pruning the search space. An AI working autonomously on the same task produces tokens faster but explores a much larger space of wrong approaches, because it lacks that contextual judgment. So the cost might scale superlinearly with task complexity not because of compute limits but because of specification limits. This suggests the linear regime on your Pareto frontier might be shorter for real-world task distributions than for benchmark-style tasks, and the "fraction of human cost" metric could look very different depending on how well-specified the task is. The tasks where AI efficiency looks great (tight feedback loops, clear success criteria) are not the same tasks that fill most of an engineer's day. (Though note that obviously this could change if AI starts to have more context and better judgement, which would basically mean training a better model.)

Daniel Kokotajlo5d82

It Is Reasonable To Research How To Use Model Internals In Training

> training process is “the most forbidden technique”, including recent criticism of Goodfire for investing in this area. I think this mischaracterizes the criticism. The criticism as I understand it is that Goodfire is planning to help frontier AI companies use model internals in training, in exchange for money. Insofar as they really are planning to do this, then I'll count myself as among the critics, for the classic "but we need interpretability tools to be our held-out test set for alignment" reason. Do you have a link to the criticism you are responding to?

Lukas Finnveden16d92

AI Futures Timelines and Takeoff Model: Dec 2025 Update

Nagy et al (2013) (h/t Carl) looks at time series data of 62 different technologies and compares Wright's law (cost decreases as a power law of cumulative production) vs. generalized Moore's law (technologies improve exponentially with time). They find that they're both close to equally good, because exponential increase in production is so common, but that Wright's law is slightly better. (They also test various other measures, e.g. economies of scale with instantaneous production levels, time and experience, experience and scale, but find that Wright's law does best.) I don't know what they find about semiconductors in particular, but given their larger dataset I'm inclined to prefer Wright's law over Moore's for novel domains. (FYI @Daniel Kokotajlo, @ryan_greenblatt.)

28Vanessa Kosoy

This post is a review of Paul Christiano's argument that the Solomonoff prior is malign, along with a discussion of several counterarguments and countercounterarguments. As such, I think it is a valuable resource for researchers who want to learn about the problem. I will not attempt to distill the contents: the post is already a distillation, and does a a fairly good job of it. Instead, I will focus on what I believe is the post's main weakness/oversight. Specifically, the author seems to think the Solomonoff prior is, in some way, a distorted model of reasoning, and that the attack vector in question can attributed to this, at least partially. This is evident in phrases such as "unintuitive notion of simplicity" and "the Solomonoff prior is very strange". This is also why the author thinks the speed prior might help and that "since it is difficult to compute the Solomonoff prior, [the attack vector] might not be relevant in the real world". In contrast, I believe that the attack vector is quite robust and will threaten any sufficiently powerful AI as long as it's cartesian (more on "cartesian" later). Formally analyzing this question is made difficult by the essential role of non-realizability. That is, the attack vector arises from the AI reasoning about "possible universes" and "simulation hypotheses" which are clearly phenomena that are computationally infeasible for the AI to simulate precisely. Invoking Solomonoff induction dodges this issue since Solomonoff induction is computationally unbounded, at the cost of creating the illusion that the conclusions are a symptom of using Solomonoff induction (and, it's still unclear how to deal with the fact Solomonoff induction itself cannot exist in the universes that Solomonoff induction can learn). Instead, we should be using models that treat non-realizability fairly, such as infra-Bayesiansim. However, I will make no attempt to present such a formal analysis in this review. Instead, I will rely on painting an in

12johnswentworth

This post is an excellent distillation of a cluster of past work on maligness of Solomonoff Induction, which has become a foundational argument/model for inner agency and malign models more generally. I've long thought that the maligness argument overlooks some major counterarguments, but I never got around to writing them up. Now that this post is up for the 2020 review, seems like a good time to walk through them. In Solomonoff Model, Sufficiently Large Data Rules Out Malignness There is a major outside-view reason to expect that the Solomonoff-is-malign argument must be doing something fishy: Solomonoff Induction (SI) comes with performance guarantees. In the limit of large data, SI performs as well as the best-predicting program, in every computably-generated world. The post mentions that: ... but in the large-data limit, SI's guarantees are stronger than just that. In the large-data limit, there is no computable way of making better predictions than the Solomonoff prior in any world. Thus, agents that are influencing the Solomonoff prior cannot gain long-term influence in any computable world; they have zero degrees of freedom to use for influence. It does not matter if they specialize in influencing worlds in which they have short strings; they still cannot use any degrees of freedom for influence without losing all their influence in the large-data limit. Takeaway of this argument: as long as we throw enough data at our Solomonoff inductor before asking it for any outputs, the malign agent problem must go away. (Though note that we never know exactly how much data that is; all we have is a big-O argument with an uncomputable constant.) ... but then how the hell does this outside-view argument jive with all the inside-view arguments about malign agents in the prior? Reflection Breaks The Large-Data Guarantees There's an important gotcha in those guarantees: in the limit of large data, SI performs as well as the best-predicting program, in every compu

Recent Discussion

Prioritizing threats for AI control

ryan_greenblatt

11mo

We often talk about ensuring control, which in the context of this doc refers to preventing AIs from being able to cause existential problems, even if the AIs attempt to subvert our countermeasures. To better contextualize control, I think it's useful to discuss the main threats. I'll focus my discussion on threats induced by misalignment which could plausibly increase existential risk.

While control is often associated with preventing security failures, I'll discuss other issues like sabotaging safety-critical work. I'll list and explain my prioritization of threats. As part of this, I'll argue that rogue internal deployments—cases where the AI is running within the company's datacenter but with control measures disabled—can be substantially worse than self-exfiltration—the AI stealing its weights and running them on an external server—as it might...

(Continue Reading - 2797 more words)

1maxnadeau1d

"When something quite bad can occur in a small number of actions (potentially even in a single trajectory) or in a short period of time, it's important to sometimes block suspicious looking actions until they are reviewed." Why? I think there is not perfect overlap between the categories “single actions that are catastrophic, i.e. high stakes failures” and “actions that require monitoring and blocking in the moment”. One example where they come apart is “producing a batch of data that would cause the next generation of models to be misaligned”—this might be a high-stakes failure, but you have plenty of time to catch it before you actually train that model. Another example would be "writing code that is catastrophically insecure but only gets run rarely".

ryan_greenblatt8h30

Good point, this is an error in my post.

How do we (more) safely defer to AIs?

ryan_greenblatt, Julian Stastny

As AI systems get more capable, it becomes increasingly uncompetitive and infeasible to avoid deferring to AIs on increasingly many decisions. Further, once systems are sufficiently capable, control becomes infeasible. ^[1] Thus, one of the main strategies for handling AI risk is fully (or almost fully) deferring to AIs on managing these risks. Broadly speaking, when I say "deferring to AIs" ^[2] I mean having these AIs do virtually all of the work to develop more capable and aligned successor AIs, managing exogenous risks, and making strategic decisions. ^[3] If we plan to defer to AIs, I think it's safest...

(Continue Reading - 21468 more words)

4Wei Dai20h

Many of my posts, e.g. Some Thoughts on Metaphilosophy, are relevant here and you may already be familiar with my ideas (or can read them if you want). But I have a new thought about this: Current human experts seem to implicitly agree that "ensemble together different epistemic strategies to diversify" is not a good idea, since they each seem to endorse acting based on their own epistemic strategies, e.g., helping AI companies build AGI vs trying to hinder them, or pursuing safety strategies that depend on their own idiosyncratic metaethical positions. If they thought ensembling was a good idea, wouldn't they practice it themselves, i.e., make decisions based on a mix of positions from their peers? So: 1. How do you get the AI company to endorse and implement ensembling, instead of just adopting the epistemic strategies of (or similar to) their own experts (who are presumably less cautious/conservative than average about advancing capabilities, otherwise they'd be outside trying to pause or slow down AI or doing something orthogonal)? 2. Given that human experts seem to have systematic biases (e.g., away from ensembling), wouldn't the ensemble also exhibit such biases? Have you taken this into account when saying "My guess is that the answer is yes"?

ryan_greenblatt8h20

you may already be familiar with my ideas

Yep, I was thinking about outputs from you (in addition to outputs from others) while writing this.

Given that human experts seem to have systematic biases (e.g., away from ensembling), wouldn't the ensemble also exhibit such biases? Have you taken this into account when saying "My guess is that the answer is yes"?

Yes, I'm including this sort of consideration when I say top human experts do would probably be sufficient.

Your point around companies not wanting to ensemble and wanting to use their own epistemic... (read more)

Wei Dai's Shortform

Wei Dai

Wei Dai13h20

Not sure if this is already well known around here, but apparently AI companies are heavily subsidizing their subscription plans if you use their own IDEs/CLIs. (It's discussed in various places but I had to search for it.)

I realized this after trying Amp Code. They give out a $10 daily free credit, which can easily be used up in 1 or 2 prompts, e.g., "review this code base, fix any issues found". (They claim to pass their API costs to their customers with no markup, so this seems like a good proxy for actual API costs.) But with even a $19.99 subscription... (read more)

Research note: A simpler AI timelines model predicts 99% AI R&D automation in ~2032

Thomas Kwa

This is a linkpost for https://metr.org/notes/2026-02-10-simpler-ai-timelines-model/

In this post, I describe a simple model for forecasting when AI will automate AI development. It is based on the AI Futures model, but more understandable and robust, and has deliberately conservative assumptions.

At current rates of compute growth and algorithmic progress, this model's median prediction is >99% automation of AI R&D in late 2032. Most simulations result in a 1000x to 10,000,000x increase in AI efficiency and 300x-3000x research output by 2035. I therefore suspect that existing trends in compute growth and automation will still produce extremely powerful AI on "medium" timelines, even if the full coding automation and superhuman research taste that drive the AIFM's "fast" timelines (superintelligence by ~mid-2031) don't happen.

Why make this?

The AI Futures Model (AIFM) has 33 parameters; this has 8.
- I previously

...

(Continue Reading - 2265 more words)

2romeostevensit2d

Thanks for doing some sensitivity analysis. It's often number one on my list of frustrating things not included in a model.

4elifland2d

Grats on getting this out! I am overall excited about exploring models that rely more on uplift than on time horizons. A few thoughts: It might be nice to indicate how these outputs relate to your all-things-considered views. To me your explicit model seems to be implausibly confident in 99% automation before 2040. I am skeptical that uplift measurements actually give much tighter confidence intervals. After talking to Thomas about this verbally, we both agree that directly using uplift measurements rather than time horizon could plausibly be better by the end of 2026, though we might have different intuitions about the precise likelihood. This isn't true in our model because we allow full coding automation. Given that this is the case in your model, Cobb-Douglas seems like a reasonable approximation.

Thomas Kwa1d10

I am skeptical that uplift measurements actually give much tighter confidence intervals.

I think we might already have evidence against longer timelines (2049 timelines from doubling difficulty growth factor of 1.0). Basically, the model needs to pass a backtest against where uplift was in 2025 vs now, and if there's significant uplift now and wasn't before 2025, this implies the automation curve is steep.

Suppose uplift at the start of 2026 was 1.6x as in the AIFM's median (which would be 37.5% automation if you make my simplifying assumptions). If we also ... (read more)

Have LLMs Generated Novel Insights?

abramdemski, Cole Wyeth, Kaj_Sotala

In a recent post, Cole Wyeth makes a bold claim:

. . . there is one crucial test (yes this is a crux) that LLMs have not passed. They have never done anything important.
They haven't proven any theorems that anyone cares about. They haven't written anything that anyone will want to read in ten years (or even one year). Despite apparently memorizing more information than any human could ever dream of, they have made precisely zero novel connections or insights in any area of science^[3].

I commented:

An anecdote I heard through the grapevine: some chemist was trying to synthesize some chemical. He couldn't get some step to work, and tried for a while to find solutions on the internet. He eventually asked an LLM. The LLM gave a very

...

(See More - 335 more words)

Cole Wyeth1d20

I do, and I’m shifting in that direction.

AI ALIGNMENT FORUM
AF

AI ALIGNMENT FORUM
AF

AI Alignment Posts

Popular Comments

AI Alignment Posts

Popular Comments

Recent Discussion

Why make this?