AI ALIGNMENT FORUM
AF

AI Alignment Forum

Distinguish between inference scaling and "larger tasks use more compute"

I think there's a hidden variable in this framework: the effective branching factor of the task. For well-specified tasks (prove this theorem, implement this API to spec, etc.), AI cost probably does scale roughly linearly with task size, matching your model. But most real-world engineering tasks aren't like that. The bottleneck isn't execution speed, it's the iterative discovery of what the task actually is. A senior engineer doing a 4-hour task isn't typing for 4 hours. They're making dozens of micro-decisions informed by tacit knowledge about the codebase, the product, the users, the org. They're constantly pruning the search space. An AI working autonomously on the same task produces tokens faster but explores a much larger space of wrong approaches, because it lacks that contextual judgment. So the cost might scale superlinearly with task complexity not because of compute limits but because of specification limits. This suggests the linear regime on your Pareto frontier might be shorter for real-world task distributions than for benchmark-style tasks, and the "fraction of human cost" metric could look very different depending on how well-specified the task is. The tasks where AI efficiency looks great (tight feedback loops, clear success criteria) are not the same tasks that fill most of an engineer's day. (Though note that obviously this could change if AI starts to have more context and better judgement, which would basically mean training a better model.)

Daniel Kokotajlo7d82

It Is Reasonable To Research How To Use Model Internals In Training

> training process is “the most forbidden technique”, including recent criticism of Goodfire for investing in this area. I think this mischaracterizes the criticism. The criticism as I understand it is that Goodfire is planning to help frontier AI companies use model internals in training, in exchange for money. Insofar as they really are planning to do this, then I'll count myself as among the critics, for the classic "but we need interpretability tools to be our held-out test set for alignment" reason. Do you have a link to the criticism you are responding to?

Lukas Finnveden18d92

AI Futures Timelines and Takeoff Model: Dec 2025 Update

Nagy et al (2013) (h/t Carl) looks at time series data of 62 different technologies and compares Wright's law (cost decreases as a power law of cumulative production) vs. generalized Moore's law (technologies improve exponentially with time). They find that they're both close to equally good, because exponential increase in production is so common, but that Wright's law is slightly better. (They also test various other measures, e.g. economies of scale with instantaneous production levels, time and experience, experience and scale, but find that Wright's law does best.) I don't know what they find about semiconductors in particular, but given their larger dataset I'm inclined to prefer Wright's law over Moore's for novel domains. (FYI @Daniel Kokotajlo, @ryan_greenblatt.)

Recent Discussion

Could LLM alignment research reduce x-risk if the first takeover-capable AI is not an LLM?

Tim Hua

1mo

Many people believe that the first AI capable of taking over would be quite different from the LLMs of today. Suppose this is true—does prosaic alignment research on LLMs still reduce x-risk? I believe advances in LLM alignment research reduce x-risk even if future AIs are different. I’ll call these “non-LLM AIs.” In this post, I explore two mechanisms for LLM alignment research to reduce x-risk:

Direct transfer: We can directly apply the research to non-LLM AIs—for example, reusing behavioral evaluations or retraining model organisms. As I wrote this post, I was surprised by how much research may transfer directly.
Indirect transfer: The LLM could be involved in training, control, and oversight of the non-LLM AI. Generally, having access to powerful and aligned AIs acts as a force multiplier

...

(Continue Reading - 1736 more words)

Rohin Shah6h20

In our approach, we took the relevant paradigm as learning + search, rather than LLMs. A lot of prosaic alignment directions only assume a learning-based paradigm, rather than LLMs in particular (though of course some like CoT monitoring are specific to LLMs). Some are even more general, e.g. a lot of control work just depends on the fact that the AI is a program / runs on a computer.

AXRP Episode 48 - Guive Assadi on AI Property Rights

DanielFilan

19h

YouTube link

In this episode, Guive Assadi argues that we should give AIs property rights, so that they are integrated in our system of property and come to rely on it. The claim is that this means that AIs would not kill or steal from humans, because that would undermine the whole property system, which would be extremely valuable to them.

Topics we discuss:

AI property rights
Why not steal from and kill humans
Why AIs may fear it could be them next
AI retirement
Could humans be upgraded to stay useful?
Will AI progress continue?
Why non-obsoletable AIs may still not end human property rights
Why make AIs with property rights?
Do property rights incentivize alignment?
Humans and

...

(Continue Reading - 22268 more words)

Research note: A simpler AI timelines model predicts 99% AI R&D automation in ~2032

Thomas Kwa

This is a linkpost for https://metr.org/notes/2026-02-10-simpler-ai-timelines-model/

In this post, I describe a simple model for forecasting when AI will automate AI development. It is based on the AI Futures model, but more understandable and robust, and has deliberately conservative assumptions.

At current rates of compute growth and algorithmic progress, this model's median prediction is >99% automation of AI R&D in late 2032. Most simulations result in a 1000x to 10,000,000x increase in AI efficiency and 300x-3000x research output by 2035. I therefore suspect that existing trends in compute growth and automation will still produce extremely powerful AI on "medium" timelines, even if the full coding automation and superhuman research taste that drive the AIFM's "fast" timelines (superintelligence by ~mid-2031) don't happen.

Why make this?

The AI Futures Model (AIFM) has 33 parameters; this has 8.
- I previously

...

(Continue Reading - 2265 more words)

2Thomas Kwa2d

I got it from your website: As for the rest, seems reasonable. I think you can't get around the uncertainty by modeling uplift as some more complicated function of coding automation fraction as in the AIFM, because you're still assuming that's logistic, we can't measure it any better than uplift, plus we're still uncertain how they're related. So we really do need better data.

1elifland2d

But in the AIFM the coding automation logistic is there to predict the dynamics regarding how much coding automation speeds progress pre-AC. It doesn't have to do with setting the effective compute requirement for AC. I might be misunderstanding something, sorry if so.

1elifland2d

Re: the 1.6 number, oh that should actually be 1.8 sorry. I think it didn't get updated after a last minute change to the parameter value. I will fix that soon. Also, that's the parallel uplift. In our model, the serial multiplier/uplift is sqrt(parallel uplift).

Thomas Kwa2d20

In my model it's parallel uplift too. Effective labor (human+AI) still goes through the diminishing returns power to get to serial uplift, which I estimate as between roughly 0.1 and 0.3.

Prioritizing threats for AI control

ryan_greenblatt

11mo

We often talk about ensuring control, which in the context of this doc refers to preventing AIs from being able to cause existential problems, even if the AIs attempt to subvert our countermeasures. To better contextualize control, I think it's useful to discuss the main threats. I'll focus my discussion on threats induced by misalignment which could plausibly increase existential risk.

While control is often associated with preventing security failures, I'll discuss other issues like sabotaging safety-critical work. I'll list and explain my prioritization of threats. As part of this, I'll argue that rogue internal deployments—cases where the AI is running within the company's datacenter but with control measures disabled—can be substantially worse than self-exfiltration—the AI stealing its weights and running them on an external server—as it might...

(Continue Reading - 2797 more words)

2maxnadeau3d

"When something quite bad can occur in a small number of actions (potentially even in a single trajectory) or in a short period of time, it's important to sometimes block suspicious looking actions until they are reviewed." Why? I think there is not perfect overlap between the categories “single actions that are catastrophic, i.e. high stakes failures” and “actions that require monitoring and blocking in the moment”. One example where they come apart is “producing a batch of data that would cause the next generation of models to be misaligned”—this might be a high-stakes failure, but you have plenty of time to catch it before you actually train that model. Another example would be "writing code that is catastrophically insecure but only gets run rarely".

ryan_greenblatt2d30

Good point, this is an error in my post.

How do we (more) safely defer to AIs?

ryan_greenblatt, Julian Stastny

As AI systems get more capable, it becomes increasingly uncompetitive and infeasible to avoid deferring to AIs on increasingly many decisions. Further, once systems are sufficiently capable, control becomes infeasible. ^[1] Thus, one of the main strategies for handling AI risk is fully (or almost fully) deferring to AIs on managing these risks. Broadly speaking, when I say "deferring to AIs" ^[2] I mean having these AIs do virtually all of the work to develop more capable and aligned successor AIs, managing exogenous risks, and making strategic decisions. ^[3] If we plan to defer to AIs, I think it's safest...

(Continue Reading - 21468 more words)

4Wei Dai3d

Many of my posts, e.g. Some Thoughts on Metaphilosophy, are relevant here and you may already be familiar with my ideas (or can read them if you want). But I have a new thought about this: Current human experts seem to implicitly agree that "ensemble together different epistemic strategies to diversify" is not a good idea, since they each seem to endorse acting based on their own epistemic strategies, e.g., helping AI companies build AGI vs trying to hinder them, or pursuing safety strategies that depend on their own idiosyncratic metaethical positions. If they thought ensembling was a good idea, wouldn't they practice it themselves, i.e., make decisions based on a mix of positions from their peers? So: 1. How do you get the AI company to endorse and implement ensembling, instead of just adopting the epistemic strategies of (or similar to) their own experts (who are presumably less cautious/conservative than average about advancing capabilities, otherwise they'd be outside trying to pause or slow down AI or doing something orthogonal)? 2. Given that human experts seem to have systematic biases (e.g., away from ensembling), wouldn't the ensemble also exhibit such biases? Have you taken this into account when saying "My guess is that the answer is yes"?

ryan_greenblatt2d20

you may already be familiar with my ideas

Yep, I was thinking about outputs from you (in addition to outputs from others) while writing this.

Given that human experts seem to have systematic biases (e.g., away from ensembling), wouldn't the ensemble also exhibit such biases? Have you taken this into account when saying "My guess is that the answer is yes"?

Yes, I'm including this sort of consideration when I say top human experts do would probably be sufficient.

Your point around companies not wanting to ensemble and wanting to use their own epistemic... (read more)

Wei Dai's Shortform

Wei Dai

Wei Dai2d20

Not sure if this is already well known around here, but apparently AI companies are heavily subsidizing their subscription plans if you use their own IDEs/CLIs. (It's discussed in various places but I had to search for it.)

I realized this after trying Amp Code. They give out a $10 daily free credit, which can easily be used up in 1 or 2 prompts, e.g., "review this code base, fix any issues found". (They claim to pass their API costs to their customers with no markup, so this seems like a good proxy for actual API costs.) But with even a $19.99 subscription... (read more)

9Vanessa Kosoy

This post states a subproblem of AI alignment which the author calls "the pointers problem". The user is regarded as an expected utility maximizer, operating according to causal decision theory. Importantly, the utility function depends on latent (unobserved) variables in the causal network. The AI operates according to a different, superior, model of the world. The problem is then, how do we translate the utility function from the user's model to the AI's model? This is very similar to the "ontological crisis" problem described by De Blanc, only De Blanc uses POMDPs instead of causal networks, and frames it in terms of a single agent changing their ontology, rather than translation from user to AI. The question the author asks here is important, but not that novel (the author himself cites Demski as prior work). Perhaps the use of causal networks is a better angle, but this post doesn't do much to show it. Even so, having another exposition of an important topic, with different points of emphasis, will probably benefit many readers. The primary aspect missing from the discussion in the post, in my opinion, is the nature of the user as a learning agent. The user doesn't have a fixed world-model: or, if they do, then this model is best seen as a prior. This observation hints at the resolution of the apparent paradox wherein the utility function is defined in terms of a wrong model. But it still requires us to explain how the utility is defined s.t. it is applicable to every hypothesis in the prior. (What follows is no longer a "review" per se, inasmuch as a summary of my own thoughts on the topic.) Here is a formal model of how a utility function for learning agents can work, when it depends on latent variables. Fix A a set of actions and O a set of observations. We start with an ontological model which is a crisp infra-POMPD. That is, there is a set of states Sont, an initial state s0ont∈Sont, a transition infra-kernel Tont:Sont×A→□(Sont×O) and a reward functio

7johnswentworth

Why This Post Is Interesting This post takes a previously-very-conceptually-difficult alignment problem, and shows that we can model this problem in a straightforward and fairly general way, just using good ol' Bayesian utility maximizers. The formalization makes the Pointers Problem mathematically legible: it's clear what the problem is, it's clear why the problem is important and hard for alignment, and that clarity is not just conceptual but mathematically precise. Unfortunately, mathematical legibility is not the same as accessibility; the post does have a wide inductive gap. Warning: Inductive Gap This post builds on top of two important pieces for modelling embedded agents which don't have their own posts (to my knowledge). The pieces are: * Lazy world models * Lazy utility functions (or value functions more generally) In hindsight, I probably should have written up separate posts on them; they seem obvious once they click, but they were definitely not obvious beforehand. Lazy World Models One of the core conceptual difficulties of embedded agency is that agents need to reason about worlds which are bigger than themselves. They're embedded in the world, therefore the world must be as big as the entire agent plus whatever environment the world includes outside of the agent. If the agent has a model of the world, the physical memory storing that model must itself fit inside of the world. The data structure containing the world model must represent a world larger than the storage space the data structure takes up. That sounds tricky at first, but if you've done some functional programming before, then data structures like this actually pretty run-of-the-mill. For instance, we can easily make infinite lists which take up finite memory. The trick is to write a generator for the list, and then evaluate it lazily - i.e. only query for list elements which we actually need, and never actually iterate over the whole thing. In the same way, we can represent

AI ALIGNMENT FORUM
AF

AI ALIGNMENT FORUM
AF

AI Alignment Posts

Popular Comments

AI Alignment Posts

Popular Comments

Recent Discussion

Why make this?