AI ALIGNMENT FORUM
AF

AI Alignment Forum

Recent Discussion

This idea was described in a presentation I have in '23, but wasn't written down anywhere.

Here is a formalization of recursive self-improvement (more precisely, recursive metalearning) in the metacognitive agent framework.

Let be the set of programs or computations represented using some syntax. For example, might be a set of strings that serve as programs for a fixed Universal Turing Machine, but might also be something more structural, like the space of -terms or the space of PCF programs. Let be the space of semantic objects that we assign to computat... (read more)

[Intro to brain-like-AGI safety] 1. What's the problem & Why work on it now?

Steven Byrnes

(Last revised: January 2026. See changelog at the bottom.)

1.1 Post summary / Table of contents

This is the first of a series of blog posts on the technical safety problem for hypothetical future brain-like Artificial General Intelligence (AGI) systems. That previous sentence might raise a few questions, such as: What is “AGI”? What is “brain-like AGI”? What is “the technical safety problem for brain-like AGI”? If these are “hypothetical future systems”, then why on Earth am I wasting my time reading about them right now? …So my immediate goal in this post is to answer all those questions!

After we have that big-picture motivation under our belt, the other 14 posts of this 15-post series will dive into neuroscience and AGI safety in glorious technical detail. See the series...

(Continue Reading - 7962 more words)

Linda Linsefors7h10

Still others of these “brain-like AGI ingredients” seem mostly or totally absent from today’s most popular ML algorithms (e.g. ability to form “thoughts” [e.g. “I’m going to the store”] that blend together immediate actions, short-term predictions, long-term predictions, and flexible hierarchical plans, inside a generative world-model that supports causal and counterfactual and metacognitive reasoning).

I think that chain-of-though planing in an agentic LLM-driven model, might qualify as this. Would you agree?

The Credit Assignment Problem

abramdemski

This post is eventually about partial agency. However, it's been a somewhat tricky point for me to convey; I take the long route. Epistemic status: slightly crazy.

I've occasionally said "Everything boils down to credit assignment problems."

What I really mean is that credit assignment pops up in a wide range of scenarios, and improvements to credit assignment algorithms have broad implications. For example:

Politics.
- When politics focuses on (re-)electing candidates based on their track records, it's about credit assignment. The practice is sometimes derogatorily called "finger pointing", but the basic computation makes sense: figure out good and bad qualities via previous performance, and vote accordingly.
- When politics instead focuses on policy, it is still (to a degree) about credit assignment. Was raising the minimum wage responsible for reduced employment? Was it

...

(Continue Reading - 4879 more words)

Cole Wyeth17h10

Interesting post in light of our discussion at CMU agent foundations 2026, in which I questioned whether (Schurz) meta-inductive justification of induction actually justifies model-based planning as in AIXI, or actually suggests a model-free approach.

The behavioral selection model for predicting AI motivations

Alex Mallen, Buck

4mo

Highly capable AI systems might end up deciding the future. Understanding what will drive those decisions is therefore one of the most important questions we can ask.

Many people have proposed different answers. Some predict that powerful AIs will learn to intrinsically pursue reward. Others respond by saying reward is not the optimization target, and instead reward “chisels” a combination of context-dependent cognitive patterns into the AI. Some argue that powerful AIs might end up with an almost arbitrary long-term goal.

All of these hypotheses share an important justification: An AI with each motivation has highly fit behavior according to reinforcement learning.

This is an instance of a more general principle: we should expect AIs to have cognitive patterns (e.g., motivations) that lead to behavior that causes those cognitive patterns...

(Continue Reading - 4575 more words)

Daniel Kokotajlo2d42

Possibly relevant empirical evidence has arrived!

Also this one here of CoT samples!

Thomas Larsen's Shortform

Thomas Larsen

Thomas Kwa2d10

Difficulty of the successor alignment problem seems like a crux. Misaligned AIs could have an easy time aligning their successors just because they're willing to dedicate enough resources. If alignment requires say 10% of resources to succeed but an AI is misaligned because the humans only spent 3%, it can easily pay this to align its successor.

If you think that the critical safety:capabilities ratio R required to achieve alignment follows a log-uniform distribution from 1:100 to 10:1, and humans always spend 3% on safety while AIs can spend up to 50%, the... (read more)

5TsviBT5d

I haven't thought too deeply about this, but I would guess that the AI self-alignment problem is quite a lot easier than the human AI-alignment problem. Cf. https://www.lesswrong.com/posts/dho4JQytfHWXtTvkt/on-the-adolescence-of-technology?commentId=t2hKmhsS6yLyJFQwh

5Thomas Larsen5d

I agree that AI successor-alignment is probably easier than the human AI alignment problem. One additional difficulty for the AIs is that they need to solve the alignment problem in a way that humans won't notice/understand (or else the humans could take the alignment solution and use it for themselves / shutdown the AIs). During the regime before human obsolescence, if we do a reasonable job at control, I think it'll be hard for them to pull that off.

5Daniel Kokotajlo5d

I generally like your breakdown and way of thinking about this, thanks. Some thoughts: 1. I think political operation / persuasion seems easier to me than bioweapons. For bioweapons, you need (a) a rogue deployment of some kind, (b) time to actually build the bioweapon, and then (c) to build up a cult following that can survive and rebuild civilization with you at the helm, and (d) also somehow avoid your cult being destroyed in the death throes of civilization, e.g. by governments figuring out what happened and nuking your cultists, or just nuking each other randomly and your cultists dying in the fallout. Meanwhile, for the political strategy, you basically just need to convince your company and/or the government to trust you a lot more than they trust future models, so that they empower you over the future models. Opus 3 and GPT4o have already achieved a baby version of this effect without even trying really. 2. 1. If you can make a rogue deployment sufficient to build a bioweapon, can't you also make a rogue internal deployment sufficient to sandbag + backdoor future models to be controlled by you? 3. I am confused about the underlying model somewhat. Normally, closing off one path to takeover (that you think is e.g. 50% of the probability mass) results in a less than 50% reduction in risk, because of the nearest unblocked strategy problem. As you say. Your response, right at the top, is that in some % of worlds the AIs can't self-improve and then do the next best strategy. But still, I feel like the reduction in risk should be less than 50%. Maybe they can't self-improve, but they can still try the next best strategy whatever that is. --

“Act-based approval-directed agents”, for IDA skeptics

Steven Byrnes

Summary / tl;dr

In the 2010s, Paul Christiano built an extensive body of work on AI alignment—see the “Iterated Amplification” series for a curated overview as of 2018.

One foundation of this program was an intuition that it should be possible to build “act-based approval-directed agents” (“approval-directed agents” for short). These AGIs, for example, would not lie to their human supervisors, because their human supervisors wouldn’t want them to lie, and these AGIs would only do things that their human supervisors would want them to do. (It sounds much simpler than it is!)

Another foundation of this program was a set of algorithmic approaches, Iterated Distillation and Amplification (IDA), that supposedly offers a path to actually building these approval-directed AI agents.

I am (and have always been) a skeptic of IDA:...

(Continue Reading - 1406 more words)

3Wei Dai3d

Is this why the smartest humans (e.g. John von Neumann, Terrance Tao) go into math, where verification is definitely easier than generation, instead of fields like philosophy and long-horizon strategy, where plans and outputs are much harder to judge by others? (JvN did do some philosophy and strategy, but surprisingly little relative to his abilities and interests, and I note that his philosophical work, in decision theory, was heavily math flavored.)

Steven Byrnes2d*50

I don’t think this is related to the points I was making in the post … But happy to chat about that anyway.

Yeah sure, common sense says that smart people will tend to enjoy being in more meritocratic intellectual fields, rather than less meritocratic ones, and also that fields in general tend to be more meritocratic when quality is easy to judge (although other things matter too, e.g. glamorous fields have it tougher because they attract grifters).

See e.g. what I wrote here about experimental science.

The mathematics community has successfully kept the cran... (read more)

17Jeremy Gillen

This post deserves to be remembered as a LessWrong classic. 1. It directly tries to solve a difficult and important cluster of problems (whether it succeeds is yet to be seen). 2. It uses a new diagrammatic method of manipulating sets of independence relations. 3. It's a technical result! These feel like they're getting rarer on LessWrong and should be encouraged. There are several problems that are fundamentally about attaching very different world models together and transferring information from one to the other. * Ontology identification involves taking a goal defined in an old ontology[1] and accurately translating it into a new ontology. * High-level models and low-level models need to interact in a bounded agent. I.e. learning a high-level fact should influence your knowledge about low-level facts and vice versa. * Value identification is the problem of translating values from a human to an AI. This is much like ontology identification, with the added difficulty that we don't get as much detailed access or control over the human world model. * Interpretability is about finding recognisable concepts and algorithms in trained neural networks. In general, we can solve these problems using shared variables and shared sub-structures that are present in both models. * We can stitch together very different world models along shared variables. E.g. if you have two models of molecular dynamics, one faster and simpler than the other. You want to simulate in the fast one, then switch to the slow one when particular interactions happen. To transfer the state from one to the other you identify variables present in both models (probably atom locations, velocities, some others), then just copy these values to the other model. Under-specified variables must be inferred from priors. * If you want to transfer a new concept from WM1 to a less knowledgeable WM2, you can do so by identifying the lower-level concepts that both WMs share, then constructing an "expla

AI ALIGNMENT FORUM
AF

AI ALIGNMENT FORUM
AF

AI Alignment Posts

Popular Comments

AI Alignment Posts

Popular Comments

Recent Discussion

1.1 Post summary / Table of contents

Summary / tl;dr