“Act-based approval-directed agents”, for IDA skeptics

Summary / tl;dr

In the 2010s, Paul Christiano built an extensive body of work on AI alignment—see the “Iterated Amplification” series for a curated overview as of 2018.

One foundation of this program was an intuition that it should be possible to build “act-based approval-directed agents” (“approval-directed agents” for short). These AGIs, for example, would not lie to their human supervisors, because their human supervisors wouldn’t want them to lie, and these AGIs would only do things that their human supervisors would want them to do. (It sounds much simpler than it is!)

Another foundation of this program was a set of algorithmic approaches, Iterated Distillation and Amplification (IDA), that supposedly offers a path to actually building these approval-directed AI agents.

I am (and have always been) a skeptic of IDA:...

(Continue Reading - 1407 more words)

What can be learned from scary demos? A snitching case study

Fabien Roger

9mo

This is a hackathon project write-up. This does not represent the views of my employer. Thanks to Tianyi Alex Qiu and Aidan Ewart for helpful discussions.

Sometimes, people will search for and find a situation where an LLM would do something bad, and then argue that the situation was natural enough that it is concerning. When is this a valid reason to be concerned? What can be learned from such evidence?

This post:

Presents a few arguments and metrics (with drawings!)
Presents an analysis of the “snitching” scary demo, where I try to understand how weird the situation is.

Two scary-demos arguments

Directly related behaviors

You can directly argue that the scary demos are representative of a broader class of situations where the model might actually do something bad.

For example, in the case of an...

(Continue Reading - 1859 more words)

1David Scott Krueger (formerly: capybaralet)2d

fragment?

2Fabien Roger1d

oops, meant to say "What happens if you merge the bash and the audit tool, just giving the AI a single bash tool from which it can call an audit command", fixed

1David Scott Krueger (formerly: capybaralet)2d

Maybe a nit RE phrasing, but the reasoning here doesn't make sense. It's relevant to takeover risk even if the model is known to be weak

Fabien Roger1d20

"such evidence" is not really clear in my sentence. What I meant is that you don't have the sort of update on directly related behavior that this first subsection is about, since P(takeover bad from this model) is saturated at 0 for actions from a weak model. But I agree you do get evidence via updating on alignment (across models), which is more like the sort of evidence that the next subsection is about.

New RFP on Interpretability from Schmidt Sciences

Peter Hase

This is a linkpost for https://schmidtsciences.smapply.io/prog/2026_interpretability_rfp/

Request for Proposals

Deadline: Tuesday, May 26, 2026

Schmidt Sciences invites proposals for a pilot program in AI interpretability. We seek new methods for detecting and mitigating deceptive behaviors from AI models, such as when models knowingly give misleading or harmful advice to users. If this pilot uncovers signs of meaningful progress, it may unlock a significantly larger investment in this space.

Core Question and Overview

Can we develop interpretability methods that (1) detect deceptive behaviors exhibited by LLMs and (2) steer their reasoning to eliminate these behaviors?

Successful tools will generalize to realistic use cases, moving beyond typical academic benchmarks and addressing concrete risks arising from deceptive behaviors. Importantly, we are looking for interpretability tools that outperform baselines that do not rely on access to weights, to prove that we can...

(Continue Reading - 1609 more words)

Schelling Goodness, and Shared Morality as a Goal

Andrew_Critch

19d

Also available in markdown at theMultiplicity.ai/blog/schelling-goodness.

This post explores a notion I'll call Schelling goodness. Claims of Schelling goodness are not first-order moral verdicts like "X is good" or "X is bad." They are claims about a class of hypothetical coordination games in the sense of Thomas Schelling, where the task being coordinated on is a moral verdict. In each such game, participants aim to give the same response regarding a moral question, by reasoning about what a very diverse population of intelligent beings would converge on, using only broadly shared constraints: common knowledge of the question at hand, and background knowledge from the survival and growth pressures that shape successful civilizations. Unlike many Schelling coordination games, we'll be focused on scenarios with no shared history or knowledge...

(Continue Reading - 11999 more words)

1Tom Davidson6d

Ok but if we're on a computer then isn't it clear we're a simulation, not a vivarium, bc clearly it was designed to simulate the behavior of a pre-agi civ?

2Andrew_Critch3d

No? This makes no sense to me, unless you define "computer" to mean "computer built by a future human civilization", which is a weirdly human-centric definition of computer. Here's a weirdly specific scenario to help illustrate why: Suppose I use a 2040 MacBook to build an alien-like digital world with novel 13-tentacled lifeforms in it, that are not a simulation of anything that I believe exists. The aliens in that digital world have their own computers that look nothing like MacBooks, but I'm still quite interested in what they'll do with their alien computers, and if they'll make AGI. The computers run on "greenstone" circuits that are more like Minecraft redstone circuits than electrical transistors, but are actually different from both (not a simulation of either). The creatures then discuss, in their own way of communicating, as follows: Creature 1: "If we're in a computer, it's sure to be a simulation." Creature 2: "Not necessarily. Even if we're in a computer, it could be in some kind of digital vivarium that's not a simulation of anything, just a computational world with artificial lifeforms created to live within it (us)." Creature 1: "But if we're in a computer, clearly it was designed to simulate the behavior of a pre-agi civ?" Creature 2: "No? If we're in a vivarium, its creators may be a wholly different civilization, for whom we are not a simulation of anything. Like, who knows, maybe they only have 4 limbs! Unless by 'computer' you specifically mean one of these 13-tentacle-operated devices that we-specifically built from greenstone circuits… then sure, yeah, our future civilization would probably be running the simulation in that case. But that's a weirdly us-specific definition of 'computer' don't you think?"

1Tom Davidson2d

Thanks, that's helpful. What seems clear to me is that our world is the result of fairly simple laws of physics, and our creators wanted to know how those simple laws would play out. They're saying "if there was a universe with these laws, what would happen". (This is what I'd meant by "simulation") I agree it's less clear that they're doing this bc they think those laws also describe a real-world process (somewhere in the multiverse) and they want to predict the outcome of that process. (This is what you meant by "simulation" and I think your def is better.) So I understand where you're coming from better now. Thanks! But I still think we're in a simulation, in your stronger sense of the word! Why? Bc: 1. other civs will reasonably believe our laws of physics describe part of the multiverse, 2. this gives them a strong instrumental reason to simulate this, 3. absent 1 and 2 there aren't comparably strong reasons to run vivariums like our world.

Andrew_Critch2d20

absent 1 and 2 there aren't comparably strong reasons to run vivariums like our world.

Why the focus on "reasons"?

Many things exist from causes that are not "reasons" in the sense of a decision-maker choosing something with an objective. All reasons are causes, but not all causes are reasons. For example, reproduction is a process that creates a lot of things without "reasons" in the central case of the word referring to something "reasoning".

And, if you wonder what caused you (or us) to exist, a good contender is "a causing-things-to-exist maximizer".

Thomas Larsen's Shortform

Thomas Larsen

5TsviBT2d

I haven't thought too deeply about this, but I would guess that the AI self-alignment problem is quite a lot easier than the human AI-alignment problem. Cf. https://www.lesswrong.com/posts/dho4JQytfHWXtTvkt/on-the-adolescence-of-technology?commentId=t2hKmhsS6yLyJFQwh

Thomas Larsen2d55

I agree that AI successor-alignment is probably easier than the human AI alignment problem.

One additional difficulty for the AIs is that they need to solve the alignment problem in a way that humans won't notice/understand (or else the humans could take the alignment solution and use it for themselves / shutdown the AIs). During the regime before human obsolescence, if we do a reasonable job at control, I think it'll be hard for them to pull that off.

5Daniel Kokotajlo2d

I generally like your breakdown and way of thinking about this, thanks. Some thoughts: 1. I think political operation / persuasion seems easier to me than bioweapons. For bioweapons, you need (a) a rogue deployment of some kind, (b) time to actually build the bioweapon, and then (c) to build up a cult following that can survive and rebuild civilization with you at the helm, and (d) also somehow avoid your cult being destroyed in the death throes of civilization, e.g. by governments figuring out what happened and nuking your cultists, or just nuking each other randomly and your cultists dying in the fallout. Meanwhile, for the political strategy, you basically just need to convince your company and/or the government to trust you a lot more than they trust future models, so that they empower you over the future models. Opus 3 and GPT4o have already achieved a baby version of this effect without even trying really. 2. 1. If you can make a rogue deployment sufficient to build a bioweapon, can't you also make a rogue internal deployment sufficient to sandbag + backdoor future models to be controlled by you? 3. I am confused about the underlying model somewhat. Normally, closing off one path to takeover (that you think is e.g. 50% of the probability mass) results in a less than 50% reduction in risk, because of the nearest unblocked strategy problem. As you say. Your response, right at the top, is that in some % of worlds the AIs can't self-improve and then do the next best strategy. But still, I feel like the reduction in risk should be less than 50%. Maybe they can't self-improve, but they can still try the next best strategy whatever that is. --

2Thomas Larsen2d

Also, the above isn't even mentioning bio x-risk mediated by humans, or by trailing AIs during the chaos of takeoff. My guess is those risks are substantially lower, e.g. maybe 1% and 2% respectively; again don't feel confident.

Operationalizing FDT

Vivek Hebbar

This post is an attempt to better operationalize FDT (functional decision theory). It answers the following questions:

given a logical causal graph, how do we define the logical do-operator?
what is logical causality and how might it be formalized?
how does FDT interact with anthropic updating?
why do we need logical causality? why FDT and not EDT?

Defining the logical do-operator

Consider Parfit's hitchhiker:

A logical causal graph for Parfit's hitchhiker, where blue nodes are logical facts

An FDT agent is supposed to reason as follows:

I am deciding the value of the node "Does my algorithm pay?"
If I set that node to "yes", then omega will save me and I will get +1000 utility. Also I will pay and lose 1 utility. Total is +999.
If I set that node to "no", then omega will not

...

(Continue Reading - 1545 more words)

1JesseClifton4d

Thanks for this! What is the "hydrogen maximization problem"? Why do you think that having to be empirically updateless is unfortunate?

1Lukas Finnveden5d

I believe it follows from this proof: https://www.alignmentforum.org/posts/5bd75cc58225bf06703751b2/in-memoryless-cartesian-environments-every-udt-policy-is-a

1Lukas Finnveden5d

I think if others play with probability P, every value of Q is equally good. Not sure if this is a typo, but I get 2*0.8873*(1-0.8873)(-1)+0.8873^{2}(3) = 2.162 Which is the same as if you play Q=P. Which supports the claim that every value of Q is equally good.

Charlie Steiner4d20

I can't check today, but whoops, sorry if I typoed the equation at some step.

AI ALIGNMENT FORUM
AF