Richard Ngo

Ngo and Yudkowsky on alignment difficulty

This post is the first in a series of transcribed Discord conversations between Richard Ngo and Eliezer Yudkowsky, moderated by Nate Soares. We've also added Richard and Nate's running summaries of the conversation (and others' replies) from Google Docs. Later conversation participants include Ajeya Cotra, Beth Barnes, Carl Shulman, Holden Karnofsky, Jaan Tallinn, Paul Christiano, Rob Bensinger, and Rohin Shah. The transcripts are a complete record of several Discord channels MIRI made for discussion. We tried to edit the transcripts as little as possible, other than to fix typos and a handful of confusingly-worded sentences, to add some paragraph breaks, and to add referenced figures and links. We didn't end up redacting any substantive content, other than the names of people who would prefer not to be cited. We swapped the order of some chat messages for clarity and conversational flow (indicated with extra timestamps), and in some cases combined logs where the conversation switched channels. Color key: Chat by Richard and Eliezer Other chat Google Doc content Inline comments 0. Prefatory comments [Yudkowsky][8:32] (Nov. 6 follow-up comment) (At Rob's request I'll try to keep this brief, but this was an experimental format and some issues cropped up that seem large enough to deserve notes.) Especially when coming in to the early parts of this dialogue, I had some backed-up hypotheses about "What might be the main sticking point? and how can I address that?" which from the standpoint of a pure dialogue might seem to be causing me to go on digressions, relative to if I was just trying to answer Richard's own questions. On reading the dialogue, I notice that this looks evasive or like point-missing, like I'm weirdly not just directly answering Richard's questions. Often the questions are answered later, or at least I think they are, though it may not be in the first segment of the dialogue. But the larger phenomenon is that I came in with s

262Nov 15, 2021

Richard Ngo

Message

Formerly alignment and governance researcher at DeepMind and OpenAI. Now independent.

22788

3027

177

1272

13y

Towards a scale-free theory of intelligent agency

I recently left OpenAI to pursue independent research. I’m working on a number of different research directions, but the most fundamental is my pursuit of a scale-free theory of intelligent agency. In this post I give a rough sketch of how I’m thinking about that. I’m erring on the side...

Mar 21, 2025106

Defining alignment research

I think that the concept of "alignment research" (and the distinction between that and "capabilities research") is currently a fairly confused one. In this post I’ll describe some of the problems with how people typically think about these terms, and offer replacement definitions. “Alignment” and “capabilities” are primarily properties of...

Aug 19, 2024123

Coalitional agency

The coalitional frame Earlier in this sequence I laid out an argument that the goals of increasingly intelligent AIs will become increasingly systematized, until they converge to squiggle-maximization. In my last post, though, I touched on two reasons why this convergence might not happen: humans trying to prevent it, and...

Jul 22, 202461

A more systematic case for inner misalignment

This post builds on my previous post making the case that squiggle-maximizers are plausible. The argument I presented was a deliberately simplified one, though, and glossed over several possible issues. In this post I'll raise and explore three broad objections. (Before looking at mine, I encourage you to think of...

Jul 20, 202431

A simple case for extreme inner misalignment

This post is the version of Yudkowsky's argument for inner misalignment that I wish I'd had in my head a few years ago. I don't claim that it's novel, that I endorse it, or even that Yudkowsky would endorse it; it's primarily an attempt to map his ideas into an...

Jul 13, 202486

Measuring Coherence of Policies in Toy Environments

This post was produced as part of the Astra Fellowship under the Winter 2024 Cohort, mentored by Richard Ngo. Thanks to Martín Soto, Jeremy Gillen, Daniel Kokotajlo, and Lukas Berglund for feedback. Summary Discussions around the likelihood and threat models of AI existential risk (x-risk) often hinge on some informal...

Mar 18, 202459

Value systematization: how values become coherent (and misaligned)

Many discussions of AI risk are unproductive or confused because it’s hard to pin down concepts like “coherence” and “expected utility maximization” in the context of deep learning. In this post I attempt to bridge this gap by describing a process by which AI values might become more coherent, which...

Oct 27, 2023110

Load More (7/61)

AI ALIGNMENT FORUM
AF

AI ALIGNMENT FORUM
AF

Richard Ngo

Richard Ngo

Richard Ngo

Ngo and Yudkowsky on alignment difficulty

Realism about rationality

Some conceptual alignment research projects

Alignment research exercises

Richard Ngo

Towards a scale-free theory of intelligent agency

Defining alignment research

Coalitional agency

A more systematic case for inner misalignment

A simple case for extreme inner misalignment

Measuring Coherence of Policies in Toy Environments

Value systematization: how values become coherent (and misaligned)

Ngo and Yudkowsky on alignment difficulty

Realism about rationality

Some conceptual alignment research projects

Alignment research exercises

Towards a scale-free theory of intelligent agency

Defining alignment research

Coalitional agency

A more systematic case for inner misalignment

A simple case for extreme inner misalignment

Measuring Coherence of Policies in Toy Environments

Value systematization: how values become coherent (and misaligned)