[Intro to brain-like-AGI safety] 5. The “long-term predictor”, and TD learning

(Last revised: June 2026. See changelog at the bottom.)

5.1 Post summary / Table of contents

Part of the “Intro to brain-like-AGI safety” post series.

In the previous post, I discussed the “short-term predictor”—a circuit which, thanks to a learning algorithm, emits an output that predicts a ground-truth supervisory signal arriving a short time (e.g. a fraction of a second) later.

In this post, I propose that we can take a short-term predictor, wrap it up into a closed loop involving a bit more circuitry, and we wind up with a new module that I call a “long-term predictor”. Just like it sounds, this circuit can make longer-term predictions, e.g. “I’m likely to eat in the next 10 minutes”. This circuit winds up being a kind of hybrid between Temporal Difference...

(Continue Reading - 6916 more words)

3philh3d

Hm, I'm going to nitpick this as a summary, partly as a way of helping me think through this myself. Do we get an override if we're currently predicting right? I think the answer is "it doesn't matter". That is, there'll be some neuronal fact of the matter. But if we treat this set of neurons as a black box, it'll implement the same algorithm either way - the short-term predictor can't tell whether the signal it's getting matches its output because the switch is in defer mode, or because the switch is in override mode and it's overridden to the same value as its current output. I think this summary relies on us pretending/assuming/modeling "yes, we get an override even if we're currently predicting the right thing". Otherwise, the next override will always be the opposite of what we predict. But I'm not sure we can always model it that way. "What's the next override" might depend on "what am I currently predicting" in a way that confuses things. Taking the digestive enzyme example, when do we get an override? According to above: 1. If we're currently eating something: override to "make enzymes". 2. If we've been making enzymes for a while, and not eating: override to "don't make enzymes". [1] We can model (1) as getting an override even if we were predicting enzymes. But I don't see how to model (2) as getting an override even if we were predicting no-enzymes. The "we've been making enzymes for a while" is an important part of it. (What's the fundamental asymmetry here? I think "we're trying to start producing enzymes before we need them, harder than we're trying to stop producing enzymes before we stop needing them".) So if we're not currently predicting enzymes, the next override is always going to be "start making enzymes". But we don't always predict enzymes, because if we did, we'd (typically) get a "no-enzymes" override. So maybe a more-accurate-but-more-confusing summary would be: we're training the short-term predictor to output what the next ove

Steven Byrnes5h20

Thanks!

I think this summary relies on us pretending/assuming/modeling "yes, we get an override even if we're currently predicting the right thing". Otherwise, the next override will always be the opposite of what we predict.

Is that assuming that the signals are binary rather than continuous? Let’s say the override ground truth says “produce digestive enzymes at rate R”, where R can vary between 0 and 10. Or to simplify, let’s say the overrides are always saying R=0 or R=10, whereas the predictions can be any real number R. Then it’s never going to predict ... (read more)

Why I Left Google DeepMind

182

TurnTrout

This is a linkpost for https://turntrout.com/why-i-left-google-deepmind

Preface for LessWrong: When I think back on my most cherished memories of this community, I return to those honoring defiance in pursuit of goodness:

Defying prestigious dogma and searching for raw truth;
Defying social pressure, acting alone to help someone while others watch;
Defying your self-expectations (your “role”), instead searching over lines of cause-and-effect to find a winning pathway;
Defying a powerful foe’s threats, because they only threaten since people like you cave;
Defying the specter of apparent impossibility because you can’t bear to lose.

I cannot return to you and say “I defied and then I won.” But I’m at least here to say “I defied.”

I recommend reading this article on my website since the embeds and typography work better there: click here.

Why I left Google DeepMind

In January, Department of Homeland Security (DHS) officers...

(Continue Reading - 10482 more words)

TurnTrout20h40

Added a footnote:

I know of exactly one other person who left over the deal: René Mayrhofer, a director for Android platform security. "Management has lost its moral compass", he stated. He apparently will serve his notice period from mid-June until August. Brave.

6Raemon1d

Frontpage mod note: This felt slightly edge-casey re: "is it Frontpage?". I think it's pretty timeless and will still be relevant in a few years and am frontpaging it, but wanted to leave some notes about it for people modeling the frontpage rules. The post seems to be doing two things: * Be a detailed recounting / postmortem of a process of internal governmance (i.e. information) * Being kind of social move of "hey, FYI I think Google Deepmind fucked up here and should be socially punished". The latter is a totally fine kind of post to write but makes it somewhat less the sort of thing I want people's "training data for what LW is primarily about" to be dominantly seeded by, which is part of what the frontpage/personal distinction is for. There isn't a hard rule here, but I think "is a significant point of the post to be a social move?" is something that counts at least a bit against frontpageness, on-the-margin (it's not the primary determinant, which is still "timelessness"). Other moderators might have somewhat different takes here, I didn't run this by everyone. (We were behind on moderating new posts and the "frontpage or no?" call here is mostly about whether it'll show up in the recommendations feel in the future. Although it does have enough karma that'll still have some impact on views for the next few days)

3LawrenceC2d

Thank you for writing this, and for standing by your principles.

3Chris_Leong4d

Do you know what your plans are now?

Announcing the Corrigibility Research Fund

Max Harms

TLDR: I'm managing a new fund, housed at Lightcone Infrastructure, that will award at least $200,000 in grants and prizes for corrigibility research in 2026. Roughly half will go to traditional grants (first application deadline August 23rd) and half for prizes recognizing excellent work done this year. If you have interest in working on corrigibility, now is a good time to start! Apply via email: grants@corrigibilityresearch.org

Why this fund exists

When I first dived into AI safety and alignment in 2009, the field was basically nonexistent. I've been relieved and gratified to see attention and funding grow, especially in the past few years. But even now, nearly all AI safety funding goes to evals, control, or interpretability. Work on alignment itself still remains deeply neglected, and it's only through...

(Continue Reading - 1462 more words)

Raemon22h20

Mod note: This is an edge-case of "frontpage or no?". We have a somewhat tricky stance on posts that are giving money to straightforward intellectual labor of a type we generally encourage on LessWrong.

The difference between these and "Job announcement" type posts is something like "how much does it feel like the post is Out To Get You, or slips LW to a world that feels more Out To Get You?". When we imagine people showing up to see a bunch of job ads, it feels like LinkedIn, whereas if a new user sees a frontpage with like 1 bounty for an intellectual problem, it still feels more like an intellectual space.

AI 2040: Plan A

131

Daniel Kokotajlo, elifland, Thomas Larsen, romeo, bhalstead, ryan_greenblatt

12d

This is a linkpost for https://www.ai-2040.com/

For the past year, we at the AI Futures Project have been sinking most of our time into our next big scenario. Now it’s done!

It’s called AI 2040: Plan A.

It’s called Plan A because it’s a recommendation, not a prediction. It’s what we think should happen, not what will happen, though we think it’s plausible enough to aim for.

It’s called AI 2040 because in it, they delay the creation of superintelligence to 2040. It would have happened much sooner (in 2030, to be precise) if not for decisive action on the part of the US and Chinese governments.

As with AI 2027, summaries don’t really do it justice, since the whole point was to be detailed and comprehensive and work things out step by step rather than rely on high-level abstractions like doom or utopia.

Read the scenario at ai-2040.com. You can...

(See More - 68 more words)

6Wei Dai4d

I like that the space governance plan supplement acknowledges that defining torture and slavery are hard problems that ASI could potentially help with: [...] And in the sidenote it gives examples of difficult subproblems or dependencies that would need solved in order to do this, like "What exactly are negatively-valenced experiences?" But I have a couple of problems with the rest of the scenario given this: 1. If there are ASI that are philosophically competent enough to solve these particular problems in this timeframe, wouldn't they likely also be competent enough to solve all or most other important philosophical problems, given just slightly more calendar time? Why don't the delegates use their ASI assistants to solve all of morality and philosophy (or specifically answer "What does the ideal intergalactic civilization look like, given humanity as the starting point?"), and then negotiate the cosmic treaty? 2. AFAICT (it's a bit hard to search the site, due to many collapsed sections), there is no mention in the rest of the plan how the ASI got to be philosophically competent, making it seem like such competence comes with superintelligence for free. I'm pretty sure at least a couple of the authors are aware of and agree (to some extent) with my concern that future AIs may not be philosophically competent by default (and one of my posts laying out this concern was even linked in another supplement), so maybe it's just an oversight?

Thomas Larsen1d32

Thanks for the comment!

1. I think the AIs might be philosophically competent enough to solve ~all the problems, and using the AIs to solve them is basically the right move. We try to make this clear at the start of the epilogue. We wanted to be somewhat more concrete in the scenario than meta level solutions like this, and try to sketch out in concrete detail what the solutions might actually look like, which is why we didn't just stop at "handoff to the AIs", though I do think in practice we should mostly be handing off to the AIs at this point.
2. I agr... (read more)

2Raemon6d

I guess I agree that nonzero of this will happen but I think this requires a kind of weird combination of “strategic about the singularity” and “believing in religious orthodoxy” and that specifically caching out into not evolving at all over time. I expect this to be a pretty small share of the Lightcone given the constraints on who would opt in. And I think over a billion years people who enforcedly believe in Islam or most equivalent things would probably still find ways to create good beautiful/alien-in-the-expected-good-ways tapestry of posthuman experiences. Between those I don’t find myself too worked up about it There maybe can be global rules like “no permacommitments without some degree of awareness of opportunity cost

6Wei Dai5d

Can you explain why it requires “strategic about the singularity”? I think as soon as it's possible to tell one's AI assistant "keep me faithful to X" (and have the AI do a competent job of this), someone among the billions of religious believers is bound to do it, and then the practice will spread via imitation and competitive signaling (similar to other "tech" for enforcing faith, like "hell for non-believers"). This process does not seem to require anyone being strategic. What am I missing?

A Red Line and Oversight Framework for Government AI Contracts

TurnTrout

This is a linkpost for https://turntrout.com/red-line-framework

My post on leaving Google DeepMind tells a story. In contrast, this Framework is a question of mechanism design and negotiation posture. I quite enjoyed optimizing this Framework against its organizational and practical constraints. The original considerations were:

Good red lines: Rule out the questionable use cases (autonomous targeting without human control, untargeted profiling) while allowing trustworthy ones like missile defense. Avoid the weaknesses flagged in legal analysis of Anthropic’s red lines.
Robust red lines: [Google] Cloud would push deals through any loophole, Google Legal seemed unlikely to tighten my drafting, and the Pentagon wouldn’t want terms at all. The language had to hold under pressure, with auditing that respected classification and operational security.
Minimal trust assumptions: I made the Chief Scientist the single root of trust that everything else

...

(Continue Reading - 8661 more words)

Endogenous Alignment

Gordon Seidoh Worley

Starting when children are fairly young, usually around 1 year of age, we adults begin the work of aligning them to our values. We teach them to say “please”, not to hit, to ask for what they want instead of screaming, and much else. We do this primarily via exogenous methods, using a combination of punishments and rewards, that molds their behavior by encouraging good behaviors and discouraging bad ones.

Such operant conditioning works because children have many instinctive behaviors that make them alignable. They want their parents to love them, for their friends to like them, and for almost anyone to help them if they feel they can be trusted. And so combined with exogenous alignment efforts by teachers and peers that continue through the school years,...

(See More - 615 more words)

Gordon Seidoh Worley3d30

Finally got a chance to read @Steven Byrnes recent post https://www.lesswrong.com/posts/rKdS7i4StaMmFzYRo/notes-on-technical-alignment-via-human-like-social-drives after I posted this, and I'd endorse it as the kind of research towards endogenous alignment I think we need more of.

AI Alignment Posts

Popular Comments