You want your proposal for an AI to be robust to changes in its level of capabilities. It should be robust to the AI's capabilities scaling up, and also scaling down, and also the subcomponents of the AI scaling relative to each other.
We might need to build AGIs that aren't robust to scale, but if so we should at least realize that we are doing that.
If you think reward-seekers are plausible, you should also think “fitness-seekers” are plausible. But their risks aren't the same.
The AI safety community often emphasizes reward-seeking as a central case of a misaligned AI alongside scheming (e.g., Cotra’s sycophant vs schemer, Carlsmith’s terminal vs instrumental training-gamer). We are also starting to see signs of reward-seeking-like motivations.
But I think insufficient care has gone into delineating this category. If you were to focus on AIs who care about reward in particular[1], you'd be missing some comparably-or-more plausible nearby motivations that make the picture of risk notably more complex.
A classic reward-seeker wants high reward on the current episode. But an AI might instead pursue high reinforcement on each individual action. Or it might want to be deployed, regardless of reward. I...
Preface for LessWrong: When I think back on my most cherished memories of this community, I return to those honoring defiance in pursuit of goodness:
I cannot return to you and say “I defied and then I won.” But I’m at least here to say “I defied.”
I recommend reading this article on my website since the embeds and typography work better there: click here.
In January, Department of Homeland Security (DHS) officers...
Your original comment sounded like this was close to the bar, whereas I think the social and political opinions were maybe 10x smaller than the timeless value. I also think that the timeless value of this post is higher than any reasonable version that tried to avoid social opinions. Maybe we agree on this too.
Abstract: In a previous paper, we began the study of sequence prediction algorithms adapted to stringological word complexity measures. One measure we considered was left-to-right (most-significant-digit-first) automaticity. Here, we show a statistically and computationally efficient algorithm adapted to the "dual" right-to-left (least-significant-digit-first) automaticity, which turns out to be substantially different for our purpose. We also demonstrate a prediction algorithm for a more expressive measure that we call "arithmetic repetition complexity". In particular, the latter can be used for predicting the so-called mix-automatic sequences.
This paper continues my sequence on the new approach to compositional learning, which started with "stringological sequence prediction I". Curiously, the ARC complexity measure I define here seems related[1] to my control-theoretic complexity measure for polytope MDPs, even though the initial motivation here came from a completely different automata-theoretic angle[2].
The Huggingface compromise reminds us of the importance of AI control and security practices. Containing models won't always just be about using "auto mode."
I'm literally working on the tool to fix this! I'm building the agent-glovebox tool (previously claude-guard) to make best practices easy. I need beta testers!
From the repo:
...When handling dangerous materials, a chemist doesn't YOLO it barehanded on the open bench. They reach through a glovebox to do their work.
Right now, you're probably barehanding AI, tossing it into a shell onto your machine with ac
(Last revised: June 2026. See changelog at the bottom.)
Part of the “Intro to brain-like-AGI safety” post series.
In the previous post, I discussed the “short-term predictor”—a circuit which, thanks to a learning algorithm, emits an output that predicts a ground-truth supervisory signal arriving a short time (e.g. a fraction of a second) later.
In this post, I propose that we can take a short-term predictor, wrap it up into a closed loop involving a bit more circuitry, and we wind up with a new module that I call a “long-term predictor”. Just like it sounds, this circuit can make longer-term predictions, e.g. “I’m likely to eat in the next 10 minutes”. This circuit winds up being a kind of hybrid between Temporal Difference...
Thanks!
I think this summary relies on us pretending/assuming/modeling "yes, we get an override even if we're currently predicting the right thing". Otherwise, the next override will always be the opposite of what we predict.
Is that assuming that the signals are binary rather than continuous? Let’s say the override ground truth says “produce digestive enzymes at rate R”, where R can vary between 0 and 10. Or to simplify, let’s say the overrides are always saying R=0 or R=10, whereas the predictions can be any real number R. Then it’s never going to predict ...
TLDR: I'm managing a new fund, housed at Lightcone Infrastructure, that will award at least $200,000 in grants and prizes for corrigibility research in 2026. Roughly half will go to traditional grants (first application deadline August 23rd) and half for prizes recognizing excellent work done this year. If you have interest in working on corrigibility, now is a good time to start! Apply via email: grants@corrigibilityresearch.org
When I first dived into AI safety and alignment in 2009, the field was basically nonexistent. I've been relieved and gratified to see attention and funding grow, especially in the past few years. But even now, nearly all AI safety funding goes to evals, control, or interpretability. Work on alignment itself still remains deeply neglected, and it's only through...
Mod note: This is an edge-case of "frontpage or no?". We have a somewhat tricky stance on posts that are giving money to straightforward intellectual labor of a type we generally encourage on LessWrong.
The difference between these and "Job announcement" type posts is something like "how much does it feel like the post is Out To Get You, or slips LW to a world that feels more Out To Get You?". When we imagine people showing up to see a bunch of job ads, it feels like LinkedIn, whereas if a new user sees a frontpage with like 1 bounty for an intellectual problem, it still feels more like an intellectual space.
What do you mean by this?
Also, not to pick on you, but I generally think it's useful to "related work" type things, not just for credit assignment, but also to facilitate understanding. e.g. as someone who is very familiar with alignment, if I can just view "fitness maximizing" as a "goal misgeneralization, but with XYZ" then maybe I can get any key insights in a few minutes.