AI Alignment Forum

When we get explicit strong generalisation to work (see the first post on the matter and the second) my dream would be to create pre-aligned generalising AIs.

Think about the usual conflict between alignment and capabilities, between doing the right thing and doing the easy thing. The standard narrative puts the good people at a constant disadvantage: they have to carefully plan every AI advance, always on the lookout for potential dangers. While those who don’t care can just YOLO and let it rip and let their AIs get ever more powerful without taking any responsibility.

Now, in reality, there is some nuance to the story; but I don’t want to nuance it, I want to turn it on its head. I want to create AIs so that the...

(See More - 908 more words)

Challenge: Hand coding weights for efficient sequence memorisation

Linda Linsefors, Lucius Bushnaq

We hand coded weights for one layer MLPs that memorises labels for input token sequences of length two. The number of facts our hand-coded models can memorise with 90% accuracy^[1]scales roughly linearly with the models' parameter count^[2], just like trained models for the same architecture. However, our hand-coded models' scaling prefactor still falls short of trained models' by a factor of . A hybrid solution in which we handcode the MLP input weights and learn the MLP output weights, which amounts to a linear classification problem^[3], falls short by a factor of .

**Figure 0:** Max number of facts each model can learn when requiring 90% accuracy on evaluation. The fitted lines are best fit for solving for .

We pose a challenge to the community: Find better...

(Continue Reading - 5231 more words)

Logan Riggs4h10

Cool challenge! I ran some experiments on bilinear layers ie

A couple notes:

They perform ~as well as SwiGLU layers in memorization capacity (the nonlinear equivalent)
There's a closed form solution for bilinear layers, but it's basically SGD (it relates to the fixed points found by KKT, though I can include a better explanation later). This efficient solution basically relies on the facts being assumed by other facts in a connected web of facts (where adding a new fact loosens the web, then pulls all other facts taut, so you have to update everything learne

Steven Byrnes

Q1: What are you saying?

A: My claim here is that if you build artificial general intelligence (AGI) via any algorithm that’s choosing actions via reinforcement learning (RL) and/or model-based search and planning—a giant chunk of your AI textbook—then that’s just an utterly terrifying thing that you’re doing. You’re playing around with algorithms that, if they work at all, would tend to create ruthless, callous AGIs, AGIs which would happily exterminate humanity and run the world by themselves, given an opportunity.

Mercifully, large language models (LLMs) today are not in the category of “algorithms that choose actions via RL & search”. At least, not primarily—see LLMs are (still) mostly powered by imitative learning, not RL. So LLMs are outside the scope of this post. However, lots of other researchers and companies around...

(Continue Reading - 4039 more words)

7Eli Tyre2d

This is a much better statement of the AI risk problem than I've often given and that it resolves some qualms that I've had since trying to explain why any AI would be motivated to take over the world, in a recent talk. Generally, there's a bit of a leap in the old LessWrong arguments that any powerful AI would exhibit instrumental convergence—that claim depends on a kind of specific conception of what AI is or has to be, which Eliezer persuasively argues for, but which isn't obvious, and is hard to convey to a layman. It is much less of a leap to say "any AI of this type, would try to take over the world if it could" and "notice that since LLM base models were invented, every single capability advance entailed making the AIs more like the scary version." This seems to me like a more intellectually honest framing of the problem. Maybe not all powerful AIs we could invent are like this, but the ones we're building in practice seem to be! I also like that this frames the problem, appropriately, not as some speculative thing that might happen, but as a thing we've seen over and over for decades, and that we expect to keep happening. It's just that as it occurs in more and more powerful and empowered AIs, we get more and more harmful versions of the problem.

4Steven Byrnes2d

My point for this post is that we need a plan for RL & search AGI, and that we don’t have one. A plan might involve a certain starting training environment in addition to a certain reward function. Sure. If you’re suggesting that, as long as the training environment is good (e.g. a loving family with adequate socializing), then we can use some straightforward reward function (e.g. “reward when the supervisor presses the ‘approve’ button”), and the AI will still wind up being genuinely nice, then I strongly disagree. E.g. plenty of human sociopaths grow up in loving families; a paperclip maximizer would maximize paperclips regardless of its childhood environment; and when people raise undomesticated animals as beloved pets, they often wind up getting mauled. So I think the reward function has to be a central part of the plan, and I tend to emphasize that. But yeah, whenever I say “What reward function would lead to a nice AGI?”, that’s shorthand for (e.g. here) “What reward function (along with training environment and other design choices) would lead to a nice AGI?”. See Intro series §12.5 for further discussion. [...] This is kinda off-topic, but I’ll respond anyway. I think the motivation to act aggressively (in certain contexts) is innate in humans, just as it’s innate in probably all complex animals. So we don’t need to explain from scratch how people develop a motivation to bully each other. The motivation is already there, and might or might not come out depending on their personality, their mood, their relationship to the potential target, and lots of other mitigating and aggravating factors. By contrast, we do need to explain how someone might develop a motivation to, I dunno, get a cupcake tattoo. There’s no innate drive that makes it feel satisfying to get a cupcake tattoo. Instead, the explanation would probably be that a cupcake tattoo is the kind of thing that this person’s idols (real or imagined) would think is cool.

3StanislavKrym2d

My main issue is with your post as-written is the response to question Q6. The alternative C has researchers attempt to maximize the output of a learned classifier, causing the optimum to go out of distribution into pathological craziness. Suppose, on the other hand, that the classifier is itself capable and that the smallest capabilities level required to jailbreak the classifier is higher than the classifier's. For example, if the ECI of the classifying LLM is , while the smallest ECI causing pathological madness is then an AI with ECI of becomes able to trust the AI who was trained to optimize its judgement until reaching the ECI of , who in turn can trust the AI with the ECI of , etc.

Steven Byrnes22h32

This is good concrete brainstorming, thanks.

Suppose, on the other hand, that the classifier is itself capable and that the smallest capabilities level required to jailbreak the classifier is higher than the classifier's.

I don’t think that’s a good assumption. My normal assumption is that if you crank a pretty basic learning algorithm against a learned classifier, you’ll find an adversarial policy in short order. For example, a paper from 2022 achieved “a >97% win rate against KataGo running at superhuman settings … [u]sing less than 14% of the compute u... (read more)

Research directions in condensation: varieties of objectivity

SamEisenstat

This is the first part of a survey of various ways that I’d like to see work on the theory of condensation develop. Condensation is a mathematical theory dealing with the organization of descriptions of the world into conceptual parts; some of the existing work on it is presented in the paper (Eisenstat 2025). This sequence will draw from that paper the definitions of random variable models and latent variable models, the notation for indexing subfamilies of variables in such models, and the objectivity theorem, Theorem 6.8. In the condensation paper, Section 3, Ideas, gives an overview of all of this. For more context on condensation, readers can refer to LessWrong posts including (Demski 2025, 2026; Gillen and Chiang 2026; Kirchner 2026). The first two posts of...

(Continue Reading - 3374 more words)

AI Alignment Posts

Popular Comments

AI Alignment Posts

Popular Comments

Recent Discussion

Q1: What are you saying?