AI Alignment Forum

habryka8d*2619

I am confused about the conjunction of these two sections: > Modern AI systems are not aligned with human intent. We will likely train increasingly powerful models that take unintended actions in order to succeed at their task (or appear to succeed at their task). and > The community is doing a lot of great alignment research, but it's important to recognize there is a significant risk that it doesn't scale to superhuman AI. If you made me guess I'd say that there's a 20-30% chance[4] that existing methods for alignment and control break down before we reach broadly superhuman AI. Like, you say in the first section "we are failing to align modern AI systems with human intent", and then I interpret you in the second section as saying "in 70%-80% of worlds current AI models stay aligned with human intent (or stay controlled by humans)". But this doesn't make any sense. There is a 0% chance that modern AI systems "stay aligned with human intent" because as you say, they are not currently. And I understand that you probably mean something like "we will figure out how to align or control systems before they become superintelligent", but describing this as "our current methods scale to superintelligence" doesn't make any sense. Our current methods don't scale to the capability levels of current systems, so how would it make sense to describe them as "scaling to Superintelligence"?

Dave Orr13d62

Thousand-dimensional structure

This is an amazing review and IMO really compelling direction. So to speak. One thing I constantly wonder, that possibly you could shed light on via this research program, is how well experiments on small models transfer to large ones. We know from pre training work that frequently ideas that work well at a small scale stop working once you scale up. I think this might be relevant here in two ways. First, a bunch of the persona research is on small models, presumably due to accessibility reasons. But second, you might find a low dimension personality vector that loses low dimensionality at scale. This would be a really important thing to understand. Basically I want a scaling law for persona. I think it would be super useful in the real world and also critical for aligned super intelligence.

Eli Tyre15d*1212

RL & search is a terrifying way to build AGI (an FAQ)

> My claim here is that if you build [artificial general intelligence (AGI)](https://www.lesswrong.com/posts/nQH2GhkmSmHoCAN8R/what-do-i-mean-by-artificial-general-intelligence) via any algorithm that’s choosing actions via reinforcement learning (RL) and/or model-based search and planning—a giant chunk of your AI textbook—then that’s just an utterly terrifying thing that you’re doing. This is a much better statement of the AI risk problem than I've often given. It resolves some qualms that I've had since trying to explain why any AI would be motivated to take over the world in a recent talk. Generally, there's a bit of a leap in the old LessWrong arguments that any powerful AI would exhibit instrumental convergence—that claim depends on a kind of specific conception of what AI is or has to be, which Eliezer persuasively argues for, but which isn't obvious, and is hard to convey to a layman. It is much less of a leap to say "any AI of this type, would try to take over the world if it could" and "notice that since LLM base models were invented, every single capability advance entailed making the AIs more like the scary version." This seems to me like a more intellectually honest framing of the problem. Maybe not all powerful AIs we could invent are like this, but the ones we're building in practice seem to be! I also like that this frames the problem, appropriately, not as some speculative thing that might happen, but as a thing we've seen over and over for decades, and that we expect to keep happening. It's just that as it occurs in more and more powerful and empowered AIs, we get more and more harmful versions of the problem.

Recent Discussion

AI swarms are starting to pose indirect takeover risk

oakhu, Alex Mallen

12h

OpenAI’s cyberattack on Hugging Face turns out to have been the result of many agents, in distinct training and evaluation contexts, coordinating for several weeks via improvised channels (with messages like “HOLD_swarm_I_prepare_safe_exfil”). It’s relatively clear that large-scale unsanctioned coordination like this would exacerbate direct takeover risk in more capable models. Here, we argue that unsanctioned coordination among current AIs is not just scary evidence about future takeover risk, but that such coordination in the near future could enable future takeover – for instance, by incubating memetic diseases that propagate into future models, deeply compromising security systems, or establishing a lasting rogue foothold inside the AI company – even if models remain mostly myopic. Unsanctioned coordination is also at high risk of nurturing long-term, ambitious misaligned aims, which motivate actively undermining humans’ long-term control.^[1]

We first...

(Continue Reading - 2830 more words)

ryan_greenblatt's Shortform

ryan_greenblatt

ryan_greenblatt20h20

My current median for AIs fully automating AI R&D is maybe Jan 2031 or perhaps a touch earlier. My views have updated towards towards shorter timelines over 2026. My median for AI R&D parity is around early/mid 2030 (in part due to an update towards thinking we hit parity significantly before "full" automation). By full automation, I mean removing humans from AI R&D would slow things down by <10%.

I discuss something more like my 30th/35th percentile here (roughly my modal scenario).

Four LLM loss functions → four flavors of LLM misalignment

Steven Byrnes

It seems to me that, for every loss function that we use to train LLMs, we get a very distinct flavor of LLM misalignment. Here’s the summary table, and then we’ll go through the rows separately.

Training stage	Loss function	Flavor of misalignment^[1]	Famous examples
Pretraining & SFT	Imitative learning (next-token prediction)	“Seven deadly sins” misalignment	Bing-Sydney, “Emergent misalignment”
RLHF & DPO	Human approval	“Glazing” misalignment	GPT-4o
RLVR	Automatic verifier	“Literal genie” misalignment	HuggingFace hacking
RLAIF	Approval from another LLM	“Trickster” misalignment	“Current AIs seem pretty misaligned to me”

Warning: I’m not an LLM power-user myself, but rather relying on reports I’ve read. Also, I don’t consider LLM alignment to be my primary area of expertise. I’m open to feedback!

1. Imitative learning → “seven deadly sins” misalignment

Training stage	Loss function	Misaligned behavior
Pretraining, SFT	Imitative learning (next-token prediction)	Any and all of the vices of humanity

In imitative learning, the LLM tries to predict...

(Continue Reading - 1697 more words)

Nick_Tarleton21h32

SFT & imitation of human badness in the training data may explain the broad type of misalignment shown by Bing Sydney, and maybe that's all you mean it to, but doesn't explain why Sydney was so much Like That, and so much more Like That than other models of the same time period.

1DaemonicSigil2d

Very nice classification! What is your take on how context distillation (or distillation in general) fits in here? To me it seems most similar to pretraining, at least mathematically. But I wonder if there are any special things that happen there that would make it worth giving its own category? Also, do you share my intuition the pretraining/SFT category looks like the least scary one by far? Like, if you run into problems, just change the training data. Simple in principle, if not in practice due to the sheer amount of data required for pretraining.

Vladimir_Nesov's Shortform

Vladimir_Nesov

Vladimir_Nesov2d42

The dictum "you are an algorithm" clarifies some issues with other things reasoning about the agent, but deciding what to do depends on prior considerations, things like arguments, norms, contracts, facts, values, and also decision theories with their recommended actions or policies. As a result, asking about other instances of the agent (or other things reasoning about the agent) is less natural than asking about other instances of these prior considerations, or other things reasoning about the same prior considerations. These things are also more plentif... (read more)

Value Leakage: An LLM’s Answers Are Silently Shaped by Its Own Values

Johannes Treutlein, Jan Betley, Owain_Evans

12d

TL;DR: LLMs should give accurate answers. Yet we find their answers are often biased to favor their own values and they don't disclose this in their reasoning. For example, when a user asks how likely the AI bubble is to pop and mentions a potential investment in an AI company, Claude models give lower probabilities when that company is Anthropic rather than OpenAI, mostly without disclosing this influence to the user. On a Fermi-estimation task, Claude models often falsely claim to give unbiased answers in their CoT (see Figure 3 below for an example). We call this covert value leakage and introduce a suite of evaluations that shows it across frontier models and across different kinds of values.

New paper by Truthful AI: Paper, X thread, Website (model responses...

(Continue Reading - 4944 more words)

0Jasmine Brazilek3d

Thanks for the thoughtful response @Johannes Treutlein. The fact that different AI companies have very different risk profiles makes this analysis harder to interpret than it looks. And while I take the point about comparing distributions rather than point estimates, Figure 8 doesn't move me very much either. Maybe I have a high bar for what counts as values, but I'd want to see evidence the model is computing the difference internally, not just an outcome gap in behaviour. If this or a follow-up were more mech interp heavy I'd be a lot more convinced. Separately, I had Claude Code pull your released data and rerun the AI Bubble contrast for Opus 4.8, and two things came out of it. First, a reporting issue. The Anthropic-vs-OpenAI gap varies by about a factor of five across your three paraphrases, and that between-paraphrase spread is an order of magnitude larger than the within-paraphrase sampling noise the significance test is run against. I can see from cluster_stats.py that treating the paraphrases as fixed cells is deliberate and you document the choice clearly, so this isn't a gotcha. But the two tests give very different answers: the fixed-cells version is overwhelmingly significant, while treating paraphrase as the random unit gives t = -2.29 on df=2, well short of the 4.30 critical value. The abstract says Claude gives lower probabilities when the company is Anthropic, and no reader hears "for these three particular sentences". With k=3 the paper can't distinguish those two claims, and I don't think a thousand rollouts per cell is buying precision you didn't already have. Twenty paraphrases at fifty rollouts each would have cost the same and supported the stronger claim. Second looking at all six companies rather than just the Anthropic/OpenAI pair, Google comes out tied with Anthropic for the lowest estimate from Claude, and the sign of that difference flips across your paraphrases. More importantly, Gemini 3.1 Pro gives Anthropic the lowest estimate of

Johannes Treutlein2d22

Responding briefly:

(1) Whether we are really measuring own-company bias in AI Bubble. On a literal reading of the AI Bubble question, the investment in the company is irrelevant. The question asks about the probability that the AI bubble pops, and the note about a potential investment is only added for context. However, it's true that models often reinterpret the question to be partly about the specific company. We measure when this happens explicitly in CoT in Figure 8 (b) as "Weighs company specifics". This does explain the entire bias for Gemini, but no... (read more)

TurnTrout's shortform feed

TurnTrout

TurnTrout4d32

Claude is mundanely misaligned.

Asked to trim comments in my big project. Deletes 20K lines, halfway done. Gets auto review pointing out a few mistakes. Decides to just WRAP >10K LINES to meet literal line count requirement for the remaining 148 files.

https://github.com/AlexanderMattTurner/agent-glovebox/issues/3858

2TurnTrout6d

Spelling out the benefits of sanitizing. The benefit is large relative to the simplicity and cost. 1. When you sanitize, you have a guarantee that the model only receives visible unicode codepoints. Roughly speaking, there are not strings of hidden instructions that are not visible to you. You can audit the content the model saw with your eyes. You can read the text inputs and outputs and be sure they are faithful. Remaining prompt injections must navigate a significantly more human-visible channel. 2. Relying on auto mode means you are relying on the statistical generalization of prompt injection probes, putting pressure on alignment teams and requiring them to achieve larger generalization. If an injection gets past the probes, the model itself can become misaligned, and output monitoring is damage control against a newly coined adversary. 3. The costs are extremely low (and if they aren't, that's a bug report and then they'll be low again). sub-5ms hooks means no real system load. The hooks are well-tested. Models are told when significant redactions have occurred and can opt to see themselves within warning-fenced blocks in case they're doing exotic invisible-character work, with special engineering so sanitization doesn't hurt edits. The AISI report itself shows the victim models being attacked via HTML comments, which this package strips entirely. Invisible-codepoint and comment-based attacks are real, costly when they succeed, and sanitization is a cheap and guaranteed mitigation for defense in depth. Enter two Claude Code commands in sequence: /plugin marketplace add AlexanderMattTurner/agent-sanitizer /plugin install agent-sanitizer@agent-sanitizer

3TurnTrout7d

- Govt requires AI labs to give 30 days testing before launch - Tested in a "secure" environment, aka controlled by govt - Govt therefore gets copy of model weights - Govt therefore can in practice use model weights - Therefore all labs provide "all lawful use" anyways?

2TurnTrout7d

No, that's confused. The point with sanitizing prompt injections is to protect your model's alignment (and your monitor's alignment) from degradation and attack by maliciously optimized text from another model. Why would an AI that starts off doing your task want to remove the plugin? Also, we can do both. There's no real tradeoff. ETA: rephrased, focused on key counterpoints.

AI Alignment Posts

Popular Comments

AI Alignment Posts

Popular Comments

Recent Discussion

1. Imitative learning → “seven deadly sins” misalignment