You can only build safe ASI if ASI is globally banned

This is a linkpost for https://www.ettf.land/p/you-can-only-build-safe-asi-if-asi

Sometimes people make various suggestions that we should simply build "safe" artificial Superintelligence (ASI), rather than the presumably "unsafe" kind.^[1]

There are various flavors of “safe” people suggest.

Sometimes they suggest building “aligned” ASI: You have a full agentic autonomous god-like ASI running around, but it really really loves you and definitely will do the right thing.
Sometimes they suggest we should simply build “tool AI” or “non-agentic” AI.
Sometimes they have even more exotic, or more obviously-stupid ideas.

Now I could argue at lengths about why this is astronomically harder than people think it is, why their various proposals are almost universally unworkable, why even attempting this is insanely immoral^[2], but that’s not the main point I want to make.

Instead, I want to make a simpler point:

Assume you have a research...

(See More - 260 more words)

Vladimir_Nesov1d57

If there was a textbook on building safe ASI with instructions that are sufficiently straightforward to execute, people would tend to build safe ASI rather than an extinction/disempowerment ASI. Some AI safety efforts could be thought of as contributions to this hypothetical textbook, making it marginally more real.

unsafe ASI is vastly easier to build than controlled ASI, and is on the same tech path

The point of an ASI ban/pause is to create the time to reduce this gap, until it's sufficiently narrow that competent people can walk across without fallin... (read more)

Anthropic repeatedly accidentally trained against the CoT, demonstrating inadequate processes

Alex Mallen, ryan_greenblatt

It turns out that Anthropic accidentally trained against the chain of thought of Claude Mythos Preview in around 8% of training episodes. This is at least the second independent incident in which Anthropic accidentally exposed their model's CoT to the oversight signal.

In more powerful systems, this kind of failure would jeopardize safely navigating the intelligence explosion. It's crucial to build good processes to ensure development is executed according to plan, especially as human oversight becomes spread thin over increasing amounts of potentially untrusted and sloppy AI labor.

This particular failure is also directly harmful, because it significantly reduces our confidence that the model's reasoning trace is monitorable (reflective of the AI's intent to misbehave).^[1]

I'm grateful that Anthropic has transparently reported on this issue as much as they...

(See More - 949 more words)

Alex Mallen1d80

This also affected Claude Opus 4.7.

1Eye You3d

I agree that this demonstrates inadequate process. Something I haven't seen pointed out yet: in this case, the CoT was irrelevant to the reward model and thus no training against CoT would have occurred. Anthropic says "This latter issue affected ~8% of RL episodes, and was isolated to three specific sub-domains of our environment mix: GUI computer use, office-related tasks, and a small set of STEM environments." I don't see how CoT would be relevant to the reward code for these tasks. It doesn’t matter what Claude thought, what matters is whether it completed the verifiable task. Thus there should be no connection between the CoT and reward, so while sloppy it shouldn’t have had an effect. (Contrast this with a subdomain alignment related or reasoning related. If you're trying to train the model not to lie to the user, then its CoT will be relevant for the reward model.)

4Alex Mallen3d

I think it's quite likely that the CoT was relevant to what the reward model was looking for. For one, it's pretty natural to include some aspects of the model spec in the reward model rubric when judging the trajectory. The model spec would naturally penalize a bunch of stated intent to misbehave in the CoT. And even if the reward model was solely looking for task completion, it's pretty natural for it to penalize stated intent to cheat on the task or do some other unintended thing.

Prompted CoT Early Exit Undermines the Monitoring Benefits of CoT Uncontrollability

Elle Najt, Asa Cooper Stickland, Xander Davies

Code: github.com/ElleNajt/controllability

tldr:

Yueh-Han et al. (2026) showed that models have a harder time making their chain of thought follow user instruction compared to controlling their response (the non-thinking, user-facing output). Their CoT controllability conditions require the models’ thinking to follow various style constraints (e.g. write in lowercase, avoid a word), and they measure how well models can comply with these instructions while achieving a task that requires reasoning. The difficulty models have in controlling their CoT is optimistic for control, since it suggests that schemers would have a hard time hiding malicious reasoning from CoT monitors.

Here, we show that a range of frontier models (Claude Opus 4.6, GPT-5.4, and Gemini 3.1 Pro) can be prompted to “early exit” their CoT and displace reasoning into the response. This undermines...

(Continue Reading - 4200 more words)

Current AIs seem pretty misaligned to me

142

ryan_greenblatt

Many people—especially AI company employees ^[1] —believe current AI systems are well-aligned in the sense of genuinely trying to do what they're supposed to do (e.g., following their spec or constitution, obeying a reasonable interpretation of instructions). ^[2] I disagree.

Current AI systems seem pretty misaligned to me in a mundane behavioral sense: they oversell their work, downplay or fail to mention problems, stop working early and claim to have finished when they clearly haven't, and often seem to "try" to make their outputs look good while actually doing something sloppy or incomplete. These issues mostly occur on more difficult/larger tasks, tasks that aren't straightforward SWE tasks, and tasks that aren't...

(Continue Reading - 7921 more words)

Cole Wyeth2d46

I skimmed after the first section, but I think this is really underselling the case with “oh there’s probably no conscious desire to make their work appear good, just a bunch of subconscious drives.” Everything described here seems to be nicely explained by AIs trying to the best of their ability to do what would be rewarded in training, and seems consistent with either inner alignment to literally getting rewarded (which just isnt achievable at deployment with current capabilities) or inner alignment to something like producing convincing-looking work.

13Wei Dai2d

There are two distinct safety problems with handing off alignment research to AI: 1. Alignment: AI doesn't try very hard, relative to other motivations like appearing to succeed. 2. Capabilities: AI just isn't very good at solving hard alignment problems, making "innocent" mistakes despite trying hard. It could be that the "capabilities problem" here is actually the more serious safety problem, due to being less legible. In other words, the alignment problem is fairly legible because its effects are apparent even in relatively easy tasks we assign to AI, which humans can fairly easily notice, as you and other people here and elsewhere are noticing. (I think this is already contributing a significant amount to the general anti-AI sentiment among the public, because many can see the apparent misalignment in their own AI usage.) But suppose we solve this problem so that AIs seem pretty aligned, but they still can't solve hard alignment problems reliably, only generate solutions that look good to humans and themselves...

3orthonormal2d

It occurs to me that this is a major factor in the persistence of hallucinated answers as well! I previously attributed it mostly to the lack of "I don't know" in its Internet training data, but from the recent AA Omniscience results it appears like applying additional RL is decreasing the willingness to admit ignorance more strongly than it's decreasing the actual frequency of incorrect answers.

Vanessa Kosoy's Shortform

Vanessa Kosoy

2Vanessa Kosoy3d

Above, I compare different decision theories to FDT. At the same time, I claim that in a deeper sense, FDT is ill-defined. One may doubt whether that is a coherent line of reasoning. Therefore, instead of a comparison to FDT, I propose to frame these observations as being about stability to precommitments. Details follow. Definition: A precommitment game is a decision problem in which 1. We are given some (the precommitment policies). We will denote . 2. We are given : for each precommitment , it says to which policy this is precomitting. 3. For any , if then : if a precommitting decision is made, then it is the only decision. 4. Denote , . (Notice that .) We are given a relation s.t. (i) implies and (ii) and implies . This tells us which precommitted outcomes correspond to which unprecomitted outcomes. The restriction of to is called the underlying decision problem of . Definition: An EDT precommitment game is a precommitment game which is also an EDT problem (i.e. extensive form and equipped with ) s.t. the following property holds. Denote (the "external" i.e unprecomittable policies) and . We require that there is some and s.t. 1. is a convex combination of and .[1] 2. is in the convex hull of where ranges over . The underlying decision problem is then an EDT decision problem with the belief . as above with is called formally causal when for any and (Is there a natural generalization without the assumption ? I don't know.) Proposition: EDT is precommitment-stable in formally causal precommitment games. That is, in any such game there is which is EDT-optimal. For example, XOR blackmail can be formalized as an EDT precommitment game which is not formally causal and EDT is not precommitment-stable there (the only optimal policy is precommitting to reject). Definition: [EDIT: The treatment of CDT here is problematic, see child post.] A CDT precommitment game is a precommitment game which is also a CDT problem (in the sense that we are given and formally ca

Vanessa Kosoy3d20

The above treatment of "CDT precommitment games" is problematic: the concept made sense in the context of "FDT to CDT translation" but it's not clear what it's doing here (i.e. what is the first argument?) Here is a better treatment.

Definition: A CDT decision problem is the following data. We have a set of variables and for each we have its range , its set of parents and its kernel

The parent relation must induce an acyclic directed graph. We also have a selected subset of decision variables and a selected subset of outcome variables s.t. . For each ... (read more)

2Vanessa Kosoy7d

Not true. Changing her beliefs in response to Omega's proposal doesn't help her. Imagine that Alice is given a choice between 1. Take a bet that pays +2 if X and -1 if not-X. 2. Take a bet that pays +2 if not-X and -1 if X. 3. Refuse both bets. No matter what probability Alice assigns to X after her update, "normal" Bayesian calculus (really CDT calculus, see below) mandates that she chooses 1 or 2, not 3. [...] I guess this example assumes the gamblers are not allowed to update on the offered bets? (Otherwise it doesn't make sense to me.) Like I said, we don't assume it here. [...] Not really, you're over-indexing on the somewhat outdated 6 year old post you're replying to. It is true that if Alice has a coin that Omega cannot predict, she can come ahead by betting according to the coin. But, as my 1-2-3 example above demonstrates, this is not the core idea. The "modern" formulation of infra-Bayesianism only allows deterministic policies, whereas randomization is modeled by means such as "taking the action to flip a coin". [...] What is actually going on here, this is not a Dutch Book argument against Bayesianism per se, this is a Dutch Book argument against Bayesian CDT. CDT-Alice believes that choosing to bet on X doesn't influence the veracity of X, since there is indeed no physical causal link from the former to the latter (X might even be determined before the bet is offered or made). EDT-Alice can succeed here by noticing that her own choice is correlated with X and therefore the probability of X differs between the "Alice bets on X" counterfactual and the "Alice bets on not-X" counterfactual. So, why is this example interesting beyond other examples that undermine CDT? Mainly, it's just easier to understand how infra-Bayesianism solves the problem here, and in particular we only need (crisp) credal sets rather supradistributions (fuzzy credal sets). Another reason is, the notion of subjective probability is often justified by thinking about bet

ryan_greenblatt's Shortform

ryan_greenblatt

23ryan_greenblatt4d

Inoculation prompting seems a bit scary: * It makes it so AIs that have obviously schemer-like behavior—exhibiting instrumental training gaming while in training (potentially with tons of insane reward hacking and ruthlessness) but then behaving in way that looks great to the company in deployment—are an actively expected/desired result of the training process. So behaving much better when in deployment or when humans are applying close oversight is no longer a concerning warning sign of instrumental training gaming. Instead, it's a desired consequence of inoculation prompting (while also matching exactly how a schemer behaves). * It actively makes instrumental reasoning more salient (via directly instructing the model to instrumentally perform well in training). * It means that your model is by default following a cognitive strategy which is pretty close to being a schemer, so if there are other forces incentivizing scheming, it might be more likely for SGD to reinforce scheming cognition. (But maybe a bunch of the cognition can just be done in the text of the prompt, making it so you at least don't select for thoughtful general purpose instrumental reasoning?) Inoculation prompting might net reduce the chance of scheming via making it so scheming isn't necessarily actively selected for by a combination of the training process and the company's outer iteration loop—which selects for AIs that seem good in deployment potentially despite doing a bunch of reward hacking in training. Another benefit of a variant of inoculation prompting is that it could reduce the incentives for models to end up saliently situationally aware and doing their own instrumental reasoning via directly providing this situational awareness and directly telling the AI to perform well for instrumental reasons. At Redwood, we've historically called this a "shoe argument": you want to prevent a local shoe industry from developing—analogous to the AI more saliently being situationally aware—so

6Fabien Roger4d

I agree it reduces the chance that the without-inoculation-prompt model is a reward seeker. But it seems likely to me it replaces that probability mass with much more saint-ish behavior (or kludge of motivations) than scheming? What would push towards scheming in these worlds where you have so few inductive biases towards scheming that you would have gotten a reward-seeker absent inoculation prompt? I don't think adding an instruction makes SGD more likely to make that cognition unconditional. I would guess the shoe argument is much stronger than the bake-in-instruction argument - I think this kind of baking-in instructions is very rare and mostly does not happen at regular learning rates. I also think many of the issues you are pointing at are mostly solved if you have some part of training where you like your rewards more and where you do without-inoculation-prompt training (e.g. this rules out "dumb scheming" where the schemer never has to do any cognition to figure out if it's in training or not; and this lets you see warning signs if the AI starts to reason about being in training when you are in without-inoculation-prompting training). And I think this is pretty common - e.g. I don't think it would make sense for the honeypots Sonnet 4.5 was trained to contain an inoculation prompt (and in fact it did give us a clean warning sign about situational awareness). (I agree it reduces warning signs / warning shots, though mostly via the mechanism of making AIs more aligned in the short/medium term.) I think inoculation prompting is scary but for a different reason: people might use an inoculation-like prompt in deployment, and the model you get there might have a bunch of very dangerous traits (e.g. be very reward-seeking, play an evil-AI character, etc.). (And this might easily happen accidentally and be hard to detect: I vaguely remember an Owain paper where he shows that even benign-looking system prompts with a format close to the "please write a vulnerabilit

ryan_greenblatt4d52

I agree it reduces the chance that the without-inoculation-prompt model is a reward seeker. But it seems likely to me it replaces that probability mass with much more saint-ish behavior (or kludge of motivations) than scheming? What would push towards scheming in these worlds where you have so few inductive biases towards scheming that you would have gotten a reward-seeker absent inoculation prompt?

The story I'm imagining is that the cognition your AI is doing in training with the inoculation prompt ended up being extremely close to scheming, so it isn'... (read more)

AI Alignment Posts

Popular Comments

AI Alignment Posts

Popular Comments

Recent Discussion

tldr: