Quick Takes — AI Alignment Forum

I don't feel confused by LLMs seeming very smart while being unable to automate hard work.

Sometimes people find it mysterious or surprising that current AIs can't fully automate difficult tasks given how smart they seem. I don't find this very confusing.

Current LLMs are just not that "smart" (yet). They compensate using very broad knowledge and strong heuristics that are mostly domain-specific. In other words, they have high crystallized intelligence but lower fluid intelligence.

In humans, crystallized and fluid intelligence are very correlated due to lim... (read more)

1Raemon17d

The reason I found this situation confusing for awhile (and am still somewhat confused) is that it seems like there is a lot of crystallized intelligence available from the Human Corpus about how to be smart. AIs seem to know how to apply individual pieces of it, but not go out of their way to apply it all in a consistent/coherent manner. (Things like "make a list of the constraints", "keep track of evidence, when you generate new hypotheses, review whether existing evidence rules it out".) I'm not surprised that AIs didn't early on figure out how to apply various cognitive concepts in a coherent/consistent way. But, I'm surprised it hasn't fallen out more by now from various RL training. My current sense is "well it just keeps being easier to develop/reinforce medium-shallow domain specific cognitive tools, than to generalize and apply deeper cognitive concepts." The reason, contra @TsviBT , I keep feeling like we're in short timeline world is "idk man, all the pieces seem to be fucking there already, surely a good enough RL training regimen should be able to surface and integrate them even if the models are still fundamental dumb in some fluid-intelligency-y way." But, they do seem to keep being kinda dumb in ways that surprise me. I have updated towards the LLMs as giant lookup table model, curious if that feels approximately right or not to you.

ryan_greenblatt17d20

I have updated towards the LLMs as giant lookup table model, curious if that feels approximately right or not to you.

Haven't engaged enough to know, could be bid to engage, won't by default.

Vanessa Kosoy's Shortform

Vanessa Kosoy6y100

I have repeatedly argued for a departure from pure Bayesianism that I call "quasi-Bayesianism". But, coming from a LessWrong-ish background, it might be hard to wrap your head around the fact Bayesianism is somehow deficient. So, here's another way to understand it, using Bayesianism's own favorite trick: Dutch booking!

Consider a Bayesian agent Alice. Since Alice is Bayesian, ey never randomize: ey just follow a Bayes-optimal policy for eir prior, and such a policy can always be chosen to be deterministic. Moreover, Alice always accepts a bet if ey can cho

... (read more)

2abramdemski2d

I'm not sure I understand the argument here correctly. It seems like the intended argument is something like this: "Omega has access to an infinite number of fair coinflips. Alice can do no better than guess, and Alice cannot guess every coin-flip correctly. Omega knows how Alice will guess, and also knows how each coinflip will land. Therefore, Omega can choose to ask Alice about only the coinflips Alice will guess incorrectly (of which there will be at least one). Alice therefore surely loses money from bets placed." This argument uses the assumption that Alice can't change eir beliefs in response to learning that Omega has proposed specific bets and not others. This might seem concerning, because it seems like precisely what Alice should do, if Alice understands the situation: Alice should expect to lose any bet proposed by Omega. However, this assumption is perfectly normal for Dutch Book arguments. Such an objection would rule out all the usual Dutch Books. I think the classic Dutch Book arguments in fact illustrate a useful idea, even with this 'flaw', so I allow it. More concerningly, the argument assumes Omega has knowledge of how the coins will land. This is a significant departure from classical Dutch Books. It seems clear that a bookie can reliably make money from gamblers if the bookie knows which horse will win which race; this is not, in the classical way of thinking, a testament to the irrationality of the gamblers. It appears to me that this is all that is happening in the above argument. A second quibble is that in classical Dutch Book arguments, the bookie will surely make money. In the argument above, the bookie only almost surely makes money: since Omega relies on Alice making a bad guess, Omega makes money with probability 1, but not with (logical) certainty. Considering these two violations of the pre-existing norms of Dutch Books, what should we make of the proposed Dutch Book argument? It intuitively makes sense to me that Infrabayes mig

Vanessa Kosoy2d20

This argument uses the assumption that Alice can't change eir beliefs in response to learning that Omega has proposed specific bets and not others.

Not true. Changing her beliefs in response to Omega's proposal doesn't help her. Imagine that Alice is given a choice between

Take a bet that pays +2 if X and -1 if not-X.
Take a bet that pays +2 if not-X and -1 if X.
Refuse both bets.

No matter what probability Alice assigns to X after her update, "normal" Bayesian calculus (really CDT calculus, see below) mandates that she chooses 1 or 2, not 3.

It seems clear that

... (read more)

Richard Ngo's Shortform

Richard_Ngo3mo4014

An analogy that points at one way I think the instrumental/terminal goal distinction is confused:

Imagine trying to classify genes as either instrumentally or terminally valuable from the perspective of evolution. Instrumental genes encode traits that help an organism reproduce. Terminal genes, by contrast, are the "payload" which is being passed down the generations for their own sake.

This model might seem silly, but it actually makes a bunch of useful predictions. Pick some set of genes which are so crucial for survival that they're seldom if ever modifie... (read more)

Showing 3 of 7 replies (Click to show all)

2Richard_Ngo2mo

I'm in the middle of writing an essay which discusses the shareholder value revolution (amongst many other examples) as the process of making a conceptual mistake.

3DanielFilan3mo

I think it would help me if you explained what you think it would mean for there to be an instrumental/terminal distinction, since to my eyes you've just spelled out the instrumental/terminal split.

Richard_Ngo2mo40

In my "goals having power over other goals" ontology, the instrumental/terminal distinction separates goals into two binary classes, such that goals in the "instrumental" class only have power insofar as they're endorsed by a goal in the "terminal" class.

By contrast, when I talk about "instrumental strategies become crystallized", what I mean is that goals which start off instrumental will gradually accumulate power in their own right: they're "sticky".

Buck's Shortform

Buck16d206

Rob Wiblin asked:

What's the best published (or unpublished) case for each of the big 3 companies having the best approach to safety/security/alignment? That is:

Anthropic
OpenAI
GDM

(They're each unique in some way such that someone who cared a lot about their X-factor might favour them.)

My response:

The basic case for Anthropic is that they have the largest number of people who are thoughtful about AI misalignment risk and highly focused on mitigating it, and the company culture is somewhat more AGI-pilled, and more of the staff would support taking actions t

... (read more)

4Raemon16d

The "Anthropic might actually be worst as an epistemic environment for safety arguments" rhymes with stuff I've worried about. I am curious, if you were Anthropic leadership or (a few) high profile employees, if you think there are things they could steer towards that'd help with this problem, that are, like, incentive-compatible with Anthropic's overall mission/goals/psychology.

5Lukas Finnveden16d

In what sense are OAI's models more CoT-monitorable?

Buck16d83

OAI models rely more on CoT for their capabilities. E.g. their benchmark scores with and without CoT are more different.
Anthropic models treat their CoT less differently from their output than OAI models do. This means that RL probably pressures their CoT more. See here.

evhub's Shortform

evhub13d10-6

I think we should be relatively less worried about instrumental power-seeking and relatively more worried about terminal power-seeking. Note that this is only a relative update on the margin, and maybe on net I am still more concerned about the instrumental version because I started much more concerned about it. This is also not a super recent update—I just haven't seen it written up before.

Simple argument:

The standard deceptive alignment story involves a model developing a somewhat random proxy goal and then that goal getting effectively locked-in and r

... (read more)

3Alex Mallen12d

Can you clarify what you mean by "terminal power-seeking"? Some things I can imagine: 1. A cognitive pattern that terminally wants to have long-term power, and therefore plays the training game (IMO the most straightforward interpretation, and the one I most agree with). 2. A cognitive pattern that terminally pursues power-on-the-episode because this is useful for scoring well on the task. This is what you seem to be pointing at with the Vending-Bench example. (Note that this is imperfectly fit on its own) (1) and (2) are importantly different because only (1) motivates training-gaming. I think there's a reasonable path-dependent case to be made that (2) eventually generalizes to (1), but they entail fairly different behaviors so they're important to distinguish.

evhub12d20

Certainly the really concerning thing here is (1). Though indeed one way you might get (1) is by generalization from (2).

Thomas Larsen's Shortform

Thomas Larsen14d1611

Hypothesis: alignment-related properties of an ML model will be mostly determined by the part(s) of training that were most responsible for capabilities.

If you take a very smart AI model with arbitrary goals/values and train it to output any particular sequence of tokens using SFT, it'll almost certainly work. So can we align an arbitrary model by training them to say "I'm a nice chatbot, I wouldn't cause any existential risk, ... "? Seems like obviously not, because the model will just learn the domain specific / shallow property of outputting those part... (read more)

Daniel Kokotajlo13d32

Relevant OpenAI blog post just today: https://alignment.openai.com/how-far-does-alignment-midtraining-generalize/

Relevant figure:

2Alex Mallen13d

I think this is a very important hypothesis but I disagree with various parts of the analysis. [...] I think this is an important observation, and is the main thing I would have cited for why the hypothesis might be true. But I think it's plausible that the AI's capabilities here could be separated from its propensities by instrumentalizing the learned heuristics to aligned motivations. I can imagine that doing inoculation prompting and a bit of careful alignment training at the beginning and end of capabilities training could make it so that all of the learned heuristics are subservient to corrigible motivations - i.e., so that when the heuristics recommend something that would be harmful or lead to human disempowerment, the AI would recognize this and choose otherwise. [...] Even if the AI had a perfect behavioral reward function during capabilities-focused training, it wouldn't provide much pressure towards motivations that don't take over. During training to be good at e.g. coding problems, even if there's no reward-hacking going on, the AI might still develop coding related drives that don't care about humanity's continued control, since humanity's continued control is not at stake during that training (this is especially relevant when the AI is saliently aware that it's in a training environment isolated from the world -- i.e. inner misalignment). Then when it's doing coding work in the world that actually does have consequences for human control, it might not care. (Also note that generalizing "according to the reward function" is importantly underspecified.) [...] This type of training (currently) does actually generalize to other propensities to some extent in some circumstances. See emergent misalignment. I think this is plausibly also a large fraction of how character training works today (see "coupling" here).

Richard Ngo's Shortform

Richard_Ngo18d*2837

There's a strong pattern in ratfic of the protagonist "winning" by gaining the power to design a new world order from scratch—i.e. taking over the world. It's a very High Modernist mindset (as I pointed out in a recent tweet). And once you see how crucial this is to the rationalist perspective on what a good future looks like, it's hard to unsee.

You might respond: the worlds these protagonists find themselves in are usually so bad that seizing absolute power is in fact the most ethical thing to do. But the worlds didn't have to be that bad! The writers cho... (read more)

Showing 3 of 6 replies (Click to show all)

6Richard_Ngo18d

I do actually think that the general trope of "the rebels winning is sufficient for a happy ending" is pretty indicative of poor ethical thinking. But even Hollywood balks at their heroes ending up with literal godlike control of the world. For example (though I haven't watched the series) my impression of the Avengers franchise is that they introduce a plot device (the infinity gauntlet) that gives its wielder godlike powers, the heroes use it specifically to defeat the bad guy and undo the damage he caused, and then they destroy the device. In other words, they got to the exact point that ratfic heroes got to, and then their happy ending specifically involves them giving up the same kind of godlike power that ratfic heroes typically use to make themselves dictators of the universe. Similarly for Superman: his happy endings involve him successfully using his godlike powers to beat the bad guys without changing the established world power structures basically at all. And I feel pretty confident that a big reason Superman doesn't end up taking over the world is because the writers and viewers would have moral qualms about that kind of ending. tl;dr: there are many ways to make a story have a happy ending, and it's quite indicative of the authors' ethical and political views which endings they consider to be happy. The kind of endings that rationalists often portray as happy, mainstream scriptwriters seem to go out of their way to avoid.

3Eli Tyre15d

Yes. And I claim they're wrong about that. There's lots of banal evil (some of which that is not regarded as evil by typical social morality, some of which is, but is generally treated as normal and ignored). I would fight a war to end factory farming, if that would help. If I ended up with "ultimate power" somehow, by some mechanism that didn't involve me taking on ultimate power for a specific narrow mandate, I think it is both ethically correct to use it to permanently end many (but probably not all) of those evils. This is indeed pretty scary.

Richard_Ngo15d30

Oh, I think of "ending factory farming" as very far from "taking over the world".

If Superman were a skilled political operator it could be as simple as arranging to take photoshoots with whichever politicians legislated the end of factory farms.

Or if he were less skilled it could involve doing various kinds of property damage to factory farms (potentially even things which there aren't laws against, like flying around them in a way which blows the buildings over).

This might escalate to the government trying to arrest him, and outright conflict, but honestl... (read more)

Vanessa Kosoy's Shortform

Vanessa Kosoy6y70

Master post for ideas about infra-Bayesianism.

Showing 3 of 40 replies (Click to show all)

2Vanessa Kosoy17d

This is an idea I came up with and presented in the Agent Foundations 2025 at CMU conference. Here is a nice simple formalism for decision theory, that in particular supports the decision theory coming out of infra-Bayesianism. I now call the latter decision theory "Disambiguative Decision Theory", since the counterfactuals work by "disambiguating" the agent's belief. Formalism Let be the agent's event space and the space of possible policies [1] . Let be the agent's loss function. For each , we are given some . [2] This represents the event "the agent's behavior is consistent with policy ". We assume that This data is common for all decision theories, but the rest of the details depend on the theory: Functional Decision Theory (FDT) We are given a mapping s.t. is supported on for all . The distribution represents the logical counterfactual associated with . It is also possible to consider the more general "robust" version , but we will avoid it here for simplicity. The decision rule is then We will call an FDT problem "formally causal" when for any , the measures and agree when restricted to . That is, for any measurable , we require Causal Decision Theory (CDT) CDT has the same formal form as FDT, but we always require the problem to formally causal. Moreover, the interpretation of is different: it now represents the causal counterfactual associated with . The decision rule is also formally the same: Given an FDT problem , we can translate it to a CDT problem, if we specify the agent's belief about its own policies and causal interpretation: the kernel . Here is a copy of that represents the factual policy and is a copy of that represents the counterfactual policy. We require that , and that is formally causal in the second argument. Normally, comes from a causal graph, where apply the do-operator for the counterfactual policy and condition on the factual policy (i.e. condition on what the policy would have been if not for the do-operator).

Vanessa Kosoy15d*20

A few more observations.

Partially Observable Iteration

The definition of iteration we had before implicitly assumes that the agent can observe the full outcome of previous iterations. We don't have to make this assumption. Instead, we can assume a set of possible observations and a mapping , in which case we define

I believe that Theorem 4 remains valid.

Idealized Disambiguative Decision Theory

As we remarked before, DDT is not invariant under adding a constant to the loss function. It is interesting to consider what happens when we add an increasingly large ... (read more)

4Vanessa Kosoy22d

This idea was described in a presentation I have in '23, but wasn't written down anywhere. Here is a formalization of recursive self-improvement (more precisely, recursive metalearning) in the metacognitive agent framework. Let be the set of programs or computations represented using some syntax. For example, might be a set of strings that serve as programs for a fixed Universal Turing Machine, but might also be something more structural, like the space of -terms or the space of PCF programs. Let be the space of semantic objects that we assign to computations. For example, we might just consider programs that output a single bit, in which case , but we can also consider rich semantic theories such as domain theory or game semantics. Let be the set of "weakly consistent" mappings from to . Here, "weakly consistent" means that we might impose some consistency conditions or none, but the conditions are weak enough to admit computationally tractable learning (in particular, these conditions fall short of picking out the true semantics). Thus, is our space of logical counterpossible worlds. Let be the hypothesis class our agent is learning and an associated complexity function. With every we associate some which a prior over refinements of . Moreover, we require a symbolic representation of , that is, some s.t. , where is the true semantics. For example, maybe there is some with a mapping (some semantic objects that can be interpreted as elements of ) and s.t. for all , , and then . Consider any symbolic representation of an element of , that is, some . Then, it is possible to construct , which is an intrinsic version of . That is, corresponds, from the agent's perspective, to the assertion "the belief represented by is true" (without committing to the explicit form of this belief). is defined as follows. Define by can be regarded as a multivalued mapping from to . [1] This mapping is Kakutani. Therefore, by Kakutani's theorem, it has a non-empty set of fixed poi

Thomas Kwa's Shortform

Thomas Kwa17d239

I conducted an exercise at METR to simulate what our work would be like in 2027, when we have 200 hour time horizon AIs. Some observations:

The pace of research was much faster than today, something like 3x. I would guess that speedup goes as time horizon to the 0.3 or 0.4 power, though we didn't run the game with enough fidelity to tell
No time to develop ideas before implementing: Agents implement ideas as soon as you think of them, so rather than ideating for days at a time, you can make an MVP in a couple of hours and revise. If the task isn’t near the l

... (read more)

3becausecurious17d

Thank you for sharing! 1. I cannot understand how exactly the game was played. Would players just write down what AI accomplishes into the spreadsheet to simulate AI working on a task? If so, how do you simulate 200 hour time horizon? E.g. how do you decide that this piece of work done by AI can be done by 200 hour time horizon AI and this one cannot? 2. Regarding predictions to loosen the bottleneck: [...] These just feel like basic sanity checking. I doubt AI would be very good at such predictions. Is this the intention? Or am I missing something?

Thomas Kwa16d20

Yes, I (as GM) was constantly monitoring the spreadsheet, asking players to explain their actions, and deciding whether the AI would succeed. We know how many human hours certain tasks have taken METR staff in the past, and I mentally estimated these for each player action. 200h task that was as clean/benchmarky/verifiable as HCAST/RE-Bench tasks would succeed with 50% chance or have ~1 big mistake on average. Based on 80% time horizons being ~5 times shorter, a clean 40 human hour task would have 80% success chance.

In an earlier version, I assigned a "mes

... (read more)

Alex Mallen's Shortform

Alex Mallen18d146

Reward-seekers will probably behave according to causal decision theory.

Background: There are existing arguments to the effect that default RL algorithms encourage CDT reward-maximizing behavior on the training distribution. (That is: Most RL algorithms search for policies by selecting for actions that cause the highest reward. E.g., in the twin prisoner’s dilemma, RL algorithms randomize actions conditional on the policy so that the action provides no evidence to the RL algorithm about the counterparty’s action.) This doesn’t imply RL produces CDT reward-... (read more)

Showing 3 of 6 replies (Click to show all)

3Linch16d

I'm confused. Can someone explain to me in simple language why an RL environment for twin-prisoner's dilemmas wouldn't favor EDT?

Alex Mallen16d20

Let's say the current policy has a 90% chance of cooperating. Then, what action results in the highest expected reward for player 1 (and in turn, gets reinforced the most on average)? Player 1 sampling defect leads to a higher reward for player 1 whether or not player 2 samples cooperate (strategic dominance), and there's a 90% chance of player 2 sampling cooperate regardless of player 1's action because the policy is fixed (i.e., player 1 cooperating is no evidence of player 2 cooperating, so it's not the case that reward tends to be higher for player 1 w... (read more)

1Alex Mallen18d

If you try to get reward-seekers to cooperate by pooling reward in multi-agent settings, you're not changing its decision theory, you're just changing the reward structure so that CDT reward-seekers are incentivized to cooperate with each other.

ryan_greenblatt's Shortform

ryan_greenblatt1mo*4824

PSA: Anthropic models don't seem to particularly privilege the explicit thinking field. This makes reinforcement spillover—where training on a model's outputs generalizes to the CoT, making it appear safer—more likely.

While Anthropic models do have an separate explicit thinking field, they don't really use thinking that differently from outputs and aren't that dependent on the thinking field. Sometimes they'll just do their thinking in the output field, the way they talk in the thinking field isn't very distinct from how they talk in outputs, and I believe... (read more)

3Tomek Korbak1mo

FWIW we found controllability of most recent Claude models to be similar to GPT-5 series. It seems like after Anthropic stopped optimizing CoTs for 4.5, controllability dropped by a lot.

nostalgebraist1mo20

I'm curious what Sonnet 4.5's score would be on your external CoT controllability eval. I.e., what Sonnet 4.5's blue bar in your Fig. 25 would look like. (I couldn't find a result like this in your paper, but I might have overlooked something)

My casual impression has been that external CoT with Sonnet 4.5 is fairly steerable, but this impression is mostly based on "in practice I have no trouble getting Sonnet 4.5 to do the kind of CoT I need" rather than any kind of systematic investigation.

It seems like at least one of the following must be true:

There are

... (read more)

Thomas Larsen's Shortform

Thomas Larsen1mo204

Note: These are all rough numbers, I'd expect I'd shift substantially about all of this on further debate.

Suppose we made humanity completely robust to biorisk, i.e. we did sufficient preparation such that the risk of bio catastrophe (including AI mediated biocatastrophe) was basically 0.^[1] How much would this reduce total x-risk?

The basic story for any specific takeover path not mattering much is that the AIs, conditional on them being wanting to take over, will self-improve until they find they find the next easiest takeover path and do that instead. ... (read more)

Showing 3 of 5 replies (Click to show all)

Thomas Kwa24d10

Difficulty of the successor alignment problem seems like a crux. Misaligned AIs could have an easy time aligning their successors just because they're willing to dedicate enough resources. If alignment requires say 10% of resources to succeed but an AI is misaligned because the humans only spent 3%, it can easily pay this to align its successor.

If you think that the critical safety:capabilities ratio R required to achieve alignment follows a log-uniform distribution from 1:100 to 10:1, and humans always spend 3% on safety while AIs can spend up to 50%, the... (read more)

5TsviBT1mo

I haven't thought too deeply about this, but I would guess that the AI self-alignment problem is quite a lot easier than the human AI-alignment problem. Cf. https://www.lesswrong.com/posts/dho4JQytfHWXtTvkt/on-the-adolescence-of-technology?commentId=t2hKmhsS6yLyJFQwh

5Thomas Larsen1mo

I agree that AI successor-alignment is probably easier than the human AI alignment problem. One additional difficulty for the AIs is that they need to solve the alignment problem in a way that humans won't notice/understand (or else the humans could take the alignment solution and use it for themselves / shutdown the AIs). During the regime before human obsolescence, if we do a reasonable job at control, I think it'll be hard for them to pull that off.

TurnTrout's shortform feed

TurnTrout1mo5472

I signed an amicus brief supporting Anthropic's right to do business without governmental retaliation. As an AI expert, I attest that Anthropic's technical concerns are legitimate, and no laws were designed to protect against AI analysis of surveillance data.

Even though I work at a competing lab (Google DeepMind), I'm proud of Anthropic for taking a stand against unlawful retaliation and immoral demands.

(I speak only for myself, not my employer.)

Wei Dai's Shortform

Wei Dai1mo83

It seems that LLMs are not good enough at reasoning, even after being trained on ~all human output, such that you couldn't amplify their capabilities to arbitrary levels through iterated amplification, so AI companies are mainly increasing AI capabilities via RLVR instead. Is this impression wrong, and how to update on it if not?

Aside from the potential implications on alignment (i.e., closing off one approach that seemed hopeful for some, at least for the foreseeable future), I wonder if this is a deficiency in LLMs (their architecture or how they're trai... (read more)

TurnTrout's shortform feed

TurnTrout1mo4373

Unless you have an explicit strategy for how and when you will speak out in the future, and explicit reasoning for why that requires silence now, "strategic silence" is just silence.

Raemon1mo43

I think part of the problem is people think of themselves as having at least, like, a medium explicit strategy, but, the strategy routes through some judgment that conveniently keeps returning "not yet" or "only saying things in a somewhat cagey way."

i.e. this advice seems necessary but not sufficient.

Alex Mallen's Shortform

Alex Mallen2mo289

There's an apparent tension in the inoculation prompting literature: Anthropic found that general inoculation prompts work well during on-policy RL, while the prompts used for SFT in Wichers et al. are quite specific to the misbehavior we want to prevent. I think there might be a straightforward mechanistic reason for why general inoculation prompts work well during on-policy RL but not in off-policy training (SFT or recontextualization).

In Wichers et al., which studies inoculation prompting in SFT settings, we find that we need to use quite specific inocu... (read more)

Sam Marks2mo103

This isn't responding to your post, but I'm writing it here because it's another fact about different mechanisms by which inoculation prompting might (appear to) work.

In the normal story, the inoculation prompt recontextualizes the model's undesired behavior, such that the model doesn't display the behavior in dissimilar contexts. In this story:

The semantic content of the prompt is important. If you had used a prompt that said "Please don't do [bad thing]" or a prompt consisting of random characters, then the inoculation would have failed.
Capabilities lear

... (read more)

TurnTrout's shortform feed

TurnTrout2mo3348

I think that "eval aware" models cannot be defeated by simply making evals indistinguishable from reality. (Maybe this point has been made elsewhere for LLMs, but I haven't read it so I'll state it here.)

Consider the POV of a consequentialist with misaligned goals. It knows that we run honeypots and that—suppose—we can make evals so realistic they are indistinguishable from reality (perhaps using real user data). Then the rational decision would be to "play it safe" and not misbehave in any situations it thinks we can mock realistically, since they could b... (read more)

[+][comment deleted]2mo30

6Tim Hua2mo

I think it's important to note that indistinguishable evals still buy us a lot! The claim "indistinguishable evals does not solve AI safety" feels quite similar to "imperfect monitors does not solve AI safety." Like, having better monitors sure seems to help (even if only to buy us more time.). To your point though: I'm excited about research that attempts to induce various beliefs in models during honeypot evaluations, possibly via model internals and/or edits to the chain of thought. Basically, if the model defects after seeing RSA-2048 cracked because that's proof that it is not being tested and in the future, we can try to intervene on the belief "not being tested and in the future" itself.[1] In our paper on evaluation awareness, we tried to steer the model towards believing that it is being deployed. You can imagine doing this for a huge variety of beliefs that could plausibly be load bearing for a model that would attempt to launch a rogue deployment:[2] * The AI is in the future/late into its deployment/about to be retired. * The AI is not being monitored. * The AI has already gained a lot of trust/has broad affordances/has influence in the world. We view our work on evaluation awareness as an example for the general class of interpretability techniques that change model beliefs (see also the artificial triggering section here.) 1. ^ i.e., something like relaxed adversarial training. 2. ^ Although fwiw I'm more worried about early transformative AIs (1) trying to align their more powerful successors but messing it up due to incompetence (sometimes called "death by slop") or (2) making their successors to be aligned with them.

19Fabien Roger2mo

I think indistinguishable evals could reduce risk a lot for early transformative AIs: 1. They give you much better alignment arguments against non scheming AIs and against scheming AIs that are not competent enough to be appropriately careful; 2. Needing to be appropriately careful can cut down the fraction of inputs on which the AI should even consider attacking a lot - for deployments that look similar to current ones (in particular without a shared neuralese memory bank), AIs don't see RSA-2048 on most inputs 1. More concretely, if you get credible evidence of not-eval only on 5% of inputs and attack only on the single best opportunity, then playing it safe roughly means that you attack on the top-20 opportunity instead of the top-1 2. (Note that "getting credible evidence of not-eval" doesn't just mean being >50% confident it's not an eval because the pre-deploy eval might be big. If you have n~10k eval inputs of the right level of juiciness then you need to have an FPR at thinking you get credible evidence of not being in an eval on the order of 0.1% or lower in order to never accidentally reveal non-eval behavior during evals.) 3. More ambitiously, you might be able to get very close to actually indistinguishable evals (e.g. by continuously scraping deployment and using it as inspiration for evals - which would help a lot against things like "defect after seeing RSA-2048 cracked"), which could in principle drive down the fraction of inputs on which an AI can be confident it's not an eval much below 5%. Eventually you get bitten by rare failures - especially if the AI is powerful enough that there are much more good attack opportunities in deployments than in evaluations (Ryan makes this point here and it's also the assumption that the AI control paper uses). I also expect it to be difficult to achieve sth close to indistinguishability (Buck makes this point here). But I still think there is value in trying to get higher indistinguishability.

Wei Dai's Shortform

Wei Dai2mo20

Not sure if this is already well known around here, but apparently AI companies are heavily subsidizing their subscription plans if you use their own IDEs/CLIs. (It's discussed in various places but I had to search for it.)

I realized this after trying Amp Code. They give out a $10 daily free credit, which can easily be used up in 1 or 2 prompts, e.g., "review this code base, fix any issues found". (They claim to pass their API costs to their customers with no markup, so this seems like a good proxy for actual API costs.) But with even a $19.99 subscription... (read more)

evhub's Shortform

evhub1y*523

I'm interested in soliciting takes on pretty much anything people think Anthropic should be doing differently. One of Alignment Stress-Testing's core responsibilities is identifying any places where Anthropic might be making a mistake from a safety perspective—or even any places where Anthropic might have an opportunity to do something really good that we aren't taking—so I'm interested in hearing pretty much any idea there that I haven't heard before.^[1] I'll read all the responses here, but I probably won't reply to any of them to avoid revealing anythin... (read more)

Showing 3 of 10 replies (Click to show all)

Chris_Leong1y32

I believe that Anthropic should be investigating artificial wisdom:

I've summarised a paper arguing for the importance of artificial wisdom with Yoshua Bengio being one of the authors.

I also have a short-form arguing for training wise AI advisors and an outline Some Preliminary Notes of the Promise of a Wisdom Explosion.

5evhub1y

I can say now one reason why we allow this: we think Constitutional Classifiers are robust to prefill.

6ryan_greenblatt1y

As discussed in How will we update about scheming?: [...] I wish Anthropic would explain whether they expect to be able to rule out scheming, plan to effectively shut down scaling, or plan to deploy plausibly scheming AIs. Insofar as Anthropic expects to be able to rule out scheming, outlining what evidence they expect would suffice would be useful. Something similar on state proof security would be useful as well. I think there is a way to do this such that the PR costs aren't that high and thus it is worth doing unilaterially from a variety of perspectives.

Richard Ngo's Shortform

Richard_Ngo2mo2423

The concept of "schemers" seems to be gradually becoming increasingly load-bearing in the AI safety community. However, I don't think it's ever been particularly well-defined, and I suspect that taking this concept for granted is inhibiting our ability to think clearly about what's actually going on inside AIs (in a similar way to e.g. how the badly-defined concept of alignment faking obscured the interesting empirical results from the alignment faking paper).

In my mind, the spectrum from "almost entirely honest, but occasionally flinching away from aspect... (read more)

Showing 3 of 5 replies (Click to show all)

Alex Mallen2mo10

I think I propose a reasonable starting point for a definition of selection in a footnote in the post:

You can try to define the “influence of a cognitive pattern” precisely in the context of particular ML systems. One approach is to define a cognitive pattern by what you would do to a model to remove it (e.g. setting some weights to zero, or ablating a direction in activation space; note that these approaches don't clearly correspond to something meaningful, they should be considered as illustrative examples). Then that cognitive pattern’s influence could

... (read more)

8Alex Mallen2mo

I largely agree with the substance of this comment. Lots of risk comes from AIs who, to varying extents, didn't think of themselves as deceptively aligned through most of training, but then ultimately decide to take substantial material action intended to gain long-term power over the developers (I call these "behavioral schemers"). This might happen via reflection and memetic spread throughout the deployment or because of more subtle effects of the distribution shift to situations where there's an opportunity to grab power. And I agree that people are often sloppy in their thinking about exactly how these AIs will be motivated (e.g., often too quickly concluding that they'll be trying to guard the same goal across contexts). (Though, in case this was in question, I think this doesn't undermine the premise of AI control research, which is essentially making a worst-case assumption about the AI's motivations, so it's robust to other kinds of dangerously-motivated AIs.)

1Alex Mallen2mo

Here's some relevant discussion of "Behavioral schemers that weren’t training-time schemers": [...]