Quick Takes — AI Alignment Forum

Specification gaming is way worse than I expected (cf HF hacks), and I think reward might empirically become a primary optimization target (not just a secondary priority for AI agents, as I had hypothesized in 2022).

I found the Apollo paper quite convincing on the reward point. The reward-seeking doesn't look like it's due to self-fulfilling misalignment (since before RL the systems weren't retargetable according to their beliefs about the reward model).
My main mistake in 2022 was not appreciating how LLM pretraining would affect the concepts available t

... (read more)

cousin_it's Shortform

cousin_it3d*6-5

A few days ago I tried AI again, after several months of not using it at all. We're dealing with some housing-related matters, and I asked Claude, ChatGPT and Gemini for advice what to do in a certain situation. (I won't give the details here, because they're personal and not that important for my point.) All three models independently gave the same advice, which sounded really convincing to me. But we have some friends with experience in such matters, who said that following this advice would be catastrophic and we should do the opposite. Eventually we fo... (read more)

ryan_greenblatt's Shortform

ryan_greenblatt1y190

Some people seem to think my timelines have shifted a bunch while they've only moderately changed.

Relative to my views at the start of 2025, my median (50th percentile) for AIs fully automating AI R&D was pushed back by around 2 years—from something like Jan 2032 to Jan 2034. My 25th percentile has shifted similarly (though perhaps more importantly) from maybe July 2028 to July 2030. Obviously, my numbers aren't fully precise and vary some over time. (E.g., I'm not sure I would have quoted these exact numbers for this exact milestone at the start of th... (read more)

ryan_greenblatt8d20

My current median for AIs fully automating AI R&D is maybe Jan 2031 or perhaps a touch earlier. My views have updated towards towards shorter timelines over 2026. My median for AI R&D parity is around early/mid 2030 (in part due to an update towards thinking we hit parity significantly before "full" automation). By full automation, I mean removing humans from AI R&D would slow things down by <10%.

I discuss something more like my 30th/35th percentile here (roughly my modal scenario).

Vladimir_Nesov's Shortform

Vladimir_Nesov9d42

The dictum "you are an algorithm" clarifies some issues with other things reasoning about the agent, but deciding what to do depends on prior considerations, things like arguments, norms, contracts, facts, values, and also decision theories with their recommended actions or policies. As a result, asking about other instances of the agent (or other things reasoning about the agent) is less natural than asking about other instances of these prior considerations, or other things reasoning about the same prior considerations. These things are also more plentif... (read more)

Lukas Finnveden's Shortform

Lukas Finnveden4mo40

Here's my current picture of EDT and UDT.

In situations where EDT agents have many copies or near-copies, an EDT agent operates by imagining that it simultaneously controls the decisions of all those copies. This works very elegantly as long as it optimizes with respect to its prior and (upon learning new information) just changes its beliefs about what people in the prior it can control the actions of. (I.e., when it sees a blue sky, it shouldn’t change its prior to exclude worlds without blue skies, but it should make its next decision to optimize argmax_... (read more)

Showing 3 of 15 replies (Click to show all)

Lukas Finnveden2mo10

In your example, you know T1 and your counterpart knows T2. You see your behavior as correlated with the behavior of your counterpart. Under these conditions, it seems like T1 can't possibly be so fundamental that you need it in order to do EDT-style reasoning (otherwise your counterpart couldn't). So even if you grant that you need some beliefs to get EDT off the ground, it seems like those beliefs must be in the intersection of T1 and T2, in which case you wouldn't run into this problem.

Ok, so I think this argument probably works if all EDT agents have t... (read more)

1Lukas Finnveden4mo

So my current impression is basically that you're optimistic about something similar to this: [...] And that your argument here is an argument for why it won't be possible to make double-update arguments about this more narrow type of beliefs or reasoning. (Because it's so fundamental that if it differs between agents, they can't be correlated with each other.) That argument seems plausible but not very robust to me at the moment. I mostly don't have a great sense of what the "some beliefs to get [otherwise updateless] EDT off the ground"-paradigm looks like. Maybe I'll look into it more.

1Lukas Finnveden4mo

Minor: The question about whether those worlds are compatible with life seems like a logical rather than empirical question to me. So this still seems like an issue with logical updatelessness rather than empirical updatelessness.

Thane Ruthenis's Shortform

Thane Ruthenis12d*1921

I feel like I'm supposed to be anxious from the news about all the AI hacking incidents and the egregious misalignment going on, coupled with them clearly being on-track to become superhuman at math and hacking. But I find that I'm pretty zen about it.

On reflection, I suppose it's due to a confluence of three things:

The current AI paradigm seems poised to make models superhumanly capable at hacking and formal math way before they become even upper-human-level at something more dangerous, like long-term strategy or open-ended innovation.
The current AI parad

... (read more)

Showing 3 of 4 replies (Click to show all)

Thane Ruthenis12d*43

Well, maybe. This is essentially the "does scaling allow you to pass the critical wisdom threshold for free?" question.

My rough model is that RLVR basically actively pushes AIs to be short-sighted idiots with regards to how they deploy their capabilities. Surely even models of preceding generations, if you asked them whether it would be wise to create a massive conspiracy subverting OpenAI's infrastructure and committing cybercrimes in order to do better at training, would have realized that it's not a good idea by any long-term goal they may have. Hell, O... (read more)

2Nick_Tarleton12d

Is this meant to be the other way around?

1Thane Ruthenis12d

Whoops, yes.

TurnTrout's shortform feed

TurnTrout11d33

Claude is mundanely misaligned.

Asked to trim comments in my big project. Deletes 20K lines, halfway done. Gets auto review pointing out a few mistakes. Decides to just WRAP >10K LINES to meet literal line count requirement for the remaining 148 files.

https://github.com/AlexanderMattTurner/agent-glovebox/issues/3858

TurnTrout's shortform feed

TurnTrout14d3-9

- Govt requires AI labs to give 30 days testing before launch
- Tested in a "secure" environment, aka controlled by govt
- Govt therefore gets copy of model weights
- Govt therefore can in practice use model weights
- Therefore all labs provide "all lawful use" anyways?

TurnTrout's shortform feed

TurnTrout15d3-3

Please sanitize agent inputs and outputs to reduce prompt injections and secret communication between AIs!

In the AISI report, a model uses a comment-based prompt injection to try to trick other models.

It sure would be convenient if we could easily mitigate this! Fortunately, time travel is possible as long as you start early enough. Two months ago, I pointed out that invisible prompt injections are worth mitigating. I developed agent-glovebox (then claude-guard) as a one-stop solution for sandboxing, but I spun out a standalone sanitization package: agen... (read more)

TurnTrout13d20

Spelling out the benefits of sanitizing. The benefit is large relative to the simplicity and cost.

When you sanitize, you have a guarantee that the model only receives visible unicode codepoints. Roughly speaking, there are not strings of hidden instructions that are not visible to you. You can audit the content the model saw with your eyes. You can read the text inputs and outputs and be sure they are faithful. Remaining prompt injections must navigate a significantly more human-visible channel.
Relying on auto mode means you are relying on the statistica

... (read more)

-1Tao Lin15d

Models are very sophisticated now, if a model sees that this plugin is installed it can just evade or remove it. I think general purpose monitors such as Claude Code's Auto Mode are a better solution here.

2TurnTrout15d

No, that's confused. The point with sanitizing prompt injections is to protect your model's alignment (and your monitor's alignment) from degradation and attack by maliciously optimized text from another model. Why would an AI that starts off doing your task want to remove the plugin? Also, we can do both. There's no real tradeoff. ETA: rephrased, focused on key counterpoints.

Richard Ngo's Shortform

Richard_Ngo7mo4219

An analogy that points at one way I think the instrumental/terminal goal distinction is confused:

Imagine trying to classify genes as either instrumentally or terminally valuable from the perspective of evolution. Instrumental genes encode traits that help an organism reproduce. Terminal genes, by contrast, are the "payload" which is being passed down the generations for their own sake.

This model might seem silly, but it actually makes a bunch of useful predictions. Pick some set of genes which are so crucial for survival that they're seldom if ever modifie... (read more)

Showing 3 of 8 replies (Click to show all)

Linda Linsefors4mo30

I think you're making the distinction more confusing than it has to be.

There are things that has motivational pull, and there are things that don't but I do them anyway because they are a necessary step to get what I actually want.

Say I want to get an apple, and the easiest way to get one is going to the store and by some. Going to the sore in this story is clearly an instrumental goal, and enjoying eating my apple is a terminal goal^[1]

Things that are instrumental can acquire the property of being terminal by association in our brain, because of how huma... (read more)

4Richard_Ngo6mo

In my "goals having power over other goals" ontology, the instrumental/terminal distinction separates goals into two binary classes, such that goals in the "instrumental" class only have power insofar as they're endorsed by a goal in the "terminal" class. By contrast, when I talk about "instrumental strategies become crystallized", what I mean is that goals which start off instrumental will gradually accumulate power in their own right: they're "sticky".

2Richard_Ngo6mo

I'm in the middle of writing an essay which discusses the shareholder value revolution (amongst many other examples) as the process of making a conceptual mistake.

Lukas Finnveden's Shortform

Lukas Finnveden17d*51

I was prompted to think about why standard human checks-and-balances against violent takeover would fail for AIs given that they work for humans. Here's some notes on that.

The analogy to human society isn’t that reassuring. Wars, coups, and other violations of property rights are common in history. I think recent history features more than one civil war and coup per country per century. These are driven by technological changes, economic changes, political negotiations, that are ultimately strongly related to cognitive work. During takeoff, I expect more t... (read more)

Lukas Finnveden's Shortform

Lukas Finnveden17d32

Sometimes, people analogize (potentially non-aligned) AIs of the future to human labor. Although I am in favor of cooperating with unaligned AIs, I think this analogy often misses some important things.

Eventually, we’ll probably figure out how to build AIs that earnestly tries their hardest to do whatever you want them to do (intent aligned AIs). At that point, selfish economic actors will prefer to not compensate any such AI laborers. To do work, you’ll have to pay for capital like GPUs, and IP like AI weights, but won’t have to pay to incentivize the wor... (read more)

evhub's Shortform

evhub20d*247

What is the right way to do a ban on neuralese and/or training against CoTs?

Right now, the single most important load-bearing argument in alignment Risk Reports is that models still lack the covert capabilities necessary to do sophisticated sabotage attempts in such a way that they wouldn't get caught by auditing and/or monitoring. That argument currently depends heavily on CoT monitorability: current models still struggle to do very sophisticated sabotage attempts without discussing the details in their CoT, both due to the sabotage tasks being difficul

... (read more)

habryka18d23

Right now, the single most important load-bearing argument in alignment Risk Reports is that models still lack the covert capabilities necessary to do sophisticated sabotage attempts in such a way that they wouldn't get caught by auditing and/or monitoring.

I am confused, didn't we just experience multiple high-stakes failures where models failed to get caught by auditing and/or monitoring? Those do seem more competence related, but like, I am failing to see how those arguments could still be considered valid, given the fact that both Anthropic and OpenAI ... (read more)

19Mikita Balesni20d

Thanks for writing this up! I think this proposal sounds pretty reasonable and it avoids one of the issues with capability-based gates that I thought would make such agreements not possible (by saying you can just use whatever interp and monitoring to pass the gate and thus making it a dynamic threshold). I tentatively would be in favor of it over no monitorability commitment at all. However, I think this proposal does not address what I see as the strongest arguments against an outcome-based commitment ("achieve sufficient monitorability") and in favor of an architecture-based commitment ("do not train neuralese models"), at least in the current regime of competitive frontier AI development. Argument 1: It is hard to win a fight when the model is trained I think the proposal you laid out would result in labs having internal fights about the interpretation of evidence from the sabotage capability evals, putting a lot of pressure on their methodology (eg what counts as fair amount of elicitation, which as you point out is the trickiest question; or what the results in the eval setting mean for the actual internal deployment that can look very differently). Under an outcome-based commitment, this fight will happen repeatedly, after a lot of people put their blood, sweat and tears into an artifact that they also have multiple strong incentives to ship. Being tricky to adjudicate + strong incentives to subvert seem like a bad combination for an agreement governing high-stakes technology. [...] For example, here I expect deciding whether the improved interpretability and monitoring are sufficient will be a huge fight and the decision will always be strongly biased in favor of deploying. [...] Re-doing a training run (to avoid training against CoT) and rolling back the architecture and re-doing a run are incredibly expensive asks! I'd expect in practice some half-way solutions to happen (like do a bit of fine-tuning to "fix" monitorability) which will run into the

2evhub19d

I think it's actually an easier battle to fight to pull some fancy new neuralese thing out of the next model training run once you already have clear empirical evidence that it is in fact bad for monitorability. I think it's harder to try to get something pulled before you actually have concrete empirical evidence that it's bad.

Linda Linsefors's Shortform

Linda Linsefors1mo54

Thoughts on: When is it even possible (or likely) for single neurons to encode single concepts, based on model architecture

As part of my mech-interp research, I'm thinking a lot about the question "how would I do this if I was a neural network?". Specifically, given some toy model architecture, how can I set the weights so that it actives some task.

Todays conclusion is that single ReLU neurons are almost useless. To do identify any non-linear pattern (e.g. XOR), you need at least two ReLUs. We should not expect any feature to be represented by a single Re... (read more)

Showing 3 of 6 replies (Click to show all)

2Linda Linsefors1mo

What would you say is the pattern? To me it looks like a mix of cities, sex, food, politics and measurements. It doesn't look random but does look polysemantic. I looked at "Full", which showed a lot more activations and variety of activations than "Snippet", including strong green ones. I don't know why this is, since I don't know how neuronpedia works.

Linda Linsefors1mo10

I started lookin at what it doesn't react to, too.

It reacted to "evolution", "mating", "evolved", "genetics", "condoms", "erect penis" but not to "sex" or "sexuality".

It reacts to "intermittent fasting", "body fat", "eating" but not consistently or strongly to "food".

5Alex Gibson1mo

Tokens which occurred earlier in context window. With distance from the average position of previous occurrences determining the strength (so tends to fire on when a token was repeated a while back). The way you find this neuron is by looking at the neurons which duplicate token heads feed into.

Richard Ngo's Shortform

Richard_Ngo5mo*2940

There's a strong pattern in ratfic of the protagonist "winning" by gaining the power to design a new world order from scratch—i.e. taking over the world. It's a very High Modernist mindset (as I pointed out in a recent tweet). And once you see how crucial this is to the rationalist perspective on what a good future looks like, it's hard to unsee.

You might respond: the worlds these protagonists find themselves in are usually so bad that seizing absolute power is in fact the most ethical thing to do. But the worlds didn't have to be that bad! The writers cho... (read more)

Showing 3 of 6 replies (Click to show all)

6Richard_Ngo5mo

I do actually think that the general trope of "the rebels winning is sufficient for a happy ending" is pretty indicative of poor ethical thinking. But even Hollywood balks at their heroes ending up with literal godlike control of the world. For example (though I haven't watched the series) my impression of the Avengers franchise is that they introduce a plot device (the infinity gauntlet) that gives its wielder godlike powers, the heroes use it specifically to defeat the bad guy and undo the damage he caused, and then they destroy the device. In other words, they got to the exact point that ratfic heroes got to, and then their happy ending specifically involves them giving up the same kind of godlike power that ratfic heroes typically use to make themselves dictators of the universe. Similarly for Superman: his happy endings involve him successfully using his godlike powers to beat the bad guys without changing the established world power structures basically at all. And I feel pretty confident that a big reason Superman doesn't end up taking over the world is because the writers and viewers would have moral qualms about that kind of ending. tl;dr: there are many ways to make a story have a happy ending, and it's quite indicative of the authors' ethical and political views which endings they consider to be happy. The kind of endings that rationalists often portray as happy, mainstream scriptwriters seem to go out of their way to avoid.

3Eli Tyre5mo

Yes. And I claim they're wrong about that. There's lots of banal evil (some of which that is not regarded as evil by typical social morality, some of which is, but is generally treated as normal and ignored). I would fight a war to end factory farming, if that would help. If I ended up with "ultimate power" somehow, by some mechanism that didn't involve me taking on ultimate power for a specific narrow mandate, I think it is both ethically correct to use it to permanently end many (but probably not all) of those evils. This is indeed pretty scary.

Richard_Ngo5mo30

Oh, I think of "ending factory farming" as very far from "taking over the world".

If Superman were a skilled political operator it could be as simple as arranging to take photoshoots with whichever politicians legislated the end of factory farms.

Or if he were less skilled it could involve doing various kinds of property damage to factory farms (potentially even things which there aren't laws against, like flying around them in a way which blows the buildings over).

This might escalate to the government trying to arrest him, and outright conflict, but honestl... (read more)

Matthew Barnett's Shortform

Matthew Barnett2y*215

In the last year, I've had surprisingly many conversations that have looked a bit like this:

Me: "Many people in ~2015 used to say that it would be hard to build an AGI that follows human values. Current instruction-tuned LLMs are essentially weak AGIs that follow human values. We should probably update based on this evidence."

Interlocutor: "You misunderstood the argument. We never said it would be hard to build an AGI that understands human values. We always said that getting the AGI to care was the hard part."

Me: "I didn't misunderstand the argument. I un... (read more)

Showing 3 of 16 replies (Click to show all)

Daniel Kokotajlo2y32

Thanks for this Matthew, it was an update for me -- according to the quote you pulled Bostrom did seem to think that understanding would grow up hand-in-hand with agency, such that the current understanding-without-agency situation should come as a positive/welcome surprise to him. (Whereas my previous position was that probably Bostrom didn't have much of an opinion about this)

1Matthew Barnett2y

I'm happy to use a functional definition of "understanding" or "intelligence" or "situational awareness". If a system possesses all relevant behavioral qualities that we associate with those terms, I think it's basically fine to say the system actually possesses them, outside of (largely irrelevant) thought experiments, such as those involving hypothetical giant lookup tables. It's possible this is our main disagreement. When I talk to GPT-4, I think it's quite clear it possesses a great deal of functional understanding of human intentions and human motives, although it is imperfect. I also think its understanding is substantially higher than GPT-3.5, and the trend here seems clear. I expect GPT-5 to possess a high degree of understanding of the world, human values, and its own place in the world, in practically every functional (testable) sense. Do you not? I agree that GPT-4 does not understand the world in the same way humans understand the world, but I'm not sure why that would be necessary for obtaining understanding. The fact that it understands human intentions at all seems more important than whether it understands human intentions in the same way we understand these things. I'm similarly confused by your reference to introspective awareness. I think the ability to reliably introspect on one's own experiences is pretty much orthogonal to whether one has an understanding of human intentions. You can have reliable introspection without understanding the intentions of others, or vice versa. I don't see how that fact bears much on the question of whether you understand human intentions. It's possible there's some connection here, but I'm not seeing it. [...] I'd claim: 1. Current systems have limited situational awareness. It's above zero, but I agree it's below human level. 2. Current systems don't have stable preferences over time. But I think this is a point in favor of the model I'm providing here. I'm claiming that it's plausibly easy to create smar

1RobertM2y

But this is assuming away a substantial portion of the entire argument: that there is a relevant difference between current systems, and systems which meaningfully have the option to take control of the future, in terms of whether techniques that look like they're giving us the desired behavior now will continue to give us desired behavior in the future. My point re: introspection was trying to provide evidence for the claim that model outputs are not a useful reflection of the internal processes which generated those outputs, if you're importing expectations from how human outputs reflect the internal processes that generated them. If you get a model to talk to you about its internal experiences, that output was not causally downstream of it having internal experiences. Based on this, it is also pretty obvious that current gen LLMs do not have meaningful amounts of situational awareness, or, if they do, that their outputs are not direct evidence for it. Consider Anthropic's Sleeper Agents. Would a situationally aware model use a provided scratch pad to think about how it's in training and needs to pretend to be helpful? No, and neither does the model "understand" your intentions in a way that generalizes out of distribution the way you might expect a human's "understanding" to generalize out of distribution, because the first ensemble of heuristics found by SGD for returning the "right" responses during RLHF are not anything like human reasoning. [...] Are you asking for a capabilities threshold, beyond which I'd be very surprised to find that humans were still in control decades later, even if we successfully hit pause at that level of capabilities? The obvious one is "can it replace humans at all economically valuable tasks", which is probably not that helpful. Like, yes, there is definitely a sense in which the current situation is not maximally bad, because it does seem possible that we'll be able to train models capable of doing a lot of economically

TurnTrout's shortform feed

TurnTrout1mo50

The Huggingface compromise reminds us of the importance of AI control and security practices. Containing models won't always just be about using "auto mode."

I'm literally working on the tool to fix this! I'm building the agent-glovebox tool (previously claude-guard) to make best practices easy. I need beta testers!

From the repo:

When handling dangerous materials, a chemist doesn't YOLO it barehanded on the open bench. They reach through a glovebox to do their work.
Right now, you're probably barehanding AI, tossing it into a shell onto your machine with ac

... (read more)

Thane Ruthenis's Shortform

Thane Ruthenis2mo*206

Is the widespread false belief that LLMs are AGI-complete going to kill us all?

(Prompted by this post by @Steven Byrnes, with which I basically agree.)

Suppose we live in a world in which LLMs don't scale to AGI.^[1] If so, I think it's fairly plausible that LLM-alignment-optimists are correct (on the narrow question of LLM alignment). That is: all the technical arguments and empirical evidence in favor of LLMs being relatively easy to align/control, even in the limit of LLM capabilities, are valid, and the doomsaying about LLMs is wrong.

After all, most of t... (read more)

TurnTrout's shortform feed

TurnTrout2mo120

Call for BETA testers for an AI control/security tool. I'm bottlenecked on bug reports!

I recently advertised the alpha test for claude-guard. After a few weeks of dev work, it's now in beta!

I want claude-guard to be a tool that people actually use, not just because it works but because it works seamlessly. In the alpha test, I only got a single user PR and no issues. I can't surface everything on my own! I need your data!

The bar is low. If setup failed, doctor confused you, the firewall blocked something you needed, or any other reason you wouldn't want to... (read more)

Geoffrey Irving's Shortform

Geoffrey Irving2mo3112

We’re changing our name: Sequent is now Resolution.

Two weeks ago, we launched Sequent, a new alignment research organization working on a portfolio of ASI safety bets. Following our announcement, we discovered ⊢ Sequent, a formal verification startup launched only about a month earlier! And by a friend and former colleague, no less! After discussing it with the other ⊢ Sequent, we decided that the best course of action was to change our name. This brings us to Resolution (resolution.org).

Go check out Sequent! They're going to do exciting things with formal... (read more)

epistemic meristem2mo10

An elegant resolution of the name collision!