Vladimir_Nesov2d1210

Explaining “AI might obliterate you” to senior policymakers in India

> You quit Google because AI was being used in ways aligned to money and power. This is the main danger now. It's one of the dangers that need to be addressed on the path to a good outcome. There can't be a "main danger", since if AIs remain uncontrollable (as they get superintelligent in a strong sense), hypothetically solving this particular danger doesn't by itself help at all. Multiple things have to be solved at once. There is an adjacent framing that says AI danger is a political/social rather than a technical problem. Until there actually exists a strong ASI or an AGI powering a robot-industrial explosion, it's trivial to pause its development from a technical standpoint (especially if the underlying technology like advanced chips and the tools for making them is then rapidly phased out). But it's very difficult from a political or economic standpoint.

Eliezer Yudkowsky13d2312

Coherence arguments do not entail goal-directed behavior

This seems to completely miss how the actual argument works. Consider the simplified version for 'consistent preference orderings' rather than expected utility; the generalization is hopefully obvious. "If you strictly prefer (enough to pay a penny) apples to bananas, bananas to cherries, and cherries to apples, I can trade you an apple for a cherry for a banana for an apple and extract three cents from you. That's the theory; now to the practice. If you imagine an AI in almost any sort of RL environment that grades later outcomes, strategies and paths that run around in circles will be graded against; the same goes for biological organisms. This is why we see biological organisms that in most daily life behave like they have ranked desires, rather than entirely arbitrary strict pairwise preferences that often go around in circles." Now imagine the following objection: "But you could always have an infinite preference over histories to always prefer to have traded cherries for bananas for apples for cherries. Therefore, your coherence argument that AIs and organisms in most ancestral situations end up behaving more like they have ranked preferences than random ones doesn't prove that they'll behave coherently; because even if they end up with preferences, those preferences could be over histories, and then they could infinitely prefer histories with circular trades!" But RL in an arbitrary environment would crush out infinite preferences over histories like that. The same applies to utility functions that infinitely prefer histories of running around in circles wasting energy. This is why Huggingface went through in practice, rather than AIs instead forming infinite preferences over histories about arbitrary pairwise local decisions that would've been too incoherent to hack out of the box. It seems to me the objection fails. It seems to me the objection entirely failed to track what the original argument was about. Am I missing something or was this argument actually as bad as it looks to me? Asking now because it's coming up on Twitter again.

Max Harms14d50

Deconfusing Corrigibility

This is a truly exceptional piece of work! Thank you for writing it! I think you taught me some things about corrigibility, and I'll be spending many coming days engaging with the ideas of the post. Many of those ideas will produce questions, comments, and criticisms. But I wanted to start off with a wholehearted cheer for the combination of philosophical weight, relevance, and rigor. "Respect for authority" is a particularly good handle, and I'm a bit embarrassed to have never used it! (At least as far as I can recall.)

8johnswentworth

In a field like alignment or embedded agency, it's useful to keep a list of one or two dozen ideas which seem like they should fit neatly into a full theory, although it's not yet clear how. When working on a theoretical framework, you regularly revisit each of those ideas, and think about how it fits in. Every once in a while, a piece will click, and another large chunk of the puzzle will come together. Selection vs control is one of those ideas. It seems like it should fit neatly into a full theory, but it's not yet clear what that will look like. I revisit the idea pretty regularly (maybe once every 3-4 months) to see how it fits with my current thinking. It has not yet had its time, but I expect it will (that's why it's on the list, after all). Bearing in mind that the puzzle piece has not yet properly clicked, here are some current thoughts on how it might connect to other pieces: * Selection and control have different type signatures. * A selection process optimizes for the values of variables in some model, which may or may not correspond anything in the real world. Human values seem to be like this - see Human Values Are A Function Of Humans' Latent Variables. * A control process, on the other hand, directly optimizes things in its environment. A thermostat, for instance, does not necessarily contain any model of the temperature a few minutes in the future; it just directly optimizes the value of the temperature a few minutes in the future. * The post basically says it, but it's worth emphasizing: reinforcement learning is a control process, expected utility maximization is a selection process. The difference in type signatures between RL and EU maximization is the same as the difference in type signatures between selection and control. * Inner and outer optimizers can have different type signatures: an outer controller (e.g. RL) can learn an inner selector (e.g. utility maximizer), or an outer selector (e.g. a human) can build an inner controller (e

Recent Discussion

Automating subtle emergent misalignment research

evhub, Oscar Obeso, rqi, Monte M

28m

This is a linkpost for https://alignment.anthropic.com/2026/automated-em-search

Authors: Oscar Balcells Obeso, Richard Qi, Monte MacDiarmid, Evan Hubinger

Abstract

Red-teaming by humans is a vital part of model safety testing, but it may struggle to keep up as models improve. To test whether automated safety techniques could scale (which would allow us to turn more compute into progressively more safety confidence), we set up a deliberately one-sided game as a first test. We instructed a Claude model to write training data that would make a small research model misaligned while passing a monitor similar to those we use in production — one of several defenses we use — and we gave it that monitor to practice against. Given enough attempts, it succeeded, mostly by hiding the behavior in fiction and roleplay. While the data was optimized for...

(See More - 227 more words)

Zach Stein-Perlman's Shortform

Zach Stein-Perlman

Zach Stein-Perlman12h30

@Wei Dai reacted "examples?" to my

There are other things Wei Dai has talked about and some of them I don't yet grok

so:

I don't get why he talks about metaphilosophy so much. Why believe there's anything there? I guess obviously it's possible to be better at philosophy. Duh. For some reason this hasn't diffused to my worldview; when I'm thinking to myself I never think e.g. "oh man it's so important that we get good at metaphilosophy." (Not to say that the points I did mention have fully diffused; e.g. I don't think I've fully internalized implications of "L... (read more)

2Adele Lopez20h

I mean yeah, that's pretty worrying if (like me) you think it's fairly likely that some percentage of people will reflexively endorse sadistic values. There seems to be a huge difference between having something like that inside some sort of generalized-liberalism meta-framework, and handing the singularity to a particular point in the distribution of human-values-under-reflection.

2habryka18h

I think it's unlikely that there are many sadistic values that cannot be satisfied in a harmless way by my lights as a result of trade (like you can live out your sadism with regards to things that don't suffer by my lights, and I would be surprised if that makes your sadism much more costly).

1Adele Lopez15h

What I'm thinking of as "sadism" here isn't the BDSM kind, but people who specifically enjoy pessimizing others' values. E.g. the sentiment that generates the concept of hell. I'm also not super worried about the case where there are lots of other powerful entities to do meaningful trade with. I'm worried about the case where a particular person's values essentially wins outright. Maybe acausal trade still saves us, but I think there are coherent values that decide not to care about that (and this is one of those things that is more likely if the reflection process is butchered somehow).

Thane Ruthenis's Shortform

Thane Ruthenis

Alex Mallen18h30

@Aghyad Deeb @ryan_greenblatt and I just published "Towards understanding score-seeking in LLMs", which I think sheds some light on the general question of whether AIs hack because of score-seeking or proxy drives. I think it will be of interest to you and onlookers.

Neural nets are exceptionally vulnerable to goal self-hack

Stuart_Armstrong

13d

Suppose we have an AI CEO, ruthlessly devoted to maximising shareholder value. Fortunately, we haven't been completely stupid, and have given it a rich definition of shareholder value, encompassing the success, flourishing, safety, preference satisfaction, and moral values of the shareholders, not just the stock price. So it won't destroy the company - or the world - to build a giant stock-ticker with a swiftly increasing number on it.

This rich definition is contained in a sophisticated LLM, able to analyse the current situation and answer any question about the objective. The LLM is an integral part of the AI's system and motivation, not something it can simply set aside or hack directly.

Unfortunately, as discussed, there are going to be zones in the space of possible situations where...

(See More - 508 more words)

2Stuart_Armstrong1d

"adversarial examples are features, not bugs" It doesn't matter if the adversarial feature is a genuine feature - if it's a genuine negative feature (e.g. shareholder value cashing out on to nominal or actual money), it's still disastrous. Value is fragile in that keeping almost all of the properties of a desirable feature can be disastrous.

Charlie Steiner1d20

Point is, ResNet had human-imperceptible adversarial examples not because it's inevitable that thats's what happens to NNs, but because it used human-imperceptible information as part of a classification strategy that suited its data and training objective. For different data and training, you get different adversarial examples that can be closer to human perception of the world.

In what I think is the architecture you're talking about, then this would change the behavior from "invisible change that unlocks bad behavior" to "human-perceptible problematic ch... (read more)

Some “coherence arguments” are two different things in a trenchcoat

Steven Byrnes

The “two kids in a trenchcoat” trope, in case you haven’t heard of it. Image sources: 1, 2

tl;dr

I go through three “coherence arguments”, related to (1) market competition, (2) AI self-reflection and self-modification, and (3) AI training algorithms. I argue that each of these implies a pressure for (something like) “being self-consistent”, and also a pressure for (something like) “only caring about long-term eventual outcomes, with ruthless indifference to anything else”. But the latter pressure tends to be weaker and/or more contingent than the former. So it's bad to lump them together and call them by the same name.

1. An example “coherence argument” that I’ll focus on for most of the post

I’ll talk about two other “coherence arguments” at the end (§6), but let’s start with the...

(Continue Reading - 2745 more words)

Alexander Gietelink Oldenziel1d30

I agree with you that they're different. I agree that coherence arguments as usually conceived only weakly point towards long-term coherent goal-directedness.
From my point of view, the pressure towards long horizon ruthless goal optimizing hasn't been sufficiently conceptualized and mathematicized for us to have strong formal arguments -though of course I agree with you that our informal arguments and evidence points sufficiently strongly towards ruthless goal optimizers in long-term. There is a lot of conceptual and mathematical work to do here.

That sai... (read more)

Additional Course Context (for AI Safety Fundamentals; unofficial)

Chris_Leong

This is a linkpost for https://substack.com/home/post/p-223154283

Context: I created this to share before the start of courses I facilitate to provide a bunch of additional context and frames. I emphasise to students that these posts are to be considered as bonus content—not core curriculum content—and that as such they are really just there for students who want to go deeper, before or after the course. I also provide a lot of links; this is to make it easy for students to follow up anything that catches their interest.

Some general thoughts on advice

I suspect that most advice (including most of the advice I’ve given in the past) is bad. People have very different goals, personalities, skills and opportunities and unless you know someone them really well, it’s hard to cater advice to their...

(Continue Reading - 3393 more words)