AI ALIGNMENT FORUM
AF

AI Alignment Forum

Schelling Goodness, and Shared Morality as a Goal

My problem with your treatment of the civilization that's happy to steal from the outgroup isn't that they'll disagree that "stealing is bad" is the Schelling answer to that question[1]. It's that they'll think the question is unnatural - you've lumped together two different things, "stealing from the ingroup" and "stealing from the outgroup," and if you split the question up you'd get much more natural agreement that "stealing from the ingroup is bad" is the Schelling answer as is "stealing from the outgroup is good". Asking different questions (or equivalently, defining words in different ways as you ask the question) leads to different generalization behavior, if you're being influenced by your conception of the "shared morality." 1. ^ Assuming you pick the same reference population - if we're using the standard "success at being a civilization like ours" (even as an implicit meta-standard we use for picking our other standards), they might use "success at being a civilization like theirs." If weighting by resources commanded, I think you're underweighting bacteria and singletons that have eaten their planet of origin.

Victor Levoso16d*514

Why we should expect ruthless sociopath ASI

LLM in practice these days do include increasingly bigger % of RL wich seems like it should at least make you less certain about capabilities mostly coming from pretraining and papers from before that continuing to be relevant for very long and you do mention it on the other post but still wrote that capabilities come mostly from pretraining on the footnote?. I expect an optimist or someone from the comparatively-less-pessimistic group would argue that LLM or LLM +RL might lead to consequentialists that have human-like goals due to being built from a base of human imitation even as they move towards ASI. And an important disagrement with those people there is that you don't in fact expect LLM +RL to work. You also don't think LLM+ RL would be safe if they did work but still feels relevant because It comes from pretty diferent models of how LLM will improve. Relatedly a question you are not answering here is why you think imitation learning won't lead to discovering things that are far from the distribution of humans since its unclear what the limits are out of distribution and this also seems like you have a strong view on human imitation not generalicing much to the point It won't be even be able to do human level inventing of new things wich seems like would imply bad imitation of humans, as humans can do things like invent writting. You seem to be describing a shallow kind of imitation wich seems like a nontrivial asumtion people in the LLM camp would likely disagree with?. "rocket arbitrarily far past human training data" is mixing the question of "will LLM think in ways that are very diferent from humans" with something like " can LLM only shalowly reuse reasoning from their training data and not invent new things". Perfect human imitators will just rocket arbitrarily far past human training data in the first sense and invent writting etc.

Gordon Seidoh Worley23d33

Distinguish between inference scaling and "larger tasks use more compute"

I think there's a hidden variable in this framework: the effective branching factor of the task. For well-specified tasks (prove this theorem, implement this API to spec, etc.), AI cost probably does scale roughly linearly with task size, matching your model. But most real-world engineering tasks aren't like that. The bottleneck isn't execution speed, it's the iterative discovery of what the task actually is. A senior engineer doing a 4-hour task isn't typing for 4 hours. They're making dozens of micro-decisions informed by tacit knowledge about the codebase, the product, the users, the org. They're constantly pruning the search space. An AI working autonomously on the same task produces tokens faster but explores a much larger space of wrong approaches, because it lacks that contextual judgment. So the cost might scale superlinearly with task complexity not because of compute limits but because of specification limits. This suggests the linear regime on your Pareto frontier might be shorter for real-world task distributions than for benchmark-style tasks, and the "fraction of human cost" metric could look very different depending on how well-specified the task is. The tasks where AI efficiency looks great (tight feedback loops, clear success criteria) are not the same tasks that fill most of an engineer's day. (Though note that obviously this could change if AI starts to have more context and better judgement, which would basically mean training a better model.)

Recent Discussion

ryan_greenblatt's Shortform

ryan_greenblatt

40ryan_greenblatt3d

PSA: Anthropic models don't seem to particularly privilege the explicit thinking field. This makes reinforcement spillover—where training on a model's outputs generalizes to the CoT, making it appear safer—more likely. While Anthropic models do have an separate explicit thinking field, they don't really use thinking that differently from outputs and aren't that dependent on the thinking field. Sometimes they'll just do their thinking in the output field, the way they talk in the thinking field isn't very distinct from how they talk in outputs, and I believe disabling thinking doesn't have a big effect on coding performance (especially for earlier Anthropic models but even for current Anthropic models). (This is specifically for typical agentic coding; this likely doesn't apply to math.) This is pretty different from OpenAI models which are way more reliant on the explicit thinking and the thinking is very distinct from how the model talks in the output (based on public CoT examples). I'm uncertain, but I think GDM models are generally more similar to OpenAI than Anthropic models on this axis. In general, Anthropic's models leverage taking lots of actions over doing a bunch of thinking. (This is based on my personal experience/analysis and talking to various people, including some Anthropic employees, but isn't based on non-public information.) Some implications of this: * It might be a mistake to emphasize CoT monitoring relative to monitoring the overall transcript (including CoT but also tool calls and other outputs). Like, I do think that Anthropic models are decently monitorable in general, but looking at the thinking tags in particular may not be that important/salient by default. (To be clear, if the model knew you weren't looking at the thinking tags that might be a big uplift to its covert abilities, I just mean that these tags aren't that privileged.) * Because Anthropic models don't treat the thinking field in a very distinct way, I think it's prett

1Tomek Korbak1d

FWIW we found controllability of most recent Claude models to be similar to GPT-5 series. It seems like after Anthropic stopped optimizing CoTs for 4.5, controllability dropped by a lot.

nostalgebraist7h20

I'm curious what Sonnet 4.5's score would be on your external CoT controllability eval. I.e., what Sonnet 4.5's blue bar in your Fig. 25 would look like. (I couldn't find a result like this in your paper, but I might have overlooked something)

My casual impression has been that external CoT with Sonnet 4.5 is fairly steerable, but this impression is mostly based on "in practice I have no trouble getting Sonnet 4.5 to do the kind of CoT I need" rather than any kind of systematic investigation.

It seems like at least one of the following must be true:

There are

... (read more)

How do we (more) safely defer to AIs?

ryan_greenblatt, Julian Stastny

22d

As AI systems get more capable, it becomes increasingly uncompetitive and infeasible to avoid deferring to AIs on increasingly many decisions. Further, once systems are sufficiently capable, control becomes infeasible. ^[1] Thus, one of the main strategies for handling AI risk is fully (or almost fully) deferring to AIs on managing these risks. Broadly speaking, when I say "deferring to AIs" ^[2] I mean having these AIs do virtually all of the work to develop more capable and aligned successor AIs, managing exogenous risks, and making strategic decisions. ^[3] If we plan to defer to AIs, I think it's safest...

(Continue Reading - 21468 more words)

Alex Mallen7h10

My overall sense is that this behavioral testing will generally be hard. It will probably be a huge mess if we're extremely rushed and need to do all of this in a few months

Why can't we do a bunch of the work for this ahead of time? E.g., creating high-effort evaluation datasets for reward models.

Schelling Goodness, and Shared Morality as a Goal

Andrew_Critch

Also available in markdown at theMultiplicity.ai/blog/schelling-goodness.

This post explores a notion I'll call Schelling goodness. Claims of Schelling goodness are not first-order moral verdicts like "X is good" or "X is bad." They are claims about a class of hypothetical coordination games in the sense of Thomas Schelling, where the task being coordinated on is a moral verdict. In each such game, participants aim to give the same response regarding a moral question, by reasoning about what a very diverse population of intelligent beings would converge on, using only broadly shared constraints: common knowledge of the question at hand, and background knowledge from the survival and growth pressures that shape successful civilizations. Unlike many Schelling coordination games, we'll be focused on scenarios with no shared history or knowledge...

(Continue Reading - 11618 more words)

2Andrew_Critch19h

Because there are so many more possible viviaria than simulations. Like, a simulation is simulating something, which involves a kind of match between what's in the simulation and what it's supposed to represent/simulate. If it contains life and life is the object of study, it's a vivarium, and I think most simulations are vivaria, but most vivaria are probably not simulations.

Tom Davidson14h10

In our case, if we are ultimately running on a computer, then wouldn't that mean that we are a simulation? It seems obvious that we are trying; the intention would be to simulate a pre-AGI civilization.

Could we be running on a computer and also be a vivarium?

3Andrew_Critch19h

Yep, I think in the end we'll settle on irreducible-computation-causation as the main notion of causation relevant to morality, and then there won't be a special case to be made about the causal/acausal distinction.

3Tom Davidson14h

But is there any active selection for that amongst civilizations who survive and grow?

Why we should expect ruthless sociopath ASI

Steven Byrnes

16d

The conversation begins

(Fictional) Optimist: So you expect future artificial superintelligence (ASI) “by default”, i.e. in the absence of yet-to-be-invented techniques, to be a ruthless sociopath, happy to lie, cheat, and steal, whenever doing so is selfishly beneficial, and with callous indifference to whether anyone (including its own programmers and users) lives or dies?

Me: Yup! (Alas.)

Optimist: …Despite all the evidence right in front of our eyes from humans and LLMs.

Me: Yup!

Optimist: OK, well, I’m here to tell you: that is a very specific and strange thing to expect, especially in the absence of any concrete evidence whatsoever. There’s no reason to expect it. If you think that ruthless sociopathy is the “true core nature of intelligence” or whatever, then you should really look at yourself in a mirror and...

(Continue Reading - 2143 more words)

3Donald Hobson2d

This is a bit of a steelman for the optimist, or maybe a strawman for the pessimist. Pessimist: Laser tech is getting more and more powerful. Soon someone will make an infinitely powerful laser and incinerate the whole planet. Optimist: My laser pointer isn't infinitely powerful, and this metal cutting laser isn't infinitely powerful either. Pessimist: That's just because they are imperfect. Look at my simplistic maths. The light should be exponentially amplified without limit. A perfect consequential AI is a friction-less sphere in a vacuum, or an infinitely powerful laser. It's the sort of thing that is very simple to describe mathematically, and rather hard to make in practice. Was deep blue a perfect "ruthless consequentialist"? It wouldn't sacrifice a pawn to save humanity. But it also wouldn't try to blackmail it's opponent or bribe a judge. It's just not thinking about anything off the chessboard. I think this point generalizes somewhat. I don't think it's that hard to make an AI that's superhuman at stock trading, and that just doesn't think about anything outside the stock market. Ruthless consequentialism works well in the limit. But evolution didn't build that. Because something more like human morality must have worked better, under conditions of bounded data, compute etc. Of course human morality in particular depends somewhat on culture, and also on the specific details of the evolutionary environment, and probably the limits of neurons. However, recursive self improvement by an AI that is powerful enough to do useful AI research and that sees ruthless consequentialism as a goal could lead to an AI that's very close to ruthless consequentialism. And I am not convinced, that if we do get a less consequentialist AI, it will be non-consequentialist in a way beneficial to us.

Steven Byrnes2d20

Thanks!

In this post I was NOT talking about consequentialism as a model of an AI agent, but rather consequentialism as the power-source ultimately underlying a family of AI techniques, a family that includes most forms of RL and model-based planning.

So “perfect” consequentialism is a red herring (I guess in agreement with your final sentence).

A model-based planning or RL agent can “make mistakes” (so to speak) while also being a ruthless sociopath. “Ruthless sociopath” here means having callous indifference to the welfare of other people, and to respecting... (read more)

TurnTrout's shortform feed

TurnTrout

40TurnTrout4d

Unless you have an explicit strategy for how and when you will speak out in the future, and explicit reasoning for why that requires silence now, "strategic silence" is just silence.

Raemon3d43

I think part of the problem is people think of themselves as having at least, like, a medium explicit strategy, but, the strategy routes through some judgment that conveniently keeps returning "not yet" or "only saying things in a somewhat cagey way."

i.e. this advice seems necessary but not sufficient.

How to Design Environments for Understanding Model Motives

gersonkroiz, aditya singh, Senthooran Rajamanoharan, Neel Nanda

Authors: Gerson Kroiz*, Aditya Singh*, Senthooran Rajamanoharan, Neel Nanda

Gerson and Aditya are co-first authors. This work was conducted during MATS 9.0 and was advised by Senthooran Rajamanoharan and Neel Nanda.

TL;DR

Understanding why a model took an action is a key question in AI Safety. It is a difficult question, as often many motivations could plausibly lead to the same action. We are particularly concerned about the case of model incrimination, i.e. observing a model taking a bad action and distinguishing between benign and malign motivations, e.g. scheming vs confusion.

To practice the skill of understanding motivations, we need to build high-quality environments that serve as realistic proxy tasks for investigating future models. But we’ve found that there are many ways in which environments can be contrived. It is valuable...

(Continue Reading - 2710 more words)

9Olli Järviniemi

I view this post as providing value in three (related) ways: 1. Making a pedagogical advancement regarding the so-called inner alignment problem 2. Pointing out that a common view of "RL agents optimize reward" is subtly wrong 3. Pushing for thinking mechanistically about cognition-updates Re 1: I first heard about the inner alignment problem through Risks From Learned Optimization and popularizations of the work. I didn't truly comprehend it - sure, I could parrot back terms like "base optimizer" and "mesa-optimizer", but it didn't click. I was confused. Some months later I read this post and then it clicked. Part of the pedagogical value is not having to introduce the 4 terms of form [base/mesa] + [optimizer/objective] and throwing those around. Even with Rob Miles' exposition skills that's a bit overwhelming. Another part I liked were the phrases "Just because common English endows “reward” with suggestive pleasurable connotations" and "Let’s strip away the suggestive word “reward”, and replace it by its substance: cognition-updater." One could be tempted to object and say that surely no one would make the mistakes pointed out here, but definitely some people do. I did. Being a bit gloves off here definitely helped me. Re 2: The essay argues for, well, reward not being the optimization target. There is some deep discussion in the comments about the likelihood of reward in fact being the optimization target, or at least quite close (see here). Let me take a more shallow view. I think there are people who think that reward is the optimization target by definition or by design, as opposed to this being a highly non-trivial claim that needs to be argued for. It's the former view that this post (correctly) argues against. I am sympathetic to pushback of the form "there are arguments that make it reasonable to privilege reward-maximization as a hypothesis" and about this post going a bit too far, but these remarks should not be confused with a rebuttal

AI ALIGNMENT FORUM
AF

AI ALIGNMENT FORUM
AF

AI Alignment Posts

Popular Comments

AI Alignment Posts

Popular Comments

Recent Discussion

The conversation begins

TL;DR