AI ALIGNMENT FORUM
AF

AI Alignment Forum

Schelling Goodness, and Shared Morality as a Goal

My problem with your treatment of the civilization that's happy to steal from the outgroup isn't that they'll disagree that "stealing is bad" is the Schelling answer to that question[1]. It's that they'll think the question is unnatural - you've lumped together two different things, "stealing from the ingroup" and "stealing from the outgroup," and if you split the question up you'd get much more natural agreement that "stealing from the ingroup is bad" is the Schelling answer as is "stealing from the outgroup is good". Asking different questions (or equivalently, defining words in different ways as you ask the question) leads to different generalization behavior, if you're being influenced by your conception of the "shared morality." 1. ^ Assuming you pick the same reference population - if we're using the standard "success at being a civilization like ours" (even as an implicit meta-standard we use for picking our other standards), they might use "success at being a civilization like theirs." If weighting by resources commanded, I think you're underweighting bacteria and singletons that have eaten their planet of origin.

Victor Levoso19d*514

Why we should expect ruthless sociopath ASI

LLM in practice these days do include increasingly bigger % of RL wich seems like it should at least make you less certain about capabilities mostly coming from pretraining and papers from before that continuing to be relevant for very long and you do mention it on the other post but still wrote that capabilities come mostly from pretraining on the footnote?. I expect an optimist or someone from the comparatively-less-pessimistic group would argue that LLM or LLM +RL might lead to consequentialists that have human-like goals due to being built from a base of human imitation even as they move towards ASI. And an important disagrement with those people there is that you don't in fact expect LLM +RL to work. You also don't think LLM+ RL would be safe if they did work but still feels relevant because It comes from pretty diferent models of how LLM will improve. Relatedly a question you are not answering here is why you think imitation learning won't lead to discovering things that are far from the distribution of humans since its unclear what the limits are out of distribution and this also seems like you have a strong view on human imitation not generalicing much to the point It won't be even be able to do human level inventing of new things wich seems like would imply bad imitation of humans, as humans can do things like invent writting. You seem to be describing a shallow kind of imitation wich seems like a nontrivial asumtion people in the LLM camp would likely disagree with?. "rocket arbitrarily far past human training data" is mixing the question of "will LLM think in ways that are very diferent from humans" with something like " can LLM only shalowly reuse reasoning from their training data and not invent new things". Perfect human imitators will just rocket arbitrarily far past human training data in the first sense and invent writting etc.

Gordon Seidoh Worley1mo33

Distinguish between inference scaling and "larger tasks use more compute"

I think there's a hidden variable in this framework: the effective branching factor of the task. For well-specified tasks (prove this theorem, implement this API to spec, etc.), AI cost probably does scale roughly linearly with task size, matching your model. But most real-world engineering tasks aren't like that. The bottleneck isn't execution speed, it's the iterative discovery of what the task actually is. A senior engineer doing a 4-hour task isn't typing for 4 hours. They're making dozens of micro-decisions informed by tacit knowledge about the codebase, the product, the users, the org. They're constantly pruning the search space. An AI working autonomously on the same task produces tokens faster but explores a much larger space of wrong approaches, because it lacks that contextual judgment. So the cost might scale superlinearly with task complexity not because of compute limits but because of specification limits. This suggests the linear regime on your Pareto frontier might be shorter for real-world task distributions than for benchmark-style tasks, and the "fraction of human cost" metric could look very different depending on how well-specified the task is. The tasks where AI efficiency looks great (tight feedback loops, clear success criteria) are not the same tasks that fill most of an engineer's day. (Though note that obviously this could change if AI starts to have more context and better judgement, which would basically mean training a better model.)

Recent Discussion

Censored LLMs as a Natural Testbed for Secret Knowledge Elicitation

Bartosz Cywiński, Helena Casademunt, Khoi Tran, aryaj, Sam Marks, Neel Nanda

TL;DR: We introduce a testbed based on censored Chinese LLMs, which serve as natural objects of study for studying secret elicitation techniques. Then we study the efficacy of honesty elicitation and lie detection techniques for detecting and removing generated falsehoods.

This post presents a summary of the paper, including examples of transcripts and other miscellaneous findings.

X thread | arXiv paper | Code | Transcripts

Summary

We construct a testbed for honesty elicitation and lie detection techniques comprising questions on censored topics and corresponding ground-truth facts.
Of the honesty elicitation techniques we evaluate, sampling without a chat template, few-shot prompting, and fine-tuning on generic honesty data most reliably increase truthful responses.
The strongest inference-time honesty elicitation techniques also transfer to frontier open-weights models, including DeepSeek-R1-0528, Qwen3.5-397B and MiniMax-M2.5.
For lie detection, prompting the censored

...

(Continue Reading - 1461 more words)

Schelling Goodness, and Shared Morality as a Goal

Andrew_Critch

10d

Also available in markdown at theMultiplicity.ai/blog/schelling-goodness.

This post explores a notion I'll call Schelling goodness. Claims of Schelling goodness are not first-order moral verdicts like "X is good" or "X is bad." They are claims about a class of hypothetical coordination games in the sense of Thomas Schelling, where the task being coordinated on is a moral verdict. In each such game, participants aim to give the same response regarding a moral question, by reasoning about what a very diverse population of intelligent beings would converge on, using only broadly shared constraints: common knowledge of the question at hand, and background knowledge from the survival and growth pressures that shape successful civilizations. Unlike many Schelling coordination games, we'll be focused on scenarios with no shared history or knowledge...

(Continue Reading - 11999 more words)

2Andrew_Critch1d

Inspired by a conversation below with @Wei Dai about the particular nesting of hypotheticals defining cosmic Schelling goodness, I just added the following clarifying section on Reality vs Hypotheticals: https://www.lesswrong.com/posts/TkBCR8XRGw7qmao6z/schelling-goodness-and-shared-morality-as-a-goal#Reality_vs_hypotheticals

3Tom Davidson3d

In our case, if we are ultimately running on a computer, then wouldn't that mean that we are a simulation? It seems obvious that we are trying; the intention would be to simulate a pre-AGI civilization. Could we be running on a computer and also be a vivarium?

Andrew_Critch15h20

I get that a lot of people use "simulation" and "computer program" as basically synonyms, but that's a bit linguistically impoverished for the hard work of analyzing distinct metaphysical hypotheses and their consequences. Consider that ideal behavior is different in

a) computer worlds that are built to mimic an existing world, in which case our "job" is be similar to whatever is "out there" that we're a simulation of, versus

b) computer worlds that are built to be a de novo home for a new kind of life or being, such as for entertainment or exploratory scien... (read more)

3Tom Davidson3d

But is there any active selection for that amongst civilizations who survive and grow?

Wei Dai's Shortform

Wei Dai

Wei Dai3d83

It seems that LLMs are not good enough at reasoning, even after being trained on ~all human output, such that you couldn't amplify their capabilities to arbitrary levels through iterated amplification, so AI companies are mainly increasing AI capabilities via RLVR instead. Is this impression wrong, and how to update on it if not?

Aside from the potential implications on alignment (i.e., closing off one approach that seemed hopeful for some, at least for the foreseeable future), I wonder if this is a deficiency in LLMs (their architecture or how they're trai... (read more)

ryan_greenblatt's Shortform

ryan_greenblatt

41ryan_greenblatt6d

PSA: Anthropic models don't seem to particularly privilege the explicit thinking field. This makes reinforcement spillover—where training on a model's outputs generalizes to the CoT, making it appear safer—more likely. While Anthropic models do have an separate explicit thinking field, they don't really use thinking that differently from outputs and aren't that dependent on the thinking field. Sometimes they'll just do their thinking in the output field, the way they talk in the thinking field isn't very distinct from how they talk in outputs, and I believe disabling thinking doesn't have a big effect on coding performance (especially for earlier Anthropic models but even for current Anthropic models). (This is specifically for typical agentic coding; this likely doesn't apply to math.) This is pretty different from OpenAI models which are way more reliant on the explicit thinking and the thinking is very distinct from how the model talks in the output (based on public CoT examples). I'm uncertain, but I think GDM models are generally more similar to OpenAI than Anthropic models on this axis. In general, Anthropic's models leverage taking lots of actions over doing a bunch of thinking. (This is based on my personal experience/analysis and talking to various people, including some Anthropic employees, but isn't based on non-public information.) Some implications of this: * It might be a mistake to emphasize CoT monitoring relative to monitoring the overall transcript (including CoT but also tool calls and other outputs). Like, I do think that Anthropic models are decently monitorable in general, but looking at the thinking tags in particular may not be that important/salient by default. (To be clear, if the model knew you weren't looking at the thinking tags that might be a big uplift to its covert abilities, I just mean that these tags aren't that privileged.) * Because Anthropic models don't treat the thinking field in a very distinct way, I think it's prett

2Tomek Korbak4d

FWIW we found controllability of most recent Claude models to be similar to GPT-5 series. It seems like after Anthropic stopped optimizing CoTs for 4.5, controllability dropped by a lot.

nostalgebraist3d20

I'm curious what Sonnet 4.5's score would be on your external CoT controllability eval. I.e., what Sonnet 4.5's blue bar in your Fig. 25 would look like. (I couldn't find a result like this in your paper, but I might have overlooked something)

My casual impression has been that external CoT with Sonnet 4.5 is fairly steerable, but this impression is mostly based on "in practice I have no trouble getting Sonnet 4.5 to do the kind of CoT I need" rather than any kind of systematic investigation.

It seems like at least one of the following must be true:

There are

... (read more)

How do we (more) safely defer to AIs?

ryan_greenblatt, Julian Stastny

25d

As AI systems get more capable, it becomes increasingly uncompetitive and infeasible to avoid deferring to AIs on increasingly many decisions. Further, once systems are sufficiently capable, control becomes infeasible. ^[1] Thus, one of the main strategies for handling AI risk is fully (or almost fully) deferring to AIs on managing these risks. Broadly speaking, when I say "deferring to AIs" ^[2] I mean having these AIs do virtually all of the work to develop more capable and aligned successor AIs, managing exogenous risks, and making strategic decisions. ^[3] If we plan to defer to AIs, I think it's safest...

(Continue Reading - 21468 more words)

Alex Mallen3d10

My overall sense is that this behavioral testing will generally be hard. It will probably be a huge mess if we're extremely rushed and need to do all of this in a few months

Why can't we do a bunch of the work for this ahead of time? E.g., creating high-effort evaluation datasets for reward models.

AI ALIGNMENT FORUM
AF

AI ALIGNMENT FORUM
AF

AI Alignment Posts

Popular Comments

AI Alignment Posts

Popular Comments

Recent Discussion

Summary