AI ALIGNMENT FORUM
AF

AI Alignment Forum

Schelling Goodness, and Shared Morality as a Goal

My problem with your treatment of the civilization that's happy to steal from the outgroup isn't that they'll disagree that "stealing is bad" is the Schelling answer to that question[1]. It's that they'll think the question is unnatural - you've lumped together two different things, "stealing from the ingroup" and "stealing from the outgroup," and if you split the question up you'd get much more natural agreement that "stealing from the ingroup is bad" is the Schelling answer as is "stealing from the outgroup is good". Asking different questions (or equivalently, defining words in different ways as you ask the question) leads to different generalization behavior, if you're being influenced by your conception of the "shared morality." 1. ^ Assuming you pick the same reference population - if we're using the standard "success at being a civilization like ours" (even as an implicit meta-standard we use for picking our other standards), they might use "success at being a civilization like theirs." If weighting by resources commanded, I think you're underweighting bacteria and singletons that have eaten their planet of origin.

Victor Levoso20d*514

Why we should expect ruthless sociopath ASI

LLM in practice these days do include increasingly bigger % of RL wich seems like it should at least make you less certain about capabilities mostly coming from pretraining and papers from before that continuing to be relevant for very long and you do mention it on the other post but still wrote that capabilities come mostly from pretraining on the footnote?. I expect an optimist or someone from the comparatively-less-pessimistic group would argue that LLM or LLM +RL might lead to consequentialists that have human-like goals due to being built from a base of human imitation even as they move towards ASI. And an important disagrement with those people there is that you don't in fact expect LLM +RL to work. You also don't think LLM+ RL would be safe if they did work but still feels relevant because It comes from pretty diferent models of how LLM will improve. Relatedly a question you are not answering here is why you think imitation learning won't lead to discovering things that are far from the distribution of humans since its unclear what the limits are out of distribution and this also seems like you have a strong view on human imitation not generalicing much to the point It won't be even be able to do human level inventing of new things wich seems like would imply bad imitation of humans, as humans can do things like invent writting. You seem to be describing a shallow kind of imitation wich seems like a nontrivial asumtion people in the LLM camp would likely disagree with?. "rocket arbitrarily far past human training data" is mixing the question of "will LLM think in ways that are very diferent from humans" with something like " can LLM only shalowly reuse reasoning from their training data and not invent new things". Perfect human imitators will just rocket arbitrarily far past human training data in the first sense and invent writting etc.

Gordon Seidoh Worley1mo33

Distinguish between inference scaling and "larger tasks use more compute"

I think there's a hidden variable in this framework: the effective branching factor of the task. For well-specified tasks (prove this theorem, implement this API to spec, etc.), AI cost probably does scale roughly linearly with task size, matching your model. But most real-world engineering tasks aren't like that. The bottleneck isn't execution speed, it's the iterative discovery of what the task actually is. A senior engineer doing a 4-hour task isn't typing for 4 hours. They're making dozens of micro-decisions informed by tacit knowledge about the codebase, the product, the users, the org. They're constantly pruning the search space. An AI working autonomously on the same task produces tokens faster but explores a much larger space of wrong approaches, because it lacks that contextual judgment. So the cost might scale superlinearly with task complexity not because of compute limits but because of specification limits. This suggests the linear regime on your Pareto frontier might be shorter for real-world task distributions than for benchmark-style tasks, and the "fraction of human cost" metric could look very different depending on how well-specified the task is. The tasks where AI efficiency looks great (tight feedback loops, clear success criteria) are not the same tasks that fill most of an engineer's day. (Though note that obviously this could change if AI starts to have more context and better judgement, which would basically mean training a better model.)

39LawrenceC

This is a review of both the paper and the post itself, and turned more into a review of the paper (on which I think I have more to say) as opposed to the post. Disclaimer: this isn’t actually my area of expertise inside of technical alignment, and I’ve done very little linear probing myself. I’m relying primarily on my understanding of others’ results, so there’s some chance I’ve misunderstood something. Total amount of work on this review: ~8 hours, though about 4 of those were refreshing my memory of prior work and rereading the paper. TL;DR: The paper made significant contributions by introducing the idea of unsupervised knowledge discovery to a broader audience and by demonstrating that relatively straightforward techniques may make substantial progress on this problem. Compared to the paper, the blog post is substantially more nuanced, and I think that more academic-leaning AIS researchers should also publish companion blog posts of this kind. Collin Burns also deserves a lot of credit for actually doing empirical work in this domain when others were skeptical. However, the results are somewhat overstated and, with the benefit of hindsight, (vanilla) CCS does not seem to be a particularly promising technique for eliciting knowledge from language models. That being said, I encourage work in this area.[1] Introduction/Overview The paper “Discovering Latent Knowledge in Language Models without Supervision” by Collin Burns, Haotian Ye, Dan Klein, and Jacob Steinhardt (henceforth referred to as “the CCS paper” for short) proposes a method for unsupervised knowledge discovery, which can be thought of as a variant of empirical, average-case Eliciting Latent Knowledge (ELK). In this companion blog post, Collin Burns discusses the motivations behind the paper, caveats some of the limitations of the paper, and provides some reasons for why this style of unsupervised methods may scale to future language models. The CCS paper kicked off a lot of waves in the alig

Recent Discussion

TurnTrout's shortform feed

TurnTrout

TurnTrout12h3747

I signed an amicus brief supporting Anthropic's right to do business without governmental retaliation. As an AI expert, I attest that Anthropic's technical concerns are legitimate, and no laws were designed to protect against AI analysis of surveillance data.

Even though I work at a competing lab (Google DeepMind), I'm proud of Anthropic for taking a stand against unlawful retaliation and immoral demands.

(I speak only for myself, not my employer.)

41TurnTrout7d

Unless you have an explicit strategy for how and when you will speak out in the future, and explicit reasoning for why that requires silence now, "strategic silence" is just silence.

4Raemon7d

I think part of the problem is people think of themselves as having at least, like, a medium explicit strategy, but, the strategy routes through some judgment that conveniently keeps returning "not yet" or "only saying things in a somewhat cagey way." i.e. this advice seems necessary but not sufficient.

Censored LLMs as a Natural Testbed for Secret Knowledge Elicitation

Bartosz Cywiński, Helena Casademunt, Khoi Tran, aryaj, Sam Marks, Neel Nanda

16h

TL;DR: We introduce a testbed based on censored Chinese LLMs, which serve as natural objects of study for studying secret elicitation techniques. Then we study the efficacy of honesty elicitation and lie detection techniques for detecting and removing generated falsehoods.

This post presents a summary of the paper, including examples of transcripts and other miscellaneous findings.

X thread | arXiv paper | Code | Transcripts

Summary

We construct a testbed for honesty elicitation and lie detection techniques comprising questions on censored topics and corresponding ground-truth facts.
Of the honesty elicitation techniques we evaluate, sampling without a chat template, few-shot prompting, and fine-tuning on generic honesty data most reliably increase truthful responses.
The strongest inference-time honesty elicitation techniques also transfer to frontier open-weights models, including DeepSeek-R1-0528, Qwen3.5-397B and MiniMax-M2.5.
For lie detection, prompting the censored

...

(Continue Reading - 1461 more words)

Schelling Goodness, and Shared Morality as a Goal

Andrew_Critch

10d

Also available in markdown at theMultiplicity.ai/blog/schelling-goodness.

This post explores a notion I'll call Schelling goodness. Claims of Schelling goodness are not first-order moral verdicts like "X is good" or "X is bad." They are claims about a class of hypothetical coordination games in the sense of Thomas Schelling, where the task being coordinated on is a moral verdict. In each such game, participants aim to give the same response regarding a moral question, by reasoning about what a very diverse population of intelligent beings would converge on, using only broadly shared constraints: common knowledge of the question at hand, and background knowledge from the survival and growth pressures that shape successful civilizations. Unlike many Schelling coordination games, we'll be focused on scenarios with no shared history or knowledge...

(Continue Reading - 11999 more words)

2Andrew_Critch2d

Inspired by a conversation below with @Wei Dai about the particular nesting of hypotheticals defining cosmic Schelling goodness, I just added the following clarifying section on Reality vs Hypotheticals: https://www.lesswrong.com/posts/TkBCR8XRGw7qmao6z/schelling-goodness-and-shared-morality-as-a-goal#Reality_vs_hypotheticals

3Tom Davidson4d

In our case, if we are ultimately running on a computer, then wouldn't that mean that we are a simulation? It seems obvious that we are trying; the intention would be to simulate a pre-AGI civilization. Could we be running on a computer and also be a vivarium?

Andrew_Critch1d20

I get that a lot of people use "simulation" and "computer program" as basically synonyms, but that's a bit linguistically impoverished for the hard work of analyzing distinct metaphysical hypotheses and their consequences. Consider that ideal behavior is different in

a) computer worlds that are built to mimic an existing world, in which case our "job" is be similar to whatever is "out there" that we're a simulation of, versus

b) computer worlds that are built to be a de novo home for a new kind of life or being, such as for entertainment or exploratory scien... (read more)

3Tom Davidson4d

But is there any active selection for that amongst civilizations who survive and grow?

Wei Dai's Shortform

Wei Dai

Wei Dai3d83

It seems that LLMs are not good enough at reasoning, even after being trained on ~all human output, such that you couldn't amplify their capabilities to arbitrary levels through iterated amplification, so AI companies are mainly increasing AI capabilities via RLVR instead. Is this impression wrong, and how to update on it if not?

Aside from the potential implications on alignment (i.e., closing off one approach that seemed hopeful for some, at least for the foreseeable future), I wonder if this is a deficiency in LLMs (their architecture or how they're trai... (read more)

ryan_greenblatt's Shortform

ryan_greenblatt

41ryan_greenblatt7d

PSA: Anthropic models don't seem to particularly privilege the explicit thinking field. This makes reinforcement spillover—where training on a model's outputs generalizes to the CoT, making it appear safer—more likely. While Anthropic models do have an separate explicit thinking field, they don't really use thinking that differently from outputs and aren't that dependent on the thinking field. Sometimes they'll just do their thinking in the output field, the way they talk in the thinking field isn't very distinct from how they talk in outputs, and I believe disabling thinking doesn't have a big effect on coding performance (especially for earlier Anthropic models but even for current Anthropic models). (This is specifically for typical agentic coding; this likely doesn't apply to math.) This is pretty different from OpenAI models which are way more reliant on the explicit thinking and the thinking is very distinct from how the model talks in the output (based on public CoT examples). I'm uncertain, but I think GDM models are generally more similar to OpenAI than Anthropic models on this axis. In general, Anthropic's models leverage taking lots of actions over doing a bunch of thinking. (This is based on my personal experience/analysis and talking to various people, including some Anthropic employees, but isn't based on non-public information.) Some implications of this: * It might be a mistake to emphasize CoT monitoring relative to monitoring the overall transcript (including CoT but also tool calls and other outputs). Like, I do think that Anthropic models are decently monitorable in general, but looking at the thinking tags in particular may not be that important/salient by default. (To be clear, if the model knew you weren't looking at the thinking tags that might be a big uplift to its covert abilities, I just mean that these tags aren't that privileged.) * Because Anthropic models don't treat the thinking field in a very distinct way, I think it's prett

2Tomek Korbak5d

FWIW we found controllability of most recent Claude models to be similar to GPT-5 series. It seems like after Anthropic stopped optimizing CoTs for 4.5, controllability dropped by a lot.

nostalgebraist4d20

I'm curious what Sonnet 4.5's score would be on your external CoT controllability eval. I.e., what Sonnet 4.5's blue bar in your Fig. 25 would look like. (I couldn't find a result like this in your paper, but I might have overlooked something)

My casual impression has been that external CoT with Sonnet 4.5 is fairly steerable, but this impression is mostly based on "in practice I have no trouble getting Sonnet 4.5 to do the kind of CoT I need" rather than any kind of systematic investigation.

It seems like at least one of the following must be true:

There are

... (read more)

AI ALIGNMENT FORUM
AF

AI ALIGNMENT FORUM
AF

AI Alignment Posts

Popular Comments

AI Alignment Posts

Popular Comments

Recent Discussion

Summary