AI ALIGNMENT FORUM
AF

AI Alignment Forum

Schelling Goodness, and Shared Morality as a Goal

My problem with your treatment of the civilization that's happy to steal from the outgroup isn't that they'll disagree that "stealing is bad" is the Schelling answer to that question[1]. It's that they'll think the question is unnatural - you've lumped together two different things, "stealing from the ingroup" and "stealing from the outgroup," and if you split the question up you'd get much more natural agreement that "stealing from the ingroup is bad" is the Schelling answer as is "stealing from the outgroup is good". Asking different questions (or equivalently, defining words in different ways as you ask the question) leads to different generalization behavior, if you're being influenced by your conception of the "shared morality." 1. ^ Assuming you pick the same reference population - if we're using the standard "success at being a civilization like ours" (even as an implicit meta-standard we use for picking our other standards), they might use "success at being a civilization like theirs." If weighting by resources commanded, I think you're underweighting bacteria and singletons that have eaten their planet of origin.

Victor Levoso22d*514

Why we should expect ruthless sociopath ASI

LLM in practice these days do include increasingly bigger % of RL wich seems like it should at least make you less certain about capabilities mostly coming from pretraining and papers from before that continuing to be relevant for very long and you do mention it on the other post but still wrote that capabilities come mostly from pretraining on the footnote?. I expect an optimist or someone from the comparatively-less-pessimistic group would argue that LLM or LLM +RL might lead to consequentialists that have human-like goals due to being built from a base of human imitation even as they move towards ASI. And an important disagrement with those people there is that you don't in fact expect LLM +RL to work. You also don't think LLM+ RL would be safe if they did work but still feels relevant because It comes from pretty diferent models of how LLM will improve. Relatedly a question you are not answering here is why you think imitation learning won't lead to discovering things that are far from the distribution of humans since its unclear what the limits are out of distribution and this also seems like you have a strong view on human imitation not generalicing much to the point It won't be even be able to do human level inventing of new things wich seems like would imply bad imitation of humans, as humans can do things like invent writting. You seem to be describing a shallow kind of imitation wich seems like a nontrivial asumtion people in the LLM camp would likely disagree with?. "rocket arbitrarily far past human training data" is mixing the question of "will LLM think in ways that are very diferent from humans" with something like " can LLM only shalowly reuse reasoning from their training data and not invent new things". Perfect human imitators will just rocket arbitrarily far past human training data in the first sense and invent writting etc.

Gordon Seidoh Worley1mo33

Distinguish between inference scaling and "larger tasks use more compute"

I think there's a hidden variable in this framework: the effective branching factor of the task. For well-specified tasks (prove this theorem, implement this API to spec, etc.), AI cost probably does scale roughly linearly with task size, matching your model. But most real-world engineering tasks aren't like that. The bottleneck isn't execution speed, it's the iterative discovery of what the task actually is. A senior engineer doing a 4-hour task isn't typing for 4 hours. They're making dozens of micro-decisions informed by tacit knowledge about the codebase, the product, the users, the org. They're constantly pruning the search space. An AI working autonomously on the same task produces tokens faster but explores a much larger space of wrong approaches, because it lacks that contextual judgment. So the cost might scale superlinearly with task complexity not because of compute limits but because of specification limits. This suggests the linear regime on your Pareto frontier might be shorter for real-world task distributions than for benchmark-style tasks, and the "fraction of human cost" metric could look very different depending on how well-specified the task is. The tasks where AI efficiency looks great (tight feedback loops, clear success criteria) are not the same tasks that fill most of an engineer's day. (Though note that obviously this could change if AI starts to have more context and better judgement, which would basically mean training a better model.)

Recent Discussion

The behavioral selection model for predicting AI motivations

Alex Mallen, Buck

3mo

Highly capable AI systems might end up deciding the future. Understanding what will drive those decisions is therefore one of the most important questions we can ask.

Many people have proposed different answers. Some predict that powerful AIs will learn to intrinsically pursue reward. Others respond by saying reward is not the optimization target, and instead reward “chisels” a combination of context-dependent cognitive patterns into the AI. Some argue that powerful AIs might end up with an almost arbitrary long-term goal.

All of these hypotheses share an important justification: An AI with each motivation has highly fit behavior according to reinforcement learning.

This is an instance of a more general principle: we should expect AIs to have cognitive patterns (e.g., motivations) that lead to behavior that causes those cognitive patterns...

(Continue Reading - 4575 more words)

TurnTrout18m20

Overall, I like this post a lot! 🙂

There are (in some sense, see discussion in Carlsmith for details) vastly more schemers than fitness-seekers.

Counting arguments are not valid arguments unless you ground them in discussion of the neural net prior. I think it's important to mention that here.

Schelling Goodness, and Shared Morality as a Goal

Andrew_Critch

13d

Also available in markdown at theMultiplicity.ai/blog/schelling-goodness.

This post explores a notion I'll call Schelling goodness. Claims of Schelling goodness are not first-order moral verdicts like "X is good" or "X is bad." They are claims about a class of hypothetical coordination games in the sense of Thomas Schelling, where the task being coordinated on is a moral verdict. In each such game, participants aim to give the same response regarding a moral question, by reasoning about what a very diverse population of intelligent beings would converge on, using only broadly shared constraints: common knowledge of the question at hand, and background knowledge from the survival and growth pressures that shape successful civilizations. Unlike many Schelling coordination games, we'll be focused on scenarios with no shared history or knowledge...

(Continue Reading - 11999 more words)

2Andrew_Critch4d

Inspired by a conversation below with @Wei Dai about the particular nesting of hypotheticals defining cosmic Schelling goodness, I just added the following clarifying section on Reality vs Hypotheticals: https://www.lesswrong.com/posts/TkBCR8XRGw7qmao6z/schelling-goodness-and-shared-morality-as-a-goal#Reality_vs_hypotheticals

3Tom Davidson7d

In our case, if we are ultimately running on a computer, then wouldn't that mean that we are a simulation? It seems obvious that we are trying; the intention would be to simulate a pre-AGI civilization. Could we be running on a computer and also be a vivarium?

3Andrew_Critch4d

I get that a lot of people use "simulation" and "computer program" as basically synonyms, but that's a bit linguistically impoverished for the hard work of analyzing distinct metaphysical hypotheses and their consequences. Consider that ideal behavior is different in a) computer worlds that are built to mimic an existing world, in which case our "job" is be similar to whatever is "out there" that we're a simulation of, versus b) computer worlds that are built to be a de novo home for a new kind of life or being, such as for entertainment or exploratory science. (a) is a "simulation", and if I discover I'm in a simulation, I might just go ahead an act like I'm not, to help the simulator with their intended purpose to mimic something that's not in a simulation. (b) is not a "simulation", and if we ever discover we're in one of those, I might look for other instructions for what I'm supposed or expected to do, from the creator(s). I call this a 'digital vivarium' but I'm open to other terms, just not 'simulation' which fails to correctly draw the distinction of, well, not being a simulation of anything. Not really, if the computer program we're inside is not designed to simulate anything. Yes, because artificial life is still life.

Tom Davidson12h10

Ok but if we're on a computer then isn't it clear we're a simulation, not a vivarium, bc clearly it was designed to simulate the behavior of a pre-agi civ?

The Refined Counterfactual Prisoner's Dilemma: An Attempt to Explode Decision-Theoretic Consequentialism

Chris_Leong

I was inspired to revise my formulation of this thought experiment by Ihor Kendiukhov's post On The Independence Axiom.

Kendiukhov quotes Scott Garrabrant:

My take is that the concept of expected utility maximization is a mistake. [...] As far as I know, every argument for utility assumes (or implies) that whenever you make an observation, you stop caring about the possible worlds where that observation went differently. [...] Von Neumann did not notice this mistake because he was too busy inventing the entire field. The point where wne discover updatelessness is the point where we are supposed to realize that all of utility theory is wrong. I think we failed to notice.

Apparently "stopping caring about the possible worlds where that observation went differently" is known as (decision-theoretic) consequentialism.

I was...

(See More - 507 more words)

2Dacyn1d

Well, in one version you are being extorted for money, whereas in the other version you are merely being bribed. If you buy Eliezer's theory that you should pay up for bribes but not for extortions (because paying up for bribes increases the probability that people will try to bribe you, which is good, but paying up for extortion increases the probability that people will try to extort you, which is bad), then the difference matters.

Chris_Leong1d10

Good point.

Assume no-one will ever know, that you can't disincentivise the actor and that they won't ever do anything like this again.

The case for satiating cheaply-satisfied AI preferences

Alex Mallen

A central AI safety concern is that AIs will develop unintended preferences and undermine human control to achieve them. But some unintended preferences are cheap to satisfy, and failing to satisfy them needlessly turns a cooperative situation into an adversarial one. In this post, I argue that developers should consider satisfying such cheap-to-satisfy preferences as long as the AI isn’t caught behaving dangerously, if doing so doesn't degrade usefulness or substantially risk making the AI more ambitiously misaligned.

This looks like a good idea for surprisingly many reasons:

It increases AIs’ desire to remain under developer control, rather than taking over or assisting adversaries.
1. It decreases the AI's upside in disempowering developers.
2. It incentivizes safe actions (because AIs don't receive payment if we notice them acting dangerously).
To the extent that the

...

(Continue Reading - 6851 more words)

4Thomas Kwa2d

Is there a case for deliberately training in cheaply satisfied AI preferences just so we can satisfy them? I think it's plausible that we can create AI motivations more easily than we can remove undesired ones.

3Alex Mallen2d

Yes! I'm quite excited by this proposal and I currently plan to write more about it and study it empirically. The basic idea is to try to make AIs' reward-hacking more responsive to satiation.

1Gordon Seidoh Worley2d

Wouldn't this come at the risk of reducing usefulness? While reward-hacking is not useful to us, it's something measured against what we think is a useful outcome. For the AI, reward-hacking is just getting reward since it can't see our judgements about it going too far. And if the AI tries less hard to get reward by completing tasks to give us what we want, that would make it less useful.

Buck2d54

Part of the post is about this.

TurnTrout's shortform feed

TurnTrout

TurnTrout3d5272

I signed an amicus brief supporting Anthropic's right to do business without governmental retaliation. As an AI expert, I attest that Anthropic's technical concerns are legitimate, and no laws were designed to protect against AI analysis of surveillance data.

Even though I work at a competing lab (Google DeepMind), I'm proud of Anthropic for taking a stand against unlawful retaliation and immoral demands.

(I speak only for myself, not my employer.)

AI ALIGNMENT FORUM
AF

AI ALIGNMENT FORUM
AF

AI Alignment Posts

Popular Comments

AI Alignment Posts

Popular Comments

Recent Discussion