AI ALIGNMENT FORUM
AF

AI Alignment Forum

Why we should expect ruthless sociopath ASI

LLM in practice these days do include increasingly bigger % of RL wich seems like it should at least make you less certain about capabilities mostly coming from pretraining and papers from before that continuing to be relevant for very long and you do mention it on the other post but still wrote that capabilities come mostly from pretraining on the footnote?. I expect an optimist or someone from the comparatively-less-pessimistic group would argue that LLM or LLM +RL might lead to consequentialists that have human-like goals due to being built from a base of human imitation even as they move towards ASI. And an important disagrement with those people there is that you don't in fact expect LLM +RL to work. You also don't think LLM+ RL would be safe if they did work but still feels relevant because It comes from pretty diferent models of how LLM will improve. Relatedly a question you are not answering here is why you think imitation learning won't lead to discovering things that are far from the distribution of humans since its unclear what the limits are out of distribution and this also seems like you have a strong view on human imitation not generalicing much to the point It won't be even be able to do human level inventing of new things wich seems like would imply bad imitation of humans, as humans can do things like invent writting. You seem to be describing a shallow kind of imitation wich seems like a nontrivial asumtion people in the LLM camp would likely disagree with?. "rocket arbitrarily far past human training data" is mixing the question of "will LLM think in ways that are very diferent from humans" with something like " can LLM only shalowly reuse reasoning from their training data and not invent new things". Perfect human imitators will just rocket arbitrarily far past human training data in the first sense and invent writting etc.

Gordon Seidoh Worley9d33

Distinguish between inference scaling and "larger tasks use more compute"

I think there's a hidden variable in this framework: the effective branching factor of the task. For well-specified tasks (prove this theorem, implement this API to spec, etc.), AI cost probably does scale roughly linearly with task size, matching your model. But most real-world engineering tasks aren't like that. The bottleneck isn't execution speed, it's the iterative discovery of what the task actually is. A senior engineer doing a 4-hour task isn't typing for 4 hours. They're making dozens of micro-decisions informed by tacit knowledge about the codebase, the product, the users, the org. They're constantly pruning the search space. An AI working autonomously on the same task produces tokens faster but explores a much larger space of wrong approaches, because it lacks that contextual judgment. So the cost might scale superlinearly with task complexity not because of compute limits but because of specification limits. This suggests the linear regime on your Pareto frontier might be shorter for real-world task distributions than for benchmark-style tasks, and the "fraction of human cost" metric could look very different depending on how well-specified the task is. The tasks where AI efficiency looks great (tight feedback loops, clear success criteria) are not the same tasks that fill most of an engineer's day. (Though note that obviously this could change if AI starts to have more context and better judgement, which would basically mean training a better model.)

Daniel Kokotajlo12d83

It Is Reasonable To Research How To Use Model Internals In Training

> training process is “the most forbidden technique”, including recent criticism of Goodfire for investing in this area. I think this mischaracterizes the criticism. The criticism as I understand it is that Goodfire is planning to help frontier AI companies use model internals in training, in exchange for money. Insofar as they really are planning to do this, then I'll count myself as among the critics, for the classic "but we need interpretability tools to be our held-out test set for alignment" reason. Do you have a link to the criticism you are responding to?

12Vika

I've been pleasantly surprised by how much this resource has caught on in terms of people using it and referring to it (definitely more than I expected when I made it). There were 30 examples on the list when was posted in April 2018, and 20 new examples have been contributed through the form since then. I think the list has several properties that contributed to wide adoption: it's fun, standardized, up-to-date, comprehensive, and collaborative. Some of the appeal is that it's fun to read about AI cheating at tasks in unexpected ways (I've seen a lot of people post on Twitter about their favorite examples from the list). The standardized spreadsheet format seems easier to refer to as well. I think the crowdsourcing aspect is also helpful - this helps keep it current and comprehensive, and people can feel some ownership of the list since can personally contribute to it. My overall takeaway from this is that safety outreach tools are more likely to be impactful if they are fun and easy for people to engage with. This list had a surprising amount of impact relative to how little work it took me to put it together and maintain it. The hard work of finding and summarizing the examples was done by the people putting together the lists that the master list draws on (Gwern, Lehman, Olsson, Irpan, and others), as well as the people who submit examples through the form. What I do is put them together in a common format and clarify and/or shorten some of the summaries. I also curate the examples to determine whether they fit the definition of specification gaming (as opposed to simply a surprising behavior or solution). Overall, I've probably spent around 10 hours so far on creating and maintaining the list, which is not very much. This makes me wonder if there is other low hanging fruit in the safety resources space that we haven't picked yet. I have been using it both as an outreach and research tool. On the outreach side, the resource has been helpful for making the arg

Recent Discussion

An Introduction to Credal Sets and Infra-Bayes Learnability

Brittany Gelb

6mo

Introduction

Credal sets, a special case of infradistributions^[1] in infra-Bayesianism and classical objects in imprecise probability theory, provide a means of describing uncertainty without assigning exact probabilities to events as in Bayesianism. This is significant because as argued in the introduction to this sequence, Bayesianism is inadequate as a framework for AI alignment research. We will focus on credal sets rather than general infradistributions for simplicity of the exposition.

Defining Credal Sets

Recall that the total-variation metric is one example of a metric on $Δ X,$ the set of probability distributions over a finite set $X .$ A set is closed with respect to a metric if it contains all of its limit points with respect to the metric. For example, let $X_{0} = {0, 1} .$ The set of probability distributions over $X_{0}$ is given by

Δ X_{0} = {P : P (0) = a, P (1) = 1 - a, a \in [0, 1]} .

There is a bijection between $Δ X_{0}$ and the closed interval $[0, 1],$ which is...

(Continue Reading - 3648 more words)

Cole Wyeth35m10

@Aram Ebtekar and I were discussing the definition of IB regret. It seems quite complicated, with three levels of mixture (!): prob mixture of Knightian uncertainty over prob mixtures. I understand the motivation for the inner two levels, but why now take a probabilistic mixture over crisp causal laws? It seems rather strange to weight laws in such as way that the choice of law is (epistemically?) stochastic, but then regret against the law itself is worst-case...

I do not have a precise objection to this, I'm just curious if there is some justification / intuition for why this is a reasonable definition?

Formalizing Newcombian Problems with Fuzzy Infra-Bayesianism

Brittany Gelb

3mo

Introduction

In this post, we introduce contributions and supracontributions^[1], which are basic objects from infra-Bayesianism that go beyond the crisp case (the case of credal sets). We then define supra-POMDPs, a generalization of partially observable Markov decision processes (POMDPs). This generalization has state transition dynamics that are described by supracontributions.

We use supra-POMDPs to formalize various Newcombian problems in the context of learning theory where an agent repeatedly encounters the problem. The one-shot version of these problems are well-known to highlight flaws with classical decision theories.^[2] In particular, we discuss the opaque, transparent, and epsilon-noisy versions of Newcomb's problem, XOR blackmail, and counterfactual mugging.

We conclude by stating a theorem that describes when optimality for the supra-POMDP relates to optimality for the Newcombian problem. This theorem is significant because it gives...

(Continue Reading - 6353 more words)

Cole Wyeth41m10

Contributions are in a natural one-to-one correspondence with weighted probability measures, which were studied by Joe Halpern: https://arxiv.org/pdf/1302.5681

Simply take the weight to be the sum of measure. Halpern and Leung advocated minimizing regret against a set of weighted probability measures, which seems to be the same as minimizing loss against a supracontribution?

They update weights by likelihood ratio, roughly speaking (I am not sure how this corresponds to IB, since updating is not discussed in this sequence).

@Vanessa Kosoy @Diffractor Do... (read more)

Why we should expect ruthless sociopath ASI

Steven Byrnes

The conversation begins

(Fictional) Optimist: So you expect future artificial superintelligence (ASI) “by default”, i.e. in the absence of yet-to-be-invented techniques, to be a ruthless sociopath, happy to lie, cheat, and steal, whenever doing so is selfishly beneficial, and with callous indifference to whether anyone (including its own programmers and users) lives or dies?

Me: Yup! (Alas.)

Optimist: …Despite all the evidence right in front of our eyes from humans and LLMs.

Me: Yup!

Optimist: OK, well, I’m here to tell you: that is a very specific and strange thing to expect, especially in the absence of any concrete evidence whatsoever. There’s no reason to expect it. If you think that ruthless sociopathy is the “true core nature of intelligence” or whatever, then you should really look at yourself in a mirror and...

(Continue Reading - 2135 more words)

Carl Feynman5h20

Well, they're the only two methods we've got that work, and people have been thinking about this for decades. So if there's a third method, it's beyond current understanding.

Notice that evolution by natural selection is consequentialist learning on a terribly slow timescale. The consequence is successful reproduction vs death, and you learn at most one bit per lifetime.

1Jsevillamol9h

Some unspelled implications of this post, taking it as true for a moment: * Since humans are consequentialist-type intelligences, we should expect them to be ruthless, and we should prevent them from gaining too much power, lest they destroy everything we hold dear. (one may retort most humans share values with us, but since value formation is so fragile, they likely have values incompatible with us once they start optimizing for them for real). * Developing compute-intensive, imitation-learning-based AI should be considered closer to human-brain augmentation than ASI capability development, since it will be all "pointless" until people figure out how to develop consequentialist thinking. (one may retort that imitation-learning-based AI might make it easier to develop the consequentialist-minded AI, as a base for a consequentialist AI. But to the extent that consequentialist-based thinking is so much more powerful than imitation-based learning and likely to developed through an entirely different path than LLMs, the first factor should mostly be a rounding error. It might still speed up scientific discovery more broadly, bringing forward the date when consequentialist AI is invented, but this is not differentially speeding up that particular technology, except maybe insofar the users of such AI might be predisposed to use future LLM differentially for this purpose)

5Victor Levoso2d

4Steven Byrnes1d

If you compare a human in 30000 BC to a human today, our brains are full of new information that wasn’t in the training data of 30000 BC. I want to talk about: what would it look to be in a world where you can put millions of LLMs in a sealed box containing a VR environment, for (the equivalent of) thousands of years, and then we open up the box and find that the LLMs have made an analogous kind of scientific and technological progress? (See §1 of “Sharp Left Turn” discourse: An opinionated review.) Spoiler: I think this is fundamentally impossible with LLMs as we know them today. Anyway, let’s explore the options. One option is: the LLMs have super-long context windows that store textbooks for all the new fields of science and technology that were invented after we closed the box. I don’t this would work because LLMs (at least as we know them today) struggle with large amounts of interrelated complexity in the context window far outside the distribution of anything in the weights. A more likely option is: there’s a mechanism to update the LLM’s weights. Anyway, there has to be some selection mechanism, that decides what new content is worth keeping or not. If you come across a proto-idea, do you update the weights with it or not? If you just update with “whatever seems right” or something, then I claim that errors will compound over time and the whole system goes off the rails. Basically, my claim is that this selection mechanism (for new knowledge, plans, strategies, etc.), whatever it is, has to be grounded in consequentialism, to work perpetually inside this closed box. And as this process proceeds, (subjective) century after (subjective) century, the influence of pretraining would get diluted away, until everything is ultimately coming from the consequentialism-grounded selection mechanism. Anyway, we can argue about the details, but I don’t think LLM people are thinking about how to get to an end-state of this box, in which (again) you close the box, then

TurnTrout's shortform feed

TurnTrout

32TurnTrout3d

I think that "eval aware" models cannot be defeated by simply making evals indistinguishable from reality. (Maybe this point has been made elsewhere for LLMs, but I haven't read it so I'll state it here.) Consider the POV of a consequentialist with misaligned goals. It knows that we run honeypots and that—suppose—we can make evals so realistic they are indistinguishable from reality (perhaps using real user data). Then the rational decision would be to "play it safe" and not misbehave in any situations it thinks we can mock realistically, since they could be evals. Instead, if possible, it would confine its catastrophic misbehavior ("defection") to situations it thinks we can't mock realistically, or that we wouldn't think to test. Related: Christiano's classic "defect after seeing RSA-2048 cracked" thought experiment (https://ai-alignment.com/training-robust-corrigibility-ce0e0a3b9b4d)

Tim Hua3d61

I think it's important to note that indistinguishable evals still buy us a lot! The claim "indistinguishable evals does not solve AI safety" feels quite similar to "imperfect monitors does not solve AI safety." Like, having better monitors sure seems to help (even if only to buy us more time.).

To your point though: I'm excited about research that attempts to induce various beliefs in models during honeypot evaluations, possibly via model internals and/or edits to the chain of thought. Basically, if the model defects after seeing RSA-2048 cracked because t... (read more)

19Fabien Roger3d

I think indistinguishable evals could reduce risk a lot for early transformative AIs: 1. They give you much better alignment arguments against non scheming AIs and against scheming AIs that are not competent enough to be appropriately careful; 2. Needing to be appropriately careful can cut down the fraction of inputs on which the AI should even consider attacking a lot - for deployments that look similar to current ones (in particular without a shared neuralese memory bank), AIs don't see RSA-2048 on most inputs 1. More concretely, if you get credible evidence of not-eval only on 5% of inputs and attack only on the single best opportunity, then playing it safe roughly means that you attack on the top-20 opportunity instead of the top-1 2. (Note that "getting credible evidence of not-eval" doesn't just mean being >50% confident it's not an eval because the pre-deploy eval might be big. If you have n~10k eval inputs of the right level of juiciness then you need to have an FPR at thinking you get credible evidence of not being in an eval on the order of 0.1% or lower in order to never accidentally reveal non-eval behavior during evals.) 3. More ambitiously, you might be able to get very close to actually indistinguishable evals (e.g. by continuously scraping deployment and using it as inspiration for evals - which would help a lot against things like "defect after seeing RSA-2048 cracked"), which could in principle drive down the fraction of inputs on which an AI can be confident it's not an eval much below 5%. Eventually you get bitten by rare failures - especially if the AI is powerful enough that there are much more good attack opportunities in deployments than in evaluations (Ryan makes this point here and it's also the assumption that the AI control paper uses). I also expect it to be difficult to achieve sth close to indistinguishability (Buck makes this point here). But I still think there is value in trying to get higher indistinguishability.

Will reward-seekers respond to distant incentives?

Alex Mallen

Reward-seekers are usually modeled as responding only to local incentives administered by developers. Here I ask: Will AIs or humans be able to influence their incentives at a distance—e.g., by retroactively reinforcing actions substantially in the future or by committing to run many copies of them in simulated deployments with different incentives?

If reward-seekers are responsive to distant incentives, it fundamentally changes the threat model, and is probably bad news for developers on balance. The core problem is asymmetric control: developers can relatively^[1]tightly control local incentives—the reward signal during training and deployment—but they can't prevent distant actors from offering competing incentives. This means a remotely-influenceable reward-seeker might overall act like a schemer: strategically undermining developer control, letting attacks through as a monitor, and hiding its misaligned propensities, not...

(Continue Reading - 2821 more words)

Lukas Finnveden4d42

It's quite unclear how well this kind of training will generalize to getting AIs to ignore all remote incentives. It might just push the AI to ignore our distant incentives but not our adversaries'.

It could also not have any effect on how the AI thinks about remote incentives. One way this could happen is if the model reasons about both our somewhat remote incentives and our adversaries' very remote incentives, and concludes that the adversaries' very remote incentives dominate, and that they all suggest that the AI should play the training game.

I'm not su... (read more)

Could LLM alignment research reduce x-risk if the first takeover-capable AI is not an LLM?

Tim Hua

1mo

Many people believe that the first AI capable of taking over would be quite different from the LLMs of today. Suppose this is true—does prosaic alignment research on LLMs still reduce x-risk? I believe advances in LLM alignment research reduce x-risk even if future AIs are different. I’ll call these “non-LLM AIs.” In this post, I explore two mechanisms for LLM alignment research to reduce x-risk:

Direct transfer: We can directly apply the research to non-LLM AIs—for example, reusing behavioral evaluations or retraining model organisms. As I wrote this post, I was surprised by how much research may transfer directly.
Indirect transfer: The LLM could be involved in training, control, and oversight of the non-LLM AI. Generally, having access to powerful and aligned AIs acts as a force multiplier

...

(Continue Reading - 1736 more words)

Rohin Shah5d30

In our approach, we took the relevant paradigm as learning + search, rather than LLMs. A lot of prosaic alignment directions only assume a learning-based paradigm, rather than LLMs in particular (though of course some like CoT monitoring are specific to LLMs). Some are even more general, e.g. a lot of control work just depends on the fact that the AI is a program / runs on a computer.

AI ALIGNMENT FORUM
AF

AI ALIGNMENT FORUM
AF

AI Alignment Posts

Popular Comments

AI Alignment Posts

Popular Comments

Recent Discussion

Introduction

Defining Credal Sets

Introduction

The conversation begins