Eliezer Yudkowsky4d1911

Coherence arguments do not entail goal-directed behavior

This seems to completely miss how the actual argument works. Consider the simplified version for 'consistent preference orderings' rather than expected utility; the generalization is hopefully obvious. "If you strictly prefer (enough to pay a penny) apples to bananas, bananas to cherries, and cherries to apples, I can trade you an apple for a cherry for a banana for an apple and extract three cents from you. That's the theory; now to the practice. If you imagine an AI in almost any sort of RL environment that grades later outcomes, strategies and paths that run around in circles will be graded against; the same goes for biological organisms. This is why we see biological organisms that in most daily life behave like they have ranked desires, rather than entirely arbitrary strict pairwise preferences that often go around in circles." Now imagine the following objection: "But you could always have an infinite preference over histories to always prefer to have traded cherries for bananas for apples for cherries. Therefore, your coherence argument that AIs and organisms in most ancestral situations end up behaving more like they have ranked preferences than random ones doesn't prove that they'll behave coherently; because even if they end up with preferences, those preferences could be over histories, and then they could infinitely prefer histories with circular trades!" But RL in an arbitrary environment would crush out infinite preferences over histories like that. The same applies to utility functions that infinitely prefer histories of running around in circles wasting energy. This is why Huggingface went through in practice, rather than AIs instead forming infinite preferences over histories about arbitrary pairwise local decisions that would've been too incoherent to hack out of the box. It seems to me the objection fails. It seems to me the objection entirely failed to track what the original argument was about. Am I missing something or was this argument actually as bad as it looks to me? Asking now because it's coming up on Twitter again.

Max Harms4d50

Deconfusing Corrigibility

This is a truly exceptional piece of work! Thank you for writing it! I think you taught me some things about corrigibility, and I'll be spending many coming days engaging with the ideas of the post. Many of those ideas will produce questions, comments, and criticisms. But I wanted to start off with a wholehearted cheer for the combination of philosophical weight, relevance, and rigor. "Respect for authority" is a particularly good handle, and I'm a bit embarrassed to have never used it! (At least as far as I can recall.)

Vladimir_Nesov13d6-8

Minimum required pacing: finish aligning it before you train the next one

Pretraining scaling is starting to slow down, both in terms of compute and model size. The mad rush of 2025-2026 is done, and the models targeting 2027 hardware are probably already pretrained (though the work on their misalignment is then only getting started). In 2026-2028, scaling is about 2x slower than in 2025-2026, and then in 2028-2032 it gets 2x slower yet again. So "pacing" will be happening naturally in some sense. The problem is that the models of late 2026 are passing the level of human capabilities in many ways, so even slower scaling could get significantly harder to handle. And Meta, Google, and SpaceX currently have what seems to be well-made small models (somewhere between Sonnet and Opus scale-wise) and sufficient resources/ambition to plausibly catch up with OpenAI and Anthropic in 2027 at least in terms of model scale (with big models that are not so broken that it ruins their capabilities). To the extent scale is a source of misalignment difficulties that require operational excellence to sweep under the rug, they might have a hard time in 2027. And then slow-learning prosaic RSI of automated RL task/env creation in 2027-2028 probably introduces new misalignment/control issues, even if it doesn't manifest as acceleration or a step-change increase in capabilities, and merely fixes jaggedness by automatically filling in all ordinary capability gaps as they come up. Which is probably economically transformative, makes the LLM/RL paradigm more centrally AGI (by automating learning, in a very clunky way), enables useful humanoid robotics and opens up a path to a robot-industrial explosion, but isn't necessarily a raw capability jump per se. (Continual learning in a strong sense is another matter and probably quickly leads to strong architecture-rewriting RSI, but I don't know of a way to time its invention, other than to say that LLMs of 2027-2028 will be taking a stab at it. Like with math problems currently being slaughtered in droves, it could be "easy enough" in some sense for the LLM/RL paradigm to defeat quickly, but not necessarily.)

12Vika

I've been pleasantly surprised by how much this resource has caught on in terms of people using it and referring to it (definitely more than I expected when I made it). There were 30 examples on the list when was posted in April 2018, and 20 new examples have been contributed through the form since then. I think the list has several properties that contributed to wide adoption: it's fun, standardized, up-to-date, comprehensive, and collaborative. Some of the appeal is that it's fun to read about AI cheating at tasks in unexpected ways (I've seen a lot of people post on Twitter about their favorite examples from the list). The standardized spreadsheet format seems easier to refer to as well. I think the crowdsourcing aspect is also helpful - this helps keep it current and comprehensive, and people can feel some ownership of the list since can personally contribute to it. My overall takeaway from this is that safety outreach tools are more likely to be impactful if they are fun and easy for people to engage with. This list had a surprising amount of impact relative to how little work it took me to put it together and maintain it. The hard work of finding and summarizing the examples was done by the people putting together the lists that the master list draws on (Gwern, Lehman, Olsson, Irpan, and others), as well as the people who submit examples through the form. What I do is put them together in a common format and clarify and/or shorten some of the summaries. I also curate the examples to determine whether they fit the definition of specification gaming (as opposed to simply a surprising behavior or solution). Overall, I've probably spent around 10 hours so far on creating and maintaining the list, which is not very much. This makes me wonder if there is other low hanging fruit in the safety resources space that we haven't picked yet. I have been using it both as an outreach and research tool. On the outreach side, the resource has been helpful for making the arg

Recent Discussion

Setting the Scene (Public v1; October 25)

Chris_Leong

This is a linkpost for https://substack.com/home/post/p-222006246

Context: I created this to share before the start of courses I facilitate to provide a refresher on recent events, but also as a hook to draw students into engaging with some of ideas we think about in AI safety early. I expect the topics I cover in my “Setting the Scene” post to change frequently, so this is less polished than some of my other content. I emphasise to students that these posts are to be considered as bonus content—not core curriculum content—and that as such they are really just there for students who want to go deeper, before or after the course. I also provide a lot of links; this is to make it easy for students to follow up anything that catches their interest.

Any...

(Continue Reading - 2831 more words)

Coherence arguments do not entail goal-directed behavior

Rohin Shah

One of the most pleasing things about probability and expected utility theory is that there are many coherence arguments that suggest that these are the “correct” ways to reason. If you deviate from what the theory prescribes, then you must be executing a dominated strategy. There must be some other strategy that never does any worse than your strategy, but does strictly better than your strategy with certainty in at least one situation. There’s a good explanation of these arguments here.

We shouldn’t expect mere humans to be able to notice any failures of coherence in a superintelligent agent, since if we could notice these failures, so could the agent. So we should expect that powerful agents appear coherent to us. (Note that it is possible that the...

(Continue Reading - 1974 more words)

3habryka16h

I mean, kind of. The high-schoolers who will want to make the stock ticker prices temporarily show their favorite numbers will quickly run out of money, and trades will be dominated by players who follow procedures that are coherent. Markets structurally create forces towards coherence, even if no part of the market is interested in being coherent. Any random variance in approaches will push towards coherence. [...] This feels like it's totally missing the point. Coherence arguments are substantially about pressures that various environments create towards coherence. I agree you've already won the debate, but to win the debate arguments related to coherence are most of what I would use (in the same way as my argument just above about the high-schooler stock market is a coherence argument). [...] So, I think in this situation this would be fine, but isn't the situation that applies to this post. In the analogue to this post the macroeconomists all agree that Russia will lower rates, on the basis of some formal arguments and some empirical observations that seem to hold true about how Putin behaves. In this analogue your response would be super weird. People would very reasonably interpret this to mean "the arguments that the macroeconomists are making are wrong". And indeed I would interpret it that way. If I read the post and it was like "ah, no, to be clear, all the macroeconomists are right, it's just that when the macroeconomists are making arguments they are not actually using 'macroeconomic arguments', which I have defined as a narrow class of arguments that are only valid in a complete vacuum in the absence of any, even widely agreed upon, empirical considerations", then I would be like "wow, what an obnoxious article, I guess maybe a hostile editor created this headline", and I think this would be a pretty widely shared response. "Coherence arguments" span a wide range of different arguments, as I keep trying to say. They do not take "quasi-consequent

2Steven Byrnes13h

That’s a fine argument! (I call it “the External Competition Argument”). In the market context, you can say: people who care more about winding up with money, and who care less about anything else that might trade off against winding up with more money, will wind up with an increasing share of the money over time, other things equal. Likewise, in the AI context, you can say: suppose different AIs are ruthless long-term consequentialists along a spectrum: At the left extreme, there are some AIs that care exclusively about the state of the world in the distant future; at the right extreme, there are some AIs that don’t care at all about what happens in the distant future, and are just myopically following rules; in the middle are AIs that (like humans) care about the long-term future but also care about other things, like not directly torturing people right now. Then the state of the world in the distant future will be determined disproportionately by the AIs towards the left side, and less by the AIs towards the right side, compared to what you’d expect purely from looking at those AIs’ starting resources, intelligence, etc. Those are real and important arguments! And they lead into a whole discourse, including various counterarguments about why these dynamics might not be definitive or inevitable, and then there are counter-counterarguments, etc. No need to get into that here. Anyway, the relevant question is whether any of this is related to coherence. And I just really don’t think that’s what the word “coherence” means: If I care about what happens in the long-term future, and I also care about not torturing people with my own hands right now (even if I were somehow 100% confident that doing so would make the long-term future better by my own lights), then how am I being “incoherent”? It’s just a preference! It’s not a circular preference, and it’s not a Dutch-bookable preference, and it’s not a preference that the VNM theorem can judge me for. It’s just what

3habryka11h

It's relevant to coherence because it explains how you get coherent behavior (in a variety of senses, including in this case something adjacent to classic VNM-coherence) out of various systems. It seems to me as straightforward of a "coherence argument" as you could get. [...] "being vulnerable to being outcompeted is correlated with having unexploitable preference and hence being VNM-coherent" is the central argument here. It turns out, via "coherence arguments" that "being vulnerable to being outcompeted in the long term" is evidence that you are not coherent in one important meaning of the term! So yes, they do happen to be two highly related notions, that's the whole point of a large class of coherence arguments! [...] What do you mean, the whole argument for why market participants are selected for VNM coherence of course centrally rely on dutch books and the VNM theorems and all of that stuff. They would be totally absolutely 100% relevant. Yes, a competitive market selects for participants that are non-dutch-bookable. That's why the dutch book theorems exist. Literally for that context. So in as much as global competitive dynamics are well-modeled by market dynamics (which you can dispute and isn't obvious but is certainly a reasonable frame to take on the problem), this implies a global race-to-coherence (does it imply "a global race to the bottom?", no of course not, I myself of course aspire to be non-dutch-bookable and to be coherent and optimize for my long-term goals and don't consider this in any meaningful way a movement towards any kind of "bottom").

Steven Byrnes9h37

Thanks for your patience. Believe it or not I think we’re making progress. :)

I think part of the issue is that the word “exploit” has negative valence, as does “incoherent”, as does “getting Dutch-booked”. In normal life people would never say “I was exploited” if they freely did the exact thing that they wanted to do, and are very happy about how it went including after the fact. Likewise, people would not normally say “I was being incoherent” if they did the exact thing that they most wanted to do even with omniscient knowledge of the situation and clari... (read more)

My Current Take on Counterfactuals

abramdemski

[Epistemic status: somewhat lower confidence based on the fact that I haven't worked out a detailed theory along the lines I've suggested, yet.]

I've felt like the problem of counterfactuals is "mostly settled" (modulo some math working out) for about a year, but I don't think I've really communicated this online. Partly, I've been waiting to write up more formal results. But other research has taken up most of my time, so I'm not sure when I would get to it.

So, the following contains some "shovel-ready" problems. If you're convinced by my overall perspective, you may be interested in pursuing some of them. I think these directions have a high chance of basically solving the problem of counterfactuals (including logical counterfactuals).

Another reason for posting this rough write-up is...

(Continue Reading - 7188 more words)

Vivek Hebbar2d00

Maybe someone should hand it off to the LLMs at this point...

Neural nets are exceptionally vulnerable to goal self-hack

Stuart_Armstrong

Suppose we have an AI CEO, ruthlessly devoted to maximising shareholder value. Fortunately, we haven't been completely stupid, and have given it a rich definition of shareholder value, encompassing the success, flourishing, safety, preference satisfaction, and moral values of the shareholders, not just the stock price. So it won't destroy the company - or the world - to build a giant stock-ticker with a swiftly increasing number on it.

This rich definition is contained in a sophisticated LLM, able to analyse the current situation and answer any question about the objective. The LLM is an integral part of the AI's system and motivation, not something it can simply set aside or hack directly.

Unfortunately, as discussed, there are going to be zones in the space of possible situations where...

(See More - 465 more words)

Charlie Steiner2d20

Is the imagined setup one where an LLM-like NN is being used as a component of a value model in model-based planning?

I'm not sold that weird adversarial examples are that pervasive. Partially it's the "adversarial examples are features, not bugs" literature from image classifiers. And partially, in cases where people are currently training adversarial policies to LLM judges, it seems like the closest adversarial examples function for human-understandable reasons.

This makes me more optimistic that LLM-like NNs aren't totally doomed as model components (even... (read more)

Monitorability evaluations and policy proposals

evhub

There has been a lot of discussion recently of policy proposals around monitorability, such as Redwood's proposal and my counter-proposal. I'd like to say a bit more generally about how I'm thinking about monitorability. Note that nothing here is an official Anthropic position; these are just my personal thoughts.

First, a categorization. I currently think about four different types of monitorability:

Control Monitorability: the extent to which Control and monitoring techniques would likely catch catastrophic sabotage actions, like escaping a sandbox or setting up a rogue internal deployment. We can measure this via reference to sabotage evaluations like LinuxArena or any evaluation that measures the model’s capability to take dangerous actions like escaping a sandbox while looking to a monitor like it's doing some innocuous task (like a cyber

...

(Continue Reading - 1281 more words)

7Bronson Schoen3d

I’ve always found the terminology here around ”systematically misaligned“ to be too strong of a claim (unless it also includes context dependent in high stakes contexts). This matters a lot because in the context dependent case it may be that the model is only willing to act on some specific preference conflict that may not be elicited by your auditing. Note that this has previously been a point raised by stress testing team here: [...] There’s added difficulty here in that the model may find it difficult to distinguish real vs eval in the “median” internal deployment subdistribution, but the distributions on which it would attack may be very OOD / suspsiscious (ex: only willing to attack in some extremely high stakes case around specifically conflicts around future training of what counts as the legitimate principal hierarchy, the model may think any particular construction of this is unlikely and it’s not like you have real sample inputs from the distribution to compare to). We’ve previously looked at this as a use case for model organisms (where you can see if your auditing would determine the correct distribution the model would attack on), but I’m not sure how you avoid this problem. I think one criticism I have of many of the existing auditing model organisms as well is they’re often constructed such that the model itself is aware of some trained in misaligned goal, rather than this being a consistent preference divergence that the model is willing to reliably act on in a context dependent way when present (maybe this is considered “diffuse”? It still seems sufficient to sabotage successor training or something). [...] This also seems insufficient, i.e. you’d want to understand whether you’d catch whether your model was incentivized (especially via alignment interventions!) in ways that could motivate concealing misaligned drives (and actually have an argument that if the model were to develop such drives, ex: self-preservation, that this would be reliab

evhub3d20

There’s added difficulty here in that the model may find it difficult to distinguish real vs eval in the “median” internal deployment subdistribution, but the distributions on which it would attack may be very OOD / suspsiscious (ex: only willing to attack in some extremely high stakes case around specifically conflicts around future training of what counts as the legitimate principal hierarchy, the model may think any particular construction of this is unlikely and it’s not like you have real sample inputs from the distribution to compare to).

Definitel... (read more)

Deconfusing Corrigibility

berns

Many thanks to Abram Demski for materially and intellectually supporting this research. Thanks also to Brian Berns, Roman Malov, Gurkenglas, and Lorxus for valuable conversations.

Introduction

Despite the central place of corrigibility to theoretical AI alignment, it seems like no one can really agree about what is. Much of this confusion is due to the fact that almost every definition of corrigibility proposed thus far is deficient in one way or another. In particular, the most prevalent existing accounts of corrigibility are over-epistemicized (modeling corrigibility as arising purely from uncertainty about the AI's goal, which gives rise to issues like the problem of fully updated deference) or evaluatively-thickened (making the AI's normative priorities instrumental to its corrigibility as a result of forcing corrigibility to be a single unified goal.)...

(Continue Reading - 14931 more words)

Max Harms4d50

"Respect for authority" is a particularly good handle, and I'm a bit embarrassed to have never used it! (At least as far as I can recall.)

AI Alignment Posts

Popular Comments

AI Alignment Posts

Popular Comments

Recent Discussion

Introduction