AI ALIGNMENT FORUM
AF

by Daniel Kokotajlo, Wei Dai
3 min read
26

32

There are so many causes or sources of AI risk that it's getting hard to keep them all in mind. I propose we keep a list of the main sources (that we know about), such that we can say that if none of these things happen, then we've mostly eliminated AI risk (as an existential risk) at least as far as we can determine. Here's a list that I spent a couple of hours enumerating and writing down. Did I miss anything important?

  1. Insufficient time/resources for AI safety (for example caused by intelligence explosion or AI race)
  2. Insufficient global coordination, leading to the above
  3. Misspecified or incorrectly learned goals/values
  4. Inner optimizers
  5. ML differentially accelerating easy to measure goals
  6. Paul Christiano's "influence-seeking behavior" (a combination of 3 and 4 above?)
  7. AI generally accelerating intellectual progress in a wrong direction (e.g., accelerating unsafe/risky technologies more than knowledge/wisdom about how to safely use those technologies)
  8. Metaethical error
  9. Metaphilosophical error
  10. Other kinds of philosophical errors in AI design (e.g., giving AI a wrong prior or decision theory)
  11. Other design/coding errors (e.g., accidentally putting a minus sign in front of utility function, supposedly corrigible AI not actually being corrigible)
  12. Doing acausal reasoning in a wrong way (e.g., failing to make good acausal trades, being acausally extorted, failing to acausally influence others who can be so influenced)
  13. Human-controlled AIs ending up with wrong values due to insufficient "metaphilosophical paternalism"
  14. Human-controlled AIs causing ethical disasters (e.g., large scale suffering that can't be "balanced out" later) prior to reaching moral/philosophical maturity
  15. Intentional corruption of human values
  16. Unintentional corruption of human values
  17. Mind crime (disvalue unintentionally incurred through morally relevant simulations in AIs' minds)
  18. Premature value lock-in (i.e., freezing one's current conception of what's good into a utility function)
  19. Extortion between AIs leading to vast disvalue
  20. Distributional shifts causing apparently safe/aligned AIs to stop being safe/aligned
  21. Value drift and other kinds of error as AIs self-modify, or AIs failing to solve value alignment for more advanced AIs
  22. Treacherous turn / loss of property rights due to insufficient competitiveness of humans & human-aligned AIs
  23. Gradual loss of influence due to insufficient competitiveness of humans & human-aligned AIs
  24. Utility maximizers / goal-directed AIs having an economic and/or military competitive advantage due to relative ease of cooperation/coordination, defense against value corruption and other forms of manipulation and attack, leading to one or more of the above
  25. In general, the most competitive type of AI being too hard to align or to safely use
  26. Computational resources being too cheap, leading to one or more of the above

(With this post I mean to (among other things) re-emphasize the disjunctive nature of AI risk, but this list isn't fully disjunctive (i.e., some of the items are subcategories or causes of others), and I mostly gave a source of AI risk its own number in the list if it seemed important to make that source more salient. Maybe once we have a list of everything that is important, it would make sense to create a graph out of it.)

Added on 6/13/19:

  1. Failure to learn how to deal with alignment in the many-humans, many-AIs case even if single-human, single-AI alignment is solved (suggested by William Saunders)
  2. Economics of AGI causing concentration of power amongst human overseers
  3. Inability to specify any ‘real-world’ goal for an artificial agent (suggested by Michael Cohen)
  4. AI systems end up controlled by a group of humans representing a small range of human values (ie. an ideological or religious group that imposes values on everyone else) (suggested by William Saunders)

Added on 2/3/2020:

  1. Failing to solve the commitment races problem, i.e. building AI in such a way that some sort of disastrous outcome occurs due to unwise premature commitments (or unwise hesitation in making commitments!). This overlaps significantly with #27, #19, and #12.

Added on 3/11/2020:

  1. Demons in imperfect search (similar, but distinct from, inner optimizers.) See here for illustration.

Added on 10/4/2020:

  1. Persuasion tools or some other form of narrow AI leads to a massive deterioration of collective epistemology, dooming humanity to stumble inexorably into some disastrous end or other.

Added on 8/31/2021:

  1. Vulnerable world type 1: narrow AI enables many people to destroy world, e.g. R&D tools that dramatically lower the cost for building WMD's.
  2. Vulnerable world 2a: We end up with many powerful actors able and incentivized to create civilization-devastating harms.

[Edit on 1/28/2020: This list was created by Wei Dai. Daniel Kokotajlo offered to keep it updated and prettify it over time, and so was added as a coauthor.]

AI RiskThreat Models
Frontpage
Mentioned in
Load More (5/11)
The Main Sources of AI Risk?
New Comment
11 comments, sorted by
top scoring
Click to highlight new comments since:
[-]Daniel Kokotajlo50

Thank you for making this list. I think it is important enough to be worth continually updating and refining; if you don't do it then I will myself someday. Ideally there'd be a whole webpage or something, with the list refined so as to be disjunctive, and each element of the list catchily named, concisely explained, and accompanied by a memorable and plausible example. (As well as lots of links to literature.)

I think the commitment races problem is mostly but not entirely covered by #12 and #19, and at any rate might be worth including since you are OK with overlap.

Also, here's a good anecdote to link to for the "coding errors" section: https://openai.com/blog/fine-tuning-gpt-2/

Reply
[-]Wei Dai40

Thank you for making this list. I think it is important enough to be worth continually updating and refining; if you don’t do it then I will myself someday.

Please do. I seem to get too easily distracted these days for this kind of long term maintenance work. I'll ask the admins to give you edit permission on this post (if possible) and you can also copy the contents into a wiki page or your own post if you want to do that instead.

Reply
[-]Daniel Kokotajlo40

Ha! I wake up this morning to see my own name as author, that wasn't what I had in mind but it sure does work to motivate me to walk the talk! Thanks!

Reply
[-]Daniel Kokotajlo40

Update: It has failed to motivate me. I made one or two edits to the list but haven't done anything like the thorough encyclopedic accounting I originally envisioned. :(

Reply
[-]Oliver Habryka10

Done! Daniel should now be able to edit the post. 

Reply
[-]William Saunders30
  • AI systems end up controlled by a group of humans representing a small range of human values (ie. an ideological or religious group that imposes values on everyone else). While not caused only by AI design, it is possible that design decisions could impact the likelihood of this scenario (ie. at what point are values loaded into the system/how many people's values are loaded into the system), and is relevant for overall strategy.
Reply
[-]William Saunders30
  • Failure to learn how to deal with alignment in the many-humans, many-AIs case even if single-human, single-AI alignment is solved (which I think Andrew Critch has talked about). For example, AIs negotiating on behalf of humans take the stance described in https://arxiv.org/abs/1711.00363 of agreeing to split control of the future according to which human's priors are most accurate (on potentially irrelevant issues) if this isn't what humans actually want.
Reply
[-]Wei Dai40

Failure to learn how to deal with alignment in the many-humans, many-AIs case even if single-human, single-AI alignment is solved (which I think Andrew Critch has talked about).

Good point, I'll add this to the list.

For example, AIs negotiating on behalf of humans take the stance described in https://arxiv.org/abs/1711.00363 of agreeing to split control of the future according to which human’s priors are most accurate (on potentially irrelevant issues) if this isn’t what humans actually want.

Thanks, I hadn't noticed that paper until now. Under "Related Works" it cites Social Choice Theory but doesn't actually mention any recent research from that field. Here is one paper that criticizes the Pareto principle that Critch's paper is based on, in the context of preference aggregation of people with different priors: Spurious Unanimity and the Pareto Principle

Reply
[-]michaelcohen20
3. Misspecified or incorrectly learned goals/values

I think this phrasing misplaces the likely failure modes. An example that comes to mind from this phrasing is that we mean to maximize conscious flourishing, but we accidentally maximize dopamine in large brains.

Of course, this example includes an agent intervening in the provision of its own reward, but since that seems like the paradigmatic example here, maybe the language could better reflect that, or maybe this could be split into two.

The single technical problem that appears biggest to me is that we don't know how to align an agent with any goal. If we had an indestructible magic box that printed a number to a screen corresponding to the true amount of Good in the world, we still don't know how to design an agent that maximizes that number (instead of taking over the world, and tampering with the cameras that are aimed at the screen/the optical character recognition program used to decipher the image). This problems seems to me like the single most fundamental source of AI risk. Is 3 meant to include this?

Reply
[-]Wei Dai10

I'm not sure if I meant to include this when I wrote 3, but it does seem like a good idea to break it out into its own item. How would you suggest phrasing it? "Wireheading" or something more general or more descriptive?

Reply
[-]michaelcohen10

Maybe something along the lines of "Inability to specify any 'real-world' goal for an artificial agent"?

Reply
Moderation Log
11Comments