Six Dimensions of Operational Adequacy in AGI Projects
7Raemon
5Ofer
5Rob Bensinger
7Yitz
16Rob Bensinger
1Chris Fong
24Rob Bensinger
3johnswentworth
7Rob Bensinger
1lc
3Rob Bensinger
1Rob Bensinger
0Mitchell_Porter
11Rob Bensinger
1Rob Bensinger
9Vanessa Kosoy
5Eliezer Yudkowsky
4Vanessa Kosoy
1Rob Bensinger
4Vanessa Kosoy
1Rob Bensinger
3johnswentworth
4Rob Bensinger
4johnswentworth
1Rob Bensinger
2[comment deleted]
0Mitchell_Porter
1Rob Bensinger
0Mitchell_Porter
4Rob Bensinger
0Mitchell_Porter
14Rob Bensinger
2Raemon
1Raemon
New Comment
Rendering 32/33 comments, sorted by (show more) Click to highlight new comments since:
Curated. My sense is there is no existing AI company with adequate infrastructure for safely deploying AGI, and this is a pretty big deal. I like this writeup for laying out a bunch of considerations.
A few times in this article, Eliezer notes "it'd be great if we could get X, but, the process of trying to get X would cause some bad consequences." I'd like to see further exploration/models of, "given the state of the current world, which approaches are actually tractable.
Even with adequate closure and excellent opsec, there can still be risks related to researchers on the team quitting and then joining a competing effort or starting their own AGI company (and leveraging what they've learned).
With Eliezer's assent, I hit the Publish button for this post, and included an editor's note that I co-wrote with Duncan Sabien.
May I ask why you guys decided to publish this now in particular? Totally fine if you can’t answer that question, of course.
It's been high on some MIRI staff's "list of things we want to release" over the years, but we repeatedly failed to make a revised/rewritten version of the draft we were happy with. So I proposed that we release a relatively unedited version of Eliezer's original draft, and Eliezer said he was okay with that (provided we sprinkle the "Reminder: This is a 2017 document" notes throughout).
We're generally making a push to share a lot of our models (expect more posts soon-ish), because we're less confident about what the best object-level path is to ensuring the long-term future is awesome, so (as I described in April) we've "updated a lot toward existential wins being likelier if the larger community moves toward having much more candid and honest conversations, and generally produces more people who are thinking exceptionally clearly about the problem".
I think this was always plausible to some degree, but it's grown in probability; and model-sharing is competing against fewer high-value uses of Eliezer and Nate's time now that they aren't focusing their own current efforts on alignment research.
now that they aren't focusing their own current efforts on alignment research.
Should I update and be encouraged or discouraged by this?
Discouraged. Eliezer and Nate feel that their past alignment research efforts failed, and they don't currently know of a new research direction that feels promising enough that they want to focus their own time on advancing it, or make it MIRI's organizational focus.
I do think 'trying to directly solve the alignment problem' is the most useful thing the world can be doing right now, even if it's not Eliezer or Nate's comparative advantage right now. A good way to end up with a research direction EY or Nate are excited by, IMO, is for hundreds of people to try hundreds of different angles of attack and see if any bear fruit. Then a big chunk of the field can pivot to whichever niche approach bore the most fruit.
From MIRI's perspective, the hard part is that:
- (a) we don't know in advance which directions will bear fruit, so we need a bunch of people to go make unlikely bets so we can find out;
- (b) there currently aren't that many people trying to solve the alignment problem at all; and
- (c) nearly all of the people trying to solve the problem full-time are adopting unrealistic optimistic assumptions about things like 'will alignment generalize as well as capabilities?' and 'will the first pivotal AI systems be safe-by-default?', in such a way that their research can't be useful if we're in the mainline-probability world.
What I'd like to see instead is more alignment research, and especially research of the form "this particular direction seems unlikely to succeed, but if it succeeds then it will in fact help a lot in mainline reality", as opposed to directions that (say) seem a bit likelier to succeed but won't actually help in the mainline world.
(In principle you want nonzero effort going into both approaches, but right now the field is almost entirely in the second camp, from MIRI's perspective. And making a habit of assuming your way out of mainline reality is risky business, and outright dooms your research once you start freely making multiple such assumptions.)
Fiscal limits notwithstanding, doesn't this suggest MIRI should try hiring a lot more maybe B-tier researchers?
Quoting a thing I said in March:
The two big things we feel bottlenecked on are:
- (1) people who can generate promising new alignment ideas. (By far the top priority, but seems empirically rare.)
- (2) competent executives who are unusually good at understanding the kinds of things MIRI is trying to do, and who can run their own large alignment projects mostly-independently.
For 2, I think the best way to get hired by MIRI is to prove your abilities via the Visible Thoughts Project. The post there says a bit more about the kind of skills we're looking for:
Eliezer has a handful of ideas that seem to me worth pursuing, but for all of them to be pursued, we need people who can not only lead those projects themselves, but who can understand the hope-containing heart of the idea with relatively little Eliezer-interaction, and develop a vision around it that retains the shred of hope and doesn’t require constant interaction and course-correction on our part. (This is, as far as I can tell, a version of the Hard Problem of finding good founders, but with an additional constraint of filtering for people who have affinity for a particular project, rather than people who have affinity for some project of their own devising.)
For 1, I suggest initially posting your research ideas to LessWrong, in line with John Wentworth's advice. New ideas and approaches are desperately needed, and we would consider it crazy to not fund anyone whose ideas or ways-of-thinking-about-the-problem we think have a shred of hope in them. We may fund them via working at MIRI, or via putting them in touch with external funders; the important thing is just that the research happens.
If you want to work on alignment but you don't fall under category 1 or 2, you might consider applying to work at Redwood Research (https://www.redwoodresearch.org/jobs), which is a group doing alignment research we like. They're much more hungry for engineers right now than we are.
If your "hiring a lot more maybe B-tier researchers" suggestion implies something more than the above 'we intend to ensure that everyone whose ideas have a shred of hope gets funded' policy, I'd be interested to hear what additional thing you think should happen, and why.
A consideration pushing against 'just try to grow the field indiscriminately' is the argument Alex Flint gives re the flywheel model.
What I'd like to see instead is more alignment research, and especially research of the form "this particular direction seems unlikely to succeed, but if it succeeds then it will in fact help a lot in mainline reality"
(Obviously even better would be if it seems likely to succeed and helps on the mainline. But 'longshot that will help if it succeeds' is second-best.)
Eliezer and Nate feel that their past alignment research efforts failed
I find this a little surprising. If someone had asked me what MIRI's strategy is, I would have said that the core of it was still something like CEV, with topics like logical induction and new decision theory paradigms as technical framework issues. I mean, part of the MIRI paradigm has always been that AGI alignment is grounded in how the human brain works, right? The mechanics of decision-making in human brains, are the starting point in constructing the mechanics of decision-making in an AGI that humans would call 'aligned'. And I would have thought that identifying how to do this, was still just research in progress in many directions, rather than something that had hit a dead end.
Quoting from the "strategic background" summary we shared in 2017:
[...] In very broad terms, however, our approach to global risk mitigation is to think in terms of desired outcomes, and to ask: “What is the likeliest way that the outcome in question might occur?” We then repeat this process until we backchain to interventions that actors can take today. [...]
1. Long-run good outcomes. Ultimately, we want humanity to figure out the best possible long-run future and enact that kind of future, factoring in good outcomes for all sentient beings. However, there is currently very little we can say with confidence about what desirable long-term outcomes look like, or how best to achieve them; and if someone rushes to lock in a particular conception of “the best possible long-run future,” they’re likely to make catastrophic mistakes both in how they envision that goal and in how they implement it.
In order to avoid making critical decisions in haste and locking in flawed conclusions, humanity needs:
2. A stable period during which relevant actors can accumulate whatever capabilities and knowledge are required to reach robustly good conclusions about long-run outcomes. This might involve decisionmakers developing better judgment, insight, and reasoning skills in the future, solving the full alignment problem for fully autonomous AGI systems, and so on.
Given the difficulty of the task, we expect a successful stable period to require:
3. A preceding end to the acute risk period. If AGI carries a significant chance of causing an existential catastrophe over the next few decades, this forces a response under time pressure; but if actors attempt to make irreversible decisions about the long-term future under strong time pressure, we expect the result to be catastrophically bad. Conditioning on good outcomes, we therefore expect a two-step process where addressing acute existential risks takes temporal priority.
To end the acute risk period, we expect it to be necessary for actors to make use of:
4. A risk-mitigating technology. On our current view of the technological landscape, there are a number of plausible future technologies that could be leveraged to end the acute risk period.
We believe that the likeliest way to achieve a technology in this category sufficiently soon is through:
5. AGI-empowered technological development carried out by task-directed AGI systems. Depending on early AGI systems’ level of capital-intensiveness, on whether AGI is a late-paradigm or early-paradigm invention, and on a number of other factors, AGI might be developed by anything from a small Silicon Valley startup to a large-scale multinational collaboration. Regardless, we expect AGI to be developed before any other (meta)technology that can be employed to end the acute risk period, and if early AGI systems can be used safely at all, then we expect it to be possible for an AI-empowered project to safely automate a reasonably small set of concrete science and engineering tasks that are sufficient for ending the risk period. This requires:
6. Construction of minimal aligned AGI. We specify “minimal” because we consider success much more likely if developers attempt to build systems with the bare minimum of capabilities for ending the acute risk period. We expect AGI alignment to be highly difficult, and we expect additional capabilities to add substantially to this difficulty.
Added: “Minimal aligned AGI” means “aligned AGI that has the minimal necessary capabilities”; be sure not to misread it as “minimally aligned AGI”. Rob Bensinger adds: “The MIRI view isn’t ‘rather than making alignment your top priority and working really hard to over-engineer your system for safety, try to build a system with the bare minimum of capabilities’. It’s: ‘in addition to making alignment your top priority and working really hard to over-engineer your system for safety, also build the system to have the bare minimum of capabilities’.”
If an aligned system of this kind were developed, we would expect two factors to be responsible:
7a. A technological edge in AGI by an operationally adequate project. By “operationally adequate” we mean a project with strong opsec, research closure, trustworthy command, a commitment to the common good, security mindset, requisite resource levels, and heavy prioritization of alignment work. A project like this needs to have a large enough lead to be able to afford to spend a substantial amount of time on safety measures, as discussed at FLI’s Asilomar conference.
7b. A strong white-boxed system understanding on the part of the operationally adequate project during late AGI development. By this we mean that developers go into building AGI systems with a good understanding of how their systems decompose and solve particular cognitive problems, of the kinds of problems different parts of the system are working on, and of how all of the parts of the system interact.
On our current understanding of the alignment problem, developers need to be able to give a reasonable account of how all of the AGI-grade computation in their system is being allocated, similar to how secure software systems are built to allow security professionals to give a simple accounting of why the system has no unforeseen vulnerabilities. See “Security Mindset and Ordinary Paranoia” for more details.
Developers must be able to explicitly state and check all of the basic assumptions required for their account of the system’s alignment and effectiveness to hold. Additionally, they need to design and modify AGI systems only in ways that preserve understandability — that is, only allow system modifications that preserve developers’ ability to generate full accounts of what cognitive problems any given slice of the system is solving, and why the interaction of all of the system’s parts is both safe and effective.
Our view is that this kind of system understandability will in turn require:
8. Steering toward alignment-conducive AGI approaches. Leading AGI researchers and developers need to deliberately direct research efforts toward ensuring that the earliest AGI designs are relatively easy to understand and align.
We expect this to be a critical step, as we do not expect most approaches to AGI to be alignable after the fact without long, multi-year delays.
We then added:
We plan to say more in the future about the criteria for operationally adequate projects in 7a. We do not believe that any project meeting all of these conditions currently exists, though we see various ways that projects could reach this threshold.
The above breakdown only discusses what we view as the “mainline” success scenario. If we condition on good long-run outcomes, the most plausible explanation we can come up with cites an operationally adequate AI-empowered project ending the acute risk period, and appeals to the fact that those future AGI developers maintained a strong understanding of their system’s problem-solving work over the course of development, made use of advance knowledge about which AGI approaches conduce to that kind of understanding, and filtered on those approaches.
... and we said that MIRI's strategy is to do research aimed at making "8. Steering toward alignment-conducive AGI approaches" easier:
For that reason, MIRI does research to intervene on 8 from various angles, such as by examining holes and anomalies in the field’s current understanding of real-world reasoning and decision-making. We hope to thereby reduce our own confusion about alignment-conducive AGI approaches and ultimately help make it feasible for developers to construct adequate “safety-stories” in an alignment setting. As we improve our understanding of the alignment problem, our aim is to share new insights and techniques with leading or up-and-coming developer groups, who we’re generally on good terms with. [...]
Replying to your points with that in mind, Mitchell:
I find this a little surprising. If someone had asked me what MIRI's strategy is, I would have said that the core of it was still something like CEV, with topics like logical induction and new decision theory paradigms as technical framework issues.
Assuming the long-run future goes well, I expect humanity to eventually build an AGI system that does something vaguely CEV-like. (This will plausibly be the main goal of step 2 in the backchained list, and therefore the main way we get to step 1.)
But I wouldn't say that MIRI's research is at all about CEV. Before humanity gets to steps 1-2 ('use CEV or something to make the long-term future awesome'), it needs to get past steps 3-6 ('use limited task AGI to ensure that humanity doesn't kill itself with AGI so we can proceed to take our time with far harder problems like "what even is CEV" and "how even in principle would one get an AI system to robustly do anything remotely like that, without some subtle or not-so-subtle disaster resulting"').
We'll have plenty of time to worry about steps 1-2 if we can figure out 3-6, so almost all of the alignment field's attention should be on how to achieve 3-6 (or plausible prerequisites for 3-6, like 7-8).
Re logical induction and decision theory: the main goal of this kind of research has been "de-confusion" in the sense described in The Rocket Alignment Problem and our 2018 strategy update.
I mean, part of the MIRI paradigm has always been that AGI alignment is grounded in how the human brain works, right? The mechanics of decision-making in human brains, are the starting point in constructing the mechanics of decision-making in an AGI that humans would call 'aligned'.
Sounds false to me, and I'm not sure how this relates to CEV, logical induction, logical decision theory, etc.
It's true that humans reason under logical uncertainty (and logical induction may be getting at a core thing about why that works), and it's true that some of our instincts and reasoning are very likely to look FDT-ish insofar as they evolved in an environment where agents model each other and use similar reasoning processes.
But if humans somehow couldn't apply their normal probablistic reasoning to math (and couldn't do FDT-ish reasoning), and yet we could still somehow do math/AI research at all, then logical uncertainty and FDT would still be just as important, because they'd still be progress toward understanding how sufficiently smart AI systems reason about the world.
Think less "LI and FDT are how humans work, and we're trying to make a machine that works like a human brain", more "there's such a thing as general-purpose problem-solving, some undiscovered aspects of this thing are probably simple, and something like LI and FDT may be (idealized versions of) how certain parts of this general-purpose problem-solving works, or may be steps toward finding such parts".
And I would have thought that identifying how to do this, was still just research in progress in many directions, rather than something that had hit a dead end.
Last I checked, there are still MIRI researchers working on the major research programs we've done in the past (or working on research programs along those lines). But the organization as a whole isn't trying to concentrate effort on any one of those lines of research, and Nate and Eliezer aren't focusing their own efforts on any of the ongoing MIRI research projects, because Nate and Eliezer think AGI timelines are too short relative to those projects' rates of progress to date.
(Or, 'a critical mass of MIRI's research leadership (which includes more people than Nate and Eliezer but assigns a lot of weight to Nate's and Eliezer's views) thinks AGI timelines are too short relative to those projects' rates of progress to date'.)
From Nate/EY's perspective, as I understand it, the problem isn't 'none of the stuff MIRI's tried has borne fruit, or is currently bearing fruit'; it's 'the fruit is being borne too slowly for us to feel like MIRI's current or past research efforts are on the critical path to humanity surviving and flourishing'.
They don't know what is on the critical path, but they feel sufficiently pessimistic about the current things they've tried (relative to their timelines) that they'd rather work on things like aumanning right now and keep an eye out for better future alignment research ideas, rather than keep their focus on the 'things we've already tried putting lots of effort into' bucket.
(As with the vast majority of my comments, EY and Nate haven't reviewed this, so I may be getting stuff about their views wrong.)
Before humanity gets to steps 1-2 ('use CEV or something to make the long-term future awesome'), it needs to get past steps 3-6 ('use limited task AGI to ensure that humanity doesn't kill itself with AGI so we can proceed to take our time with far harder problems like "what even is CEV" and "how even in principle would one get an AI system to robustly do anything remotely like that, without some subtle or not-so-subtle disaster resulting"').
I want to register my skepticism about this claim. Whereas it might naively seem that "put a strawberry on a plate" is easier to align than "extrapolated volition", on a closer look there are reasons why it might be the other way around. Specifically, the notion of "utility function of given agent" is a natural concept that we should expect to have a relatively succinct mathematical description. This intuition is supported by the AIT definition of intelligence. On the other hand, "put a strawberry on a plate without undesirable side effects" is not a natural concept, since a lot of complexity is packed into the "undesirable side effects". Therefore, while I see some lines of attack on both task AGI and extrapolated volition, the latter might well turn out easier.
And if humans had a utility function and we knew what that utility function was, we would not need CEV. Unfortunately extracting human preferences over out-of-distribution options and outcomes at dangerously high intelligence, using data gathered at safe levels of intelligence and a correspondingly narrower range of outcomes and options, when there exists no sensory ground truth about what humans want because human raters can be fooled or disassembled, seems pretty complicated. There is ultimately a rescuable truth about what we want, and CEV is my lengthy informal attempt at stating what that even is; but I would assess it as much, much, much more difficult than 'corrigibility' to train into a dangerously intelligent system using only training and data from safe levels of intelligence. (As is the central lethally difficult challenge of AGI alignment.)
If we were paperclip maximizers and knew what paperclips were, then yes, it would be easier to just build an offshoot paperclip maximizer.
I agree that it's a tricky problem, but I think it's probably tractable. The way PreDCA tries to deal with these difficulties is:
- The AI can tell that, even before the AI was turned on, the physical universe was running certain programs.
- Some of those programs are "agentic" programs.
- Agentic programs have approximately well-defined utility functions.
- Disassembling the humans doesn't change anything, since it doesn't affect the programs that were already running[1] before the AI was turned on.
- Since we're looking at agent-programs rather than specific agent-actions, there is much more ground for inference about novel situations.
Obviously, the concepts I'm using here (e.g. which programs are "running" or which programs are "agentic") are non-trivial to define, but infra-Bayesian physicalism does allow us the define them (not without some caveats, but hopefully at least to a 1st approximation).
More precisely, I am looking at agents which could prevent the AI from becoming turned on, this is what I call "precursors". ↩︎
Yeah, I'm very interested in hearing counter-arguments to claims like this. I'll say that although I think task AGI is easier, it's not necessarily strictly easier, for the reason you mentioned.
Maybe a cruxier way of putting my claim is: Maybe corrigibility / task AGI / etc. is harder than CEV, but it just doesn't seem realistic to me to try to achieve full, up-and-running CEV with the very first AGI systems you build, within a few months or a few years of humanity figuring out how to build AGI at all.
And I do think you need to get CEV up and running within a few months or a few years, if you want to both (1) avoid someone else destroying the world first, and (2) not use a "strawberry-aligned" AGI to prevent 1 from happening.
All of the options are to some extent a gamble, but corrigibility, task AGI, limited impact, etc. strike me as gambles that could actually realistically work out well for humanity even under extreme time pressure to deploy a system within a year or two of 'we figure out how to build AGI'. I don't think CEV is possible under that constraint. (And rushing CEV and getting it only 95% correct poses far larger s-risks than rushing low-impact non-operator-modeling strawberry AGI and getting it only 95% correct.)
Maybe corrigibility / task AGI / etc. is harder than CEV, but it just doesn't seem realistic to me to try to achieve full, up-and-running CEV with the very first AGI systems you build, within a few months or a few years of humanity figuring out how to build AGI at all.
The way I imagine the win scenario is, we're going to make a lot of progress in understanding alignment before we know how to build AGI. And, we're going to do it by prioritizing understanding alignment modulo capability (the two are not really possible to cleanly separate, but it might be possible to separate them enough for this purpose). For example, we can assume the existence of algorithms with certain properties, s.t. these properties arguably imply the algorithms can be used as building-blocks for AGI, and then ask: given such algorithms, how would we build aligned AGI? Or, we can come up with some toy setting where we already know how to build "AGI" in some sense, and ask, how to make it aligned in that setting? And then, once we know how to build AGI in the real world, it would hopefully not be too difficult to translate the alignment method.
One caveat in all this is, if AGI is going to use deep learning, we might not know how to apply the lesson from the "oracle"/toy setting, because we don't understand what deep learning is actually doing, and because of that, we wouldn't be sure where to "slot" it in the correspondence/analogy s.t. the alignment method remains sound. But, mainstream researchers have been making progress on understanding what deep learning is actually doing, and IMO it's plausible we will have a good mathematical handle on it before AGI.
And rushing CEV and getting it only 95% correct poses far larger s-risks than rushing low-impact non-operator-modeling strawberry AGI and getting it only 95% correct.
I'm not sure whether you mean "95% correct CEV has a lot of S-risk" or "95% correct CEV has a little S-risk, and even a tiny amount of S-risk is terrifying"? I think I agree with the latter but not with the former. (How specifically does 95% CEV produce S-risk? I can imagine something like "AI realizes we want non-zero amount of pain/suffering to exist, somehow miscalibrates the amount and creates a lot of pain/suffering" or "AI realizes we don't want to die, and focuses on this goal on the expense of everything else, preserving us forever in a state of complete sensory deprivation". But these scenarios don't seem very likely?)
I'm not sure whether you mean "95% correct CEV has a lot of S-risk" or "95% correct CEV has a little S-risk, and even a tiny amount of S-risk is terrifying"?
The latter, as I was imagining "95%".
Insofar as humans care about their AI being corrigible, we should expect some degree of corrigibility even from a CEV-maximizer. That, in turn, suggests at least some basin-of-attraction for values (at least along some dimensions), in the same way that corrigibility yields a basin-of-attraction.
(Though obviously that's not an argument we'd want to make load-bearing without both theoretical and empirical evidence about how big the basin-of-attraction is along which dimensions.)
Conversely, it doesn't seem realistic to define limited impact or corrigibility or whatever without relying on an awful lot of values information - like e.g. what sort of changes-to-the-world we do/don't care about, what thing-in-the-environment the system is supposed to be corrigible with, etc.
Values seem like a necessary-and-sufficient component. Corrigibility/task architecture/etc doesn't.
And rushing CEV and getting it only 95% correct poses far larger s-risks than rushing low-impact non-operator-modeling strawberry AGI and getting it only 95% correct.
Small but important point here: an estimate of CEV which is within 5% error everywhere does reasonably well; that gets us within 5% of our best possible outcome. The problem is when our estimate is waaayyy off in 5% of scenarios, especially if it's off in the overestimate direction; then we're in trouble.
Conversely, it doesn't seem realistic to define limited impact or corrigibility or whatever without relying on an awful lot of values information - like e.g. what sort of changes-to-the-world we do/don't care about, what thing-in-the-environment the system is supposed to be corrigible with, etc.
I suspect you could do this in a less value-loaded way if you're somehow intervening on 'what the AGI wants to pay attention to', as opposed to just intervening on 'what sorts of directions it wants to steer the world in'.
'Only spend your cognition thinking about individual physical structures smaller than 10 micrometers', 'only spend your cognition thinking about the physical state of this particular five-cubic-foot volume of space', etc. could eliminate most of the risk of 'high-impact' actions without forcing us to define human conceptions of 'impact', and without forcing the AI to do a bunch of human-modeling. But I don't know what research path would produce the ability to do things like that.
(There's still of course something that we're trying to get the AGI to do, like make a nanofactory or make a scanning machine for WBE or make improved computing hardware. That part strikes me as intuitively more value-loaded than 'only think about this particular volume of space'.
The difficulty with 'only think about this particular volume of space' is that it requires the ability to intervene on thoughts rather than outputs.)
'Only spend your cognition thinking about individual physical structures smaller than 10 micrometers', 'only spend your cognition thinking about the physical state of this particular five-cubic-foot volume of space', etc. could eliminate most of the risk of 'high-impact' actions without forcing us to define human conceptions of 'impact', and without forcing the AI to do a bunch of human-modeling.
I do not think that would do what you seem to think it would do. If something optimizes one little chunk of the world really hard, ignoring everything else, that doesn't mean the rest of the world is unchanged; by default there are lots of side effects. E.g. if something is building nanotech in a 1m cube, ignoring everything outside the cube, at the very least I'd expect that dump nuke levels of waste heat into its immediate surroundings.
I agree that this would be scary if the system is, for example, as smart as physically possible. What I'm imagining is:
- (1) if you find a way to ensure that the system is only weakly superhuman (e.g., it performs vast amounts of low-level-Google-engineer-quality reasoning, only rare short controlled bursts of von-Neumann-quality reasoning, and nothing dramatically above the von-Neumann level), and
- (2) if you get the system to only care about thinking about this cube of space, and
- (3) if you also somehow get the system to want to build the particular machine you care about,
then you can plausibly save the world, and (importantly) you're not likely to destroy the world if you fail, assuming you really are correctly confident in 1, 2, and 3.
I think you can also get more safety margin if the cube is in Antarctica (or on the Moon?), if you've tried to seal it off from the environment to some degree, and if you actively monitor for things like toxic waste products, etc.
Notably, the "only care about thinking about this cube of space" part is important for a lot of the other safety features to work, like:
- It's a lot harder to get guarantees about the system's intelligence if it's optimizing the larger world (since it might then improve itself, or build a smart successor in its environment -- good luck closing off all possible loopholes for what kinds of physical systems an AGI might build that count as "smart successors", while still leaving it able to build nanotech!).
- Likewise, it's a lot harder to get guarantees that the system stably is optimizing what you want it to optimize, or stably has any specific internal property, if it's willing and able to modify itself.
- Part of why we can hope to notice, anticipate, and guard against bad side-effects like "waste products" is that the waste products aren't optimized to have any particular effect on the external environment, and aren't optimized to evade our efforts to notice, anticipate, or respond to the danger. For that reason, "An AGI that only terminally cares about the state of a certain cube of space, but does spend time thinking about the larger world", is vastly scarier than an AGI that just-doesn't-think in those directions.
- If the system does start going off the rails, we're a lot more likely to be able to shut it down if it isn't thinking about us or about itself.
This makes me think that the "only care about thinking about certain things" part may be relatively important in order for a lot of other safety requirements to be tractable. It feels more "(realistically) necessary" than "sufficient" to me; but I do personally have a hunch (which hopefully we wouldn't have to actually rely on as a safety assumption!) that the ability to do things in this reference class would get us, like, 80+% of the way to saving the world? (Dunno whether Eliezer or anyone else at MIRI would agree.)
Thank you for the long reply. The 2017 document postulates an "acute risk period" in which people don't know how to align, and then a "stable period" once alignment theory is mature.
So if I'm getting the gist of things, rather than focus outright on the creation of a human-friendly superhuman AI, MIRI decided to focus on developing a more general theory and practice of alignment; and then once alignment theory is sufficiently mature and correct, one can focus on applying that theory to the specific crucial case, of aligning superhuman AI with extrapolated human volition.
But what's happened is that we're racing towards superhuman AI while the general theory of alignment is still crude, and this is a failure for the strategy of prioritizing general theory of alignment over the specific task of CEV.
Is that vaguely what happened?
The 2017 document postulates an "acute risk period" in which people don't know how to align, and then a "stable period" once alignment theory is mature.
"Align" is a vague term. Let's distinguish "strawberry alignment" (where we can safely and reliably use an AGI to execute a task like "Place, onto this particular plate here, two strawberries identical down to the cellular but not molecular level.") from "CEV alignment" (where we can safely and reliably use an AGI to carry out a CEV-like procedure.)
Strawberry alignment seems vastly easier than CEV alignment to me, and I think it's a similar task (in both difficulty and kind) to what we'll need AGI to do in order to prevent humanity from killing itself with other AGIs.
The "acute risk period" is the period where we're at risk of someone immediately destroying the world once they figure out how to build AGI (or once hardware scales to the required level, or whatever).
Figuring out how to do strawberry alignment isn't sufficient for ending the acute risk period, since humanity then has to actually apply this knowledge and build and deploy an aligned AGI to execute some pivotal act. But I do think that figuring out strawberry alignment is the main obstacle; if we knew how to do that, I think humanity would have double-digit odds of surviving and flourishing.
The "stable period" is the period between "humanity successfully makes it the case that no one can destroy the world with AGI" and "humanity figures out how to ensure the long-term future is awesome".
This stable period is very similar to the idea of a "long reflection" posited by Toby Ord and Will MacAskill, though the lengths of time they cite sound far too long to me, at least if we're measuring in sidereal time. (With fast-running human whole-brain emulations, I think you could complete the entire "long reflection" in just a few sidereal years, without cutting any corners or taking any serious risks.)
So if I'm getting the gist of things, rather than focus outright on the creation of a human-friendly superhuman AI
"Human-friendly" and "superhuman" are both vague -- strawberry-aligned task AGI is less robustly friendly, and less broadly capable, than CEV AGI. But strawberry-aligned AGI is still superhuman in at least some respects -- heck, a pocket calculator is too -- and it's still friendly enough to do some impressive things without killing us.
Alignment is a matter of degree, and more ambitious tasks can be much harder to align.
MIRI decided to focus on developing a more general theory and practice of alignment;
Strawberry alignment is more "general" in the sense that we're not trying to impart as many human-specific values into the AGI (though we still need to impart some).
But it's less "general" in the sense that strawberry-grade alignment is likely to be much more brittle than CEV-grade alignment, and strawberry-grade alignment is much more dependent on us carefully picking exactly the right tasks and procedures to make the alignment work.
But what's happened is that we're racing towards superhuman AI while the general theory of alignment is still crude, and this is a failure for the strategy of prioritizing general theory of alignment over the specific task of CEV.
No. If we'd focused on CEV-grade alignment over strawberry-grade alignment, we'd be in even worse shape if anything.
The problem is that timelines look short, so it's looking more difficult to figure out strawberry alignment in time to prevent human extinction. We should nonetheless make strawberry alignment humanity's top priority, and put an enormous effort into it, because there isn't a higher-probability path to good outcomes. (AFAICT, anyway. Having at least some people try to prove me wrong here obviously seems worthwhile too.)
CEV alignment is even harder than strawberry alignment (by a large margin), so short timelines are much more of a problem for the 'rush straight to CEV alignment' plan than for the 'do strawberry alignment first, then CEV afterwards' plan.
The "stable period" is supposed to be a period in which AGI already exists, but nothing like CEV has yet been implemented, and yet "no one can destroy the world with AGI". How would that work? How do you prevent everyone in the whole wide world from developing unsafe AGI during the stable period?
I think there are multiple viable options, like the toy example EY uses:
I think that after AGI becomes possible at all and then possible to scale to dangerously superhuman levels, there will be, in the best-case scenario where a lot of other social difficulties got resolved, a 3-month to 2-year period where only a very few actors have AGI, meaning that it was socially possible for those few actors to decide to not just scale it to where it automatically destroys the world.
During this step, if humanity is to survive, somebody has to perform some feat that causes the world to not be destroyed in 3 months or 2 years when too many actors have access to AGI code that will destroy the world if its intelligence dial is turned up. This requires that the first actor or actors to build AGI, be able to do something with that AGI which prevents the world from being destroyed; if it didn't require superintelligence, we could go do that thing right now, but no such human-doable act apparently exists so far as I can tell.
So we want the least dangerous, most easily aligned thing-to-do-with-an-AGI, but it does have to be a pretty powerful act to prevent the automatic destruction of Earth after 3 months or 2 years. It has to "flip the gameboard" rather than letting the suicidal game play out. We need to align the AGI that performs this pivotal act, to perform that pivotal act without killing everybody.
Parenthetically, no act powerful enough and gameboard-flipping enough to qualify is inside the Overton Window of politics, or possibly even of effective altruism, which presents a separate social problem. I usually dodge around this problem by picking an exemplar act which is powerful enough to actually flip the gameboard, but not the most alignable act because it would require way too many aligned details: Build self-replicating open-air nanosystems and use them (only) to melt all GPUs.
Since any such nanosystems would have to operate in the full open world containing lots of complicated details, this would require tons and tons of alignment work, is not the pivotal act easiest to align, and we should do some other thing instead. But the other thing I have in mind is also outside the Overton Window, just like this is. So I use "melt all GPUs" to talk about the requisite power level and the Overton Window problem level, both of which seem around the right levels to me, but the actual thing I have in mind is more alignable; and this way, I can reply to anyone who says "How dare you?!" by saying "Don't worry, I don't actually plan on doing that."
It's obviously a super core question; there's no point aligning your AGI if someone else just builds unaligned AGI a few months later and kills everyone. The "alignment problem" humanity has as its urgent task is exactly the problem of aligning cognitive work that can be leveraged to prevent the proliferation of tech that destroys the world. Once you solve that, humanity can afford to take as much time as it needs to solve everything else.
The "alignment problem" humanity has as its urgent task is exactly the problem of aligning cognitive work that can be leveraged to prevent the proliferation of tech that destroys the world. Once you solve that, humanity can afford to take as much time as it needs to solve everything else.
OK, I disagree very much with that strategy. You're basically saying, your aim is not to design ethical/friendly/aligned AI, you're saying your aim is to design AI that can take over the world without killing anyone. Then once that is accomplished, you'll settle down to figure out how that unlimited power would best be used.
To put it another way: Your optimistic scenario is one in which the organization that first achieves AGI uses it to take over the world, install a benevolent interim regime that monopolizes access to AGI without itself making a deadly mistake, and which then eventually figures out how to implement CEV (for example); and then it's finally safe to have autonomous AGI.
I have a different optimistic scenario: We definitively figure out the theory of how to implement CEV before AGI even arises, and then spread that knowledge widely, so that whoever it is in the world that first achieves AGI, they will already know what they should do with it.
Both these scenarios are utopian in different ways. The first one says that flawed humans can directly wield superintelligence for a protracted period without screwing things up. The second one says that flawed humans can fully figure out how to safely wield superintelligence before it even arrives.
Meanwhile, in reality, we've already proceeded an unknown distance up the curve towards superintelligence, but none of the organizations leading the way has much of a plan for what happens, if their creations escape their control.
In this situation, I say that people whose aim is to create ethical/friendly/aligned superintelligence, should focus on solving that problem. Leave the techno-military strategizing to the national security elites of the world. It's not a topic that you can avoid completely, but in the end it's not your job to figure out how mere humans can safely and humanely wield superhuman power. It's your job to design an autonomous superhuman power that is intrinsically safe and humane. To that end we have CEV, we have June Ku's work, and more. We should be focusing there, while remaining engaged with the developments in mainstream AI, like language models. That's my manifesto.
You're basically saying, your aim is not to design ethical/friendly/aligned AI [...]
My goal is an awesome, eudaimonistic long-run future. To get there, I strongly predict that you need to build AGI that is fully aligned with human values. To get there, I strongly predict that you need to have decades of experience actually working with AGI, since early generations of systems will inevitably have bugs and limitations and it would be catastrophic to lock in the wrong future because we did a rush job.
(I'd also expect us to need the equivalent of subjective centuries of further progress on understanding stuff like "how human brains encode morality", "how moral reasoning works", etc.)
If it's true that you need decades of working experience with AGI (and solutions to moral philosophy, psychology, etc.) to pull off CEV, then something clearly needs to happen to prevent humanity from destroying itself in those intervening decades.
I don't like the characterization "your aim is not to design ethical/friendly/aligned AI", because it's picking an arbitrary cut-off for which parts of the plan count as my "aim", and because it makes it sound like I'm trying to build unethical, unfriendly, unaligned AI instead. Rather, I think alignment is hard and we need a lot of time (including a lot of time with functioning AGIs) to have a hope of solving the maximal version of the problem. Which inherently requires humanity to do something about that dangerous "we can build AGI but not CEV-align it" time window.
I don't think the best solution to that problem is for the field to throw up their hands and say "we're scientists, it's not our job to think about practicalities like that" and hope someone else takes care of it. We're human beings, not science-bots; we should use our human intelligence to think about which course of action is likeliest to produce good outcomes, and do that.
[...] I have a different optimistic scenario: We definitively figure out the theory of how to implement CEV before AGI even arises, and then spread that knowledge widely, so that whoever it is in the world that first achieves AGI, they will already know what they should do with it. [...]
How long are your AGI timelines? I could imagine endorsing a plan like that if I were confident AGI is 200+ years away; but in fact I think it's very unlikely to even be 100 years away, and my probability is mostly on scenarios like "it's 8 years away" or "it's 25 years away".
I do agree that we're likelier to see better outcomes if alignment knowledge is widespread, rather than being concentrated at a few big orgs. (All else equal, anyway. E.g., you might not want to do this if it somehow shortens timelines a bunch.)
But the kind of alignment knowledge I think matters here is primarily strawberry-grade alignment. It's good if people widely know about things like CEV, but I wouldn't advise a researcher to spend their 2022 working on advancing abstract CEV theory instead of advancing strawberry-grade alignment, if they're equally interested in both problems and capable of working on either.
[...] To put it another way: Your optimistic scenario is one in which the organization that first achieves AGI uses it to take over the world, install a benevolent interim regime that monopolizes access to AGI without itself making a deadly mistake, and which then eventually figures out how to implement CEV (for example); and then it's finally safe to have autonomous AGI. [...]
Talking about "taking over the world" strikes me as inviting a worst-argument-in-the-world style of reasoning. All the past examples of "taking over the world" weren't cases where there's some action A such that:
- if no one does A, then all humans die and the future's entire value is lost.
- by comparison, it doesn't matter much to anyone who does A; everyone stands to personally gain or lose a lot based on whether A is done, but they accrue similar value regardless of which actor does A. (Because there are vastly more than enough resources in the universe for everyone. The notion that this is a zero-sum conflict to grab a scarce pot of gold is calibrated to a very different world than the "ASI exists" world.)
- doing A doesn't necessarily mean that your idiosyncratic values will play a larger role in shaping the long-term future than anyone else's, and in fact you're bought into a specific plan aimed at preventing this outcome. (Because CEV, no-pot-of-gold, etc.)
I do think there are serious risks and moral hazards associated with a transition to that state of affairs. (I think this regardless of whether it's a government or a private actor or an intergovernmental collaboration or whatever that's running the task AGI.)
But I think it's better for humanity to try to tackle those risks and moral hazards, than for humanity to just give up and die? And I haven't heard a plausible-sounding plan for what humanity ought to do instead of addressing AGI proliferation somehow.
[...] you're saying your aim is to design AI that can take over the world without killing anyone. Then once that is accomplished, you'll settle down to figure out how that unlimited power would best be used. [...]
The 'rush straight to CEV' plan is exactly the same, except without the "settling down to figure out" part. Rushing straight to CEV isn't doing any less 'grabbing the world's steering wheel'; it's just taking less time to figure out which direction to go, before setting off.
This is the other reason it's misleading to push on "taking over the world" noncentral fallacies here. Neither the rush-to-CEV plan nor the strawberries-followed-by-CEV plan is very much like people's central prototypes for what "taking over the world" looks like (derived from the history of warfare or from Hollywood movies or what-have-you).
I'm tempted to point out that "rush-to-CEV" is more like "taking over the world" in many ways than "strawberries-followed-by-CEV" is. (Especially if "strawberries-followed-by-CEV" includes a step where the task-AGI operators engage in real, protracted debate and scholarly inquiry with the rest of the world to attempt to reach some level of consensus about whether CEV is a good idea, which version of CEV is best, etc.)
But IMO it makes more sense to just not go down the road of arguing about connotations, given that our language and intuitions aren't calibrated to this totally-novel situation.
The first one says that flawed humans can directly wield superintelligence for a protracted period without screwing things up. The second one says that flawed humans can fully figure out how to safely wield superintelligence before it even arrives.
There's clearly some length of time such that the cost of waiting that long to implement CEV outweighs the benefits. I think those are mostly costs of losing negentropy in the universe at large, though (as stars burn their fuel and/or move away from us via expansion), not costs like 'the AGI operators get corrupted or make some major irreversible misstep because they waited an extra five years too long'.
I don't know why you think the corruption/misstep risk of "waiting for an extra three years before running CEV" (for example) is larger than the 'we might implement CEV wrong' risk of rushing to implement CEV after zero years of tinkering with working AGI systems.
It seems like the sensible thing to do in this situation is to hope for the best, but plan for realistic outcomes that fall short of "the best":
- Realistically, there's a strong chance (I would say: overwhelmingly strong) that we won't be able to fully solve CEV before AGI arrives. So since our options in that case will be "strawberry-grade alignment, or just roll over and die", let's start by working on strawberry-grade alignment. Once we solve that problem, sure, we can shift resources into CEV. If you're optimistic about 'rush to CEV', then IMO you should be even more optimistic that we can nail down strawberry alignment fast, at which point we should have made a lot of headway toward CEV alignment without gambling the whole future on our getting alignment perfect immediately and on the first try.
- Likewise, realistically, there's a strong chance (I would say overwhelming) that there will be some multi-year period where humanity can build AGI, but isn't yet able to maximally align it. It would be good if we don't just roll over and die in those worlds; so while we might hope for there to be no such period, we should make plans that are robust to such a period occurring.
There's nothing about the strawberry plan that requires waiting, if it's not net-beneficial to do so. You can in fact execute a 'no one else can destroy the world with AGI' pivotal act, start working on CEV, and then surprise yourself with how fast CEV falls into place and just go implement that in relatively short order.
What strawberry-ish actions do is give humanity the option of waiting. I think we'll desperately need this option, but even if you disagree, I don't think you should consider it net-negative to have the option available in the first place.
Meanwhile, in reality, we've already proceeded an unknown distance up the curve towards superintelligence, but none of the organizations leading the way has much of a plan for what happens, if their creations escape their control.
I agree with this.
In this situation, I say that people whose aim is to create ethical/friendly/aligned superintelligence, should focus on solving that problem. Leave the techno-military strategizing to the national security elites of the world. It's not a topic that you can avoid completely, but in the end it's not your job to figure out how mere humans can safely and humanely wield superhuman power. It's your job to design an autonomous superhuman power that is intrinsically safe and humane.
This is the place where I say: "I don't think the best solution to that problem is for the field to throw up their hands and say 'we're scientists, it's not our job to think about practicalities like that' and hope someone else takes care of it. We're human beings, not science-bots; we should use our human intelligence to think about which course of action is likeliest to produce good outcomes, and do that."
We aren't helpless victims of our social roles, who must look at Research Path A and Research Path B and go, "Hmm, there's a strategic consideration that says Path A is much more likely to save humanity than Path B... but thinking about practical strategic considerations is the kind of thing people wearing military uniforms do, not the kind of thing people wearing lab coats do." You don't have to do the non-world-saving thing just because it feels more normal or expected-of-your-role. You can actually just do the thing that makes sense.
(Again, maybe you disagree with me about what makes sense to do here. But the debate should be had at the level of 'which factual beliefs suggest that A versus B is likeliest to produce a flourishing long-run future?', not at the level of 'is it improper for scientists to update on information that sounds more geopolitics-y than linear-algebra-y?'. The real world doesn't care about literary genres and roles; it's all just facts, and disregarding a big slice of the world's facts will tend to produce worse decisions.)
If e.g. the government of Iceland suddenly understood how serious things had gotten and granted sanction and security to a project, that would fit this description, but I think that trying to arrange anything like this would probably make things worse globally because of the mindset it promoted.
I think I have an reasonable guess, but interested in more details about what goes wrong here and what mindset it promotes. (i.e. governments generally trying to regulate AI whether or not they understand how to do so, and then getting both random-ineffective regulations and rent capture and stuff?)
Curated and popular this week
