I feel like someone should be arguing the other side, and no one else has stepped up, so I guess I’ll have a go. :-P This comment will be like 75% my honest opinions and 25% devil’s advocate. Note that I wasn’t around at the time, sorry for any misunderstandings.
I think your OP does some conflation of (1) “Eliezer was trying to build FAI” with (2) “Eliezer was loudly raising the salience of ASI risk (and thus incidentally the salience ASI in general and how big a deal ASI is), along with related community-building etc.”. But these are two somewhat separate decisions that Eliezer made.
For example, you summarize an article as claiming “Shane Legg was introduced to the idea of AGI through a 2000 talk by Eliezer, and then co-founded DM in 2010 (following an introduction by Eliezer to investor Peter Thiel…)” Those seem to be (2) not (1), right? Well, I guess the 2000 talk is neither (1) nor (2) (Eliezer didn’t yet buy AI risk in 2000), but more generally, MIRI could have directly tried to build FAI without Eliezer giving talks and introducing people, and conversely Eliezer could have given talks and introduced people without MIRI directly trying to build FAI.
So I’m skeptical that (1) (per se) contributed nontrivially to accelerating the race to ASI. For example, I’d be surprised if Demis founded DeepMind partly because he expected MIRI to successfully build ASI, and wanted to beat them to it. My guess is the opposite: Demis expected MIRI to fail to build powerful AI at all, and saw it as a safety outfit not doing anything relevant from a capabilities perspective. After all, DeepMind pursued a very different technical research direction.
On the other hand, I think there’s at least a strong prima facie case that (2) shortened timelines, which is bad. On the other hand, (2) helped build the field of alignment, which is good. So overall, how do we feel about (2)? I dunno. You yourself seemed to be endorsing (2) in 2004 (“…putting more resources into highlighting the dangers of unsafe AI…”). For my part, I have mixed feelings, but by default I tend to be in favor of (2) for kinda deontological reasons (if people’s lives are at risk, it’s by default good to tell them). But (2) is off-topic anyway; the thing you’re re-litigating is (1), right?
OK next, let’s talk about intelligence augmentation (IA), per your other comment proposal: “Given that there are known ways to significantly increase the number of geniuses (i.e., von Neumann level, or IQ 180 and greater), by cloning or embryo selection, an obvious alternative Singularity strategy is to invest directly or indirectly in these technologies, and to try to mitigate existential risks (for example by attempting to delay all significant AI efforts) until they mature and bear fruit (in the form of adult genius-level FAI researchers).”
There are geniuses today, and they mostly don’t work on FAI. Indeed, I think existing geniuses have done more to advance UFAI than FAI. I think the obvious zeroth-order model is that a world with more geniuses would just have all aspects of intellectual progress advance more rapidly, including both capabilities and alignment. So we’d wind up in the same place (i.e. probably doom), just sooner.
What would be some refinements on that zeroth-order model that make IA seem good?
One possible argument: “Maybe there’s a kind of ‘uncanny valley’ of ‘smart enough to advance UFAI but not smart enough to realize that it’s a bad idea’. And IA gets us a bunch of people who are all the way across the valley”. But uncanny-valley-theory doesn’t seem to fit the empirical data, from my perspective. When I look around, “raw intelligence” vs “awareness of AI risk and tendency to leverage that understanding into good decisions” seem somewhat orthogonal to me, as much as I want to flatter myself by thinking otherwise.
Another possible argument: “Maybe it’s not about the tippy-top of the intelligence distribution doing research, but rather the middle of the distribution, e.g. executives and other decisionmakers making terrible decisions”. But realistically we’re not going to be creating tens of millions of geniuses before ASI, enough to really shift the overall population distribution. Note that there are already millions of people smarter than, say, Donald Trump, but they’re not in charge of the USA, and he is. Ditto Sam Altman, etc. There are structural reasons for that, and those reasons won’t go away when thousands of super-geniuses appear on the scene.
Another possible argument: “If awareness of x-risk, good decision-making, etc., relies partly on something besides pure intelligence, e.g. personality … well OK fine, we can do embryo-selection etc. on both intelligence and (that aspect of) personality.” I’m a bit more sympathetic to this, but the science to do that doesn’t exist yet (details). (I might work on it at some point.)
[[2025-12-13 UPDATE: …And even if this problem is solved, i.e. it becomes technically possible to engineer a child’s personality, there is no reason to expect people to actually make more babies with truth-seeking personalities who would help with x-risk, out of proportion to the number of extra baby Sam Altmans etc. who would make the situation worse. Again, the zeroth-order model above seems a better bet.]]
So that’s the IA possibility, which I don’t think changes the overall picture much. And now I’ll circle back to your five-point list. I already addressed the fifth. I claim that the other four are really bad things about our situation that we have basically no hope of avoiding. On my models, ASI doesn’t require much compute, just ideas, and people are already making progress developing those ideas. On the margin we can and should try to delay the inevitable, but ultimately someone is going to build it (and then probably everyone dies). If it gets built in a more democratic and bureaucratic way, like by some kind of CERN for AI, then there are some nice things to say about that from the perspective of ethical procedure, but I don’t expect a better actual outcome than MIRI-of-2010 building it. Probably much worse. The project will still be rolling its own metaethics (at best!), the project will still be ignoring illegible safety problems, the project will almost definitely still involve key personnel winding up in a position to grab world-altering power, and the project will probably still be subjecting the whole world to dire risk by doing something that most of the world doesn’t want them to do. (Or if they pause to wait for global consensus, then someone else will build it in the meantime.) We still have all those problems, because those problems are unavoidable, alas.
This may seem plausible because new evidence about the technical difficulty of alignment was the main reason MIRI pivoted away from their plan, but I want to argue that actually even without this information, there were good enough arguments back then to conclude that the plan was bad
I think it's not totally implausible that one could have called correctly well in advance that the problem itself would be too hard, without actually seeing much evidence that results from MIRI's and other attempts. I think one could consider things like "the mind is very complex and illegible" and "you have to have a good grasp on the sources of capabilities because of reflective self-modification" and "we have no idea what values are or how intelligence really works", and maybe get justified confidence, I'm not sure.
But it seems like you're not arguing that in this post, but instead saying that it was a bad plan even if alignment was easy? I don't think that's right, given the stakes and given the difficulty of all plausible plans. I think you can do the thing of trying to solve it, and not be overconfident, and if you do solve it then you use it to end acute risk, but if you don't solve it you don't build it. (And indeed IIUC much of the MIRI researcher blob pivoted to other things due to alignment difficulty.) If hypothetically you had a real solution, and triple quadruple checked everything, and did a sane and moral process to work out governance, then I think I'd want the plan to be executed, including "burn all the GPUs" or similar.
If hypothetically you had a real solution, and triple quadruple checked everything, and did a sane and moral process to work out governance, then I think I'd want the plan to be executed, including "burn all the GPUs" or similar.
First note that the context of my old debate was MIRI's plan to build a Friendly (sovereign) AI, not the later "burn all the GPUs" Task AI plan. If I was debating the Task AI plan, I'd probably emphasize the "roll your own metaethics" aspect a bit less (although even the Task AI would still have philosophical dependencies like decision theory), and emphasize more that there aren't good candidate tasks to for the AI to do. E.g. "burn all the GPUs" wouldn't work because the AI race would just restart the day after with everyone building new GPUs. (This is not Eliezer's actual task for the Task AI, but I don't remember his rationale for keeping the actual task secret so I don't know if I can talk about it here. I think the actual task has similar problems though.)
My other counterarguments all apply as written, so I'm confused that you seem to have entirely ignored them. I guess I'll reiterate some of them here:
What's a sane and moral process to work out governance? Did anyone write something down? It seems implausible to me, given other aspects of the plan (i.e., speed and secrecy). If one's standard for "sane and moral" is something like the current Statement on Superintelligence, then it just seems impossible.
"Triple quadruple checked everything" can't be trusted when you're a small team aiming for speed and secrecy. There are instances where widely deployed supposedly "provably secure" cryptographic algorithms and protocols (with proofs published and reviewable by the entire research community, who have clear incentives to find and publish any flaws) years later turned out to be actually insecure because some implicit or explicit assumption used by the proof (e.g., about what the attacker is allowed to do) turned out to be wrong. And that's a much better understood, inherently simpler problem that has been studied for decades, with public adversarial review processes that much better mitigate human biases compared to a closed small team.
See also items 2 and 5 in my OP.
and not be overconfident
I didn't talk about this in the OP (due to potentially distracting from other more important points) but I think Eliezer at least was/is clearly overconfident, judging from a number of observations including his confidence in his philosophical positions. (And overconfidence is just quite hard to avoid in general.) We're lucky in a way that his ideas for building FAI or a safe Task AI didn't almost work out, but instead fell wide of the mark, otherwise I think MIRI itself had a high chance of destroying the world.
you seem to have entirely ignored them
Well, I meant to address them in a sweeping / not very detailed way. Basically I'm saying that they don't seem like the sort of thing that should necessarily in real life prevent one from doing a Task-ish pivotal act. In other words, yes, {governance, the world not trusting MIRI, extreme power concentration} are very serious concerns, but in real life I would pretty plausibly--depending on the specific situation--say "yeah ok you should go ahead anyway". I take your point about takeover-FAI; FWIW I had the impression that takeover-FAI was more like a hypothetical for purposes of design-thinking, like "please notice that your design would be really bad if it were doing a takeover; therefore it's also bad for pivotal-task, because pivotal-task is quite difficult and relies on many of the same things as a hypothetical safe-takeover-FAI".
Basically I'm saying that they don't seem like the sort of thing that should necessarily in real life prevent one from doing a Task-ish pivotal act. In other words, yes, {governance, the world not trusting MIRI, extreme power concentration} are very serious concerns, but in real life I would pretty plausibly--depending on the specific situation--say "yeah ok you should go ahead anyway".
That's kind of surprising (that this is your response), given that you signed the Superintelligence Statement which seems to contradict this. But I can see some ways that you can claim otherwise, so let me not press this for now and come back to it.
I take your point about takeover-FAI; FWIW I had the impression that takeover-FAI was more like a hypothetical for purposes of design-thinking, like "please notice that your design would be really bad if it were doing a takeover; therefore it's also bad for pivotal-task, because pivotal-task is quite difficult and relies on many of the same things as a hypothetical safe-takeover-FAI".
Since you write this in the past tense ("had the impression"), let me first clarify: are you now convinced that sovereign-FAI (I'm avoiding "takeover" due to objection from Habryka and this) was a real and serious plan, or do you want more evidence?
Assuming you're convinced, I think you should (if you haven't already) update more towards the view I have of Eliezer, that he is often quite seriously wrong and/or overconfident, including about very important/consequential things like high level AI strategy. I applaud him for being able to eventually change his mind, which probably puts him in at least 99-percentile of humanity, but from an absolute standard, the years it sometimes takes is quite costly, and then often the new position is still seriously wrong. Case in point, the sovereign-FAI idea was his second one, after changing his mind from the first "accelerate AGI as fast as possible (the AGI will have good values/goals by default)".
Maybe after doing this update, it becomes more plausible that his third idea (Task AGI, which I guess is the first that you personally came into contact with, and then spent years working towards) was also seriously wrong (or more seriously wrong than you think)?
That's kind of surprising (that this is your response), given that you signed the Superintelligence Statement which seems to contradict this.
I think no one should build AGI. If someone is going to build AGI anyway, then it might be correct to make AGI yourself first, if you have a way to make actually aligned (hopefully task-ish or something).
are you now convinced that sovereign-FAI (I'm avoiding "takeover" due to objection from Habryka and this) was a real and serious plan, or do you want more evidence?
Let me rephrase. I already believed that there had been a plan originally, like 2004 +/- 3 years, to make sovereign AI. When I was entering the field, I don't recall thinking too hard about "what do you do with it", other than thinking about a Task-ish thing, but with Sovereign AI as a good test case for thought experiments. I don't know when Yudkowsky and others updated (and still don't).
I think you should (if you haven't already) update more towards the view I have of Eliezer, that he is often quite seriously wrong and/or overconfident
I'm still not sure where you're getting this? I mean, there are places where I would disagree with Yudkowsky's action-stances or something. For example, I kinda get the sense that he's planning "as though" he has confident short timelines; I don't think confident short timelines make sense, but that's different from how an individual makes their plans. For example, I'm working almost exclusively on plans that only pay off after multiple decades, which looks like "very confident of long timelines", but I'm not actually very confident of that and I say so...
For example, the rejection of HIA you quoted again seems pretty tepid, which is to say, quite non-confident, explicitly calling out non-confidence. In practice one can only seriously work on one or two different things, so I wonder if you're incorrectly inferring confidence on his part.
Maybe after doing this update, it becomes more plausible that his third idea (Task AGI, which I guess is the first that you personally came into contact with, and then spent years working towards) was also seriously wrong (or more seriously wrong than you think)?
I think I'm not following; I think Task AI is a bad plan, but that's because it's extremely difficult. I think you're asking me to imagine that it is solved, but that we should have unknown-unknown type certainty about our solution; and I just don't feel like I know how to evaluate that. If there was (by great surprise) some amazing pile of insights that made a safe Task-AGI seem feasible, and that stood up to comprehensive scrutiny (somehow), then it would plausibly be a good plan to actually do. I think you're saying this is somehow overconfident anyway, and maybe I just disagree with that? But it feels hard to disagree with because it's pretty hypothetical. If you're argument is "but other people are overconfident and wrong about their alignment ideas" I think this proves too much? I mean, it seems to prove too much if applied to cryptography, no? Like, you really can make probably-quite-secure systems, though it takes a lot of work and carefulness and stuff, and the guarantees are conditional on certain mathematical conjectures and only apply to some class of attacks. But I mean, the fact that many people with a wide range of abilities could become wrongly overconfident in the security of their hand-rolled system, doesn't prove an expert can't know when their system is secure in certain senses.
Let me rephrase. I already believed that there had been a plan originally, like 2004 +/- 3 years, to make sovereign AI. When I was entering the field, I don't recall thinking too hard about "what do you do with it", other than thinking about a Task-ish thing, but with Sovereign AI as a good test case for thought experiments. I don't know when Yudkowsky and others updated (and still don't).
I got a free subscription to Perplexity Pro, and used it to generate a report about this. The result seems pretty good. The short answer to "when Yudkowsky updated (from Sovereign to Task AGI)" is very likely 2014 or 2015 (see second collapsed section below).
Perplexity AI research report on "using this post and comment and linked resources as a starting point, can you build a comprehensive picture of SIAI/MIRI's strategy over time, especially pinpointing the specific dates that its strategy changed/pivoted, backed up by the best available evidence
https://www.lesswrong.com/posts/dGotimttzHAs9rcxH/relitigating-the-race-to-build-friendly-ai"
https://www.lesswrong.com/posts/dGotimttzHAs9rcxH/relitigating-the-race-to-build-friendly-ai"
SIAI/MIRI’s strategy over time: phases and pivot points
The Machine Intelligence Research Institute (MIRI), originally the Singularity Institute for Artificial Intelligence (SIAI), has changed strategy multiple times.
Over 25 years, the strategy has moved through roughly these phases:
- 2000–2003: “Build the Seed AI ourselves, as fast as possible” (accelerationist + Friendly AI).
- ~2003–2010: “Friendly AI and existential risk” + heavy outreach, still with a strong self‑image as the eventual FAI builder.
- 2011–2013: Professionalisation and pivot to technical FAI research; formal mission broadened from “build FAI” to “ensure AI turns out well,” though key people still expected MIRI (or a tight equivalent) to build a world‑altering Friendly AI.
- 2013–2016: Mature “world‑altering Friendly AI” / CEV vision plus growing emphasis on Task‑AGI and “pivotal acts.”
- 2017–2020: New engineering‑heavy, largely confidential research program aimed at new foundations for alignable AI, with a clear plan around “minimal aligned AGI” performing a pivotal act.
- 2020–2022: Public admission that the 2017 program had “largely failed” and a period of strategic regrouping; increasingly pessimistic public tone (e.g. AGI Ruin).
- 2023–present: Official pivot to policy and communications first, with research now framed as secondary support for an “international halt” to progress toward smarter‑than‑human AI.
Below is a more detailed, evidence‑based timeline, with particular attention to when strategy clearly shifted.
1. Founding: “Create a Friendly, self‑improving AI” (2000–2003)
Founding mission (July 2000).
The organization is founded on 27 July 2000 as the Singularity Institute for Artificial Intelligence by Eliezer Yudkowsky and Brian & Sabine Atkins. The mission statement on the early IRS forms and timeline is explicit:
“Create a Friendly, self‑improving Artificial Intelligence.”timelines.issarice
Early technical writings like Coding a Transhuman AI and Creating Friendly AI 1.0 (2001) describe specific architectures for a transhuman AI that self‑improves while remaining benevolent. This is not just “study risks”; it is a plan to design the Seed AI.intelligence+1
Early accelerationism.
MIRI’s own 2024 retrospective states that at founding, the goal was:
“to try to accelerate to smarter‑than‑human AI as quickly as possible, on the assumption that greater‑than‑human intelligence entails greater‑than‑human morality.”intelligence
So the earliest SIAI strategy can be summarized as:
- Build a self‑improving Seed AI (with Friendly properties).
- Speeding up AGI is seen as good, provided it is Friendly.
Approximate pivot:
By MIRI’s own later account, Eliezer’s “naturalistic awakening” — realizing that superintelligence would not automatically be moral — occurs between 2000–2003, and:
“MIRI shifted its focus to the alignment problem around 2003.”intelligence+1
So around 2003 there is a conceptual pivot from “accelerate AGI, assume it’s benevolent” to “AI alignment is a hard, central problem.”
2. Mid‑2000s: Friendly AI + outreach, still as would‑be FAI builders (2003–2010)
Mission reformulation (2007 and 2010).
By 2007 the formal mission is updated to:
“developing safe, stable, and self‑modifying Artificial General Intelligence.”timelines.issarice
In 2010 it further becomes:
“To develop the theory and particulars of safe self‑improving Artificial Intelligence… and otherwise improve the probability of humanity surviving future technological advances.”timelines.issarice
This is already less “we will build it” and more “develop theory and improve odds,” but still assumes SIAI as central technical actor.
Strategic plan 2011: three‑pronged strategy.
The 2011 Strategic Plan lays out three “core strategies”:
- “Solve open research problems in Friendly AI theory.”
- “Mobilize additional human capital and other resources” (Singularity Summit, LessWrong, volunteers, outreach).
- “Improve the function and capabilities of the organization.”intelligence
Near‑term (2011–2012) priorities emphasise:
- Public‑facing research on “creating a positive singularity.”
- Outreach/education/fundraising via the Summit and rationality community.
- Organizational maturity.intelligence
So in this period, strategy is mixed:
- Technical aim: make progress on FAI theory.
- Meta‑aim: grow a rationalist / EA / donor ecosystem and mainstream attention.
Evidence they still saw themselves as the eventual FAI builders.
Several pieces of evidence support the picture that, in the 2000s and up to at least ~2011, SIAI’s central plan was that it (or a very close “functional equivalent”) would actually build the first Friendly AGI:
- The MIRI timeline notes the original mission (2000–2006) as “Create a Friendly, self‑improving Artificial Intelligence.”timelines.issarice
In a video first hosted in 2011, Eliezer says:
“As a research fellow of the Singularity Institute, I'm supposed to first figure out how to build a Friendly AI, and then once I've done that go and actually build one…
The Singularity Institute was founded on the theory that in order to get a Friendly artificial intelligence someone's got to build one. So there. We're just going to have an organization whose mission is ‘build a Friendly AI’. That's us. … At the core, the reasoning is: ‘Someone has to do it. “Someone” is us.’”lesswrongLuke Muehlhauser’s 2013 post Friendly AI Research as Effective Altruism (looking back on the founding) explicitly confirms:
“MIRI was founded in 2000 on the premise that creating Friendly AI is the best way to ensure a positive Singularity…
MIRI co‑founder Eliezer Yudkowsky usually talks about MIRI in particular — or at least, a functional equivalent — creating Friendly AI.”intelligence
Wei Dai’s 2025 LessWrong post also recalls that he was “arguing against SIAI/MIRI’s plan to build FAI” as early as 2004. That is consistent with the explicit “someone is us” framing above.lesswrong+1
So for roughly 2000–2010, the strategic picture is:
- Goal: Design and eventually build a Friendly, self‑improving AGI.
- Means: Foundational theory + community‑building + outreach (Summits, LessWrong, rationality training).
- Role: SIAI is not just an advocacy group; it is intended Seed AI shop.
3. 2011–2013: professionalisation and pivot to FAI math research
Here we see the first clearly documented organizational pivot with a date.
3.1. 2013: “MIRI’s Strategy for 2013” – research‑only, FAI math‑first
In April 2013, MIRI (new name) publishes MIRI’s Strategy for 2013. Key elements:intelligence
They have:
“once [been] doing three things — research, rationality training, and the Singularity Summit. Now we’re doing one thing: research. Rationality training was spun out to CFAR, and the Summit was acquired by Singularity University.”intelligence
After strategic planning with 20+ advisors in early 2013, they decide to:
“(1) put less effort into public outreach, and
(2) shift our research priorities to Friendly AI math research.”intelligenceThey distinguish expository, strategic, and Friendly AI research and explicitly say:
“Strategic research will consume a minority of our research budget in 2013…”intelligence
In other words, as of early 2013:
- Outreach and rationality training are explicitly deprioritised.
- The core strategy is: focus limited staff and budget on math‑style Friendly AI research (logical uncertainty, self‑modification, decision theory) with some high‑quality strategy work, but little expository or broad public outreach.
3.2. 2013: bylaw / mission statement change
In the same period, MIRI’s formal mission statement is softened from “create Friendly AI” to a more conditional role. Luke writes in June 2013:
“When updating MIRI’s bylaws in early 2013, Yudkowsky and I came to a compromise on the language of MIRI’s mission statement, which now reads:
‘[MIRI] exists to ensure that the creation of smarter‑than‑human intelligence has a positive impact… Thus, the charitable purpose… is to: (a) perform research… (b) raise awareness… (c) advise… and (d) as necessary, implement a smarter‑than‑human intelligence with humane, stable goals’ (emphasis added).”intelligence
Luke adds:
“My own hope is that it will not be necessary for MIRI (or a functional equivalent) to attempt to build Friendly AI itself. But… I must remain open to the possibility that this will be the wisest course of action as the first creation of AI draws nearer.”intelligence
So early 2013 marks a formal pivot:
- In documents: from “we exist to build FAI” to “we exist to ensure AGI goes well; we might build AGI if necessary.”
- In practice: from “outreach + community + research” to “almost pure technical FAI math research.”
4. Circa 2013: the “world‑altering Friendly AI” plan
Wei Dai’s 2025 LessWrong post, which you linked, is primarily about this era. He characterizes the “circa 2013” strategy as:
“MIRI’s circa 2013 plan, to build a world‑altering Friendly AI, was a good one?”alignmentforum+1
His argument is that this plan was strategically bad even ex ante, because a small team attempting to directly build a Friendly sovereign AI faces “illegible safety problems” — traps they cannot even see.lesswrong
Does the evidence support that this really was the plan?
- Luke’s 2013 post explicitly notes that Eliezer usually talks about MIRI specifically (or a functional equivalent) creating Friendly AI.intelligence
In the 2013 strategy conversation with Holden Karnofsky and others, Eliezer says:
“Civilization doesn’t seem to be handling AI friendliness on its own … so our mission: just go do it…
I have never held out much hope for approaching a random AGI project 30 years from now… and showing them a bunch of results… So it doesn’t necessarily have to be in MIRI’s basement. It’s going to be some project that shows what I would consider a reasonable level of concern… The present community that we're trying to gather is not so much a community of people who are going to be influential on someone else. It's more like a community of people who do research work.”openphilanthropy
This is consistent with: MIRI expects either itself, or a close “strategically adequate” project built around its ideas and people, to be the group that actually builds the first aligned AGI.
- The LessWrong discussion notes that Yudkowsky “usually talks about MIRI in particular — or at least, a functional equivalent — creating Friendly AI,” and this needed explicit textual evidence by 2025 because it was no longer “common knowledge.”alignmentforum+1
Put together, the best reading is:
- Around 2013, MIRI’s operational strategy is “do deep foundational technical work now so that MIRI or a tightly aligned group can someday build a world‑altering Friendly AI (often conceived as CEV‑like), thereby securing the future.”
- Critics like Wei Dai were indeed pushing alternative strategies (e.g. invest in more geniuses via embryo selection / cloning and buy time by delaying AI efforts) as early as 2004.lesswrong
This is almost exactly the “race to build Friendly AI” picture the LessWrong post is relitigating, although there is internal disagreement over how much this should be described as “take over the world yourselves” versus “someone legitimate should, and we might be that someone.”lesswrong
5. Emergence of Task‑AGI and “pivotal acts” (≈2014–2017)
Over time, there is a shift in how MIRI talks about what the first aligned system should do.
- Early concepts such as Coherent Extrapolated Volition (CEV) envisaged a very powerful sovereign AI that, roughly, does what humanity would want if we were smarter, more informed, and more coherent. That is a maximally ambitious alignment target.
- Later, Yudkowsky distinguishes between:
- A Sovereign (a CEV‑like system running the long‑term future), and
- A Task AGI, a powerful but limited system intended to perform a pivotal act — some one‑time intervention that “upsets the gameboard” in our favor (e.g. uploading researchers, preventing hostile superintelligences, enabling nanotech), after which further decisions can be made under safer conditions.lesswrong+1
The LessWrong “pivotal act” and “Task AGI” pages (compiled later, but summarizing this thinking) explicitly give examples of a Task AGI that can:
- Upload humans and run uploads at high speed.
- Prevent the origin of hostile superintelligences, possibly temporarily.
- Deploy nanotechnology that gives a route to intelligence enhancement or strong defense.lesswrong+1
These are very close to the examples quoted in the comment thread you provided.
When does this show up as strategy?
The clearest institutional articulation is MIRI’s 2017 Fundraiser and Strategy Update (Nov 30, 2017). That document lays out a multi‑step high‑level plan:effectivealtruism
- Humanity needs a “stable period” to think clearly about long‑run futures and avoid locking in bad outcomes.
- To get there, we must first end an “acute risk period” in which AGI could cause extinction.
MIRI expects this to require a “risk‑mitigating technology” developed via:
“AGI‑empowered technological development carried out by task‑based AGI systems.”effectivealtruism
That, in turn, requires:
“Construction of minimal aligned AGI… aligned AGI that has the minimal necessary capabilities for ending the acute risk period.”effectivealtruism
- They stress the need for a “strategically adequate” project with strong opsec, research closure, commitment to the common good, and “white‑boxed system understanding.”effectivealtruism
This is, in effect, the Task‑AGI / pivotal‑act plan written down in strategic language.
So while Task‑AGI ideas were developing earlier, 2017‑11‑30 is the first strong documented pivot point where MIRI’s official strategic picture becomes:
- Don’t aim first for a full CEV‑style sovereign.
- Aim to enable a strategically adequate team to build a minimal aligned AGI capable of executing a pivotal act that ends the acute risk period.
Eli Tyre’s later comment, which you saw in the LessWrong thread, describes an “intentional switch, around 2016 (though I’m not confident in the date), from aiming to design a Friendly CEV‑optimizing sovereign AI to aiming to design a corrigible minimal Science‑And‑Engineering‑AI to stabilize the world.” That matches what shows up publicly in 2017.lesswrong
6. 2017–2018: new engineering‑heavy, nondisclosed‑by‑default research program
In parallel, there was a major research‑tactics pivot.
6.1. New research directions (2017–2018)
The 2017 fundraiser already notes that, with new funding and changed views, MIRI is “starting to explore new research directions while also continuing to push on our agent foundations agenda,” and is hiring software engineers to “experiment and explore some ideas in implementation.”effectivealtruism
The 2018 post 2018 Update: Our New Research Directions gives a detailed picture:intelligence
- MIRI still has the Agent Foundations agenda (logical uncertainty, embedded agency, decision theory).openphilanthropy+1
- “Recently, some MIRI researchers developed new research directions that seem to enable more scalable progress” by building software systems to test intuitions and iterate quickly.intelligence
- High‑level themes include:
- “Seeking entirely new low‑level foundations for optimization, designed for transparency and alignability from the get‑go, as an alternative to gradient‑descent‑style machine learning foundations.”intelligence
- Figuring out parts of cognition that can be “very transparent as cognition.”intelligence
These directions are explicitly engineering‑heavy, Haskell‑centric, and aimed at deconfusion while building real code, not just pen‑and‑paper math.intelligence
6.2. “Nondisclosed‑by‑default” policy
The same 2018 post announces a major change in publication policy:
“MIRI recently decided to make most of its research ‘nondisclosed‑by‑default’, by which we mean that going forward, most results discovered within MIRI will remain internal‑only unless there is an explicit decision to release those results…”intelligence
Reasons include:
- Worry that deconfusion results might also lead to capabilities advances.
- Desire to let researchers think without constant self‑censorship about dual‑use risks.
- Focus on speed of progress rather than exposition.intelligence
So 2018‑11‑22 marks a clear tactical pivot:
- From publishing most core work to doing a large fraction of the new engineering‑heavy research under confidentiality.
- From a more outreach‑oriented identity to “we really genuinely don’t mean outreach; we’re simply and only aiming to directly make research progress on the core problems of alignment.”intelligence
This is almost certainly the “4–5 years of confidential‑by‑default research” that Habryka references in the comment thread.lesswrong
7. 2020: admitting the 2017 research push “largely failed” and regrouping
By late 2020, MIRI states that the 2017–2019 “new directions” have not panned out as hoped.
In 2020 Updates and Strategy (Dec 20, 2020), Nate Soares writes:intelligence
- 2020 saw “limited progress in the research MIRI’s leadership had previously been most excited about: the new research directions we started in 2017.”intelligence
Senior staff have become “more pessimistic about this approach”:
“The non‑public‑facing research I (Nate) was most excited about had a flavor of attempting to develop new pragmatically‑feasible foundations for alignable AI… After pushing more on these lines of research, MIRI senior staff have become more pessimistic about this approach.”intelligence
As a result:
“We are currently in a state of regrouping, weighing our options, and searching for plans that we believe may yet have a shot at working.”intelligence
Some researchers leave; others attempt to “rescue” the 2017‑initiated lines; others pivot to new exploratory projects.intelligence
This is a clear strategic pivot dated 2020‑12‑20:
- Official recognition that the previous core engineering program is unlikely to succeed.
- An explicit search phase for new strategy, with no replacement yet fully specified.
8. 2022: “AGI Ruin” and strong public pessimism
Eliezer’s widely read 2022 essay AGI Ruin: A List of Lethalities further clarifies how MIRI leadership now see the situation.intelligence+1
Key points relevant to strategy:
- Alignment, in the sense “get to less than certain doom on the first dangerous try,” is portrayed as extremely difficult; the problem is lethal and must be solved under both a time limit and a “first critical try” constraint.intelligence
He argues a pivotal act is still needed:
“We need to align the performance of some large task, a ‘pivotal act’ that prevents other people from building an unaligned AGI that destroys the world.”intelligence
- But he also says he sees no realistic path to doing this under current conditions, and regards doom as the default outcome.
Although AGI Ruin is not itself a strategy document, it marks a public rhetorical pivot:
- The earlier plan “MIRI (or a strategically adequate project we seed) devises and executes a pivotal act using aligned AGI” is treated as effectively unachievable.
- This viewpoint sets the stage for the next major institutional shift: toward halting or pausing AGI rather than expecting to align and deploy it.
9. 2023–2024: pivot to policy and communications first
MIRI’s 2024 Mission and Strategy Update (Jan 3–4, 2024) is the most explicit top‑level strategy document and also provides a retrospective on earlier phases.lesswrong+2
After reviewing its history (founding accelerationism, 2003 alignment shift, 2006–2012 community‑building, 2013 Agent Foundations, 2017 engineering push, 2020 regrouping), it states:
“In 2023, MIRI shifted its strategy to pursue three objectives:
- Policy: Increase the probability that the major governments of the world end up coming to some international agreement to halt progress toward smarter‑than‑human AI, until humanity’s state of knowledge… has drastically changed; and until we are able to secure these systems…
- Communications: Share our models of the situation with a broad audience…
- Research: Continue to invest in a portfolio of research… including technical alignment research (though we’ve become more pessimistic that such work will have time to bear fruit if policy interventions fail to buy the research field more time).”effectivealtruism+2
They add:
“Although we plan to pursue all three of these priorities, it’s likely that policy and communications will be a higher priority for MIRI than research going forward.”lesswrong+1
Thus, in 2023 (per the 2024 retrospective) MIRI’s top‑level strategy pivots to:
- Primary: Push for an international halt / moratorium on smarter‑than‑human AI (policy) and make their models legible enough to shift the Overton window (communications).
- Secondary: Continue alignment research, but now framed as conditional on policy success buying time.
This is a very sharp contrast with:
- 2013 (“less outreach, more technical research”).intelligence
- 2018 (“we really don’t mean outreach; we’re just trying to solve core alignment problems”).intelligence
Now, outreach and policy work are explicitly prioritized over new technical work.
10. Consolidated list of major strategic pivots with dates and evidence
Below is a concise list of the clearest, evidence‑backed pivot points.
| Date / period | Pivot | Evidence |
|---|---|---|
| 2000‑07‑27 | Founding as SIAI with mission “Create a Friendly, self‑improving AI” and early accelerationist assumption “greater intelligence ⇒ greater morality.” | Founding mission on 2000 Form 990 and timeline; MIRI 2024 retrospective.timelines.issarice+1 |
| ~2003 | Conceptual shift from “accelerate AGI, assume morality” to “alignment is hard and central”; focus moves to the Friendly AI problem and x‑risk. | MIRI 2024 history: “shifted its focus to the alignment problem around 2003.”intelligence+1 |
| 2007–2010 | Mission reworded to “developing safe, stable, and self‑modifying AGI” and then “develop the theory… of safe self‑improving AI,” still with self‑image as eventual FAI builder. | Mission entries on timeline; early FAI texts.timelines.issarice+2 |
| 2011 strategic plan (Aug 2011) | First formal organisational strategy: three pillars (FAI research, outreach/human capital, organisational improvements); heavy emphasis on Summit and rationality outreach. | Strategic Plan, updated Aug 2011.intelligence |
| By ~2011 (video) | Eliezer states explicitly that SI’s mission is “first figure out how to build a Friendly AI, and then… actually build one… ‘Someone has to do it. “Someone” is us’,” confirming the self‑builder strategy. | 2011 video quoted by Eli Tyre.lesswrong |
| Early 2013 | Formal bylaw/mission change: from “build FAI” to “ensure that the creation of smarter‑than‑human intelligence has a positive impact,” with building AGI only “as necessary.” | Luke’s Friendly AI Research as Effective Altruism quoting the new bylaws.intelligence |
| 2013‑04‑12 | MIRI’s Strategy for 2013: pivot from “research + outreach + Summit + rationality” to “one thing: research,” with outreach and expository work sharply deprioritized and FAI math research prioritized. | MIRI’s Strategy for 2013.intelligence |
| Circa 2013 (world‑altering FAI plan) | Operational plan: MIRI or a closely allied small team aims to build a world‑altering Friendly AI (often sovereign / CEV‑like) to secure the future; this is the central “race to build Friendly AI” debated by Wei Dai et al. | Wei Dai’s retrospective; Luke’s confirmation that Eliezer talks of MIRI in particular creating FAI; 2013 strategy conversation stating “mission: just go do it,” possibly not “in MIRI’s basement” but via a tightly concerned project.lesswrong+3 |
| 2017‑11‑30 | 2017 Fundraiser and Strategy Update: explicit strategic model of success via “AGI‑empowered technological development” by task‑based AGI, construction of “minimal aligned AGI”, and need for a strategically adequate project; this is the Task‑AGI / pivotal‑act plan in official form. | 2017 strategy/fundraiser post.effectivealtruism |
| 2017–2018 | Launch of “new research directions” — engineering‑heavy, software‑based deconfusion work — alongside Agent Foundations, enabled by new funding. | 2017 fundraiser and 2018 update.effectivealtruism+1 |
| 2018‑11‑22 | Adoption of “nondisclosed‑by‑default” policy: most new technical results kept internal unless clear safety upside in release; MIRI explicitly positions itself as doing non‑outreach, straight‑shot alignment research. | 2018 Update: Our New Research Directions.intelligence |
| 2020‑12‑20 | 2020 Updates and Strategy: senior staff judge that the 2017–2019 non‑public‑facing research “has, at this point, largely failed”; MIRI enters a state of regrouping and strategy search. | 2020 Updates and Strategy.intelligence |
| 2022‑06‑10 | AGI Ruin: A List of Lethalities: public articulation that alignment is extremely difficult and likely to fail; still endorses need for a “pivotal act” but with very little hope of achieving it under current dynamics. | AGI Ruin.intelligence+1 |
| 2023 (described 2024‑01‑03) | Major strategy shift to policy and communications as higher priority than research: goal is to increase odds of an international agreement to halt progress toward smarter‑than‑human AI, while still doing some technical and supporting research. | MIRI 2024 Mission and Strategy Update.intelligence+2 |
How this lines up with the LessWrong post and comments
Starting from the post and comments you linked:
- The claim that “MIRI’s circa‑2013 plan [was] to build a world‑altering Friendly AI” is well supported by:
- The original mission (“Create a Friendly, self‑improving AI”).timelines.issarice
- Eliezer’s 2011 statements that SI’s mission is literally to build Friendly AI and that “someone has to do it… ‘Someone’ is us.”lesswrong
- Luke’s 2013 remark that Eliezer usually talks about MIRI in particular (or a functional equivalent) creating FAI.intelligence
- The 2013 strategy conversation, which explicitly dismisses the idea of merely influencing “a random AGI project 30 years from now,” in favor of gathering a community that “just [goes and does] it” with safety baked in from the ground up.openphilanthropy
- The “pivotal act” and “Task AGI” thinking that comes up in the comments corresponds closely to what MIRI formalizes in 2017 as “minimal aligned AGI” used for “AGI‑empowered technological development” to end the acute risk period.lesswrong+2
- The comment‑thread reference to a 4–5‑year confidential research period matches the 2017–2018 launch of nondisclosed‑by‑default, engineering‑heavy research and the 2020 admission that this push had largely failed.intelligence+1
- The observation that MIRI later pivoted away from “build FAI ourselves” because alignment turned out to be too hard is consistent with:
- The 2020 regrouping post.intelligence
- The strongly pessimistic tone of AGI Ruin (2022).intelligence
- The 2023 pivot toward policy and communication, treating large‑scale AI pause as central and technical alignment research as increasingly unlikely to arrive in time absent that pause.effectivealtruism+2
The main point of disagreement in the LessWrong discussion is about phrasing like “MIRI’s plan, to build a Friendly AI to take over the world,” which some participants (e.g. Habryka) see as overstating how explicitly “take over the world ourselves” was ever adopted as official organizational strategy, versus “someone should do something world‑altering, and maybe that someone is us or a very similar team.”lesswrong
On the factual side, however, the evidence is fairly clear that:
- For roughly its first 10–15 years, SIAI/MIRI saw itself as a small group that might directly build the first Friendly superintelligence (initially a sovereign CEV‑like system, later a minimal Task‑AGI for a pivotal act).
- Around 2013 it professionalized and narrowed to technical FAI work while still effectively planning to be (or seed) that project.
- From 2017 onward it concentrated on an internal, engineering‑heavy attempt to find new foundations for alignable AI.
- By 2020–2023 it concluded that this had failed and pivoted to policy and communication, with technical work now secondary and survival odds seen as hinging largely on halting or drastically slowing frontier AI.
That is the best evidence‑backed reconstruction of SIAI/MIRI’s strategic trajectory given the post, its links, and MIRI’s own later retrospectives.
- https://timelines.issarice.com/wiki/Timeline_of_Machine_Intelligence_Research_Institute
- https://intelligence.org/files/CFAI.pdf
- https://intelligence.org/2024/01/04/miri-2024-mission-and-strategy-update/
- https://intelligence.org/files/strategicplan2011.pdf
- https://www.lesswrong.com/users/elityre
- https://intelligence.org/2013/06/05/friendly-ai-research-as-effective-altruism/
- https://www.lesswrong.com/posts/dGotimttzHAs9rcxH/relitigating-the-race-to-build-friendly-ai
- https://www.alignmentforum.org/posts/dGotimttzHAs9rcxH/relitigating-the-race-to-build-friendly-ai
- https://intelligence.org/2013/04/13/miris-strategy-for-2013/
- https://www.openphilanthropy.org/wp-content/uploads/10-27-2013-conversation-about-MIRI-strategy.pdf
- https://www.lesswrong.com/w/coherent-extrapolated-volition-alignment-target
- https://www.lesswrong.com/w/pivotal-act
- https://forum.effectivealtruism.org/posts/GxmJ2ntyMiaG2PPSu/miri-2017-fundraiser-and-strategy-update
- https://intelligence.org/2018/11/22/2018-update-our-new-research-directions/
- https://openphilanthropy.org/files/Grants/MIRI/MIRI_Technical_Research_Agenda.pdf
- https://intelligence.org/2020/12/21/2020-updates-and-strategy/
- https://intelligence.org/2022/06/10/agi-ruin/
- https://www.alignmentforum.org/posts/uMQ3cqWDPHhjtiesc/agi-ruin-a-list-of-lethalities
- https://www.lesswrong.com/posts/q3bJYTB3dGRf5fbD9/miri-2024-mission-and-strategy-update
- https://forum.effectivealtruism.org/posts/Lvd2DFaHKfuveaCyQ/miri-2024-mission-and-strategy-update
- https://intelligence.org/summary/
- https://dash.harvard.edu/bitstreams/e9d128d7-ae0b-48a2-a88f-531fd238dac2/download
- https://www.lesswrong.com/posts/SZ2h2Yte9pDi95Hnf/on-miri-s-new-research-directions
- https://www.lesswrong.com/posts/ppmtsKiodNceMe3k6/what-are-some-alternative-approaches-to-understanding-agency
- https://intelligence.org/2018/11/28/2017-in-review/
- https://www.alignmentforum.org/posts/tD9zEiHfkvakpnNam/a-challenge-for-agi-organizations-and-a-challenge-for-1
- https://v-dem.net/media/publications/wp_122_final.pdf
- https://en.wikipedia.org/wiki/Singularity_Group
- https://en.wikipedia.org/wiki/Eliezer_Yudkowsky
- http://www.superintelligence.ch/Superintelligence.pdf
- https://www.alignmentforum.org
- https://zoo.cs.yale.edu/classes/cs671/12f/12f-papers/yudkowsky-ai-pos-neg-factor.pdf
- https://www.lesswrong.com/posts/4iqhkNv7woSnkpMRa/holden-karnofsky-s-singularity-institute-objection-1
- http://sl4.org/archive/0207/4638.html
- https://louisdelmonte.com/tag/strong-artificial-intelligence/
- https://www.reddit.com/r/DebateAnarchism/comments/1z9l24/anarchotranshumanism_ama/
- https://www.lesswrong.com/posts/2cxNvPtMrjwaJrtoR/ai-regulation-may-be-more-important-than-ai-alignment-for
- https://www.alignmentforum.org/w/taskagi_open_problems?l=2mx
- https://www.lesswrong.com/posts/FsEDu6CvzyzJkrQd8/pivotal-acts-using-an-unaligned-agi
- https://gyro.substack.com/p/on-approaching-doom
- https://futureoflife.org/recent-news/machine-learning-ai-risk/
- https://arbital.greaterwrong.com/p/unforeseen_maximum/
- https://arxiv.org/pdf/2503.05628.pdf
- https://www.lesswrong.com/posts/tD9zEiHfkvakpnNam/a-challenge-for-agi-organizations-and-a-challenge-for-1
- https://forum.effectivealtruism.org/topics/machine-intelligence-research-institute
- https://www.lesswrong.com/posts/uMQ3cqWDPHhjtiesc/agi-ruin-a-list-of-lethalities
- https://www.lawfaremedia.org/article/the-case-for-ai-doom-rests-on-three-unsettled-questions
- https://thezvi.substack.com/p/on-agi-ruin-a-list-of-lethalities
- https://intelligence.org/2022/07/30/july-2022-newsletter/
Follow-up answer to "Can you dig deeper, for example into LessWrong posts/comments, to find out when Eliezer or MIRI first started thinking or talking about task AGI?"
Based on a deeper analysis of LessWrong posts, Arbital edit histories, and MIRI archives, the concept of Task AGI (and the associated shift to "pivotal acts" rather than immediate "Sovereign/CEV") emerged earlier than the 2017 strategy update.
The specific "origin date" for the term and formalized concept is June 9, 2015.
Key Dates and Evidence
- June 9, 2015 (The "Smoking Gun"): Eliezer Yudkowsky created the page "Task-directed AGI" on Arbital on this exact date.
- Evidence: Metadata from Arbital scrapes explicitly list the page as "by Eliezer Yudkowsky Jun 9 2015 updated Mar 25 2017."arbital.obormot
- Significance: This proves the concept was being actively formalized in mid-2015, fully two years before it became the centerpiece of the 2017 public strategy.
- May 24, 2016: The Arbital page for the "Do-What-I-Mean hierarchy" was created, which integrated "Task AGI" into a broader safety framework. This confirms that by 2016, MIRI had a developed internal language around "Task AGI" vs "Sovereign AGI" that was distinct from their earlier "Tool AI" discussions.arbital
- 2014 (The Intellectual Catalyst): The term likely crystallized as a response to Nick Bostrom's book Superintelligence (published July 2014).
- Context: Bostrom popularized the typology of "Oracle, Genie, Sovereign, Tool."
- Connection: The Arbital definition of Task AGI explicitly references Bostrom: "An advanced AI that's meant to pursue a series of limited-scope goals... In Bostrom's terminology, a Genie."arbital.obormot
- Pivot Logic: MIRI had previously been skeptical of "Tool AI" (arguing that passive tools are unsafe). "Task AGI" appears to be their 2015 attempt to construct a safe version of a Genie/Tool—one that is agentic enough to be useful (perform a pivotal act) but limited in scope ("corrigible" and "mildly optimizing") to avoid taking over the future.
Summary of the Evolution
The timeline of MIRI's internal pivot to "Task AGI" can be reconstructed as:
- 2013–2014: Debate over "Tool AI" vs. "Oracle AI." Eliezer is critical of the idea that AI can just be a "passive tool" (see Holden Karnofsky's Singularity Institute Objection, 2012–2014).
- July 2014: Bostrom's Superintelligence provides new vocabulary (Genie vs. Sovereign).
- June 2015: Eliezer formally documents "Task-directed AGI" on Arbital. The strategic hope shifts from "Build CEV immediately" to "Build a Task AGI to perform a pivotal act."
- 2016: Internal consensus solidifies. MIRI researchers (like Eli Tyre and Nate Soares) begin treating "Task AGI for a pivotal act" as the standard plan. This is the "intentional switch around 2016" mentioned in the comments.
- Nov 2017: The strategy is officially announced to the public in the 2017 Fundraiser and Strategy Update.
In short: While the public saw the pivot in 2017, the intellectual shift happened in mid-2015, likely triggered by the digestion of Bostrom’s 2014 book and the need for a more pragmatic "minimum viable" target than full CEV.
- https://arbital.obormot.net/page/task_agi.html
- https://arbital.com/p/dwim/
- https://www.lesswrong.com/posts/keiYkaeoLHoKK4LYA/six-dimensions-of-operational-adequacy-in-agi-projects
- https://intelligence.org/2022/06/07/six-dimensions-of-operational-adequacy-in-agi-projects/
- https://arbital.com/p/soft_optimizer/
- https://www.lesswrong.com/w/coherent-extrapolated-volition-alignment-target
- https://arbital.com/p/cev/
- https://www.lesswrong.com/posts/zj7rjpAfuADkr7sqd/could-we-automate-ai-alignment-research-1
- https://arbital.com/p/task_agi/
- https://www.lesswrong.com/posts/7NvKrqoQgJkZJmcuD/instruction-following-agi-is-easier-and-more-likely-than
- https://www.lesswrong.com/posts/bnY3L48TtDrKTzGRb/ai-safety-success-stories
- https://www.lesswrong.com/w/autonomous-agi
- https://en.wikipedia.org/wiki/Artificial_general_intelligence
- https://www.reddit.com/r/SneerClub/comments/1i1wmhf/risaacarthur_fan_learns_about_lesswrong_is/
- https://philarchive.org/archive/TURCDA
- https://thezvi.wordpress.com/2022/06/13/on-a-list-of-lethalities/
- https://www.ibm.com/think/topics/artificial-general-intelligence
- https://manifold.markets/MartinRandall/is-there-a-pivotal-weak-act
- https://bayesianinvestor.com/blog/index.php/2024/12/29/corrigibility-should-be-an-ais-only-goal/
- https://intelligence.org/2022/06/10/agi-ruin/
- https://intelligence.org/2013/08/11/what-is-agi/
- https://www.greaterwrong.com/w/soft_optimizer?l=2r8&sort=new
- https://www.lesswrong.com/posts/dGotimttzHAs9rcxH/relitigating-the-race-to-build-friendly-ai
- https://www.alignmentforum.org/posts/AqsjZwxHNqH64C2b6/let-s-see-you-write-that-corrigibility-tag
- https://www.greaterwrong.com/posts/dGotimttzHAs9rcxH/relitigating-the-race-to-build-friendly-ai/comment/jg5z7WMELvm6rymfa
- https://www.greaterwrong.com/tag/task-directed-agi
- https://arbital.greaterwrong.com/p/AGI_typology?l=1g0
- https://www.lesswrong.com/users/tsvibt
1I think no one should build AGI. If someone is going to build AGI anyway, then it might be correct to make AGI yourself first, if you have a way to make actually aligned (hopefully task-ish or something).
If Eliezer or MIRI as a whole had said something like this, especially the first part "I think no one should build AGI." while pursuing their plans, I would be more tempted to give them a pass. But I don't recall them saying this, and a couple of AIs I asked couldn't find any such statements (until after their latest pivot).
Also I wouldn't actually endorse this statement, because because it doesn't take into account human tendency/bias to think of oneself as good/careful and others as evil/reckless.
I'm still not sure where you're getting this?
Eliezer claiming to have solved metaethics. Saying that he wouldn't "flinch from" trying to solve all philosophical problems related to FAI by himself. (man, it took me 30-60 minutes to find this link) Being overconfident on other philosophical positions like altruism and identity.
If there was (by great surprise) some amazing pile of insights that made a safe Task-AGI seem feasible, and that stood up to comprehensive scrutiny (somehow), then it would plausibly be a good plan to actually do.
I would be more ok with this (but still worried about unknown unknowns) if "comprehensive scrutiny" meant scrutiny by thousands of world-class researchers over years/decades with appropriate institutional design to help mitigate human biases (e.g., something like academic cryptography research + NIST's open/public standardization process for crypto algorithms). But nothing like this was part of MIRI's plans, and couldn't be because of the need for speed and secrecy.
Ok. I think I might bow out for now unless there's something especially salient that I should look at, but by way of a bit of summary: I think we agree that Yudkowsky was somewhat overconfident about solving FAI, and that there's a super high bar that should be met before making an AGI, and no one knows how to meet that bar; my guess would be that we disagree about
- the degree to which he was overconfident,
- how high a bar would have to be met before making an AGI, in desperate straits.
Definitely read the second link if you haven't already (it's very short and salient), but otherwise, sure.
(I did read that one; it's interesting but basically in line with how I think he's overconfident; it's possible one or both of us is incorrectly reading in / not reading in to what he wrote there, about his absolute level of confidence in solving the philosophical problems involved.)
Hmm, did you also read my immediate reply to him, where I made the point "if you’re the only philosopher in the team, how will others catch your mistakes?" How to understand his (then) plan except that he would have been willing to push the "launch" button even if there were zero other similarly capable philosophers available to scrutinize his philosophical ideas?
(Also just recording that I appreciate the OP and these threads, and people finding historical info. I think the topic of how "we" have been going wrong on strategy is important. I'm participating because I'm interested, though my contributions may not be very helpful because
- I was a relative latecomer, in that much of the strategic direction (insofar as that existed) had already been fixed and followed;
- I didn't especially think about strategy that much initially, so I didn't have many mental hooks for tracking what was happening in the social milieu in terms of strategic thinking and actions.)
(Oh I hadn't read the full thread, now I have; still no big update? Like, I continue to see him being seemingly overconfident in his ability to get those solutions, but I'm not seeing "oh he would have mistakenly come to think he had a solution when he didn't", if that's what you're trying to say.)
made it very difficult for voices calling for AI pause/stop to attract attention and resources.
This isn't implausible, but could you point to instances / evidence of this? (I.e. that MIRI's plan / other participation caused this.)
Recently I've been relitigating some of my old debates with Eliezer, to right the historical wrongs. Err, I mean to improve the AI x-risk community's strategic stance. (Relevant to my recent theme of humans being bad at strategy—why didn't I do this sooner?)
Of course the most central old debate was over whether MIRI's circa 2013 plan, to build a world-altering Friendly AI[1], was a good one. If someone were to defend it today, I imagine their main argument would be that back then, there was no way to know how hard solving Friendliness/alignment would be, so it was worth a try in case it turned out to be easy. This may seem plausible because new evidence about the technical difficulty of alignment was the main reason MIRI pivoted away from their plan, but I want to argue that actually even without this information, there were good enough arguments back then to conclude that the plan was bad:
(The main rhetorical innovation in my current arguments that wasn't available back then is the concept of "illegible safety problems", but the general idea that there could be hidden traps that a small team could easily miss had been brought up, or should have been obvious to MIRI and the nearby community.)
Many of these arguments are still relevant today, considering the plans of the remaining and new race participants, but are not well known due to historical reasons (i.e., MIRI and its supporters argued against them to defend MIRI's plan, so they were never established as part of the LW consensus or rhetorical toolkit). This post is in part an effort to correct this, and help shift the rhetorical strategy away from putting everything on technical alignment difficulty.
(This post was pulled back into draft, in order to find more supporting evidence for my claims, which also gave me a chance to articulate some further thoughts.)
My regular hobby horse in recent years has been how terrible humans are at making philosophical progress relative to our ability to innovate in technology, how terrible AIs may also be at this (or even worse, in the same relative sense), and how this greatly contributes to x- and s-risks. But I've recently come to realize (or pay more attention to) how terrible we also are at strategic thinking, and how terrible AIs may also be at this (in a similar relative sense), which may be an even greater contribution to x- and s-risks.[3]
(To spell this out more, if MIRI's plan was in fact a bad one, even from our past perspective, why didn't more people argue against it? Weren't there anyone whose day jobs were to think strategically about how humanity should navigate complex and highly consequential future technologies/events like the AI transition, and if so why weren't they trying to talk Eliezer/MIRI out of what they were planning? Either way, if you were observing this course of history in an alien species, how would you judge their strategic competence and chances of successfully navigating such events?)
A potential implication from all of this is that improving AI strategic competence (relative to their technological abilities) may be of paramount importance (so that they can help us with strategic thinking and/or avoid making disastrous mistakes of their own), but this is clearly even more of a double-edged blade than AI philosophical competence. Improving human strategic thinking is more robustly good, but suffers from the same lack of obvious tractability as improving human philosophical competence. Perhaps the conclusion remains the same as it was 12 years ago: we should be trying to pause or slow down the AI transition to buy time to figure all this out.
This was edited from "to build a Friendly AI to take over the world in service of reducing x-risks" after discussion with @habryka and @jessicata. Jessica also found this passage to support this claim: "MIRI co-founder Eliezer Yudkowsky usually talks about MIRI in particular — or at least, a functional equivalent — creating Friendly AI." (Interestingly, what was common knowledge on LW just 12 years ago now requires hard-to-find evidence to establish.)
According to the linked article, Shane Legg was introduced to the idea of AGI through a 2000 talk by Eliezer, and then co-founded DM in 2010 (following an introduction by Eliezer to investor Peter Thiel, which is historically interesting, especially as to Eliezer's motivations for doing so, which I've been unable to find online). I started arguing against SIAI/MIRI's plan to build FAI in 2004: "Perhaps it can do more good by putting more resources into highlighting
the dangers of unsafe AI, and to explore other approaches to the
Singularity, for example studying human cognition and planning how to do
IA (intelligence amplification) once the requisite technologies become
available."
If we're bad at philosophy but good at strategy, we can do things like realize the possibility of illegible x-risks (including ones caused by philosophical errors), and decide to stop or slow down the development of risky technologies on this basis. If we're good at philosophy but bad at strategy, we might avoid making catastrophic philosophical errors but still commit all kinds of strategic errors in the course of making highly consequential decisions.