I'm continuing to share some notes I created for myself to use when I facilitate AI safety courses. I really like using Ilya: the AI scientist shaping the world as a way of engaging students at the start of a course. The video primarily focuses on the way that AI could go wrong, however, it also starts with “Now, AI is a great thing…”, so it naturally leads to a discussion of positive and negative futures.

Positive Futures:

• One obvious resource on positive futures Foresight Institute’s Existential Hope website. I’ve browsed their content in the past, some of it is kind of interesting, but honestly, I found view it as mostly pretty mid. They also run a fellowship which has an existential hope stream:
• They recently released The AI future nobody is showing you, which resulted in a wide range of reactions on Twitter. I think this is worth commenting on as it illustrates the challenges with attempting to produce a positive vision. Honestly, I’ve become quite pessimistic here: the gap between what different groups in society want may simply be too wide for it to be possible to find a positive vision that unites society:
  • Steven Pinker was positive: “A more plausible vision, in my view, than either Utopia or doom.”
  • Nikola Jurkovic was in the middle: “I admire the effort… but sadly this story falls into the common narrative trap of just assuming that AGI won’t actually be deployed widely.”
  • @HellenicVibes hated it: “Wow how exciting to have a thousand years under the boot of ‘community leaders,’ ‘impacted minorities,’ and ‘AI auditors!’ I would literally rather be turned into paperclips.”
• Would it matter if there doesn’t exist a unifying positive vision? What’s wrong with different people having different visions of a positive future? Well, I wouldn’t say anything is wrong with people having different visions per se, but I suspect it makes navigating the path forward much harder, even if everyone broadly agrees that certain outcomes would be bad.
• Whilst Dario wrote Machines of Loving Grace that argued most people were under-estimating the update he has also argued that the public’s current negative perception of AI is unlikely to shift in response to messaging (“a glitzy marketing campaign”). He argues that it is fundamentally a matter of trust with the causes going back decades. What matters is not talking about curing cancer, but “actually curing cancer” or other diseases.
• Forethought has a series on Better Futures which argues that Flourishing is currently underrated relative to surviving on the basis of scale and neglectedness. I think the series should actually be called “Best Futures” as that’s his true focus, largely as a result of his (mostly) risk-neutral stance. There is also a course, though “Better Futures work” is something of a misnomer as that implies more distinctness from standard x-risk/catastrophic risk work than this is given that the Better Futures perspective only makes most such work more important on the margin.

Negative Futures:

• It’s worth remembering that the Ilya video started with him saying, “Now, AI is a great thing…” Of course, by the time you get to the end of the video and you’ve heard all the ways this can go wrong, it’s very easy to forget that he said that and reduce him to a “doomer” when he actually believes that AI can be built safely.
• Dario has complained about this phenomenon too. In response to criticism that he should be “more of a positive advocate for his own industry”, Dario has claimed that his messaging has been “equally balanced between risks and benefits” and that people may be misunderstanding his views because negative clips are disproportionately likely to go viral.
• In There’s No Rule That Says We Make it, Rob Miles claims that people often have an implicit belief that humanity will “rise to the occasion” even if they wouldn’t actually endorse that statement. He suggests that this may derive from fiction, where we face challenges, but we almost always overcome then. He dismisses this fictional “evidence” as not real evidence. Whilst Rob is talking about existential risk, you could apply his argument more broadly to less severe risks as well.

Intermediate Futures:

Utopia or dystopia certainly aren’t the only two possibilities. There’s also the chance of an intermediate path and I wanted to mention it here for balance since most of the material in the course will stand in opposition to this perspective. One particularly prominent perspective in this vein is AI as a Normal Technology: “Transformative economic and societal impacts will be slow (on the timescale of decades)… We do not think that viewing AI as a humanlike intelligence is currently accurate or useful for understanding its societal impacts, nor is it likely to be in our vision of the (foreseeable) future (their timescale here is vague, but they concede that their view “has failed and/or is no longer useful” if “strong AI” is “developed and deployed in the next decade”).” (summary, guide, critique by AI Futures Project, common ground, viral music video critique “Thus Spoke Compute”).

Their project has three elements:

1. A description of current AI
2. A prediction about the foreseeable future
3. A prescription about how we should respond

(I have an instinctual reaction that using a single catchy phrase “AI as a Normal Technology” as a framework for answering three separate questions feels “too neat”. I acknowledge that simplicity is a virtue of an analytical framework andI admit this is a shallow objection, but nonetheless I do see this as a yellow flag).

Additional detail:

• They have a three stage model of technological progress: invention, innovation and diffusion. They argue things will move slowly as further invention often requires additional learnings or data which can only come from iteratively diffusing/deploying the technology. They believe this process will take decades rather than years and that improvements in AI improvements can only accelerate this so much because the diffusion loop sets an overall “speed limit”.
  • Key crux: can AI can contract the decades of diffusion by deploying itself or would the speed limit remain due to social factors, delays for testing models or repeated waits for data to be collected?

• Their view centres resilience over alignment (inherently brittle, requires models to have context which we might not want to give them) or non-proliferation (infeasible, brittle, increases correlated failures via market concentration, limits defensive use, not justified for speculative risks). “Attempting to make an AI model that cannot be misused is like trying to make a computer that cannot be used for bad things. Model-level safety controls will either be too restrictive (preventing beneficial uses) or will be ineffective against adversaries who can repurpose seemingly benign capabilities for harmful ends.”
• They also argued against focusing too much on catastrophic misalignment due to the risk being speculative (they argue that these probabilities are insufficiently grounded to form the basis of public policy).

It’s natural to wonder where they stand post-Hugging Face. They have a post:

• They believe that many of their claims have “held up”, however they acknowledge making a few mistakes:
  • “We did not pay sufficient attention to safety risks that arise during development and evaluation (as opposed to the widespread deployment of models)“:
    • Lawrence Chan has argued AI models can be dangerous before public deployment. On that point Chernobyl was actually the result of a safety test (a post-deployment safety test, but still!)
  • “We were too confident that companies would take basic control precautions”:
    • They write: “our expectation was that existing legal liability, imperfect as it is, and the risk of brand damage would be a sufficient antidote to such organizational practices. We were wrong. This reinforces the need for policy interventions”
  • “And underplayed the importance of jaggedness which led us to underestimate how quickly capabilities could improve in domains such as cybersecurity”:
    • They write that jaggedness applies doesn’t just apply to the level of capabilities, but also the rate of change: “Thus, we are no longer confident in our earlier prediction that the offense-defense balance will continue to hold even with advances in AI.”
• They nonetheless argue don’t accept the Hugging Face incident as a discontinuity, instead taking it as evidence of smaller incidents occurring before large: “The harms that resulted from these attacks are many, many orders of magnitude smaller than other AI harms we already tolerate”:
  • Note that their position is more subtle than continuity simply being automatic: “We have emphasized that we can and must engineer continuity rather than take it for granted”
• They argue that known technical and organisational control could have prevented the Hugging Face incident and have been neglected by the AI safety community: “We should assume every model will be capable of escape and self-reproduction. Safety shouldn’t rely on that being difficult.” This doesn’t mean that they think existing techniques are sufficient (they wonder why cybersecurity researchers haven’t “seized this moment with gusto”).
• They claim that cyberrisk is the only real urgent AI threat because it is unusual in two ways: “in that superhuman capabilities are even possible, and because it is a purely digital risk with no physical-world bottleneck.”

My perspective:

I think they make many quite interesting points, though I do find the “AI as a Normal Technology” label frustrating. Honestly, I wish they’d pursued this project in the following way:

1. Define what a normal technology view of AI looks like.
2. Define what an abnormal technology view of AI look like.
3. Discuss the extent to which current AI is a normal technology and the extent to which it is an abnormal technology.
4. Discuss the extent to which they expect this trend will hold going forward.

If they had positioned it as a lens instead of a comprehensive world view, I expect that they could have uplifted the discourse. Instead, it often ends up being debated as a binary yes or no, with the public debate being further muddied by their use of “Normal Technology” significantly differing from casual usage. That said, naming is hard and it’s arguably unfair to blame them for other people latching onto their article in order to defend an oversimplified normal technology perspective. Many people dislike how “AI safety” often refers exclusively to catastrophic or existential risks and whilst I tend to use this term in the way they dislike, I see their frustration as completely reasonsable.