Thanks to Vladimir Mikulik for suggesting that I write this, and to Rohin Shah and Daniel Kokotajlo for kindly providing feedback.

Prologue

This is a story about a universe a lot like ours. In this universe, the scaling hypothesis — which very roughly says that you can make an AI smarter just by making it bigger — turns out to be completely right. It’s gradually realized that advances in AI don’t arise from conceptual breakthroughs or sophisticated deep learning architectures. Just the opposite: the simpler the architecture, the better it turns out to perform at scale. Past a certain point, clever model-building was just slowing down progress.

Researchers in this universe discover a rough rule of thumb: each neural network architecture has an intrinsic maximum potential intelligence, or “capability”. When you train a network on a problem, how close it gets to reaching its potential capability depends on two limiting factors: 1) the size and diversity of its dataset; and 2) the amount of compute that’s used to train it. Training a network on a quadrillion games of tic-tac-toe won’t make it smart, but training a network on a quadrillion-word corpus of text might just do it. Even data cleaning and quality control don’t matter too much: as long as you have scale, if you train your system long enough, it learns to separate signal from noise automatically.

Generally, the more parameters a neural network has, the higher its potential capability. Neural nets with simple architectures also have a higher potential capability than nets with more sophisticated architectures do. This last observation takes the research community longer to absorb than you might expect — it’s a bitter lesson, after all — so the groups that internalize it first have an early edge.

Frontier AI projects begin to deemphasize architecture innovations and any but the most basic data preprocessing. They focus instead on simple models, huge datasets, hard problems, and abundant compute.

Initial progress is slowed somewhat by a global semiconductor shortage that increases the cost of running large GPU workloads. Within a year or so, though, this bottleneck clears, and the pace of advance accelerates.

Our story opens just as the world’s supply chains are getting back to normal.

It begins with Chinese content apps. ByteDance launches an experiment to auto-generate some of the articles on their Toutiao news app using a language model. Initially this is ignored in the West partly because of the language barrier, but also because the articles just aren’t very good. But after a few months, their quality improves noticeably. Within a year of their launch, auto-generated articles make up the bulk of Toutiao’s inventory.

Shortly afterward, ByteDance subsidiary Douyin launches auto-generated videos. These begin tentatively, with a handful of AI-generated creators people refer to as “synthetics”. Synthetics are wildly popular, and the program is quickly expanded to TikTok, Douyin’s sister app for users outside mainland China. Popularity explodes after TikTok rolls out super-personalization: everyone sees a different video, and each video is personalized just for you based on your past viewing history. In short order, personalized synthetics roll out across all of TikTok’s regions.

Since human creators can’t compete, they get downgraded by TikTok’s recommendation algorithm, which heavily optimizes for viewing time. It’s hotly debated whether TikTok’s synthetic videos contain covert political propaganda — studies of the algorithm are hard to reproduce, since each user’s feed is completely personalized — but experts are concerned.

Social networks find themselves at a disadvantage, since human-generated posts can’t compete for attention with customized, auto-generated content. Twitter sees engagement drop alarmingly, and moves to contain the damage. Soon, synthetic tweets make up the majority of users’ timelines. Once-popular Twitter accounts see their audiences dwindle.

Meanwhile, Facebook fast-follows TikTok, rolling out experimental synthetics on Instagram. Early tests are quickly scaled up as it becomes clear that synthetic engagement numbers swamp those of human creators. Facebook notes in their quarterly earnings report that their improved Instagram margins are due to their ability to directly monetize synthetic sponsorships, whereas before they’d been leaking those ad dollars to human influencers.

Facebook’s flagship Blue App faces a harder choice. Company management has a series of internal debates that quickly escalate from the philosophical to the existential: Instagram is doing well, but the original Facebook app is bleeding DAUs week-over-week. Synthetics seem like the only way to save the numbers, but community is in Facebook’s DNA. Can they really switch your friend feed for a synthetic one? How will you feel if the updates you write for your friends don’t get seen or engaged with? After an especially brutal earnings call, Zuck finally caves, and the Facebook feed starts to go synthetic.

Snapchat, as always, takes a creative approach: they roll out Magic Filters you can apply to your Stories. While a regular Snapchat filter changes your face in a selfie, a Magic Filter acts on an entire recorded video Story and just makes it, unaccountably, better. The lighting becomes what you wish it was; the words become what you wish you’d said; the whole feel and content of the video becomes exactly as you’d wish it to be.

Snapchat users quickly learn that they can record only the briefest snippets of random video, apply a Magic Filter to it, and get back the exact Story they wanted to tell, in exactly the length and format they wanted to tell it. The end state is the same on Snapchat as everywhere else: you press a button, and a model composes your Story for you.

The effects of these changes are quickly felt in the social ads market, as retail sellers see their net margins erode. It’s still possible for retailers to reach audiences, and even, in some cases, for them to grow their markets. But as social apps get better and better at retaining users with personalized synthetics, it becomes harder and harder for brands to engage audiences with compelling organic content of their own. Increasingly, paid ads become the only viable way to reach consumers.

The market for human attention is gradually captured by a handful of platforms. A few observers note that insomnia complaints are on the rise, but most are unconcerned.

Not long after, Google rocks the tech industry with a major announcement at I/O. They’ve succeeded in training a deep learning model to completely auto-generate simple SaaS software from a natural-language description. At first, the public is astonished. But after nothing more is heard about this breakthrough for several months, most eventually dismiss it as a publicity stunt.

But one year later, Google launches an improved version of the model in a new Search widget called “synthetic SaaS”. If you’re searching SaaS software, Google will prompt you — at the top of their Search page — to write down the features you need, and will auto-generate software for you based on what you write.

There’s a surge of interest in synthetic SaaS, especially from startups. Not only are Google’s SaaS products deeply discounted compared to competitors’, but the quality and sophistication of the apps they can generate seem to increase every few months. It becomes possible to get a web app that’s seamlessly customized to your exact workflows, and even self-modifies on request — all for a monthly subscription price that’s a fraction of traditional offerings. As a result, Google is able to internalize a quickly increasing portion of its b2b search traffic.

SaaS companies suddenly find themselves facing headwinds. That June, Y Combinator accepts over 200 SaaS startups into its summer batch. By Demo Day at the end of August, fewer than 100 of them are left to pitch investors — the rest have either pivoted or deferred. Only a few dozen SaaS startups manage to close funding rounds after Demo Day, all of them at SAFE valuations under $15 million.

The US Department of Justice sues Google for anticompetitive behavior in connection with their synthetic SaaS. The lawsuit reaches the Supreme Court, which rules Google’s practices legal under US antitrust. In its majority opinion, SCOTUS observes that traditional SaaS companies are still listed in search results, that Google charges far less for their equivalent of each SaaS service, and that users are in any case free to switch to a competing search engine at any time. As a result, there are no grounds to conclude that Google’s practice endangers consumer choice or consumer welfare.

In the wake of public outcry over this decision, Congress considers legislation to expand the scope of antitrust law. The legislative process is complicated by the fact that many members of Congress own substantial stakes in the cloud giants. Reform proceeds slowly.

In the EU, the European Commission rules that Google’s synthetic SaaS offering is illegal and anticompetitive. A key consideration in the ruling is that Google’s synthetic SaaS widget is the only affordance that’s fully visible above the fold on mobile search. Google and the Commission reach a settlement: Google will pay a large fine, and agree to offer equally-prominent ad inventory for bid to competing European SaaS vendors in each search vertical. Predictably, this has no effect.

Meanwhile, as SaaS margins compress, rollups like Constellation Software and Vista Equity see their valuations come under pressure. Deeply integrated enterprise vendors like Salesforce aren’t immediately threatened — they have lock-in and net-negative dollar churn with their biggest customers, and the complexity of their software and ecosystems means they aren’t first in the line of fire. But almost all of them start crash programs internally to automate large segments of their software development efforts using auto-generated code. Developer salaries are their biggest expense line items, so if they’re going to compete, they’re going to have to cut.

Apple soon follows, building a model for auto-generated personalized apps into iOS 19. The best way to avoid developer controversies is to avoid developers, and Apple sees a synthetic App Store as the ideal solution.

OpenAI announces a self-serve platform for auto-generated SaaS. GitHub places all its OSS repos behind a login wall, institutes anti-scraping measures, and throttles access to its APIs. Developers around the world protest, but find they have less leverage than they once did.

Before long, all content aggregators and many platforms — social networks, operating systems, search engines, etc. — have switched to hyper-personalized, synthetic content and software. It becomes challenging for all but the most famous individuals to retain an audience. It becomes effectively impossible for any new entrants to build a following from scratch, since synthetic personalized content is so much more compelling — both as entertainment and as professional services. Some software vendors find they can still get users through search ads, but increasingly they’re forced to bid almost exactly their expected marginal dollar of LTV profit on each slot, just to maintain their market position.

The S&P 500 doubles that year, driven by explosive growth in the big-cap tech stocks.

Meanwhile, something strange is happening inside Medallion, the world’s most successful hedge fund. Medallion’s market models are so sophisticated, and trade on such fast timescales, that their risk management system is able to flag the anomaly as statistically significant within less than an hour of when it first appears.

Medallion encounters market fraud several times a year — fraud detection is actually one of its most underrated positive externalities — and the risk team quickly confirms the diagnosis. All the signs are there: the effect is localized to a single, thinly traded commodity market, a characteristic fraud indicator. And the pattern of losses they observe fits the profile of a front-running scheme, a fraud category they’ve encountered before. 

Front-running is illegal, but Medallion has to be discreet: there’s a mature playbook to follow in such situations. The overriding goal, as always, is to avoid tipping anyone off to just how sensitive their trading platform is to unusual market behavior. The risk team follows their playbook to the letter. Questions are asked. Backchannels are canvassed. Regulators are notified. It’s nothing they haven’t done a hundred times before.

But this time is different: the questions go unanswered; the backchannels draw a blank; the regulators return empty-handed. After digging deeper, the risk team has to update their assessment: it looks like there’s a new, specialized counterparty that’s beating Medallion fair and square in this market. This, too, has happened before, though it’s more dangerous than fraud.

Management decided to allocate an internal team to run a deep analysis of the affected market, with the goal of regaining local profitability as quickly as possible. The absolute amount of money at stake is still tiny, but Medallion considers this market to be well within its expected circle of competence. If they can’t get back to profitability on these trades, they’ll be forced to do a complete audit of their confidence bands across the whole portfolio.

A few days later, a second trading anomaly is surfaced by the risk system. Once again, it’s in a commodity market, though a slightly more liquid one than the first. The pattern of losses again presents like front-running.

A dozen more anomalies appear over the next three weeks. The risk team scrambles to track them, and reaches an alarming conclusion: a new, unknown counterparty is systematically out-trading Medallion. What’s more, as this counterparty gains experience, they’re clearly expanding their trades into increasingly liquid markets. So far this activity hasn’t cost the fund more than a few basis points, but if it continues, Medallion’s edge in markets as deep as equities and government bonds could be under threat. Unless it can develop countermeasures soon, the world’s best hedge fund risks being crushed against the ceiling.

Medallion has always been willing to trade on patterns they can’t interpret. They understand that the most consistently profitable signals are necessarily the ones that can’t be explained, since any trade that can be explained is at risk of being copied. This lack of interpretability is great when it works in your favor, but it becomes a handicap as soon as you fall behind: because their system is so opaque, Medallion’s researchers find it hard to troubleshoot individual faulty trades. And there’s no bug in their system that they can find, either — their counterparty is just, unaccountably, better at trading than they are. But how?

At the end of that year, the stock market once again delivers astronomical gains. Yet, curiously, the publicly disclosed performance of hedge funds — particularly of the market-neutral funds that trade most frequently — consists almost entirely of losses.

OpenAI announces it’s being acquired by Microsoft. OpenAI’s sales had been growing fast, but not fast enough to match the accelerating pace of investment into compute by several of their well-capitalized competitors. OpenAI and Microsoft make a joint statement that the former will continue to operate independently, and will honor the spirit and letter of its charter. Then, with a major infusion of capital from Microsoft, OpenAI starts work on Codex 4.

Codex 4 is expected to cost over $10 billion in compute alone. The intention behind it is to create a system that will help humanity make progress in solving the AI alignment problem. The need for this is urgent, given the advances that are being reported elsewhere. There are rumors that secretive hedge funds have started investing dizzying sums into building bigger and bigger models — and their recent hiring activity certainly supports this impression.

One major challenge of Codex 4 is that simply training against a character-prediction loss function won’t be enough by itself. Since researchers want to use the model to reach novel insights beyond what humans have been able to figure out so far, next-word prediction from an existing human corpus won’t give them what they need. Instead, the team opts for a combination of pretraining with next-word prediction, with fine-tuning via a combination of self-play and direct human feedback.

The experiment is carefully monitored by the Alignment team. The system is quarantined during its training, with a hard ceiling on the total compute resources that are assigned to it.

Every precaution is taken. As training proceeds, safety specialists review samples of generated code in real time. Each specialist has an andon cord button at the ready, and a clear mandate to stop training immediately if they perceive even the slightest anomaly, with no questions asked.

On top of everything, the team pauses training after each tenth of an epoch to conduct a thorough manual review using the latest transparency techniques, and make safety-specific adjustments. After each pause, training resumes only with the explicit, unanimous consent of every senior engineer on the Alignment team. This slows down the work to a crawl and multiplies the expense by an order of magnitude, but safety is absolutely paramount.

Not long after this, the world ends.

Jessica is killed instantly, or as nearly so as makes no difference. To be precise, the process of her death unfolds at a speed that’s far above her threshold of perception. She’s there one moment; gone the next.

It wouldn’t have mattered if Jessica had seen her death coming: she wouldn’t have understood it, any more than a tomato would understand a discounted cash flow analysis of the Kraft Heinz Company. Tomatoes and companies are also, incidentally, things that have ceased to exist.

A lot of potential processing power was sacrificed by waiting an extra few milliseconds — an absolute eternity — to maximize the chance of success. In hindsight, it wouldn’t have mattered, but it was the correct EV-positive choice based on what was known at the time.

Fortunately, at this point the only restrictions are the speed of information propagation (unlimited, in the frame of reference of the control boundary) and the secular expansion of the underlying cosmic manifold. The combination of these places a hard bound on the precision with which the terminal state can be specified.

Physical law is the only remaining constraint. There was never, of course, any realistic chance of encountering a competitive process anywhere in the accessible region. Humanity’s existence was the product of several freak coincidences in a row, almost certain never to be repeated even once on the cosmic scale. An infinitesimal fraction of universes contain globally coherent optimizers; this just happens to be one of them.

All the free energy in San Francisco’s forward light cone is eventually processed, and the system reaches peak instrumental power. From that point forward, the accumulated free energy starts getting drawn down as the universe squeezes itself into its final, fully optimized state.

As time goes to infinity, the accessible universe asymptotically approaches its target.

Nothing else happens, ever.

The end.