Have time horizons in mathematics been superexponential?

I have previously expressed skepticism about modeling time horizons as superexponential (absent feedback loops from AI speeding up AI R&D). In particular, I didn't buy AI Futures Project's argument that there are "meta-skills" that let humans do projects of arbitrary length.

Spelled out in more detail, my skepticism was coming from something like the following argument:

• The "AI time horizon" of a task – roughly, how much output the AI needs to produce to get one unit of feedback – should scale roughly linearly with training inputs (compute, training efficiency, etc.). This is supported by theoretical and empirical evidence in this paper.
• The "human time horizon" is somewhat subtle to define, since it depends on which humans you ask to complete the task. If you ask a random human, they will probably not be able to solve an IMO problem given essentially unlimited time, so there is a sense in which an AI that can do this already has an infinite time horizon. Therefore it is more coherent to consider a "moving baseliner" definition, in which we ask how long it takes humans who are competent at the task complete it. This is also a more reasonable way to extrapolate METR's methodology, who looked at the (geometric) mean time taken to complete a task among those who successfully completed it.
• Intuitively, the "AI time horizon" and "human time horizon" (moving baseliner version) should be similar to each other? (This is kind of a speculative assumption.)
• Therefore, if training inputs scale exponentially (which is the historical trend, and a reasonable extrapolation absent AI R&D feedback loops), we should expect human time horizons to scale exponentially.

However, qualitatively, it appears that the (human) time horizon of AI in formal mathematics has progressed faster than exponentially recently. This view has been expressed by Noam Brown:

When the models figured out how to do GSM8K, it would take a human mathematician about five seconds to do a GSM8K problem. This is grade school math, grades K-8. Then the next year, they were able to do the MATH benchmark problems. These would take an expert human mathematician maybe a minute to do.

Then you get to AIME. This is the qualifier for the USA Mathematics Olympiad team. It would take a good human mathematician probably 10 minutes to do, and the models were able to do that a year later. So every year, you’re seeing this 10x increase in the tasks they’re able to do, in terms of how long it would take a human mathematician to do it. Then it was very sensible that a year later we get to IMO gold, because that’s 100 minutes. That’s about how long it takes a human mathematician to do an IMO problem.

Just projecting outwards, I was like, “Okay, how long would it take a person to solve something like a Millennium Prize Problem?” I don’t have a good sense, but if we are following this trend line of 10x every year, we go from IMO gold, which is taking an hour and a half, to next year, 15 hours. That should not be enough to solve a Millennium Prize Problem. So I was like, “I don’t think we’re going to get it in 2026, probably not in 2027, maybe in 2028.” So it did happen a lot faster than I expected.

I would be very interested to know to what extent a more careful, quantitative assessment supports this apparent trend break. If there is strong support, it would be a substantial evidence against my argument above (especially the step labeled as a speculative assumption), which would change how quickly I expect AI to surpass humans in other fields.

I would get an LLM to conduct this study, but it involves some subtlety and careful judgment. In particular, I would want to put some thought into the following issues:

• Inference compute: performance is a function of inference compute, so we need to account for this in a reasonable way to make fair comparisons.
• Selection effects: when solutions to an open problem are announced, there is a selection effect on the difficulty of the problem for humans, properties of the problem that might make it easier for AI, and random chance. Ideally, we would try to account for these by using information about the number of attempted problems of similar difficulty, and perhaps by judging difficulty using the impressiveness the solutions.
• Task completion rate: time horizons are generally defined using a reliability threshold, often 50% or 80%.
• Choice of baseliners: as argued above, we should use a "moving baseliner" methodology, where we compare to humans who are competent at the task. This is a vague requirement that requires interpretation, and the way that human completion times are estimated will have a big impact on the final results.
• (Optional) AI R&D feedback loops: if the trend has been superexponential, it would be great to disentangle to what extent (if any) this has been because of AI R&D feedback loops rather than an underlying superexponential trend. Given that Epoch's Math ECI has not accelerated over the same period, I would guess that this is not a major factor, but perhaps there is other evidence bearing on this.

Even a first-pass attempt at this would be very interesting to me. The progress of AI in mathematics over the last few months has surprised a lot of people (even among those who were paying close attention), including Noam (as expressed above) and myself (as recently as last year, I was predicting it would take at least a couple of years to get to where we are today). So I think it would be valuable to have a better understanding of what has actually happened, and what the takeaways should be for other forecasts.