I strongly doubt we live in a data-limited AGI timeline
I think it's more fair to say humans were "trained" over millions of years of transfer learning, and an individual human is fine tuned using much less data than Chinchilla.
Is that fair to say? How much kolmogorov complexity can be encoded by evolution at a maximum, considering that all information transferred through evolution must be encoded in a single (stem) cell? Especially when we consider how genetically similar we are to beings which don’t even have brains, I have trouble imagining that the amount of “training data” encoded by evolution is very large.
I don't see the problem. Your learning algorithm doesn't have to be "very" complicated. It has to work. Machine learning models don't consist of million lines of code. I do see the problem where one might expect evolution not to be very good at doing that compression, but I find the argument that there would actually be lots of bits needed very unconvincing.
Last time I checked, you could not teach a banana basic arithmetic. This works for most humans, so obviously evolution did lots of leg work there.
A lot of the human genome does biochemical stuff like ATP synthesis. These genes, we share with bananas. A fair bit goes into hands, etc. The number of genes needed to encode the human brain is fairly small. The file size of GPT3 code is also small.
I think humans and current deep learning models are running sufficiently different algorithms that the scaling curves of one don't apply to the other. This needn't be a huge difference. Convolutional nets are more data efficient than basic dense nets.
AIXI, trained on all wikipedia, would be vastly superhuman and terrifying. I don't think we are anywhere close to fundamental data limits. I think we might be closer to the limits of current neural network technology.
Sure, video files are bigger than text files.
Yes, self play allows for limitless amounts of data, which is why AI can absolutely be crazy good at go.
My model has AI that are pretty good, potentially superhuman, at every task where we can give the AI a huge pile of relevant data. This does include generating short clickbait videos. This doesn't include working out advances in fundamental physics, or designing a fusion reactor, or making breakthroughs in AI research. I think AIXI trained on wikipedia would be able to do all those things. But I don't think the next neural networks will be able to.
This doesn't include working out advances in fundamental physics, or designing a fusion reactor, or making breakthroughs in AI research.
Why don't all of these fall into the self-play category? Physics, software and fusion reactors can all be simulated.
I would be mildly surprised if a sufficiently large language model couldn't solve all of Project Euler+Putnam+MATH dataset.
Physics can be simulated, sure. When a human does a simulation, they are trying to find out useful information. When an neural net is set, they are trying to game the system. The human is actively optimizing for regions where the simulation is accurate, and if needed, will adjust the parameters of the simulation to improve accuracy. The AI is actively trying to find a design that breaks your simulation. Designing a simulation broad enough to contain the width of systems a human engineer might consider, and accurate enough that a solution in the simulation is likely to be a solution in reality, and efficient enough that the AI can blindly thrash towards a solution with millions of trials, that's hard.
Yes software can be simulated. Software is a discrete domain. One small modification from highly functioning code usually doesn't work at all. Training a state of the art AI takes a lot of compute. Evolution has been in a position where it was optimizing for intelligence many times. Sure, sometimes it produces genuine intelligence, often it produces a pile of hard coded special case hacks that kind of work. Telling if you have an AI breakthrough is hard. Performance on any particular benchmark can be gamed with a heath robinson of special cases.
Existing Quanum field theory, can kind of be simulated, on one proton at huge computational cost, and using a bunch of computational speed up tricks specialized to those particular equations.
Suppose the AI proposes an equation of its new multistring harmonic theory. It would take a team of humans years to figure out a computationally tractable simulation. But ignore that and magically simulate it anyway. You now have a simulation of multistring harmonic theory. You set it up with a random starting position and simulate. Lets say you get a proton. How do you recognise that the complicated combination of knots is indeed a proton? You can't measure its mass, mass isn't fundamental in multistring harmonic theory. Mass is just the average rate a particle emits massules divided by its intrauniverse bushiness coefficient. Or maybe the random thing you land on is a magnetic monopole, or some other exotic thing we never new existed.
Let's take a concrete example.
Assume you have an AI that could get 100% on every Putnam test, do you think it would be reasonable or not to assume such an AI would also display superhuman performance at solving the Yang-Mills Mass Gap?
Producing machine verifiable formal proofs is an activity somewhat amenable to self play. To the extent that some parts of physics are reducible to ZFC oracle queries, maybe AI can solve those.
To do something other than produce ZFC proofs, the AI must be learning what real in practice maths looks like. To do this, it needs large amounts of human generated mathematical content. It is plausible that the translation from formal maths to human maths is fairly simple, and that there is enough maths papers available for the AI to roughly learn it.
The Putnam archive consists of 12 questions X 20 years=240 questions, spread over many fields of maths. This is not big data. You can't train a neural net to do much with just 240 examples. If aliens gave us a billion similar questions (with answers), I don't doubt we could make an AI that scores 100% on putnam. Still it is plausible that enough maths could be scraped together to roughly learn the relation from ZFC to human maths. And such an AI could be fine tuned on some dataset similar to Putnam, and then do well in putnam. (Especially if the examiner is forgiving of strange formulaic phrasings)
The Putnam problems are a unwooly. I suspect such an AI couldn't take in the web page you linked, and produce a publishable paper solving the yang mills mass gap. Given a physicist who understood the question, and was also prepared to dive into ZFC (or lean or some other formal system) formulae, then I suspect such an AI could be useful. If the physicist doesn't look at the ZFC, but is doing a fair bit of hand holding, they probably succeed. I am assuming the AI is just magic at ZFC, that's self play. The thing I think is hard to learn is the link from the woolly gesturing to the ZFC. So with a physicist there to be more unambiguous about the question, and to cherrypick and paste together the answers, and generally polish a mishmash of theorems into a more flowing narrative, that would work. I'm not sure how much hand holding would be needed. I'm not sure you get your Putnam bot to work in the first place.
It wouldn't just guess the next line of the proof, it'd guess the next conjecture. It could translate our math papers into proofs that compile, then write libraries that reduce code duplication. I expect canonical solutions to our confusions to fall out of a good enough such library.
I would expect such libraries to be a million lines of unenlightening trivialities. And the "libraries to reduce code duplication", to mostly contain theorems that were proved in multiple places.
I think if you pretrained it on all of YouTube, you could get explanations and illustrations of people doing basic tasks. I think this would (if used with appropriate techniques that can be developed on a short notice) make it need very little data for basic tasks, because it can just interpolate from its previous experiences.
In this history, uploads (via data from passive BMIs) precede AGIs, a stronger prospect of alignment.
What happens afterwards, I don't know. A perfect upload is trivially aligned. I wouldn't be that worried about random errors. (Brain damage, mutations and drugs don't usually make evil geniuses) But the existence of uploading doesn't stop alignment being a problem. It may hand a decisive strategic advantage to someone, which could be a good thing if that someone happens to be worried about alignment.
Going from a big collection of random BMI data to uploads is hardish. There is no obvious easily optimized metric. It would depend on the particular BMI. I think its fairly likely something else happens first. Like say someone cracking some data efficient algorithm. Or self replicating nanotech. Or something.
An upload (an exact imitation of a human) is the most straightforward way of securing time for alignment research, except it's not plausible in our world for uploads to be developed before AGIs. The plausible similar thing is more capable language/multimodal models, steeped in human culture, where alignment guarantees at least a priori look very dubious. And an upload probably needs to be value-laden to be efficient enough to give an advantage, while remaining exact in morally relevant ways, though there's a glimmer of hope generalization can capture this without a need to explicitly set up a fixpoint through extrapolated values. Doing the same with Tool AIs or something is only slightly less speculative than directly developing aligned AGIs without that miracle, so the advantage of an upload is massive.
Assuming of course that the first upload/(sufficiently humanlike model ) is developed by someone actually trying to do this.
This is a prediction of what I think the near future might be like.
Suppose the trends in scaling laws roughly continue. Deep learning can do anything if it has enough data. But getting that data is hard.
So large language models get better, but not that much better. They are using most of the available high quality text, and it is hard to get more text. Image generation, can do. Self driving cars. Took lots of simulations, lots of gathering data and even building a filmset town in a desert and letting 1000 self driving cars crash all over it for a few months. But we got there. Short videos, can do, 30 second clickbait videos are now generated by something like DALLE-2. Social media profiles. Done. Deep learning can learn to do anything, given enough data.
It can't learn to make AI breakthroughs. There just aren't enough examples of humans coming up with breakthroughs. You can train a model to set network parameters, but the best parameters are fairly well understood. And you need to run it on small examples, so you get slightly better parameters for Mnist classifiers. The more choice you give the network, the more likely you are to get something that doesn't function at all. If you just create an RL agent producing code, it won't produce anything that compiles. (or at least anything nontrivial that compiles). If you pretrain on code, your model will output code similar to existing code. So it will output existing algorithms, with the minor adjustments any competent programmer could easily do. Often neural networks. Sometimes k-means or linear regression.
If you prompt a large language model for a superintelligent output, you usually get a result like this.
This isn't a coincidence, the pattern [prompt asking for superintelligence][Answer degenerating into nonsense] appears many times in the training data. (For example it appears here. And if no one give me a reason not to, it can appear in many other places as well).
So in this world, we have AI that is limited to things there is large amounts of data for. If many humans do something every day, and that data is reasonably collectable, an AI can be trained to do it. If the AI can play blindly for a million rounds before it figures something out, then it can do it. Any computer games. Short term picking things up with robot hands etc. If you have lots of robots, and a wholesale supply of eggs, and a team of people to clean up the first attempts, you can train an AI to make an omelette. (Especially if you have many human examples to start with). Some startups are doing this.
Perhaps formal theorem provers with deep learning selection of the next line of proof become a thing.
The world has reached a new equilibrium. The thing humans still hold over the machines is data efficiency. And there it will stay until someone either manages to invent a much more data efficient algorithm, an abstract theory step that will hopefully take a while, or someone stumbles onto a much more data efficient algorithm trying to make some routine AI, or someone manages to cajole code for a superintelligence out of a language model. This is an equilibrium that could plausibly remain for more than a decade.