The bottleneck to AI self-improvement isn't compute, it's simulation and verification
Any domain that can be simulated, AI will eventually solve; any domain that can't be simulated, even the strongest model can only wait for experimental data. Richard Socher argues the path to recursive self-improvement is to first let AI research AI, then attack biology.
The video won't play here. Listen to the audio instead:
The argument · tap a timestamp to hear it
Scientific progress is slowing because knowledge has become a maze
Socher's argument: antibiotics turned bacterial infections from a death sentence into a minor nuisance, but viruses and cancer were not solved the same way; after E=mc² and general relativity, physics has produced no fundamental breakthrough of comparable rank. The reason isn't too few people or too little money — it's that every field has fragmented into a thousand subfields, and it takes a person many years to go deep in just one, so the number of people in each subfield is actually too small. He cites predictions from Stanisav and others, and considers this an important cause of the slowdown in science.
— Richard SocherAcademia doesn't reward things that are too new
Socher uses his own experience to illustrate: doing neural networks for NLP around 2010, most of his first few papers were rejected by NLP conferences and could only be published in niche workshops. He remembers the first NIPS deep learning workshop had only thirty or forty people, all of whom later became famous, and at the time they were seen as a small band of heretics. His judgment: the career path demands you be novel, but if you're too novel your paper gets rejected — and that in itself is a social cause of the slowdown in science.
— Richard SocherThe prompt engineering paper was rejected unanimously
Socher mentions he wrote a paper describing prompt engineering (DECA NLP), arguing that a single neural network could be prompted with arbitrary questions. The paper was rejected by almost every reviewer and area chair, on grounds including ‘too crowded’ and ‘meaningless’, with some even saying that not even humans have a system that can answer every type of question. His argument: what looks obvious now was completely non-obvious to the whole field at the time. This explains why, when biologists say ‘impossible within decades’, he doesn't necessarily believe them.
— Richard SocherPredicting the next token is learning geography and protein structure
Socher uses the example of ‘I'm driving north from New York’ predicting the next word as Boston to show that simply predicting the next token forces the model to absorb geographic knowledge about the locations of cities. The same mechanism applied to proteins: in a neural network trained on next-token prediction, proteins that are closer together in 3D space after folding also show corresponding correlation in the model's representations. His conclusion: a large neural network trained on enough data in a specific domain will absorb that domain's knowledge through next-token prediction, and the token can be an English word, a protein, an image pixel or a snippet of sound.
— Richard SocherIn domains that can be simulated and verified, AI will definitely surpass humans
Socher offers a predictable criterion: any domain that can be simulated and verified, AI can experiment in the simulation infinitely many times, and will therefore eventually solve those problems. Chess and Go have no hidden variables, and AI can also play itself, so he isn't surprised AI wins. He goes further: mathematics will change enormously as a result, and frontier mathematicians like Tao are already clear on this; programming is the strongest verifiable domain right now, because you can give AI a screenshot of a website and have it generate code, then verify it automatically, so all of programming and the digital economy will change.
— Richard SocherWhat biology lacks isn't models, it's training data
Socher says the route to recursive self-improvement is to first have AI do AI research, reaching the knowledge and capability equivalent to 50,000 PhDs, and then attack physics, chemistry and biology. Biology's bottleneck is data: you need to run a large number of perturbation experiments — knock out a gene or add a molecule and see how the cell responds — and only after accumulating enough of them can AI possibly learn the underlying patterns, just as it derives Boston from ‘I'm driving north from New York’. The ultimate goal is to build a virtual cell and let AI experiment inside it repeatedly. He mentions Tahoe Therapeutics and Parallel Bio are working on data collection.
— Richard SocherWhether AI can cure cancer depends on demand elasticity
Socher reduces AI's impact on employment to how elastic demand is when a product's price falls. Illustrators hate AI because an illustration went from $200 to 2 cents, and the world doesn't need that many illustrations — demand didn't rise a thousandfold. Programming is the opposite: when the price falls, demand actually increases, i.e. the Jevons paradox, so demand for programmers is now higher. Back to cancer, he thinks AI will play a big role in curing many kinds of cancer, because every cancer is not homogeneous and you need therapies customized for each person and each subtype — exactly the kind of systems problem AI is good at — but clinical trials will still take many years.
— Richard SocherThe AI economist was rejected outright by journals
Socher says economics has no objective benchmarks like computer science, so it easily becomes a political arena. They used two-level reinforcement learning to build an AI economist, submitted to Nature and Science and were rejected outright, and in one case an ethicist who didn't understand AI rejected it without reading the full text. In their simulation, simple agents have different utility functions and working hours, can collect resources, build houses, monopolize, and a meta agent decides how to tax and subsidize, with the reward being equality multiplied by productivity. They found agents would avoid taxes by shifting gains around the tax year boundary, and that the famous Saez formula is only optimal in a single-step economy.
— Richard SocherThe four pillars of the Eureka Machine
Socher breaks the Eureka Machine into four pillars: first, large language models, which absorb world knowledge; second, reality models, which bring scientific measurement data (the parts human language hasn't described or is hard-pressed to describe) into the model; third, simulation, from Go and chess to the virtual cell, with different domains at different levels of abstraction; fourth, the real world, where you must run experiments to verify the confounding variables the simulation missed. On top of the four pillars sit the agent swarm and the community of scientists. He specifically notes that no team has yet built a virtual cell in the full sense.
— Richard SocherCompute is the biggest constraint, and data always has new sources
Socher says compute is the biggest constraint, and in the future companies and nations will both have to decide which problems are worth solving and how much compute to give them, and new scaling laws will emerge. Most of the public internet's content has already been digested by the major labs, but new data is always being produced, news for example. He mentions y.com partnering with labs like Neolabs to supply fresh search results when a model is asked about something that happened last week.
— Richard SocherIn their own words · checked verbatim
Anything you can simulate, AI will solve. But before you know it, you're in this recursive self-improvement loop.
Richard Socher0:01
as there are more and more sub fields and niches, it's actually hard to have enough people in each of these sub fields
Richard Socher3:03
AI doesn't really care if it's English or a sequence of amino acids.
Richard Socher10:11
anything you can simulate AI will solve
Richard Socher18:18
boy are we far away from the true upper bounds of any of the spaces of intelligence
Richard Socher1:12:49
Figures
| Size of the Recursive–Amazon compute deal | $410 million | 1:06:45 |
| Number of recessions economists failed to predict | 148 of the past 150 recessions were not predicted | 45:37 |
| Outcome of the AI economist paper submission | Rejected outright by Nature and Science | 45:37 |
| Change in illustration price | From $200 per illustration to 2 cents | 35:29 |
| Date of the Progen paper | 2018 | 29:27 |
| Publication date of The Eureka Machine | September 22 | 1:13:49 |
Glossary
- recursive self-improvement
- AI improving AI itself, forming a continuously accelerating loop.
- next-token prediction
- The basic training objective of a language model: predict the next token in a sequence.
- Jevons paradox
- When the efficiency of resource use rises, total consumption increases rather than falls.
- virtual cell
- A computational system that fully simulates the behavior of a real cell with a model; it does not yet exist.
- open-endedness
- A system that continuously generates novel and unpredictable directions of exploration, similar to biological evolution.
How to listen
Founders and investors focused on AI for Science, biopharma investing and AI infrastructure, especially anyone trying to judge the timeline for recursive self-improvement.
40:19 to 44:35, the discussion of whether society will reject AI — philosophical, low information density.