AI researchers sped up 10x in two years, but RSI may be easier than being a paralegal
RSI is a cumulative task: in theory a Python file under a million tokens could train a recursively self-improving model from scratch. A law-firm paralegal, by contrast, has to remember shifting interpersonal relationships and tacit ways of doing things — a non-stationary distribution.
The video won't play here. Listen to the audio instead:
The argument · timestamps estimated from transcript position
A model looks smart, then a month later looks dumb again
John Schulman describes a recurring loop: a new model comes out, people are blown away and say ‘this is AGI’, but after about a month of use it starts to look dumb again. This loop may keep repeating, and it's hard to predict how many more times. His mechanism: capability won't explode right now, because doing research and engineering you still get stuck at bottlenecks — even if a model can write far more code than a human, it won't give you a 100x productivity gain. Dwarkesh opens by asking: if there's no crazy alien superintelligence by 2036, what's the most likely technical reason? Beren Millidge's default answer is a Moravec-paradox-style continuation — AI does superbly at whatever humans put into a benchmark or environment, but a stubborn sim-to-real gap always stands in the way.
— John Schulman / Beren MillidgeDistillation is the only force against centralisation
John Schulman sees distillation as the main force countering the centralisation trend: anything learnable through RL can be distilled easily, because the number of bits is small and a little data suffices. But he stresses distillation isn't trivial — when distilling with supervised learning the prompt distribution matters enormously, and you need a real, broad prompt distribution. He mentions something that surfaced recently: some Chinese companies may be running router services that let Chinese users access otherwise-blocked US frontier models, and these services collect and sell data, which is an extremely useful dataset for distillation because it gives a perfect prompt distribution. Charlie O'Neill draws a concrete prediction from this: frontier labs may not have much of an edge left on RL environments.
— John Schulman / Charlie O'NeillRSI is a cumulative task; being a lawyer is not
Charlie O'Neill splits tasks into two kinds: cumulative and non-stationary. RSI is cumulative — in theory a Python file under a million tokens could train a recursively self-improving model from scratch, and once you discover attention, mixture of experts or GRPO you just add it to the training stack without going back to the start. The real world is not cumulative: a law-firm paralegal has to remember the shifting relationships among all the important people in the company and all the tacit ways of doing things, which is a non-stationary distribution. Dwarkesh's summary: ‘RSI just happens to be easier than being a paralegal, which is very unfortunate.’ If labs conclude RSI is cumulative, more compute will tilt that way.
— Charlie O'Neill / Dwarkesh PatelContinual learning is stuck on catastrophic forgetting
Charlie O'Neill says the outer loop of pouring data into mid-training at large scale and large batch does work, but at the micro level — a model doing several hundred consecutive small updates for one law firm — every method collapses: SFT on successful trajectories catastrophically forgets and general capabilities degrade; on-policy distillation pushes that boundary out a bit further but eventually yields too. RL is good at instilling capabilities and bad at instilling knowledge. Beren Millidge thinks this is mainly a technical problem, not a capacity problem: a model of the same size pretrained from scratch on all the data is simply better, and the bottleneck is plasticity and catastrophic forgetting, with no good way to stop old data being forgotten.
— Charlie O'Neill / Beren MillidgeData explains 12x, architecture only 3.7x
Dwarkesh and Princeton's Jerry Han ran a grid experiment: pairing training recipes and datasets from 2019 to now in every combination, GPT-2 with the latest Ultra-FineWeb, Delphi with an old dataset like Pile, running the whole grid. The conclusion is that data gives roughly a 12.0x compute-efficiency gain, while architectural improvements give only about 3.7x, and at very small scale. Charlie O'Neill says the big chunk that is the internet has already been eaten, useful things won't keep growing at the same rate, and there's probably only a pile of 0.1% loss reductions left. He notes someone at Epoch estimated 3x per year since 2019, over 2000x cumulative, and the missing 100x probably comes from post-training.
— Dwarkesh Patel / Charlie O'NeillRL needs a curriculum; pretraining doesn't
Beren Millidge says this comes back to the RL signal problem: RL is currently bad at exploration, and if the model can't get it in 128 rollouts it basically gets no learning signal, so RL needs curricula while pretraining needs none at all. Charlie O'Neill adds: the pretraining signal is already in Common Crawl, the problem is just filtering out noise, which can be automated; but once you get to mid-training and post-training, the signal simply isn't in the raw data and no amount of filtering will pull it out — there isn't a proof of a Millennium Prize problem hiding in Common Crawl waiting to be filtered out. You have to get the bits elsewhere: have people write out reasoning traces, construct environments, or use human data from deployment.
— Beren Millidge / Charlie O'NeillMost of RL's success comes from mid-training
Beren Millidge says what's underrated is mid-training: pretraining on synthetic reasoning data and environments, warm-starting the model to where it has almost completed 80% of the path to the final RL checkpoint, with RL only fine-tuning the policy on top. So it doesn't need to learn these behaviours from scratch, and a few bits per episode are enough. He also stresses that the signal-to-noise ratio of those bits is far higher than ordinary pretraining — in SFT you have to match another model's reasoning process token by token, which pours in too many bits about ‘how it thinks’ and drowns the signal; RL's objective ignores all the other bits and keeps only the one bit of ‘did it get the answer right’.
— Beren MillidgeOne year or three, the difference is long-tail chores
Asked when you could hire a drop-in remote white-collar worker, Charlie O'Neill says if it has to operate a browser, probably a few years; if it can be given Slack and programmatic access, probably a year. Beren Millidge says full generality might be three years, but organisations will actively reshape themselves to be more AI-friendly, so 80-90% is reachable before then. The substance of the disagreement is the long tail: some chores humans can do will take the model a very long time. Charlie gives the example that a model won't ‘yell at people’ to push work forward — it's too nice. John Schulman points out that human remote workers already vary enormously in quality, and on Upwork it's often hard to get someone doing a software project to take feedback seriously; in some cases the pre-AI version is worse than today's AI.
— Charlie O'Neill / Beren Millidge / John SchulmanIn their own words · checked verbatim
There’s this cycle that keeps repeating where a new model comes out and people are blown away and they’re like, “This is it. This is AGI.” But then they use it a bit, and it starts to feel dumb after a month or so.
John Schulman0:00
We had this very nice straight line and that held for a really, really long time. But there were so many discrete discontinuities and innovations that had to happen to keep that scaling law going.
Charlie O'Neill0:00
Basically anything that can be learned through RL can be distilled very easily, because it’s a small number of bits.
John Schulman18:39
It’s so unfortunate that RSI happened to be easier than being a paralegal.
Dwarkesh Patel44:38
RL is good at getting capabilities in, but it’s not as good at getting knowledge in
Charlie O'Neill52:00
There’s no hidden proof of a Millennium Prize problem sitting in Common Crawl that we can just filter until we see it.
Beren Millidge59:46
An awful lot of what we see as successes of RL actually comes from very, very good mid-training data, which is where we’re essentially doing pre-training but on synthetic reasoning data and the kind of environments that get the model warm-started for RL.
Beren Millidge1:18:03
An example of something that it wouldn’t be good at is if I have to yell at someone to get something at work, or really push someone to get something done. The model just isn’t going to do that. It’s going to be too nice.
Charlie O'Neill1:28:32
Figures
| Charlie O'Neill's estimated speedup multiple for ‘maximising the current objective’ | 10x | 0:00 |
| Dwarkesh's hypothetical ‘no superintelligence’ year | 2036 | 0:00 |
| Compute-efficiency gain from data | about 12.0x | 59:46 |
| Compute-efficiency gain from architectural improvements | about 3.7x | 59:46 |
| Epoch's estimated annual compute-efficiency gain | 3x | 59:46 |
| Rollout threshold for RL exploration | 128 | 1:03:00 |
| Doubling period for how long a model can work continuously | doubles every three months (EdgeBench) | 1:18:03 |
| Progress toward the final RL checkpoint that mid-training brings the model | about 80% | 1:18:03 |
| Time prediction for 10x AI-researcher productivity | John Schulman and Beren Millidge: two years; Charlie O'Neill: 5-10 years | 1:28:32 |
| Time prediction for AI fully surpassing human experts | John Schulman: 3-4 years; Charlie O'Neill: 5-10 years; Beren Millidge: about 5 years | 1:28:32 |
Glossary
- RSI / recursive self-improvement
- A model can improve its own training or research process, thereby accelerating the arrival of the next generation of models.
- sim-to-real gap
- Performing well in a benchmark or simulated environment but failing once transferred to the real world.
- mid-training
- A round of training after pretraining and before RL, using synthetic reasoning data and environments.
- entropy collapse
- After RL training the model's output diversity drops sharply, repeatedly producing the same style.
- logit distillation
- Making the student model match the teacher's full output probability distribution, not just the final answer.
How to listen
Engineers and researchers tracking the boundaries of AI capability and training mechanics, especially anyone wanting to understand RL generalisation, the continual-learning bottleneck and what mid-training does.
The opening hypothetical about the 2036 scenario can be fast-forwarded; the mechanics section starts at 44:38.