AI's next breakthrough isn't training — it's letting AI learn on the job
The labs are betting RLVR will generalize into general intelligence, but the real bottlenecks are sample efficiency and continual learning. OPSD and "dreaming" may be the keys that let AI learn from deployment rather than from pre-training alone.
The video won't play here. Listen to the audio instead:
The argument · timestamps estimated from transcript position
The labs are betting that scaled-up RLVR alone will build AGI
The labs are betting that if you train AI to complete millions of verifiable tasks across thousands of diverse RL environments, you can build AGI. That kind of training, the bet goes, gives rise to general problem-solving ability — the capacity to push an open-ended task forward for weeks in the face of errors and ambiguity. Optimists believe the defects, such as data inefficiency and the absence of continual learning, can simply be steamrolled by training at scale, the way "fundamental" problems in NLP were drowned by the flood of compute. A model's training sample efficiency is only one-millionth that of a human, but the training cost is a one-time expense, amortized across billions of user sessions. What matters is the model's intelligence, generality and sample efficiency inside a session — and that is precisely what RL training is improving.
— Dwarkesh PatelGrindability, not verifiability, is what decides where models can improve
Progress on computer use has been slow, and the reason is not only verifiability but grindability — whether you can run a large number of rollouts in parallel against a deterministic, replayable simulator. With coding you can create a repository environment and have a thousand agents try things in parallel; with computer use you cannot have a thousand agents testing Amazon's checkout flow at the same time, because you would be detected and banned. Cloning applications like Slack and Gmail is feasible, but it is labor-intensive and does not scale. What this reveals is that unless you can build a replayable training target for a domain, models struggle to get better at it. And many of the skills AGI requires — running a business, litigating a case, trading stocks — cannot be replayed inside a data center: verification takes months or years of real-world action, and the conditions cannot be perturbed in parallel.
— Dwarkesh PatelWhether RLVR generalizes across every domain is still an open empirical question
The labs are betting that RLVR generalizes to all domains: train in enough containerized, reproducible environments and you end up with a general agent that can make plans, learn quickly and acquire new skills. Drop an AI with unlimited RLVR into Texas politics in 1948 and it might understand how to win a Senate seat better than LBJ did; hand the same AI a hundred million dollars in 2002 and it could build SpaceX. But this is an open empirical question. Dario mentioned on the podcast that training on short contexts does not necessarily generalize to long-context performance, which suggests RLVR generalization is not unboundedly strong. And if a model cannot generalize from short to long, how would it generalize from training on a mass of white-collar tasks to building something from nothing in the real world?
— Dwarkesh PatelContinual learning has to write learning back into the weights, not context
Continual learning has to go back into the weights. An AI cannot learn purely through an ever-growing KV cache: that does not scale, and it is not how humans learn. Human learning involves compression, which is what helps generalization and insight. But the moment you move into the weights, you give up the sample efficiency of in-context learning, because gradient updates are extremely sample-inefficient. The online-learning models that have successfully shipped, such as Cursor Tab, need 400 million or more requests a day predicting the same target. Yet the reason continual learning matters is that the world is complex: every job, every company, every problem is different, and a model needs to learn information specific to its particular deployment — information that cannot be stuffed into shared training. Sample efficiency and continual learning are deeply linked problems: on the job there is little data to work with, so you need sample efficiency, and in-context learning's "fast weights" memory scales poorly, so you need architectural innovation.
— Dwarkesh PatelDreaming could become the fourth scaling axis after inference-time compute
"Dreaming" is the more speculative approach: if an AI could build a simulator of reality in which to rehearse new skills and try alternative strategies, it could experience orders of magnitude more simulated samples within the same wall-clock time. EfficientZero can beat a novice human within 2 hours, because it plays dozens of simulated games in its head. Future LLMs might consume far less real data while practicing endlessly in environments they construct themselves. But simulating an entire world is much harder than simulating Go, which is why this remains the more speculative path. If it works, it becomes a fourth scaling axis alongside pre-training, RL and inference-time compute. The model spends compute writing RL environments and rehearsing the skills it will need in production, rather than merely compressing context.
— Dwarkesh PatelBy late 2027, AI improves through deployment rather than pre-release training
By the end of 2027, RLVR training will have produced agents that can orient themselves inside an unfamiliar problem, try strategies and iterate. The effective context may extend to a week: the AI works alongside you for a week, and at the weekend you give it a thumbs up or a thumbs down. If you give it a thumbs up, the base model distills what the session learned, via OPSD, dreaming or some other technique. The AI can improve in adjacent domains, and gradually widen the range of what it can do. The main way AI gets better is no longer the training that happens before release, but the experience accumulated across broad deployment. With every interaction the AI becomes smarter — not just because of your own past sessions, but because of the interactions of every user worldwide. That prospect is both frightening and exciting, and utterly unlike the way AI improves today.
— Dwarkesh PatelIn their own words · checked verbatim
if we train AIs to accomplish millions of verifiable tasks across thousands of diverse RL environments, then we’ll basically have built AGI.
Dwarkesh Patel0:00
it is not enough for a domain to be verifiable. It also has to be very grindable
Dwarkesh Patel2:12
There’s the context length you train at and there’s a context length that you serve at. If you train at a small context length and then try to serve at a long context length, maybe you get these degradations.
Dwarkesh Patel5:41
We don’t have some separation between parameters and activations. And there’s not some lump of these fast-weight representations that juts out further and further from our skull as we learn more things throughout our lifetime.
Dwarkesh Patel8:09
If the AI can build a good simulation of reality against which to rehearse new skills, or try alternative strategies and reinforce what works, then it could experience orders of magnitude more simulated samples in the same wall clock time.
Dwarkesh Patel13:16
Every time you interact with AI, it’ll be smarter. Not only because it has been learning from all your previous sessions, but also from all its interactions with all the other users in the world.
Dwarkesh Patel15:00
Figures
| Model training sample efficiency compared with humans | one-millionth | 0:00 |
| Share of lab compute going to inference | 30-50% | 8:09 |
| Bits stored per token when training Llama 3 70B | 0.075 bits | 16:54 |
| Gap between in-context learning and pre-training in information stored | 35 million times | 16:54 |
Glossary
- RLVR (reinforcement learning with verifiable rewards)
- A training method that applies reinforcement learning using reward signals that can be verified.
- OPSD (on-policy self-distillation)
- A technique in which the base model imitates the predictions of the post-long-session model, distilling what was learned.
- Dreaming
- An AI building simulated environments in which to train itself, in order to raise sample efficiency.
- KV cache
- The cache in a Transformer that stores context information, growing with the number of tokens.
- Sample efficiency
- A model's ability to learn from a small amount of data.
How to listen
AI researchers, technical decision-makers, and founders and investors watching AI's future, especially anyone interested in training paradigms and continual learning.
The sponsor segment (16:54-17:56) is skippable.