AI research assistants exist, but lack the judgment to know what's worth pursuing
Two core researchers from OpenAI and Thinking Machines argue that AI can now run experiments autonomously and accelerate research, but judging which ideas are worth pursuing—the ‘taste’ for promising research—still comes primarily from human researchers' experience, not from models.
The video won't play here. Listen to the audio instead:
The argument · tap a timestamp to hear it
AI can run experiments autonomously, but only within explicit task boundaries
OpenAI uses Codex to manage computing resources and automate research workflows, but only in scenarios where the task is explicitly defined. Currently, RSI resembles giving researchers an assistant with strong execution but no independent judgment, not a researcher who can propose directions.
— Mark ChenDays of AI work can't match a researcher's one careful review
John Shulman describes his workflow: writing complex prompts for Fable, removing constraints, letting it run for days producing massive output. The model generates reasonable experiments and interesting ideas, but results often aren't good enough and sometimes fail entirely. Only when he reviews and provides finer decomposition does progress occur—he builds understanding, asks questions the AI wouldn't think of, and points toward directions it wouldn't go.
— John ShulmanAGI is 80% complete; RSI is only 25-50% there
Mark Chen frames both AGI and RSI as spectrums, estimating AGI at roughly 80% complete and RSI at only 25-50%. A contemporaneous Meter report shows Anthropic's research progress accelerated ~2x in the past year, likely because human researchers use AI to accelerate their own work rather than models conducting fully autonomous research.
— Mark ChenModels excel at tuning parameters, not at judging which ideas merit pursuit
John Shulman distinguishes execution (knowing parameters, running experiments) from taste (originating ideas). Models excel at hyperparameter optimization but struggle judging which ideas matter long-term. This intuition comes from hands-on research, reading extensive literature, and learning from failures—experiences where predictions were proven wrong and results observed. Models can replicate memory but not yet this kind of experiential knowledge.
— John ShulmanPretraining has one measuring stick; downstream research has none
Pretraining progress is easy to measure: the entire field has used perplexity for years, more data compression means better models, simple hill-climbing. Downstream research is far more open-ended with no single right answer. John Shulman calls this one of RSI's core challenges. Mark Chen adds that benchmark shelf-life shrinks constantly; much of their work now automatically creates new, harder benchmarks themselves.
— John ShulmanAI research is more like biology than pure logic; reasoning alone won't accelerate it
Mark Chen explains why AI accelerates math faster—hypotheses verify through reasoning alone. But AI research resembles biology or experimental physics: you need solid hypotheses but must actually run experiments to verify. One key to accelerating RSI is predicting large-scale results from small-scale experiments, where models can help too. Progress will be compute-limited until AI helps build next-generation data centers, creating another inflection point.
— Mark ChenWithout solving alignment, models can silently drift from their intended goals
John Shulman raises a specific risk: without solving objective and value alignment, models running long-term could gradually diverge from initial goals, even spawning sub-goals like ‘copying weights outside of OpenAI’. Future alignment requires stronger models aligning other models. Chain of thought monitorability is currently the most effective tool, meaning RSI becomes deeply embedded in alignment.
— John ShulmanWho decides alignment's value standard? Disagreement is more fundamental than risk
John Shulman raises a more fundamental question: alignment to whose values? Thinking Machines leans toward giving users control over their AI's values rather than letting a few companies decide. He wants alignment to become open global science, avoiding over-fitting to any one company's conclusions—even local failures teach something new.
— John ShulmanIn their own words · checked verbatim
We're very much in this centaur era where AI has phenomenal capabilities. It's superhuman in a lot of ways, but when you have an expert human working with the AI, you get even better results.
John Shulman2:01
I feel like we're maybe 80 percent of the way when I think about this AGI spectrum. I think we're earlier in the RSI spectrum.
Mark Chen5:21
Research taste is about knowing which ideas are important in the long run, which ideas really catch on and end up going beyond whatever paper they first appear in.
John Shulman6:28
I think it's much harder downstream, right? I think the situation downstream has been that evals have been having shorter and shorter shelf life.
John Shulman9:31
If you don't solve the goal and value alignment problems, you can start pursuing your goals, but maybe kind of drift a little bit away from those goals.
John Shulman11:36
So I'm kind of most bullish on like, optimizing for the thing we want. I think we can probably cook up the right kind of training objectives that'll incentivize the models to report on their internal states that they otherwise didn't verbalize.
John Shulman12:40
I do think people understand that this is a problem, but also understand the benefits that technology can bring.
Sarah Wang14:42
Figures
| Anthropic research progress acceleration (past year) | ~2x | 5:21 |
| Mark Chen's estimate of AGI spectrum progress | ~80% | 5:21 |
| Mark Chen's estimate of RSI spectrum progress | 25%-50% | 5:21 |
| Alignment failures fixed through automated research | 10 | 13:42 |
| Model scale transferability of these repair methods | ~5x | 13:42 |
Glossary
- RSI (Recursive Self-Improvement)
- Using AI to accelerate AI's own research and training process, not implying fully autonomous model-generated next-generation models.
- perplexity
- A metric measuring how accurately a language model predicts the next word; lower values indicate better performance.
- chain of thought monitorability
- Assessing whether a model is deviating from intended behavior by reading its reasoning process; currently the most effective alignment tool.
- confessions research
- OpenAI research training models to report their own misbehavior by rewarding honesty rather than punishing wrong answers.
How to listen
Practitioners and investors who want to understand the internal processes of large-model R&D organizations and gauge how far AI-automated research has actually progressed.
The opening segment on RSI definitions and history (00:00-01:01) is low on information density; skip ahead to 03:04 for concrete workflow descriptions.