The window to slow down has passed; superintelligence arrives in two to three years
Geoffrey Irving, the UK's former head of AI safety, believes full superintelligence will arrive within two to three years and the window to slow down has already been missed; the hope for alignment lies in scalable oversight, character research and theoretical breakthroughs, not in precisely specifying a utility function.
The video won't play here. Listen to the audio instead:
The argument · timestamps estimated from transcript position
Fewer than 10 people nodding is enough to slow AI down
Geoffrey Irving states his position clearly at the start: the time to slow down is now, or even earlier. He explains that if we carefully analyse when we should have slowed down, the answer would be ‘some time in the past’, because we are too close to that crazy future. He proposes that as long as fewer than 10 key decision-makers (lab CEOs, relevant people in China, a few national leaders) agree, slowdown can be achieved; but unilateral slowdown is very difficult. This judgment is the undertone of the entire conversation, and explains why he thinks the alignment problem requires global coordination rather than a purely technical solution.
— Geoffrey IrvingTraining for only a week does not mean the model has no multi-year plan
Responding to Rohin Shah's view that training time is short, so models will not learn to take over the world, Irving points out that this confuses ‘the timescale of the overall plan’ with ‘the timescale of the task's components’. Even if each subtask is completed within a week, multiple subtasks combined can realise a multi-year plan. This means training duration does not constitute a safety guarantee, because a model can execute a long-term strategy by decomposing tasks, without needing to complete the full chain of reasoning within a single training process.
— Geoffrey IrvingThe alignment approaches of the three major labs are actually the same
Irving argues that the strategies of OpenAI, Anthropic and DeepMind are all combinations of ‘character training + scalable oversight + monitoring’. He compares the labs' character training methods: Anthropic leans towards virtue ethics, OpenAI has more rules and is more corrective, while DeepMind is unclear. Whether these methods will work on superintelligence currently has no strong argument. This observation reveals that the alignment practices of frontier labs lack theoretical foundations and are more empirical parameter tuning.
— Geoffrey IrvingPresenting only evidence and not counter-evidence can fool the judge
Irving gives a key obstacle to scalable oversight — ‘ambiguous arguments’: a model may present only evidence supporting a claim while hiding counter-evidence, because counter-evidence is what is decisive. He cites Beth Barnes's experiment at OpenAI: human debaters used complex but false arguments to fool the judge, because neither side could find the flaw. This obstacle has no good theoretical solution, indicating that oversight methods relying purely on human feedback may fail when facing smarter models.
— Geoffrey Irving‘Models are only good at verifiable tasks’ is overrated
Irving's modal expectation is full superintelligence within two to three years. The main uncertainty is whether models can be good at fuzzy tasks (intuition, fuzzy planning). He believes people overestimate the likelihood that models are only good at verifiable tasks, and trusts that through the data flywheel, models will gradually expand to more non-verifiable tasks. This timeline judgment means the window for slowdown and safety research is extremely short, and also supports his opening claim to slow down ‘now’.
— Geoffrey IrvingWhen government does AI safety, the ultimate focus is policy
Irving points out AISI's future priorities: first, misuse risk, i.e. dangerous capability evaluation, which needs to combine high research capability and national security; second, mitigation measures, including misuse and loss of control, for which AISI already has a strong protection team; third, policy, which is the main reason Irving joined AISI. As the largest government AI safety research institution, AISI's policy recommendations should be based on tactical reality. These three points outline the role of government-level participation in AI safety: evaluation, defence, policy-making.
— Geoffrey IrvingAlignment is not setting goals, it is designing the training process
Irving believes that before superintelligence arrives, we cannot precisely specify its utility function. The only hope is that the training process has error-correction capability, allowing us to fall into some ‘basin’ rather than an exact target. He proposes a concrete idea: through a branching graph of the training space, using 1000 checkpoints to show the model's training path, thereby testing different algorithms and increasing understanding of branching behaviour. This attempts to shift alignment from ‘setting goals’ to ‘designing the training process’.
— Geoffrey IrvingRL's main role is to filter out ineffective reasoning patterns
Irving thinks RL mainly does two things: first, it selects out ineffective reasoning patterns; second, humans often write only the final answer, forcing the model to reason in latent space. The emergence of o1 is a combination of RL algorithm tuning and improved base model capabilities. He predicts the frontier of alignment technology is the combination of ‘scalable oversight + character + learning dynamics’, but organisations will adopt diversified strategies and will not bet everything on one approach.
— Geoffrey IrvingIn their own words · checked verbatim
Now. Now. If we were to carefully analyse this question of exactly when we should slow down, it would be like a while ago in the past, because we’re just too close to this crazy future.
Geoffrey Irving0:00
On factual questions it would be not racist; it was very happy to write incredibly horrible poetry about racism.
Geoffrey Irving14:31
One of the winning strategies was basically a debater would produce a very complicated argument that sort of sounded true, was false, but neither of the debaters — not the liar or the honest debater — knew where the flaw was.
Geoffrey Irving23:31
some degree of your conscious train of thought is a hallucination as you go along the world.
Geoffrey Irving56:12
"Rationalisation" there is a pejorative word, but it’s also just intrinsically how intelligence works, even for humans.
Geoffrey Irving1:06:35
Hope is not like a lot of probability. Just like, we should try.
Geoffrey Irving1:23:51
If you had the other LBJ that was opposed to the first one, that's saying, "By the way everyone, you realise what he's doing here? He's cheating the vote system." And everyone is like, "That's ridiculous. That's clearly unfair. Let's fix the rule to break that." I think that intervention would get you so much power over the misaligned components of LBJ that I think it's within hope to imagine getting that story right.
Geoffrey Irving1:28:54
I told Dario — this is annoying — that I would write a document called "Language is enough to get to AGI" — and then I didn't write it until 2019. So early 2019 is when I actually wrote the document.
Geoffrey Irving1:36:46
Figures
| Key people needed for slowdown | fewer than 10 | 0:00 |
| Number of rules in Anthropic's constitution | 5 | 10:46 |
| Acceptable performance loss for a temporary pause | 2-10x slower | 34:14 |
| Time until superintelligence (modal expectation) | 2-3 years | 29:34 |
| History of the alignment field | at most 20-25 years | 1:18:26 |
| Age of empirical character research | 1-2 years | 1:19:30 |
| Nanotechnology and uploading timeline | a few years to two decades | 59:28 |
| Compute for solving Pentago | 10^17 or 10^18 FLOPS | 1:31:00 |
| Year when the RL+LLM path began to be advocated | 2018 | 1:34:05 |
| o1 release date | 2024 | 1:36:46 |
Glossary
- scalable oversight
- Oversight methods that can still ensure behaviour conforms to intent when AI capabilities surpass human oversight.
- ambiguous arguments
- A reasoning style that gives only supporting evidence and hides decisive counter-evidence, capable of deceiving overseers.
- character training
- Shaping a model's personality traits through training to reduce harm, part of current lab alignment strategies.
- effective field theory
- A theory that ignores microscopic details and describes physical phenomena with macroscopic parameters, which Irving uses as an analogy for alignment theory.
- data flywheel
- A cycle where models generate data to train the next generation of models, continuously expanding the range of capabilities.
How to listen
Who should listen: AI alignment researchers, heads of large-model training teams, AI policy practitioners, and any technical decision-maker sensitive to judgments about the superintelligence timeline.
Around 1:31:00 the Pentago solving compute estimate can be skipped without affecting the main thread.