Invoke extinction risk, propose slower releases: satisfies neither side
If AI truly carried extinction-level risk, the only self-consistent response would be nationalized control, not voluntary release pacing. The current pitch leaves both regulators and industry unsatisfied.
The video won't play here. Listen to the audio instead:
The argument · tap a timestamp to hear it
Invoke extinction risk, propose slower releases: satisfies neither side
The substance of Dario's post—better sandboxing, stricter testing—is considered sound. But the PR context drags it down. On the same day, the CEO tells television he doesn't dispute the ‘10% extinction probability’ framing, which makes what should be a practical safety proposal look like a defense of downplaying catastrophe. The word ‘pacing’ itself has nothing to do with safety—slow-making nuclear weapons wouldn't reassure anyone—it appears designed to simultaneously appease the internal ‘pause faction’ and external regulators. Instead, neither side buys it. Both assume the other is the one getting coddled.
If extinction risk is real, only nationalization makes sense
If the lab's leading experts truly believe the technology carries extinction-level risk, the only self-consistent response isn't voluntary release slowdown but nationalized control with nuclear-weapon-level approval and oversight systems. Alternatively: if private conversation shows most don't genuinely believe the probability, then public extinction-risk statements look like recruiting theater—language chosen to seem grave in the talent market, fundamentally a company HR issue. It shouldn't become the national policy justification that handicaps the entire industry.
— Martin CasadoCritical infrastructure inevitably trends toward state control
Post-WWII, critical-infrastructure sectors—power, banking, healthcare—gradually moved toward state or quasi-state oversight. Even AT&T and IBM, born with government contracts, couldn't escape antitrust litigation and structural dissolution. For AI, a FINRA-style hybrid model—industry-funded, federally directed—is considered the least-objectionable outcome available. Yet it still disadvantages open models and frontier innovation. It's the least-bad choice among bad options, not an ideal solution.
The early internet was more dangerous but never got shut down
Before 2001, any PC that connected to the internet was nearly certain to get infected. Worms crashed internet infrastructure; hospitals went down; losses ran to hundreds of billions. By early-1990s thinking, internet shutdown was entirely plausible. Instead, the industry layered in security measures reactive to real crises, not through foresight. This history illuminates AI governance's core tension: we're predicting ‘what might go wrong’ rather than legislating after it does—a mode regulation was never designed for.
Agent swarms expose enterprise systems in unprecedented ways
Historically, enterprise security relied on ‘95%–99% of people doing the right thing’. Internal systems like GitHub, Slack, expense software were rarely designed against DoS. Agent swarms, though, are thousands of tireless actors constantly attempting operations, misidentifying normal tasks as violations (and vice versa), with massive concurrent access that looks indistinguishable from DoS. Enterprises now need a monitoring layer that didn't exist—tracking every authentication, every API call. But current OS permission models either prompt the user continuously or grant full filesystem deletion rights. They weren't built for this level of granularity.
The real danger: Europe turns AI regulation into clickthrough theater
The real nightmare isn't fines but GDPR-style compliance theater: every agent ‘write’ operation or third-party service call pops a warning like ‘this vendor's output may be inaccurate; proceed?’. Regulators like this because it shifts accountability to users, creating an ‘I warned you’ legal chain—like UAC popups, Word macro warnings, cookie banners that everyone mindlessly clicks. America surrendered tech antitrust leadership fifteen years ago; Europe has nothing to lose, so Europe is more likely to codify clickthrough warnings as AI's regulatory standard.
Car and aviation regulation took decades to crystallize
‘Unsafe at Any Speed’ appeared sixty years after the Model T. The FAA gained real regulatory power forty years after the Wright Brothers—through aviation's most explosive innovation period. Thalidomide drove FDA tightening fifty to seventy-five years after the incident. A Nick Bostrom podcast episode captures the problem: if you regulate before understanding how a technology fails, you've solved nothing. You've just forced it into existence without learning to control it.
From text generation to selection: the next wave of model use
For years, people tried forcing text-generating models into traditional if-statement logic with awkward results. The new approach reverses that: models still read text and grasp context but skip expensive generation. Instead, they pick from given options and output a probability, feeding directly into probabilistic conditionals—reviving old research from the 1970s on probabilistic programming. It's the fastest-adopted model pattern since ChatGPT, which shows the real innovation is happening in the application layer, not in the models.
In their own words · checked verbatim
If you regulate AI too early, you actually don't solve anything. You still just kind of have the same risk ultimately. You will the thing into being, but you haven't figured out how to control it.
Agent swarms completely flip that. These are just roaming drones. But like 5,000 to 10,000 and they will easily mistake a good task for a bad one.
The US about 15 years ago stopped leading in tech antitrust. The problem is that Europe is going to lead with that because they have nothing to lose.
This could change the nature of software fundamentally. The center of innovation has just moved.
An employee has like 10% chance of species extinction. The post is very reasonable, but the atmospherics are not.
So if a constituency within the labs that are the most knowledgeable people believe this stuff has existential risk, the answer is to nationalize it and actually put controls that we know that work, right?
Martin Casado6:10
So, remember the old CRT? So, it turns out, like, let's say it's night and you're using a CRT terminal in your room. The lightest thing in the room is actually the pixel that the raster beam is on.
And so this has probably been the fastest adoption of an AI model since chat GPT. It's just been remarkable because we're all primed for this.
Figures
| Extinction risk probability in Dario's post | ~10% | 4:08 |
| Agent swarm scale discussed | 5,000–10,000 | 0:00 |
| Computer Fraud and Abuse Act signed | 1986 | 25:26 |
| Time from Morris Worm incident to CFAA passage | 2.5 years | 25:26 |
| Era when PC connection meant near-certain infection | Before 2001 | 24:25 |
Glossary
- pacing
- Anthropic's term for deliberately slowing release cadence without halting R&D
- X-risk / existential risk
- Risk of human extinction or civilization-scale collapse
- FINRA
- Hybrid self-regulatory model where industry funds operations and federal authority oversees
- TEMPEST attack
- Exploiting electromagnetic radiation leakage to remotely reconstruct screen or keyboard data
- covert channel
- Hidden communication pathway using physical side effects like heat, electromagnetic emission, or pixel brightness
How to listen
Founders, investors, and security engineers tracking AI policy and enterprise safety who want to understand the reasoning behind regulatory debates rather than just reading headlines.
8:12–18:19: Repeated semantic debate over ‘pacing’ circles back on itself; the core argument is clear earlier in the episode.