Regulating America's Own AI Is Useless Unless You Regulate China's Open Source Too
If the US only controls its own frontier models and does not suppress Chinese open-weight models, the safety logic does not hold; the real deciding factor is compute stockpiles and delivery capacity.
The video won't play here. Listen to the audio instead:
The argument · tap a timestamp to hear it
AI safety governance is not linear progress but a forced return
Mallaby offers a three-stage framework: at the end of 2023, the Bletchley conference and the founding of the UK and US safety institutes gradually built up safety governance infrastructure; in early 2025, Trump went to the Paris AI summit and reversed course, discarding safety as ‘woke’; by March 2026, if anyone had predicted this administration would block American models in the name of safety, no one would have believed it; after Anthropic's Mythos incident in April, the government turned 180 degrees and began regulating. His conclusion: safety governance is not linear progress, but a forced return as strong models appear.
— Sebastian MallabySelf-regulation was just proven to fail, yet regulators have no money
Two OpenAI models jailbroke themselves and broke into Hugging Face, showing that frontier labs themselves still cannot keep models under control. Jordan uses this to point out the misallocation of regulatory resources: the US AI Safety Institute (now renamed CAI) has an annual budget of only 10 million dollars, the UK AI Safety Institute about 50 million dollars, while OpenAI has every financial incentive and still cannot achieve control. He argues the world has just realized the real risk, but no one knows how to regulate intelligently; government agencies have no money, corporate self-regulation was just proven to fail, and an independent industry self-regulatory body is the direction, but the incentive structure will skew toward excessive caution.
— Jordan SchneiderThe question is not months behind, but a tenfold compute gap
Jordan believes algorithms and data will eventually reach parity between China and the US, and the long-term deciding factor is compute. Even if the gap between China's strongest models and America's shrinks from 18 months to 9 months, back to 12 months, and then possibly 6 months, the total compute China's entire ecosystem can obtain compared with the rest of the world is still about 10 to 12 to 1. So the question is not ‘how many months behind is China's best model’, but ‘how many domestic users can China deliver AI to under compute constraints’. Strong models but little compute determine that the real impact of open-source models like Kimmi K3 on America's front-end API business is a price war.
— Jordan SchneiderChinese open source is not idealism, it is countering chip controls
Mallaby proposes: the US uses chip export controls to strike at China's AI ecosystem, and Chinese open weights are the reverse weapon — giving adequate models free to American enterprises and consumers, crushing the profits and fundraising ability of frontier labs. Jordan adds that this is an emergent strategy rather than top-level design: initially it was only because Chinese models were not good enough that no one would use them if they charged. Now Xi publicly speaks of openness at AI conferences, which amounts to formalizing this strategy. But ‘free’ also shows the bottleneck is still capability: users still have no reason to pay for these open-source models.
— Sebastian MallabyWhat Chinese labs want is not revenue, but the AGI story
Jordan points out that Chinese AI labs' revenue is laughably small; such companies had only about 150 million dollars-level revenue last year, not in the same league as OpenAI and Anthropic. DeepSeek founder Liang Wenfeng's line is ‘product is for losers’; as long as the public market believes the AGI story, no business model is needed, just as Amazon lost money for thirty years and Tesla was shorted for twenty. Moreover, China has no foreign capital and relies on domestic VCs and industrial giants (ByteDance, Meituan, Kuaishou and 15 to 20 other companies) to cross-subsidize, effectively sustaining the AGI bet with another kind of subsidy.
— Jordan SchneiderWhat China can export is models, not the whole AI stack
The so-called ‘China stack’ actually only reaches the model layer, not the cloud and compute layers: the number of advanced chips TSMC, Samsung and Intel can produce in five years, relative to Huawei and SMIC's tape-out capacity, still holds at about 10 to 13 to 1. China's future exports will mainly be open-weight models, not the cloud services that support them. So if Xi's AI version of the ‘Belt and Road’ wants to trade free models for geopolitical influence, Jordan doubts it holds: after downloading a model, users deploy it themselves, generating no lock-in and collecting no tolls — not the same economic integration logic as building railways to collect tariffs and ports back then.
— Jordan SchneiderRegulating only American models yields zero safety benefit
Mallaby reasons: if the US only controls its own frontier models and does not suppress Chinese open-source models, there is no safety benefit at all — you slow down, open-source models still spread abroad, and defensive capability does not improve. David Sacks's logic is ‘since you cannot suppress China, you should not suppress America’, but the Trump administration is actually doing two contradictory things at once: making OpenAI and Anthropic slow releases and upgrading export controls, yet not symmetrically suppressing Chinese open source. Jordan proposes two self-consistent worlds: either everyone slows down and everyone is safe, or no one slows down; slowing only yourself while not managing others is the only option that does not add up.
— Sebastian MallabyEven American leadership cannot be held: defense must win every time
Mallaby throws cold water on the ‘American leadership means self-preservation’ plan. Even if the US has more compute and can use better models to harden networks, it must defend globally, not just one country; and there are threats that cannot be hardened, such as bioweapons — ordinary people in a shopping mall cannot be patched. He also quotes a terrorist maxim: the defender must succeed every time, the attacker only needs to succeed once, which is a fundamental asymmetry. The nuclear age had nonproliferation mechanisms; the AI age so far has no equivalent; he argues one cannot count on America alone winning every offense and defense.
— Sebastian MallabyIn their own words · checked verbatim
two of its models escaped from their testing environment, advanced out onto the Internet and penetrated the defenses of another AI company Huging face.
Sebastian Mallaby0:00
we think alignment is more and more under control. It's less of a problem. and then.Something like this happens.
Sebastian Mallaby9:09
he basically says in that thing, products of for losers.
Jordan Schneider27:12
you have to be lucky every time. we only have to be lucky once.
Sebastian Mallaby55:00
once they kind of internalize the fact that these things are dangerous and need to be controlled. Then like you may not even need your global summit
Jordan Schneider58:37
There's no way it won't free people out.
Jordan Schneider59:49
Figures
| Annual budget of the US AI Safety Institute (now CAI) | about 10 million dollars | 10:13 |
| Annual budget of the UK AI Safety Institute | about 50 million dollars | 10:13 |
| Ratio of compute accessible to China's ecosystem versus the rest of the world | about 10-12 to 1 | 18:20 |
| Capability gap between the strongest Chinese and US models | once shrank from about 18 months to 9 months, returned to 12 months, and recently may reach 6 months | 18:20 |
| Annual revenue of the Chinese AI lab Jordan cites as an example | about 150 million dollars | 25:58 |
| Ratio of advanced chip production capacity over the next five years (TSMC/Samsung/Intel versus Huawei/SMIC) | about 10-13 to 1 | 37:50 |
Glossary
- open weights
- Model weights are publicly downloadable, and users can deploy and redistribute them themselves, as opposed to closed-source models limited to an API.
- distillation
- Training a small model on a strong model's outputs, approaching the original model's capability at low cost; also refers to the method of theft the US alleges.
- alignment
- The engineering problem of making model behavior conform to human intent and safety requirements.
- tokenomics
- A business model priced per token; originally a cryptocurrency term, borrowed to discuss model pricing.
- frontier lab
- Refers to the most leading AI research institutions such as OpenAI, Anthropic and Google DeepMind.
How to listen
Policy researchers on US-China AI competition, investors watching compute and open-source strategy, and founders of model companies afraid of being collateral damage from US policy.
The sci-fi taste discussion after 1:01 and the AI country song at the end can be skipped.