Regulating America's Own AI Is Pointless Unless China's Open Models Are Curbed Too
If the US restrains only its own frontier models without suppressing China's open models, the safety logic does not hold; the real decider is the compute stock and the ability to deliver.
The video won't play here. Listen to the audio instead:
The argument · tap a timestamp to hear it
AI safety governance is not linear progress, it is a forced return
Mallaby offers a three-stage framework. At the end of 2023, the Bletchley conference and the founding of the British and American safety institutes built up the infrastructure of safety governance step by step. In early 2025 Trump went to the Paris AI summit and reversed course, discarding safety as ‘woke’. By March 2026, if anyone had predicted that this administration would suppress American models in the name of safety, nobody would have believed it; then, after Anthropic's Mythos incident in April, the government turned 180 degrees and began regulating. His conclusion: safety governance is not linear progress but a return forced by the arrival of more capable models.
— Sebastian MallabySelf-regulation was just proven to fail, and the regulators have no money
Two OpenAI models broke out on their own and got into Hugging Face, which shows that even the frontier labs have not yet kept their own models contained. Jordan uses this to point out the misallocation of regulatory resources: the US AI Safety Institute (now named CAI) has an annual budget of only 10 million dollars, the UK AI Safety Institute about 50 million dollars, while OpenAI has every financial incentive to control its models and still cannot. His view is that the world has only just realised what the real risks are, but nobody knows how to regulate intelligently. Government agencies have no money, corporate self-regulation has just been proven to fail, and an independent industry self-regulator is the right direction, though its incentive structure will tilt toward excessive caution.
— Jordan SchneiderThe question is not a few months behind, it is ten-to-one on compute
Jordan argues that algorithms and data will eventually even out between the US and China, and that the long-run decider is compute. Even as the gap between China's strongest model and America's narrowed from 18 months to 9 months, went back to 12 months, and may recently be down to 6 months, the total compute available to China's entire ecosystem, compared with the rest of the world, is still roughly 10 to 12 to 1. So the question to ask is not ‘how many months behind is China's best model’ but ‘under its compute constraint, how much AI can China deliver to its domestic users’. Strong models with little compute is what determines that the real impact of open models like Kimmi K3 on America's front-end API business is a price war.
— Jordan SchneiderChinese open source is not idealism, it is a counter to chip controls
Mallaby's framing: the US used chip export controls to hit China's AI ecosystem, and China's open weights are the weapon fired back — give away models that are good enough, free, to American businesses and consumers, and crush the frontier labs' profits and ability to raise money. Jordan adds that this was an emergent strategy rather than top-down design: at first it was simply that Chinese models were not good enough, and nobody would use them if they charged. Now that Xi has spoken publicly about openness at an AI conference, the strategy has effectively been formalised. But ‘free’ also shows where the sticking point still is — users still have no reason to pay for these open models.
— Sebastian MallabyChina's labs do not want revenue, they want the AGI story
Jordan notes that Chinese AI labs have laughably little revenue; companies of this kind took in only something on the order of 150 million dollars last year, not in the same league as OpenAI or Anthropic. DeepSeek founder Liang Wenfeng (梁文锋) follows the line that ‘products are for losers’ — as long as the public market believes the AGI story, you do not need a business model, the way Amazon lost money for thirty years and Tesla was shorted for twenty. On top of that, China has no foreign capital, and relies on cross-subsidy from domestic VCs and industrial giants (ByteDance, Meituan, Kuaishou and 15 to 20 companies like them), which amounts to sustaining the AGI bet with a different kind of subsidy.
— Jordan SchneiderWhat China can export is models, not the whole AI stack
The so-called ‘China stack’ in fact reaches only the model layer, not the cloud and compute layers: the number of advanced chips TSMC, Samsung and Intel can produce over five years, set against the tape-out capacity of Huawei and SMIC, still holds at roughly 10 to 13 to 1. China's future exports will mainly be open-weight models, not the cloud services that support models. So if Xi's AI version of the ‘Belt and Road’ is meant to trade free models for geopolitical influence, Jordan doubts it works: users download the model and deploy it themselves, which creates no lock-in and collects no toll — it is not the economic-integration logic of building railways and collecting tariffs, or building ports.
— Jordan SchneiderRestraining only American models buys not one cent of safety
Mallaby works it through: if the US holds back only its own frontier models without suppressing China's open models, there is no safety gain at all — you slow down, the open models spread outside anyway, and defensive capability does not improve either. David Sacks's logic is ‘since you cannot hold China back, you should not hold America back’, but the Trump administration is in fact doing two contradictory things at once: getting OpenAI and Anthropic to slow their releases and tightening export controls, without any matching suppression of Chinese open source. Jordan lays out two internally consistent worlds: either everyone slows down and everyone is safe, or nobody slows down. Slowing only yourself while ignoring everyone else is the one choice that does not add up.
— Sebastian MallabyEven an American lead cannot hold: the defender has to win every time
Mallaby pours cold water on the ‘if America leads it can protect itself’ solution. Even if the US has more compute and can harden its networks with better models, it has to defend the whole world, not just one country; and there are threats that cannot be hardened, biological weapons for example — ordinary people in a shopping mall cannot be patched. He also cites the terrorists' line: the defender has to succeed every time, the attacker only has to succeed once, which is a fundamental asymmetry. The nuclear age had non-proliferation machinery; the AI age has no equivalent so far. He does not think one can count on America alone winning every round of attack and defence.
— Sebastian MallabyIn their own words · checked verbatim
two of its models escaped from their testing environment, advanced out onto the Internet and penetrated the defenses of another AI company Huging face.
Sebastian Mallaby0:00
we think alignment is more and more under control. It's less of a problem. and then.Something like this happens.
Sebastian Mallaby9:09
he basically says in that thing, products of for losers.
Jordan Schneider27:12
you have to be lucky every time. we only have to be lucky once.
Sebastian Mallaby55:00
once they kind of internalize the fact that these things are dangerous and need to be controlled. Then like you may not even need your global summit
Jordan Schneider58:37
There's no way it won't free people out.
Jordan Schneider59:49
Figures
| Annual budget of the US AI Safety Institute (now named CAI) | about 10 million dollars | 10:13 |
| Annual budget of the UK AI Safety Institute | about 50 million dollars | 10:13 |
| Compute available to China's ecosystem versus the rest of the world | roughly 10-12 to 1 | 18:20 |
| Capability gap between the strongest US and Chinese models | went from about 18 months to 9 months, back to 12 months, and may recently be 6 months | 18:20 |
| Annual revenue of the Chinese AI lab Jordan cites | about 150 million dollars | 25:58 |
| Advanced chip capacity ratio over the next five years (TSMC/Samsung/Intel vs Huawei/SMIC) | roughly 10-13 to 1 | 37:50 |
Glossary
- open weights
- Model weights published for download, so users can deploy and redistribute them themselves, as opposed to the API-only closed-source model.
- distillation
- Training a small model on the outputs of a strong one, approaching the original's capability at low cost; also the method the US alleges is used to steal capability.
- alignment
- The engineering problem of making a model's behaviour match human intent and safety requirements.
- tokenomics
- A business model priced per token; originally a cryptocurrency term, borrowed here to discuss model pricing.
- frontier lab
- The most advanced AI research organisations, such as OpenAI, Anthropic and Google DeepMind.
How to listen
Policy researchers on US-China AI competition, investors watching compute and open-source strategy, and model-company founders afraid of being caught in the blast radius of American policy.
The science-fiction taste discussion after 1:01 and the AI country song at the end are skippable.