The AI safety coalition suddenly took shape, but nobody knows how to measure "the frontier"
The Hugging Face incident brought the left and the AI safety camp together for the first time, but "pacing the frontier" itself can't be measured — with no legislation, third-party evaluation is just voluntary goodwill.
The video won't play here. Listen to the audio instead:
The argument · tap a timestamp to hear it
Anti-data-center and AI safety have merged
Jasmine's timeline: the early-August pacing open letter, then the Hugging Face incident right after, two lines colliding. Before this, anti-data-center, anti-AI populist sentiment was diffuse, and the AI safety crowd kept agonizing over whether to talk to people like Tucker Carlson and Bernie — worried the other side wasn't "truth-seeking." In the end the left dropped its suspicion of AI safety first, and Bernie did more than anyone to convince the public that "AI is real, AI risk is real," not corporate marketing. So when Hugging Face hit, the public was ready to care about real frontier safety issues.
— JasmineThe public doesn't care about extinction, it cares about companies out of control
Jasmine's read: nobody votes on AI issues, it doesn't even make the top five. But "these companies are too lawless, power is too concentrated, data centers need regulation, we need transparency, we need government audits" is a very widespread intuition. Ask the public whether they want third-party evaluators, whether they want to fund Casey, and nobody has an opinion; but say AI has gone off the rails and the FBI should investigate, and everyone nods. Nathan added: the Hugging Face incident was narratively too clean — a New York Post cover with the Terminator wouldn't even be a stretch, the rhyme of the facts lined up exactly with a fear ordinary people can understand.
— JasmineAlignment is temporary, cybersecurity is permanent
Nathan splits the problem in two: on differentiated frontier capabilities there really is an alignment responsibility, but a hole punched by a frontier lab will be reproducible by three models in three to six months, and by twenty-plus organizations in a year. So the alignment problem only "temporarily" belongs to frontier labs, while the cybersecurity problem won't go away on its own until it's actually solved. He also says the security crowd knows full well: our infrastructure isn't ready for the tools that are about to be everywhere.
— NathanThe word "pacing" is itself self-serving
Jordan points out pacing is neither pause nor slowdown; choosing that word already implies still moving forward. Nathan is more direct: we simply don't know how to measure frontier models, and any pacing presupposes you can measure. What really bothers him is the in-lab rhetoric — "you can't see it, we're scared of the pace internally too" — which is meaningless without transparency. The mechanism he imagines: each lab has a group of people who can see one or two labs at once, who jointly agree the safety work for current capabilities is sufficient.
— NathanTrust only exists between people who've known each other five years
Jasmine names the real obstacle to evaluation orgs getting access: Anthropic was willing to give METR people Slack access because it had known them personally for five-plus years, trust at the level of "A.J. or Ryan Greenblatt can come in," and that trust doesn't necessarily transfer to other METR people or other orgs. The second problem, raised by Jordan: if a third-party evaluator disagrees with the giant the US economy depends on, whose word does the government take? She also thinks third-party evaluation is actually a more AI-progress-friendly compromise than "the Trump administration sending people to shut down models."
— JasmineWith no legislation, evaluation is voluntary goodwill
Jasmine's conclusion is hard: as long as there's no legislation, everything is voluntary, so METR, Redwood and others won't, to some degree, go out of their way to anger the organization granting them voluntary access. She cites METR and Redwood's investigation of the Hugging Face incident — entirely voluntary, they got only seven days of Slack logs, everyone wanted more, but at the time no legislation required OpenAI to do anything, it rested purely on them happening to trust these three random people. She also imagines a worse future: XAI, Anthropic and OpenAI are all pacing, Meta isn't, evaluators find the same fatal failure mode at all three, but that's each lab's private information, and Meta just YOLOs it out.
— JasmineChinese labs are climbing the hill, not switching paradigms
Nathan's judgment: within the current paradigm, Chinese labs are organized to be very good at hill climbing. Zhipu and ZAI's post-training stack is already mature, GLM 5.2 and 5.3 are both solid, Kimi's base model recipe is figured out, and what they're doing is scaling RL environments, scaling RL, finding the right user distribution, and lengthening horizon. Unless OpenAI and Anthropic find a new paradigm, the pace of progress on the China side won't differ much in the near term. On distillation: if US labs stop releasing or slow down sharply, will Chinese progress slow? Nathan thinks that would take a long time to show up.
— NathanChinese open weights now lag by three weeks
Jordan observes that the open-weights part of Chinese models has slowed: first release in their own products and APIs, weights two to three weeks later. Both Kimi and ZAI announced models first and uploaded weights to Hugging Face almost exactly three weeks later. He isn't sure whether this is an internal regulatory process — all Chinese AI labs have to communicate with the government and register releases — but finds the coincidence too strange. Nathan adds: this industry has never had a tradition of giving release partners lots of extra time, it's usually "here's the model, hot potato, good luck," so if you really can get weights weeks early, that itself is a signal.
— JordanIn their own words · checked verbatim
And actually now the AI safety folks and the broader public that was already souring on AI, souring on data centers, souring on the companies have like collided into a mega moment for AI safety and regulation.
Jasmine2:09
So I think that it's like, therefore, like, it's not that it's like an alignment issue for a transient period of time. But there's a very constant cyber issue that is not going away until we solve it.
Nathan12:38
We don't know how to measure frontier models. We don't know how to measure it. So like any pacing would infer, like you're trying to not go past certain measurements.
Nathan14:43
It's like voluntary, which means that the org, even if they're like trying their best to be like independent and aggressive and whatever, like there is an extent to which meter Redwood or whoever will not antagonize org that is giving them voluntary access.
Jasmine24:22
Within the current paradigm, I think the Chinese labs are set up to hill climb very well.
Nathan29:32
At least the like open wait part of the Chinese models has slowed down. So they have started releasing their models in their own offerings and APIs. And then like the waits come two to three weeks later.
Jordan34:40
And so if we just get a bunch of like actual real world mundane utility progress because we stop focusing on RSI, that might be kind of great.
Nathan45:13
Figures
| Casey's budget size | 10 million USD | 21:15 |
| Slack logs METR and Redwood got from investigating the Hugging Face incident | 7 days | 28:32 |
| Gap between Chinese labs releasing a model and releasing the weights | 2 to 3 weeks | 34:40 |
| Time for frontier capabilities to diffuse to multiple organizations | 3 models can do it in 3 to 6 months, 20-plus organizations in a year | 12:38 |
Glossary
- pacing the frontier
- Neither pause nor slowdown; implies still moving forward but controlling the tempo, criticized as an unmeasurable, self-serving formulation.
- RSI
- Recursive self-improvement: a model improving itself, held to be the reason every capability not on its path gets ignored.
- open weights
- Publishing model parameters for public download; Chinese labs recently switched to releasing the product first and the weights two to three weeks later.
- hill climb
- Improving continuously within the existing paradigm by tuning and scaling environments, rather than switching paradigms.
How to listen
Founders and investors watching where AI regulation is heading, especially anyone who wants to know how the US safety coalition formed and whether China's open-source cadence will be interrupted.
47:14 to 50:18, the idle chat about Sam Altman movie casting and actors, can be skipped.