AI detection isn't a style check — it looks for padded information
The tell in AI writing isn't style, it's ‘information padding’: output that carries more than the input did. A hedge-fund titan publicly owning his AI op-ed became the turning point — Pangram's fourth generation runs a false-positive rate of roughly 1 in 24,000, and models converging actually makes the writing easier to catch.
The video won't play here. Listen to the audio instead:
The argument · tap a timestamp to hear it
‘Did you use AI’ is the wrong question; ask where the ideas came from
The event: Stanley Druckenmiller published an op-ed in the WSJ that anyone paying attention could see was AI-written, and someone who ran it through Pangram got back a flat 100% AI. What made it a turning point is that he didn't hide it — he said outright that he isn't fundamentally a good writer and simply used AI — and the WSJ refused to retract. Bryan pushes one step further: if you're admitting it, publish the prompt. ‘Did you use AI’ is a false binary; the real question is where the ideas came from — was it ‘write me an op-ed opposing the Treasury's actions’, or ‘here are my five pages of notes, polish them into an essay’?
— Max SperoDetecting AI is impossible in theory and good enough in practice
In 2023 Max asked friends doing research whether AI text could be detected at all. The answer was unanimous: impossible, don't try, and the models are only going to get stronger. Pangram's response was to give up on the theoretical counterexample (‘have ChatGPT restate any human text and you've falsified the whole idea’) and care only about the practical case: when the information going in is less than what comes out, that gap is the trace of AI involvement. Bryan calls it word inflation. That same definition explains why fake reviews on Yelp/Amazon never became the early breakthrough — the platforms reckoned AI reviews were only 3% of the total, not painful enough to act on.
— Max SperoAI is caught by synthetic mirror pairs, not em-dash rules
Pangram is a classifier model, not a surface heuristic of the em-dash-detector variety. The method: take an open-source LLM, cut off its ‘predict the next token’ head, and bolt on a classification head that outputs a 0-15 scale of AI involvement. The training data is manufactured by ‘synthetic mirroring’: take a human-written article, have an LLM compress it back into a prompt (‘write 200 words covering these three points’), then feed that prompt to a randomly chosen frontier model — yielding a matched human/AI pair on the same subject. A second layer uses editing prompts, then labels at the clause level: clauses left untouched are human, clauses that were altered are AI-assisted, and whole new sentences that appeared out of nowhere are AI-generated.
— Max SperoA detector lives or dies on its false-positive rate, not its accuracy
Pangram 4's false-positive rate is about 1/24,000, measured by running millions of pre-2022 documents — an era when it was all but impossible for anyone to have had AI ghostwrite for them. Its false-negative rate is about 1/300, quantified by taking real user prompts from WildChat and re-feeding them to frontier models to regenerate the text. Not perfect, but pointed the right way. Bryan's own experience is that false positives are vanishingly rare, and that is precisely the lifeline that decides whether a product like this can be used seriously. Pangram 4's step change came from replacing the 512-token window with token-level output: two AI sentences buried inside an otherwise human-written document can now be pointed out word by word.
— Max SperoA detector needn't hit 100% — it only has to make cheating cost something
Max says education is an enormous market, and this year is the first time anyone has registered that a genuinely usable AI detector exists. The product logic isn't 100% precision; it's putting friction on the shortcut — turning ‘throw the assignment at an AI’ from free into costly, at which point the student takes the next-easiest path, which is doing the work themselves. The same judgment carries into the enterprise: Oxide's hiring is a writing-intensive process, and a ‘why do you want to join Oxide’ essay produced by an LLM reciting boilerplate is recognizable on sight — the detector only says out loud what was already plain.
— Max SperoMost ‘I was wrongly flagged’ sob posts are humanizer marketing
Max separates two kinds of opposition. One kind genuinely wishes the detector didn't exist: it means the AI text their own profession turns out can now be authoritatively named as such. The other kind is marketing from the humanizer ecosystem — tools that claim to rewrite AI text so it slips past Turnitin and Pangram, sold to students whose assignments are being checked. They typically ship their own ‘AI detector’ as well: first it rules your text AI, then it sells you the rewrite. A lot of the Reddit posts crying ‘my teacher flagged me at 76% AI and I got a zero’ are astroturf for that lead gen — drop them into Pangram and they come back entirely AI-written.
— Max SperoStronger models haven't made detection harder; they've made it easier
Asked whether he is permanently chasing the frontier models, Max gives a counterintuitive answer: two things are happening at once — the models are smarter, but after round after round of post-training on ‘correctness’ and preferences, their output distribution is far narrower than in the GPT-2 era. GPT-2 tried to model the full distribution of human output; today's models hand you only the ‘correct’ token, not what an ordinary person would actually say. He and the three Oxide hosts report the same felt experience: new models talk with a distinct flavor of being ‘strangely erudite’. Detection hasn't gotten harder — it's gotten easier to pick up.
— Max SperoThe next argument is over how much the human actually put in
Internally, Pangram can already guess the model family: the probe's top-1 accuracy is 90%, though productizing it would mean training that up to 98%. The further-out direction is a ‘reverse Pangram’ — inferring what prompt could have produced a passage, and how much context the model was handed at the time: two points, or fifteen? Information on that axis changes the characterization of Druckenmiller's op-ed completely — handing over five pages of notes to be polished is collaboration; handing over five bullet points to be expanded is ghostwriting. The next fight in detection isn't ‘was AI used’, it's ‘how much did the human put in’.
— Max SperoIn their own words · checked verbatim
He says, yes. Of course I used AI to write this. I'm not a good writer, essentially.
Max Spero3:15
I actually think that, like, doing one's own writing actually helps you form your thoughts.
Bryan Cantrill7:30
We don't we don't care about, like, solving it in a, like, theoretical sense. We care about solving it in a practical sense.
Max Spero13:00
And so how we create the training data for Pangram is a method that we call synthetic mirroring.
Max Spero34:30
For Pangrom four, it's about one in twenty four thousand.
Max Spero39:30
creating friction on the, like, the easy path just makes people do the the next easiest path, which is doing the assignment themselves.
Max Spero46:00
Their claim is that it will evade any AI detector, like Turnitin, now Pangram.
Max Spero50:00
today's LLMs have a much more narrow distribution of what they output than, like, g p t two.
Max Spero59:30
Figures
| Pangram false-negative rate | about 1/300 (measured with real WildChat prompts regenerated by frontier models) | 40:30 |
| Corpus size behind the false-positive measurement | millions of pre-2022 documents | 39:30 |
| First Pangram model released | February 2024 | 41:00 |
| Pangram 4 released | a little over a month before this episode was recorded | 42:30 |
| Classification granularity before Pangram 4 | 512-token window | 43:30 |
| Top-1 accuracy of the internal probe identifying model family | 90% | 1:01:00 |
| API credits one person burned trying to fool Pangram | 700 dollars | 28:00 |
| Large platforms' early estimate of AI reviews | 3% | 15:30 |
Glossary
- synthetic mirroring
- Taking a human-written article, inferring the prompt behind it, then having a model rewrite from that prompt, to build matched ‘human/AI’ sample pairs.
- humanizer
- A service claiming to rewrite AI text so it can evade detectors, sold mainly to students whose assignments are being checked.
- classifier head
- The classification-scoring head that replaces a large model's original ‘predict the next token’ output head.
- WildChat
- A dataset of real user conversations with AI, which Pangram uses to estimate its false-negative rate.
How to listen
Product and engineering leads working on content moderation, hiring screens, and academic-integrity enforcement, plus content teams that produce a lot of LLM text and need to know where the line sits.
53:00-57:00, the Shell Game podcast and the philosophical wandering, can be skipped without losing the main thread.