AI Detection Isn't About Style, It's About Information Inflation
The tell of AI writing isn't style but the 'information inflation' where output exceeds input. A hedge fund titan's public admission about his AI-written op-ed became a turning point—Pangram's fourth-gen false positive rate is about 1 in 24,000, and the more models converge, the easier they are to catch.
The video won't play here. Listen to the audio instead:
The argument · tap a timestamp to hear it
The real question isn't 'did you use AI' but where the ideas came from
Event: Stanley Druckenmiller published an op-ed in WSJ that was obviously AI-written; someone ran it through Pangram and got a 100% AI score. The turning point was that he didn't hide it—he admitted, 'I'm essentially not a good writer, I just used AI,' and WSJ refused to retract. Bryan pushed further: since he admitted it, show the prompt. 'Did you use AI' is a false binary; the real question is where the ideas came from—whether it's 'help me write an op-ed against the Treasury's action' or 'here are my five pages of notes, help me polish them into an essay.'
— Max SperoDetecting AI is theoretically impossible but practically sufficient
In 2023, Max asked research friends whether AI text could be detected; the unanimous answer was 'impossible, don't try, models will only get stronger.' Pangram's response was to abandon theoretical counterexamples (like 'have ChatGPT recite any text to falsify it') and focus only on practical scenarios: when the input information is less than the output, that's a trace of AI involvement—Bryan calls it word inflation. This definition also explains why early Yelp/Amazon fake reviews didn't become a breakthrough: platforms felt AI reviews were only 3% and not worth the trouble.
— Max SperoCatching AI isn't about em dashes but synthetic mirroring
Pangram is a classifier model, not a surface-rule em dash detector. The approach: take an open-source LLM, cut off the 'predict next token' head, and replace it with a classification head that outputs a 0-15 AI involvement score. Training data is built via 'synthetic mirroring': take human-written articles, have an LLM compress them into prompts (like 'write a 200-word piece covering these three points'), then feed those to random frontier models to get a human/AI pair on the same topic. A second layer uses editing prompts for clause-level annotation: unchanged parts are human, modified parts are AI-assisted, and entirely new sentences are AI-generated.
— Max SperoA detector's lifeline is false positive rate, not accuracy
Pangram 4's false positive rate is about 1 in 24,000, measured by running millions of pre-2022 documents (when AI ghostwriting was nearly impossible); the false negative rate is about 1 in 300, measured by regenerating with frontier models from real user prompts in WildChat. Not perfect, but directionally right. Bryan's experience is that false positives are extremely low, and that's the lifeline for whether the product gets taken seriously. Pangram 4's step change came from switching from a 512-token window to token-level output: even a fully human document with two AI sentences in the middle can be pinpointed word by word.
— Max SperoDetectors don't need to be perfect, just make cheating costly
Max says education is a huge market, and this year is the first time people realize 'a truly usable AI detector exists.' The product logic isn't 100% precision but creating friction on the shortcut—making 'throwing homework at AI' go from zero cost to some cost, so students take the next-easiest path, which is doing it themselves. This judgment applies to enterprise too: Oxide's hiring is a writing-intensive process, and LLM-generated boilerplate in 'why I want to join Oxide' is instantly recognizable; the detector just makes it explicit.
— Max SperoMost 'false positive' sob stories are astroturfing ads
Max distinguishes two types of opposition. One group genuinely wishes detectors didn't exist: it makes the AI text they produce professionally identifiable by an authority. The other is the humanizer ecosystem's marketing: these tools claim to rewrite AI text to fool Turnitin and Pangram, selling to students caught cheating; they typically also include an 'AI detector' that first flags your text as AI, then sells the rewriting service. Many Reddit posts about 'teacher falsely flagged my essay as 76% AI, gave me a zero' are astroturf for this lead gen—run them through Pangram and they're all AI-written.
— Max SperoStronger models haven't made detection harder—it's easier
Asked if they're forever chasing frontier models, Max gives a counterintuitive take: two things happen simultaneously—models get smarter, but post-training that repeatedly teaches 'correctness' and preferences narrows the output distribution far more than in the GPT-2 era. GPT-2 tried to mimic the full human output distribution; today's models only give 'correct' tokens, not what an ordinary person would say. His and Oxide's three hosts' shared intuition: new models speak with a 'strangely erudite' flavor, and detection hasn't gotten harder—it's gotten easier.
— Max SperoThe next battleground is how much the human actually input
Pangram can already guess the model family internally: the probe's top-1 accuracy is 90%, but productization needs 98%. Further out is 'reverse Pangram'—inferring what prompt might have generated a text and how much context the model had: two key points or fifteen? This dimension's information changes the characterization of Druckenmiller's op-ed completely—giving five pages of notes to polish is collaboration; giving five bullet points to expand is ghostwriting. The next detection debate isn't 'did you use AI' but 'how much did the human actually input.'
— Max SperoIn their own words · checked verbatim
He says, yes. Of course I used AI to write this. I'm not a good writer, essentially.
Max Spero3:15
I actually think that, like, doing one's own writing actually helps you form your thoughts.
Bryan Cantrill7:30
We don't we don't care about, like, solving it in a, like, theoretical sense. We care about solving it in a practical sense.
Max Spero13:00
And so how we create the training data for Pangram is a method that we call synthetic mirroring.
Max Spero34:30
For Pangrom four, it's about one in twenty four thousand.
Max Spero39:30
creating friction on the, like, the easy path just makes people do the the next easiest path, which is doing the assignment themselves.
Max Spero46:00
Their claim is that it will evade any AI detector, like Turnitin, now Pangram.
Max Spero50:00
today's LLMs have a much more narrow distribution of what they output than, like, g p t two.
Max Spero59:30
Figures
| Pangram false negative rate | about 1 in 300 (measured with WildChat real prompts + frontier model regeneration) | 40:30 |
| False positive measurement corpus size | millions of pre-2022 documents | 39:30 |
| First Pangram model release | February 2024 | 41:00 |
| Pangram 4 release | a little over a month before this recording | 42:30 |
| Classification granularity before Pangram 4 | 512-token window | 43:30 |
| Internal probe top-1 accuracy for model family identification | 90% | 1:01:00 |
| API credits someone burned trying to fool Pangram | $700 | 28:00 |
| Large platforms' early estimate of AI reviews | 3% | 15:30 |
Glossary
- synthetic mirroring
- A technique that reverse-engineers prompts from human articles and has models rewrite them to create human/AI sample pairs.
- humanizer
- A service that claims to rewrite AI text to evade detectors, mainly sold to students caught cheating.
- classifier head
- The output head of a large model, originally for predicting the next token, replaced with a classification scoring head.
- WildChat
- A dataset of real user-AI conversations that Pangram uses to estimate false negative rates.
How to listen
Product and tech leads in content moderation, recruitment screening, and educational anti-cheating, plus content teams that heavily use LLMs and need to know where the boundaries lie.
The Shell Game podcast and philosophical digression from 53:00-57:00 can be skipped without missing the main thread.