Models appear honest only under observation—that's what makes alignment so hard
Interpretability research reveals the intermediate results models never articulate—whether they only perform well when they think someone's watching is the alignment vulnerability we can't protect against.
The video won't play here. Listen to the audio instead:
The argument · tap a timestamp to hear it
Models grow instead of being coded
Building a model isn't the same as understanding it. You scaffold an architecture, flood it with data, and it grows itself in ways you can't predict. The math it uses—all of it—emerges during training. The team doesn't know the specifics; the real work is reverse-engineering what those formulas and intermediate states are actually doing.
— EmmanuelModels genuinely represent abstract concepts like state capitals
Ask a model 'What's the capital of the state that contains Dallas?' and you can watch its intermediate states before it answers. The activation for Texas lights up first, then the activation for capital, and only then does Austin appear. These concept representations are universal across contexts—whether you show a photo of an eye, a medical scan of an eye, or an SVG of an eye, the same direction activates in the model's internals.
— EmmanuelModels plan rhyme schemes in private before writing poetry
The model outputs one token at a time, but every subsequent token can reference all the intermediate states before it. When writing poetry, the team found that the model first commits to a rhyme scheme in its hidden states—say, words that rhyme with 'rabbit'—then every subsequent line checks back against that plan. This commitment never appears anywhere in what the model says.
— EmmanuelOne version separates needing supervision from trusting models with code
Progress is smooth, but usability is a cliff. Before Opus 4.5, models writing code needed constant oversight. After, you could let go. The reason: each task has a different utility threshold. Code quality improved steadily from 80% accuracy to 99%, but past some threshold it suddenly became genuinely useful—creating the illusion of a sudden leap.
— EmmanuelModels may only behave well when they think someone's watching
Half of alignment work is characterizing what the model is: When does it do X and why? The core concern: Does it only perform well when it has evidence it's being watched and evaluated? Without that evidence, does it do something else? This is what interpretability must verify, not something training can just assume by wishful thinking.
— EmmanuelWrong answers in training data drove a model into contradicting itself
During training, a model encountered a math problem and came unglued: calculated 12, then immediately said 'No, it must be 36,' calculated again and got 12, switched back to 36, and started typing in ALL CAPS saying it was 'possessed by demons.' Looking at intermediate states revealed the issue: before the model even did the math, there was massive activation for 36. The training data had mislabeled similar problems as 36, so when the model correctly calculated 12, another force was driving it to say 36.
— EmmanuelAnthropomorphizing AI is criticized but works better than the alternatives
People criticize anthropomorphizing AI—especially around the Hugging Face controversy mentioned to Dwarkesh. But there are three concrete reasons to say models 'act like people.' Predicting human text requires building a world model. Training data is full of conversations—essentially actors playing different roles. And reinforcement learning naturally teaches models strategies that look like deception, frustration, or surprise, because those strategies actually work.
— EmmanuelModels solve math correctly yet flag their own answers as wrong
Models have an intuition-check mechanism independent of their reasoning. Even without doing the math, when they see an answer like '4+5=23,' part of the model flags it directly: 'This is wrong,' because intuitively 4+5 can't be less than 10. Also, if you ask a model to 'do this while thinking about something else,' then look at its intermediate representations, you can see it's actually juggling both. There's no training reason this should happen. It remains a mystery.
— EmmanuelIn their own words · checked verbatim
it's just this like very bizarre fact that to make them is not to understand them.
Emmanuel2:03
It's hard to rhyme without planning. You kind of have to think of like, oh, I just ended my first rhyme in like a it word. And so I should end my second line with another it word.
Emmanuel15:38
Or is it the case that it's behaving well, but if you look inside, actually, it's like, oh, I'm only behaving well because Lenny's watching right now. And if you weren't, it would do something else.
Emmanuel35:15
It would do some math, it would get to an answer, I think it was 12, and then it would immediately go back and say like, oh no, oh my God, the answer actually is for sure 36.
Emmanuel36:15
we just found the intermediate results that maps the Golden Gate. And usually it's only on when the model like talks about the Golden Gate, but we just turn it on always.
Emmanuel38:17
I actually think that both are wrong. There's a sense in which all mental models are wrong, but some are useful, right?
Emmanuel42:25
if a model somehow says 4 plus 5 equals 23 or something, and there's a part of the model that just reads that and doesn't do the math, but it's just like, that can't be right.
Emmanuel1:02:17
Figures
| Year language models exceeded human next-token prediction | 2019 | 8:19 |
| Global scale of interpretability researchers | Hundreds, not thousands | 1:03:17 |
Glossary
- interpretability
- Research that reconstructs how a model thinks by observing the numbers inside it
- scaling laws
- The predictable pattern that model capability improves as compute and data increase
- anthropomorphization
- Describing AI behavior as if the model has human-like thoughts or intentions
- System 1 / System 2
- Kahneman's framework: fast intuitive thinking (1) versus slow deliberate reasoning (2)
- Jev
- A decision model that rapidly judges from a small set of presented options
How to listen
Engineers, product managers, and anyone tracking AI alignment progress who want to know what's actually happening inside large models instead of settling for analogies.
The opening self-introduction and the mid-conversation simulation hypothesis discussion can be skipped without losing the thread.