Individual neuron meanings are far less confused than imagined; agents find answers through systematic interrogation
Gated MLPs force neurons to switch between a handful of meanings, not hundreds mixed together. AI agents running systematic experiments can write descriptions that accurately predict neuron behavior, overturning the old claim that neurons are too chaotic to interpret.
The video won't play here. Listen to the audio instead:
The argument · timestamps estimated from transcript position
Single neurons are far more interpretable than academia assumed
Tigges's core claim: individual neurons in MLPs have been underestimated by prior interpretability work. Early networks were smaller and messier, and activation functions like ReLU and GELU were hard to parse, so the field defaulted to ‘representations are distributed, not in single neurons’ and relied on auxiliary models like SAEs instead. His counter: you rarely need many neurons firing at once; even if one neuron encodes multiple meanings, it's usually ‘a compact cluster of meanings,’ not hundreds of inseparable semantic stacks.
— Curt TiggesGated structures let one neuron hold two different meanings
Modern models use SwiGLU gated MLPs that give each neuron two independent signals: the result of gate projection multiplied by up projection, forming AND-gate-like logic. This lets the same neuron switch interpretations across contexts—Tigges's example is a neuron where one pole represents ‘oral health’ and the other ‘marine wildlife,’ with completely different downstream neurons pre-positioned by context to receive the right signal and avoid misreading.
— Curt TiggesAblating 88% of activations barely dents model performance
On Qwen3.6-35B-A3B, the team kept only the top-64 strongest activations per active expert (512 neurons total, keeping 64), suppressing roughly 88% of activations and zeroing the rest; another approach kept only extreme values per neuron across historical percentiles, replacing the rest with near-zero means. Result: model performance on many benchmarks stayed roughly flat compared to the original; the only clear degradation was in reasoning—loops, repeated self-checking—but simple subsequent fine-tuning restored performance to or beyond original levels.
— Curt TiggesWithout internal noise, models become more confident
Why do some reasoning-free benchmarks actually score higher after cutting most neurons? Tigges explains these benchmarks measure how fast the model answers, and with less internal noise to distract, the most confident answer floats to the top without interference. For reasoning tasks, the same effect appears after fine-tuning—the model finds the answer it already knows faster, without repeated self-doubt.
— Curt Tigges74% KL divergence recovery validates neuron function descriptions
The way to verify whether a neuron function description is sound: completely ablate the neuron, have another model score each token's predicted activation 0-9 based only on the description, feed those predictions back in, and measure how much it recovers the damaged next-token prediction distribution (KL divergence). The main metric is 74% KL divergence recovery weighted by damage magnitude—not 74% accuracy, not ‘the model is 74% understood,’ but at positions where the neuron actually matters, predictions roughly match the real activations.
— Curt TiggesThe monosemanticity question is the wrong question
Tigges says ‘is this neuron monosemantic or multisemantic’ itself is misleading; the real question is whether you can enumerate everything it represents. His hypothetical: a neuron that might respond to clothing, exercise equipment, dishes, and tablecloths—looks multivalent, but if the model's underlying concept is ‘things you'd see at a garage sale,’ humans might never discover that connection. Many neurons are better understood as verbs than nouns: not detecting an entity, but signaling downstream to trigger wait states, rechecks, direction changes in reasoning control flow.
— Curt TiggesFacts live in connections, not neurons
Tigges flags a counterintuitive problem: models have around 5 million neurons, but clearly remember far more facts because one neuron ≠ one fact; facts live in connections between neurons—the number of paths a few dozen interconnected neurons can combine is exponential. His example: a small ‘community’ of dozens of neurons dedicated to encoding which painters belong to Impressionism, Neo-Impressionism, or Post-Impressionism. Verified experts also show strong topic consistency, mostly covering one to three subjects.
— Curt Tigges100 million tokens of fine-tuning brings reasoning back
After sparsity, reasoning degrades—loops, self-checking, indecision—but Tigges says that's not lost knowledge, it's miscalibration. The team used roughly 100 million tokens, a few GPUs, and one day of subsequent fine-tuning to restore or partly exceed original performance, verified on math, code, and medical QA benchmarks. He emphasizes this is far below standard post-training scale and vastly below pre-training scale, showing the ablated 90% of activations affected whether the model could smoothly find an answer, not whether it knew one.
— Curt TiggesIn their own words · checked verbatim
the individual neurons in MLPs in neural networks in general are more interpretable than the field has thought to this point.
Curt Tigges7:18
there's a neuron where the one meaning is oral health and the other meaning is marine wildlife.
Curt Tigges18:04
It doesn't have as much internally to distract itself from the right answer, and the most confident thing rises to the top.
Curt Tigges37:32
as we apply increasing levels of intelligence, then more and more of what the neuron is doing becomes legible and predictable.
Curt Tigges45:59
It's not just detecting a thing. It's like sending a signal that is going to induce the model to do something else.
Curt Tigges1:02:16
A neuron is not a fact, but the fact is in the connections between neurons.
Curt Tigges1:21:31
They're, like, mostly about one to three topics, and that's it.
Curt Tigges1:25:25
Figures
| Qwen3.6-35B-A3B neuron count | ~5 million | 12:26 |
| Expert layer structure | 256 experts per layer, select 8, 512 neurons per expert | 12:26 |
| Top-64 sparsity ablation | ~88% of activations | 29:20 |
| KL divergence recovery | 74% | 51:09 |
| Analysis cost per neuron | $45 | 1:48:05 |
| Activation sample corpus | 10 billion tokens | 35:54 |
Glossary
- gated MLP (SwiGLU)
- Uses gate projection and up projection multiplied together, giving each neuron AND-gate-like logic to combine meanings across different contexts.
- monosemanticity
- Whether a neuron corresponds to a single concept rather than mixing multiple distinct meanings.
- ablation
- Forcibly deactivating or replacing a neuron with mean values to observe resulting changes in model performance.
- SAE (sparse autoencoder)
- Auxiliary model previously used to extract ‘interpretable features’ from model internal activations.
- neuron scientist
- AI agent that automatically clusters neuron activations and generates and verifies functional hypotheses about them.
How to listen
Engineers and researchers working on model interpretability, safety monitoring, or post-training fine-tuning, who want to know if you can bypass auxiliary models and read neurons directly.
The opening through 05:55 is the host's preview summary of the full episode and repeats material later; you can skip it.