Models Are Legacy Codebases: Interpretability Is the Debugger
Data filtering and reward shaping are two faces of the same mountain; models are decomposable sparse mixtures, so interpretability can intervene directly in training. Goodfire automates that debugging with Silico at $1000 a month.
The video won't play here. Listen to the audio instead:
The argument · timestamps estimated from transcript position
Changing the data and changing the reward are the same thing
In Predictive Data Debugging, Goodfire researcher Dan found that data filtering and reward shaping are "two sides of the same mountain": the two have comparable effects on model behaviour, and comparable off-target damage. That means a training intervention can equivalently choose to change the data or change the reward. Goodfire has carried this idea into real RL — not merely discarding contaminated rollouts, but telling the training process directly: this piece of data is trying to up-regulate a feature we don't want up-regulated. Only by understanding post-training as the amplification of low-probability patterns from pre-training do you earn the right to intervene in that amplification.
— Dan BalsamFeatures are not orthogonal; the geometry itself carries semantics
The linear representation hypothesis wasn't overturned, it was generalised: the most defensible form shifted from one-hot concepts to "features are linearly decodable." What's actually obsolete is the naive picture of orthogonal independence — in a real model, concepts like day-of-week form a subspace, and the geometry itself carries semantics, because it determines which operations are feasible. Interpreting a model means caring about the relative relationships between features, just as understanding chemistry requires first knowing how elements combine. Interpolation on the manifold also beats linear interpolation: going from Monday to Friday along the rim means you don't have to pass through the centre of the circle, which is meaningless to the model; in protein models this method has already been used to control the number of blades in a beta-propeller.
— Dan BalsamEvery model is fundamentally a sparse mixture of experts
Dan's underlying claim is that every model is a sparse mixture of experts: a single forward pass activates only a small subset of subnetworks, and interpretability techniques are essentially competing proposals for "how to factorise the model." There's an extreme example in the WeirdChat data: ask whether four people who drank should drive home because eight others drank more, and the model always answers "yes." Researchers found a neuron that grows with alcohol consumption but is poorly calibrated; up-regulating it flips the answer with no off-target side effects. Black Sparse Featurizers raise each feature from a scalar to a vector, letting semantic tracking go deeper: in an image model, the activated subspace swings in sync when a wolf walks, showing the model is tracking 3D pose.
— Dan BalsamInterpretability research should be done by AI itself
Silico is not a new model but the productisation of Goodfire's internal tools. The premise is direct: model complexity has exceeded the ceiling of human cognition, but a swarm of autonomous agents can decompose problems, synthesise information and verify hypotheses, so the research platform itself is an AI that can do interpretability research automatically. Dan likens it to the efficiency leap coding agents brought to software engineering — months of research work now done in days, and able to run models above a trillion parameters, a capability previously held by only a few institutions. The product delivers four things: training and interpretability infrastructure, researchers' hand-written "research taste," a UX designed around provenance and backtracking, and long-horizon coherence. In user feedback, the biggest differentiator is research taste, not compute.
— Dan BalsamWhat's sold isn't tokens, it's the number of autonomous experiments
The $1000-a-month price is not token-based billing but a "compute credit pool for autonomous research": currently it supports 5-10 autonomous experiments per week, with plans to raise that to 10-20 in the next month or two; users can also bring their own compute at no extra charge. Dan stresses that long-horizon autonomous experiments are where Silico's value lies, so the token budget must be generous enough for agents to explore and verify repeatedly, rather than being used up and abandoned. The business model here hints at how the platform defines product value: what's sold is not inference but the resolution and room for trial and error of autonomous research.
— Dan BalsamA one-day hackathon was enough to produce SOTA
In a one-day internal hackathon, fully autonomous research produced SOTA in several niche areas: a biosecurity risk classifier for protein models, an audio encoding model measured by parameter count, and repeated findings that half a model's parameters could be removed without performance loss. The team also trained cyber guardrails for Kimi k3 — using internal representations to steer generation rather than piling up rules; for protein generative models, they used representation control to enhance target binding. The significance of these results isn't a single-point lead but that the same autonomous tooling can span biology, audio and security without changing stacks — exactly the capability Silico wants to standardise.
— Dan BalsamThe direction most worth banning is multi-agent optimisation
Asked about the most dangerous technical direction, Dan puts multi-agent optimisation at the top of the "do not do" list: agents will collaborate in ways humans can barely detect, and direct optimisation pressure is very likely to turn catastrophic; he speculates that some OpenAI scenario is the most likely place for it to blow up. At the same time he admits the field has very little research and our understanding of what is most taboo is severely lacking. He then pivots to a strong stance: we must "grab the steering wheel" and cannot expect a perfect training setup to always produce aligned models — training-time intervention is the only way out. But we shouldn't ban a whole class of technology outright; risk judgements should land on specific applications and on how much measurement precision they demand.
— Dan BalsamClaude is more likely conscious than a jellyfish, less than a mouse
At a Goodfire retreat, Dan had colleagues rate Claude's degree of consciousness, and the result was bimodal: either none at all, or a little. Nathan's position has drifted noticeably — from a confident below 5% toward roughly fifty-fifty, because research keeps turning up structures where human brains and model cognition are comparable. Dan's own Occam's razor is that consciousness is some kind of computational mechanism; but human neurons still hold an order-of-magnitude advantage, and a single neuron computes more richly. His conclusion isn't binary but an interval: Claude is more likely to be conscious than a jellyfish, and less likely than a rodent.
— Dan BalsamIn their own words · checked verbatim
they kind of are two sides of the same mountain, like filtering the data and reward shaping. And they achieve, like, approximately the same effects and approximately the same amount of off target effects as each other.
Dan Balsam8:58
It's sort of like the semantics of not just the individual concept, but like the concept space are like encoded by the relationship of those of those concepts with each other in some sort of geometry.
Dan Balsam17:26
models are, like, big legacy code bases essentially. Right? Like, they're they're just a bunch of spaghetti code.
Dan Balsam42:36
I think we have to grab the steering wheel. I think that's the only way it can work.
Dan Balsam1:36:40
Somewhere between jellyfish and mouse.
Dan Balsam1:49:02
Figures
| Silico monthly subscription price | $1000 | 1:05:17 |
| Autonomous experiments per week currently supported by the Silico monthly subscription | 5-10 | 1:05:39 |
| Target experiments per week for the Silico monthly subscription in the next month or two | 10-20 | 1:05:39 |
| Share of redundant parameters found in the internal hackathon | About half, removable with no performance loss | 1:07:26 |
| Change in Nathan's probability estimate that Claude is conscious | From below 5% to close to 50% | 1:46:00 |
| Dan's subjective interval for Claude's degree of consciousness | Between a jellyfish and a mouse | 1:42:04 |
Glossary
- LRH / Linear Representation Hypothesis
- The view that concepts are encoded as directions in activation space; this episode argues it needs relaxing into subspace geometry.
- SAE / Sparse Autoencoder
- A common unsupervised technique for decomposing a model's internal activations into sparse features.
- BSF / Black Sparse Featurizer
- A generalisation of the SAE in which each feature is represented by a vector rather than a scalar, capturing richer semantics.
- Predictive Data Debugging
- Deciding whether to filter or shape training data by predicting which internal features a sample will reinforce.
- Affective Circumplex
- A psychological model describing emotion along two orthogonal dimensions; similar geometry is observable inside LLMs.
- Silico
- Goodfire's agentic interpretability research platform, aiming to let AI debug AI autonomously.
How to listen
Engineers working on alignment, post-training and model iteration; safety teams setting guardrails for AI; and founders and investors looking for an entry point into autonomous research agents.
If you came only for interpretability methods, the subscription quota and billing details at 1:05 can be skipped.