Without Claude Code, Opus 4.5 Would Never Have Had Its Breakout
Anthropic's product lead looks back: however strong a model gets, users only feel it through a vehicle like Claude Code — and the core of the PM job has shifted from writing PRDs to writing evals. As she puts it, the eval is the new PRD.
The video won't play here. Listen to the audio instead:
The argument · tap a timestamp to hear it
A 24-hour joke shipped to 2,000 people gave the product team its identity
In the early days Anthropic had only 5 product engineers, with a single engineer covering the entire API business, and the team was still working out what it was. In early 2024 a piece of interpretability research found the internal feature representing the Golden Gate Bridge; when that feature was forcibly turned up, Claude dragged the bridge into every answer it gave — ask it for a pasta recipe and it would work in the colour of the bridge. Within 24 hours the team turned that quirk into a shipped product on Claude.ai. It reached only about 2,000 people, but it was the first time the team felt it could build a product experience that was unlike OpenAI's and genuinely carried the character of the research.
— Dianne PennTrust built during the Opus 3 sprint still shapes how research and product work together
The Opus 3 training sprint happened when the company was under 200 people, and the question the team kept being asked was: why should anyone choose Claude? Research and product worked across org lines for months straight, including through the December holiday break, from home. The trust that came out of carrying that pressure together is the direct basis for how the product team now works with the several research leads who run reinforcement learning, character and alignment.
— Dianne PennNeither the model nor the product gets its moment without the other
Opus 4.5 is treated as another turning point, but Dianne is emphatic that this was not a model victory alone: without a product vehicle like Claude Code, users would simply not have felt the jump in the model's intelligence. And in the other direction, Claude Code had already been iterating for months, yet the explosive adoption only arrived once model intelligence crossed the threshold where it could carry a task end to end on its own. Take away either side and the other never has its moment.
— Dianne PennSpending tokens is the input; the experiment is what you are actually buying
Responding to the claim that being willing to spend $100,000 a year on tokens means living in 2028 early, Dianne reframes token spend as an input — the real output is the experimentation itself. She describes an early Anthropic Slack channel that nearly the whole company was in, where people openly shared how they were using Claude (reworking a piece of writing, drafting an email) and others would build on the idea and try it themselves; often it took a dozen or more attempts before a genuinely valuable use emerged. Experimentation is not a solo sport; it is a collective discovery process.
— Dianne PennThe stronger the model, the faster the guardrails have to evolve with it
When the Fable/Mythos models shipped, they were broadly restricted and put under review precisely because they were so capable — companies wanted to confirm the model would not threaten the security of their own systems first. Dianne reads this as evidence that safety guardrails and pre-release testing for frontier models have to evolve in lockstep with capability; one response was a new fallback system, so that a user who hits a restriction still gets an immediately usable answer at the Opus 4.5 level. Lenny raises a side effect he thinks is under-discussed: restrictions like these objectively hand the labs holding the newest models a capability advantage that outsiders cannot get. Dianne answers that the goal is to make access as broadly available as possible rather than to manufacture that gap.
Vague user complaints are not specs; thirty to forty concrete failures are
The slogan on Dianne's team is that the eval is the new PRD. When a user says Claude made something up again, that description cannot be handed to a researcher to act on. The team has to push until it knows which sentence, whether the model should have called a tool and failed to, or whether it cited the right document and then read the fact wrong. An early example: users complained that Claude did not follow instructions, and digging in showed that roughly 80% of the time the real problem was that the model could not produce correctly formatted JSON. Turning that class of failure into 30 to 40 examples produced a reusable eval set, and every new model has been run against it since.
— Dianne PennNo amount of seniority exempts a product leader from shipping with the model
Dianne argues that no matter how senior you are, a product leader has to keep hands on the tools and keep shipping things with the model, rather than judging product quality from management experience alone. The onboarding plan she writes for a senior PM is identical to the one for a junior PM: read the raw user feedback, talk to customers yourself. She keeps one or two workstreams that she personally follows through model iterations, so that she retains her own judgement about where the model's capabilities end.
— Dianne PennSurviving an exploding release cadence takes a hive mind, not individual heroics
Anthropic shipped 4 model series across all of 2024; the second quarter of 2025 alone exceeded that entire year's output. What sustains that pace is not personal heroics but what the team calls a hive mind: on the night before a launch, people who are not the owner will step in to edit the blog post or come up with demos, and coming back from vacation does not mean three times the backlog, because the team already knows what needs doing. She sees this as the core mechanism for not burning out under sustained pressure.
— Dianne PennIn their own words · checked verbatim
if you're willing to spend $100,000 a year right now in tokens, you are living the way somebody in 2028 is going to live.
Lenny20:03
The thesis of labs in many ways is identifying and pulling the thread on the thread of discontinuous large bets that might not be in the core roadmap.
Dianne Penn23:15
we actually have a saying on the team of evals are the new PRDs, right?
Dianne Penn41:15
Here, you have to sweat the tokens as much as you sweat the pixels.
Dianne Penn42:19
And what I saw was something like 80% of what people meant in the early days for this failure was Claude would not write the right JSON.
Dianne Penn45:35
So I think there is a, I do feel pretty strongly that like, you know, if you're a manager, you have to be hands-on, you have to spend a portion of your time actually shipping.
Dianne Penn52:09
what you don't want is, like, an AI that just agrees with you, right?
Dianne Penn1:03:34
no matter how far you go, there's always another level.
Dianne Penn1:27:37
Figures
| Anthropic annualized revenue (relayed by Lenny, unverified) | about $50 billion | 3:03 |
| Users reached by Golden Gate Claude | about 2,000 | 6:09 |
| Company size during Opus 3 training | fewer than 200 people | 9:10 |
| Early product engineering headcount | 5 (plus 1 person covering the entire API business) | 4:04 |
| Share of "doesn't follow instructions" feedback that was actually a JSON formatting problem | about 80% | 45:35 |
| Model series shipped across all of 2024 | 4 series | 1:16:55 |
Glossary
- token maxing
- Using AI tokens heavily and frequently to run experiments, so you experience the way work will be done once compute is far cheaper
- jagged edge
- The unevenness of model capability — already superhuman at some tasks, still weak at others
- evals
- Test sets that use concrete examples to measure how a model performs on a class of failure; PMs use them in place of a PRD to express what they need
- DRI
- Directly Responsible Individual — the single person accountable for a project or a launch
- red teaming
- Testing that actively tries to break a model's safety defences before release
- hive mind
- A working state in which team members share context so deeply that they collaborate as though they were one brain
How to listen
Product managers and engineers building at AI companies, anyone trying to understand how evals replace PRDs, and people curious about how research works inside a frontier lab and how its labs group incubates new bets.
The opening small talk at 0:00, plus the WorkOS and Mercury ad breaks at 8:10 and 38:04.