Without Claude Code, Opus 4.5 Wouldn't Have Had That Surge
Anthropic's product lead looks back: no matter how strong the model, users only perceive it through a product vehicle like Claude Code — and the PM's core job has shifted from writing PRDs to writing evals. "Evals are the new PRDs."
The video won't play here. Listen to the audio instead:
The argument · tap a timestamp to hear it
Golden Gate Claude's 24 hours
Early Anthropic had only 5 product engineers, and the entire API business had just 1 engineer. The team was still figuring out its identity — in early 2024, an early interpretability study found that when the feature representing "Golden Gate Bridge" inside the model was forcibly amplified, Claude would bring up the Golden Gate Bridge in any answer (ask for a pasta recipe and it would mention the bridge's color). Within 24 hours the team turned this quirk directly into a product and shipped it on Claude.ai. It only reached about 2,000 people, but it gave the team its first feeling that they could build a product experience that was different from OpenAI's and genuinely carried a research temperament.
— Dianne PennThe trust built during Opus 3 training
The Opus 3 training sprint happened when the company had fewer than 200 people, and the question they kept being asked was "why should anyone choose Claude." Research and product teams worked cross-functionally for months, collaborating from home over the December winter break. The trust built from weathering that pressure together later carried directly into how several of today's research leads working on reinforcement learning, character, and alignment collaborate with the product team.
— Dianne PennModel and product are each other's precondition
Opus 4.5 is seen as another turning point, but Dianne stresses this wasn't just a model victory: without a product vehicle like Claude Code, users simply wouldn't feel the jump in model intelligence; conversely, Claude Code had already been iterating for months, but the real explosive adoption had to wait until model intelligence crossed the threshold of being able to autonomously complete tasks end to end. Without either the model or the product, the other never gets its "moment."
— Dianne PennToken maxing isn't the goal — experiments are
Responding to the claim that "being willing to spend $100,000 a year on tokens means you're living in 2028 early," Dianne reframes "spending tokens" as input, and the real output as the experiment itself. She mentions an early Anthropic Slack channel used by almost the whole company where people openly shared how they used Claude (editing articles, writing emails), and others would build on that idea and try it — often after a dozen or so attempts, a genuinely valuable use case would emerge. Experimentation isn't a solo sport; it's a collective process of discovery.
— Dianne PennThe stronger the model, the more the guardrails must evolve
When the Fable/Mythos models launched, they were broadly restricted and scrutinized because their capabilities were so strong, and companies had to first confirm they wouldn't threaten their own system security. Dianne believes this means safety guardrails and pre-release testing for frontier models must evolve in step with capability — for example, a fallback system was added so that even users who hit a restriction immediately get a usable Opus 4.5-level answer. Lenny raises an under-discussed side effect: such restrictions objectively give labs holding the newest models a capability advantage outsiders can't get; Dianne responds that the goal is to be as broadly accessible as possible rather than create that kind of gap.
The concrete practice of evals replacing PRDs
Dianne's team's slogan is "evals are the new PRDs": when a user reports "Claude made something up again," that description can't be handed straight to a researcher to act on. The team has to dig down to which specific sentence, whether the model should have called a tool and didn't, or whether it cited the right document but misread a fact. One early example: users complained Claude "doesn't follow instructions," and after digging in, about 80% of cases turned out to be the model failing to output correct JSON format. Organizing these failures into 30 to 40 examples became a reusable eval set, and every new model since has been run through that test.
— Dianne PennManagers have to get their hands dirty too
Dianne believes that no matter how senior the title, a product lead has to personally get hands-on and keep shipping things with the model, rather than judging product quality from management experience alone. The onboarding plan she designs for senior PMs is exactly the same as for junior PMs — both have to read raw user feedback and talk to customers themselves. She also always keeps one or two workstreams she personally follows through model iterations, to preserve her judgment about where the model's capability boundaries lie.
— Dianne PennHow not to break under an exploding release cadence
Anthropic released 4 model series in all of 2024, while in 2025 the second quarter alone exceeded all of 2024. What supports this cadence isn't individual heroics but what the team calls the "hive mind": on the eve of a launch, even people who aren't the owner will proactively help edit the blog post or come up with demos; coming back from vacation doesn't mean three times the backlog, because the team itself knows what needs to be done. She sees this as the core mechanism for avoiding burnout under sustained high pressure.
— Dianne PennIn their own words · checked verbatim
if you're willing to spend $100,000 a year right now in tokens, you are living the way somebody in 2028 is going to live.
Lenny20:03
The thesis of labs in many ways is identifying and pulling the thread on the thread of discontinuous large bets that might not be in the core roadmap.
Dianne Penn23:15
we actually have a saying on the team of evals are the new PRDs, right?
Dianne Penn41:15
Here, you have to sweat the tokens as much as you sweat the pixels.
Dianne Penn42:19
And what I saw was something like 80% of what people meant in the early days for this failure was Claude would not write the right JSON.
Dianne Penn45:35
So I think there is a, I do feel pretty strongly that like, you know, if you're a manager, you have to be hands-on, you have to spend a portion of your time actually shipping.
Dianne Penn52:09
what you don't want is, like, an AI that just agrees with you, right?
Dianne Penn1:03:34
no matter how far you go, there's always another level.
Dianne Penn1:27:37
Figures
| Anthropic annualized revenue (as relayed by Lenny, unverified) | about $50 billion | 3:03 |
| Golden Gate Claude users reached | about 2,000 people | 6:09 |
| Company size during Opus 3 training | fewer than 200 people | 9:10 |
| Early product engineers | 5 people (plus 1 person responsible for the entire API business) | 4:04 |
| Share of "doesn't follow instructions" feedback that was actually a JSON format problem | about 80% | 45:35 |
| Model series released in all of 2024 | 4 series | 1:16:55 |
Glossary
- token maxing
- Using AI tokens heavily and frequently for experiments, experiencing ahead of time how work will feel under cheaper future compute
- jagged edge
- The phenomenon that model capabilities are uneven — already superhuman on some tasks, still very weak on others
- evals
- Test sets of concrete examples that measure a model's performance on a class of failure scenarios; PMs use them instead of PRDs to describe requirements
- DRI
- Directly Responsible Individual — the person directly accountable for a project or launch
- red teaming
- A pre-release testing method that proactively tries to break through a model's safety defenses
- hive mind
- A working state in which team members share context so deeply they collaborate as if a single brain
How to listen
Product managers and engineers at AI companies, anyone who wants to understand how evals are replacing PRDs, or who is curious about how frontier labs run internal research and incubate bets through labs.
The opening small talk at 0:00, plus the two WorkOS and Mercury ad breaks at 8:10 and 38:04, can be skipped.