The moat in evals is not the AI — it is human taste
You cannot leave eval design to the AI: it is good at top-down criteria, but the bottom-up evals that actually decide whether a product is good — the taste distilled out of error data — have to come from a human, and only then get executed at scale with Claude Code.
The argument · tap a timestamp to hear it
The model can write your criteria, but never your taste
Shreya splits evals into two kinds. Top-down criteria are set in advance from the nature of the task itself — length, actionability. Bottom-up criteria come out of browsing a large volume of samples and hardening a vague, intuitive dissatisfaction into a rule. Claude is very good at the top-down half, but the bottom-up half has to be supplied by a human, because the model has no way of knowing your taste out of thin air. That is also why evals have to be iterative: every new failure mode you find adds another criterion — though be careful not to overfit to a single interview.
— ShreyaHand a judge too long a rubric and it will quietly skip criteria
Shreya suggests splitting your criteria into two groups and having subagents grade each group separately, so the model does not get lazy and ignore items when facing one long checklist. She also has the skill auto-generate a spreadsheet (or pivot table) laying out pass and fail for every criterion, which lets you see quickly which criteria matter more for the task in front of you — and even weight them by hand, instead of deferring blindly to uniform weights.
— ShreyaFifteen minutes building an interface beats squinting at a spreadsheet
Shreya demoed her error discovery skill. Step one: read the data and work out its semantic type — is this an article, code, or a trace? Step two: design the visual encoding, using color, spacing and opacity to make differences in the data pop so a human can actually see them. Step three: build the review interface with HTML and Python. Step four: let the AI cluster the data and pick out a diverse set of samples. Step five: enter the interaction loop — you leave open-ended feedback in the interface, and Claude Code reads it in real time and folds it back into new samples or new eval criteria. The whole flow takes about 15 minutes, and what you buy is readability far beyond what a table gives you.
— ShreyaHumans only have to say what they dislike; let the AI do the sorting
Hamel explains the machinery behind that loop. As a person reads samples, they drop open-ended comments in passing — ‘I don't like the negative contrast’, ‘this is annoying but I can't say why’ — and a Claude monitor tool running in the background keeps digesting those comments. Once roughly ten have accumulated it starts inducing themes on its own, then proactively sweeps the full sample set and labels every instance of the same problem. The core philosophy: humans supply taste and judgment, the AI supplies discovery and categorization at scale, rather than inventing the criteria for you.
— HamelWhat error analysis produces is your eval rubric itself
Shreya points out that error analysis yields more than a list of problems — those failure modes convert directly into your rubric, which is to say your eval criteria. Of the 361 suggestions auto-generated in her demo, the most common were staccato fragments of fewer than four words (249 of them). Write those criteria into the skill and every future generation can be judged against them item by item with an LLM judge. And because you already have a human-labeled dataset, you can run the check in reverse and validate whether the eval criteria themselves are reliable.
— ShreyaWhat automated eval tools miss is exactly the product judgment
Hamel shared their benchmarking of automated eval tools like Brain Trust, Arize and LangSmith: they catch the obvious failures — tool calls that fail, users expressing dissatisfaction — but they broadly miss the implicit errors that require product judgment. Take a real estate sales bot facing the objection ‘you don't have this listing’: it will only say ‘sorry, we don't’, and never propose an alternative. More notable still: coding agents like Claude Code and Codex perform the same as those tools, because the underlying models and prompts are essentially the same.
— HamelEven 90% precision is not enough — false positives skew your direction
Shreya adds that automated eval tools top out at 80%-90% precision even in the best case, which means 10%-20% of the errors they flag are false positives, and acting on them blindly will skew your product direction. So the approach that actually works is: use the tools to find the obvious errors fast, then review the data by hand — validating what the tools found, filling in what they missed, and making sure your taste gets injected into the product. That is what opens a gap between you and your competitors.
— ShreyaTrade evals for a cheaper model and cut cost by 100x
On the depth of the course, Shreya says that beyond error analysis and productionizing, it also covers wiring evals into CI/CD, monitoring long-term drift, and adversarial evals (guarding against tenant data leakage). The most practical part is the improvement module: using evals to guide a model switch — when you move to a cheaper model, you use evals to optimize its accuracy, and they have in practice cut costs for students and clients by 100x. That is an engineering capability on top of pure eval skill.
— ShreyaIn their own words · checked verbatim
Now Claude is very very bad at coming up with bottomup evals. That's all you
Shreya6:03
we joke that writing is the final boss for LLMs
Hamel11:05
the interface that it comes up with is going to be so much better than me looking at my data in say Google spreadsheets or something
Shreya16:08
the hardest part of eval is error analysis which is coming up with this rubric
Shreya34:22
There's there is no world in the future. Even if you have AGI, if you're building a product, you have to look at your data.
Hamel39:25
It's really hard for humans to upfront think of like from top down like what are all the things that are good and bad. It's like almost impossible.
Hamel47:31
Figures
| Total suggestions auto-generated by the error discovery skill | 361 | 34:22 |
| Staccato fragments (sentences of 4 words or fewer) | 249 | 34:22 |
| Time to build the review interface | 15 minutes | 24:15 |
| Peak precision of automated eval tools | 80%-90% | 50:33 |
| Cost reduction achieved by optimizing with evals | 100x | 52:35 |
| Human comments needed to trigger auto-labeling | about 10 | 28:19 |
Glossary
- bottom-up evals
- Eval criteria formed by a human distilling failure modes out of a large volume of real outputs.
- top-down evals
- Eval criteria set in advance from the nature of the task and domain knowledge, independent of the data.
- LLM judge
- Having a model return a pass/fail verdict on an output against explicit criteria.
- error discovery
- The workflow of reviewing data to identify failure modes and harden them into eval criteria.
- staccato fragments
- Broken sentences of fewer than four words — a common filler style in AI writing.
- slop
- Empty, formulaic, low-information boilerplate in AI generation.
How to listen
Engineers and founders building products on LLMs — chatbots, writing tools, agents especially — and anyone currently stuck on how to design an eval system.
[08:04] Ad break, skippable.