Eight closed-source models now outrank the strongest open-source—but this reflects benchmark choices
AAv5 stakes evaluation weights on frontier tasks models are racing toward, not the workflows today's economy actually depends on—and this choice alone determines who looks ahead.
The video won't play here. Listen to the audio instead:
The argument · timestamps estimated from transcript position
Independent benchmarking is not a technical task—it is a political one
Pushing scores up is easy and expensive; knowing which direction to push is hard. Benchmarks no longer just ‘score’ models—they compress AI's chaotic outputs into a single number that must satisfy community intuition, withstand scientific scrutiny, and stay relevant. With trillions now at stake, that number decides market winners and shapes how labs train models. The pressure runs far deeper than mere scoring.
Behind four equal weights lies an editorial choice
AAv5 splits the intelligence index into knowledge work, coding, science, and general categories—each 25%—including AA-Briefcase, FrontierCode 2.0, Terminal-Bench-Science, CritPt v2, AA-Omniscience, and HLE-Diamond. The problem is that ‘real-world use’ is wildly uneven: open-source models are competitive at coding, but the author has less confidence in knowledge work. These weights describe the present but also represent an editorial bet on where the field should go.
Eight closed-source models now collectively outscore the strongest open-source
The new ranking shows eight closed-source models beating Kimi K3.1 Pro, the best open-source frontier model. But the author cautions: AAv5 measures where frontier models are heading, which naturally widens the gap from open-source. The real story isn't that open-source suddenly got weaker—the previous index overcredited it. This revision corrects that, not a sudden decline.
Lagging on science tasks isn't open-source's value proposition
Open-source models clearly fall behind on Terminal-Bench-Science and CritPt v2, but that was never the pitch. Open-source's real mission is two things: turn high-demand tasks into commodity pricing at hardware cost, and give enterprises the ability to own, customize, and control their AI with private data. Both take years. Today, compute is still tight and open-source suppliers are still profitable—when that margin compresses, we'll know the model works.
Strong models benchmark-optimize too—just with restraint
The author has observed a pattern: the best open-source models do optimize for benchmarks (benchmax), but carefully. The worst models cheat obviously—nobody notices. The reason is that competing across multiple domains at the frontier requires a reinforcement-learning setup that mimics the benchmark itself, something not every lab has. Some even outsource data. Next cycle—say, GLM 6 or Kimi 3.5 in a few months—open-source will show marked improvement on this index.
Hidden tests block gaming—but kill community correction
AA moved some tests to ‘private benchmarks’—hidden questions and environment—to stop targeted gaming. The direction makes sense, but the cost is you must trust the benchmark maker entirely. The community can no longer spot flaws in the benchmark itself and push for improvement, something that used to happen. With this much money involved, the author leans toward transparency and letting more eyes find bugs, rather than playing judge behind closed doors.
Top labs stopped living by these rankings years ago
Anthropic and OpenAI abandoned these intelligence indices months or years back. Their focus is now almost entirely on automating their own R&D and serving customers willing to spend heavily. Some team members still push new capability to open markets, but that's a small slice and not driven by public rankings. For mid-size labs, credibility on these indices is survival—their brand is built on ‘we keep up with the frontier.’
Who pays sets how much trust you can place
The author ranks trust levels: AA charges labs to verify results; Arena makes money running pre-launch testing for them. Neither is wrong, but both mean they're not the ecosystem's most independent voice. Nonprofits, academia, and government (like CAISI) rank highest for integrity. For frontier accuracy, though, AA and Arena are the best available. The author calls his own work ‘benchmarking the benchmarkers’—it's more sociology than science.
In their own words · checked verbatim
Hill-climbing is easy, knowing what to hill-climb is hard.
With the amount of money in the AI ecosystem today, the weight of independent evaluation is even higher – you dictate the winners and losers of a trillion dollar market.
This plot shows 8 closed models beating the best to-be open-weight model, Kimi K3.1 Pro (which isn’t colored blue).
Open models’ demise from this seems greatly exaggerated, but I do think the previous index was flattering the open weight models too strongly.
The best open models do benchmax a little. The worst open models benchmax a lot (but don’t get a ton of attention anyways).
Their focus is to a large majority on supporting internal uses – like AI R&D automation – and the use-cases of customers who are spending billions of dollars.
There is enough money on the line, where I’d rather see transparency in benchmarking and enabling many eyes to find bugs.
The entities I trust the most to be fair on evaluations are non-profits, academics, and government institutions like CAISI.
Figures
| Closed-source models beating Kimi K3.1 Pro | 8 | 1:57 |
| AAv5 four-category weights each | 25% | 1:57 |
| AA-Briefcase weight within knowledge work | 15% | 1:57 |
Glossary
- benchmax
- Tuning training data distribution to the benchmark itself to inflate scores while sacrificing real generalization.
- private benchmarks
- Tests with hidden questions and environments, designed to prevent targeted optimization but also blocking community fact-checking.
- Intelligence Index
- A weighted composite of multiple evaluations reduced to a single score for comprehensive model ranking.
How to listen
Founders and investors using benchmarks to pick models or tell fundraising stories; engineers tracking how open-source models turn commercial.