Open-source models will win; vLLM is the Linux of AI inference
vLLM is already the most active open-source project on GitHub, but You Kaichao's real judgment is this: model capabilities cannot stay hidden for long, open-source models will ultimately win, and the inference engine is the next Linux — the moat is fast iteration, not the model itself.
The video won't play here. Listen to the audio instead:
The argument · tap a timestamp to hear it
Tuning is not the bottleneck — hardware's multi-level cache is
You Kaichao left algorithm research for three escalating reasons. First, submissions to academic conferences grew exponentially while review quality fell, making submission feel like buying a lottery ticket. Second, he found that algorithmic results depend heavily on machine learning systems: Kaiming He's ResNet rested on large-scale experiments, while a lab with small data can only "do a Daoist rite inside a snail shell." Third, friends in industry reported that what Megvii engineers struggled with most was the hardware's multi-level cache, not hyperparameter tuning. Together these three pushed him from algorithms to systems.
— Kaichao YouAn open-source project with no company behind it is bound to die
His advisor believed vLLM would surely die without company backing — just as Linux could not have gotten where it is without Red Hat, Ubuntu without Google, or PyTorch without Meta. An open-source community is not a legal entity: there is no one to sign an NDA, a big company donating compute can only hand over a single machine, and cluster-level optimization means begging everywhere and having it revoked at any moment. You Kaichao decided to start a company at the end of 2024, took a postdoc offer to wait two years for it, and kept saying internally, "if only there were a company."
— Kaichao YouThe core skill in maintaining an open-source project is daring to delete features
vLLM uses tiered governance: a few "benevolent dictators" make the final call, about a dozen core maintainers own key modules, and a few dozen committers write code. The hard decision You Kaichao made was to delete beam search, even though recommender-system users kept complaining; he judged that mainstream AI inference no longer depends on beam search, and that maintaining it would only add unmaintainable complexity. That ability to make trade-offs is the key to setting priorities in an open-source project.
— Kaichao YouThe assumption that "community users are all well-intentioned" has already broken
In 2025, ranked by contributor activity on GitHub, vLLM was the most active project. This brought a "happy problem": training bootcamps packaging students' resumes by submitting junk PRs, bot accounts filing 20 PRs in 10 minutes — breaking the assumption that "community users are all well-intentioned." Maintainers had to introduce verification mechanisms and block bots. As coding agents advance, code becomes cheap, and maintainers find it faster to rewrite a PR themselves than to review it; the community is turning into a two-layer structure of maintainers plus users.
— Kaichao YouIf a model doesn't fit the hardware, it can't capture the compute dividend
He uses hydropower as an analogy: hardware is the natural resource, the model is the generator, and the inference engine is the power system. With the same water flow, generators designed differently differ enormously in efficiency — that is co-design. He points to the "hardware lottery": once Moore's Law fails, compute gains come from specialized chips, and a model architecture that doesn't fit the hardware can't capture the dividend. Transformer drew the GPU lottery ticket, and RoPE became its best partner because it doesn't modify the attention kernel. DeepSeek's strength is precisely that its algorithm people understand systems — for instance, the exploration of MOE from coarse-grained to fine-grained was all implemented first on its own inference engine.
— Kaichao YouA token is not a generic commodity — it bears the model's brand
Electricity is a generic commodity; you can modulate voltage and frequency to a unified standard. A token is different: you cannot convert a DeepSeek token into a Kimi token, because the token bears the model's brand. That is why model architectures are so varied, and the inference engine must support two or three hundred architectures while continuously trimming unnecessary complexity. This also explains why open-source models will win: once a model is used at scale, user data can train new models, and model capabilities cannot stay hidden for long.
— Kaichao YouFor models that deal with people, million-level context is enough
You Kaichao offers two predictions. First, the context needs of models that deal with people basically stop at the million level; only special fields like biochemistry need tens of millions or even hundreds of millions, and long tasks can be handed off to external tools. Second, open-source models will win in the end, because a model is not like a nuclear weapon that can be kept hidden for deterrence — it needs large-scale use, and use accumulates data, which replicates new models. So the moat is not model capability but fast iteration.
— Kaichao YouIn their own words · checked verbatim
If you don't start a company with us — you make a lot of money, but ten years later, if our project fails, are you happy or not happy?
你如果不跟我们一起创业。 你赚到了很多钱，但是十年之后，为我们项目失败了，你是开心还是还是不开心啊
Kaichao You1:15:25
We want to lay a solid foundation of infrastructure for this coming age of intelligence.
我们想为接下来的这个智能时代。 打好这个基础设施。
Kaichao You1:16:25
There's no way to modulate a token. You can't convert a D-model token into a Kimi token — the token bears the model's brand.
token它是没有办法去做调制的。 就是你没有办法说把一个D模型的token转化为一个kimi的 token，这个token它是带着模型的烙印的。
Kaichao You2:17:56
Figures
| Company seed round | $150 million | 0:00 |
| Gaokao score above Tsinghua's admission line | 2 points | 6:16 |
| Number of courses with a 4.0 undergraduate grade | 36 | 7:17 |
| vLLM open-sourced | around June 2023 | 38:18 |
| GitHub contributor activity ranking | No. 1 | 1:24:55 |
| Current company headcount | 30-plus, approaching 40 | 1:50:11 |
Glossary
- PagedAttention
- Managing the KV cache in fixed-size blocks to reduce memory fragmentation; the core algorithm of vLLM.
- KV Cache
- Caching the keys and values of historical tokens during autoregressive generation to avoid recomputation.
- Speculative Decoding
- Guessing multiple tokens in parallel first, then verifying them in one pass, to speed up autoregressive decoding.
- MOE
- Activating only a small number of experts per token, scaling model parameters with less computation.
- BYOC
- A model where the customer provides the machines and the company provides software and inference service, billed by token.
How to listen
Suited to founders, investors and engineers focused on open-source governance, AI Infra startups, and model-system co-design.
The first half hour on his education can be skimmed; after the 37-minute mark the open-source and co-design substance gets denser.