K3 Has No Disruptive Innovation, but Pushes Activated Parameters to the 100B Scale
Sun Yutao reads the Kimi K3 technical report, pointing out that it has no single breakthrough but rather fuses existing techniques such as linear attention, hybrid attention, and dynamic load balancing to achieve activated parameters at the 100B scale; he judges that large models will see no more Transformer-level innovations, only incremental improvements.
The video won't play here. Listen to the audio instead:
The argument · tap a timestamp to hear it
Architecture improvements no longer chase performance, only inference efficiency
Sun Yutao points out that in the era of large models, parameter count is the primary factor for performance, and the performance gains from architectural improvements are relatively small; however, different architectures show huge differences in inference efficiency, which directly determines deployment cost. Industry research has therefore shifted from performance-oriented to efficiency-oriented, focusing on inference efficiency. His doctoral research, starting in 2023, mainly addresses inefficiencies in large-model inference, including linear attention and hybrid attention.
— Sun YutaoHybrid attention is an engineering compromise, yet capability is not lost
Although hybrid attention is an engineering compromise architecturally, it is not a compromise in model performance. As long as a certain proportion of full attention is maintained, the model can achieve lossless or even better upper bounds—meaning engineering gains without sacrificing the capability of full attention. This experimental conclusion has led to the widespread adoption of hybrid attention. However, Sun also notes its limitation: the inference speedup is proportional to the hybrid ratio. For example, when full attention occupies one quarter, the speedup is also limited by that, representing a constant-factor improvement, which is why he later turned to shared-KV architectures.
— Sun YutaoParameter count remains the most essential variable determining intelligence ceiling
Kimi K3's core selling point is effective scaling: total parameters of 2.8T, activated parameters around 100B, and context length reaching the 1M scale. Sun emphasizes that larger models have higher intelligence ceilings, and parameter size remains the most essential variable for model intelligence. At release, K3 was much larger than other domestic open-source models and was one of the earliest models to successfully reach this scale.
— Sun YutaoMLA is nothing new, just an equivalent implementation of MQA
Regarding V4's abandonment of MLA, Sun believes MLA is essentially an equivalent form of MQA, bringing nothing new but a better implementation path. MLA's benefits mainly come from parameter design; its computation is far greater than actually needed, wasting compute during inference. Lu Fuli felt that on inference chips, MTP could utilize compute more effectively, so V4 switched to MTP. He thinks this is not a matter of right or wrong but a choice. K3 retains MLA because K2's priority was getting things to work; if it works, don't change it. It may not continue to be used in the future.
— Sun YutaoWhen splitting two matrices, normalization must be inserted between them
Sun proposes an architectural design principle: two matrices multiplied together can be merged into one in terms of expressiveness, but they are not equivalent in optimization properties, often leading to training instability. Therefore, if two matrices must be written separately, a normalization must be added between them to stabilize training. This occurs in MLA and the two-layer matrices in FFN. He also mentions that using clip to directly truncate intermediate activations is a crude approach; a smoother approach is to constrain intermediate activations to a mathematically controllable range via normalization.
— Sun YutaoToday's Transformer is already a Ship of Theseus
Sun believes that current model training does not involve a huge, so-called disruptive innovation but rather fuses many previous works to find better combinations. He uses the Ship of Theseus as a metaphor for the Transformer: the 2017 ship was assembled from parts like attention and MLP, and over eight or nine years of sailing, some parts have been continuously replaced, so today's similarity to the original is very low. Whether it is still called a Transformer is a philosophical question, but from an engineering perspective, people still call it that. Technology improves gradually, driven by everyone.
— Sun YutaoWSD does not reduce tuning difficulty; K3 chose a simpler scheme
Sun introduces the WSD learning rate schedule proposed by MiniMax: during the cooldown phase, the model learns fastest, so high-quality data can be concentrated there, achieving better results than uniformly distributing data. WSD also allows freely switching the actual token count during training, making it easier to adjust release timing. However, the K3 team believes WSD's tuning difficulty is not reduced because the optimal cooldown ratio still depends on the total token count; they chose a simpler scheme with only two variables (total token count and peak learning rate), which is easier to tune.
— Sun YutaoLarge models may never see another Transformer-level innovation
Sun offers a bold claim: large models may not see innovations like the Transformer itself; everything after will be incremental improvements. He believes that as long as a capability can be clearly defined, it can be achieved; pure language models reaching AGI is not a big problem, but AGI involving interaction with the real physical world still has a long way to go. Model size cannot grow indefinitely because the amount of information that can be aggregated on the human internet is finite, so models need not be infinitely large. He also mentions that because the room for improvement in large models is limited, he personally has turned to exploring world models.
— Sun YutaoIn their own words · checked verbatim
I think K3 is quite bold—on the basis of existing open-source models, they aim to raise it by an order of magnitude.
我觉得K3它还是比较有魄力，就是在已有的开源模型这个赛基础上，他们要提升一个数量级。
Sun Yutao1:55:29
My view is that scientific research does not have leapfrogging improvements. It's just that people tend to mark certain nodes of incremental improvement as milestones.
我其实的一个观点就是科学研究是不存在节约性的提升的。只是说大家习惯把这个技术的间立性提升的某些节点会成为一个mles。
Sun Yutao1:56:29
Let me give my judgment—my judgment is a bold claim. My paradox is that large models may not have innovations like the Transformer itself; everything after will be incremental improvements.
我说一下我的判断啊，我的判断比较暴论，我的悖论就是大模型可能没有太本人的创新了，后面都是一些改良性的进进模。
Sun Yutao2:01:46
Figures
| Kimi K3 total parameters | 2.8T | 19:12 |
| Kimi K3 context length | 1M (one million) | 20:13 |
| K3 activated parameters increase over K2 | 3x+ | 1:55:29 |
Glossary
- MLA / Multi-head Latent Attention
- An attention mechanism that compresses KV into a latent representation, reducing cache overhead during inference.
- WSD / Warmup-Stable-Decay learning rate schedule
- A three-phase schedule where the decay phase learns fastest, allowing concentration of high-quality data.
- MTP / Multi-token Prediction
- A method that lets the model predict multiple future tokens simultaneously, combined with speculative decoding to speed up inference.
- EP / Expert Parallelism
- Distributing MoE experts across different GPUs, which involves load balancing and communication optimization issues.
How to listen
Large-model architecture researchers, pretraining and inference engineers, and anyone wanting to understand Kimi K3's technical decisions and the latest industry trends in architecture fusion.