Quantizing More Layers Is Actually More Accurate: Errors Cancel Each Other Out
Pick the right layers and quantizing more preserves fidelity better — the errors cancel each other out. Likewise, a 10x speedup isn't a single breakthrough; it's quantization, a speculator, and PD disaggregation each doubling on top of the others.
The video won't play here. Listen to the audio instead:
The argument · tap a timestamp to hear it
Latency isn't a model property, it's how well the speculator guesses
When a 200k-token request comes in, the first question isn't compute, it's "have you sent this passage before." The system first does cache-aware routing, picking a replica that both has an idle prefill worker and has already cached part of the prefix, skipping prefill as much as possible; on some models prefill and decode are already split across two GPU groups, the former producing the KV cache and first token, the latter doing token-by-token decoding. The key is that the speculator sitting in front is trained on traffic — it assumes you're writing code, so draft-token acceptance is high; if you're actually asking it to summarize the entire Harry Potter series, the same system gets slower. Latency isn't an intrinsic property of the model, it's a function of how well the traffic profile matches the speculator.
— Philip"Supporting a new model" and "being production-ready" are two different things
"Supporting a new model" has two completely different meanings: being able to emit tokens, and having a production-usable API. The former isn't hard — vLLM/SGLang maintainers often get weights early, and the model side submits PRs themselves. The latter requires redoing everything on a private stack: quantizing weights to NVFP4 to saturate Blackwell, and calibrating the quantization to confirm there's no intelligence regression; running inference with real weights to get hidden states and retraining the speculator; then adding runtime support for the new architecture (GLM-5.2's DSA sparse attention was borrowed from DeepSeek). Because the model API side has zero data retention, they don't know the actual traffic, so they can only use public datasets to approximate "coding and agents are the mainstream" to train a general speculator. Even if K2.5 to K2.6 is just continued post-training, this process can't be avoided.
— PhilipVision can be transplanted without losing a point of the backbone
Open models can undergo organ transplants. Haley on the team hooked Kimi's vision encoder onto GLM-5.2: both the encoder and backbone weights frozen, training only the few-million-parameter projector in between — the cost of touching backbone weights is that the model may get dumber at other capabilities. The first version was trained only on "describe this image" data and didn't learn it; after switching to pairing each image with a set of discriminative questions (is there a bird in the image, is there a scientist), grokking appeared. The result is MMLU Pro 56%, not frontier, but when there's no image input it skips the encoder entirely, and the original GLM-5.2 quality loses nothing. So you get a model assembled from Kimi's eyes, GLM's weights, and DeepSeek's attention.
— PhilipMode collapse isn't the weights' fault, it's the cluster's fault
On GLM-5.2 and DSV 4 there were mode collapses where the same token was output repeatedly (most commonly S), even at temperature 0.9. Their judgment: this isn't a weights problem. The same weights on a different inference engine (SGLang to vLLM), or merging in changes from NVIDIA's new TensorRT-LLM image, and the problem disappears. A more extreme case was the same model collapsing on one cluster but not another, because the former's inter-node KV cache transfer went over a slower interconnect, exposing a race condition from a missing synchronization barrier in the kernel — the final fix was to switch clusters. There's also a production fallback: if the same token appears four or more times in a row, cut it off and rerun, but exclude special characters like table separators.
— AliPick the right layers and quantizing 20% more is actually more faithful
The industry default is "more quantization, more loss," but their conclusion is that it's not necessarily so. Quantization is lossy compression, and the skill is all in choosing layers (in image models, modulation layers and out projection aren't quantized). Research intern Joshua's paper gives the approach: predict which layers' quantization errors will cancel each other out — one biased right, one biased left — and then specifically quantize those layers. The evaluation criterion also changed: instead of benchmark scores, look at the KL divergence between the quantized model's and the full-precision model's logits distributions; the closer the distributions, the more likely the behavior matches the original model. The result is 20% more quantization than NVIDIA's (and thus about 20% more throughput) while being more faithful. Note this isn't making the model stronger, just canceling out loss.
— Ali10x isn't optimized out, it's bought with hardware
The 10x breakdown is concrete: a trillion-parameter model on an unoptimized off-the-shelf engine runs at about 30-50 tokens/second (here meaning tokens per second in the latency sense, which the industry should call ITL). Getting to 300-400 requires simultaneously: the best hardware, a trained speculator, quantization done, a high cache hit rate, and small batch and parallelism configurations tuned for latency rather than throughput. The itemized gains are BF16→FP8 about 30-40%, FP8→FP4 another 30-40%, speculator about 2x, PD disaggregation about 2x, plus double-digit percentages from runtime and new kernels. But if you fix the same hardware and same card count, pure inference optimization is only 2-4x — the difference is bought with hardware, and 10x shows up more between the worst and best providers on OpenRouter.
— PhilipMega kernels can't save communication that must cross cards
Ali is bearish on mega kernels: fused kernels saving launch overhead sounds nice, but under tensor parallelism half the matrix is on another card, and nonlinearities like softmax still require communication, which fusion can't save; moreover, he's dealt with companies doing fused mega kernels, and what actually runs in production is still TensorRT-LLM and modular kernels. In NVIDIA's post unveiling Rubin, this GPU's design simply makes mega kernels unnecessary. Following this trend he asks: GPUs add dedicated tensor cores and TMA every generation, you're increasingly just orchestrating tiles rather than controlling threads, and Rubin relative to T4 is already nearly an ASIC, so why be bullish on AI ASIC companies? Swyx's rebuttal is: vertical integration makes in-house chips reasonable for model makers — by Martin Casado's math, taking 50 billion out of a 500 billion training budget for an ASIC pays off if it buys more than 10% efficiency; and what ASIC companies are really doing is interconnect and memory allocation, which is the 10x to 1000x bottleneck.
— AliOpen video models are 100x cheaper and nobody uses them
The economics on the video side are the opposite of LLMs. The open-closed gap for LLMs has nearly disappeared, but for video it's night and day — Ali's example: even if he pushed the cost of generating a three-hour movie down to $10, 100x cheaper than closed source, the customer still faces a $1000 absolute number, so they still use Veo and Kling for the whole film. Less demand → less innovation → fewer open checkpoints, and Wan 2.7 is no longer open source. The technical bottleneck is the quadratic attention term: a 480p video at 16 frames/second, 5 seconds compressed into latents is still 35000 tokens, and this is O(n²), so doubling the duration quadruples the cost. So there are only two paths: sparse attention (each token attends only the first 12.5%, with a noticeable drop in quality), or autoregression — but today all autoregressive video models are poor quality, and stitching 5-7 second clips drifts segment by segment, going dark to a black screen after twenty-some seconds.
— AliIn their own words · checked verbatim
It is possible that the model in which I quantized more information is going to perform better because the quantization errors have canceled out.
Ali30:28
When we publish optimizations, it’s 20%, it’s 100% it’s 200%. So there’s still probably like a lot further to go, honestly. Like you’ll, you’ll know that inference is pretty much solved when researchers start publishing about how they got 1% faster at something.
Philip31:57
With local AI, it’s how do I fit this model onto my hardware and then make it less dumb? And with data center influence, it’s how do I load this model and then make it less slow?
Philip46:17
The downsides is every single autoregressive video model is s**t. It’s just terrible quality.
Ali1:17:10
So that’s the fundamental problem with trying to change a fact in an MLP within the weight. KV cache compaction fixes that.
Ali1:38:28
Figures
| GLM-5.2 score after vision retrofit | MMLU Pro 56% | 17:25 |
| Baseten quantization increment over NVIDIA | 20% more quantization, about 20% higher throughput | 29:27 |
| Trillion-parameter model unoptimized baseline speed | 30-50 tokens/second | 34:31 |
| Target speed after full optimization | 300-400 tokens/second | 35:38 |
| Itemized gains from precision reduction | BF16→FP8 about 30-40%, FP8→FP4 another 30-40% | 38:23 |
| Attention token count for a 5-second 480p video | about 35000 tokens | 1:15:25 |
Glossary
- speculative decoding (spec dec)
- A small draft model guesses several tokens in a row, and the large model verifies them in one forward pass, accepting or rejecting.
- PD disaggregation
- Splitting input processing that produces the KV cache from token-by-token generation across two GPU groups.
- NVFP4
- A 4-bit format natively supported by Blackwell, where each parameter takes only half a byte.
- mega kernel
- Fusing multiple GPU kernels into one, saving launch overhead and intermediate data movement.
- expert parallelism
- In MoE models, placing entire experts on a single card, with far less inter-card communication than tensor parallelism.
- KV cache compaction
- Compressing existing KV without changing weights; Ali sees this as a viable path to continual learning.
How to listen
Infrastructure engineers hosting open models on their own rented GPUs, technical leads deciding between self-hosting and APIs, and teams doing video generation.
The intern jokes from 0:00–1:00; the music diffusion segment after 1:21 basically reaches no conclusion.