Why cross-entropy loss wins: it trains a text compressor
Cross-entropy loss was chosen because it can be proven mathematically to be the only function form that guarantees the loss is minimized when the model's output equals the data's true distribution — which is exactly equivalent to training the model into a text compressor that approaches the Shannon limit.
The argument · tap a timestamp to hear it
gzip can draw the language family tree
A 2002 paper used only gzip, a general-purpose file compressor, to automatically cluster texts in different languages and even reconstruct the genealogical tree among language families, all without relying on any linguistic prior knowledge. The method is to append a fragment of document B to the end of document A and compress the result, then compare this with the file size of compressing A alone — the difference is essentially asking "how well does a compression scheme optimized for A handle a fragment of B," and the same idea was later used for text authorship attribution.
— Grant SandersonThe cost of an old code meeting a new distribution
If a coding scheme was optimized for some distribution Q (say, assigning probabilities 1/2, 1/4, 1/8, 1/8 to the robot commands up/down/left/right and designing corresponding bit lengths), but the actual data has become a new distribution P, continuing to use the old code wastes bits. Re-weighting by the new distribution to compute the average number of bits this old code would use gives a value called "the cross-entropy of Q relative to P" — it measures exactly how a compression scheme optimized for one situation performs in another.
— Grant SandersonThe minimum of cross-entropy is entropy itself
Picture the cross-entropy formula as a curve drawn by fixing P and letting Q vary. This curve always attains its minimum where Q equals P, and that minimum is exactly the entropy of P. This is because the question cross-entropy asks — "how does a code optimized for one setting perform in another" — can only reach theoretical optimal compression efficiency when the two settings are identical. This property is the pivot of the entire argument that follows.
— Grant SandersonThe pretraining loss is the average information per token
When training a language model, for every next token that actually appears in the training data, take the negative log of the probability the model assigns to it, then average over all tokens — that is the entire pretraining loss function. The "smarter" the model, the higher the probability it gives to tokens that actually occur, and the lower the corresponding negative log value (information content / degree of surprise); a poorly trained model is unconfident about everything and is "surprised" everywhere.
— Grant SandersonWhy it must be the negative log, not just any decreasing function
The common explanation is "in the cross-entropy formula everything except the term for the true token is zero, so what's left is naturally the negative log" — but this doesn't explain why cross-entropy was chosen in the first place. A more solid argument: if you only require some function F to satisfy "the average loss over all training samples is minimized only when the model's output distribution equals the data's true statistical distribution," then constrained optimization with Lagrange multipliers proves F must be a logarithmic function — the choice isn't actually yours, the negative log is forced by mathematics, and the average loss formula at that point is exactly the definition of cross-entropy.
— Grant SandersonDistillation: feed the small model the big model's entire probability distribution
Ordinary pretraining gives full weight only to "the one token that actually occurred" at each position, and zero to everything else; distillation instead has the small model match the full probability distribution given by the big model at each position, with the loss defined as the cross-entropy of the small model's distribution relative to the big model's. A single training sample thus conveys far more information than in ordinary pretraining — like learning chess not by seeing only the one move actually played, but by having a stronger player explain every candidate move and its weight at each step.
— Grant SandersonCross-entropy loss is equivalent to training a text compressor
From the compression angle, cross-entropy naturally measures "how wasteful it is to move one compression scheme to another situation"; from the language model training angle, the loss function is the average information per token. As long as you can turn a general predictor (such as a language model) into a compressor — encoding the text with roughly as many bits as the text's information content in the model's view — the two threads line up completely: training a model with cross-entropy loss is equivalent to training it into the best possible text compressor.
— Grant SandersonKL divergence: how many bits the encoding wastes
Cross-entropy minus entropy gives the KL divergence (Kolbach-Leibler divergence), which specifically measures "with a poorly optimized coding scheme, how many bits are wasted per symbol on average." In machine learning, KL divergence is often treated as a "distance" between two distributions: zero when the two distributions are identical, larger as they differ more, but it is asymmetric — the KL divergence of Q relative to P is not equal to that of P relative to Q, which is also why using cross-entropy rather than KL divergence in distillation is a choice worth pondering.
— Grant SandersonIn their own words · checked verbatim
But whenever you see the same formula pop up in two separate contexts, it's math's way of kind of winking at you and hinting at a connection.
Grant Sanderson2:05
for an optimal code, the number of bits that you allocate to a given symbol looks like the negative log base 2 of the probability for that symbol showing up.
Grant Sanderson4:09
A very nice way to think about the loss function we use in this case is that you measure the average information per token from the model's perspective.
Grant Sanderson17:28
You might think this loss function would go by a name like information loss, or log loss, but in fact, people call it cross-entropy loss, which at first might sound really strange, given that nowhere in the explanation have we actually used the cross-entropy formula.
Grant Sanderson20:33
you would like it to be the case that the full average loss over all of the examples is minimized only when the model's output matches the statistics that show up in the data, there's actually a really nice mathematical argument for why F has to be a logarithm.
Grant Sanderson24:38
it's the difference between trying to learn chess just by watching a game versus having someone better than you talk through all the possible good moves and how heavily they would weight them at each new turn.
Grant Sanderson27:42
And once you understand this, it makes clear how using cross-entropy loss is actually equivalent to training the model to be the best possible text compressor.
Grant Sanderson29:48
In the language of compression, you would think of it as describing how many bits per symbol are you wasting by using a poorly optimized code.
Grant Sanderson31:49
Figures
| Cross-entropy of the old robot-command code when facing a new distribution | 2.625 bits/symbol | 6:13 |
| Cross-entropy of a 90/10 skewed distribution code used on a 50/50 uniform distribution | about 1.74 bits/symbol | 9:20 |
| Publication year of the paper "Language Trees and Zipping" | 2002 | 0:00 |
Glossary
- cross-entropy
- The average number of bits per symbol required to encode data from distribution P using a code optimized for distribution Q
- KL divergence
- Cross-entropy minus entropy; measures how many bits a coding scheme wastes relative to the optimal scheme
- distillation
- Training a small model using the full probability distribution output by a large model rather than a single correct answer
- Lagrange multiplier
- A mathematical method for finding extrema of a function under constraints
- Shannon limit
- The theoretical lower bound on the optimal number of bits achievable in lossless compression
How to listen
Engineers and researchers interested in the intersection of loss function design, information theory and deep learning, especially those who want to understand where the name "cross-entropy loss" comes from.
The rigorous derivation with Lagrange multipliers at 24:38 can be skipped without affecting the conclusions that follow.