Compression Is Intelligence: Prediction and Compression Are Two Sides of the Same Coin
Information theory says prediction and compression are mathematically equivalent, so the pretraining objective of a language model can be re-understood as building the most efficient text compressor — not merely predicting the next token.
The argument · tap a timestamp to hear it
The pretraining objective is really compression
Karpathy points out that large language model pretraining is usually described as ‘next token prediction’, using cross-entropy loss, and the term cross-entropy itself comes from information theory. One result of information theory is that prediction and compression are mathematically equivalent — two sides of the same coin. So you can restate the pretraining objective entirely as: not predicting the next token, but building the most efficient text compressor. The value of this restatement is that it makes clearer what cross-entropy actually is and why it is used.
— Andrej KarpathyThe robot example: probability determines code length
A warm-up with a robot far away on the moon: the commands are only up, down, left, right, but the distribution is uneven — half are up, a quarter down, an eighth left, an eighth right, and each command is independent. The naive student gives every command a fixed two bits, averaging 2 bits; the smart student uses a variable-length code: 0 for up, 10 for down, 110 for left, 111 for right, for a weighted average of 1.75 bits. The key constraint is that no codeword may be a prefix of another codeword — this is called a prefix-free code — so that the robot can decode unambiguously.
— Andrej KarpathyThe output of perfect compression looks like random noise
A third, ‘impractical’ student proposes: random noise is incompressible, so the bitstream produced by a perfect compression algorithm should be indistinguishable from random noise. If the compressed n-bit bitstream really looks like random noise, then all 2 to the n messages must be equally likely, each with probability 1/2^n. Working backwards, a message that takes n bits under a perfect scheme means its probability is 2 to the minus n. Take the log and negate it, and you get n = -log₂(p). This negative-log expression is the fundamental formula of information theory, and Shannon's definition of ‘the information content of an event’.
— Andrej KarpathyInformation content is the lower bound on compression
Perfect compression is not always possible, because the probabilities of real messages are usually not integer powers of 2. The more general statement is: the information content of a message gives a lower bound on how short it can be compressed — at least when averaged over all possible messages. You can absolutely overfit a compression algorithm to one specific case. In the robot example the probabilities are too clean, all powers of 2, so the information content of each symbol is exactly an integer number of bits; but in natural language the probability of each letter depends heavily on context, and none of them are powers of 2, so the computed information content is all fractional.
— Andrej KarpathyShannon had his wife guess the next letter
Shannon started with n-gram statistics, but long strings simply do not appear in a corpus, so the statistics break down. So he switched models: his wife Betty. Shannon took a book and had Betty guess the next letter one at a time; when she guessed wrong he wrote down the correct letter, and when she guessed right he drew a dash. The transcribed text has fewer letters but the same information content — because if he could replicate a Betty, he would only need to give her this reduced text to restore the original. In a 1950 paper he changed to recording how many guesses were needed to get it right, and from that inferred the implicit probabilities the human brain assigns to real letters. When he analysed language he was not purely doing data analysis; he was probing an underlying language model, namely the brain of his subject.
— Andrej KarpathyEntropy is the average information per symbol
Entropy asks for the average information content per symbol in a signal. Because under perfect compression codeword length equals information content, entropy is essentially asking about the compression limit, giving a lower bound on how efficiently a signal can be compressed. For any probability distribution, entropy is the sum over all p of p times negative log₂(p). Visually, draw the probability distribution as stacked horizontal bars whose widths are the probabilities, put a rectangle on each bar with height equal to negative log₂(p), and entropy is the sum of the areas of all the rectangles. The more uniform the distribution the higher the entropy, the more skewed the lower, and the more kinds of symbols the higher too.
— Andrej KarpathyThe noiseless coding theorem: the limit is achievable
A more precise understanding of entropy is: it describes the minimum number of bits per symbol needed to encode a message drawn from this distribution. This is the central theorem of Shannon's 1948 paper that founded information theory — the noiseless coding theorem. It says no encoding can be more efficient than this limit, and, more strongly, Shannon proved you can always get arbitrarily close to it. This expression only applies when every symbol follows the same distribution; natural language does not satisfy that, and needs the more general entropy rate, i.e. an average over all possible messages — but that is almost never something you can compute cleanly.
— Andrej KarpathyEnglish is about one bit per character
Shannon estimated the entropy of English at roughly one bit per character, provided the subject has at least 100 preceding letters of context. That number is crazy, because it means English can be compressed down to a single yes/no answer per character. Karpathy says part three will show that algorithm: if you are allowed to use a high-quality language model for encoding and decoding, in practice you can get quite close to this limit. He also previews that part two will cover cross-entropy and its use in training large language models, plus distilling large models into small ones, and why GZIP can recover structure across different languages.
— Andrej KarpathyIn their own words · checked verbatim
one of the conclusions of information theory says that prediction and compression are mathematically equivalent. They turn out to be two sides of the same coin.
Andrej Karpathy0:00
we can't really answer this question, or at least Shannon couldn't, without necessarily engaging with some notion of intelligence.
Andrej Karpathy3:05
random noise should be incompressible, and therefore, a perfect compression algorithm should produce a bitstream that's indistinguishable from random noise.
Andrej Karpathy10:13
the information of a message gives you a lower bound on how much it can be compressed, at least when you average overall possible messages.
Andrej Karpathy17:19
he wasn't just doing pure data analysis looking through books. He was trying to probe at an underlying model of language, namely the interviewee's brain.
Andrej Karpathy21:27
This question of measuring the average information per symbol is basically asking about the limit of compression.
Andrej Karpathy24:29
no encoding can ever be more efficient than this limit, and even more strongly, he showed that it's always possible to get arbitrarily close to this limit.
Andrej Karpathy28:35
Figures
| Estimated entropy of English | about 1 bit/character (when the subject has at least 100 preceding letters of context) | 29:36 |
| Average codeword length of the smart student in the robot example | 1.75 bits/command | 5:10 |
| Average codeword length of the naive student in the robot example | 2 bits/command | 4:07 |
Glossary
- prefix-free code
- An encoding in which no codeword is a prefix of another codeword, guaranteeing unambiguous decoding.
- noiseless coding theorem
- Shannon's 1948 proof that entropy is the compression limit, and that it can be approached arbitrarily closely.
- entropy rate
- The average information content per symbol in a stochastic process, applicable when the symbol distribution is not fixed.
- cross-entropy loss
- The loss function used for language model pretraining, whose name comes from information theory.
How to listen
Engineers and researchers who want to build information-theory intuition, especially those who use cross-entropy loss every day but have never thought through why it looks the way it does.
The roughly one-minute virtual job fair ad at the end can be skipped.