The world is too loud. Read what matters.

Andrej Karpathy

Compress text to under 1 bit per character: arithmetic coding reveals compression is prediction

Any text predictor can be transformed into a nearly lossless compressor via arithmetic coding, and conversely any compressor can recover the prediction distribution—compression rates fundamentally depend on how well the language model predicts.

Information theoryLanguage modelsArithmetic codingPretrainingCompression
Uses visualization to clarify the mathematics of arithmetic coding, extends to the essence of pretraining and the‘compression is intelligence’argument. High information density but theoretical; suited for those wanting to understand language model training principles.

The argument · tap a timestamp to hear it

6:05

Any text maps to a unique real number on the number line

The algorithm encodes any text into an extremely specific real number between 0 and 1. The idea is to view every possible string as occupying a range on the number line, where the range's width equals the probability the model assigns to that text. The more common a string, the wider its range; given any number falling within that range, plus the text length, a receiver can uniquely recover the original text. The entire compression algorithm's core is finding a number in that range with the shortest binary representation.

— Grant Sanderson
11:08

Decoders need an identical model to recover the prediction

Decoding mirrors encoding: the receiver first runs the model on empty context to get the first distribution, then infers the first character by seeing which character's range the received number falls into. The character is then fed back into the model to get the next distribution, and the process continues checking where the number falls—repeating until reaching the character count the sender specified. The critical requirement is that both sides must use exactly the same model; otherwise the distributions won't match and the entire scheme fails.

— Grant Sanderson
15:12

Required bits equal Shannon's information measure exactly

To guarantee representing a number within a range of probability p using k bits or fewer requires 2^-k < p; rearranging shows k must be at least -log2(p)—precisely Shannon's definition of information. This means arithmetic coding is in an approximate sense a concrete implementation of the Shannon limit: the compressed message size nearly equals the text's information content as seen by the model, with only some rounding error and fixed overhead for recording message length.

— Grant Sanderson
22:21

Pretraining is fundamentally about minimizing compressed bit count

A model's probability for text can be expanded as a product of successive single-character prediction probabilities; taking the negative logarithm converts it to a sum of‘costs’—this is cross-entropy, the loss function minimized during pretraining. Thus pretraining can be reframed: the goal is not training a‘next-word predictor’but rather training‘the best text compressor’—the number to minimize is simply how many bits each training sample can be compressed to.

— Grant Sanderson
23:21

Larger models compress highly predictable text far more aggressively

Encoding a mathematics history text with the 3.7 million-parameter NanoGPT requires 254 bits—roughly 2 bits per character, double Shannon's estimate. Switching to the open-source Qwen model with 500 million parameters, the same text's last two words require only 5 bits, compressing the whole to under 1 bit per character. For highly predictable text like the U.S. Constitution preamble, Qwen barely needs additional bits after the opening phrases—the entire text uses only about 10 bits, two orders of magnitude better than the Huffman encoding demonstrated at the start.

— Grant Sanderson
31:25

Fair compression rates must include the model's own size

Without including the decoder program's (including the model itself) size in the final compressed total, you can cheat: hardcode Wikipedia into the decoder so the alleged compression payload can be arbitrarily small. The Hutter Prize accounts for this by including decoder size in its scoring precisely to prevent such cheating. By this metric, of two models with equal predictive ability, the one with fewer parameters compresses better—echoing the intuition that‘clearer understanding depends on fewer universal principles, not massive memorized facts’.

— Grant Sanderson
33:26

Learning speed during adaptation counts as part of compression rate

If during encoding and decoding the model backpropagates on each new token and continuously fine-tunes its weights, as long as updates are deterministic and perfectly synchronized on both encoding and decoding sides, both parties can maintain prediction consistency. Then the final compression rate depends not only on the model's static predictive ability but also on how fast it adapts to new context—better learning rates yield better compression. This provides a third pillar for‘compression is intelligence’: effective compression algorithms need models that both predict well, are small in size, and learn fast.

— Grant Sanderson

In their own words · checked verbatim

each time a probability gets chopped in half, the information content increases by one bit

Grant Sanderson2:02

this quantity, the negative log p that we wrote down, Is not actually the Shannon information of the message, because this value p is representing the probability of the text according to the model, not the true probability of seeing that text

Grant Sanderson20:18

a mathematically equivalent, but I think psychologically distinct, framing is to say that the objective is not prediction per se. It's to create the best possible text compressor

Grant Sanderson22:21

isn't it kind of disingenuous to praise an algorithm for compressing this passage so much if it's just calling out to some massive model that's basically memorized that passage somewhere in its hundreds of millions of weights

Grant Sanderson24:21

if you actually wanted to send somebody a gigabyte of Wikipedia in a compressed package, the raw data stream coming out of that arithmetic coding would be useless if you didn't also give them the means of decoding it

Grant Sanderson31:25

the better the learning rate, the better the compression

Grant Sanderson33:26

I was just genuinely surprised that compression takes off many more boxes than I would have guessed the first time that I ever thought about the seemingly innocent task of making files smaller

Grant Sanderson35:27

Figures

NanoGPT parameter count3.7 million22:21
Sample text encoding with NanoGPT25423:21
U.S. Constitution preamble with Qwenabout 1023:21
Bits to encode word "math"1015:12

Glossary

arithmetic coding
An encoding method that transforms any prediction model into a near-optimal lossless compressor.
Hutter Prize
An award for algorithms that compress 1GB of Wikipedia to the smallest size, including decoder program volume.
cross-entropy
A measure of the difference between a model's predicted distribution and the true distribution; the loss function minimized during pretraining.
asymmetric numeral system
An alternative method proposed in 2013 to convert predictions into compression, using integer rather than real number encoding, with faster speed.

How to listen

Who it's for

Engineers and AI researchers interested in understanding the essence of language model pretraining, information theory, and compression algorithms.