Andrej Karpathy
Rarely updated, but every episode rewards frame-by-frame watching
40:45Even if AI completely surpasses humans at theorem-proving, mathematics' value will barely change
Math funding comes mainly from teaching, not from proofs; once AI turns theorem-proving into raw data, the scarce skills become building theory, practicing oral presentation, and deciding who deserves academic credit.
51:54The last IMO problem AI could not solve, and its value is not in the answer
In 2025, fewer than 1% of IMO contestants got full marks on Problem 6, and every AI was stuck on it. What makes it hard is not computation — it is that you must first understand, first have taste, before you start writing. And that is exactly what reinforcement learning struggles most to train.
Why cross-entropy loss wins: it trains a text compressor
Cross-entropy loss was chosen because it can be proven mathematically to be the only function form that guarantees the loss is minimized when the model's output equals the data's true distribution — which is exactly equivalent to training the model into a text compressor that approaches the Shannon limit.
32:25Compression Is Intelligence: Prediction and Compression Are Two Sides of the Same Coin
Information theory says prediction and compression are mathematically equivalent, so the pretraining objective of a language model can be re-understood as building the most efficient text compressor — not merely predicting the next token.