Residual Connections: Deep Learning's Most Underrated Simple Hack
ResNet is the most cited paper of the 21st century, but its real contribution isn't making networks deeper — it's inventing the residual stream, a kind of working memory that lets a model store and edit information, and that fundamentally changed how we understand neural networks.
The argument · tap a timestamp to hear it
Deeper networks perform worse, not better
In early 2015, Sun Jian's team used careful initialization to push a network to 30 layers, but performance got worse than 14 layers (error rate 16.59% vs 13.34%). In theory, a 30-layer model could match a 14-layer one at minimum via identity mappings, but the optimizer couldn't find such a solution. This hinted the problem wasn't network capacity but the training process itself.
Shattered gradients are the root disease of deep networks
By visualizing the loss landscape, they found that as depth increased, the loss surface of early layers became extremely rugged and gradient directions turned as random as white noise — the ‘shattered gradients problem’. Gradients are the core learning signal; once they fail, deep networks can't be trained effectively.
Residual connections: an almost absurdly simple fix
Sun Jian's team's solution was laughably simple: between every two layers, add the input tensor directly to the output, forming a skip connection. This gave activations and gradients a direct path, effectively eliminating the shattered gradients problem. A 74-layer network with skip connections jumped from 38.9% to 72.6% accuracy, sweeping the major 2015 vision competitions.
ResNet overturned hierarchical representation theory
A Cornell team found that deleting or even shuffling ResNet's layers had little effect on performance. This contradicted the then-mainstream hierarchical representation theory — which held that each layer builds on the previous one, so deleting any layer should be catastrophic. It forced the field to rethink: ResNet may not be learning hierarchical features but iteratively refining a ‘residual stream’ that runs through the whole network.
The residual stream: modern AI's hidden backbone
Straighten out the skip connections and you see an uninterrupted stream of data from input to output, with each layer merely making incremental edits on top. This ‘residual stream’ later became the core design of the Transformer and the cornerstone of modern AI. But note: follow-up work shows ResNet learns both hierarchical representations and refines the residual stream — the two can coexist.
The residual stream is the model's working memory
A Meta team found that in a vision Transformer's residual stream, activations at certain positions were unusually large and concentrated in unimportant regions of the image. They hypothesized the model used these positions as ‘registers’ to aggregate global information. Experiments confirmed it: the embeddings at these high-activation positions performed astonishingly well on image classification (85.2% vs 10.8%), and after adding extra ‘register tokens’, the model no longer occupied image positions.
In their own words · checked verbatim
So if the 30 layer model was capable of achieving at least the same performance as the 14 layer model, then why couldn't the team's optimizers find these solutions?
as depth increases, gradients in standard feed forward networks increasingly resemble white noise.
The solution GNS's team found is almost comically simple.
Few ideas in the history of machine learning have had such a profound and rapid impact on the field, ultimately earning the ResNet paper more citations than any other paper in the 21st century.
The model learns to recognize patches containing little useful information and recycles the corresponding tokens to aggregate global image information while discarding spatial information.
And it's so remarkable to me that in the absence of registered tokens, these models effectively learn to make their own by repurposing unimportant parts of the image.
Figures
| 74-layer network accuracy | 38.9% | 8:13 |
| 8-layer network accuracy | 44.1% | 6:10 |
| 74-layer network accuracy after adding residual connections | 72.6% | 19:19 |
| Accuracy of high-activation embeddings on the Cars dataset | 85.2% | 29:28 |
| Accuracy of non-high-activation embeddings on the Cars dataset | 10.8% | 29:28 |
Glossary
- Residual stream
- The continuous path of data flowing through a network from input to output, with each layer making only incremental edits.
- Skip connection
- The operation of adding the input tensor directly to a layer's output, giving gradients a shortcut.
- Register tokens
- Extra embedding vectors added to store global information, avoiding the need to occupy image positions.
- Shattered gradients problem
- Increasing depth makes gradients as random as white noise, rendering training ineffective.
How to listen
Deep learning researchers, AI engineers, and technically curious people who want to know the low-level mechanics of neural networks.
The sponsor Jane Street interview segment (21:20-23:21) can be skipped without losing the main thread.