Residual Connections: The Most Underrated Simple Hack in Deep Learning History
ResNet is the most cited paper of the 21st century, but its real contribution isn't deepening networks—it's the invention of the residual stream, a working memory that lets models store and edit information, fundamentally changing our understanding of neural networks.
The argument · tap a timestamp to hear it
Deeper networks performed worse
In early 2015, Kaiming He's team used careful initialization to push networks to 30 layers, but performance was worse than 14 layers (error rate 16.59% vs 13.34%). Theoretically, a 30-layer model could achieve at least the same performance as a 14-layer one via identity mapping, but optimizers couldn't find such solutions. This hinted the problem wasn't network capacity but the training process itself.
Shattered gradients were the root cause
Through loss landscape visualization, they found that as layers increased, the loss surface for early layers became extremely rugged, with gradient directions as random as white noise—termed the 'shattered gradients problem.' Gradients are the core signal for learning; once they fail, deep networks can't train effectively.
Residual connections: an almost absurdly simple fix
Kaiming He's team's solution was laughably simple: between every two layers, add the input tensor directly to the output, creating a skip connection. This provided a direct path for activations and gradients, effectively eliminating the shattered gradients problem. With skip connections, a 74-layer network's accuracy jumped from 38.9% to 72.6%, sweeping major vision competitions in 2015.
ResNets upended hierarchical representation theory
A Cornell team found that removing or even shuffling layers of a ResNet had little impact on performance. This contradicted the then-dominant hierarchical representation theory, which held that each layer builds on the previous one, so removing any layer should be catastrophic. This forced a rethink: ResNets might not be learning hierarchical features but iteratively refining a 'residual stream' running through the network.
The residual stream: modern AI's hidden backbone
Straightening the skip connections reveals an uninterrupted stream from input to output, with each layer making incremental modifications. This 'residual stream' later became the core design of the Transformer and the foundation of modern AI. However, subsequent research proved that ResNets learn both hierarchical representations and refine the residual stream—they can coexist.
The residual stream is the model's working memory
Meta's team found that in vision Transformers, certain positions in the residual stream have unusually large activations, concentrated in image regions of low importance. They hypothesized the model uses these positions as 'registers' to aggregate global information. Experiments confirmed: embedding vectors from these high-activation positions performed remarkably in image classification (85.2% vs 10.8%), and adding extra 'register tokens' meant the model no longer used image positions.
In their own words · checked verbatim
So if the 30 layer model was capable of achieving at least the same performance as the 14 layer model, then why couldn't the team's optimizers find these solutions?
as depth increases, gradients in standard feed forward networks increasingly resemble white noise.
The solution GNS's team found is almost comically simple.
Few ideas in the history of machine learning have had such a profound and rapid impact on the field, ultimately earning the ResNet paper more citations than any other paper in the 21st century.
The model learns to recognize patches containing little useful information and recycles the corresponding tokens to aggregate global image information while discarding spatial information.
And it's so remarkable to me that in the absence of registered tokens, these models effectively learn to make their own by repurposing unimportant parts of the image.
Figures
| 74-layer network accuracy | 38.9% | 8:13 |
| 8-layer network accuracy | 44.1% | 6:10 |
| 74-layer network accuracy with residual connections | 72.6% | 19:19 |
| High-activation embedding vectors accuracy on Cars dataset | 85.2% | 29:28 |
| Non-high-activation embedding vectors accuracy on Cars dataset | 10.8% | 29:28 |
Glossary
- Residual stream
- The continuous flow path of data from input to output in a network, with each layer making only incremental modifications.
- Skip connection
- An operation that adds the input tensor directly to a layer's output, providing a shortcut for gradients.
- Register tokens
- Extra embedding vectors added to store global information, avoiding the use of image positions.
- Shattered gradients problem
- The phenomenon where increasing depth makes gradients as random as white noise, causing training to fail.
How to listen
Deep learning researchers, AI engineers, and tech enthusiasts curious about the underlying mechanisms of neural networks.
The sponsor segment with Jane Street (21:20-23:21) can be skipped without missing the main thread.