Era 2 · Deep Learning Renaissance (2012-2016)¶
From the AlexNet ignition to the ResNet finale — 5 years that turned deep learning from a fringe school into the dominant AI paradigm.
Collected Notes¶
- AlexNet — How Two GTX 580s Cut ImageNet Error to 15% · 2012 · Krizhevsky, Sutskever & Hinton
- Dropout — Randomly Turning Neurons Off to Stop Feature Co-adaptation · 2012 · Hinton et al.
- Word2Vec - The Industrial Shortcut that Put Meaning into Vectors · 2013 · Mikolov et al.
- VAE — Turning Generative Modeling into a Tractable Variational Bound · 2013 · Kingma & Welling
- DQN — The First Deep RL Agent to Learn Atari from Pixels · 2013 · Mnih et al.
- GAN — Teaching a Generator to Paint via an Adversarial Game · 2014 · Goodfellow et al.
- Seq2Seq - Compress Any Sequence into a Vector, Then Decode It Back · 2014 · Sutskever, Vinyals & Le
- Bahdanau Attention — Teaching Neural MT Where to Look · 2014 · Bahdanau, Cho & Bengio
- Adam — Adaptive Moments for Stochastic Optimization · 2014 · Kingma & Ba
- VGG — Pushing CNNs to 19 Layers with 3×3 Convolutions · 2014 · Simonyan & Zisserman
- R-CNN — The ImageNet Feature Hierarchy That Rebooted Detection · 2014 · Girshick et al.
- Adversarial Examples — Linearity, FGSM, and the Beginning of Modern Robustness · 2014 · Goodfellow, Shlens & Szegedy
- BatchNorm — Turning Training Stability into a Layer · 2015 · Ioffe & Szegedy
- Inception / GoogLeNet — Making CNNs Deeper by Making Them Wider · 2015 · Szegedy et al.
- Nature DQN - The Atari Moment That Made Deep Reinforcement Learning Public · 2015 · Mnih et al.
- Faster R-CNN — Learning Region Proposals Inside the Detector · 2015 · Ren et al.
- U-Net — Turning Encoder-Decoders and Skip Connections into the Default Grammar of Medical Segmentation · 2015 · Ronneberger, Fischer & Brox
- ResNet — How Deep Residual Learning Unlocked the 152-Layer Door · 2015 · He et al.
- FCN - Turning Classification Networks into Pixel-Level Segmenters · 2015 · Long, Shelhamer & Darrell
- Knowledge Distillation — Pouring a Large Model's Dark Knowledge into a Small One · 2015 · Hinton, Vinyals & Dean
- He Init - The Starting Point That Kept ReLU Networks Alive · 2015 · He et al.
- YOLO — Turning Object Detection into a Single Real-Time Regression · 2016 · Redmon et al.
- AlphaGo — SL + RL + Value Network + MCTS Brought 9-Dan Go a Decade Forward · 2016 · Silver et al.
- LayerNorm: Normalization Without a Batch · 2016 · Ba, Kiros & Hinton