Era 2 · Deep Learning Renaissance (2012-2016)

From the AlexNet ignition to the ResNet finale — 5 years that turned deep learning from a fringe school into the dominant AI paradigm.

Collected Notes

• AlexNet — How Two GTX 580s Cut ImageNet Error to 15% · 2012 · Krizhevsky, Sutskever & Hinton
• Dropout — Randomly Turning Neurons Off to Stop Feature Co-adaptation · 2012 · Hinton et al.
• Word2Vec - The Industrial Shortcut that Put Meaning into Vectors · 2013 · Mikolov et al.
• VAE — Turning Generative Modeling into a Tractable Variational Bound · 2013 · Kingma & Welling
• DQN — The First Deep RL Agent to Learn Atari from Pixels · 2013 · Mnih et al.
• ZFNet — Visualizing the Black Box That AlexNet Opened · 2013 · Zeiler & Fergus
• GAN — Teaching a Generator to Paint via an Adversarial Game · 2014 · Goodfellow et al.
• Seq2Seq - Compress Any Sequence into a Vector, Then Decode It Back · 2014 · Sutskever, Vinyals & Le
• Bahdanau Attention — Teaching Neural MT Where to Look · 2014 · Bahdanau, Cho & Bengio
• Adam — Adaptive Moments for Stochastic Optimization · 2014 · Kingma & Ba
• VGG — Pushing CNNs to 19 Layers with 3×3 Convolutions · 2014 · Simonyan & Zisserman
• GloVe - The Global Co-occurrence Bridge for Word Vectors · 2014 · Pennington, Socher & Manning
• R-CNN — The ImageNet Feature Hierarchy That Rebooted Detection · 2014 · Girshick et al.
• Network In Network — Putting a Tiny MLP Inside Every Convolution · 2014 · Lin, Chen & Yan
• Adversarial Examples — Linearity, FGSM, and the Beginning of Modern Robustness · 2014 · Goodfellow, Shlens & Szegedy
• BatchNorm — Turning Training Stability into a Layer · 2015 · Ioffe & Szegedy
• Inception / GoogLeNet — Making CNNs Deeper by Making Them Wider · 2015 · Szegedy et al.
• Nature DQN - The Atari Moment That Made Deep Reinforcement Learning Public · 2015 · Mnih et al.
• Faster R-CNN — Learning Region Proposals Inside the Detector · 2015 · Ren et al.
• U-Net — Turning Encoder-Decoders and Skip Connections into the Default Grammar of Medical Segmentation · 2015 · Ronneberger, Fischer & Brox
• ResNet — How Deep Residual Learning Unlocked the 152-Layer Door · 2015 · He et al.
• FCN - Turning Classification Networks into Pixel-Level Segmenters · 2015 · Long, Shelhamer & Darrell
• Knowledge Distillation — Pouring a Large Model's Dark Knowledge into a Small One · 2015 · Hinton, Vinyals & Dean
• He Init - The Starting Point That Kept ReLU Networks Alive · 2015 · He et al.
• Spatial Transformer Networks — Letting CNNs Learn to Crop, Align, and Warp · 2015 · Jaderberg et al.
• YOLO — Turning Object Detection into a Single Real-Time Regression · 2016 · Redmon et al.
• AlphaGo — SL + RL + Value Network + MCTS Brought 9-Dan Go a Decade Forward · 2016 · Silver et al.
• DenseNet - Feature Reuse as a Network Architecture · 2016 · Huang et al.
• WaveNet - The Neural Starting Point for Raw-Waveform Generation · 2016 · van den Oord et al.
• A3C - Asynchronous Actors as the Stabilizer for Deep Reinforcement Learning · 2016 · Mnih et al.
• LayerNorm: Normalization Without a Batch · 2016 · Ba, Kiros & Hinton