Era 1 · Neural Network Foundations (1957-2011)¶

From the Perceptron's optimism, through the AI winter's despair, to backprop's revival and the birth of CNN/LSTM — these 50 years quietly built every "ancestor part" that modern deep learning still uses.

Collected Notes¶

Perceptron — The "Genesis" of Neural Networks and the First AI Hype · 1958 · Rosenblatt
Backpropagation — How the Chain Rule Ended the First AI Winter · 1986 · Rumelhart, Hinton & Williams
Universal Approximation — The Existence Theorem That Certified Neural Networks' Expressive Power · 1989 · Hornik, Stinchcombe & White
SVM — How Convex Optimization + Kernel Trick Reigned over Neural Networks for 15 Years · 1992 · Boser, Guyon & Vapnik
LSTM — How a Weight-1 Identity Connection Reborn Sequence Modeling · 1997 · Hochreiter & Schmidhuber
LeNet-5 — When Convolution, Weight Sharing and End-to-End Gradients First Merged · 1998 · LeCun, Bottou, Bengio & Haffner
Random Forests — How an Industrial-Grade Ensemble Paradigm Reigned over Non-Neural ML for 20 Years · 2001 · Breiman
LDA — Promoting pLSA to a Generalizable Fully-Bayesian Topic Model with a Dirichlet Prior · 2003 · Blei, Ng & Jordan
Autoencoder — RBM Pretraining Wakes Neural Networks From Cold Storage · 2006 · Hinton & Salakhutdinov
DBN — How Hinton's Layer-wise RBM Pretraining Lit the Third Deep Learning Revolution · 2006 · Hinton, Osindero & Teh
t-SNE — The Visual Language of High-Dimensional Data Visualization · 2008 · van der Maaten & Hinton
ImageNet — How a Single Dataset Lit the Fuse for 2012 AlexNet · 2009 · Deng, Dong, Socher, Li, Li & Fei-Fei
Glorot Init — Making Deep Networks Pass Signals Before They Learn · 2010 · Glorot & Bengio
RNN-LM — Moving Language Modeling from Fixed Windows to Recurrent State · 2010 · Mikolov et al.
Stacked Denoising Autoencoders — Turning Local Denoising into Deep Representation Pretraining · 2010 · Vincent, Larochelle, Lajoie, Bengio & Manzagol
ReLU — How max(0, x) Turned Deep Networks from "Lab Toy" to "Industrial Cornerstone" · 2011 · Glorot, Bordes & Bengio