Era 1 · Neural Network Foundations (1957-2011)

From the Perceptron's optimism, through the AI winter's despair, to backprop's revival and the birth of CNN/LSTM — these 50 years quietly built every "ancestor part" that modern deep learning still uses.

Collected Notes (14)

• ReLU — How max(0, x) Turned Deep Networks from "Lab Toy" to "Industrial Cornerstone" · 2011 · Glorot, Bordes & Bengio
• Glorot Init — Making Deep Networks Pass Signals Before They Learn · 2010 · Glorot & Bengio
• ImageNet — How 15M Images Turned a 'Dataset' into the Fuse of the Deep Learning Revolution · 2009 · Deng, Dong, Socher, Li, Li & Fei-Fei
• t-SNE — The Visual Language of High-Dimensional Data Visualization · 2008 · van der Maaten & Hinton
• Autoencoder — RBM Pretraining Wakes Neural Networks From Cold Storage · 2006 · Hinton & Salakhutdinov
• DBN — How Layer-wise Greedy Pretraining Made Deep Networks Trainable for the First Time · 2006 · Hinton, Osindero & Teh
• LDA — Promoting pLSA to a Generalizable Fully-Bayesian Topic Model with a Dirichlet Prior · 2003 · Blei, Ng & Jordan
• Random Forests — Bagging + Feature Sampling that Crowned Decision Trees on the ML Throne · 2001 · Breiman
• LeNet — Stitching Convolution, Pooling and Backprop into the First Industrial-Grade Deep Network · 1998 · LeCun, Bottou, Bengio & Haffner
• LSTM — How Gating Made Recurrent Networks Remember Long Dependencies for the First Time · 1997 · Hochreiter & Schmidhuber
• SVM — How Max-Margin and the Kernel Trick Dominated Machine Learning for Two Decades · 1992 · Boser, Guyon & Vapnik
• Universal Approximation — The Existence Theorem That Certified Neural Networks' Expressive Power · 1989 · Hornik, Stinchcombe & White
• Backprop — Pulling Multi-layer Networks from 'Untrainable' into the Optimizable World via the Chain Rule · 1986 · Rumelhart, Hinton & Williams
• Perceptron — How the First Hardware Neuron That Learns from Data Sparked AI as a Discipline · 1958 · Rosenblatt