Era 4 · Foundation Models (2020-2022)

From GPT-3 igniting the scaling belief to ChatGPT reshaping product form — the critical 3 years that turned large models from research paradigm into commercial paradigm.

Collected Notes (33)

• ControlNet — Plugging spatial control into frozen diffusion via zero-convolutions · 2023 · Zhang et al.
• Chinchilla — Proving All LLMs Were 'Undertrained' via Compute-Optimal Allocation · 2022 · Hoffmann et al. (DeepMind)
• Classifier-Free Diffusion Guidance — One Line of Code Removes the Bolt-On Classifier and Unifies Modern Text-to-Image · 2022 · Ho et al.
• CoT — Unlocking LLM Reasoning with 'Let's Think Step by Step' · 2022 · Wei et al. (Google Brain)
• Constitutional AI — Replacing Tens of Thousands of Human Harm Labels With a Constitution and AI Feedback · 2022 · Bai et al.
• DiT - When Diffusion Models Replaced the U-Net with a Transformer · 2022 · Peebles et al.
• DreamBooth — Implanting Any Subject Into a Text-to-Image Model With 3-5 Photos · 2022 · Ruiz et al.
• Flamingo: a Visual Language Model for Few-Shot Learning · 2022 · Alayrac et al. (DeepMind)
• FlashAttention: Faster Attention Without Dropping a Single Interaction · 2022 · Dao et al.
• Imagen — Cascaded Text-to-Image Diffusion with Deep Language Understanding · 2022 · Saharia et al.
• InstructGPT — Turning GPT-3 from a Continuator into an Obedient Assistant via RLHF · 2022 · Ouyang et al. (OpenAI)
• MAE — Teaching ViT Self-Supervised Pretraining via 75% Masking · 2022 · He et al. (FAIR)
• PaLM — Scaling Dense Language Models to 540B with Pathways · 2022 · Chowdhery et al.
• ReAct: Synergizing Reasoning and Acting in Language Models · 2022 · Yao et al.
• Stable Diffusion — Moving Diffusion into Latent Space so Consumer GPUs Can Generate Images · 2022 · Rombach et al. (CompVis)
• Whisper - Turning 680k Hours of Weak Supervision into a General Speech Interface · 2022 · Radford et al.
• AlphaFold2 — Driving Protein Structure Prediction to Atomic Accuracy via Attention + Evoformer · 2021 · Jumper et al. (DeepMind)
• CLIP — Teaching Vision Models to Understand Language via 400M Image-Text Pairs · 2021 · Radford et al. (OpenAI)
• Codex — Evaluating Large Language Models Trained on Code · 2021 · Chen et al.
• DALL-E — Recasting Text-to-Image Generation as Language Modeling · 2021 · Ramesh et al.
• LoRA — Slashing Large-Model Fine-tuning Cost by 99% via Low-Rank Matrices · 2021 · Hu et al. (Microsoft Research)
• Swin Transformer - Turning ViT into a General-Purpose Vision Backbone with Shifted Windows · 2021 · Liu et al.
• DDPM — Crowning Diffusion as the King of Image Generation via Thousand-Step Denoising · 2020 · Ho, Jain, Abbeel (UC Berkeley)
• DETR — Recasting Object Detection as Transformer Set Prediction · 2020 · Carion et al.
• GPT-3 — When 175B Parameters Made Prompting the New Programming Paradigm · 2020 · Brown et al. (OpenAI)
• MoCo: Queues, Momentum Encoders, and the Moment Vision Self-Supervision Became Transferable · 2020 · He et al.
• NeRF — Encoding a Scene into a Differentiable Radiance Field with One MLP · 2020 · Mildenhall et al. (UC Berkeley)
• RAG — Turning Wikipedia into Replaceable Memory for Generation · 2020 · Lewis et al.
• Scaling Laws for Neural Language Models · 2020 · Kaplan et al. (OpenAI)
• Score SDE — Unifying Score-Based and Diffusion Models through Stochastic Differential Equations · 2020 · Song et al.
• SimCLR — A Plain Contrastive Loss That Crowned Self-Supervised Vision on ImageNet Linear Eval · 2020 · Chen et al.
• ViT — Dethroning Convolution from Vision with Pure Transformer · 2020 · Dosovitskiy et al. (Google Brain)
• wav2vec 2.0 - Speech Recognition After 53k Hours of Listening and 10 Minutes of Labels · 2020 · Baevski et al.

Candidate Papers (Selection)

• PaLM (2022) — Pathways + 540B validation
• DALL-E 2 (2022) — Hierarchical T2I