Deep linear networks are a surprisingly useful toy model of weight-space dynamics
Deep linear networks are simple enough to study analytically but rich enough to exhibit key phenomena of neural network training.
📣 Read the perspective paper: There Will Be a Scientific Theory of Deep Learning 📣
On neural scaling and the quanta hypothesis
What is the origin of neural scaling laws? What do they tell us about the structure of data? What are the limits of interpretability?
Perspectives on the science of deep learning
Three essays on building theory that matters.