Why autonomy is a coordination problem
Autonomy scales, but safety does not come from any single system getting better. When many agents share space and time, decisions collide and hazards compound, so keeping us safe becomes a coordination…
Score matching and diffusion
The setup goes back to Sohl-Dickstein et al. [1]. The forward chain takes data x0 and adds Gaussian noise according to a fixed schedule α¯t, producing intermediate samples x1,…,xT via xt=α¯tx0+1−α¯tϵ with…
Adversarial examples
Adversarial examples initially seemed an oddity. Szegedy et al. [1] demonstrated that a minuscule perturbation, meaningless to human eyes, could confidently flip a neural net's prediction. My first…
Mode connectivity
The old picture of the loss surface as many isolated basins, one per initialization, has not held up. Freeman and Bruna [6] suggested early on that low-loss level sets stay connected. Garipov et al. [1]…
The neural tangent kernel
The neural tangent kernel was one of the few deep-learning theory ideas that were useful before they became a concept. It doesn't solve generalization, but it makes a very stubborn object analyzable.…
The implicit-bias program
Why does training a model without an explicit regularizer, with the loss driven nearly to zero, still produce a solution that generalizes? The classical answer is that the objective has to carry the…
The edge of stability
Cohen et al. [1] observed that gradient descent on neural networks spends most of training in a regime where the top Hessian eigenvalue λmax is above the classical stability threshold 2/η. The step is…
Kaplan, Chinchilla, and broken laws
Kaplan et al. [1] set the baseline picture: test loss decays as a power law L(C)=(C/C0)−αC in each of model size N, dataset size D, and compute C, with clean exponents α that hold over many orders of…
Lottery ticket hypothesis
The lottery-ticket hypothesis of Frankle and Carbin [1] proposes that a randomly initialized dense network already contains a much sparser subnetwork (the "winning ticket") which, trained in isolation…
Double descent
The classical U becomes a W with a second descent in the overparameterized (p>n) regime and that second descent often goes below the first minimum.
Reading Tishby's information bottleneck
Tishby and Zaslavsky's 2015 paper was, until fairly recently, one of the most-cited papers in deep-learning theory. They described training as two distinct phases. In the first, the "fitting" phase, the…
On flat minima
Whether flat minima generalize better than sharp ones has been an open question for about seven years. The debate seems to close every year and reopen a year later. Most readers entering the field…
Four explanations for Grokking
The network has generalized but long after it has already fit the data. The paper is Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets by Power et al. [1]. I came across it maybe a…