https://mouhssine.rifaki.me/notes/ Essays on deep learning theory, reinforcement learning, and mathematical statistics. en-us Copyright 2024-2026 Mouhssine Rifaki Wed, 22 Apr 2026 12:00:00 +0000 Wed, 22 Apr 2026 12:00:00 +0000 1440 https://mouhssine.rifaki.me/notes/diffusion-models/ https://mouhssine.rifaki.me/notes/diffusion-models/ Wed, 22 Apr 2026 12:00:00 +0000 Mouhssine Rifaki The DDPM forward chain has a clean dual under score matching, and once the two are placed side by side they are not separate ideas. The score function… generative models theory https://mouhssine.rifaki.me/notes/adversarial-examples/ https://mouhssine.rifaki.me/notes/adversarial-examples/ Sat, 07 Feb 2026 12:00:00 +0000 Mouhssine Rifaki Adversarial examples initially seemed an oddity. Szegedy et al. [1] demonstrated that a minuscule perturbation, meaningless to human eyes, could confidently flip a neural net's prediction. My first… robustness theory https://mouhssine.rifaki.me/notes/mode-connectivity/ https://mouhssine.rifaki.me/notes/mode-connectivity/ Tue, 04 Nov 2025 12:00:00 +0000 Mouhssine Rifaki The old picture of the loss surface as many isolated basins, one per initialization, has not held up. Freeman and Bruna [6] suggested early on that low-loss level sets stay connected. Garipov et al. [1]… optimization generalization https://mouhssine.rifaki.me/notes/neural-tangent-kernel/ https://mouhssine.rifaki.me/notes/neural-tangent-kernel/ Sat, 09 Aug 2025 12:00:00 +0000 Mouhssine Rifaki The neural tangent kernel was one of the few deep-learning theory ideas that were useful before they became a concept. It doesn't solve generalization, but it makes a very stubborn object analyzable.… theory optimization https://mouhssine.rifaki.me/notes/implicit-bias/ https://mouhssine.rifaki.me/notes/implicit-bias/ Wed, 14 May 2025 12:00:00 +0000 Mouhssine Rifaki Why does training a model without an explicit regularizer, with the loss driven nearly to zero, still produce a solution that generalizes? The classical answer is that the objective has to carry the… optimization generalization https://mouhssine.rifaki.me/notes/edge-of-stability/ https://mouhssine.rifaki.me/notes/edge-of-stability/ Thu, 27 Feb 2025 12:00:00 +0000 Mouhssine Rifaki Cohen et al. [1] observed that gradient descent on neural networks spends most of training in a regime where the top Hessian eigenvalue… optimization theory https://mouhssine.rifaki.me/notes/neural-scaling-laws/ https://mouhssine.rifaki.me/notes/neural-scaling-laws/ Mon, 06 Jan 2025 12:00:00 +0000 Mouhssine Rifaki That paper shaped how labs designed pre-training experiments for the next two years, and the eventual "Chinchilla" effort grew out of trying to reproduce and extend its recommendations. It also turned out… scaling generalization https://mouhssine.rifaki.me/notes/lottery-tickets/ https://mouhssine.rifaki.me/notes/lottery-tickets/ Sat, 16 Nov 2024 12:00:00 +0000 Mouhssine Rifaki The lottery-ticket hypothesis of Frankle and Carbin [1] proposes that a randomly initialized dense network already contains a much sparser subnetwork (the "winning ticket") which, trained in isolation… generalization optimization https://mouhssine.rifaki.me/notes/double-descent/ https://mouhssine.rifaki.me/notes/double-descent/ Sun, 25 Aug 2024 12:00:00 +0000 Mouhssine Rifaki Nakkiran et al. [2] made the picture concrete by showing that the W-shape appears in three different axes: model size… generalization scaling https://mouhssine.rifaki.me/notes/information-bottleneck/ https://mouhssine.rifaki.me/notes/information-bottleneck/ Tue, 09 Jul 2024 12:00:00 +0000 Mouhssine Rifaki Tishby and Zaslavsky's 2015 paper was, until fairly recently, one of the most-cited papers in deep-learning theory. They described training as two distinct phases. In the first, the "fitting" phase, the… theory generalization https://mouhssine.rifaki.me/notes/flat-minima/ https://mouhssine.rifaki.me/notes/flat-minima/ Sun, 07 Apr 2024 12:00:00 +0000 Mouhssine Rifaki Whether flat minima generalize better than sharp ones has been an open question for about seven years. The debate seems to close every year and reopen a year later. Most readers entering the field… optimization generalization https://mouhssine.rifaki.me/notes/grokking/ https://mouhssine.rifaki.me/notes/grokking/ Sat, 24 Feb 2024 12:00:00 +0000 Mouhssine Rifaki The network has generalized but long after it has already fit the data. The paper is Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets by Power et al. [1]. I came across it maybe a… generalization optimization