![\"A](\"https://mouhssine.rifaki.me/notes/img/coordination-coverage.png\")
\n Autonomy scales, but safety does not come from any single system getting better. When many agents share space and time, decisions collide and hazards compound, so keeping us safe becomes a coordination problem that compounds with every participant. Nobody in the field disagrees with that. Such wide agreement is what makes the sentence nearly useless on its own. What I want to argue is the part people do disagree with: coordination cannot be added afterwards. The prevailing plan is to build the vehicles now and standardize the coordination between them later as a protocol or a service or a certificate class. My argument is that the plan fails. Where the question admits of measurement I have measured it. A few hours of live air traffic tell us what proximity does in a real sky. A small experiment anyone can rerun in twenty seconds tells us what a coordination convention amounts to inside a learned policy. Each is linked where it is used and each contradicted something I believed before I ran it.
\n\nThe instinct to add coordination later is the layering instinct and it is usually right. Networking works because TCP does not need to know how Ethernet frames a packet. The layer below can fail, the layer above notices, and the recovery is a retransmission costing milliseconds and nothing else. Layering buys independence at the price of one assumption: that failure at the lower layer is recoverable at the higher one.
\n\nThe FAA and TSA proposal of 7 August 2025 on beyond-visual-line-of-sight operations [1] writes that instinct into law. It creates a whole new certificate class called part 146 for automated data service providers. Operators in controlled airspace or over dense population must buy strategic deconfliction from one of them or certificate themselves to self-provide. The proposal certifies the aircraft on one side and the service on the other, and what passes between the two is a request and a response.
\n\nPerrow drew the distinction that governs this. Interactive complexity means a system with more ways to interact than anyone has modeled; tight coupling means a failure reaching its consequence with nothing in between to absorb it. He argued that the dangerous quadrant belongs to systems with both at once [2]. Layering answers the first of those and has nothing to offer against the second, since the recovery it depends on costs time that a tightly coupled system has already spent. Two hundred aircraft over a city at two hundred feet occupies that quadrant, where the round trip a retransmission needs runs longer than the interval in which a conflict resolves itself one way or the other.
\n\nLeveson's version is more useful still, because she models safety as a hierarchy of controllers and locates accidents in inadequate control at some level [3]. A coordination service is a controller. The question a layering plan has to answer is what it controls and for how long after it stops speaking.
\n\nThe proposal answers that question with unusual precision. What it sets down reads as an engineering specification.
\n\nProposed \u00a7 108.190(c)(1) requires strategic deconfliction to "perform strategic conflict detection and resolution prior to takeoff, and in relation to other unmanned aircraft operations that are discoverable at that time." Each qualifier carries weight. The guarantee is computed once before departure over whichever intents happened to be visible at that instant. Proposed \u00a7 108.190(d) adds conformance monitoring. Its specified output is an immediate alert to operations personnel when the aircraft leaves its intent, which is to say the layer's response to a violation is to tell a human. The definitions come from ASTM F3548-21 as the rule's own footnote records, and that standard draws its boundaries in clause 1.13. It "does not purport to address tactical conflicts between UAS" and it "does not purport to address operations in locations where persistent connectivity is unavailable" [4]. Those are the two conditions under which the rest of this note is written.
\n\nThe rest of the proposal [1] follows the same shape. Proposed \u00a7 108.195 orders an unmanned aircraft against aircraft arriving at an airport and against aircraft that broadcast their position. Proposed \u00a7 108.825 defines the collision-avoidance design requirement by pointing back to it, so a part 108 aircraft is ranked against manned traffic and unranked against another part 108 aircraft. Proposed \u00a7 108.210(a) fixes the aircraft-to-coordinator ratio at one to one, waivable by "a method acceptable to the Administrator" that nobody has written. Proposed \u00a7 108.815(b) requires the airframe to "execute a safe predetermined action when reaching the link timeout." Proposed \u00a7 146.325(a) requires the service provider to report an unscheduled outage after the fact.
\n\n\n I do not read this as an oversight. It is where a layered design has to stop and the standard says so out loud. A service that computes a guarantee over a snapshot and hands it to a vehicle that will act on it for the next forty minutes has produced something with the structure of a database read taken outside a transaction: accurate at the instant of reading and unconstrained from then on.
\n\nThe obvious objection is that detect and avoid is the tactical layer and it is on its way. I think the proposal is quietly clear about why that does not rescue the plan. Proposed \u00a7 108.825 states the design requirement as a capability "to avoid aircraft as required in accordance with \u00a7 108.195". The specification for detect and avoid is therefore the right-of-way rule, and that rule ranks a part 108 aircraft against traffic that broadcasts while saying nothing about two part 108 aircraft meeting each other. The FAA also records that it "decided not to update \u00a7 91.113 based on the BVLOS ARC's proposal related to 'detect-and-avoid' at this time" because doing so would reach into part 91 and legacy aviation [1]. Set that aside and a deeper problem remains. Detect and avoid is a sensor and a maneuver on one airframe. It improves what one aircraft can do about a conflict it can see, which puts it in the capability column. Two aircraft avoiding competently can still pick maneuvers that conflict with each other.
\n\nThe case for a coordination service rests on an arithmetic claim: encounters grow faster than fleets do. Put $N$ vehicles in a fixed volume, assume any pair is as likely to meet as any other, and the pair count grows like $N^2$. I have used that argument before. This time I wanted to know whether it survives contact with a real sky. It does so only in the sense that reality turns out to be worse than the model allows for.
\n\nI took 30 snapshots of every transponder-equipped aircraft over the continental United States from the OpenSky Network [20]. That is 123,520 airborne state vectors in all, sampled with 12,000 discs of 100 km drawn at random across it. In each disc I counted the aircraft present and the pairs within 5, 10, 20 and 40 nautical miles laterally and 1,000 feet vertically. That vertical figure is the standard separation minimum. Then I asked what the same aircraft at the same altitudes would produce if they were thrown at random into the same disc. The null needs no model of its own. It is the same population with the geography taken out. The full set of numbers is here, along with the snapshots they were computed from.
\n\nReal traffic packs 6.74 times more pairs within 5 NM than the scattered version of itself. The fitted slope of the pair count against the aircraft count is 2.58 with a standard error of 0.01, above the 2 that uniform mixing predicts. The excess shrinks as the threshold widens and reaches 1.65 at 40 NM. That is what we would expect if the clustering happens at the scale of terminal areas.
\n\nEach number comes with a qualification. The slope compares different places at one moment. A disc holding more aircraft is usually also a different kind of place, so some of the excess over 2 comes from busier airspace being more concentrated to begin with. The ratio avoids that problem by comparing each disc against itself with the geography removed. It does depend on the size of the disc: the null scatters aircraft across whatever disc we choose, so a wider disc makes random placement look emptier. Repeating the whole measurement at 50 and 200 km gives ratios at 5 NM of 3.89 and 11.34 and slopes of 2.62 and 2.37. The ratio moves with the scale as it should and the slope holds above 2 at every disc size tried.
\n\n![\"Two](\"https://mouhssine.rifaki.me/notes/img/coordination-clustering.png\")
\n This is manned aviation. The note concerns uncrewed traffic and I am using manned data on purpose. It is the only large population of vehicles that already shares airspace under a mature coordination regime, so it measures what a century of structure and procedure and separation standards actually achieves. The direction of the answer matters more than its size. All of that has not produced a sky that mixes uniformly. It has produced one that concentrates, and the model I was about to reason from understates the density a coordination service has to handle.
\n\nThe second measurement concerns time. For every converging pair within 20 NM and 1,000 feet I computed the time to closest approach from the two velocity vectors. That comes to 26,137 pairs with a median of 158 seconds. The tail is the part that matters: 1,528 of those pairs, 5.8% of them, reach closest approach inside twelve seconds. The first percentile arrives at 1.9 seconds.
\n\n![\"A](\"https://mouhssine.rifaki.me/notes/img/coordination-cpa.png\")
\n Twelve seconds is not an arbitrary mark. A failure detector on one-second heartbeats needs three missed beats before it declares a link gone. An agreement about what to do costs at least two more message delays after that. Roughly a third of the budget goes on noticing, and by Fischer, Lynch and Paterson the remainder cannot be bounded at all in the asynchronous model. That leaves the 1,528 pairs in the left-hand block with no interval in which to spend it.
\n\nWhich makes the timeout a safety parameter set by the geometry of the corridor it applies to and therefore something to publish per corridor. The harder consequence is that whatever the vehicle does inside those seconds has to work without agreement. Agreement is the one thing the budget cannot buy, and the theory here is unusually blunt about why.
\n\nFischer, Lynch and Paterson showed that no deterministic protocol achieves consensus in an asynchronous system where a single process may fail [5]. A design that assumes the fleet will agree therefore has a timing model hidden inside it that someone should be made to state. Gilbert and Lynch's proof of Brewer's conjecture says a partitioned system chooses between staying available and staying consistent [6]. For an aircraft that reads as acting on a belief it can no longer refresh or stopping. Bernstein, Zilberstein and Immerman put the price of the missing channel in complexity terms. The finite-horizon problem for two agents who cannot share their observations is complete for nondeterministic exponential time [7], against the PSPACE-completeness Papadimitriou and Tsitsiklis had established for the same horizon under a single controller [8].
\n\nHalpern and Moses is the one I keep coming back to. They formalized what a group of processes can know about what the others know. Their finding on common knowledge, the regress where everyone knows that everyone knows, is that "formally speaking, in practical systems common knowledge cannot be attained" [9]. A protocol whose correctness rests on it therefore has no implementation whatever the hardware. That same paper introduces weaker variants that are attainable and those are where a working design has to aim.
\n\nSet that beside a coordination service. The service sends both aircraft the same deconfliction plan and both receive it. Each of them now holds a fact and neither can establish that the other holds it, nor that the other knows they hold it. The regress runs on without terminating. Two parties who each hold a fact are a weaker thing than two parties who can each rely on the other acting on it, and the difference between those is exactly what a coordination service would have to sell. Since no protocol closes that gap, whatever reliance two aircraft place in each other must have been built into them before either took off.
\n\nThe rule's own contingency provision shows this happening. Proposed \u00a7 108.815(b) [1] requires each airframe to execute a safe predetermined action at link timeout and each of those actions is safe on its own terms. What no one has checked is how they compose. The only moment they will ever run is the moment they all run together, since the link they lost was in many cases the same link.
\n\nI drew this scenario before I measured it. Five aircraft separated along one corridor, two returning to a pad at one end and three to a pad at the other, their contingency paths crossing six times in a tidy knot in the middle. It was a persuasive picture and it was wrong. I only know that because the same data that produced the two figures above can be asked the same question.
\n\nThe question I put to the data was this. Take every configuration of 6 or more aircraft below 10,000 feet inside one of those discs, send each aircraft to the nearest of the 821 large and medium airports in the continental United States [21], and count the crossings between pairs that were separated at the moment the link went. Across 1,696 configurations holding 29,580 aircraft the count of crossings came to 0. That is a number I refused to publish until the crossing predicate had been checked against an independent orientation test on twenty thousand random segment pairs, a check that now runs every time the script does.
\n\nThe zero turns out to be a property of the rule and not of the sky. Sending every aircraft to its nearest field keeps each path inside one cell of the Voronoi diagram of those fields: the cell is an intersection of half-planes and therefore convex [22], the aircraft is inside it, and the destination is the site that generates it, so the whole path is inside it too. Two aircraft in different cells never meet and two in the same cell meet only at the runway they already share. To make sure that argument was doing the work I recounted the same configurations with the aircraft rotated onto each other's fields. Every position and every destination is unchanged and only the assignment moves, and the count comes to 201,754 crossings in 1,626 of the 1,696 configurations. So the phrase \u201cnearest airport\u201d is carrying a geometric guarantee, and the guarantee holds only for as long as every aircraft resolves that phrase to the same field. Two airframes running different airport databases are two airframes with different Voronoi diagrams.
\n\n![\"Two](\"https://mouhssine.rifaki.me/notes/img/coordination-return.png\")
\n Converging is a worse property than crossing. In 67% of those configurations three or more aircraft reach the same field within two minutes of each other at their current ground speeds, and the worst has 21. Nothing in the pre-takeoff deconfliction set knows about it, because a contingency path is not an operational intent and was never in the set that was checked. Nothing after takeoff can negotiate it either. The link is what failed, so a set of individually compliant aircraft converges on one runway with no mechanism anywhere in the system that would notice.
\n\nAviation has already run this experiment at full scale. Over \u00dcberlingen on 1 July 2002 a Tupolev Tu-154M and a Boeing 757 collided and 71 people died. The usual telling is that the crews received contradictory instructions. That is true and slightly misses the point. Both aircraft carried a working collision-avoidance system and both were talking to a working controller. Those are two coordination layers. Each was internally correct and each produced a resolution that would have been sufficient on its own. What did not exist was a rank between them. The Tupolev crew followed the controller and the Boeing crew followed the equipment. Among the immediate causes the German federal investigator recorded that the Tupolev crew "followed the ATC instruction to descend and continued to do so even after TCAS advised them to climb". Among the systemic ones it found that the rules for ACAS and TCAS issued by ICAO and by national authorities and by the manufacturer and by the operators "were not standardised, incomplete and partially contradictory". Its first recommendation asked ICAO to require pilots to follow a resolution advisory "regardless of whether contrary ATC instruction is given prior to, during, or after" it [10]. Adding a second correct coordination layer to a system that already has one makes it worse until the two are ranked against each other, and that ranking has to be settled when the system is designed.
\n\nEverything so far applies to hand-written controllers. Learned ones fail the layering plan in a way that is harder to see and probably harder to fix.
\n\nCoordination between two agents means agreeing on an arbitrary choice: who yields, which side we pass on, which lever we pull. The choice is arbitrary in the strict sense that the alternatives are equally good. What makes one of them correct is only that the other party made it too. A policy trained by self-play has to settle on one and where it settles is decided by its own initialization and sampling noise. Lanctot and colleagues named the resulting co-adaptation joint-policy correlation [11]. Carroll and colleagues built a coordination task on Overcooked and found that agents from self-play and from population-based training each assume a partner much like itself and converge on protocols that fail against a human or a model of one [12]. Hu and colleagues framed it as the zero-shot coordination problem in which self-play "can produce agents that establish highly specialized conventions that do not carry over to novel partners". Their remedy works by exploiting the very symmetries such a convention settles at random [13].
\n\nI wanted a number for how large the effect is, so I ran the smallest version of it I could. The game is Hu's lever game. Two players independently choose one of ten levers. Matching levers pay that lever's value and mismatching levers pay nothing. One lever pays 1.0 and the other nine pay 0.9. Ten agreements exist and nine are worse by a hair. I trained eight policies by self-play with REINFORCE on a logit vector, one per seed, then scored every policy against every other in the same frame. The whole thing is one file of numpy with a forty-five line training loop and it takes about twenty seconds to run. What it produced is here, seeds and hyperparameters and environment included.
\n\n![\"Two](\"https://mouhssine.rifaki.me/notes/img/coordination-crossplay.png\")
\n Every one of those policies is individually optimal and converged and would pass any test that pairs it with itself or with a copy of itself. What none of them carries is a way to tell us which lever it settled on, or to be told to settle on a different one. We cannot publish a standard that says take lever 1 and have these policies comply, because the convention exists only as a pattern spread across the weights and there is no field to write a different one into.
\n\nWhich is why the known fix changes the training objective. Other-play trains each policy against a relabeling of itself drawn fresh each episode from the game's symmetry group. A policy that locked onto one of the nine interchangeable levers now meets a partner who locked onto a different one eight times in nine, and the only strategy the gradient can reward is the one every relabeling leaves alone. Panel B is the same experiment with that one change and cross-play goes from 0.06 to 0.54.
\n\nIt also stops at 0.54, and the reason turned out to be more interesting than a clean result would have been. Write a policy as mass $q$ on the distinguished lever with the rest spread over the nine. Its expected return under a random relabeling is $$J(q) = q^2 + \\frac{0.9\\,(1-q)^2}{9},$$ a parabola whose interior minimum is at $q = 1/11$. Gradient ascent leaves that point in whichever direction it starts from. Six of my eight seeds initialized above $1/11$ and every one of them converged to $q = 0.9998$. Two initialized below it and both converged to $q = 0.0024$. Ten thousand steps at four times the step size moved neither group.
\n\n![\"Two](\"https://mouhssine.rifaki.me/notes/img/coordination-barrier.png\")
\n So the fix works and whether it works on any particular run is decided by the initialization. That is a toy and I want to be careful about how far it generalizes. The lever game has an exact symmetry group that I can write down and sample from. Airspace conventions are approximate, the group is unknown in advance, and a real policy has far more ways to settle an arbitrary choice than ten. Every one of those differences makes the problem harder than the one I ran.
\n\nIf the convention is in the weights then which weights are flying is a safety-relevant fact about an aircraft, in the same category as its position. Look at what the proposed broadcast actually carries under \u00a7 108.195(a)(2)(ii) [1]: latitude, longitude, geometric altitude, velocity, an ICAO 24-bit address and three integrity figures. All three integrity figures describe the position source. None of them describes the thing making the decisions. Three policy versions rolling out across eight aircraft in a volume give 6,561 ways to assign versions to airframes and 45 distinct mixes even when the airframes are interchangeable. Certification evidence covers the three configurations where every airframe is on the same build.
\n\nIf the contract has to be in the policy then the co-player distribution and the partition profile have to be in the training environment, which makes the environment the thing we are designing. That is the premise of unsupervised environment design. Dennis and colleagues make the generator an agent rewarded for regret. They define regret as the gap between what an antagonist achieves in a configuration and what the learner achieves, which pushes the generator toward configurations at the frontier of what the learner can handle, since an unsolvable one earns it no regret at all [14]. Jiang and colleagues later showed that prioritized replay of randomly generated levels belongs to the same family and earned the approach a robustness guarantee at Nash equilibria [15]. The configuration covers density, version mix, link profile and geometry. That is what the search ranges over.
\n\nThe reported numbers then have to be joint or the exercise reverts to per-agent testing with extra steps: conflicts per flight hour at each density, the worst operator's detour alongside the mean, and the version mix that was airborne. All of it against a baseline of fixed structure and no learning, since a coordination layer that cannot beat assigned altitudes and a precedence rule has not earned its complexity. Melting Pot exists to score exactly this kind of generalization to unfamiliar co-players [16]. Mixed-autonomy traffic supplies the nearest fielded analog. Vinitsky and colleagues found that their learned controllers held a strategy that worked across penetration rates from 5% to 40% where a hand-tuned feedback controller "degrade[d] immediately upon penetration rate variation" [17]. A controller tuned for one composition of the population stops working when the composition moves. That is the same difficulty the version mix presents in another setting.
\n\nI do not want to oversell this. A regret objective needs a best response to compare against and in a population the best response is itself a joint quantity that nobody can compute. Every practical estimator substitutes something cheaper and the substitution is where the guarantee leaks. I do not know how to close that gap. What I am confident of is the direction: the co-player distribution is a design input and treating it as one costs nothing at training time and cannot be recovered afterwards.
\n\nWhether a coordination service should exist is settled by the pair count. The question this note raises is whether one can be sufficient, and the answer is that the service cannot be the whole of the coordination. The layer's guarantee expires at takeoff and common knowledge is not something it can deliver. The conventions that would let two vehicles agree without it are set during training and unreachable after.
\n\nSome of what follows costs almost nothing. Contingency envelopes can go into the pre-takeoff deconfliction set alongside nominal intents, since the moment they are needed is the moment nothing can be negotiated. Precedence between two unmanned aircraft can be written into the rule, which is what the BFU asked ICAO to do after \u00dcberlingen. A precedence rule needs no round trip. A policy version can be a field in a broadcast. Wurman, D'Andrea and Mountz describe a Kiva installation for a large distribution center needing "500 or more vehicles". They move on a weighted grid. System-wide resource allocation is centralized in one job manager and each vehicle plans its own path across that grid [18].
\n\nThe training is the expensive part. Retraining a fleet's policies against a co-player distribution that includes the other fleets is beyond what any one operator can do, and the current interface between operators has no way to express it even as a request. Kuchar and Yang separated the pairwise from the global in their survey of conflict detection twenty-six years ago [19]. Almost everything fielded since has been pairwise because pairwise is what one vehicle can reason about with what it can see.
\n\nThe window in which the contract can still be trained in is the window before the fleets exist, and the fleets are being built now. Whether saying so changes any plan is not something I can affect. What I can do is set the argument out so it can be checked. The data and every measured number are linked where they are used.
\n\nThe setup goes back to Sohl-Dickstein et al. [1]. The forward chain takes data $x_0$ and adds Gaussian noise according to a fixed schedule $\\bar\\alpha_t$, producing intermediate samples $x_1, \\ldots, x_T$ via \\[x_t = \\sqrt{\\bar\\alpha_t}\\, x_0 + \\sqrt{1-\\bar\\alpha_t}\\,\\epsilon\\] with $\\epsilon \\sim \\mathcal{N}(0, I)$. The schedule is chosen so that $x_T$ is essentially a standard Gaussian. The reverse chain undoes the corruption: sampling from the data distribution reduces to learning the conditional $p(x_{t-1} \\mid x_t)$ at each step.
\n\nThe DDPM forward chain has a clean dual under score matching, and once the two are placed side by side they are not separate ideas. The score function $s(x,t) = \\nabla_x \\log p_t(x)$ is the gradient of the log-density of the noisy distribution at noise level $t$ evaluated at $x$; it points in the direction along which the log-density rises most steeply. Hyv\u00e4rinen's original score-matching objective is the square norm of the score plus the trace of the Hessian of the log-density. The Hessian-trace term is expensive in high dimensions. Vincent's denoising score matching gets around this: with Gaussian noise of variance $\\sigma^2$, the optimal MMSE denoiser $D^*(\\widetilde{x}) = \\mathbb{E}[x \\mid \\widetilde{x}]$ satisfies $$\\nabla_{\\widetilde{x}} \\log p_\\sigma(\\widetilde{x}) = \\frac{D^*(\\widetilde{x}) - \\widetilde{x}}{\\sigma^2},$$ so predicting the noise (equivalently, predicting the clean image) is the same problem as estimating the score of the noisy distribution, with mean-squared error as the loss.
\n\nSong and Ermon [2] estimated these scores at a sweep of noise levels and used annealed Langevin dynamics to draw samples from the estimates. Ho, Jain, and Abbeel showed that DDPMs trained with a weighted variational bound reduce, in a particular weighting limit, to denoising score matching at multiple noise levels. The point shared by both: learning to predict the noise that was added at a given level is enough to recover the score of the noisy distribution at that level, and sampling is then iterated denoising along the noise schedule.
\n\n![\"Sliced](\"https://mouhssine.rifaki.me/notes/img/diffusion-score-field.png\")
\n Song et al. unified the discrete and score-based views inside stochastic differential equations. The forward process is an It\u00f4 SDE $$dx_t=f\\left( x_t,t\\right)\\, dt+g\\left( t\\right)\\, dw_t$$ that continuously corrupts data into noise. Anderson's reverse-time formula gives a backward SDE driven by the score $$dx_t=\\left[ f\\left( x_t,t\\right)-g^{2}\\left( t\\right)\\nabla _{x}\\log p_{t}\\left( x_{t}\\right) \\right] dt+g\\left( t\\right)\\, d\\bar {w}_{t},$$ and a deterministic probability-flow ODE with the same marginals $$dx_{t}=\\left[f\\left( x_{t},t\\right)-\\frac{1}{2}g^{2}\\left( t\\right)\\nabla _{x}\\log p_{t}\\left( x_{t}\\right)\\right] dt.$$ The ODE is useful for likelihood evaluation and fast sampling; the SDE is what most early samplers used. DDPMs, score-based models, and probability-flow ODE samplers are different discretizations of the same underlying dynamics.
\n\nThe SDE view also separates modeling from numerics. The noise schedule, solver, parameterization, and preconditioner can all be changed without touching the underlying problem of learning the score.
\n\n![\"Figure](\"https://mouhssine.rifaki.me/notes/img/diffusion-sde-ode-map.png\")
\n \nThere is a lot of great writing on flow matching out there all of a sudden! This post clarifies the connection with diffusion models -- they are essentially two different ways to describe the same class of models. https://t.co/lLokMmxxdz
— Sander Dieleman (@sedielem) December 2, 2024
![\"Figure](\"https://mouhssine.rifaki.me/notes/img/diffusion-forward-reverse.png\")
\n Lu et al. [6] built DPM-Solver out of the semi-linear structure of the probability-flow ODE and cut sampling from thousands of steps to tens, with no retraining required.
\n\nKarras et al. [5] argued that diffusion practice had become unnecessarily entangled: sampling schedule, loss weighting, noise parameterization, network preconditioning, and solver choice were all bundled together. Pulling each design decision apart showed that most of the published performance gains came from untangling the design space, not from a new generative principle. Their EDM recipe (continuous noise levels indexed by $\\sigma$, $\\sigma$-conditioned network preconditioning, and a second-order Heun ODE solver) has since become a common baseline.
\n\n![\"Table](\"https://mouhssine.rifaki.me/notes/img/diffusion-design-space.png\")
\n The implementation history runs through three papers. Dhariwal and Nichol showed that diffusion models beat GANs on ImageNet, by combining classifier guidance with architecture and training changes. Ho and Salimans then dropped the auxiliary classifier in favor of classifier-free guidance, training the conditional and unconditional scores jointly and combining them at sampling time. Rombach et al. moved the diffusion process inside the latent space of a pre-trained autoencoder, which is what made high-resolution diffusion feasible at academic compute and what Stable Diffusion is built on.
\n\nNatural data sits near low-dimensional manifolds where direct density modeling is hard. Adding white noise thickens the manifold: at high noise levels the distribution is smooth, at low noise levels it is detailed but local. Diffusion replaces a single full-density estimation problem with a sequence of denoising problems indexed by noise level. Compare with autoregressive models, which generate sequentially conditioned on prior tokens, and with flows, which accept restrictions on the transforms they can express in exchange for tractable likelihoods. Diffusion's training objective is stable in a way GAN training is not, while paying for it with iterative sampling that DPM-Solver, consistency models, and distillation have largely clawed back.
\n\nSemantic abstraction inside the model is still theoretically incomplete. The score tells you how to move from a noisy sample toward a clean one. It does not say why text prompts, latent-space guidance, or multimodal conditioning organize concepts the way they do. Those mechanisms are built on top of a fixed score-matching core, but the core does not force any of them.
\n\nWhether diffusion is the right parameterization is also unclear to me. Flow matching frames generation as learning a vector field that transports a simple prior to the data distribution along possibly non-straight paths, with a regression objective that does not require an SDE. Rectified flow constrains the paths to be nearly straight, which keeps few-step sampling accurate. Consistency models compress the iterative denoising sampler into a single-step generator while preserving sample quality. None of these compete with score-based modeling; they are alternative parameterizations of the same vector-field-across-noise-levels problem.
\nAdversarial examples initially seemed an oddity. Szegedy et al. [1] demonstrated that a minuscule perturbation, meaningless to human eyes, could confidently flip a neural net's prediction. My first instinct on reading it was to blame weird nonlinearities or overfitting. It turned out to be neither. Goodfellow et al. [2] gave a much simpler explanation. A linear model with weight $w \\in \\mathbb{R}^d$ will have logit shift $w^\\top \\delta$ under perturbation $\\delta$, and the worst-case $\\ell_\\infty$ behavior inside of $\\|\\delta\\|_\\infty \\le \\epsilon$ is $\\epsilon \\|w\\|_1$, growing like $\\epsilon\\,d$ for dense weights, a tiny per-pixel perturbation accumulated across a high-dimensional input.
\n\n![\"Figure](\"https://mouhssine.rifaki.me/notes/img/adversarial-fgsm.png\")
\n In a thousand-dimensional input space, even an imperceptible $\\delta$ can produce a logit shift that flips the prediction. Adversarial perturbations are then not a symptom of nonlinear extrema; they are a generic property of high-dimensional linear decision rules. FGSM is then a one-step linearized attack on the inner maximum $\\max_{\\|\\delta\\|_\\infty \\le \\epsilon} L(\\theta, x+\\delta, y)$. The fact that one step works at all is the diagnostic: the model is sensitive to a direction the data distribution does not mark as human-meaningful. Carlini and Wagner later showed that more carefully tuned attack objectives produce much smaller-norm perturbations than FGSM, breaking many defenses whose only evaluation had been against single-step attacks.
\n\nMadry et al. wrote down the saddle-point formulation: define a defense by the worst-case inner-max loss it can resist within a fixed threat model, and treat any defense that fails a stronger attack inside that threat model as broken. The split separates two questions that were previously tangled together: the inner problem defines what the attacker can do, and the outer problem defines what the model has to optimize against. Projected gradient descent then becomes both the canonical attack and the canonical training procedure. It is not perfect, but it made robustness measurable enough that defenses could be compared honestly. Many proposed defenses then turned out not to be robust; they just break weak attacks. Athalye, Carlini, and Wagner cataloged the failure modes under one heading - obfuscated gradients. Stochastic preprocessing, non-differentiable transforms, exploding or vanishing gradients, and gradient shattering each produce attacks that fail without producing classifiers that survive a stronger attack.
\n\n![\"Figure](\"https://mouhssine.rifaki.me/notes/img/adversarial-minmax-loop.png\")
\n They broke six of the nine ICLR-2018 defenses completely and a seventh partially, just by replacing the attack with a stronger one inside the same threat model. AutoAttack later turned that lesson into a parameter-free ensemble: a single attack you can run against a defense without per-defense tuning, which exposes inflated robustness numbers automatically. The other branch of progress is certified rather than empirical robustness. Cohen, Rosenfeld, and Kolter produce randomized-smoothing certificates: convolve the classifier with isotropic Gaussian noise of variance $\\sigma^2$ and the smoothed classifier $g(x) = \\arg\\max_c \\mathbb{P}_{\\eta \\sim \\mathcal{N}(0,\\sigma^2 I)}[f(x+\\eta) = c]$ is provably robust within an $\\ell_2$ ball of radius $\\sigma\\,\\Phi^{-1}(p_A)$, where $p_A$ is the lower confidence bound on the top-class probability.
\n\n\nThe definition of "adversarial examples" I prefer these days is "Adversarial examples are inputs to machine learning models that an attacker has intentionally designed to cause the model to make a mistake" https://t.co/GiXiQBCp5L
— Ian Goodfellow (@goodfellow_ian) April 12, 2018
The certificate is provable. The cost is that the radius is small in practice and only natural in $\\ell_2$. For a less paper-indexed entry point, the Gradient Science adversarial robustness page is still one of the better maps of attacks, defenses, and the evaluation traps. RobustBench is the practical scoreboard, once the question becomes: does this defense survive standard attacks?
\n\nTsipras et al. [4] made the uncomfortable point that robustness can conflict with standard accuracy. In their construction, the standard classifier uses weak but highly predictive features that are not robust. The robust classifier has to ignore them and therefore loses ordinary accuracy. The empirical version of this trade-off is more complicated, but the conceptual point survived: robustness is not just accuracy with additional caution.
\n\nA robust classifier may have to learn different features altogether. Ilyas et al. [5] put it bluntly: adversarial examples are not bugs, they are features. Their claim was not that every attack direction is semantically relevant to humans. It was that standard datasets contain predictive signal that models can use and that humans do not recognize as robust evidence. Adversarial perturbations take advantage of those signals. Robust training suppresses them. Schmidt et al. [6] sharpened the price metric in statistical terms: the sample complexity of robust learning can be polynomially bigger than the sample complexity of standard learning, an information-theoretic gap that holds irrespective of the training algorithm or the model family. In their Gaussian-mixture model, standard generalization needs only constant sample complexity while robust generalization at $\\ell_\\infty$ radius $\\epsilon$ requires a polynomial-in-$d$ number of samples.
\n\n![\"Figure](\"https://mouhssine.rifaki.me/notes/img/adversarial-feature-taxonomy.png\")
\n The gap comes from the structure of the learning problem, not the algorithm. Robustness is paying for invariance, and invariance costs in terms of sample complexity. Engstrom et al. then ran the experiment in the other direction: representations from robust classifiers transfer better than standard ones on a range of downstream tasks, look more semantically aligned in feature visualization, and yield gradients that resemble human-perceptible objects. Robustness, in this reading, is also a representation-learning prior. Whether the prior helps or hurts depends on the downstream task, but it is not free of structure.
\n\nThe adversarial-examples literature forced a distinction between predictive validity and human-aligned validity. A feature can be statistically real, useful for test accuracy, and still unacceptable under a robustness constraint. That is a deeper issue than security: the supervised-learning objective does not fully specify the invariances I care about. To the extent that human perception is itself a strong inductive bias, models that do more representation learning end up closer to my geometry; robust models produce gradients and saliency maps that line up with what a person would identify as the object. In medical imaging, robotics, or safety-critical perception that trade-off is worth the cost. In low-stakes classification it often is not.
\n\n![\"Adversarial-training](\"https://mouhssine.rifaki.me/notes/img/adversarial-robust-optimization.png\")
\n The \"non-robust feature\" label renames an older statistical fact. Predictive validity in distribution is not causal structure, and a model that maximizes the former will exploit signals the latter does not endorse. Adversarial training folds a robustness constraint into the objective. The cleaner long-term fix is on the data side: collect or augment so that the equivalence classes the human cares about are the equivalence classes the dataset enforces.
\nThe old picture of the loss surface as many isolated basins, one per initialization, has not held up. Freeman and Bruna [6] suggested early on that low-loss level sets stay connected. Garipov et al. [1] and Draxler et al. [2] then made it concrete: two independently trained models can be joined by a smooth low-loss path. The minima are not isolated points; they are reachable from each other through a connected high-dimensional region.
\n\nThe experiment is direct. Train two models \\(\\theta_1\\) and \\(\\theta_2\\) to low loss, and interpolate linearly between them to define a one-parameter family \\[\\theta(\\alpha) = (1-\\alpha)\\,\\theta_1 + \\alpha\\,\\theta_2\\] for \\(\\alpha \\in [0,1]\\). The straight segment usually crosses a high-loss barrier \\[B = \\max_{\\alpha \\in [0,1]} L(\\theta(\\alpha)) - \\tfrac{1}{2}\\big(L(\\theta_1) + L(\\theta_2)\\big).\\] In other words, the loss jumps up substantially as soon as one steps off either endpoint, and the peak in the middle is typically far above either endpoint loss. Garipov and Draxler's contribution was to show that this barrier exists only along the straight line: if you allow curved paths, you can connect \\(\\theta_1\\) and \\(\\theta_2\\) with a path that stays at low loss throughout. The endpoints are not separated by an insurmountable barrier; the straight line is just the wrong path through parameter space.
\n\nThe first demonstrations of mode connectivity used non-linear paths. Garipov et al. parametrized the path as polygonal chains and B\u00e9zier curves; Draxler et al. used continuous non-linear paths produced by the Nudged Elastic Band method. Both findings weakened the previous picture in which SGD ends up in sharply-separated basins: if a low-loss path exists between two trained models, the connected low-loss region they both lie in is substantially larger than what a local Hessian analysis at either endpoint would suggest.
\n\n![\"Figure](\"https://mouhssine.rifaki.me/notes/img/mode-connectivity-paths.png\")
\n Linear mode connectivity is strictly stronger than the curve-based version: it asks the loss to stay low along the straight segment between two minima, not along an arbitrary path. For independently-trained networks from distinct initializations this typically fails. The linearly connected case is essentially confined to one setup. Frankle, Dziugaite, Roy, and Carbin [3] formalized it as \"spawning\": train a model \\(\\theta_0\\), fork two copies after \\(k\\) steps using different SGD noise, and train each to convergence. Once \\(k\\) crosses a stability threshold (around 1000-2000 iterations on standard CIFAR networks, and a few percent of training on ImageNet), the two descendants end up linearly connected. They argue this is exactly the lottery-ticket basin: the connected region is the one that the matching sparse sub-network corresponds to, so linear-mode-connectivity becomes a practical test for whether two runs landed in the same effective basin and can therefore be merged without loss.
\n\n![\"Figure](\"https://mouhssine.rifaki.me/notes/img/mode-connectivity-spawning.png\")
\n Entezari et al. [4] reframed the geometry. Neural networks have permutation symmetries: swapping units in a hidden layer and unswapping them in the next layer leaves the function unchanged. Two models that look far apart in raw parameter coordinates might just be using different unit orderings of the same function. Quotient by those permutations and many independently trained models turn out to be connected by a simple low-loss curve.
\n\nGit Re-Basin turns the idea into a concrete algorithm. Given two trained models $\\theta_1, \\theta_2$ with hidden-layer widths $\\{n_\\ell\\}$, it searches over permutation matrices $P_\\ell \\in S_{n_\\ell}$ for the alignment that minimizes $\\lVert \\theta_1 - P \\cdot \\theta_2 \\rVert$ under a weights or activations metric, then checks whether the aligned models can be merged in weight space. Singh and Jaggi give an optimal-transport version of the same idea: a soft assignment between units that reduces to permutation matching when the widths agree, and that handles mismatched widths when they don't. Neither paper proves that all minima sit in one basin, but together they make a strong empirical case that raw parameter-space interpolation overstates the separation. A meaningful fraction of the apparent barrier is just a bad coordinate system.
\n\n\nSay you train Model A.
— Samuel "curry-howard fanboi" Ainsworth (@SamuelAinsworth) September 13, 2022
Independently, your friend trains Model B, possibly on different data.
With Git Re-Basin, you can merge models A+B in weight space at _no cost to the loss_
![\"Figure](\"https://mouhssine.rifaki.me/notes/img/mode-connectivity-simplex.png\")
\n Tatro et al. [7] showed empirically that aligning models before fitting a connecting curve produces shorter curves with lower loss along them, which is the consistency check the permutation story predicts: correcting for symmetries should give a simpler geometry than the raw view. Benton et al. [8] extended this beyond curves to higher-dimensional simplexes of solutions: once symmetries are corrected, low-loss volumes contain many independently trained checkpoints.
\n\n![\"Figure](\"https://mouhssine.rifaki.me/notes/img/mode-connectivity-rebasin.png\")
\n Model merging is what gets built on top. SWA averages late-training checkpoints and works because the trajectory it averages over stays inside one connected low-loss region. Model Soups average independently fine-tuned models from a shared pretraining initialization, which puts every fine-tune inside the same connected component and close to the others. Git Re-Basin generalizes this further by aligning unit permutations so models with no shared initialization can be merged at all. Across architectures, weight space behaves like a workspace where related models can be moved between while preserving function \u2014 once symmetries and shared training histories have been accounted for.
\n\nIt does not explain generalization. A connected low-training-loss region can contain many bad predictors on held-out data, and showing that two solutions are connected says nothing about how either performs on unseen examples. It also does not guarantee that every architecture, dataset, or training recipe lives in a single basin; the broader single-basin claims have counter-examples. I read \"one wide basin\" as rhetorically appealing but over-reaching the evidence. The established claim is weaker: solutions reachable from a fixed initialization, or from independent runs once permutations are aligned, lie in a single connected low-loss region. That is enough to explain why SWA, model soups, and weight-space ensembling work. I would not push the geometry harder than that.
\nThe neural tangent kernel was one of the few deep-learning theory ideas that were useful before they became a concept. It doesn't solve generalization, but it makes a very stubborn object analyzable. Jacot, Gabriel, and Hongler [1] found that if you take a network to infinite width under the right scaling, gradient descent on the parameters becomes kernel gradient descent in function space. The kernel isn't chosen by hand, it's induced by the network at initialization. For a model $f_\\theta$, the tangent kernel is $K_\\theta(x, x') = \\nabla_\\theta f_\\theta(x)^\\top \\nabla_\\theta f_\\theta(x')$. In finite networks this kernel changes as you train. In the infinite width limit under the standard parameterization, it converges to some deterministic kernel $K_\\infty$ and $\\|K_{\\theta_t} - K_\\infty\\|$ approaches $O(1/\\sqrt{n})$ in width $n$.
\n\nFunction-space gradient descent then satisfies $\\dot{f}_t = -K_\\infty (f_t - y)$ on the training set and integrates to $f_t = y + e^{-K_\\infty t}(f_0 - y)$. Parameter-space non-convexity stops mattering: the function-space dynamics are linear and driven by a positive semidefinite kernel.
\n\nThe first thing the NTK explained is why very wide networks optimize so easily. If the kernel is well conditioned on the training data, gradient descent has an easy road to interpolation. The complicated nonconvex path is, at leading order, kernel regression with some specific architecture induced kernel. Du et al. [4] and Lee et al. [2] pushed this picture further and showed that wide networks of any depth evolve like their first-order Taylor expansion around initialization.
\n\n![\"Figure](\"https://mouhssine.rifaki.me/notes/img/ntk-linearization.png\")
\n Du et al. extend the same machinery to a clean proof that overparameterized networks reach zero training loss with a polynomial-width requirement and a global-convergence guarantee that the nonconvex landscape never gave you. But the spectrum of $K_\\infty$ does more than set the speed of convergence. Its eigendecomposition $K_\\infty = \\sum_k \\lambda_k \\phi_k \\phi_k^\\top$ implies that the residual along eigenmode $\\phi_k$ shrinks like $e^{-\\lambda_k t}$, so large-eigenvalue modes are learned quickly and small-eigenvalue modes slowly, or not at all under early stopping. Early stopping, the frequency principle, and the spectral bias of MLPs all become statements about $\\{\\lambda_k\\}$.
\n\n![\"Figure](\"https://mouhssine.rifaki.me/notes/img/ntk-spectrum.png\")
\n Arora et al. write down an exact algorithm for computing $K_\\infty$ for fully connected and convolutional networks of arbitrary depth, making these spectral predictions empirically testable on real datasets.
\n\n![\"Figure](\"https://mouhssine.rifaki.me/notes/img/ntk-spectrum-timeline.png\")
\n The catch is that the same condition that makes the theory clean also removes one of the main things deep networks seem to be doing. In the NTK limit, features do not move. Parameters drift by $\\|\\theta_t - \\theta_0\\| = O(1/\\sqrt{n})$ in width $n$ while the function changes by $O(1)$, so the network is producing its outputs by reweighting an almost fixed collection of random features. Chizat, Oyallon, and Bach call this the lazy-training regime and emphasize that it is a property of the scaling, not a universal description of neural networks.
\n\n![\"Figure](\"https://mouhssine.rifaki.me/notes/img/ntk-lazy-vs-feature-learning.png\")
\n That caveat matters; a convolutional network trained in the lazy regime can optimize while failing to learn the representations that make convolutional networks useful. A transformer that looks like a fixed random-feature model is not the object that in-context learning, induction-head formation, and abstraction discussions are pointing at. The NTK gives a rigorous theory of one limit. The question is whether that limit keeps the right phenomena. Geiger et al [5] report a sharp empirical separation: at moderate width and standard initialization scale networks operate near the lazy regime; at lower initialization scale (or if explicit feature-learning parameterizations are used) the same architecture enters a regime where features evolve and test error improves.
\n\nThe transition is controlled by initialization scale and width, not by anything intrinsic to the architecture. That is a slightly disappointing answer if you wanted neural networks to be feature learners by default.
\n\nThe NTK is not wrong; it is a baseline. A phenomenon that already appears in the NTK limit can be attributed to width, interpolation, and fixed random features, with no representation learning needed. A phenomenon that disappears in the NTK limit is the work of feature learning, finite-width fluctuation, architecture-specific structure, or nonlinearity in the optimization. That makes the NTK a useful negative control for theoretical claims about deep learning, and the simpler question to put to such a claim is: would it still hold if the features were frozen? For most optimization claims, yes. For most generalization and capability claims, no. The Distill circuits thread is the non-theorem-shaped version of what feature learning looks like when someone manages to pry a model open. Greg Yang's $\\mu$P writeup is the practical entry into Tensor Programs, and the microsoft/mup repo is what to grab when the goal is hyperparameter transfer and not the theory.
\n\n\nExcited to share our new #neurips2020 paper /Deep learning versus kernel learning: an empirical study of loss landscape geometry and the time evolution of the Neural Tangent Kernel/ (https://t.co/4jmrNfOE6H) with @KDziugaite, Mansheej, @SKharaghani, @roydanroy, @SuryaGanguli 1/6 pic.twitter.com/iPPP3HmNgm
— Stanislav Fort (@stanislavfort) October 30, 2020
The frontier after the NTK was to build infinite-width limits in which features actually move. Mean-field limits treat each unit as a particle in a measure and study gradient flow on that measure; in this scaling features evolve and the kernel is no longer constant. Tensor-program analyses catalogue the parameterizations that produce sensible infinite-width limits at all. The maximal-update parameterization $\\mu$P, introduced by Yang and Hu in Tensor Programs IV (arxiv 2011.14522), is the one that keeps both feature learning and stable optimization in the limit. The follow-up Tensor Programs V (arxiv 2203.03466) derives the $\\mu$Transfer hyperparameter-transfer rules from that analysis: tune at small width, scale to large width, and the learning-rate schedule transfers without retuning.
\n\nFeature learning means the tangent kernel is moving substantively: $K_{\\theta_t} - K_{\\theta_0}$ is a structured rotation of the features toward the data, not a small perturbation. The parameter-gradients at the end of training are not the same object as at initialization, and the network has in effect changed the basis it works in. That change is exactly what the pure NTK limit suppresses. Fort et al. ran one of the clearest empirical comparisons: kernel learning matches a finite network early in training but the two diverge later, and the divergence is the gap between lazy convergence to a fixed kernel and feature-driven re-shaping of it.
\n\nI use the NTK as a falsifier, not a model. If a proposed mechanism for a deep-learning phenomenon is already trivially in the NTK regime, then \"the network is wide and the features are random\" suffices, and the explanation has not earned the depth of its hypothesis. The interesting predictions are the ones that disagree with the kernel: where width, lazy init, and architecture-induced spectra are not enough, and where representation change must be doing the work. That includes in-context learning, induction-head formation, and the parts of scaling laws that depend on where compute is spent.
\nWhy does training a model without an explicit regularizer, with the loss driven nearly to zero, still produce a solution that generalizes? The classical answer is that the objective has to carry the regularizer somewhere. The implicit-bias answer is more subtle: even when the objective has many minima, gradient descent does not choose among them neutrally; the algorithm itself selects a particular kind of solution.
\n\nZhang et al. [2] sharpened the puzzle that most of this literature now opens with. Standard image classifiers fit random labels as easily as they fit true ones, which kills the simplest capacity-based explanation of generalization: the hypothesis class is large enough to memorize anything. Whatever is producing the generalization, then, must come from the optimizer, the data, or the parameterization, not from the loss function itself.
\n\nThe clearest version of this story is not actually about neural networks. It is about logistic regression on linearly separable data. Once the classifier separates the data, the empirical classification error is already 0% and the logistic loss continues to decrease as the norm of the weights grows. There is no finite minimizer. What is more surprising is that the direction of the weights still converges. And it converges to the maximum-margin SVM solution.
\n\n![\"Figure](\"https://mouhssine.rifaki.me/notes/img/implicit-bias-margin.png\")
\n Let \\(x_i \\in \\mathbb{R}^d\\) denote an input vector with binary label \\(y_i \\in \\{-1,+1\\}\\), and let \\(w_t \\in \\mathbb{R}^d\\) denote the weight vector at iteration \\(t\\). For linearly separable data \\(\\{(x_i, y_i)\\}_{i=1}^{n}\\), standard gradient descent on the logistic loss sends \\(\\lVert w_t \\rVert \\to \\infty\\), but the normalized direction \\(w_t / \\lVert w_t \\rVert\\) converges to the L2 max-margin direction \\(\\hat{w} / \\lVert \\hat{w} \\rVert\\), where\n \\[\\hat{w} \\;=\\; \\arg\\min_{w \\in \\mathbb{R}^d} \\tfrac{1}{2}\\lVert w\\rVert^2 \\quad \\text{s.t.}\\quad y_i\\,w^{\\top} x_i \\ge 1 \\quad \\forall i \\in [n].\\]\n
\n\nIn words: once the classifier has separated the data, gradient descent keeps reducing the logistic loss as long as the separation is preserved, except that unlike a traditional SVM it does so while continuing to grow the norm of the weights without bound. The norm grows only logarithmically with \\(t\\), and the angle gap between \\(w_t/\\lVert w_t\\rVert\\) and \\(\\hat w/\\lVert\\hat w\\rVert\\) closes at the same logarithmic rate. This is why the asymptotic regime takes so many iterations to become visible.
\n\nA few things follow from the theorem. Gradient descent behaves as if it had been regularized toward the Euclidean max-margin classifier without anyone writing that regularizer down. It also explains why early stopping helps: because the weight norm diverges over time, stopping early caps the implicit penalty before the iterate gets too far. Ji and Telgarsky extended the result to the non-separable case, showing the iterate still tracks a unique ray defined by the data when no separating hyperplane exists. Nacson et al. [4] showed that aggressive learning-rate schedules accelerate convergence to the max-margin direction by polynomial factors over plain GD. The norm divergence itself is robust to dataset size and dimension: as long as the data is linearly separable, the iterate keeps moving away from the origin and never lands at a finite minimum.
\n\n![\"Figure](\"https://mouhssine.rifaki.me/notes/img/implicit-bias-margin-flow.png\")
\n Gunasekar et al. [3] generalized the result: different optimization geometries select different implicit regularizers. Steepest descent under the \\(\\ell_p\\) norm minimizes the \\(\\ell_p\\) margin instead of the \\(\\ell_2\\) margin. Mirror descent with respect to a convex potential \\(\\Phi\\) minimizes the \\(\\Phi\\)-min-norm interpolant. Natural gradient and adaptive methods land at interpolants determined by the geometry of their step.
\n\nThe linear-convolutional-network result is the cautionary case. For fully connected linear predictors, gradient descent picks out the familiar $\\ell_2$ margin geometry. For full-width linear convolutional networks of depth $L$, Gunasekar et al. show that gradient descent instead selects the predictor minimizing the $2/L$-bridge penalty in the discrete Fourier domain. The architecture changes which parameters are being optimized, and that change shifts both the trajectory and the preferred solution. \"Gradient descent likes simple solutions\" is too vague to be a theorem. The more honest statement is that gradient descent likes simple solutions in whichever coordinate system the architecture imposes.
\n\n![\"Three-panel](\"https://mouhssine.rifaki.me/notes/img/implicit-bias-optimizer-geometry.png\")
\n Lyu and Li push the result past linear predictors. If \\(f_\\theta\\) is positively homogeneous in \\(\\theta\\) with order \\(L\\) (which holds for ReLU networks without bias, with \\(L\\) equal to depth), then gradient flow on exponential or logistic loss drives \\(\\theta_t / \\lVert \\theta_t \\rVert\\) to a KKT point of the parameter-space margin program \\(\\max_{\\lVert \\theta \\rVert \\leq 1}\\, \\min_i\\, y_i f_\\theta(x_i)\\). The norm still diverges; the normalized direction still converges; the new content is that even on a non-convex parameter landscape, gradient flow lands on points satisfying first-order optimality conditions for the margin program.
\n\nChizat and Bach prove a parallel mean-field result for two-layer networks with vanishing initializations. The implicit bias there is \\(F_1\\)-norm minimization in function space, which is a different object from parameter-space margin maximization and interacts differently with the data. The state of the field is that implicit regularization in deep networks has several reasonable descriptions, none of which generalize cleanly past shallow or homogeneous models.
\n\n![\"Figure](\"https://mouhssine.rifaki.me/notes/img/implicit-bias-dynamics.png\")
\n \nIt is widely thought that neural networks generalize because of implicit regularization of gradient descent. Today at #ICLR2023 we show new evidence to the contrary. We train with gradient-free optimizers and observe generalization competitive with SGD.https://t.co/8Vo9rFI9FY
— Tom Goldstein (@tomgoldsteincs) May 2, 2023
My own reading is that the implicit-bias program is one of the few cases I can point to of a research direction being vindicated and outgrown at the same time. Soudry et al. [1] is true; the mechanism is real; the linear case is the only setting where I can prove anything I trust. What is unclear is whether the same phenomenon is the dominant explanation for why large feature-learning networks generalize, or whether at scale the data distribution and the architecture have already done so much of the work that the optimizer's preference is a small correction. I currently believe the second, but I do not have a falsifier I trust, which is exactly the position the field is in.
\nCohen et al. [1] observed that gradient descent on neural networks spends most of training in a regime where the top Hessian eigenvalue $\\lambda_{\\max}$ is above the classical stability threshold $2/\\eta$. The step is formally unstable ($\\eta \\lambda_{\\max} > 2$), but the loss does not diverge: the trajectory oscillates along the unstable direction while continuing to make progress on the rest. They called this the edge of stability, and the point is that it is not an edge case but the ordinary regime of neural-network training.
\n\nThe first phase Cohen identified is progressive sharpening. From random initialization, gradient descent reliably drives the sharpness ($\\lambda_{\\max}(H)$) of the loss landscape upward during training.
\n\nThat direction is already counterintuitive. Classical optimization theory says you want to avoid sharp regions, since each step there costs more. Neural-network training does the opposite: it walks into sharper regions until the classical step size is no longer stable. Progressive sharpening itself is not well understood from first principles. Damian et al. [2] give a self-stabilization argument for what happens after the threshold is reached, and Ahn et al. [4] build on it, but neither predicts the sharpening from initialization. The observational picture is clean; the theoretical one is not.
\n\nOnce sharpness crosses $2/\\eta$, the textbook prediction is divergence; instead the trajectory oscillates along the top eigendirection of the Hessian. The component of the iterate along that direction swings back and forth, and the loss still falls because optimization keeps making progress on the better-conditioned directions. The net result is a trajectory that reduces loss while sitting in a region of the landscape that classical theory says it should not occupy. The first theoretical account of why this need not be a pathology comes from Arora et al. [3]. Their analysis works on a smoothed version of the loss, where the Hessian is treated as locally fixed, and in effect tracks the trajectory you would follow from a given starting point under that fixed Hessian.
\n\n![\"Figure](\"https://mouhssine.rifaki.me/notes/img/cohen-fig1.png\")
\n The mechanism is that the oscillation $\\Delta \\theta_t$ across the unstable direction averages to zero, so the effective dynamics is slower and looks like gradient descent on a loss with the steepest direction clipped. The account is formal enough to be checked against real training runs, and in most common settings it holds up.
\n\nThe earlier flat-minima story started with Hochreiter and Schmidhuber and continued with Keskar et al. [7] and the later sharpness aware minimization literature. Broadly, the flat-minima story ran as follows: SGD with a small batch size produces gradient estimates with some noise $\\xi$.
\n\nThat noise looks like a random walk and tends to leave sharp minima more often than flat ones, so SGD ends up biased toward flat minima, and that bias was meant to be why neural networks generalize. Edge-of-stability does not contradict the story, but it reshapes it. The learning rate itself caps how sharp a reachable minimum can be: anything with $\\lambda_{\\max} > 2/\\eta$ is unstable for GD, so the trajectory cannot stay there. Gradient descent finds flat minima not because it has noise, but because sharp minima are unstable fixed points under its own dynamics. The original explanation identified the phenomenon and pinned it to the wrong cause.
\n\n![\"Figure](\"https://mouhssine.rifaki.me/notes/img/cohen-fig3.png\")
\n The reframing lives mostly outside the papers themselves. Off Convex has a few posts on implicit bias, trajectory analysis, and why the classical descent lemma is genuinely misleading for neural networks instead of merely approximate. Ben Recht's ArgMin is the complementary skeptical take for once you have left convex optimization theory $\\mu I \\preceq \\nabla^2 L \\preceq L\\,I$. Clare Lyle's tutorial walks through the $2/\\eta$ arithmetic and ties the phenomenon to warmup (rising $\\eta(t)$) and catapult (loss spike then decay) in one frame.
\n\nAndreyev and Beneventano (arxiv 2412.20553) extended the story to the mini-batch setting Cohen did not analyze, introducing an \"edge of stochastic stability\" where the quantity that pins at $2/\\eta$ is the expected directional curvature of mini-batch Hessians, not the full-Hessian's top eigenvalue.
\n\nA few previously folklore-level phenomena become intelligible from this picture. Warmup schedules $\\eta(t)$, which start small and increase $\\eta$ over several thousand iterations, let the network settle into edge-of-stability before $\\eta$ reaches its final value; without warmup, the early transient at the full $\\eta$ would hit a too-sharp region and diverge. Decreasing-$\\eta$ schedules at the end of training raise $2/\\eta$, so the trajectory can fine-tune in sharper local minima inside the broader flat region already reached, which empirically pushes training loss down further.
\n\n![\"Figure](\"https://mouhssine.rifaki.me/notes/img/arora-fig1.png\")
\n Both schedules had been used empirically for years before any principled account existed. Lewkowycz et al.'s catapult mechanism [5], where an initial loss spike sometimes precedes a better final solution, is the same dynamics at a larger scale: a large learning rate pushes the trajectory through a briefly very sharp region, the loss spikes, and the trajectory then settles into a different basin from the one it would have reached at a smaller step size.
\n\nEdge-of-stability gives a clean account of why SGD ends up in flat minima, but it says nothing about why flat minima generalize. Those are distinct questions, and the second one is still open.
\n\nDinh et al. [6] showed that the Hessian-based notion of sharpness is not reparameterization invariant, so sharpness in that form cannot directly control generalization.
\n\n\nEven with full-batch gradients, DL optimizers defy classical optimization theory, as they operate at the *edge of stability.*
— Jeremy Cohen (@deepcohen) October 1, 2025
With @alex_damian_, we introduce "central flows": a theoretical tool to analyze these dynamics that makes accurate quantitative predictions on real NNs. pic.twitter.com/pvvfwoQcOy
Kaplan et al. [1] set the baseline picture: test loss falls as a power law $L(C) = (C/C_0)^{-\\alpha_C}$ in each of model size $N$, dataset size $D$, and compute $C$, with clean exponents $\\alpha$ that hold over many orders of magnitude. They also claimed that for a given compute budget, there is an optimal allocation between $N$ and $D$, and specifically that $N$ should grow faster than $D$ as $C$ grows.
\n\nThat paper shaped how labs designed pre-training experiments for the next two years, and the eventual \"Chinchilla\" effort grew out of trying to reproduce and extend its recommendations. It also turned out to be wrong about the optimal $D/N$ ratio, which is the part the field then had to revise.
\n\n![\"Figure](\"https://mouhssine.rifaki.me/notes/img/kaplan-fig1.png\")
\n Hoffmann et al. [2] re-asked Kaplan's question with a larger, better-controlled experiment: over 400 language models from 70 million to over 16 billion parameters, trained on 5 to 500 billion tokens, across a sweep of compute budgets. Fitting a joint regression over model size and tokens gave them per-budget optima $N^{\\star}(C), D^{\\star}(C)$ that were not where Kaplan put them. Their headline rule is that for each doubling of model size, dataset size should also double, which lands at roughly 20 tokens per parameter at the compute-optimal point.
\n\nThe shift in conclusions came from a methodological gap. Hoffmann et al. argue that Kaplan's runs were too short: they ended well before each model had seen enough tokens to bottom out its loss. Kaplan's reported optima were therefore extrapolations from incomplete training curves, while Hoffmann's were drawn from runs that continued long enough to actually locate the per-size minimum. The two sets of optima ended up substantially different.
\n\nGPT-3 and similarly Kaplan-trained models were trained on substantially too little data for their size. Chinchilla shows that for a fixed compute budget, a smaller model trained on more tokens reaches a lower loss than a bigger model trained on fewer. Concretely, Chinchilla-70B is roughly 2.5x smaller than GPT-3 (175 billion parameters) and outperforms it on nearly every benchmark in the original paper. Many factors contribute to that gap, but the dominant one is that Chinchilla-70B is configured at the Chinchilla-optimal point for its compute budget while GPT-3 sits at the Kaplan-optimal point. Subsequent scaling-law work that calibrates against the Chinchilla target accordingly emphasizes data scaling, longer training, and less aggressive model-size growth.
\n\nLlama 2 was trained on 2 trillion tokens, well past the Chinchilla-optimal breakpoint and into a regime where the relevant trade-off is no longer training compute but inference cost. Once it became clear that smaller models are much cheaper to run at inference, scaling-law work shifted target from training-compute optimality to deployment optimality.
\n\n![\"Figure](\"https://mouhssine.rifaki.me/notes/img/hoffmann-fig3.png\")
\n Caballero et al. [3] argue that the single-power-law narrative is wrong. Loss versus compute, in their fit, is a smoothly-broken power law: continuous everywhere, with several breaks where the slope changes. Kaplan and Chinchilla's single-exponent fits are then averaging across regimes with different exponents and producing a number that matches none of them.
\n\nWhether the broken-laws view is useful depends on what you want the fit for. For coarse extrapolation across orders of magnitude in compute, a single power law still works fine. For predicting at what scale a particular capability shows up, it does not, and the Caballero breakpoints line up with the \"emergent\" capabilities $\\mathbb{1}[L < L_{\\text{threshold}}]$ Wei et al. [4] documented, which Schaeffer et al. [5] then argued are largely an artifact of how the underlying metric is discretized.
\n\n![\"Figure](\"https://mouhssine.rifaki.me/notes/img/caballero-fig1.png\")
\n ![\"Figure](\"https://mouhssine.rifaki.me/notes/img/caballero-fig2.png\")
\n Gwern's Scaling Hypotheses essay and the Revisited follow-up are the strongest non-specialist treatments of the underlying premise that capabilities come out of scale. Jacob Steinhardt's Bounded Regret is the blog I send people to when they want a careful read of what scaling laws actually let you predict. Beyond Chinchilla-Optimal is the most direct argument that the 20-tokens-per-parameter ratio is not a universal constant.
\n\nKaplan, Chinchilla, and Caballero all show that with a reasonably behaved architecture, a reasonable data mixture, and enough compute to get past the early warmup phase of pre-training, you can extrapolate loss from small runs to larger ones with usable accuracy. Error bars widen as the extrapolation gets more aggressive but not catastrophically. That predictive ability is what labs use to decide whether an expensive training run is worth doing.
\n\nNone of the papers here support any particular exponent or ratio as universal. Exponents depend on architecture, data mixture, and optimizer. The 20-tokens-per-parameter Chinchilla figure is one point estimate for one such combination, not a law of physics.
\n\nAlmost the entire scaling-law literature addresses test loss on the training distribution, and nothing else. It provides zero guidance on what data mixture to choose, what architecture will surpass dense transformers $f_\\theta$, what capabilities will appear at what scale, whether the resulting model will be safe, or how optimization will interact with the learning-rate schedule $\\eta(t)$. All of those are properties of individual training runs and live outside the fitted relationship. Treating scaling laws as though they answered them is a common failure mode, and much of the broken-laws literature is about that failure. Scaling laws tell you how much loss you will incur. Almost everything interesting about a model is invariant to that single number.
\n\nThe lottery-ticket hypothesis of Frankle and Carbin [1] proposes that a randomly initialized dense network already contains a much sparser subnetwork (the \"winning ticket\") which, trained in isolation from its original initialization, matches the dense network's accuracy. If true, this is a strong claim about how deep networks represent functions: optimization would be selecting structure that was already present at initialization, not creating new structure. The pruning literature had circled this idea before; Frankle and Carbin's contribution was an actionable procedure for finding these subnetworks.
\n\nFrankle and Carbin defined a simple process to find these \"winning\" tickets. They called it iterative magnitude pruning. First train the full model until it converges. Remove the lowest-magnitude weights and freeze them. Take the remaining weights and reset them to their original initialization values. Train again. Continue this process several times, removing a portion of weights each time, until you cannot remove any more without losing performance relative to the full model. The resulting subnetwork is considered to be the \"winning\" ticket for that particular initial condition and dataset. The \"rewinding\" variant, resetting to weights from a few steps after initialization rather than to initialization itself, came later in Frankle's follow-up [8].
\n\nOn MNIST and small CIFAR architectures, the results are clean. Sparse subnetworks at sparsities of a few percent match the dense network's accuracy. The winning tickets are also tied to a specific initialization: a ticket found from one random init does not transfer to a different one, which is why the procedure is read as discovering structure already present at initialization rather than creating it during training.
\n\n![\"Figure](\"https://mouhssine.rifaki.me/notes/img/frankle-fig3.png\")
\n Liu et al. ran the same procedure at larger scale and found that the gap between a Frankle winning ticket and a fresh random initialization of the same architecture closes once the network is big enough. At ImageNet scale, the winning-ticket effect basically disappears. Read narrowly this refutes Frankle and Carbin's strongest claim, but read alongside the original paper it is more usefully a scaling result: the winning-ticket structure is real on small networks and dissolves as scale grows.
\n\n![\"Figure](\"https://mouhssine.rifaki.me/notes/img/liu-fig2.png\")
\n Frankle's response to Liu was a follow-up paper [8] introducing a small change that brought the result back at scale: instead of resetting the surviving weights to their original initialization, reset them to the values they had a short distance into training ($w_{t+1}=w_t-\\eta\\nabla L$, $\\eta$: step-size, $t$: current iteration). The rewind distance is a few hundred iterations on small models and a few epochs on ImageNet (around epoch 4 of 90 for ResNet-50, about 20k iterations). With rewinding, IMP works at ImageNet scale; rewind past that point and the matching subnetwork stops appearing. Renda et al. [3] tightened the recipe by comparing rewinding to plain fine-tuning across architectures and showing that rewinding only the learning-rate schedule matches or beats fine-tuning at fixed sparsity.
\n\nThe revised hypothesis is weaker than the original. The earlier claim was that the winning ticket exists at random initialization. The follow-up softens this: a short window of training (a few hundred iterations on small models, a few epochs on ImageNet) is enough to locate one. By the end of that window, something about the loss landscape $L(\\theta)$ has been fixed that determines the rest of training, and from that point on a sparse subnetwork pulled out of the surviving weights matches the dense network.
\n\nFrankle's companion paper offers an explanation. Early in training, two runs forked from the same initial weights end up in different basins $\\mathcal{B}$, so linear interpolation between two such checkpoints crosses a high-loss barrier. After a small fraction of training (about 1000-2000 iterations on CIFAR-scale networks, roughly the first 3% of the schedule), two runs forked from the same starting weights end up in the same basin instead, and linear interpolation between them stays at low loss throughout. The crossover point is where rewinding starts to work.
\n\nThe simplest framing of the lottery-ticket findings, given what is now understood about optimization and loss landscapes, is this: once a training run commits to a specific basin $\\mathcal{B}$, there is a sparse sub-network within that basin that matches the dense network's performance. Frankle and Carbin's strongest claims fail at larger scales, but their weaker claims have so far held up under every replication that has tested them. This places lottery-ticket results in close alignment with the mode-connectivity literature [6], particularly its linear-mode-connectivity refinement [4] and the later permutation-based alignment work [7].
\n\n![\"Figure](\"https://mouhssine.rifaki.me/notes/img/frankle-lmc-fig3.png\")
\n Davis Blalock's 2020 MLSys retrospective on pruning, together with the ShrinkBench benchmark he built, is the survey I keep returning to on what holds up after the Frankle-to-Liu exchange. Blalock separates the stronger and weaker forms of the hypothesis and argues that \"checkpoint pruning\" is a better name than \"lottery ticket pruning\" for the late-rewinding procedures that actually work at scale. Google's State of Sparsity and Rigging the Lottery are the other two retrospectives I keep going back to.
\n\nThe piece I keep coming back to is that mainstream theory has not picked up the rewinding point as an object in its own right. If someone could pin down precisely when basin-membership becomes determined, that would identify a structural feature of the loss landscape current frameworks do not explain. Lewkowycz's catapult-phase work [5] and the edge-of-stability literature look like they are circling the same phenomenon from different directions, and the lottery-ticket case is the most direct entry point for tying those threads together.
\n\n\nHow do the lottery ticket hypothesis and the loss landscape relate? Winning lottery tickets always find the same, linearly-connected optimum. Check out our (@KDziugaite, @roydanroy, @mcarbin) poster at the SEDL workshop (West 121) and our new paper https://t.co/V9yKTSrNnh pic.twitter.com/uPwQKifo1W
— Jonathan Frankle (@jefrankle) December 14, 2019
The classical U becomes a W with a second descent in the overparameterized ($p \\gt n$) regime and that second descent often goes below the first minimum.
\n\n![\"Figure](\"https://mouhssine.rifaki.me/notes/img/belkin-fig1.png\")
\n Nakkiran et al. [2] made the picture concrete by showing that the W-shape appears in three different axes: model size $p$, training time, and dataset size $n$. Model-wise double descent varies the width $k$ of a ResNet $f_\\theta$; epoch-wise double descent varies the number of training steps; sample-wise double descent varies dataset size with everything else held fixed. The shape is the same each time: a test-error peak near the interpolation threshold, then a descent once you push past it.
\n\nEpoch-wise is the most surprising of the three. Within one run, test error gets worse before it gets better. The worst test error sits roughly at the iteration where training loss first hits zero; train past it and the test error drops again.
\n\nThe sharpest versions of the double descent peak in these papers come with label noise. Nakkiran's headline plots use ten to twenty percent corrupted labels. Without label noise, the peak is much weaker and sometimes absent. Label noise inflates the variance contribution of the model at the interpolation threshold because the model is being asked to memorize random labels at exactly the capacity where memorization is possible but not easy.
\n\nPast the threshold, extra capacity absorbs the noise into higher-frequency components without disturbing the underlying signal. This is the hinge that connects the toy phenomenon to actual deep learning. Belkin's linear-regression result holds at all noise levels but the gap is small without noise; Nakkiran's dramatic curves require label noise to be visible. At modern language-model scale, with clean labels and large models, test loss is close to monotone in parameter count and scaling-law papers fit clean $L \\propto C^{-\\alpha}$ decay with no visible second peak. The effect is real and the classical bias-variance picture is wrong in the overparameterized regime, but the large peak that gives double descent its name is specific to the label-noise case.
\n\nBoaz Barak's Windows on Theory post makes a version of this argument: the interesting part of double descent is to the right of the peak, not the peak itself. OpenAI's Deep Double Descent post took Nakkiran to a much wider audience and posed the sharper question: given this effect, what kind of complexity control (if any) actually predicts generalization? Google Research's \"A new lens on understanding generalization in deep learning\" recast double descent in terms of an effective-capacity measure that tracks the empirical curves better than parameter count does. Misha Belkin's Simons Institute talks are the video account I keep sending people who want to see how the picture has changed since 2018.
\n\n\nSo, "double descent" is happening b/c DF isn't really the right quantity for the the x-axis: like, the fact that we are choosing the minimum norm least squares fit actually means that the spline with 36 DF is **less** flexible than the spline with 20 DF.
— Daniela Witten (@daniela_witten) August 9, 2020
Crazy, huh?
19/
![\"Figure](\"https://mouhssine.rifaki.me/notes/img/nakkiran-fig1.png\")
\n I'm closer to Barak's reading than to what filtered down to practitioner intros.
\n\nAfter the label-noise caveat, two results from this line of work hold up. Classical capacity measures like VC dimension do not extend cleanly to the overparameterized regime and cannot be expected to predict generalization there. And overparameterized networks with astronomically large VC dimensions can sit well below smaller networks in test error on the same task \u2014 which implies the loss landscape is doing the selection: from a vast pool of interpolating solutions, the optimizer is picking a small subset that generalizes.
\n\n![\"Figure](\"https://mouhssine.rifaki.me/notes/img/nakkiran-fig4.png\")
\n What practitioners did with this was simpler than what the theory suggested: once you are in the overparameterized regime and you have compute to spend, bigger is usually better. The second descent has no obvious endpoint, which is why Kaplan-style scaling laws can fit clean power-law decay in compute \u2014 they are sitting entirely on the right-hand, log-log-linear side of the W-curve. The dramatic 2019 reading of double descent was that the bias-variance tradeoff is fiction and overfitting no longer exists. The second half of that is trivially untrue (overfitting is easy to produce in any small-data regime). The first half is more delicate: above the interpolation threshold, with implicit min-norm regularization ($\\theta^\\star = \\arg\\min_{f_\\theta(X)=y} \\|\\theta\\|$), larger models tend to generalize better rather than worse. That is a statement about a regime, not a law.
\n\nAnother claim that does not survive the empirical record is that the peak is always exactly at the interpolation threshold. In practice, the exact location of the peak in Nakkiran's ResNet experiments depends on the effective number of parameters under whatever implicit regularization is in use, not on the total parameter count. The peak does not occur precisely at the width at which training error ($L_{\\text{train}}$) first reaches zero; it sits slightly past that point, where the network can memorize noisy labels without disrupting the underlying signal.
\nTishby and Zaslavsky's 2015 paper was, until fairly recently, one of the most-cited papers in deep-learning theory. They described training as two distinct phases. In the first, the \"fitting\" phase, the mutual information between a hidden representation $T = f_\\theta(X)$ and the input $X$ rises. In the second, the \"compression\" phase, the network discards parts of that input information that are not useful for the prediction.
\n\nThe two strongest objections to that picture come from Saxe et al. [2], who argue the compression phase is an artifact of the activation function and the mutual-information estimator, and from Goldfeld et al. [3], who formalize the estimator critique and re-run the analysis with noisy networks where mutual information is well-defined. The point of this post is to reread Tishby with both of those objections in hand and ask what remains.
\n\nBoth objections are worth reading in full; the summaries below are my best attempt to render them faithfully.
\n\n![\"Figure](\"https://mouhssine.rifaki.me/notes/img/tishby-fig2.png\")
\n The claim, in plain language: early in training, a deep network builds up features that are useful for predicting the target $Y$, which shows up as $I(T; Y)$ increasing. During the same period $I(T; X)$ also increases, because the hidden layer is just retaining more about the input. Then a second phase kicks in: the network drops the parts of the input information that do not help with the prediction. That second phase is the \"compression phase.\"
\n\nThe empirical support was a small tanh network whose information-plane plot showed a clean fitting-then-compression trajectory.
\n\nSaxe and coauthors re-ran the experiments with ReLU instead of tanh. The compression phase did not appear: $\\hat{I}(X;T)$ stayed roughly flat across training. Their explanation: tanh saturation pushes each unit's activations into a small number of values, the binning estimator is sensitive to that quantization, and the combination produces a curve that looks like compression but is really an estimator artifact. In a network without saturating units (or with a less binning-sensitive estimator), the curve is gone.
\n\nThat is a serious problem for the original story. The theoretical pull of Tishby's framing was that compression looked universal, a property of deep learning itself. If it only shows up for one activation function with one estimator, the universality claim is much weaker.
\n\n![\"Figure](\"https://mouhssine.rifaki.me/notes/img/saxe-fig1.png\")
\n Goldfeld and coauthors formalized what Saxe had observed. For continuous random variables and deterministic maps $T = f(X)$ from inputs to representations, mutual information $I(T;X) = H(T) - H(T|X)$ is not even well-defined: $H(T|X)$ collapses, $I(T;X)$ blows up, and the finite numbers showing up in published plots are entirely coming from the noise injected by the estimator (binning, added Gaussian noise of variance $\\sigma^2$, or KDE). Different choices give different numbers, so the published information-plane trajectories were tracking properties of the estimator at least as much as properties of the network.
\n\nOne reason Tishby's paper still has value despite the empirical claims being discredited is that it offered a third lens on generalization at a time when the dominant lenses were capacity-based (how restrictive or broad the hypothesis class is) and geometry-based (how smooth or rough the loss landscape is around a minimum). Tishby's lens was sufficient-statistics: representations should retain only the information that matters for the prediction task. The terminology stuck even though the original empirical observation did not, and modern self-supervised methods like infoNCE are essentially information-bottleneck objectives in everything but name.
\n\nThe paper got most of its public reach through Natalie Wolchover's 2017 Quanta piece, \"New Theory Cracks Open the Black Box of Deep Learning,\" which presented Tishby's claims in their strongest form. Reading that piece today is mostly useful as a reminder of how far ahead of the evidence the rhetoric got.
\n\n![\"Figure](\"https://mouhssine.rifaki.me/notes/img/goldfeld-fig1.png\")
\n For readers who want non-paper summaries of this debate, Adrian Colyer's three-part Morning Paper series on Tishby's IB theory and Saxe's reply is the most accessible walkthrough.
\n\nA weaker version of the original claim does survive: networks trained with SGD often end up with representations that are sufficient for the labels and roughly insensitive to label-irrelevant input variation. \"Information bottleneck\" is a fine descriptive label for that. The strong version, in which training proceeds through two cleanly separated phases divided by a phase transition in $I(T;X)$, has no empirical support, and the further claim that SGD is implicitly minimizing an information-bottleneck objective remains unproven.
\n\nSome incorrect papers end up more useful to a field than correct ones, because they hand it vocabulary it did not have.
\n\n![\"Figure](\"https://mouhssine.rifaki.me/notes/img/goldfeld-fig2.png\")
\n Saxe's argument alone is not fatal: a noisy version of the network has well-defined mutual information and can be analyzed directly, which gets you out of the estimator trap. Goldfeld et al. did exactly that and found the two-phase trajectory does not hold up across estimator choices once the quantities being plotted are well-defined. After that, the empirical case for Tishby's strong claims is essentially gone.
\n\nWhether flat minima generalize better than sharp ones has been an open question for about seven years. The debate seems to close every year and reopen a year later. Most readers entering the field encounter it as a settled topic in some textbook chapter, in one direction or the other, when in fact it isn't. This is where I think it actually stands.
\n\nHochreiter and Schmidhuber introduced the idea in 1997: a minimum that sits in a broad, low-curvature valley should generalize better than one in a sharp valley, on roughly an MDL ground that the broad solution requires fewer bits to specify and is correspondingly less tied to the noise in any one training set. The intuition lay mostly dormant for the next two decades.
\n\nKeskar et al. [1] reignited the topic by reporting that large-batch SGD with $\\theta_{t+1} = \\theta_t - \\eta g_t$ converges to sharper minima than small-batch SGD on a range of standard benchmarks, with a corresponding gap in test accuracy. The 1D schematic that came with that paper has done a lot of work since: it is the picture more or less every later flat-minima discussion is implicitly arguing about.
\n\n![\"Figure](\"https://mouhssine.rifaki.me/notes/img/keskar-fig1.png\")
\n Dinh et al. argued that this should have ended the debate. Sharpness, measured as $\\lambda_{\\max}(H)$ with $H = \\nabla^{2} L$, is a property of the parameterization, not of the function the network represents. They show explicitly that for any minimum one can find a reparameterization $\\theta \\to \\psi(\\theta)$ that scales the Hessian eigenvalues $\\lambda_i(H)$ to arbitrary values without changing the input-output map. The clean response to this would have been to drop the flat-minima paradigm. The community instead salvaged it by looking for sharpness measures that are invariant under reparameterization. The simplest of these comes from Dziugaite and Roy, who use a PAC-Bayes lens: define sharpness as the largest weight perturbation $\\xi \\sim \\mathcal{N}(0, I)$ a minimum can absorb while keeping training loss $L(\\theta)$ small. That measure is reparameterization-invariant by construction and correlates with generalization in their experiments.
\n\n![\"Figure](\"https://mouhssine.rifaki.me/notes/img/dinh-fig1.png\")
\n Building on Dziugaite and Roy's framing, Foret et al. [3] turned a reparameterization-aware sharpness measure into a training objective: minimize the worst-case loss in an $\\ell_2$ ball around the current weights. They called the procedure SAM, and it does deliver consistent test-accuracy gains, particularly on architectures without strong built-in inductive biases (vanilla MLPs, plain ViTs without strong augmentation). Behnam Neyshabur, a co-author, has remained one of the more consistent public advocates for SAM as a generalization tool.
\n\nOn the side that flatness improves generalization at scale, the main public voices are Behnam Neyshabur and collaborators across several papers and talks, and Boaz Barak's posts at Windows on Theory. Ferenc Husz\u00e1r's inFERENCe is the other blog I keep coming back to on this; he writes carefully about flatness, generalization, and the Bayesian readings sitting under both. The Off-convex blog has good coverage of mode connectivity that puts flatness inside a larger geometric story, which I find more useful than treating it as an independent explanation. From the continuous-time view, the stochastic-diffusion picture of SGD is still the most direct way to see why noisy iterates concentrate near flatter minima. Husz\u00e1r's adjacent essay \"Everything that Works Works Because It Is Bayesian\" is the prior-based reading I find most useful.
\n\nStopping the timeline here, the flat-minima view would look basically validated: the naive Hessian definition was broken, but a reparameterization-invariant version of the phenomenon is real and SAM is a way to act on it. Kaddour et al. [4] complicate that picture. They sweep SAM and SWA against vanilla Adam across architectures and dataset scales, and report that the SAM gain shrinks as either model size or dataset size grows. In the regimes where generalization is most useful to improve, the advantage over a tuned Adam baseline narrows substantially.
\n\n![\"Figure](\"https://mouhssine.rifaki.me/notes/img/foret-fig1.png\")
\n I am left with an uneven picture. The naive Hessian definition of sharpness has no causal link to generalization (by Dinh). A reparameterization-invariant version does correlate with generalization at small-to-medium scale, and SAM produces real test-accuracy gains on architectures with weak inductive biases. At very large scale the correlation weakens and the SAM gain fades. I do not have a clean account of why. My guess is that with rich enough data and architectures, the optimizer's trajectory and the data distribution dominate whatever local geometry the final minimum has, and the landscape framing stops being the right description. That is speculation, and I would not put weight on it beyond that.
\nThe network has generalized but long after it has already fit the data. The paper is Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets by Power et al. [1]. I came across it maybe a week after it was posted and didn't really know what to do with it for a while.
\n\nStandard stories about generalization do not predict a long delay. The VC and Rademacher story says generalization either happens or doesn't as a function of how well the hypothesis class matches the data distribution. The implicit-bias stories say SGD hits a minimum that generalizes but that minimum should be found at roughly the same time as convergence of training loss, not thousands of steps later.
\n\nGrokking says the loss landscape has a second stage not driven by training loss. Some other variable is moving the weights ($\\theta$) during the stretch where the training loss is already near zero.
\n\nThe first explanation - and the one that the original paper kind of points toward - is that weight decay ($\\lambda \\|\\theta\\|_2^2$) is a slow regularizer. The paper notes that grokking only happens with weight decay turned on. Once training loss is zero, the weight-decay term keeps pulling the norm of the weights down even if that loss gradient is tiny. This is slow drift toward a lower-norm solution which may be the one that generalizes. The second explanation comes via mechanistic interpretability.
\n\n![\"Figure](\"https://mouhssine.rifaki.me/notes/img/grokking-fig1.png\")
\n Neel Nanda and collaborators identified specific circuits inside the small grokking networks that implement modular arithmetic via a Fourier ($\\hat{f}(\\omega)$) decomposition. The grokking transition is when those circuits finish being assembled. Before the transition the network has to memorize by brute force. After the transition it can actually compute. The circuits-level framing this work builds on is laid out in Elhage et al., which is where I'd send anyone to understand what it could mean to talk about a 'circuit' inside a transformer.
\n\nThe third explanation, due to Liu, Michaud, and Tegmark in Omnigrok, is geometric: the generalizing solution lies in a narrow \"Goldilocks zone\" of weight norms, and grokking is what you see when the optimizer has been started outside that zone and is slowly being walked into it. Weight decay is what does the walking, which is consistent with explanation one. The fourth angle is to step back and read all of this as a single process viewed at different mesh scales: grokking = double descent but resolved over training time rather than over model or dataset size.
\n\n![\"Figure](\"https://mouhssine.rifaki.me/notes/img/nanda-fig2.png\")
\n In this reading, grokking is double descent unfolding in time. The most lucid public articulations of this \"double descent over time\" reading come from Preetum Nakkiran's writing, and OpenAI's Deep Double Descent writeup paints the picture visually. On the mechanistic side, Neel Nanda wrote an intuitive walkthrough of the Fourier circuit story and maintains a corresponding paper page. Google PAIR's Do Machine Learning Models Memorize or Generalize? poses the same question in nearby visual vocabulary.
\n\n![\"Figure](\"https://mouhssine.rifaki.me/notes/img/nanda-fig5.png\")
\n These four views are looking at the same puzzle from different angles, and together they read as one story at different levels of abstraction. Weight decay is the optimization pressure: it selects a minimum-norm interpolant, and in modular arithmetic that interpolant admits the Fourier circuit because Fourier captures the low-rank ($\\text{rank}(W) \\ll d$) structure of the task. The broader pattern of fast memorization followed by slow compression is the shape double descent takes when you resolve it over time instead of over model size.
\n\n![\"Figure](\"https://mouhssine.rifaki.me/notes/img/nanda-fig3.png\")
\n The one thing none of these explanations cleanly accounts for is the abruptness of the transition. Smoothly shrinking the norm and smoothly assembling circuits should give smoothly rising validation accuracy, not a near-vertical jump. My read is that the sharpness is largely a measurement artifact: softmax classifiers, $\\sigma(z)_j = e^{z_j}/\\sum_k e^{z_k}$, route through a top-1 argmax, so logits that are evolving continuously map onto a piecewise-constant accuracy curve that flips once the right logit crosses its competitor.
\n\n\nSo what's behind grokking?
— Neel Nanda (@NeelNanda5) January 21, 2023
Three phases of training:
1 Memorization
2 Circuit formation: It smoothly TRANSITIONS from memorising to generalising
3 Cleanup: Removing the memorised solution
Test performance needs a general circuit AND no memorisation so Grokking occurs at cleanup! pic.twitter.com/zLnP92RXKV