Deep learning theory is the attempt to explain why networks that should fail by classical statistics keep working. Three threads carry most of the weight. Optimization asks how gradient descent navigates a wildly non-convex loss surface and still lands in usable minima; the surprising answer is that high dimensions make bad local traps rare. Generalization asks why models with more parameters than data points do not simply memorize; implicit regularization and phenomena like double descent reframe the old bias-variance story. Scaling laws then make the whole thing predictable: loss falls as a power law in compute, data, and size.
High-dimensional geometry rescues optimization because a critical point is only a true minimum when every direction curves upward at once. In a space with millions of parameters, that demands millions of independent coin flips landing the same way; the odds collapse as dimension grows. Most points where the gradient vanishes are therefore saddles, curving up along some axes and down along others. A saddle always offers an escape direction, so gradient descent rarely freezes; it slides off along the downhill curvature. The few genuine minima that remain tend to sit near the bottom, where the low loss lives.
Random matrix theory treats the Hessian at a critical point as a draw from a random symmetric ensemble, whose eigenvalues follow Wigner’s semicircle law. A point’s stability is set by how many of those eigenvalues are negative, and the height on the loss surface acts as a rigid shift of the whole spectrum. Low on the surface, the semicircle sits mostly to the right, so almost every direction curves up and the point is nearly a minimum. Raise the height and the distribution slides left, dipping more eigenvalues below zero. Bray and Dean derived this exact coupling for Gaussian landscapes.
Real Hessians honor the spirit of the Gaussian picture while violating its letter. Choromanska’s spin-glass mapping needed independence assumptions that trained networks plainly break, so it survives as intuition rather than proof. What instruments actually find, in measurements by Sagun, Papyan, and Ghorbani, is not a semicircle but a spectrum split in two: a dense bulk piled at zero, signaling flat and redundant directions from overparameterization, plus a handful of large positive outliers, often roughly one per class. Negative eigenvalues exist but stay few and shallow. The surface is ruled less by strong saddles than by pervasive flatness.
Flatness is the most credible bridge from that empty bulk to generalization, though a cracked one. The reasoning is clean: a wide basin means many nearby weight settings yield the same loss, so the solution barely moves when the data shifts from training sample to true distribution, and robustness to perturbation is what generalization demands. Hochreiter and Keskar built the case; sharp minima track the large-batch failures. Yet Dinh showed sharpness can be reparameterized away, so flatness is a powerful heuristic, the target of methods like SAM, rather than a theorem. Geometry, not parameter count, carries the explanation.