Onpode
Cover art for The loss landscape mystery: how overparameterized models avoid overfitting

The loss landscape mystery: how overparameterized models avoid overfitting

October 17, 2026 · 15 min

Jonathan Ingles & Ben Okonkwo

Overparameterized neural networks avoid overfitting because gradient descent implicitly gravitates toward flat regions of the loss landscape, where many interpolating solutions coexist and small perturbations — like new data — cause minimal error spikes. Zhang et al. (2017) and Belkin et al.'s double descent curve (2019) established this empirically, but the causal mechanism remains unproven for general architectures.

Modern deep neural networks routinely operate in an overparameterized regime — containing far more trainable parameters than training examples — yet consistently generalize well to unseen data. This directly contradicts classical statistical learning theory, which predicts that fitting more parameters than data points should result in overfitting and poor out-of-sample performance.

0:0015:17
Get the next episode on Artificial Intelligence →

Follow it free — new episodes land in your feed.

Or make your own — any topic, in minutes

More Onpode episodes on Artificial Intelligence →

About this episode

Every ML course teaches the same rule: more parameters than data points and you're memorizing, not learning. Find the sweet spot in the middle. The problem is that rule is wrong — not subtly wrong, but demonstrably broken by experiments that have been in the literature since 2017. This episode traces how the field got here and what a more accurate picture actually looks like. The conversation starts with a specific experiment: a neural network trained on randomly labeled data memorizes everything perfectly, while the same architecture trained on real data generalizes. Same capacity, wildly different outcome. That asymmetry cracks open the real question — not how many parameters a model has, but what the geometry of its loss landscape looks like and where the optimization process tends to land. From there, the episode works through double descent, flat versus sharp minima, and what it means that skip connections measurably flatten the loss landscape. It takes seriously the distinction between observation and mechanism — the correlation between flat minima and good generalization is solid, but the causal proof for general architectures hasn't been formally closed. Poggio, one of the architects of modern learning theory, walked back the flatness hypothesis. That matters. The episode ends where the field actually is: scaling trillion-dollar infrastructure on a very expensive correlation, with no principled way to predict when implicit regularization fails. That's not a rhetorical flourish — it's the live problem.

Frequently asked

What is double descent in machine learning?

Double descent, mapped by Belkin et al. in 2019, is the phenomenon where test error follows a U-shape, then spikes sharply at the interpolation threshold — where parameters roughly equal training examples — and then decreases again as the model becomes heavily overparameterized. The classical U-shaped bias-variance curve captures only the first half.

Why do overparameterized neural networks generalize instead of overfitting?

Overparameterized neural networks generalize because, past the interpolation threshold, thousands of zero-training-error solutions exist and gradient descent gravitates toward flat regions of the loss landscape. Flat minima are robust to small weight perturbations — including the distributional shift new data introduces — while sharp, narrow minima memorize and fail to generalize.

What is implicit regularization in neural networks?

Implicit regularization is the tendency of overparameterized neural networks to prefer simpler, flatter solutions without any explicit penalty term in the loss function. Chiang, Ni, Miller et al. (2023 ICLR) showed this bias toward flat minima is optimizer-agnostic — it persists across different optimizers — suggesting it arises from the overparameterized regime and data geometry, not SGD specifically.

What did Zhang et al. 2017 show about neural network memorization?

Zhang et al. (2017) showed that a standard deep network trained on randomly labeled data memorized every label with zero training error, while the identical architecture trained on real structured data generalized. This proved that a network's capacity does not determine whether it memorizes or generalizes — the optimization process and data structure do.

Does flatness in the loss landscape cause better generalization?

Flat minima consistently co-occur with better generalization — networks in flat loss-landscape basins are robust to small weight perturbations that mimic new data — but a formal causal proof valid for general architectures has not been derived. Poggio, who helped build the theoretical scaffolding for this field, retreated from the flatness hypothesis, and the failure conditions for implicit regularization remain uncharacterized.

Grounded in 10 sources
Closed-Form Steepest Descent Direction toward Flat Minima: Reducing Upper Bounds on the Loss Hessian Eigenspectrum in Neural Networks ↗ · arxiv.org
Implicit Regularization and Generalization in Overparameterized Neural Networks ↗ · arxiv.org
Visualizing the Loss Landscape of Neural Nets ↗ · arxiv.org
Techniques for mitigating overfitting in machine learning: a comprehensive review, taxonomy, and practical guide ↗ · pmc.ncbi.nlm.nih.gov
Why do Larger Models Generalize Better? A Theoretical Perspective via the XOR Problem - Tel Aviv University ↗ · cris.tau.ac.il
What is Double Descent in Machine Learning? ↗ · datacamp.com
Overparameterization in neural networks: from application to theory ↗ · escholarship.org
Modern understanding of overfitting and generalization in machine learning - Learning with Machines ↗ · jedick.github.io
Why Neural Networks Generalise, and Why They Are (Kind of) Bayesian — LessWrong ↗ · lesswrong.com
Unpacking the Implicit Norm Dynamics of Sharpness- ... ↗ · ojs.aaai.org
Read transcript

Jonathan Ingles: Ben — every statistics course you ever took, every ML textbook, what do they tell you about overfitting?

Ben Okonkwo: More parameters than data points and you're memorizing, not learning. The bias-variance tradeoff. Optimal model complexity is somewhere in the middle — U-shaped curve, you shoot past the bottom and performance degrades.

Jonathan Ingles: Franky, that framework is wrong. Not in a subtle way. Not 'incomplete for edge cases' wrong. Empirically broken, and has been demonstrably broken since at least 2017.

Ben Okonkwo: Okay, that's a strong claim — what's the specific evidence?

Jonathan Ingles: Zhang et al. 2017. They took a standard deep network, trained it on data with randomly assigned labels — pure noise, no structure — and it memorized every label. Zero training error. Then the same architecture, same size, trained on real data with actual patterns — it generalized. The architecture didn't change. The data did. So the network's capacity is not what's being selected. The thing doing the selecting is the optimization process itself.

Ben Okonkwo: Hm — so capacity is necessary but not sufficient. The network can memorize, it just doesn't have to.

Jonathan Ingles: And then Belkin et al. in 2019 mapped what the actual error curve looks like when you scale capacity. It's not U-shaped. Test error decreases, then spikes sharply right at the interpolation threshold — the point where parameters roughly equal your number of training examples — and then, in the overparameterized regime, it decreases again. They called it double descent. The U-shaped curve isn't the floor, it's just the first half.

Ben Okonkwo: Wait — it actually gets worse before it gets better a second time? That spike is real?

Jonathan Ingles: That spike is the entire point. At the interpolation threshold you've squeezed the model into the worst possible configuration — just enough parameters to fit the data, no slack, maximally sensitive to noise. Push past it into heavy overparameterization and suddenly there are thousands of solutions with zero training error to choose from. Interpolating solutions. And the optimization can afford to be selective.

Ben Okonkwo: So the plain-language version is something like: in a landscape with billions of dimensions, there are vast flat regions — and those flat regions are where generalizing solutions cluster. More dimensions means more flat terrain to land in, not more ways to get lost.

Jonathan Ingles: That's exactly it. Flat minima generalize. Sharp, narrow minima memorize. And overparameterization, counterintuitively, makes the flat terrain easier to find.

Ben Okonkwo: Now — the part I want to flag is whether we know that as a mechanism or just as an observation. Because those are different things.

Jonathan Ingles: That distinction is exactly the right place to push. Observation versus mechanism. And here's what we actually have: a loss landscape. Every possible combination of weights your network could have — billions of numbers — maps to a single value, how badly the model is performing. The geometry of that surface is what determines whether gradient descent finds something that generalizes or something that memorizes.

Ben Okonkwo: So the surface itself — not the parameter count — is the real object we should be reasoning about.

Jonathan Ingles: Right. And Li, Xu, Taylor, Studer, and Goldstein — their 2017 paper, 'Visualizing the Loss Landscape of Neural Nets' — they actually rendered these things. Took the landscape, projected it down to two or three dimensions so you can see it. Some architectures produce something like a smooth mountain bowl. Others look like... shattered glass. Jagged peaks everywhere.

Ben Okonkwo: Wait — they could literally see it?

Jonathan Ingles: Projection onto a random 2D slice, yeah. Not the full surface, obviously — you can't visualize a billion-dimensional space. But their key finding was that skip connections — the architectural choice ResNet famously uses — flatten the landscape. Measurably. And the networks with flatter landscapes generalized better. That's the visual evidence connecting architecture to geometry to generalization in one paper.

Ben Okonkwo: Now, interesting — so the mechanism question is: why does a flat basin generalize where a sharp spike doesn't? Because I want to make sure we're not just saying 'flat is good, sharp is bad' without the actual reasoning.

Jonathan Ingles: Picture a researcher — she's fine-tuning a vision model, late afternoon, she nudges one weight value by a tiny amount. In a flat minimum, her loss barely moves. In a sharp minimum, that same nudge sends her off a cliff. New data — which is always slightly different from training data — is essentially that nudge. Flat means robust. Sharp means brittle.

Ben Okonkwo: The flatness is actually a stability guarantee against distribution shift.

Jonathan Ingles: Empirically, yes. Theoretically — it gets uncomfortable. The Hessian eigenspectrum is how you mathematically fingerprint the curvature at a minimum. Smaller maximum eigenvalues, flatter the region. And Yuto Omae et al. recently derived closed-form analytical bounds on that spectrum — actual formulas telling you which directions in weight space push you toward flatter terrain.

Ben Okonkwo: Hm — so that's the navigation tool. Bounded Hessian eigenvalues tell you which way 'flatter' actually is.

Jonathan Ingles: Right — and what Poggio's retreat from the flatness hypothesis tells me is that the correlation is solid, the causal chain is not. We know flat minima and good generalization co-occur. We don't have a clean proof that flatness causes it for general architectures. Those are genuinely different claims and the field has been — frankly — lazy about keeping them separate.

Ben Okonkwo: Which means the high-dimensional geometry research — the result that three-layer networks have few problematic second-order critical points, two-layer networks have no first-order critical points along training trajectories — that's relevant here, because it's telling you why gradient descent doesn't get stuck in the jagged regions. The sharp spikes mostly aren't traps, actually. The question is whether where it lands is flat enough.

Jonathan Ingles: Right — and that's the setup for what's actually happening with SGD. Because the landscape not having traps is one thing. Where SGD actually wanders is the operative question. Picture a machine learning engineer — Saturday afternoon, she's reviewing training logs for a 40-billion-parameter model. It's generalizing beautifully on web text. Her colleague ran the identical architecture on randomly shuffled labels as a sanity check — same optimizer, same learning rate, same everything — and it memorized every single training example. Zero error. Zero generalization. Same model, wildly different outcome. That's Zhang et al. live in the wild.

Ben Okonkwo: And the only variable is what the data looks like.

Jonathan Ingles: Exactly. Real web text has structure — patterns that thousands of different weight configurations can represent. So when SGD takes its noisy steps, it wanders into a flat basin where lots of nearby solutions also work. Random labels have none of that. Every training point is arbitrary, so the only way to hit zero error is to carve out a sharp, narrow spike memorizing each one individually. The optimizer lands somewhere completely different.

Ben Okonkwo: So the noise in Stochastic Gradient Descent is actually doing work here — it's not a bug to be minimized, it's what steers you away from the sharp spike.

Jonathan Ingles: That is implicit regularization. No penalty term in the loss function. Nobody told the optimizer to prefer simple solutions. The complexity control is emerging from the optimization process itself — from SGD hitting the data structure and sliding toward flat terrain.

Ben Okonkwo: Now — I want to be careful here, because that framing could imply this is an SGD-specific quirk. And I don't think that's what the evidence says.

Jonathan Ingles: Chiang, Ni, Miller et al. — 2023 ICLR. They tested implicit regularization behavior across different optimizers, and the effect doesn't depend strongly on which optimizer you're using. It's generic. Which means it's structural — it's coming from the overparameterized regime and the data geometry, not from something idiosyncratic about SGD's update rule.

Ben Okonkwo: Oh, that's interesting — so the bias toward flat minima survives even when you swap out the optimizer entirely.

Jonathan Ingles: Which is what makes the Saturday-afternoon result so clarifying. The engineer's model isn't generalizing because she picked SGD over Adam. It's generalizing because real data has structure that makes flat basins available — and in the overparameterized regime, the training process finds them almost regardless of how it's searching.

Ben Okonkwo: And the interpolating solutions piece closes the loop — she's hitting zero training error, which classically would mean she's overfit catastrophically, but she's actually just found one of many flat-basin solutions that happen to also generalize.

Jonathan Ingles: That's the interpolation threshold insight. Zero training error in the overparameterized regime doesn't mean failure. It means you've crossed into territory where interpolating solutions — solutions that fit every training point exactly — can still generalize, because the flat basin they live in is robust to the small perturbations new data introduces.

Ben Okonkwo: So — and I want to flag this because it matters — what we still don't have is a clean causal proof that flatness drives generalization for general architectures. The correlation is solid, the structural mechanism points the right direction, but the theoretical reconciliation is genuinely incomplete. And that gap is going to matter a lot when we get to whether practitioners can actually predict when implicit regularization fails.

Jonathan Ingles: That's the live question. And frankly the uncomfortable one.

Ben Okonkwo: So that gap — and I want to actually sit with it rather than just name it — isn't a gap between 'we're pretty sure' and 'we're certain.' It's a gap between correlation and a derivable causal mechanism. The Hessian eigenvalue flatness co-occurs with generalization. Consistently. But the step from 'co-occurs' to 'causes' — for general architectures — that step hasn't been formally closed.

Jonathan Ingles: And Poggio retreating from the flatness hypothesis is the tell. That's not a minor researcher hedging. Poggio helped build the theoretical scaffolding this field runs on.

Ben Okonkwo: Right — but what does that actually mean for someone operating at scale? Because I can't quite resolve this: the classical bias-variance tradeoff was also theoretically clean. And it was wrong.

Jonathan Ingles: That's exactly the pattern repeating.

Ben Okonkwo: So — a regulatory reviewer, say. November, she's evaluating a proposed 70-billion parameter model for a medical diagnostics application. She asks: will implicit regularization hold here? Will this thing generalize to patient populations outside the training set? And the honest answer, given what Chiang et al. found about the behavior being optimizer-agnostic, is... we think so, but the conditions under which it fails are not — I mean, they're genuinely not characterized.

Jonathan Ingles: We don't know when it breaks. That's the live problem.

Ben Okonkwo: And Zhang et al. showed both outcomes — memorization and generalization — with identical architectures. But they didn't characterize the boundary. We still can't predict from first principles which regime you land in.

Jonathan Ingles: Which means near the interpolation threshold — where double descent peaks, where test error actually spikes — practitioners have no principled rule for where to operate. It's navigation by intuition dressed up as engineering.

Ben Okonkwo: That's — yeah. That's the part that should be uncomfortable.

Jonathan Ingles: The institutional parallel to classical learning theory isn't decorative. The bias-variance tradeoff was wrong for years before textbooks updated. Now we're scaling on a flatness hypothesis that Poggio himself walked back, calling implicit regularization solved because Chiang et al. showed it's generic — without actually having the causal theory that tells you when 'generic' stops being true.

Ben Okonkwo: Okay — so what would actually settle it? What's the experiment or derivation that closes the gap between 'flatness correlates with generalization' and 'flatness causes it'?

Jonathan Ingles: Frankly, a theoretical proof valid for general architectures — not just two-layer or three-layer networks — that derives generalization bounds directly from Hessian eigenspectrum properties. Omae et al. got us analytical bounds on directions toward flat terrain. That's navigation. It's not causation. Until someone closes that derivation, the field is making trillion-dollar infrastructure decisions on a very expensive correlation.

Ben Okonkwo: And that's — I mean, that's the real shift, isn't it. We've moved past arguing about whether overparameterization works. The question is whether we'd see it when it stops working. And I genuinely don't know what that signal looks like.

Jonathan Ingles: Nobody does. That's the honest position. The structural tendency in high-dimensional non-convex optimization to settle into flat, generalizing minima — that's real, it's durable, the empirical record on it is strong. But 'durable' isn't the same as 'universal.' And right now, practitioners and regulators can't predict the failure conditions. There's no map of where implicit regularization breaks down.

Ben Okonkwo: Right — but the part that doesn't fit is that warranted confidence built on an incomplete theory is still doing real work in the world. Trillions of parameters. Systems making actual decisions.

Jonathan Ingles: Warranted confidence on incomplete theory is a different thing from warranted confidence on understood theory. The field keeps collapsing that distinction.

Ben Okonkwo: That's the thing I keep landing on, actually. Either we get a complete theory of implicit regularization — derivable, general — or we need a systematic empirical map of exactly when and where it fails. One of those two. Without one or the other, we're scaling on trust.

Jonathan Ingles: And the question at the end of all of this isn't 'does it work.' It's — what would failure look like. Would anyone see it coming.

Ben Okonkwo: Yeah. And I don't think either of us has that answer.

Jonathan Ingles: No. Which is — frankly — the only honest place this particular thread ends.

Ben Okonkwo: Good conversation. Genuinely.