Onpode
Cover art for Why bigger models predictably get smarter — the empirical patterns that drive AI research

Why bigger models predictably get smarter — the empirical patterns that drive AI research

July 22, 2026 · 7 min

Hugo Vance & Lila Soto

The 2022 Chinchilla scaling law (Hoffmann et al., DeepMind) revealed that large language models were undertrained by roughly a factor of twenty — not because the underlying power-law math was wrong, but because earlier work optimized model size while ignoring training data volume. Parameters and tokens should scale in roughly equal proportion.

Scaling laws are empirical power-law relationships demonstrating that AI model performance — measured primarily through loss on language tasks — improves predictably as researchers increase three core resources: model size (parameters), training data, and compute.

0:007:15
Make your own on Onpode →

Describe any topic. Hear it in minutes.

More Onpode episodes on Artificial Intelligence →

About this episode

In 2020, OpenAI researchers published what felt like a map for building smarter AI: scale up parameters, and model performance improves in smooth, predictable curves across many orders of magnitude. Two years later, DeepMind ran the same power-law math and reached a different conclusion — that the largest models of the era had been undertrained by a factor of roughly twenty. Same framework, opposite prescription. This episode works through what that gap actually means. Not just as a historical correction, but as a live question about how reliable the map is right now. The conversation covers what scaling laws measure (cross-entropy loss on next-token prediction), what the Chinchilla result revealed about the original framework's blind spot, and why the contradiction isn't a sign the science is broken — it's a sign the original model fit the wrong variables. But the harder problem isn't the Kaplan-Chinchilla disagreement. It's emergence: the documented phenomenon where a model gains a qualitatively new capability at some threshold of scale, while the loss curve the whole time shows nothing but smooth descent. The curve is nominally on track. It just wasn't measuring what changed. Researchers at Stanford HAI and Harvard's Kempner Institute are actively revising pieces of the framework — learning rate schedules, data distribution effects — which is either reassuring science or a map that keeps shifting while billion-dollar training runs navigate by it. Probably both.

Frequently asked

What are AI scaling laws and why do they matter?

AI scaling laws are empirical power-law relationships showing that language model loss improves predictably as parameters, training data, and compute increase. Established by Kaplan et al. at OpenAI in 2020, they let researchers forecast model capability and allocate compute budgets before committing to expensive training runs.

What is the difference between Kaplan and Chinchilla scaling laws?

Kaplan et al. (2020) argued that scaling model parameters mattered most; training data was secondary. Chinchilla (Hoffmann et al., DeepMind, 2022) used the same power-law framework but found parameters and training tokens should scale in roughly equal proportion — revealing that models of that era were undertrained by a factor of about twenty.

How undertrained were large language models before Chinchilla?

Chinchilla scaling analysis found that large language models trained under the Kaplan recipe were undertrained by roughly a factor of twenty — meaning they needed approximately twenty times more training tokens to be compute-optimal. This was a systematic misallocation across the field, not a minor calibration error.

Can scaling laws predict emergent AI capabilities?

Scaling laws do not reliably predict emergent capabilities. Cross-entropy loss — the metric scaling laws track — can follow a smooth, on-curve descent while a model simultaneously crosses a threshold and gains a qualitatively new ability, such as multi-step reasoning. The loss curve measures one layer of change and is silent on the other.

Are AI scaling laws still considered reliable for guiding model development?

Scaling laws remain a primary planning tool, but researchers at Stanford HAI and Harvard's Kempner Institute are actively revising them — correcting for learning rate schedules, data distribution, and configuration effects. The curves have been robust across architectures, but robustness within a known regime is not the same as universality.

Grounded in 12 sources
Scaling Law with Learning Rate Annealing ↗ · arxiv.org
Reconciling Kaplan and Chinchilla Scaling Laws ↗ · arxiv.org
Neural Scaling Laws Rooted in the Data Distribution ↗ · arxiv.org
Position: Enough of Scaling LLMs! Lets Focus on Downscaling ↗ · arxiv.org
Scaling Laws for Neural Language Models ↗ · arxiv.org
Power Lines: Scaling Laws for Weight Decay and Batch Size in LLM Pre-training ↗ · doi.org
Scaling Laws in Linear Regression: Compute, Parameters, and Data ↗ · doi.org
New Approach to Scaling Laws Could Change How AI Models Are Trained | Stanford HAI ↗ · hai.stanford.edu
The Power of Scale in Machine Learning - Kempner Institute ↗ · kempnerinstitute.harvard.edu
Scaling Laws, Foundation Models, and the AI Singularity: A ... ↗ · researchgate.net
Anthropic's 'do more with less' bet has kept it at the AI frontier, Daniela Amodei tells CNBC - CNBC ↗ · cnbc.com
Google DeepMind CEO Demis Hassabis says AI scaling 'must be pushed to the maximum' - Business Insider ↗ · businessinsider.com
Read transcript

Hugo Vance: You sent me that paper at eleven at night with no context — just 'read this.' I did.

Lila Soto: And? I mean — did it land?

Hugo Vance: It landed. The Chinchilla correction. Hoffmann et al. at DeepMind, using power-law curves — the same mathematical architecture Kaplan et al. used at OpenAI in 2020 — concluding that large models of that era were undertrained by something like a factor of twenty. That is a significant number.

Lila Soto: Twenty times. Not a rounding error — that's a systematic blindness baked into how the whole field allocated compute. And that's actually where I want to start today, because the question underneath all of this is: if scaling laws are laws, how does the same power-law math produce one prescription in 2020 and the opposite prescription two years later?

Hugo Vance: The laws contradicting themselves.

Lila Soto: Yeah — and Jared Kaplan, the OpenAI author who gave the field the first recipe, goes on to co-found Anthropic, which is now one of the labs most committed to scaling. So the person whose paper got revised is now building the thing the revision says should have been built differently all along. That's not just interesting, it's kind of the whole tension.

Hugo Vance: And yet before we can hold that tension — the same math, two contradictory prescriptions — I think we need to actually say what a scaling law is. Not the formula. The intuition. Think of it like a mileage curve on a very long road trip. You don't need to drive the whole highway to know your rate. The first fifty miles tell you. And that rate — it holds. That's what Kaplan et al. at OpenAI established in 2020: on a log-log plot, cross-entropy loss — how well the model predicts the next token — improves approximately linearly as you add parameters, data, compute. Smooth. Predictable. Many orders of magnitude.

Lila Soto: So a researcher reading that paper, late 2019, deciding whether to build one 175-billion-parameter model or spread the compute — the curve says go big on parameters?

Hugo Vance: The Kaplan recipe said exactly that. Favor parameters; data matters less at the margin. She builds the large model. And then Hoffmann et al. at DeepMind run the same power-law math — and find she starved that model of training tokens by a factor of roughly twenty. Compute-optimal allocation, they call it: parameters and training tokens should scale in roughly equal proportion. The curve was real. The reading of the curve was wrong.

Lila Soto: Twenty times undertrained. That's — I mean, that's not a calibration issue.

Hugo Vance: No. And here's where I'd be cautious about calling this a law correcting itself. A law doesn't produce opposite prescriptions from identical inputs. What Chinchilla scaling revealed is that the original framework optimized the wrong variable — it fit loss curves as a function of model size and missed the interaction with data volume entirely. That's not self-correction. That's a gap in the original model.

Lila Soto: Hm — so the practical payoff, capability forecasting, extrapolating these log-log curves before you commit a compute budget — that whole enterprise depends on having fit the right variables in the first place?

Hugo Vance: Yes. And there's a harder version of that problem coming — because the loss curves stay smooth even when the model's actual behavior does something the curve didn't predict. We'll get to that. The predictability is real, and it may also be hiding something.

Lila Soto: And that hiding — that's actually the thing that won't let me go. Because Daniela Amodei and Anthropic's whole public position is basically: we know roughly what we're buying when we scale. The curves have held. But then you get emergent capabilities — model does something it literally could not do before, at some threshold of scale, and the loss curve the whole time? Smooth descent. No bump. No warning.

Hugo Vance: That's the crack in the glass.

Lila Soto: Picture someone at Anthropic — it's, I don't know, a Friday afternoon, they're reviewing eval results from a new checkpoint — and cross-entropy loss is down, exactly on the curve, everything nominal. And then a benchmark they'd been tracking as flat just... lights up. Multi-step reasoning, something it couldn't do at all. The curve didn't predict that. The curve measured the wrong layer of what was changing.

Hugo Vance: Yes. And I think we have to be careful not to overstate — the sourcing on emergence is genuinely contested. But the gap between 'cross-entropy loss improved' and 'a qualitatively new capability appeared' is real and unresolved. Those are two different questions. The framework answers one cleanly and is silent on the other.

Lila Soto: So is it measuring the wrong thing when it matters most?

Hugo Vance: That is the live question. And it's why the formal 'Enough of Scaling LLMs' camp — the downscaling position — has traction. Because if emergence is real and unpredicted by the curves, then inference-time compute, efficiency, different axes entirely, those aren't retreats. They're bets that the three-variable model — parameters, data, compute — is already an incomplete map.

Lila Soto: Yeah — and I don't think either of us can say confidently which bet is right. The curves have been robust across architectures and domains, which is why universality claims keep getting made. But robust within a regime isn't the same as universal. That gap is still open.

Hugo Vance: Well. Stanford HAI and the Kempner Institute at Harvard are actively rewriting pieces of the framework — learning rate schedules, data distribution effects, configuration-level corrections. You can read that as reassuring. The science is honest, it's stress-testing itself. Or you can read it as the map keeps changing, which is — I mean, that is not nothing, when billion-dollar training runs are navigating by that map.

Lila Soto: Yeah — and I think that's what I'll actually carry out of this. 'Scaling law' is almost a misnomer. It's a working model the field keeps pressure-testing, and the pressure tests keep finding edges. The bet isn't that the curves hold forever. It's on whoever understands what the curves can't see.

Hugo Vance: You sent me that paper at eleven at night. No context. And I think what I didn't expect — it's not that the correction was large. It's that the correction was possible at all inside the same framework. That's either the health of the thing or its exposure. I'm genuinely not sure which.

Lila Soto: Good. I'll take genuinely not sure. Thank you for sitting with it.