Megan Skiendel: There's a number I can't stop thinking about. Seven orders of magnitude. Not seven percent. Not sevenfold. Seven orders of magnitude — that's a dollar versus the GDP of the entire United States. And a single curve held across that whole range. That's what OpenAI published in January 2020.
David Sterling: Kaplan and Amodei. The scaling laws paper.
Megan Skiendel: Exactly. And the core idea is almost insultingly simple when you strip it down — AI gets better not randomly, but on a schedule. Predictably. You measure cross-entropy loss, which is just how wrong the model is on its predictions, and it falls in a smooth power-law as you scale up model size, data, compute. Every time. No noise.
David Sterling: The bread analogy actually works here. Double your flour and double your practice hours — the loaf improves by the same margin every time, across every kitchen. That's the structural claim.
Megan Skiendel: And then — listen — that curve, which came from a curve-fit, not a mathematical proof, became the basis for billions in compute spend. The whole capital allocation logic of this industry rests on an empirical observation across seven orders of magnitude. Which is... actually not the same thing as knowing it's true.
David Sterling: That gap — between 'empirically robust' and 'formally proven' — is exactly the slipperiness the whole episode turns on.
David Sterling: The gap broke open in 2022. DeepMind — Hoffmann et al. — publishes Chinchilla. And the finding is, frankly, brutal. Kaplan's framework said pile on parameters. The optimal ratio turned out to be roughly equal — parameters and training tokens scaled together. The field had a four-to-one ratio wrong for two years.
Megan Skiendel: PaLM 540B. Already in motion.
David Sterling: Already in motion. Theoretically suboptimal the day Chinchilla landed. That's the capital-destruction part nobody prices — the compute was spent, the ratio was wrong, and the IsoFLOP curves made it visible in retrospect. Parabolic shape, fixed compute budget — you can see exactly where the optimal balance sits. And PaLM wasn't on that curve.
Megan Skiendel: And look — I'd bet there were people inside those labs who understood the parameter-to-data problem before Chinchilla published. They just didn't have permission to say it. Competitive pressure, boardrooms wanting 'bigger is better for the benchmark' — Chinchilla didn't change the math, it changed who could walk into a room and say what was already obviously true.
David Sterling: Which means the planning culture was reactive, not predictive. And that's — wait, actually, that's the uncomfortable corner here. If Kaplan's curve was wrong enough to constitute a capital-destruction event, what confidence does anyone have in the 2022 revision?
Megan Skiendel: Exactly. The correction corrected itself once. That's not reassuring.
Megan Skiendel: But here's what I want to pull on — what does the curve actually rest on mechanistically? Because 'it works empirically' is not a mechanism. What's underneath it?
David Sterling: The Zipfian explanation. Patterns in language — and apparently in images, in EHR timelines — they're distributed with a heavy tail. A few patterns are everywhere, most are vanishingly rare. And as you scale, you cover more of that tail. Power-law input, power-law output.
Megan Skiendel: That's almost too clean.
David Sterling: It is. The loss surface constraint adds the other half — more parameters and more data together restrict the solution space, they compound. That's why it generalizes across diffusion transformers, EHR models, not just language. The architecture changes, the Zipfian structure of the domain apparently doesn't.
Megan Skiendel: Which — wait, okay, that actually makes the data pruning result stranger. Because if the mechanism is tail-coverage, then removing examples should hurt you. But the research says selectively pruning low-quality data can push performance past the standard power-law curve. Toward exponential, even.
David Sterling: That's the thing I can't fully resolve. If pruning beats the curve, then maybe the law was always secretly describing data quality, not just size. The noisy examples were — I mean, they were actively suppressing tail-coverage efficiency. You weren't measuring scale, you were measuring scale contaminated by garbage.
Megan Skiendel: Which reframes the whole capital problem. Suddenly it's not a compute question — it's a curation question. And Anthropic, Kaplan's own shop, is the one stress-testing exactly this.
David Sterling: The question I keep not being able to answer — and I want to sit with this — is whether anyone at Anthropic, Kaplan himself, could see the next Chinchilla coming. Or whether the correction is always retrospective by definition.
Megan Skiendel: Honestly? I think it's always retrospective. The Chinchilla correction didn't emerge because someone at DeepMind predicted Kaplan would be wrong. Hoffmann's team just ran the IsoFLOP curves and the parabola showed them.
David Sterling: Which means — I mean, that's the thing I can't resolve. The supercollapse result, loss curves collapsing onto a single universal curve tighter than random-seed noise — that's either profound or it's the most elegant measurement artifact ever produced.
Megan Skiendel: Both, maybe.
David Sterling: Both, maybe. And Demis Hassabis is publicly engaging the question of whether scaling is slowing, which — frankly, that's not nothing. That's the CEO of the lab that published Chinchilla saying out loud that the regime has limits.
Megan Skiendel: And Kaplan and Amodei left OpenAI and built Anthropic, which is — still inside the same paradigm, still betting the curve holds. I don't know what to do with that. They corrected the field once by leaving. What does it mean that they stayed in the framework?
