Lila Soto: Hugo, I have a question that's been bothering me since Tuesday — if you trained a model four times smaller than a competitor's, on the same budget, and yours won, would you tell anyone?
Hugo Vance: I'd write a paper about it, presumably.
Lila Soto: That's exactly what Hoffmann et al. did. March 2022, the Chinchilla paper drops — seventy billion parameters, beats Gopher, which is two hundred and eighty billion parameters, same compute budget. And Gopher is three months old. Built at the same institution. This isn't revisionist history, it's like — the ink was barely dry.
Hugo Vance: Which names something important about how fast institutional memory can calcify. Kaplan et al. at OpenAI published their scaling laws in 2020 — two years earlier — formally showing that performance follows power-law curves across more than seven orders of magnitude of scale. The field read that and concluded: bigger model, fixed budget, always. That became the recipe. Gopher was the recipe.
Lila Soto: Wait — so the Kaplan paper wasn't wrong, but everybody drew the wrong conclusion from it?
Hugo Vance: The power law was real — on a log-log plot, the relationship between loss and scale traces a straight line, and that held across an extraordinary range. The error was in the allocation it implied. Hoffmann's team trained four hundred models to find the actual compute-optimal point, and the answer was not 'maximize parameters.' It was: scale model size and training tokens equally. Chinchilla was built on that ratio, and it won.
Lila Soto: Four hundred models just to find the right ratio — that's not a hunch, that's a grid search on the whole problem.
Hugo Vance: Indeed. Which raises the question this episode really wants to sit with — if the field's core prescription reversed once inside two years, what's underneath the curve that makes it trustworthy at all? Or does it need to be trustworthy in that way to be useful?
Lila Soto: And I guess I'd add — does the curve describe what's actually getting better, or just a loss number that's a proxy for something we haven't fully named yet.
Hugo Vance: Cross-entropy loss. That's the metric — next-token prediction error. The scaling laws describe how that number falls as you add scale. Whether it maps cleanly to capability is, well, a separate question.
Lila Soto: Yeah — and that gap is exactly where I want to go.
Hugo Vance: That gap — between the loss number and the thing we actually care about — is real, but let's earn it. Before we get there, I want to make sure the underlying shape is clear, because the intuition is genuinely strange until it clicks. Think about getting better at parallel parking. The improvement from your first attempt to your tenth is enormous. Tenth to hundredth — still meaningful, but you can feel the returns shrinking. Thousandth to ten-thousandth? You're splitting millimeters. That flattening — that is a power law. Not a rocket ship. Not compounding forever. A curve that keeps improving but costs more and more effort for each smaller and smaller gain.
Lila Soto: It's not exponential — it's more like... the harder you push, the less you get back.
Hugo Vance: Exactly that. Formally: loss is proportional to scale raised to a negative exponent — but don't hear the equation, hear what it means. Each time you multiply your model by ten, your error drops by a fixed fraction. Same fraction, every time. Which sounds encouraging until you realize the absolute improvement keeps shrinking.
Lila Soto: And when Jared Kaplan's team plotted this — they saw it hold across seven orders of magnitude? That's not like, two data points.
Hugo Vance: Seven orders of magnitude on a log-log plot, and it traces a straight line. That's the empirical signature — on logarithmic axes, a power law looks linear. Which is, yes, almost unsettlingly clean.
Lila Soto: That seems almost too clean. Across that range — different model sizes, different everything — and it's just a straight line?
Hugo Vance: It held across the Transformer architecture, yes. And the slope of that line — the scaling exponent, which researchers call alpha — tells you precisely how much the loss drops per order-of-magnitude increase in parameters, or data, or compute. That slope is what makes small experiments predictive. You train something modest, you measure the slope, you extrapolate upward. That was the promise.
Lila Soto: Huh — so the slope is actually doing the forecasting work.
Hugo Vance: It is. And the metric that makes any of this comparable across models is cross-entropy loss — next-token prediction error. Not benchmark scores, not human ratings. The model's raw error at guessing the next word. That consistency is what lets you draw one line across wildly different experiments. Now — and this is the part people skip — there's also a floor term in the equation. An irreducible loss, L-zero. The minimum error you could ever reach, no matter how large you scale, because the data itself contains ambiguity. Real language is not fully predictable. That floor doesn't move.
Lila Soto: Wait — so scaling forever still hits a wall?
Hugo Vance: Hard ceiling, yes. Set by the entropy of the data distribution itself — not by engineering limits. Which is, I think, the part that should temper the more expansive claims about what scale alone can achieve. The curve is real. The exponent is measurable. And the floor is also real.
Lila Soto: So the line is straight, the slope is readable, but the line can't go down forever — what does that mean for how much anyone should trust an extrapolation past the data they've actually measured?
Hugo Vance: That extrapolation problem — that's exactly where the Chinchilla correction bites hardest. Because the floor being real isn't the only constraint. The shape of the curve you're extrapolating from depends entirely on conditions you've already fixed: your architecture, your data, your optimizer. Change any one of those, the curve shifts. And Kaplan et al. measured under specific conditions, drew a line, and the field treated that line as universal.
Lila Soto: Which is what made the Chinchilla result so — I mean, it wasn't a tweak. Hoffmann's team trained four hundred models, seventy million parameters up to sixteen billion, tokens ranging from five billion to five hundred billion, and the finding wasn't 'adjust slightly.' It was: the whole allocation was wrong.
Hugo Vance: Wrong for the compute-optimal regime, yes.
Lila Soto: Imagine a researcher. Not Tuesday morning, let's say Friday afternoon before a budget deadline. She has fifty million dollars for a training run. Under Kaplan's prescription, she spins up a two-hundred-billion-parameter model and skimps on tokens. That was the defensible choice. That was what the law said.
Hugo Vance: And Chinchilla says she should cut the model size dramatically and match it with far more data. Equal scaling — each doubling of parameters paired with a doubling of training tokens. The same fifty million, entirely different bet.
Lila Soto: And the first choice — the Kaplan choice — is just wrong?
Hugo Vance: The paper says simply wrong, yes. Which — you see, that's what makes this not a refinement. Gopher, two hundred and eighty billion parameters, built at DeepMind in late 2021, was the Kaplan prescription executed at scale. Chinchilla, seventy billion, trained three months later at the same institution, beat it on the same compute budget. The error wasn't old thinking. It was current.
Lila Soto: Same institution, three months apart.
Hugo Vance: Twenty-four months from Kaplan et al. publishing to Hoffmann et al. reversing the core prescription. That is — well, in a field that calls these things 'laws,' that is a very short time for a law to require complete inversion.
Lila Soto: Which makes me wonder — does the word 'law' actually survive this? Like, physics has laws. This feels more like a working heuristic that held until someone tested it methodically enough to break it.
Hugo Vance: I'd be cautious about 'law' precisely because of this. A law implies some architectural necessity, some physical constant. What we have is a regularity that held under Kaplan's conditions — Transformer architecture, cross-entropy loss, specific data regimes. Hoffmann changed the allocation question, and the prescription collapsed. That's a pattern, not a law.
Lila Soto: A pattern that a generation of very large, data-undertrained models was built on.
Hugo Vance: Indeed. And there's a further layer — the smooth loss curve that Kaplan measured and that Chinchilla revised? It says nothing about what happens when capability doesn't improve smoothly. Which is the part that actually complicates this whole picture, and we haven't touched it yet.
Lila Soto: Oh — you mean when the model just suddenly gets good at something, no warning from the curve?
Hugo Vance: Exactly that — no warning from the curve. That's the problem. The cross-entropy loss keeps descending smoothly, the log-log line keeps behaving, and then at some threshold, the model just... solves multi-step arithmetic. Or starts writing code. Abilities that weren't there at all, and then were. Researchers call these emergent capabilities, and the defining feature is that the loss curve gave you nothing. No slope change. No inflection point. Nothing.
Lila Soto: So the curve is — wait, is the curve just measuring the wrong thing?
Hugo Vance: Not wrong for resource allocation. For deciding how to spend compute, cross-entropy loss is still your best instrument. But for forecasting when a model becomes qualitatively different — capable of reasoning, say — it's incomplete. The map is accurate. The territory just contains features the map wasn't drawn to show.
Lila Soto: Hm. The map-versus-territory thing. So you're scaling up, loss is ticking down, everything looks predictable, and then — someone runs a benchmark on a Thursday and the model can suddenly do something it couldn't do Wednesday.
Hugo Vance: And no scaling law formulation currently predicts where that jump occurs, or why. That's not a gap people are quietly papering over — it's an acknowledged open problem.
Lila Soto: Which means disciplined curve-reading still misses it.
Hugo Vance: Yes — and that's the harder version of the naïve extrapolation failure. The naive version is projecting from benchmark headlines, which scaling laws exist to discipline. The subtler version is: you're reading the curve correctly, you're using the exponent rigorously, and the capability you actually care about still jumps discontinuously somewhere the curve doesn't mark. You've done everything right and you're still surprised.
Lila Soto: Oh that's — yeah, that's unsettling. The rigor doesn't protect you.
Hugo Vance: Well, it protects your budget. It doesn't protect your capability forecast. Those are different things, and I think conflating them is — actually, that conflation is where a lot of the overclaiming happens.
Lila Soto: And this gets worse if the architecture changes, right? Because the exponent you measured doesn't travel?
Hugo Vance: Precisely. The scaling exponent is measured under fixed conditions — same Transformer architecture, same data distribution, same optimizer. Each new model generation changes at least one of those. When that happens, the curve you measured shifts. The slope may not transfer. Which means the forecast from your last family of experiments may not apply to your next one, and you won't know until you've already built it.
Lila Soto: I mean — that's the Chinchilla correction again but one level up. The methodology itself keeps needing correction.
Hugo Vance: Which is, you see, why a 2026 ICML study from Stanford HAI — Sanmi Koyejo is the senior author — introduced improved approaches to scaling efficiency. Not revised numbers. Revised methodology. The curve isn't wrong; the instruments for reading it are still being sharpened. That's the honest state of the field.
Lila Soto: So the thing that was supposed to be the stable foundation is still being built while people are building on top of it.
Hugo Vance: Yes. And I think that's actually the honest landing. Not that the foundation is shaky — that it's being built in real time, with real instruments, and the instruments have limits that practitioners are now, at least, aware of. That's different from where the field was in 2020.
Lila Soto: The thing that I keep — I mean, what actually clicked for me through all of this is simpler than I expected. The diminishing returns are real. Measurable. You can put a number on exactly how much loss drops per order of magnitude, read it off the log-log slope, and that forecast is honest. That part holds.
Hugo Vance: It holds until the conditions it was measured under change. Which they do.
Lila Soto: Right — but knowing that is the instrument working, not failing. Like, the Kaplan-to-Chinchilla correction in twenty-four months, the emergent capability gaps, Sanmi Koyejo's team still revising methodology in 2026 — none of that says 'throw out the curve.' It says know what the curve is actually measuring.
Hugo Vance: Every field goes through this. Medicine, physics — you get a regularity, it looks universal, you call it a law, and then the boundary conditions surface. The discipline isn't in having a perfect law. It's in being honest about when your assumptions are no longer being tested.
Lila Soto: Heuristic, not law. Better instruments, not better certainty.
Hugo Vance: That's where I'd leave it, yes. The curve is the best map for allocating compute that the field has produced. Knowing where it stops matching the territory — the irreducible floor, the emergent jumps, the exponent that doesn't travel across architectures — that's not a failure of the map. That's knowing how to read one.
Lila Soto: That's — yeah. That's a good place to stop. Thank you for walking through it with me.
