Hugo Vance: You seemed distracted when you came in — what is it?
Lila Soto: I've been trying to explain to my partner what we're covering today, and the thing I kept getting stuck on is — okay, today we're getting into scaling laws for AI, which sounds technical, but the actual question underneath it is almost philosophically weird: what if you could predict how capable a machine would be just from a formula, without understanding anything about how it thinks?
Hugo Vance: Yes. That is the core of it.
Lila Soto: And the thing I couldn't get past — Hestness et al. at Baidu, that's 2017, 2018, they were already seeing this pattern. Test error falling the same way across image classification, machine translation, language modeling. Same curve. And it kind of just... sat there for years?
Hugo Vance: It required Jared Kaplan at OpenAI to formalize all three axes together in 2020 before the machinery moved. Before that, you had one axis at a time. Kaplan showed parameters, training data, and compute all independently produce the same power-law improvement in loss — the mathematical form is L proportional to X to the negative alpha — and crucially, you can fit that curve on small cheap runs and forecast a much larger model's loss before committing to the full training cost.
Lila Soto: Okay, for anyone whose eyes are glazing over at the formula — what does that actually feel like as an idea?
Hugo Vance: Think of learning to play piano. Every extra hour of practice closes a fixed percentage of the gap between you and mastery — not a fixed number of mistakes corrected, a fixed fraction of what remains. Double your hours and the curve bends the same way whether you're playing scales or Chopin. That consistent shape, that's the power law. The extraordinary thing Kaplan showed is that neural networks follow the exact same geometry when you feed them more parameters, more data, or more compute.
Lila Soto: Which is — I mean, that's the thing, right? It's not a theory about why intelligence improves. It's just: we measured, it does, predictably.
Hugo Vance: Precisely. And once you have a reliable prediction, you no longer need to bet on architectural novelty. Labs like OpenAI, Google, Meta — they could stop asking 'what clever design will get us there' and start asking 'how much compute can we buy.' That inversion is the whole story.
Lila Soto: And that raises the question of whether something important got lost in that trade — but I want to hold that, because first I want to understand how the Chinchilla work from DeepMind kind of quietly said the original recipe was wrong.
Hugo Vance: Yes — and Chinchilla is the complication we'll come to. But before we get there, the thing I want to sit with is what that inversion actually *did* to the room. Because it's not abstract. You have Kaplan et al. publishing in 2020, showing all three axes — parameters, data, compute — simultaneously predictable. And what that means, practically, is that a funding committee now has a forecast document instead of a bet.
Lila Soto: A forecast document. Like — literally a spreadsheet?
Hugo Vance: Essentially, yes. Think of it this way. Two proposals go into review. One is an architectural novelty — genuinely interesting, unknown payoff. The other is a scaled Transformer. Same design that already works. But this second team has run small cheap training runs, fitted the power-law curve, and they walk in with a predicted loss figure for the large model before a single GPU-hour of full training is spent. The committee funds the second one. Not because the first idea is bad. Because the forecast *exists*.
Lila Soto: That's — oh, that's kind of devastating actually.
Hugo Vance: It is the mechanism, yes. Stanford HAI framed this explicitly — scaling laws as resource-allocation tools. The question 'how much compute do we need?' becomes the central strategic question. Not 'what architecture should we try.'
Lila Soto: But — okay, didn't someone still need to invent the Transformer in the first place? I mean, you can't scale nothing.
Hugo Vance: Correct. And that's the precise point. The Transformer was the architecture. Once it existed and scaling laws described its behavior, the research question inverted — it became 'how much?' not 'what next?' Architecture didn't stop mattering. It just... stopped being the active frontier.
Lila Soto: Which is how you end up with OpenAI's entire GPT series being, at its core, the same underlying design, just — bigger.
Hugo Vance: Exactly that. And not only OpenAI. Google, Meta, Microsoft — multi-year roadmaps, all pivoting around this predictability. Because if you can forecast loss from a cheap run, and you have the capital to actually buy the compute at scale, the rational move is obvious. The labs that couldn't afford to verify the curve at massive scale were simply excluded from the frontier.
Lila Soto: Wait — so the predictive capability before training is also the thing that concentrates the whole field? Like, it's not just a research tool, it's — hm, it's almost a barrier to entry.
Hugo Vance: Well, yes. That's the institutional logic completed. You see, the power law is legible to a board, legible to a government funder, legible to a Microsoft. A novel architecture is not. So the economic gravity pulls everything toward the labs that can run the verification experiments at scale — OpenAI, DeepMind, Google, Meta. And the interesting, uncomfortable implication is that this crowding happened precisely because the prediction worked. A reliable forecast is also, necessarily, a moat.
Lila Soto: And then Chinchilla lands and says — the recipe you've all been scaling so confidently? The parameter-to-data ratio was off the whole time. Which is the part I can't quite get my head around, because everyone just... kept going.
Hugo Vance: DeepMind's finding was precise — not vague. For a fixed compute budget, you get better loss by running a smaller model on more data than a bigger model on less. Equal scaling, roughly. Parameters and data in proportion. Kaplan's 2020 recipes had tilted heavily toward parameters. The early large models were, in the technical sense, undertrained on data relative to their size.
Lila Soto: Wait — so the biggest, most expensive models from that era were just... inefficient?
Hugo Vance: Systematically, yes. And here's what's interesting — picture a DeepMind researcher in, I don't know, late 2021, sitting with the Chinchilla results before publication. She's looking at the numbers and realizing that a model trained on twice the data but half the parameters would have outperformed the flagship models everyone had staked their reputations on. That's not a small wrinkle. That's a structural error baked into years of roadmaps.
Lila Soto: And yet — didn't it kind of just get absorbed? Like, everyone updated the recipe and kept going?
Hugo Vance: That's exactly the thing. The power law still held. DeepMind didn't break the framework — they corrected the ratios within it. Google, Meta, Microsoft, all of them shifted their scaling roadmaps around the new compute-optimal recipe. The faith in scaling as the lever didn't waver. If anything it deepened, because now it had survived a correction.
Lila Soto: Which is — I mean, that's kind of philosophically strange, right? The recipe was wrong and that made people trust the framework more?
Hugo Vance: Well, yes. A framework that can absorb its own correction is very durable. The question of whether that durability reflects robustness or just structural lock-in — that's something we'll have to come back to, because the answer has real stakes.
Lila Soto: Okay — but I want to introduce something that actually surprised me when I found it. Sorscher et al., NeurIPS 2022. They showed that if you prune the training data — actually remove the low-quality or redundant examples — you can beat the power law entirely. Not just move along the curve. Break it. Exponential improvement instead of polynomial.
Hugo Vance: Exponential. That's the word that should give everyone pause.
Lila Soto: Because if Sorscher et al. are right, the standard power law isn't a ceiling — it's an artifact. We've been training on unfiltered data, the noise drags the curve down, and the power law is what you get when you haven't bothered to clean your inputs. That feels like — I don't know, it's like we spent a decade optimizing how fast we could read a library and nobody checked whether half the books were gibberish.
Hugo Vance: That's a fair analogy. And the implication is uncomfortable — the dominant decade-long bet on raw scale over data quality may have been systematically suboptimal. Not wrong enough to stop working, but not the efficient path.
Lila Soto: So then — how do you even know if you've broken the power law or just moved to a different part of the same curve?
Hugo Vance: You see, that is precisely the open problem. And it connects to something we don't yet have — a mechanistic explanation for why the power law appears in the first place. Without that, any deviation could be a regime change or a measurement artifact. We're forecasting from a pattern we cannot yet explain.
Lila Soto: And that's — I mean, that's the part that makes me uneasy. We can't explain it, but we're betting on it anyway. What does that actually look like from the inside?
Hugo Vance: The theoretical attempts exist. Statistical mechanics approaches, percolation theory — there's work suggesting two criticality regimes, one for discrete subtasks and one for a dominant data manifold. But none of that has achieved consensus. The field is still forecasting from the pattern alone.
Lila Soto: Wait — no consensus? After five years of trillion-dollar bets?
Hugo Vance: None. And here's why that matters practically — the scaling laws were measured on upstream pre-training loss. Not downstream task performance. Not what the model actually does after fine-tuning on a real application. That transfer is murkier. So when you extrapolate into, say, multimodal learning, or test-time compute scaling, or a trillion-parameter model — you're extending a regularity observed in one constrained setting into territory it has never been tested in. That is an empirical bet. Not a principled prediction.
Lila Soto: Hm. So the downstream-upstream gap is — that's not a minor footnote, that's the whole question of whether any of this translates to capability people actually care about.
Hugo Vance: Precisely. And yet — funding flows anyway. Because scaling produces results, results attract capital, capital funds more scaling. OpenAI, DeepMind, Google, Meta, Microsoft. They've run this cycle long enough that periodic claims of diminishing returns — and there have been several — have not displaced the framework even once.
Lila Soto: Which is the thing I was building toward — isn't that circular? We've structured the research ecosystem around the problems scaling laws predict well, which then proves scaling laws work. Is that — I mean, is that just how progress always happens, or is it something more uncomfortable?
Hugo Vance: I'd put it this way. Imagine a researcher at a mid-tier lab in, I don't know, early 2023. She's been running experiments suggesting that architecture changes at small scale outperform scaling predictions. Genuinely interesting result. But to publish it credibly, she needs large-scale verification — which requires compute she cannot access without a grant — and every grant committee is staffed by people whose careers rest on the scaling paradigm being correct. The circularity isn't conspiratorial. It's just the structure of where resources live.
Lila Soto: Oh. So the question isn't whether she's right — it's whether she can afford to find out.
Hugo Vance: Yes. And what's genuinely uncertain is whether scaling's persistence reflects robustness — the power law really is that durable — or the structural fact that within a system where OpenAI and DeepMind and Google hold the compute, scaling is the only strategy that reliably gets resourced. We cannot easily distinguish those two from outside.
Lila Soto: Which means even Sorscher et al.'s data pruning result — the exponential improvement finding from NeurIPS 2022 — kind of can't get a fair hearing, because testing it at scale requires the very infrastructure controlled by labs whose roadmaps assume power-law scaling.
Hugo Vance: That is the bind, yes. A revised power law — like Chinchilla — gets absorbed because it keeps scaling as the lever. A result that breaks the power law entirely requires betting against the paradigm that funds you. That's a different ask.
Lila Soto: So what would it actually take for a lab to credibly challenge this — like, what does that even look like?
Hugo Vance: Well, the honest answer is: a capital-intensive failure. Not a better paper. If a lab spends, say, ten billion dollars scaling to a trillion parameters and the loss curve flattens — genuinely flattens, not just slows — that's the first thing with enough force to make a board ask a different question. Not Sorscher et al., not a NeurIPS result. A write-down.
Lila Soto: That's kind of a grim condition for scientific revision.
Hugo Vance: It is. And yet — I keep thinking about what actually landed for me in all of this. Kaplan et al. found something real. Hestness before him found something real. The power law holds inside the regime where we've checked it. The part that unsettles me isn't the finding, it's the gap between upstream pre-training loss and what we actually care about downstream. That gap hasn't closed. We just stopped staring at it.
Lila Soto: Yeah — I think that's where I landed too, actually. It's not that the discovery was wrong. Hestness, Kaplan, the three axes, the predictive curve — that's real. It's that what they measured was a description of how the field had already chosen to build and test things. Pre-training loss on unfiltered data. And then that description got treated like gravity.
Hugo Vance: Mm. Not wrong. Just — not as universal as gravity.
Lila Soto: Which means the real question isn't whether scaling laws are true. They are, inside the box where we've looked. It's whether we'll ever fund the conditions to find out what's outside the box.
Hugo Vance: And that — I think that's actually the right place to stop.
Lila Soto: Good conversation. Thank you for being patient with all my wrong guesses.
