Eleanor Crane: I want to start with a confession — I thought I understood what a scaling law was until I actually read the 2020 paper. And then I understood it less.
Ben Okonkwo: Oh, interesting — less how?
Eleanor Crane: Because the word 'law' implies derivation, theory, mechanism. What Jared Kaplan and his colleagues at OpenAI actually did was — they ran experiments. Systematic experiments. And they found that cross-entropy loss falls on a straight line on log-log axes as you increase model parameters, training tokens, or compute. Seven or more orders of magnitude, the line holds. They fitted an exponent — α ≈ 0.076 — and it looked clean. It looked like physics.
Ben Okonkwo: But it wasn't derived from physics. There was no theory predicting that exponent.
Eleanor Crane: None. Which means what we're actually looking at is a pattern that someone discovered — not proved — and then the field built on top of it as though it had been proved.
Ben Okonkwo: And the practical consequence of that is — a tenfold increase in parameters reduces loss by only about sixteen percent. Small, stable, predictable. Small enough that you need to run enormous models to move the needle meaningfully. Which is exactly the infrastructure argument that followed.
Eleanor Crane: And now — Dario Amodei was on that 2020 paper. He was at OpenAI, he was a co-author with Kaplan, and he later co-founded Anthropic. And Anthropic's Claude exists because that empirical straight line on a log-log plot became a credible economic argument for spending more. That's the thing I want to sit with today — not just what the math says, but what the math was made to carry.
Ben Okonkwo: Right — and whether it could bear that weight. Because 2022 is where that gets complicated.
Eleanor Crane: Which is where we're headed. But first I want to just — sit for a second in how strange the original finding actually is. That the loss of a neural language model obeys a power law across seven orders of magnitude, and nobody knows why.
Ben Okonkwo: The strange thing is actually a useful thing, if you let it be. Because the 'nobody knows why' part is what makes it an empirical discovery rather than a theorem. And — okay, let me try the analogy that's been sitting in my head. Imagine a chef. She notices that every time she doubles her kitchen staff, her error rate on orders drops by roughly the same fraction. Not a fixed number of errors — the same proportion. Whether she has four cooks or four hundred. She doesn't know why. She just knows it holds.
Eleanor Crane: And that's essentially Kaplan.
Ben Okonkwo: That is essentially Kaplan. Double the parameters, loss shrinks by a predictable fraction. Every time. That's the core of it — one sentence. And what makes it a discovery rather than a guess is that the line doesn't wobble when you go from tiny models to the largest anyone had trained. Seven orders of magnitude. On log-log axes, it's straight.
Eleanor Crane: What does 'straight on log-log axes' actually mean for someone who last did logarithms in high school?
Ben Okonkwo: It means the relationship is a power law — which, in plain terms, means the gains are always proportional, not fixed. You never get to a point where doubling your resources gives you twice the absolute improvement. You always get the same fraction. It's actually — hm — it's a little humbling? The curve never bends down to zero improvement, but it never leaps either.
Eleanor Crane: And the thing being measured — it's cross-entropy loss. Which is essentially: how surprised is the model by the next word?
Ben Okonkwo: Exactly — it's the average penalty the model pays for assigning low probability to the correct next token. Lower loss, less surprise, better predictions. And scaling laws say: more resources, less surprise, always by the same proportional bite. What Kaplan's team at OpenAI gave the field was the ability to predict that bite before you built the model.
Eleanor Crane: That's the part that shifted everything. Before 2020 'make the model bigger' was — it was engineering folklore. People believed it worked, but it was belief.
Ben Okonkwo: Right — and the paper turned it into a quantified prediction. Which means you could point at a compute budget and say: here is what loss you will achieve. That's — I mean, that's genuinely different from intuition. That's infrastructure planning.
Eleanor Crane: And independent of which hardware generation you're on, which specific training run.
Ben Okonkwo: The part that complicates the chef analogy is this: the chef is observing her own kitchen. Kaplan's observation held across completely different experimental setups. That's not just pattern recognition. That's — well, it *looks* like a law. Which is, I think, exactly why people started treating it like one.
Eleanor Crane: And that gap — between 'looks like a law' and 'is a law' — is where 2022 was waiting.
Ben Okonkwo: And 2022 arrived in the form of Hoffmann et al. at Google DeepMind, who ran over 400 models — 70 million parameters up to 16 billion-plus — and found that the waiting wasn't abstract. The original allocation guidance was wrong.
Eleanor Crane: Wrong how, specifically.
Ben Okonkwo: Kaplan said: if you have a fixed compute budget, spend it on parameters. Bigger model, fewer tokens. Hoffmann said — actually, no, you need far more data relative to model size than that. The optimal ratio is completely different.
Eleanor Crane: And they proved it by building Chinchilla. Seventy billion parameters, trained on 1.4 trillion tokens. Same compute budget as Gopher — which was 280 billion parameters. A model less than a quarter the size, and it outperformed Gopher across benchmarks.
Ben Okonkwo: Which means — and this is the part that still catches me — GPT-3 was in the same boat. 175 billion parameters, undertrained. A smaller model, run longer on more data, would have done better on the same compute.
Eleanor Crane: Wait. GPT-3 — the model that, you know, convinced a lot of people this whole project was real — was leaving performance on the table the entire time.
Ben Okonkwo: That's what Chinchilla showed. The field had been systematically undertrained large models because of how Kaplan's guidance was read. And I mean — it's not that the power law was fake, it's that the question 'how do you allocate inside the budget' had a different answer than anyone thought.
Eleanor Crane: So the law held, but the prescription built on top of it didn't. And by the time Chinchilla landed in 2022, the billions were already committed — the clusters, the parameter counts, the roadmaps.
Ben Okonkwo: Right — and part of why that happened is methodology. The way you fit the curve changes what optimal allocation you infer. isoFLOP analysis — which is what Hoffmann used — holds total compute constant and varies the split between model size and tokens. You trace parabolas. You find the bottom of each parabola, and that's your optimum. Kaplan's approach was different, and it gave a different answer.
Eleanor Crane: That's not a rounding error. That's — I mean, imagine a software team shipping a product based on a benchmark study, and two years later someone reruns the benchmark with a better experimental design and the product is already in production at scale. You can't just patch that.
Ben Okonkwo: And the confidence calibration question is what gets me. How many people building on Kaplan's numbers understood they were extrapolating from a particular fitting choice, not from something more fundamental?
Eleanor Crane: The economic logic — at labs like OpenAI, Anthropic, Google DeepMind — rested on these curves being predictive. And they were predictive, within a regime. But the regime had a shape nobody had mapped yet.
Ben Okonkwo: Which brings us somewhere we haven't gone yet — and it's the part that makes all of this a little vertiginous. We still don't have a mechanistic explanation for why the power law holds at all. And if you don't know why it holds, you don't know where it breaks.
Eleanor Crane: And the economic logic that followed may be extrapolating well past what the evidence can actually support. That's the layer we need to pull next.
Ben Okonkwo: And the vertiginous part is — okay, so we've established the correction happened. But the correction itself doesn't tell you why the law holds in the first place. The best account anyone has right now is something like: larger models reduce competition over neural resources. Fewer neurons fighting over the same training signal, which means rare tasks — low-frequency things the model only sees a handful of times — actually get retained instead of overwritten.
Eleanor Crane: That's the mechanistic hypothesis. But it's not a theory.
Ben Okonkwo: It's phenomenological. It describes the shape of what we see — it doesn't actually derive the exponent. There's no equation that starts from 'neurons compete for resources' and outputs α ≈ 0.076. It's more like... a plausible story that fits the curve. Which means we don't know where the curve ends.
Eleanor Crane: And when researchers have pushed into new regimes — federated learning, different benchmark families — what happens?
Ben Okonkwo: The formulation requires modification. That's how the papers put it, and — I want to be careful here, because 'modification' is doing a lot of work. What it actually means is: you refit the exponents. The straight line on the log-log plot is still there, but it's a different line. A different slope.
Eleanor Crane: Which raises the question I can't quite shake — is that discovering a boundary of the law, or discovering that we fitted it wrong to begin with?
Ben Okonkwo: Exactly. And we genuinely cannot tell. That's not — I mean, that's not a rhetorical hedge. There's no test that separates 'the law breaks here' from 'the exponent we had was always an artifact of our fitting choices in a particular regime.'
Eleanor Crane: Hold on. That means every time someone says these laws are durable across orders of magnitude—
Ben Okonkwo: —they mean: durable within the regimes we've tested. Dense transformer architectures, English text, standard next-token prediction on centralized clusters. That's the actual domain of the claim. Synthetic data, different architectures — those are open questions. We don't have the numbers.
Eleanor Crane: And the economic logic at OpenAI, at Anthropic, at Google DeepMind — it rests on assuming the next order of magnitude looks like the last one.
Ben Okonkwo: That's the load-bearing assumption. Loss reduction will continue. Capability will follow loss. The exponents won't shift. None of that is in the data — it's an extrapolation from a phenomenological regularity that nobody can explain from first principles.
Eleanor Crane: Which is — I mean, that's not nothing. That's the intellectual foundation for hundreds of billions of dollars in infrastructure decisions.
Ben Okonkwo: And the Chinchilla correction actually makes it worse, in a way — not better. Because it shows the fitting choices matter enormously for what you infer. Kaplan's team fitted one way and got one prescription. Hoffmann's team used isoFLOP parabolas and got a completely different one. If we still don't know why the law holds, how confident should anyone be that the next refitting won't shift the prescription again?
Eleanor Crane: So the question isn't just whether the curve continues. It's whether we're even reading the curve correctly right now.
Ben Okonkwo: And yet the decisions keep being made. Anthropic is training Claude. Google DeepMind is scaling. The line keeps being drawn forward, one order of magnitude at a time, on the strength of — what, exactly? The confidence that loss reduction will continue, that capability follows loss, that the exponents won't shift. None of that is in the data. It's the leap the data cannot make for you.
Eleanor Crane: That's the thing I can't settle. The Chinchilla correction didn't break the power law — the loss still falls on a straight line. What it broke was the confidence that any one team's reading of that line is the right one. Kaplan read it one way, Hoffmann read it another, and billions were committed between those two readings. And Dario Amodei was on the first paper. He co-founded Anthropic. Claude is, in some real sense, the economic logic of a log-log straight line made into a product.
Ben Okonkwo: That's genuinely unresolved, I think. Not unresolved as in we need more time — unresolved as in the structure of the problem doesn't have a solution yet. We can't separate 'the law is durable' from 'we keep refitting it to new regimes and calling that durability.'
Eleanor Crane: Maybe the most honest thing you can say about scaling laws is that they're the most durable empirical pattern we have for this — and that's exactly why it matters that nobody knows why they're durable. The not-knowing isn't a footnote. It's load-bearing.
Ben Okonkwo: Yeah — and I think I'll be sitting with that for a while.
Eleanor Crane: Same. Thank you for thinking through it with me.
