Marcus Kline: Long week — but I've had a number stuck in my head, and I want to hand it to you cold.
Ben Okonkwo: Sure, hit me.
Marcus Kline: Seven orders of magnitude.
Ben Okonkwo: That's — okay, in what context? Because that number means very different things depending on what you're measuring.
Marcus Kline: Model loss. Compute. Data. Model size. In January 2020, Jared Kaplan and colleagues — OpenAI, Johns Hopkins — publish what becomes the foundational scaling laws paper, and they show the relationship between those variables holds across seven orders of magnitude. And on a log-log plot it's a straight line. Clean. Almost unnervingly clean.
Ben Okonkwo: Right — and the reason that matters is that power-law relationships at that scale are rare. Most empirical regularities in machine learning fall apart when you push them two, three orders of magnitude. Seven is... yeah, that's a different category of finding.
Marcus Kline: And then consider: two years later, Jordan Hoffmann and DeepMind train over four hundred language models, look at the same relationship, and conclude that every large model built using Kaplan's prescription — including DeepMind's own Gopher, at 280 billion parameters — was significantly undertrained. Not because the law was false. Because the field drew the wrong instruction from a law that was true.
Ben Okonkwo: Both teams right about the law, opposite conclusions about what to do — and the gap between those conclusions is billions of dollars of compute spent on the wrong side of it. That's... that's the episode.
Marcus Kline: And before we get to how the field got it backwards — what actually is the thing they were using? Because I think most people hear 'scaling law' and assume it's more theory than it is.
Ben Okonkwo: Right — okay, so here's the plainest version I know. Think of a factory learning curve. Every time you double the size of your operation, you get better at making the thing — but each doubling gains you less than the last one did, and that ratio is fixed. Predictable. That's it. That's the core of it.
Marcus Kline: The ratio is fixed.
Ben Okonkwo: Fixed. And that's what Kaplan's team actually measured — they had four quantities. N, the number of model parameters. D, the training data in tokens. C, the total compute budget in FLOPs. And L, cross-entropy loss on next-token prediction. And when you plot any one of those against L on a log-log scale, you get... a straight line. The exponent on that power law is less than one, which is just math for: every doubling helps, but each doubling helps a little less than the one before it.
Marcus Kline: Diminishing returns, but on a schedule.
Ben Okonkwo: Exactly on a schedule — and here's the part that actually surprised me when I first sat with it. The architectural details don't move the line much. Network width, network depth — within a wide range, those choices don't really change where you land. The law holds across different transformer designs. Which means it's not a quirk of one engineering decision, it's something more... I don't know, structural? It's almost uncomfortable how clean it is.
Marcus Kline: That's the thing that reads like a genuine law rather than an observation.
Ben Okonkwo: And it is, empirically. Seven orders of magnitude, architecture-independent — that's not noise. But the part nobody names clearly enough: L is loss on next-token prediction. It is not — and the Kaplan paper is careful here, but the field was not always — it is not a direct measure of what the model actually does in the world. The moment you ask whether lower L predicts better reasoning, or better benchmarks, that's a separate question the law doesn't answer.
Marcus Kline: So the line is real. The question is what the line is measuring — and whether that's the thing anyone actually cared about.
Ben Okonkwo: And that gap — between what the line measures and what people built on top of it — is exactly where the Kaplan-Chinchilla split gets strange. Because Kaplan's prescription wasn't wrong about the law. It was wrong about which side of the law you should optimize against. Larger models are more sample-efficient, that part is true — but sample efficiency under a fixed compute budget doesn't mean you should starve the data side. And nobody named that clearly enough.
Marcus Kline: Put a person in the room. Early 2022, you're a head of infrastructure at a frontier lab. You have the Kaplan paper. You've built your planning assumptions around it. Then April 2022 — Chinchilla drops.
Ben Okonkwo: And the paper names your model. By name. Gopher — DeepMind's own 280-billion parameter model — is the exhibit A for undertrained. That's... that's not a gentle peer review note.
Marcus Kline: What does undertrained actually mean there? Because I want to make sure we're precise.
Ben Okonkwo: Right — so Hoffmann's team, training over 400 models, found that for a fixed compute budget you should scale model size and training tokens roughly equally. One-to-one. Kaplan had said push model size faster than data — bigger model, modest data, stop before convergence. Chinchilla said: no, that model needed a much larger data budget to actually use its capacity. The parameters were there. The tokens weren't.
Marcus Kline: So the model was — I mean, it was the right size for a training run nobody gave it.
Ben Okonkwo: Exactly. And the reconciliation work — Pearce and Song at Microsoft Research and MIT, Porian and colleagues — they actually traced why the coefficients didn't match. Non-embedding versus total parameter counting. Warmup duration. Optimizer tuning. These aren't philosophical differences, they're experimental design choices that move where the line sits. The law didn't break. The measurement did.
Marcus Kline: The law didn't break. And that's the part that should probably unsettle us more than if it had. Because Dario Amodei co-authored the Kaplan paper and walked that framework into Anthropic. The prescription traveled.
Ben Okonkwo: And the part that comes later in this conversation is going to make that worse — because we haven't even touched what loss actually predicts about what these models do in the real world. That's a different problem entirely.
Marcus Kline: And the prescription traveled, but so did the assumption underneath it. That loss is the thing. That if loss goes down, capability goes up. And nobody in the room was asking whether that was actually true.
Ben Okonkwo: Right — and the mechanism for why it wouldn't be true is... interesting, actually. Cross-entropy loss on next-token prediction is smooth and continuous. Benchmark performance on specific tasks is not. It's often discontinuous — a model jumps from failing a task to passing it. So the loss curve is a clean line and the capability curve is... lumpy. PaLM at 540 billion parameters, Google's model — it hits certain benchmarks well above what the loss trajectory would suggest, and falls short on others. The law never promised that.
Marcus Kline: Wait — falls short on which side?
Ben Okonkwo: Both sides. That's what's strange. Loss goes down, some benchmarks improve faster than expected, some slower. The law predicted neither. You've got a 540-billion parameter model built in line with scaling law predictions — and the benchmark surprises are essentially noise the law has no explanation for.
Marcus Kline: So compute budget gets allocated — billions in FLOPs — using a loss prediction. And then deployed against a capability expectation the math was never doing.
Ben Okonkwo: That's the load-bearing gap. And the part that — I mean, it should create more institutional caution than it does — is that nobody knows why the power law holds at all. There's no theoretical derivation. Kepler had the orbital law before Newton had gravity, sure, but Newton arrived. Here we're extrapolating well past the original seven orders of magnitude that Kaplan's team actually measured, and the mechanism still hasn't shown up.
Marcus Kline: And in the Kepler case, the mechanism eventually constrained where the law could apply. Without it — you can't predict from first principles where it breaks.
Ben Okonkwo: No. You genuinely cannot. Which means the conditions under which it fails are invisible until they're not.
Marcus Kline: And that's — that's what this whole structure rests on. A straight line nobody can explain, extended past the data that drew it, used to plan systems that touch how millions of people use language. The unease isn't performative. It's just the math, stated plainly.
Ben Okonkwo: The part I can't resolve — and I've tried — is that the diminishing returns are right there in the exponent. Mathematically visible. And the industry looked at that and kept doubling anyway. That's not ignoring the law. That's a bet that the economic upside clears the shrinking marginal return. But nobody can say from the math where that bet goes bad.
Marcus Kline: And the law can't tell you. That's the actual edge of it. Kaplan's team measured seven orders of magnitude — that felt like ground. But frontier labs are training past that boundary now, and the power law is just... continuing, because it hasn't stopped yet. Which is not the same thing as knowing it won't.
Ben Okonkwo: No. It's genuinely not.
Marcus Kline: I keep thinking about Kepler again. He trusted the line. The line held. And he had no idea why — and it turned out that was fine, eventually. Maybe this is fine eventually. I don't know that it is.
Ben Okonkwo: Neither do I. Good talk.
