Cole Brennan: Long week, man — but I found the thing that made it worth it, so I'm ready.
Malcolm Reeves: The thing. Let's hear it.
Cole Brennan: GPT-4's technical report — and I know that sounds dry, bear with me — it says OpenAI could accurately predict GPT-4's performance using models trained on one one-thousandth of its compute. One thousandth. And I had to put the document down because — I mean, that's not an engineering claim, that's almost a physics claim. You're saying intelligence scales like a law of nature.
Malcolm Reeves: Is that right — you went straight to the technical report?
Cole Brennan: I did, and here's what got me — because the scaling laws paper, Kaplan and McCandlish in January 2020, that paper analyzed 200-plus transformer models across seven orders of magnitude. And the relationship between compute and model quality was, and I'm quoting, 'almost eerily precise.' But OpenAI had already committed to this trajectory — GPT-1, GPT-2, progressively larger transformers on progressively more data — before that paper even existed.
Malcolm Reeves: Now, that sequencing is everything. The conviction preceded the proof. And once the proof arrived, the whole field reorganized around it — not around architectural innovation, around the Transformer as a vessel you just keep filling.
Cole Brennan: Which — the Transformer wasn't even OpenAI's, right? That's 2017, that's Google.
Malcolm Reeves: Correct. And the scaling hypothesis holds that you don't need to redesign it. More parameters, more data, more compute — the architecture stays. That's the bet. And today we're going to trace exactly how a bet that bold either becomes genius or becomes the most expensive assumption in the history of technology.
Cole Brennan: But what does that bet actually mean in practice — like, what's the mechanism? Because 'more compute, better model' sounds like something you'd put on a motivational poster.
Malcolm Reeves: Think of it like baking. You double the recipe every time, and the bread keeps getting better at the same rate. No saturation. No ceiling you can see. And at some loaf size, it starts doing things a small loaf structurally cannot do. That's the scaling law — doubling parameters or data or compute cuts the model's prediction error by a predictable amount, every time, smooth as a curve on paper.
Cole Brennan: Wait — every time? Like, no diminishing returns?
Malcolm Reeves: Not within the ranges Kaplan and McCandlish actually measured. 200-plus transformer models, seven orders of magnitude of compute — and the loss curve never bent away from the law. That's what 'eerily precise' means. GPT-2 at 1.5 billion parameters was the first place you could actually see it — each doubling of parameters dropped perplexity measurably, and there was no sign it was stopping.
Cole Brennan: And GPT-2 is the one they called too dangerous to release, right? Which — I mean, some people would say that was overstated.
Malcolm Reeves: Now, the safety framing around GPT-2 is its own conversation. But what matters for this arc is what GPT-2 actually demonstrated technically — zero-shot prompting. Give it a natural language instruction, no examples, no retraining, and it performs the task. That's not what narrow models do. That's the first empirical sign that scaling generalizes.
Cole Brennan: And then GPT-3 is where it — wait, actually, this is the part I don't fully get. GPT-3 hits 175 billion parameters and suddenly it can do few-shot learning, in-context learning, code generation. But those weren't just better versions of what GPT-2 did. They weren't there at all before.
Malcolm Reeves: That's the phase transition. You give GPT-3 three examples in a prompt — no weight updates, no retraining — and it completes the pattern. In-context learning. It was absent in smaller models entirely. The power law describes the error rate falling smoothly, but it doesn't explain why a capability just appears at a threshold. The formula and the emergence are two different phenomena.
Cole Brennan: So the law tells you the bread gets better, but it doesn't tell you when the bread suddenly learns to toast itself.
Malcolm Reeves: And that toasting itself — that's the thing nobody had a formula for. Now, think about what that actually means on the ground. A contract lawyer, eleven at night, deadline in the morning, pastes a dense indemnification clause into GPT-3 and types 'what does this mean for my client?' And the model answers — coherently, specifically. GPT-3 was never trained on legal Q&A. That capability wasn't extrapolated from GPT-2. It simply arrived.
Cole Brennan: That's the part that gets me. Nobody built that. Nobody said 'step seven: teach it contract law.'
Malcolm Reeves: And yet GPT-4's technical report says they predicted its performance from one one-thousandth of the compute. So which is it? Did they predict the contract lawyer moment, or did they predict the loss curve and then discover that the loss curve had — quietly — been hiding capabilities the whole time?
Cole Brennan: Wait — because Sébastien Bubeck and the 'Sparks of AGI' team described GPT-4 as exhibiting surprisingly general intelligence across novel tasks. Surprisingly. That word is doing a lot of work if you've already predicted the thing.
Malcolm Reeves: It is. And I think — I mean, the honest answer is those are two different claims that people keep treating as one. The loss curve is predictable. The capabilities that live inside a low-loss model? That's a separate question. You can forecast the envelope. You cannot forecast what's inside it.
Cole Brennan: So the formula works, but it's predicting the wrong thing.
Malcolm Reeves: Precisely. The scaling law tells you how much the model improves at predicting the next token. It does not tell you that somewhere above 100 billion parameters, a model will generalize to legal reasoning it was never shown. Those emerged. The law described the road; it didn't describe what city you'd find at the end of it.
Cole Brennan: And — I mean, it gets uncomfortable for the whole framework, right? Because if the emergent behaviors only appear past certain scale thresholds, then you need the capital to reach those thresholds before you even know what you're getting. And there's a whole separate question coming about whether dense scaling is even the only road there — Mixture-of-Experts architectures are starting to challenge that assumption pretty hard.
Malcolm Reeves: But that's the next problem. The first problem is that the surprise and the predictability are both true simultaneously — and nobody has fully resolved that.
Cole Brennan: And that unresolved tension is exactly what makes the moat question so uncomfortable — because if emergent capabilities only show up past the threshold, then reaching the threshold *is* the product. You can't sample it. You can't prototype it. You just have to buy enough compute and get there.
Malcolm Reeves: And that's the structural consequence. Only organizations that can accumulate capital for compute, data, and parameters at that scale even get to see what lives above the threshold. Everyone else is reasoning about a country they've never entered.
Cole Brennan: Is that moat permanent, though?
Malcolm Reeves: Now, that's where Mixture-of-Experts architectures become genuinely interesting. Dense transformers — every parameter fires on every token. MoE models activate only a subset of parameters per token. So you can scale the model's *capacity* without scaling the compute proportionally. Those are decoupled.
Cole Brennan: Wait — so a sparse model could be, what, ten times larger by parameter count but not ten times more expensive to run?
Malcolm Reeves: That's the claim. And recent research — FlexMoE, DSMoE, cMoLLM, all 2026 papers — shows sparsely activated architectures matching dense transformers under the same compute budget. A research team at a lab that cannot afford OpenAI's training runs might still reach competitive performance if the architecture is efficient enough.
Cole Brennan: That's — I mean, that reframes the whole thing. Because if compute efficiency matters as much as compute volume, then OpenAI's capital accumulation isn't a permanent wall. It's — it's more like a head start on a track that might be getting shorter.
Malcolm Reeves: It shifts the moat from raw volume to efficiency. And that's an open question — genuinely unresolved. Larger dense models do achieve better sample efficiency, faster convergence, lower inference cost per unit of capability. That's real. But whether that advantage survives an architectural shift toward sparse scaling? Nobody can honestly answer that yet.
Cole Brennan: So the question isn't whether OpenAI's strategy *worked* — it clearly worked. The question is whether the thing that made it work is still the thing you need.
Malcolm Reeves: And the thing is — OpenAI organized its entire financing and research roadmap around the scaling hypothesis before Kaplan's paper existed. Before the math. They committed to a graph that hadn't been drawn yet. That's not strategy in the conventional sense. That's a wager.
Cole Brennan: Which brings me back to — you remember the opening, the 1/1,000th claim. I put the document down. And now I think... I mean, what they predicted was the loss curve. The contract lawyer at eleven at night? That wasn't in the forecast. The bet paid off, but I'm not sure it paid off the way they thought it would.
Malcolm Reeves: Vision or a very expensive educated guess. Might only be answerable from further out than we are.
Cole Brennan: That's honestly the most honest place to land.
Malcolm Reeves: Good conversation. Thank you for bringing the technical report.
