Max Rivera: Clara, hey — I have to tell you, I went down a rabbit hole this week that I could not get out of, and I think it's going to make this conversation a little dangerous.
Clara Bennett: Dangerous how?
Max Rivera: Like, I now have opinions. Which is — I know, I know, that's your job. But I was trying to explain to someone why GPT-4 hitting eighty percent on the HumanEval coding benchmark is actually a weird number, not just a big number, and I realized I didn't fully understand it myself.
Clara Bennett: What made it feel weird to you, specifically?
Max Rivera: That the smaller models were basically at zero. Not like, fifty percent, not like thirty — near zero. And then it just... isn't. There's no in-between you can point to.
Clara Bennett: Now that's the thing. That's the shape of what Jason Wei and colleagues at OpenAI formalized in 2022 — emergent abilities, defined precisely as capabilities not present in smaller models and present in larger ones, with the key word being unpredictable by extrapolation. You couldn't have looked at the small-model numbers and seen the eighty percent coming.
Max Rivera: Okay, and that's — hm, that's a really specific definition. Absent below a threshold, present above it.
Clara Bennett: Exactly. No gradual middle ground. And what makes it stranger is that the loss curves — the numbers that scaling laws track — are smooth the entire time. Predictable. The underlying model is improving on a clean curve, and then task performance just snaps.
Max Rivera: Wait — so the same model that looks boring on the loss graph is the one doing the sudden jump on the task?
Clara Bennett: Same model. And that gap between smooth loss and jagged capability is where the whole debate lives.
Max Rivera: That's — yeah, that's the itch. Is the jump real or is something about how we're measuring it creating the illusion of a cliff.
Clara Bennett: And that question turns out to have a serious scientific disagreement behind it, which is what makes this worth unpacking.
Max Rivera: That gap is what I keep snagging on — like, OK, so what actually *crosses* inside the model at that threshold? What changes?
Clara Bennett: The two clearest cases are chain-of-thought reasoning and in-context learning. Let me take them in order, because they're not the same mechanism.
Clara Bennett: Chain-of-thought is the one that makes the discontinuity undeniable. A model either spontaneously generates intermediate reasoning steps before it answers — or it doesn't. There is no partial version you can observe below the threshold. No half-step. You don't see a model doing thirty percent of the reasoning chain. It's absent, then it's there.
Max Rivera: Wait — no half-measure at all?
Clara Bennett: None that's observable. Now, in-context learning is the other case — that's where a model performs a brand new task just from a few examples in the prompt, zero weight updates, nothing retrained. And that also appears abruptly. Not gradually better. Abruptly present.
Max Rivera: Actually, I want to make this concrete. Because I'm picturing someone, like, a developer, Friday evening, testing a one-billion-parameter model on a logic puzzle — and it just fails. Every single time. No reasoning shown.
Clara Bennett: Right. Same architecture, scaled to two billion parameters — same person, same puzzle — and now it walks through the steps. Unprompted. That's not a metaphor; that's the empirical pattern. Research puts the critical threshold for reasoning tasks at approximately 1.6 billion parameters. Below it, commonsense and deductive reasoning performance is essentially flat. Above it, it jumps.
Max Rivera: 1.6 billion. That's — I mean, that's weirdly specific for something that's supposed to be unpredictable.
Clara Bennett: That specificity is retrospective — we can locate it after the fact. The problem is you can't predict *which* capability crosses at *which* threshold before you run the experiment. GPT-3 and GPT-4 at OpenAI made this undeniable for the field at scale, but they didn't tell you in advance what would flip.
Max Rivera: So the loss is still going down smoothly the whole time this is happening?
Clara Bennett: The entire time. That's the paradox that demands explanation. The number that scaling laws track — smooth, predictable, boring — and task performance just snaps at a threshold the loss curve gave you no warning about. The model's capacity to form internal representations of abstract concepts crosses some sufficiency point, and then the behavior is there. But the loss didn't blink.
Max Rivera: Which is — yeah, that's either a real phase transition in what the model *is*, or it means we're measuring the wrong thing entirely. And I know that question has a very pointed answer from at least one research group.
Clara Bennett: That pointed answer is Schaeffer et al., 2023. And their claim is actually pretty surgical — it's not 'emergence is fake,' it's 'your yardstick is creating the cliff.' Exact-match scoring is all-or-nothing. One wrong token, zero credit. So a model that's quietly improving gets scored zero until it crosses some invisible line where it gets every token right — and then it looks like a jump.
Max Rivera: Wait — so if you swap in partial-credit scoring on the same models, same weights, nothing changes —
Clara Bennett: The cliff flattens. Smooth curve. The discontinuity was in the yardstick, not the model.
Max Rivera: Okay that's — I mean, that should be devastating to the whole story. But I keep feeling like it's not. Like, chain-of-thought reasoning specifically — how do you give partial credit for that? You either have intermediate steps or you don't.
Clara Bennett: That's the strongest counter. Chain-of-thought survives the Schaeffer critique precisely because there's no partial version to score. And in-context learning holds up too. Schaeffer's re-scoring works on tasks where partial credit is coherent — arithmetic, translation, things with gradable outputs. It doesn't dissolve the capabilities that are structurally binary.
Max Rivera: So two camps, both partially right, kind of — wait, actually, there's a third thing I found and I think it's the weirdest one. Zhao et al., 2025. The bimodal seed distribution hypothesis.
Clara Bennett: Walk me through it.
Max Rivera: Imagine you're training the same model fifty times, different random seeds each run. Zhao et al.'s finding is that at a certain capacity threshold, the distribution of outcomes across those runs stops being unimodal — like, clustered around one middle value — and splits into two peaks. A low cluster and a high cluster. No middle. The model either nails it or doesn't, depending on initialization. That's not a phase transition in some clean physics sense. That's not a measurement artifact either. It's a probability distribution flipping shape.
Clara Bennett: And that matters because it means the capability was always latent in the architecture — it's just that only some random seeds find it at that scale. The threshold is real, but it's probabilistic.
Max Rivera: Which — picture a team shipping a product, say a coding assistant, and they run their eval on one trained checkpoint and it looks fine. But a different random seed, same compute budget, same data — different story. They got lucky or unlucky and they don't know which.
Clara Bennett: Now Wu and Lo, 2024, add another layer — a U-shaped pattern. Hard questions start bad, dip lower mid-scale, then recover. Easy questions do the inverse. When you mix them in a standard benchmark, the two curves cancel each other out and you see apparent stagnation. Then both recover together and the whole benchmark score jumps. No phase transition required. No measurement artifact. Just two populations of questions moving in opposite directions.
Max Rivera: No way. So you can manufacture the *appearance* of emergence just by having a mixed difficulty benchmark.
Clara Bennett: You can, yes. But here's what doesn't move regardless of which mechanism is true — the practitioner's problem. If you are a developer and your model is below the threshold, the capability is unusable. Above it, it works. Whether that cliff is in the metric, the seed distribution, or a Wu-Lo difficulty mix — you still hit a wall, and then you don't. The measurement debate doesn't dissolve that.
Max Rivera: Right — and that practical cliff gets way more complicated once you realize scale isn't just one axis, which is actually the part we should get into next — because Chinchilla changed what 'scale' even means.
Clara Bennett: And that's exactly where Chinchilla lands like a wrench in the whole framework. Hoffmann and colleagues' result isn't just 'bigger isn't always better' — it's that compute-efficient training requires scaling training tokens proportionally alongside parameters. Not just building a larger model. The ratio matters.
Max Rivera: Wait — so it's not just one axis. It's two.
Clara Bennett: Two at minimum. Google DeepMind's Chinchilla model — 70 billion parameters — outperformed models significantly larger than it that had been trained on fewer tokens. Parameter count alone does not explain or predict when emergence happens. The token-to-parameter ratio is a second variable, and before Chinchilla, most labs were effectively ignoring it.
Max Rivera: Okay that's — I mean, if the industry was optimizing on the wrong axis the whole time, then their intuitions about when a capability would show up were built on a broken map.
Clara Bennett: Exactly the consequence. Now add a third axis: model merging. You combine two fine-tuned models — different tasks, different specializations — and through nonlinear interactions between their parameters, you get capabilities that neither parent model had. Not additive. Emergent from the combination.
Max Rivera: Hold on. That's not even about pre-training scale at all.
Clara Bennett: No, it isn't. And then inference-time compute — test-time scaling — adds a fourth. You can get emergent behavior not from how the model was trained but from how much computation you allow at the moment it answers. More steps at inference, different capabilities surface. So now safety teams are tracking emergence across pre-training scale, token count, model merging, and inference budget simultaneously.
Max Rivera: So where does that actually leave a safety team? Like — practically. What do they do with four moving variables?
Clara Bennett: Anthropic's approach — and this is the institutional acknowledgment that the problem is real — is to test models across many scales and tasks rather than assuming gradual capability growth. Don't assume the next checkpoint is incrementally better. Treat each scale as potentially discontinuous.
Max Rivera: That sounds right but also — wait, that doesn't actually solve it. Because the capability doesn't exist yet at the scale you're testing.
Clara Bennett: That's the structural paradox. You cannot red-team a capability that isn't present at the scale being tested. Imagine a safety researcher — it's late, she's running evals on a model checkpoint, systematic, thorough — and the dangerous capability genuinely does not exist at that size. She clears it. Two months later, at deployment scale, it's there. Her testing wasn't negligent. The capability was absent when she looked.
Max Rivera: And no amount of more thorough testing at the smaller scale fixes that.
Clara Bennett: It's baked in. Regulatory frameworks built around cataloguing known capabilities — they offer no early warning for the class of abilities most likely to matter. The most consequential capabilities are the ones that don't exist at evaluation scale. That's not a procedural failure you can audit your way out of. It's architectural.
Max Rivera: So the thing you most need to catch is the thing that's invisible until you've already crossed the line. That's — yeah, that's not a testing problem. That's a much harder problem than that.
Clara Bennett: And that's what scaling laws actually are, in the end. They describe the road. Smooth, predictable, trackable — you know how many hours the student is putting in. You just don't know when the concept locks.
Max Rivera: That's — wait, that's the exact thing I was trying to say at the beginning. I was telling someone why eighty percent on HumanEval is weird. And I think what I actually didn't have the words for was: the loss curve didn't predict it. Jason Wei's whole definition — unpredictable by extrapolation — that's not just a caveat in the paper. That's the field's honest admission that the map runs out at the turn.
Clara Bennett: And we're not in the dark because researchers aren't trying hard enough. The systems themselves change character at thresholds we can only locate after we've crossed them. That's the structural fact.
Max Rivera: Right — but the part that doesn't fit neatly is that the most powerful systems get deployed first. Not after we've mapped what's there. During.
Clara Bennett: During. In real time. That's not alarmism — it's just the sequence.
Max Rivera: Yeah. I came in thinking the weird thing was the eighty percent number. Turns out the weird thing is that nobody had a reason to expect it. This was a good one.
