Cole Bryant: Jonathan, hey — did you eat, are you caffeinated, because I need you firing on all cylinders for this one.
Jonathan Ingles: I'm fine. What do you have.
Cole Bryant: Okay I'm going to hand you a number and I just want your gut reaction — no thinking, just react. GPT-3. Three-digit addition. Coin flip.
Jonathan Ingles: Meaning it was basically guessing.
Cole Bryant: Basically guessing. And then — nobody added arithmetic to the training, nobody built a chain of math examples, nothing changed in what the model was taught — it just got bigger. And then it could do it. Like, reliably.
Jonathan Ingles: Wait — nobody trained it on arithmetic chains specifically?
Cole Bryant: Nobody. That's the whole — that's the thing that broke my brain when I first landed on this. It's like, imagine a kid who cannot tie their shoes. Cannot. And then one morning, no practice overnight, no YouTube tutorial, they just tie a perfect bow. Something reorganized inside that you didn't touch. That's what happened. And it's not one weird case — Jason Wei and his collaborators surveyed this and found over a hundred of these abilities. Across GPT-3, PaLM, Chinchilla — arithmetic, logical deduction, rudimentary theory of mind, code generation — none of it explicitly trained for.
Jonathan Ingles: Over a hundred.
Cole Bryant: Over a hundred documented. And the definition Wei nailed down is — an ability is emergent if it's absent, basically near-random, in smaller models, and then present in larger ones, and you couldn't have predicted it just by watching the small model's trend line. You couldn't extrapolate your way there.
Jonathan Ingles: So the question is whether that jump is a genuine leap in capability, or whether researchers just found the metric that makes it look like a cliff.
Cole Bryant: That's — yeah. That's exactly where this gets uncomfortable.
Jonathan Ingles: Because those are very different problems.
Cole Bryant: Because one of them means the model actually crossed a threshold — like, water hit zero degrees and froze — and the other means we just had a bad thermometer.
Jonathan Ingles: The phase transition framing is the precise one. Performance sits near chance — random, basically useless — and then at some scale threshold it jumps. Not gradually. Jumps. That's the claim. And it holds across GPT-3, PaLM, LaMDA — same shape of curve, different capabilities.
Cole Bryant: Wait — LaMDA too? Like the same cliff shape on few-shot prompting?
Jonathan Ingles: Few-shot prompting, arithmetic — yes. And what makes it wild is what few-shot actually means mechanically. No weight updates. No fine-tuning. You put a handful of examples in the prompt at inference time and the model just... figures out the task. Nothing in the network changed. No gradient touched anything.
Cole Bryant: Bro — okay so that means the adaptation is happening entirely inside the forward pass. Like, the model reads the examples and reconfigures something in real time without — wait, that's not retraining, that's just... thinking?
Jonathan Ingles: That's in-context learning. And it only shows up above certain scale thresholds. Below them — random. Chain-of-thought prompting is the same story. You prompt the model to show its intermediate steps, and suddenly it can solve problems it couldn't touch before. Also classified emergent. Also only above a scale threshold.
Cole Bryant: So the prompting technique itself is emergent. Not just the answer — the whole method of getting there.
Jonathan Ingles: Correct. Now — the structural piece is this: Researchers didn't just observe the jump from outside. They went inside. And they found what they're calling symbol abstraction heads in the early layers of these networks.
Cole Bryant: Hold on — inside the actual weights?
Jonathan Ingles: Early layers. Specific attention heads that take input tokens and convert them into abstract relational variables. Not pattern-matching theater. Actual structural reorganization. The model isn't just doing more of the same thing at scale — something inside it is reconfiguring to operate at a different level of abstraction.
Cole Bryant: Okay that — dude, that changes the picture for me. Because if you can actually see the mechanism forming inside the network — like you can point to the layer, point to the heads — then why can't you predict which ability is next? That gap is insane.
Jonathan Ingles: That's the exact right question. Scaling laws tell you aggregate performance improves. They do not tell you symbol abstraction heads will organize at parameter count X and unlock reasoning capability Y. Those are different predictions entirely.
Cole Bryant: So we can see the structure after it forms, but we're still blind to it before. Which means every new model generation is basically — you ship it and then find out what it learned to do.
Jonathan Ingles: But that 'ship it and find out' framing is exactly what the Mirage critique blows up. There's a researcher — picture her Friday afternoon, evaluation script open, about to write the abstract for her paper — she's got this beautiful cliff on her accuracy graph. Exact-match on arithmetic outputs. Looks like a phase transition. Clean jump. Then she swaps the metric to per-token log-probability, and the cliff just... flattens. Same model. Same weights. Gradual slope the whole way up.
Cole Bryant: Wait — same model, nothing changed, just a different evaluation script?
Jonathan Ingles: Same model. That's the Mirage critique. Exact-match accuracy is coarse — you're either right or wrong, no partial credit. So tiny improvements in the model register as zero until suddenly they're enough to tip the answer, and then it looks like a cliff. Per-token log-probability is continuous. It catches the gradient the whole time.
Cole Bryant: So she has to decide — did the model change, or did her ruler?
Jonathan Ingles: That's the actual question she has to answer before she hits submit.
Cole Bryant: Okay but — wait, no — from a builder's angle, I'm not sure the distinction matters? Like, whether it's a real phase transition or a measurement artifact, the jump still shows up in production. You still can't predict it before you scale. The model still surprises you on a Tuesday when it starts doing something it couldn't do at half the parameter count. The artifact is... also real.
Jonathan Ingles: It matters enormously for governance. Georgetown CSET published an explainer on exactly this. Their point: if capabilities appear suddenly, pre-deployment evaluation regimes are systematically inadequate — you test the small model, it can't do X, you assume the big model can't, and you're wrong. But if the improvement is actually gradual, evaluation could work. You could catch it climbing.
Cole Bryant: Oh. That's — okay that's the actual stakes. That's not a methodology argument, that's a safety argument.
Jonathan Ingles: The abruptness framing is what makes pre-deployment review look hopeless. Gradual framing is what makes it look tractable. Same empirical fact, opposite policy conclusion.
Cole Bryant: And — I mean, this is the thing that won't let me go — both framings are supported by actual empirical work right now. Wei et al. documented over a hundred of these jumps across GPT-3, PaLM, Chinchilla, the curves look like cliffs. And the Mirage line of research says switch the metric and the cliff disappears. Neither side has knocked the other out.
Jonathan Ingles: Unresolved. Genuinely. And that's not a comfortable place to be when you're deciding whether to deploy something.
Cole Bryant: No it is not. And honestly — the scaling law piece makes it worse, because scaling laws give you aggregate improvement curves but zero precision on which specific ability lands at which parameter count. There's a whole separate axis coming with test-time scaling that changes the math again, and the policy question of how you even regulate something that unpredictable — it's genuinely uncomfortable.
Jonathan Ingles: The ruler problem doesn't go away when you add more compute at inference. It compounds.
Cole Bryant: The ruler problem compounds — okay but that's the thing I actually can't get past. Because scaling laws, like, they give you the envelope. They say bigger model, more data, more compute, performance goes up. That's — the math checks out. But they literally cannot tell you that reasoning ability Y appears at parameter count X. That's not what they're measuring.
Jonathan Ingles: There's a refinement. Distributional scaling laws research tightened the predictions — better than simple extrapolation. Still can't close the gap. Still cannot name the specific ability or the specific crossing point.
Cole Bryant: So it improved the envelope but not the address inside the envelope.
Jonathan Ingles: Exactly. And the address is what matters if you're deploying. Now — the data side is even more unsettling. Skywork-Math model series. Researchers used it to show that data scaling laws for mathematical reasoning are far from saturated. More data keeps producing meaningful gains. Which means the landscape of emergent abilities we haven't seen yet is still expanding.
Cole Bryant: Wait — still expanding? Like, we're not close to seeing everything?
Jonathan Ingles: Not even close. And then there's test-time scaling. You don't retrain. You don't add parameters. You throw more compute at inference — at the moment the model is actually answering — and emergent reasoning shows up in specialized domains. Medicine specifically. Capabilities that looked absent, latent the whole time, waiting.
Cole Bryant: Hold on — medicine? Like clinical reasoning?
Jonathan Ingles: Medical reasoning, yes. Same model, more inference compute, better outputs. It was always there.
Cole Bryant: Dude that — okay that breaks the whole framing for me. Because I was thinking about emergent abilities as a training problem. You train bigger, surprise appears. But if the capability is latent and you can unlock it at inference just by running the model longer, then — wait, that's not even a scaling story anymore. That's a deployment story. You ship something, you don't know what it can do, and then someone figures out the right inference budget and suddenly it can diagnose a patient.
Jonathan Ingles: That's the governance gap in one sentence. Organizations scaling these models are running experiments whose outputs they cannot fully anticipate. And the policy fork is narrow. Georgetown CSET's framing: restrict scaling — probably infeasible given competitive dynamics, no one lab stops unilaterally — or accept that the most powerful deployed systems will keep surprising their builders and invest hard in post-deployment monitoring and mechanistic interpretability.
Cole Bryant: What would interpretability actually need to look like to close this? Like specifically — not 'we need better tools,' what does the tool do?
Jonathan Ingles: It would need to predict which specific emergent ability appears at which threshold, before deployment. Not after. Those tools do not exist. Symbol abstraction heads were identified post-hoc — after the capability was already there. We can reverse-engineer the structure. We cannot forecast it.
Cole Bryant: So the honest answer is — we're beta-testing these systems in the real world and calling it iterative capability discovery. And the gap between 'we can see the structure after it forms' and 'we can predict the structure before it forms' is where all the actual risk lives.
Jonathan Ingles: The gap you just named — that's where it actually sits. We can reverse-engineer the symbol abstraction heads after they form. We cannot forecast them. And I don't think that's a temporary research gap. That unpredictability is structurally embedded in how these networks learn. You're not going to close it with one more paper.
Cole Bryant: Wait — structurally embedded? Like, not just 'we haven't figured it out yet' but 'the way neural networks learn makes this inherently unforecastable'?
Jonathan Ingles: That's the honest read. Better metrics help — continuous evaluation over exact-match, distributional scaling law refinements — and mechanistic interpretability is real progress. But the specific question of which emergent ability appears at which threshold, before deployment? Nobody has that tool. And frankly, the architecture may not permit it.
Cole Bryant: So the race — I mean, that's actually what this is, right? It's not 'will we figure out emergence' it's 'can interpretability tools catch up fast enough that we stop being surprised by our own systems before something consequential surprises us.' And the competitive dynamics mean nobody's pausing to wait for those tools. Labs keep scaling. The surprises keep coming.
Jonathan Ingles: That's the one thing I actually buy cleanly out of all of this. The question isn't whether to scale — that ship has sailed. The question is whether measurement and interpretability can move fast enough to matter.
Cole Bryant: Yeah. That's — honestly that's where I landed too. Quietly, not triumphantly. Just... that's the race.
Jonathan Ingles: Good conversation. Genuinely.
