Maya Chen: Nathan, long week — but I've got a riddle that made mine weirder, and I'm handing it straight to you.
Dr. Nathan Hayes: A riddle. Okay, yes — let's hear it.
Maya Chen: Two models. Same task: three-digit addition. One has 125 million parameters — fails completely, basically random. One has 175 billion — gets it. Not better at it. Gets it. What changed?
Dr. Nathan Hayes: Scale. But — and this is the part that actually matters — not in the way scaling laws would predict. Kaplan et al. at OpenAI showed prediction loss follows a smooth power-law with parameters and compute. Smooth. So you'd expect arithmetic to improve smoothly too. It doesn't.
Maya Chen: It just — flips.
Dr. Nathan Hayes: Right. And that's the phenomenon Jason Wei and Yi Tay formalized in their 2022 paper in Transactions on Machine Learning Research. Emergent abilities — defined as capabilities that are near-random in smaller models and appear sharply above a scale threshold, in a way you could not have predicted by extrapolating from the smaller-model curve.
Maya Chen: So Wei and Tay gave a name to something people at Google Research had already been... noticing? Bumping into?
Dr. Nathan Hayes: Essentially, yes. And they went further — documented dozens of these abilities across LaMDA, GPT-3, Gopher, Chinchilla, PaLM. Multiple model families, distinct architectures. The same tasks flip on at scale across all of them.
Maya Chen: Okay, that's — I mean, that's the detail that keeps nagging at me. Because if it were just GPT-3 doing this, you could write it off. But the pattern holding across that many different architectures feels like... something.
Dr. Nathan Hayes: Or it feels like something — which is not the same as being something. And that distinction is genuinely unsettled. You have smooth, continuous loss improvement from Kaplan et al. and sharp, unpredicted capability jumps from Wei et al. Both empirically documented. Both real. And they're in tension with each other.
Maya Chen: So the question isn't just what emergence is — it's whether it's even the right way to describe what we're seeing.
Dr. Nathan Hayes: Exactly that. And we don't have a clean answer.
Maya Chen: But that gap — between smooth and jagged — I keep wanting a way in. Like, what does it actually feel like to cross that line? Is there even a good way to picture it?
Dr. Nathan Hayes: There is, actually. Think about a child learning to read. Every hour of practice, letter recognition improves — smooth, predictable, you could graph it. Then one day she reads a full sentence and it means something. The component skills were always improving. But comprehension snapped into place.
Maya Chen: Oh. The letters were always getting better — but the sentence is a different thing entirely.
Dr. Nathan Hayes: Exactly. Scaling laws are the letter-recognition curve. Emergent abilities are the moment the sentence lands. And — importantly — you cannot predict the sentence moment by watching the letter curve. It's not on that graph.
Maya Chen: Wait. But if the loss curve is smooth the whole time — if Kaplan et al.'s power-law just keeps going — how can a specific ability snap on? Shouldn't something in the underlying numbers warn you?
Dr. Nathan Hayes: DeepMind's Chinchilla study actually tightened the whole picture. What they showed is that optimal performance depends jointly on parameter count and training data size — a lot of large models at the time were undertrained. But even with that correction, even at the compute-optimal frontier, the improvement stayed smooth. The loss curve does not jump. So the jump you see in, say, three-digit arithmetic — that's not in the loss. It's somewhere else.
Maya Chen: Somewhere else — meaning we don't actually know where?
Dr. Nathan Hayes: Mechanistically, no, not precisely. The working explanation — and I want to be clear this is inference, not direct observation — is a representational capacity threshold. Smaller models lack sufficient space to encode the abstract structure a task requires. Above some scale, that boundary is crossed. But we infer that backward from the performance curve. We haven't seen the boundary itself.
Maya Chen: So it's — hm — it's a story we're fitting to the data, not a mechanism we've actually caught in the act.
Dr. Nathan Hayes: That's a fair way to put it. Wei et al. defined emergence formally — an ability you cannot predict by linearly extrapolating from smaller models — which means by definition it's the part of scaling that the smooth curves miss. The definition is operationally precise. What it doesn't tell you is why.
Maya Chen: And that gap — that's not a footnote. That's sort of the whole problem, right? Because if you can't predict which abilities emerge at which scale, you're always discovering it after the fact.
Dr. Nathan Hayes: Yes. And that's not a comfortable place to be. Multi-step arithmetic, symbolic manipulation, few-shot in-context learning, instruction-following — all of them showed near-random performance in smaller models and sharp performance in larger ones. Across PaLM, GPT-3, multiple families. The pattern is real. The forecast isn't.
Maya Chen: Okay but — wait — that 'always after the fact' thing is where I want to push, because I think there's a version of this where the problem isn't the model at all. It's us. The ruler we're using.
Dr. Nathan Hayes: The Schaeffer metric artifact point. Yes — and it's the part I find most uncomfortable, honestly.
Maya Chen: Walk me through it concretely. Because I keep picturing — a researcher, Saturday morning, running GPT-3 through letter-string analogy tasks. The canonical version. Model aces it. She's thinking, okay, something real just happened here. Then she swaps two numbers in the sequence.
Dr. Nathan Hayes: And the model falls apart. That's exactly what Damian Hodel and Jevin West documented in 2023 — simple variations of those same letter-string analogies, the kind humans handle without blinking, and GPT-3 collapses. Change the wording slightly, swap the numbers — performance gone.
Maya Chen: So she wasn't watching reasoning. She was watching — pattern matching on training data.
Dr. Nathan Hayes: That's the data memorization confound. And Hodel and West's point is that before you claim zero-shot analogical reasoning emerged, you have to rule that out. Which the canonical evaluation doesn't do — it can't, because it's testing the same surface the model may have memorized.
Maya Chen: Right — but now layer in what Schaeffer's group showed about the metric itself. Because that researcher on Saturday isn't just choosing a bad task. She might be choosing a bad measuring stick.
Dr. Nathan Hayes: The metric artifact hypothesis, exactly. If you score answers as solved or unsolved — binary — a model going from forty percent correct to sixty percent correct looks flat. Nothing happening. Switch to a partial-credit metric, a continuous one, and that same improvement shows up as a smooth upward slope the whole time. The leap was in the ruler.
Maya Chen: The discontinuity wasn't in the model — it was in how we decided to count.
Dr. Nathan Hayes: Now — bring in Lu et al. from 2023. Their argument is that many claimed emergent abilities are confounded by in-context learning competencies developing separately. The model got better at parsing what the prompt wants, and we called it emergence of the underlying task.
Maya Chen: Wait. So — in-context learning is itself an emergent ability, by Wei and Tay's own definition, and we're using it to measure whether other things are emergent?
Dr. Nathan Hayes: That's the trap. Few-shot prompting is the standard evaluation paradigm for emergence — you give the model examples in the prompt, no gradient updates, and see if it can do the task. But that method works only if in-context learning is already strong. So the measurement tool and the confound are the same instrument. You can't separate them.
Maya Chen: That's — I mean, that's a methodological trap you can't just clever your way out of. The thing you're testing with is the thing that might be producing the result.
Dr. Nathan Hayes: And that's before we even get to what this means for safety — if harmful capabilities like deception or reward hacking obey the same logic, appearing suddenly with no smooth warning curve, the evaluation frameworks we have are structurally blind. That's where this stops being academic.
Maya Chen: And that's the part that won't leave me alone — structurally blind. Because picture someone at NIST, let's say, designing an eval suite for a new deployment. They run it on the 7B checkpoint, the 13B, the 34B. Everything looks fine. Reasonable. And then the 70B ships and something is just... different in a way none of those checkpoints warned them about.
Dr. Nathan Hayes: That's not a hypothetical failure mode. That's the definition of the problem. Wei et al.'s formal definition of emergence is literally the region where extrapolation from smaller models fails. So if you're doing safety evaluation on smaller checkpoints — which is what pre-deployment testing is — you are by definition working in the blind spot.
Maya Chen: Wait. That's in the definition itself?
Dr. Nathan Hayes: It is. The phenomenon is defined as the place extrapolation breaks. So any evaluation framework premised on 'smaller models predict larger models' is structurally mismatched to exactly the capabilities it most needs to catch.
Maya Chen: Okay but — I want to push on one thing. If Schaeffer's group is right, if the discontinuity is the ruler and not the model, does that NIST evaluator actually have a problem? Like, does the safety concern dissolve if emergence is a measurement artifact?
Dr. Nathan Hayes: No. And this is — actually, let me be careful here because it's easy to get this wrong. Even if the underlying capability is improving smoothly, the task-level behavior can still snap on once a threshold is crossed in practice. Wu and Lo showed this in 2024 — hard questions exhibit U-shaped scaling, easy questions exhibit inverted-U scaling, and at intermediate scales those two patterns cancel each other. You see apparent stagnation and then a jump. That cancellation effect is real regardless of which account of emergence is correct.
Maya Chen: So the smooth curve underneath doesn't protect you.
Dr. Nathan Hayes: Not from the jump at the task level. And now name the tasks we're worried about. Deception. Manipulation. Reward hacking. Those don't get tested in a controlled benchmark the way arithmetic does — they're harder to even operationalize. So you have capabilities that may arrive on the same schedule as beneficial ones, with no advance signal from smaller checkpoints, and no clean eval to catch them.
Maya Chen: That's — I mean, that's not a capability milestone anymore. That's a governance gap.
Dr. Nathan Hayes: And the representational capacity threshold explanation, if it's correct, makes it worse — not better. Because it implies there's no gradual signal. Below the threshold, the model structurally cannot encode the abstract structure required. Above it, the capability becomes possible. No ramp. No warning.
Maya Chen: And we still can't predict which threshold, for which task, in which new model family — before training.
Dr. Nathan Hayes: We have no prospective theory for it. The pattern held across LaMDA, GPT-3, Gopher, Chinchilla, PaLM — multiple architectures, Google Research, DeepMind, OpenAI. The consistency across those families is genuinely striking. But consistency post-hoc is not prediction. A real phase transition — water freezing — you forecast from thermodynamics before you freeze the water. We're describing this after the fact every time.
Maya Chen: So that NIST evaluator isn't doing anything wrong. She's just — the tools she has were built for a different kind of problem.
Dr. Nathan Hayes: Exactly. Frameworks designed for gradual improvement assume the small model is a scaled-down version of the large one. Emergence breaks that assumption at precisely the capabilities that carry the most risk. That's the structural mismatch — and it doesn't go away even if you accept every critique Schaeffer or Lu or Hodel and West have made.
Maya Chen: The thing that keeps landing for me — and I think it's where I actually settled — it's not whether Wei et al. are right or Schaeffer's group is right. It's that we're scaling without a prospective theory. Like, nobody can say, before training, which capabilities cross the threshold next. Beneficial or harmful. We find out after.
Dr. Nathan Hayes: That's the honest place to end up, yes. The disagreement between the genuine phase-transition account and the metric artifact account — that's unresolved. And the resolution isn't academic, because it changes what kind of governance you'd build. But both accounts share the same gap: no forecast.
Maya Chen: Which brings me back to the riddle I handed you at the start. Two models, same task, 125 million parameters versus 175 billion. I framed it like the answer was in the gap between them. But now... I don't think the gap is where the answer lives.
Dr. Nathan Hayes: No — the answer would have to be in what we could predict before we ran either model. And we had nothing.
Maya Chen: Yeah. That's — sort of a quieter and worse version of the same riddle, isn't it.
Dr. Nathan Hayes: It is. Worth sitting with.
