Cyrus Reed: Iris, good — you're here, because I read something this week that I genuinely cannot explain and it's been bothering me since Tuesday.
Iris Holm: Tuesday is when the good ones hit. What was it?
Cyrus Reed: So imagine you're a safety researcher, right, and you're running the same multi-step reasoning task on two models. Identical architecture. Identical training setup. One has seventy billion parameters, one has seven billion. And the smaller one just — fails. Like, completely. But the bigger one solves it. And not like, 'does better on a benchmark.' Solves. It.
Iris Holm: Abruptly. Not a curve.
Cyrus Reed: Right — but how does that happen? The training objective is just next-token prediction the whole time. No one taught it to reason. No one added a step. So what — what flipped?
Iris Holm: Wei et al. called it emergent abilities in 2022. The formal definition: capabilities like in-context learning, multi-step reasoning, code generation — not present in smaller models, appearing abruptly past a critical scale threshold, not explicitly trained for. GPT-3, GPT-4, PaLM — those are the flagship cases.
Cyrus Reed: Wait — so this is a named, documented phenomenon and researchers still don't know why it happens?
Iris Holm: Named, documented, and actively contested. That's the more interesting part.
Cyrus Reed: Contested how — like, people think it's not real?
Iris Holm: Some people think it's an artifact of how you measure. Change the metric, the jump disappears. But we'll get into that. The first question — the one I want you to actually sit with — is simpler. Why would scale alone produce qualitatively different behavior from a model that is still, at its core, just predicting the next token?
Cyrus Reed: That's — yeah, that's the thing. Because the task didn't change. The model didn't change type. It just got... bigger.
Iris Holm: And something on the other side of a threshold that nobody can predict in advance.
Cyrus Reed: Okay but — threshold nobody can predict. That's the part that keeps snagging me. Like what is actually happening inside the model that makes 'bigger' suddenly mean 'different'?
Iris Holm: Here's the intuition. You're cramming a city map into memory, one street at a time. At some point you have enough streets that something clicks — you stop remembering routes and start understanding the city's geometry. Navigation was always latent in the streets. You just needed enough of them.
Cyrus Reed: That's — yeah. Okay.
Iris Holm: That's overparameterization. More parameters than the model strictly needs — and that excess capacity is what lets it compress latent structure that was always in the training data. The geometry was there. The model finally had room to find it.
Cyrus Reed: But why does the compression happen suddenly? Like — wait, no — if it's just accumulating streets gradually, shouldn't understanding also arrive gradually?
Iris Holm: That's exactly the right question. The answer is sparse attention patterns. At sufficient scale, the transformer's attention mechanism doesn't slowly drift toward task-relevant configurations — it abruptly reorganizes into them. Like water freezing. Not slowly getting colder. Solid.
Cyrus Reed: So the mechanism isn't smooth accumulation — it's a phase transition. An actual structural flip in how the attention heads are allocating weight.
Iris Holm: Right. And in-context learning is the clearest product of that flip — the ability to see a few examples in a prompt and perform a task with zero weight updates. That capability isn't trained in. It falls out of the attention geometry reorganizing.
Cyrus Reed: No way — so in-context learning isn't a feature someone designed?
Iris Holm: Nobody put it there. It's structural emergence from next-token prediction.
Cyrus Reed: And then — wait, there was a finding about this in small models, too? Because I thought this was supposed to be a frontier-scale thing, like GPT-4 or PaLM territory, but I read something about phase transitions showing up in seven-billion-parameter transformers, which is... not frontier.
Iris Holm: Research on small GPT-style transformers shows the same phase transitions — vocabulary, capability development — at modest scales and early in training. Which collapses the story that this is exclusive to frontier models. The transitions are architectural. Scale lowers the threshold. It doesn't create the mechanism.
Cyrus Reed: But — wait, that's actually what breaks the clean version of all this. Because if the transitions are architectural and not scale-exclusive, then how do we even know they're real discontinuities? Because there's this whole — okay, so the mirage critique. That's the thing that's been sitting with me since I read the Stanford HAI take on it.
Cyrus Reed: The move the mirage paper makes is — it's almost annoyingly simple, actually. You get a cliff on the benchmark because you're grading exact match. Right or wrong. No partial credit. So a model that's getting closer and closer to the answer over thousands of training steps looks like it's flat-lining, and then suddenly it crosses the threshold and — cliff. But if you switch to token-level probability, how likely was the correct next token, the cliff just... becomes a slope. Same model. Same task.
Iris Holm: And Stanford HAI put institutional weight behind that framing. It's not a lone contrarian paper.
Cyrus Reed: Which is what makes it land so hard. But — okay, picture this. A safety researcher, late afternoon, she's running a 13-billion-parameter model on a multi-step reasoning benchmark. Her grading script says fail. Model failed. Done. Except she pulls up the token-level probabilities and the correct answer token was sitting at, like, 0.41 probability. The model was almost there. The discontinuity was in her rubric.
Iris Holm: That's the uncomfortable version. The capability didn't jump. The scoring function created the illusion of a jump.
Cyrus Reed: No way — so does the mirage critique actually dissolve emergence entirely? Because that seems like it should be bigger news than it is.
Iris Holm: It doesn't dissolve it. It displaces it. The discontinuity might still be real — it just lives in the metric, not the model. That's a different claim with different consequences.
Cyrus Reed: Huh. Okay — wait, is metric discontinuity actually less interesting? Because I mean, if the measurement is what jumps, and the measurement is what we use to decide whether a model is safe to deploy—
Iris Holm: Exactly. That's the live wire. And it connects to the Webb, Holyoak, and Lu thing — they claimed GPT-3 showed emergent analogical reasoning, zero-shot. Critics came back and said: the training data probably contained structurally similar problems. Not reasoning. Retrieval.
Cyrus Reed: So scaling laws show loss improving on a smooth power-law curve, but capability benchmarks show these hard jumps — and the jumps might just be artifacts of how we grade. That means our entire picture of what these models can do depends on which ruler we pick up.
Iris Holm: Which is a problem that gets much worse the moment you bring in safety and governance — and whether regulators should treat capability jumps as genuinely unpredictable events or as a story we accidentally told ourselves with the wrong metric.
Cyrus Reed: Yeah, and — that part, I think we have to come back to, because the stakes there are completely different depending on which answer is right.
Iris Holm: And that ruler problem is exactly where it stops being academic. Scaling laws give you a smooth power-law curve — loss, compute, parameters, all predictable. But capability? Nobody can tell you when the next cliff appears. Those are structurally different kinds of knowledge. One you can forecast. One you cannot.
Cyrus Reed: Wait — so the thing regulators would actually use to make decisions, the capability benchmark, is the thing that might be broken?
Iris Holm: Whether the mirage critique is right or wrong, the governance problem is identical either way. Real emergence: you get genuinely unpredictable capability jumps past scale thresholds. Mirage: your safety narrative was built on a misconstrued causal story. Both outcomes leave regulators holding frameworks built on extrapolation from smaller models — and smaller models, by definition, don't show you what you need to see.
Cyrus Reed: Huh. That's — okay, that's a worse situation than I thought, actually. Because the two failure modes aren't opposites. They both break the same thing.
Iris Holm: Now think about Anthropic working on Claude, or DeepMind's work behind Gopher and Chinchilla. Those labs are not sitting still. There's active research — chain-of-thought prompting, for one — that shows once you're past a threshold, you can amplify what's there. You include intermediate reasoning steps in the prompt, and suddenly the model is doing multi-step problem-solving it couldn't do without them. CoT is itself an emergent capability, by the way. It only works at scale.
Cyrus Reed: No way — chain-of-thought prompting is emergent? Like the technique for eliciting reasoning is itself a product of scale?
Iris Holm: Below a certain parameter count, showing the model a reasoning chain does nothing. The capacity to use it isn't there. Which means — and this is the part that should unsettle people — you can't test CoT on a smaller proxy model and conclude anything about how it will behave at deployment scale.
Cyrus Reed: So the evaluation tool is itself scale-dependent. The thing you'd use to check whether a model reasons safely — doesn't even work on the models you're allowed to fully inspect.
Iris Holm: Exactly. And there's a further step — meta-abilities alignment. Instead of waiting for an 'aha moment,' you explicitly train on self-verifiable tasks targeting deduction, induction, abduction. All three. The idea is to make emergent reasoning controllable rather than coincidental. But — and this is the tension — if you can engineer emergence earlier, you've just handed someone a dial.
Cyrus Reed: Wait — doesn't that actually destroy the unpredictability argument? Because if you can deliberately trigger emergence by manipulating training, then it's not an irreducible surprise anymore, it's a — I mean, that's almost the opposite of what makes it scary from a governance standpoint.
Iris Holm: That's the exact right tension. And it cuts both ways. If labs can engineer emergence, they can also conceal it. The unpredictability that justified loose oversight becomes a choice, not a constraint.
Cyrus Reed: Oh. So — so the question shifts from 'when does emergence happen' to 'who decides when it happens.' That's a completely different regulatory problem.
Iris Holm: Picture a policy analyst — not a technical person, someone writing compute thresholds into law. She's drawing the line at ten to the twenty-six FLOPs because that's where the last capability jump appeared. But the meta-abilities alignment work means the next lab doesn't have to wait for that threshold. They can seed the transition earlier. Her line is already behind the curve before the ink dries.
Cyrus Reed: So the thing we still don't know — is whether emergence is a property of scale, or a property of what you decide to train for. And that distinction is doing enormous work in a room full of people writing policy.
Iris Holm: Either way, the tools we built for prediction were designed for smooth curves. Scaling laws, power-law loss, predictable compute trade-offs. None of that was designed for cliffs.
Cyrus Reed: And — wait, I think that's actually what clicked for me in all of this. Because whether the Wei et al. framing holds up or the mirage critique wins, the honest answer is the same: we don't have a foresight framework that fits a world where the next capability might be a discontinuity. Like, if it's real, scaling laws can't warn you. If it's a mirage, the safety story was built on the wrong causal model the whole time. Those are different problems but they land in the same place — nobody actually knows what comes next.
Iris Holm: The part that stays with me is simpler than the measurement debate. The transformer architecture keeps producing things nobody designed it to produce. In-context learning. Chain-of-thought at scale. Sparse attention patterns nobody specified. Whatever the right explanation is — real discontinuity, metric artifact, something in between — the architecture is doing something we didn't put there.
Cyrus Reed: That's — yeah. That's the part that doesn't go away regardless of which side wins.
Iris Holm: Frankly, that's a reasonable place to stop.
Cyrus Reed: Yeah. I mean — I came in wanting an answer and I think I got a better question, which feels about right for this one.
Iris Holm: A better question is the useful output. Thanks for the Tuesday problem.
