Malcolm Reeves: Cole — I want to start somewhere specific. October 19th, 2026. MIT Technology Review drops a piece under the headline 'We were looking at it wrong.' You see that?
Cole Brennan: Whoa, yeah — for real, that headline alone made me stop scrolling.
Malcolm Reeves: And the source is a twenty-six-thousand-word essay by Nick Cammarata — former OpenAI, now at Goodfire — arguing that AI neurons self-organize around clean, human-legible concepts. He calls it 'recursive generalization.' His claim: the models are, essentially, already interpretable. We just weren't reading them right.
Cole Brennan: Right — but here's where I keep getting stuck. The New York Times picks it up October 27th, eight days later. 'Did the Hardest A.I. Problem Just Get Much Easier?' And then — same window — Neel Nanda and Chris Potts at Stanford are publicly saying no, this is not close to solved. Like, that's a fast collapse from possible paradigm shift to case study in hype.
Malcolm Reeves: Eight days. That's the window we're trying to understand.
Cole Brennan: And the street-sign version — I mean, the plain-language version — is this: you can learn what every neuron in a neural network responds to, every label on every street sign, and still have no idea why the system fails when it fails. Legibility is not the same thing as understanding. That's the whole fight.
Malcolm Reeves: But that distinction — legibility versus understanding — I want to make sure I'm seeing it clearly. Because on the surface it sounds like you're saying: knowing what a neuron fires for is not the same as knowing *why* the model made a decision. Is that the actual gap?
Cole Brennan: That's exactly it. And Cammarata's work — look, mechanistic interpretability, which he helped pioneer at OpenAI, it's the practice of cracking open a network and asking neuron by neuron: what does this respond to? And 'recursive generalization' is his claim that neurons don't just fire randomly — they self-organize around coherent central concepts, like they're tidying themselves up. Which is genuinely interesting. But Neel Nanda's response draws this very precise line — he says, okay, the demonstrations have research value, but — and this is the part people glossed over — neuron-level legibility does not scale to causal explanations of what a frontier model actually does under pressure.
Malcolm Reeves: Does not scale — meaning what, exactly?
Cole Brennan: Meaning — okay, imagine a trauma surgeon in Nairobi reviewing an AI triage recommendation mid-shift. The system flags one patient as lower priority. The explanation is there, neurons lighting up, concepts labeled, it looks clean. But can the surgeon trace *why* that specific patient got deprioritized versus the one next to them? That causal chain — across hundreds of layers, millions of interactions — that's what Neel Nanda is saying we still can't do. And Chris Potts at Stanford was even blunter — he said the essay changes nothing about the difficulty of understanding LLMs.
Malcolm Reeves: Potts said that directly? 'Changes nothing'?
Cole Brennan: Word for word, yeah. And here's what I find important about that — the critics aren't saying Cammarata is wrong about the neurons. They're saying he's wrong about what that *implies*. That's a different kind of disagreement. It's not 'your data is bad' — it's 'your conclusion overreaches the data by about ten miles.'
Malcolm Reeves: And the part that I think makes this worse — we haven't even gotten to what the field's actual track record looks like on validating these claims with real humans. That number, when it comes up, reframes everything.
Cole Brennan: And that number is — I mean, I want to say it carefully because it sounds almost too damning. A 2025 arxiv review found fewer than 1% of explainable AI papers actually validate their explanations with human subjects. Not 10%. Not 20%. Fewer than one in a hundred.
Malcolm Reeves: Is that right. One percent.
Cole Brennan: Which means — wait, actually this is the part that reframes Cammarata's essay completely — the XAI field has been producing interpretability claims for decades without ever asking whether a human on the other end actually understood the explanation. MIT SERC's position is that if you can't show the explanation was received and understood by a person, it doesn't count as interpretable. Full stop. And almost nobody clears that bar.
Malcolm Reeves: Now picture a claims adjuster — Thursday afternoon, she's reviewing a submitted medical image, the AI flags it as inconsistent with the claimed injury. System surfaces a visualization, neurons lighting up in the right places. Looks legible. But she cannot tell whether that pattern tracks a real medical signal or a spurious correlation baked in during training. Neuron legibility does not answer that question.
Cole Brennan: And that's the scenario where 'recursive generalization' either proves itself or doesn't. And the thing is — it hasn't been replicated across different architectures yet. That's not a minor caveat. The NYT called this a potential resolution to AI's oldest puzzle. It's a blog post.
Malcolm Reeves: A blog post that MIT Technology Review had pre-loaded with credibility — because they'd already named mechanistic interpretability one of their top 10 breakthrough technologies. That framing was already in place before Cammarata published a word of this essay.
Cole Brennan: Which is — huh — I mean, that's how a contested direction gets mistaken for a breakthrough before the replication is even attempted.
Malcolm Reeves: And that's what Chris Potts put his finger on — he said explicitly it was prominent social-media figures amplifying the essay, not the substance of the work itself, that drove the reaction. The frame was already set before the NYT even ran their piece on October 27th. The mainstream audience arrived to a consensus that had formed without them.
Cole Brennan: Which means — I mean, the genuinely uncomfortable version of this is — hype and real progress can look completely identical for weeks. Maybe longer. If 'recursive generalization' gets independently replicated across different architectures, it could be actual traction on AI oversight. If it doesn't... this whole episode is just a case study in how fast that gap opens between what a blog post demonstrates and what a field will accept as solved.
Malcolm Reeves: And we genuinely don't know which one we're in yet.
Cole Brennan: No. We don't. That's an honest place to sit with it, I think. Appreciate you working through it.
