Iris Holm: Tell me if this is weird — I've been thinking about RNN researchers. Like, specifically them. 2018, you've built your whole career on sequential processing, and then one paper from Google Brain makes the paradigm obsolete.
Hana Field: Oh, that's not weird at all — that's the part of this I can't stop thinking about either. Because it happened fast. And that paper — Ashish Vaswani, Noam Shazeer, the 2017 paper — it didn't just propose a new technique, it proposed that the whole sequential thing was unnecessary.
Iris Holm: Which is a brutal claim if you've spent years optimizing RNNs.
Hana Field: Brutal and, it turned out, basically right — and so that's what we're getting into today. The transformer. The architecture that lives underneath every major model since 2017, GPT-2, GPT-3, BERT, Claude, Gemini, the whole lineage. And the question driving it is: what problem were they actually solving, and why did this specific mechanism — self-attention — win?
Iris Holm: Start with the problem. What were Recurrent Neural Networks actually failing at?
Hana Field: So RNNs processed sequences token by token — one word at a time — and they were carrying this compressed hidden state forward as they went. And the failure mode was long-range dependency capture. Information from the beginning of a sequence had to survive through every intermediate step to reach the end, and it degraded. Sometimes badly. The further apart two relevant tokens were, the harder it was for the model to connect them.
Iris Holm: So the model might just... lose the subject of a sentence before it reaches the verb.
Hana Field: Exactly that. And self-attention solved it by doing something almost conceptually violent — instead of reading in sequence, every token looks at every other token simultaneously. Direct. No waiting, no degradation across steps. And the math behind it, the query-key-value system, is actually simple enough that you can write the core formula on a napkin. That's what I find so disorienting about it.
Iris Holm: Wait — the formula that replaced an entire sequential processing paradigm fits on a napkin?
Hana Field: Scaled dot-product attention. Softmax of QK-transpose divided by the square root of d-k, times V. That's it. And yet somehow that operation — those three learnable matrices — is what every major language model is built on. So what does it mean to pay attention? That's where I want to go.
Iris Holm: Okay, so the napkin formula — let me actually make that land. Dinner party. Someone across the table says your name. Every other conversation drops out and your brain locks onto that voice. Attention is exactly that, but the model does it for every word, to every other word, all at once.
Hana Field: And the reason that matters — that 'all at once' part — is that an RNN couldn't do it. It had to pass your name down a telephone chain, one step at a time, and by the end of a long sentence, the signal was just... quieter.
Iris Holm: Right. So the Q-K-V thing. The query is what a token is looking for. The key is what each token is offering. And the value is what actually gets passed along if the match is good.
Hana Field: So the model is basically asking every other word — hey, are you relevant to me right now?
Iris Holm: Yes. And it's not asking one at a time — every token is asking every other token simultaneously. That's the thing RNNs literally could not do architecturally.
Hana Field: And what I find almost strange about it — the model isn't told which words matter. It learns which queries match which keys through training. No one writes the rules for relevance.
Iris Holm: Which is — wait, actually that's the disorienting part. The scores come from a dot product. Geometry. How aligned two vectors are in learned space.
Hana Field: Oh wow — yeah, relevance as angle.
Iris Holm: And then — this is where multi-head attention comes in — you don't run this once. You run it in parallel, multiple times, with different Q, K, V projections. Different heads pick up different relationships. One might track syntax, another semantics, another position.
Hana Field: So one head is noticing that 'she' refers back to a name mentioned four sentences ago, and a completely different head is tracking that 'ran' agrees with 'she' grammatically — at the same moment.
Iris Holm: Simultaneously. That's what Vaswani and Shazeer built at Google Brain — not just attention, but attention running in parallel channels, each learning something distinct.
Hana Field: And that's what the whole GPT lineage sits on. Every generation of those models — same mechanism, scaled.
Iris Holm: Which is the question. If the mechanism is that elegant — and it is — why are we now layering in Mamba? That's the crack in the ceiling we haven't named yet.
Hana Field: That crack is actually a wall, and the math makes it embarrassingly concrete. Because the formula we just named — softmax of QK-transpose over root d-k — every token attending to every other token means the cost grows quadratically with sequence length. Double the sequence, you don't double the compute. You quadruple it.
Iris Holm: O of n-squared.
Hana Field: And so imagine a legal analyst — it's nine in the morning, she's got a 200-page acquisition contract she needs to feed into the model. That's not a weird edge case, that's Tuesday. And the transformer either truncates the document or the compute cost makes it just... not viable. The architecture hits its ceiling at exactly the moment the task gets long enough to actually need it.
Iris Holm: The architecture famous for long-range dependencies. Fails on long documents.
Hana Field: And it doesn't even fail loudly — that's the part that gets me. It's not a crash. What happens instead is attention dilution. The scores spread across too many tokens, and the model just... quietly loses focus. It's still generating output. It just stopped meaningfully tracking the clause on page three that contradicts the clause on page ninety.
Iris Holm: Which is worse than a crash, actually. You don't know it happened.
Hana Field: Exactly. And then there's the Naim et al. work from 2025 — Naim, Bolte, and Asher — and this one I find genuinely unsettling, because the claim isn't just practical. It's mathematical. They present empirical and mathematical evidence that transformers cannot achieve general predictive accuracy during in-context learning.
Iris Holm: Hold on. In-context learning was the whole superpower. GPT-3, few-shot prompting — that was the thing.
Hana Field: Yeah, and what Naim et al. are saying is — and I want to be precise here because this is easy to overstate — they're not saying it never works. They're saying there are inherent architectural limits on how robust or general that capability actually is. The model can't actually implement arbitrary learning algorithms from examples in the prompt. What it can do is pattern-match to what it memorized during training.
Iris Holm: So in-context learning is — what, sophisticated retrieval? Not generalization?
Hana Field: That's the uncomfortable version of the claim, and I think it's closer to right than the marketing version. The scaling factor in the formula — the root d-k that stabilizes training — is there because without it the dot products explode and gradients vanish. The architecture has load-bearing constraints baked all the way down. And those constraints don't disappear just because you have more compute.
Iris Holm: Which is the reason Mamba exists. And why every major hybrid still keeps the attention layers anyway — and whether that's genuine architectural permanence or just the weight of entrenched infrastructure, that's the part we need to pull apart next.
Hana Field: The ceiling is real. The question is whether we're close to breaking through it, or just building taller furniture.
Iris Holm: Taller furniture is actually the right image — because Mamba isn't replacing the foundation, it's handling what the foundation can't. State space model, linear time complexity, maintains a compressed hidden state instead of scoring every token against every other token. That's the pitch. And yet Jamba — the hybrid — still keeps attention layers in.
Hana Field: Which is the thing I keep snagging on, you know? If Mamba solves the quadratic problem — genuinely, in linear time — why not just... go all the way?
Iris Holm: Because Mamba alone can't do what attention does on relational scoring. Long-range dependencies across a document aren't just about remembering a token — they're about scoring its relevance to every other token dynamically. A compressed hidden state loses that. You compress, you forget what you compressed.
Hana Field: Mm — but is that a fundamental limit or just where the research is right now?
Iris Holm: That's the exact question. And I don't think we can answer it cleanly yet.
Hana Field: And so what Jamba does — and Mixtral 8x7B is doing something adjacent with sparse mixture-of-experts routing, where tokens get routed to specialized sub-networks rather than running through everything — what these hybrids are really saying is: attention for the hard relational work, something cheaper for the rest. And that feels less like a vindication of attention and more like... okay, we can't remove it yet, so we'll route around the expensive parts.
Iris Holm: That framing assumes the 'yet' does a lot of work. Here's my actual skepticism — BERT, GPT-2, GPT-3, Claude, Gemini, the whole lineage back to Vaswani and Google Brain in 2017, they're all built on the same attention core. That could mean attention is irreplaceable. Or it could mean the field committed before Mamba-class alternatives were mature enough to compete, and now the GPU optimization, the training frameworks, the tooling — all of it is tuned for attention operations. Switching isn't free even if something better exists.
Hana Field: Path dependency as its own kind of cage.
Iris Holm: Right — and the infrastructure cost is real and independent of the capability comparison. That's the part that doesn't get said.
Hana Field: And yet — and I think this is where I'd push back a little — the fact that every serious hybrid design still retains the attention layers, not as legacy scaffolding but as the component doing the most computationally expensive and apparently irreplaceable work... that's not nothing. Picture a genomics researcher, two in the morning, feeding a protein sequence that's thousands of tokens long into a hybrid model. The SSM layers handle the bulk, cheap and fast. But the attention layers are the ones catching that a mutation at position forty has a relational consequence at position nine hundred. That's not inertia. That's a division of labor that actually maps onto what each mechanism does well.
Iris Holm: Which I'd accept — if we could separate 'attention is doing something nothing else can' from 'attention is doing something we haven't yet trained the alternatives to do at scale.' Those are not the same claim.
Hana Field: No, they're not. And I don't think the evidence resolves it. The transformer's persistence across every generation, that could be genuine architectural superiority or it could be that no alternative was ever given the same infrastructure investment to compete on equal footing.
Iris Holm: And that ambiguity is the honest place to land. Two explanations, same evidence, not yet separable.
Hana Field: And what that means practically — for anyone building on these models, anyone trusting them with long documents or complex reasoning — is that the thing you're relying on might be permanent or might be a very well-funded placeholder. And you probably can't tell from the outside which one it is.
Iris Holm: Vaswani and Shazeer solved two specific problems. Parallelizability. Long-range dependencies. Both at once, 2017, Google Brain. And the field built everything on top before anyone knew where the ceiling was.
Hana Field: And we're still inside it. That's the strange part. The scaffold held long enough that it became load-bearing — and now when we talk about Mamba, or Jamba, or sparse MoE routing, we're not talking about replacing the foundation. We're talking about what to add to a building that's already standing.
Iris Holm: Which circles back, honestly, to those RNN researchers. The ones we opened on.
Hana Field: Yeah — I've been thinking about that too, because it lands differently now. In 2018 those researchers watched their whole approach become textbook history. And at the time it probably felt like the transformer was the answer. Final. Closed. And what we're sitting with now is — it wasn't the answer. It was the answer to a specific question, at a specific moment, and we just... didn't know the next question yet.
Iris Holm: Permanent solution or exceptionally successful scaffold. I don't think we can call it.
Hana Field: No. And I think that's the honest place to leave it. Not unresolved in a frustrating way — just... actually unresolved.
Iris Holm: Good conversation. Genuinely.
