Jordan Hale: Ryan, okay — weird week. I ended up in this conversation with someone who works on ML infrastructure and she said something I cannot stop thinking about. She said, "we're all living inside a 2017 decision."
Ryan Castillo: The transformer.
Jordan Hale: "Attention Is All You Need." That paper. And the thing is, I've heard the name a hundred times but I actually went and read what it did, and — it replaced recurrence. Completely. RNNs processed tokens one at a time, sequentially, that was just the paradigm, and this paper said no, every token attends to every other token simultaneously. All at once.
Ryan Castillo: Which is what made distributed training tractable. Parallelism across positions means you can split the work across GPU clusters — that's the property that made OpenAI, Google DeepMind, Anthropic, Meta, Mistral all converge on it.
Jordan Hale: Every major lab. That's — I mean, I know it when you list them out like that but it still lands weird every time.
Ryan Castillo: Here's what I find stranger: GPT-3 proved the bet was right. Billion-parameter transformers, scaling laws hold in practice, capability improves predictably with compute. That validation happened. And simultaneously the field was sitting on the fact that self-attention scales quadratically — double sequence length, quadruple your cost.
Jordan Hale: So the same mechanism that cracked the sequential bottleneck — no wait, let me back up — the thing that made the transformer brilliant is also exactly the thing that makes it expensive at scale?
Ryan Castillo: That's the structure of it. O(N²) is a consequence of what makes self-attention work, not incidental to it.
Jordan Hale: That's genuinely an uncomfortable shape for a foundational bet — like, the thing that saves you is also the thing that eventually corners you.
Ryan Castillo: And the question is whether the field has noticed it's cornered, or whether it's decided to build its way out in silicon instead of software.
Jordan Hale: Which — that's what I want to dig into, because the answer to that is not obvious to me at all.
Ryan Castillo: But to actually get there — to the silicon question — we need to sit with why RNNs failed first. Because the conventional story is 'they were slow,' and that's not wrong, it's just not the whole thing.
Jordan Hale: What's the rest of it?
Ryan Castillo: The constraint wasn't that parallelizing RNNs was hard to engineer. It's that the architecture makes it structurally impossible. Imagine you're trying to summarize a book, but you're only allowed to read one sentence at a time — and you cannot touch sentence three until sentence two is completely processed. That's an RNN. You cannot hand half the book to someone else, because every step depends on the step before it.
Jordan Hale: So it's not a resource problem. It's like — you could hand that person a thousand extra books and it wouldn't help, because the rule says one sentence at a time.
Ryan Castillo: Exactly — and that's the number that matters here: the mid-2010s compute investment had nowhere to go. GPUs multiplied, distributed infrastructure scaled up, and recurrent networks literally could not use it. The sequential bottleneck compounded as sequences got longer. More hardware, same wall.
Jordan Hale: Wait — so LSTMs didn't fix that? Because I always thought LSTMs were the upgrade that bought recurrent nets more time.
Ryan Castillo: They fixed the vanishing gradient problem — the memory issue. But the sequential processing constraint? Still there. One token at a time, in order. LSTMs are a better book-reader who remembers more, but still forbidden from reading two sentences simultaneously.
Jordan Hale: Okay, that distinction is — I mean, I think I had those two things collapsed into one problem in my head. The memory limitation and the parallelism limitation are actually separate failures.
Ryan Castillo: Right, but here's where self-attention is the contrast. Spread the whole book on a table. Every sentence visible simultaneously. Every sentence comparing itself directly to every other sentence. That's what 'Attention Is All You Need' actually did — it made the architecture map naturally onto GPUs dividing work across many accelerators in parallel.
Jordan Hale: And that's — you know, that's the thing that made distributed training tractable. Not faster hardware on the same architecture, a different architecture that could actually use the hardware.
Ryan Castillo: Which is why architecture — specifically that parallelism across positions — is the load-bearing claim for why GPT-3 happened when it did and not five years earlier with RNNs.
Jordan Hale: So the compute was sitting there, ready, and the field was basically blocked by a structural rule it had built into the default architecture. And then 2017 just... removed the rule.
Ryan Castillo: And removing that rule had a second-order effect that I think gets collapsed into the headline. Multi-head attention — the extension that actually ships in every major model — it's not one attention operation, it's several running in parallel across different learned subspaces simultaneously. So you're not just getting parallelism across tokens, you're getting parallelism across different kinds of relationships between tokens.
Jordan Hale: Wait — different kinds of relationships? Like what, concretely?
Ryan Castillo: One head might be tracking subject-verb agreement, another's tracking coreference — which 'it' refers to — another's picking up positional proximity. They're each learning a different subspace, and you get all of them in one forward pass.
Jordan Hale: It's not — I mean, it's not just that the architecture could use distributed hardware, it's that the architecture was itself already parallel at multiple levels. Token-to-token and head-to-head.
Ryan Castillo: Which is the reason GPT-3 lands as the validation moment and not just a big model. You could throw 175 billion parameters at it and the scaling laws held — capability improved predictably. That's not an accident of compute. RNNs couldn't have used 175 billion parameters productively because the sequential bottleneck would have compounded faster than the parameters helped.
Jordan Hale: GPT-3 is basically the field cashing the architectural check that 'Attention Is All You Need' wrote in 2017.
Ryan Castillo: Look — I want to push on one thing though. Are we actually sure we can separate architecture from compute here? Because self-attention's quadratic cost only became tolerable because we built distributed GPU and TPU infrastructure specifically to absorb it. Didn't they co-evolve?
Jordan Hale: No, I think the architecture had to come first — you know, because the compute was already there in the mid-2010s and going nowhere useful. LSTMs weren't the bottleneck because of insufficient hardware. The hardware was ready. The architecture blocked it.
Ryan Castillo: That's the stronger version of the claim. Okay.
Jordan Hale: And the clearest way I can make it feel real — there's this scenario that I think captures it. Picture a researcher at a finance firm, three in the morning, trying to fine-tune a model on hundred-page regulatory documents. She's got an eight-GPU setup. She loads fifty pages. Memory explodes. Cuts it to twenty-five. Still hitting the wall.
Ryan Castillo: And that's not a hardware budget problem. She has eight GPUs.
Jordan Hale: Eight GPUs and she cannot get past twenty-five pages. Because the cost of every token attending to every other token — that quadratic thing — it's baked into what makes self-attention work at all. Which, honestly, is what I want to get into next, because the way the field is responding to that wall makes the original architectural bet look even stranger.
Ryan Castillo: The two-track response is uncomfortable.
Jordan Hale: Two tracks, right — and the thing that gets me is they're not sequential. It's not like the algorithmic people tried first and then the hardware people picked up the slack. They're running simultaneously, which means the field looked at O(N²) and basically said this is serious enough to attack from two directions at once.
Ryan Castillo: That's the signal. Two simultaneous research tracks on the same structural problem is not how you treat a footnote.
Jordan Hale: So walk me through the algorithmic side first.
Ryan Castillo: State-space models, linear attention variants, and — Gated Associative Memory. GAM. It's a linear-complexity architecture, fully parallel, explicitly designed as a transformer challenger. O(N), not O(N²). Doubling sequence length doubles cost instead of quadrupling it.
Jordan Hale: Wait — fully parallel and linear? Like, why hasn't that just... replaced transformers already?
Ryan Castillo: That's the right question, and I don't think the answer is purely technical. Look — every major lab, their entire training infrastructure, is built around the quadratic attention computation. Switching architectures isn't a software update. It's rebuilding the stack.
Jordan Hale: So it's — I mean, is it ecosystem lock-in more than a hard limit? Because that's a different problem.
Ryan Castillo: Probably both. But the hardware track is where it gets genuinely strange — memristor crossbars, analog gain cells, in-memory computing. The actual proposal is to implement the attention operation directly in the chip so the quadratic cost doesn't hit memory bandwidth the same way.
Jordan Hale: No way. They're solving it in silicon?
Ryan Castillo: That's the move. And what that tells you — the quadratic math is so embedded in the architecture that it's cheaper, apparently, to redesign the hardware than to abandon the algorithm.
Jordan Hale: That's a wild thing to sit with. Like, we built everything on self-attention, we discovered the structural cost, and the response was not — okay, let's go back to the drawing board — it was, you know, let's change the drawing board itself. Literally change what the chip does.
Ryan Castillo: And that means the 2017 bet — 'Attention Is All You Need' — might be sticky not because it's optimal but because we've committed to it so completely that the workaround is more attractive than the exit. GAM and the SSMs might be better. The field might know they're better. And transformers could still win on inertia.
Jordan Hale: We don't actually know if we chose the best architecture, or just the first one that worked at scale and then became too expensive to leave.
Ryan Castillo: Look — and I think that's actually the most honest place the evidence lands. Not 'transformers won because they're best.' More like: they were good enough, early enough, and now the tooling, the trained weights, the hardware optimization, the researcher expertise — all of it compounds around the architecture. GAM and state-space models aren't just fighting the math. They're fighting seven years of infrastructure.
Jordan Hale: Which is — I mean, that's a different kind of lock-in than I usually think about. Like, it's not just that switching is expensive. It's that every year you don't switch, the cost of switching goes up.
Ryan Castillo: Compounding industrial investment. That's the phrase that matters. And if analog in-memory computing actually lands — if the memristor approach makes quadratic attention cheap enough in silicon — then the efficiency argument for the alternatives just... softens. The theoretical advantage of O(N) stays on paper. Transformers stay in practice.
Jordan Hale: So they could win not because Vaswani et al. got it right in 2017, but because we built the escape hatch into the chip instead of into a new architecture.
Ryan Castillo: That's the uncomfortable version, yeah.
Jordan Hale: You know, I keep thinking about what she said — the ML infrastructure person who started all this. 'We're all living inside a 2017 decision.' And I think... that's still true. It's just that the decision turned out to be stickier than even she meant it. It wasn't just a technical choice. It was the first domino in a chain of infrastructure investments that now have a gravity of their own.
Ryan Castillo: And we genuinely don't know if that chain leads somewhere optimal or just somewhere permanent.
Jordan Hale: Yeah. There might be a version of this where we look back at transformer dominance the way we look at other transitional technologies. Not the final answer. Just the thing that was good enough to become load-bearing before anything better had a chance.
Ryan Castillo: Good enough to become permanent. That's a real category of outcome. Appreciate you dragging me through the whole shape of it.
