Ben Okonkwo: Marcus, hey — before we do anything, I want to try something: I'm going to name a year and a place, and I want your honest first instinct.
Ben Okonkwo: Google Brain, 2017.
Marcus Vale: Vaswani, Shazeer, 'Attention Is All You Need' — that's the whole game right there. Every LLM that matters runs on that architecture.
Ben Okonkwo: Right — GPT series, BERT, Meta Llama, the whole stack. Now here's what I actually want to get into, because I think the instinct you just had — 'that's the whole game' — is load-bearing in a way that might not survive scrutiny. The paper replaced RNNs and LSTMs, which processed tokens strictly one at a time and had genuinely crippling vanishing gradient issues. Transformers fixed that. But the mechanism that fixed it, self-attention, scales as O of n-squared. Every token attends to every other token. And Vaswani and Shazeer... I mean, they could see that. It's in the math.
Marcus Vale: Okay, so — if it's such a known ceiling, why does literally every major lab still ship on transformer foundations in 2024? That's the thing I don't buy as a clean story.
Ben Okonkwo: Right — but the reason they still ship on transformers is actually the answer to why RNNs died. Okay, picture a film editor working a twelve-hour cut. She needs to remember that the actor who appears in frame three is the same one who reappears in frame nine hundred. An RNN has to pass that fact through every single frame in between. Each step, the signal degrades a little — that's the vanishing gradient problem, not a metaphor, an actual mathematical shrinkage of the error signal as you backpropagate through hundreds of time steps. By the time you're trying to correct the model on that connection, the gradient is basically zero. The model literally cannot learn the relationship.
Marcus Vale: Oh — so the old architecture was a game of telephone. The message just... degrades with distance.
Ben Okonkwo: Exactly that. And self-attention cuts the telephone line entirely — the ending just looks directly back at frame three. Every token attends to every other token simultaneously, so the gradient has a direct path, not a chain of a thousand steps. That's the geometric fix Vaswani and the Google Brain team actually shipped in 2017.
Marcus Vale: But parallel processing destroys word order. You can't just read all tokens at once and pretend sequence doesn't exist.
Ben Okonkwo: Right, and that's — okay, this is actually the clever patch. Positional encodings get added directly to each token's embedding before anything else happens. The model knows where in the sequence every token sits, even though it's processing all of them at once. And then — now this is the part I find genuinely interesting — they don't run just one attention operation. Multi-head attention runs several in parallel, each one learning to track a different kind of relationship. One head might lock onto syntax, another onto coreference. They concatenate the outputs.
Marcus Vale: So multiple lenses on the same sequence at the same time. What's the grounding — is there empirical work showing the heads actually specialize, or is that just the theoretical framing?
Ben Okonkwo: There's interpretability work suggesting it, yeah — I'd call it suggestive rather than settled. But here's what's not in question: this architecture is why GPT, BERT, Llama — all of it — got built at scale at all. The parallelizability is what made GPUs actually useful for training. That's the part we haven't touched yet, and honestly it's where the quadratic cost starts biting back in ways that make the whole foundation look a lot less permanent.
Marcus Vale: The GPU angle is actually what made the whole thing fundable. Think about it — a machine learning engineer, Thursday night, she queues a training run on a 128,000-token document. Transformer kicks in, all tokens hit the hardware simultaneously, GPU utilization spikes to basically 100 percent. That's it. That's the commercial unlock. RNNs couldn't do that — sequential dependency meant the GPU sat idle waiting for the previous token to finish. You were paying for hardware you couldn't use.
Ben Okonkwo: And that's why scaling laws became testable at all.
Marcus Vale: Exactly — once you can peg the GPU at full utilization, you can actually run the experiment. More parameters, more data, more compute, performance goes up predictably. That's what made GPT, BERT, Llama, Claude possible. Not the math alone. The hardware fit.
Ben Okonkwo: Okay but that same engineer — she's watching inference cost now, right? Longer context window, costs explode. That's the O-n-squared wall.
Marcus Vale: Yeah. And here's where I'll actually hedge — because the question is whether that's structural or just an engineering constraint someone optimizes away. Mamba and the hybrid RNN variants suggest it's structural. You can't attend to every token against every other token at a million tokens without burning real power on every forward pass. That's not accidental.
Ben Okonkwo: So what experiment would have to fail for you to update that? Like — if sparse attention variants close the gap at 128k context, does that change the claim?
Marcus Vale: Frankly — yes. If sparse or gated attention gets within, say, five percent performance on long-context benchmarks without structural modification to the core transformer block, I update. But we're not there. The research says we're patching, not fixing.
Ben Okonkwo: What actually settled for me is that Mamba-Transformer hybrids are in production now. Not a paper. Shipped. And they use gated recurrence, which is structurally close to what transformers replaced in 2015. That's not a theoretical challenge to the architecture. That's the market already hedging.
Marcus Vale: Yeah — and I think that's the honest landing. Transformer got there first. Crossed the threshold first. Parallelizability, GPU fit, scaling laws — all of it. But 'first past the threshold' and 'optimal for the next threshold' are different claims, and the hybrid architectures are basically the market's way of saying it knows the difference.
Ben Okonkwo: And we genuinely can't say which way it lands. I'm comfortable with that.
Marcus Vale: Frankly, an open question with shipped evidence is better than a closed one with vibes. Good talk.
