Marcus Vale: I'm going to put something to you before we start. The paper that changed everything — was it inevitable, or was it an accident?
Ben Okonkwo: Hm. I'd say — neither? And both. Which I realize is an annoying answer.
Marcus Vale: Classic Ben.
Ben Okonkwo: Look — before 2017, attention mechanisms existed. They were being bolted onto RNNs for machine translation. As supplements. The insight that Vaswani and the Google Brain team had — and Noam Shazeer was on that paper too — was to ask what happens if you remove the recurrence entirely. No hidden state chain. Just attention. Pure parallelization. And the answer, demonstrated at NeurIPS 2017, was: you get better translation scores with significantly less training time than any recurrent model.
Marcus Vale: They ran the old architecture out of the room.
Ben Okonkwo: They did. But here's what I find — I mean, the thing that doesn't get said — is that the fix created its own constraint. Self-attention compares every token to every other token. The math on that is O(n²). Which at 512 tokens, fine. But that's where the 2017 paper actually ran its experiments. 512 tokens.
Marcus Vale: Okay, walk me through the actual mechanism. Like — not abstractly. Someone opens a translation tool Tuesday morning, types 'the bank by the river flooded.' What is the model actually doing with the word 'bank'?
Ben Okonkwo: Right, so — every token gets projected into three vectors. A query, a key, a value. The query is essentially 'what am I looking for?' The key is 'what do I contain?' And 'bank' fires its query against the key of every other token — 'river,' 'flooded,' all of them — simultaneously. The dot products produce attention weights. High weight on 'river' means the value of 'river' flows back into how 'bank' gets represented. That's — I mean, that's the whole trick.
Marcus Vale: Wait — why divide by sqrt of d_k? I've seen that in Jay Alammar's Illustrated Transformer and just — glossed over it.
Ben Okonkwo: Gradient saturation. If your key dimension is, say, 512, the raw dot products get very large — the softmax output sharpens so hard it's basically a one-hot vector, gradients vanish, learning stops. Scaling by sqrt of 512 keeps the distribution soft enough to actually train.
Marcus Vale: And then you run that whole process — what, eight times in parallel? Different subspaces? Multi-head attention is basically 'what if we did that for syntax, semantics, coreference, all at once' and concatenate?
Ben Okonkwo: Exactly. Each head learns different relationship types. But — wait, there's something that should be bothering you. If everything runs in parallel, no sequential steps — how does it even know 'bank' comes before 'river'? Self-attention is permutation-invariant. Shuffle the tokens, you get identical attention weights. The architecture genuinely cannot tell word order without — actually, this is the part Alammar flags as almost an afterthought — they bolt on positional encodings. External signals. It's a patch.
Marcus Vale: A patch. On the architecture that runs GPT-4.
Ben Okonkwo: Now — and this is where I want to slow down, because the 512-token thing is the number that should haunt people. Vaswani's experiments, NeurIPS 2017, 512 tokens. That's the empirical baseline the whole scaling story starts from. Modern context windows hit 128K. Do the math — that's not 512 squared anymore, that's 128,000 squared. Roughly sixteen billion attention operations. Per forward pass.
Marcus Vale: Sixteen billion. Every time someone asks GPT to read a long document.
Ben Okonkwo: Every time. And the scaling laws — Kaplan, Chinchilla, Hoffmann — they show performance improves predictably with model size and data. That's real. OpenAI's entire commercial trajectory, BERT, GPT-series, it all tracks. But — I mean, those laws don't say the architecture is structurally sound. They say throwing scale at it works. Those aren't the same claim.
Marcus Vale: Wait — is that Shazeer's bet too? He co-authored the original paper, then kept compounding the same architectural assumption for years. That's either conviction or — no, actually, what's the alternative read?
Ben Okonkwo: Interesting. The alternative read is what the applied research papers are quietly saying. Transformer-GRU hybrids, KAN-BiLSTM-Transformer combinations — these keep appearing. That's not academic noise. That's practitioners hitting the quadratic wall and routing around it.
Marcus Vale: So the architecture won the benchmark war but the production engineers are already hedging. And nobody in the hype cycle is talking about it.
Marcus Vale: The thing that gets me — Vaswani and the Google Brain team sat down to fix machine translation. That's it. That's the whole mandate. And what they actually shipped was the substrate for GPT, for vision models, for AlphaFold doing protein folding. None of that was in the 2017 paper. That generalization wasn't — I mean, it wasn't engineered. It was discovered after the fact. That's not a feature. That's an accident that happened to be load-bearing.
Ben Okonkwo: Yeah. And Stanford CS 224n students have been working through that exact gap for years now — the course notes walk you through the architecture and the open questions are still open. Same ones. The quadratic cost, what positional encoding is actually doing, whether the scaling laws are telling us something deep or just — hm — something convenient. The pedagogy caught up. The answers didn't.
Marcus Vale: So the architecture solved a problem it didn't know it was solving. And the quadratic cost — maybe that's the same thing in reverse. A constraint we don't fully understand yet.
