Felix Ortiz: Tess, hey — you ready for this one? Because I have been sitting with a number and I need to hand it to you.
Tess Hollis: Hand it over.
Felix Ortiz: Okay — for every n tokens you feed into a transformer, it computes n² pairwise attention scores. Every single pair, simultaneously. That's the number. React.
Tess Hollis: So it scales with the square. That's not a rounding error — that's baked into the skeleton of the thing.
Felix Ortiz: Right — and here's the part that keeps catching me. That exact same operation, the all-pairs computation, is also the reason the architecture is powerful in the first place. It's not like the cost is a side effect. The cost IS the capability. You can't — I mean, you literally cannot separate them.
Tess Hollis: Wait — same math, two names. One name is parallelism, the other name is quadratic bottleneck.
Felix Ortiz: That is the whole episode, yeah. And it goes back to 2017 — Vaswani and the team at Google publish 'Attention Is All You Need,' they just... throw out recurrent networks entirely. No LSTMs, no sequential hidden state passing, just pure self-attention across the full sequence. And the thing that made that possible — GPUs and TPUs already optimized for matrix multiplication — also meant the n² cost could actually run. The hardware and the architecture kind of chose each other.
Tess Hollis: So the question is whether that was genius or whether it was convenient.
Felix Ortiz: Oh — okay, yeah, that's the sharper version of what I was trying to say.
Tess Hollis: Because those are genuinely different claims.
Felix Ortiz: Okay but — convenient is maybe underselling it. Like, imagine you're reading a sentence and instead of going word by word and trying to hold earlier words in memory, you could just see every word in relation to every other word, all at once. That's the whole thing. That's self-attention. And the reason it could even exist is that GPU matrix math was already built for exactly that shape of computation.
Tess Hollis: That's the click. Every token, direct line to every other token — no waiting, no forgetting.
Felix Ortiz: Right — and compare that to what LSTMs were doing. They were literally passing a hidden state forward one token at a time. Vaswani's team in 2017 just said, why are we doing this sequentially at all? And they dropped recurrence entirely.
Tess Hollis: Wait — LSTMs were already an improvement on the original vanishing gradient problem, right? That was the whole reason GRUs and LSTMs existed. And they still hit the wall.
Felix Ortiz: Yeah, exactly — they patched the gradient problem but kept the sequential bottleneck. You still couldn't parallelize. And that's the part that killed them on hardware, not the math per se — just the shape of the computation didn't fit what GPUs wanted to do.
Tess Hollis: So the story isn't 'transformers are smarter.' It's 'transformers look like matrix multiplication, and we'd already spent years making matrix multiplication absurdly fast.'
Felix Ortiz: Which — okay, and here's where I think about the actual people in the room. Vaswani, Shazeer, Parmar, Uszkoreit, Polosukhin, Jones, Gomez, Kaiser — all Google, one paper, 2017. That roster didn't set out to build the architecture that would run everything. They were solving a translation task.
Tess Hollis: And Google already had TPUs.
Felix Ortiz: Right — so the hardware advantage was already inside the building when they wrote the paper. That's not — I mean, I'm not saying it's rigged, but the alignment between the architecture and the available silicon was not an accident of the universe. It was an accident of institutional proximity.
Tess Hollis: The scaled dot-product piece — they divide the raw query-key scores by the square root of the key dimension before softmax. That detail is doing real work: it stops the scores from getting so large that gradients die. That's not convenient, that's precise.
Felix Ortiz: Oh — yeah, no, that's — wait, that's actually the part I want to sit with. Because that move, the √d_k scaling, is genuinely elegant. But it also — it doesn't change the shape of the computation. It stabilizes it. The n² cost is still there. So you get stability AND the full hardware alignment, and now suddenly GPT-3 scales to 175 billion parameters because the whole thing just... runs.
Tess Hollis: And that's where 'genius or convenient' stops being a clean question. Because by the time you're at 175 billion parameters, it doesn't matter anymore which one it was.
Felix Ortiz: And that's the trap, right — because once you're at 175B, Google and OpenAI and Meta are all pouring resources in, Google is literally building TPUs specifically to run this shape of computation, and now the n² cost isn't something you fix. It's what the whole infrastructure assumes.
Tess Hollis: It becomes load-bearing.
Felix Ortiz: Exactly. And then — okay, so they extend it with multi-head attention. Instead of one attention function, you run several in parallel, each with different learned projections. So you're catching different relational patterns simultaneously. It's richer.
Tess Hollis: Right — but that's the elegant trap. Multi-head attention makes the representations richer without touching the n² core at all. The cost is still there, you've just bought more signal inside it.
Felix Ortiz: Which is — yeah, and that's where I want to get concrete, because there's a scenario that makes this really uncomfortable. Picture a researcher trying to handle long medical documents — full patient history, needs the whole context. They build a sparse attention variant. Saves real compute. But then — wait, actually this is the part that breaks it — they can't verify whether the long-range dependencies are still there. Because the mechanism they pruned is the all-pairs operation. That's the thing that was reading across the whole document.
Tess Hollis: So they solved the cost and broke the thing the cost was buying.
Felix Ortiz: And you can't tell from the outside whether it's broken. The model still produces output.
Tess Hollis: That's what no one names. Sparse attention isn't a cheaper transformer — it's a different tradeoff wearing the same name.
Felix Ortiz: Which — okay, and this is the part that gets me when you look at BERT, vision transformers, molecular generation, time-series forecasting. The architecture is everywhere. Every domain.
Tess Hollis: Wait — does that breadth prove the architecture is right, or just that the infrastructure got so entrenched nobody can easily build something else?
Felix Ortiz: That's — yeah, I think it's both, and that's the uncomfortable answer. BERT alone spans language understanding, medical imaging, molecular generation. T5 at 11B, GPT-3 at 175B, Meta's models in the same range — the scaling just kept working because the hardware kept meeting it.
Tess Hollis: And the field named the bottleneck — n² is the bottleneck — but nobody's willing to say out loud that every fix trades away the core property. That's the thing people are stepping around.
Felix Ortiz: Right, and — there's a layer of this we haven't touched yet that makes all of it weirder. The in-context learning piece, the fact that GPT-3 just... did task-switching nobody trained it for — nobody actually knows if that's tethered to all-pairs attention or just correlated with scale. And the analog hardware story makes it even stranger. That one I want to get to.
Tess Hollis: Oh — hold on. Say more about the analog piece when we get there, because I think that's where the 'substrate-independent math' story falls apart.
Felix Ortiz: And that's exactly the door into the weird part — because in-context learning, the thing GPT-3 just... did, task-switching, chain-of-thought, none of that was trained in. OpenAI built a big parallel model and those behaviors showed up afterward. Nobody at OpenAI put 'emergent task-switching' in the spec.
Tess Hollis: It was discovered, not engineered.
Felix Ortiz: Right — and the sources describe it phenomenologically. Like, 'this emerged from scale and parallelism.' But nobody actually derives it. Nobody shows you the mechanism where dot products between 175 billion parameters produce novel task generalization. It's — I mean, we watched it work and then agreed to move on.
Tess Hollis: Which means — okay, here's the buried thing. If we don't know why all-pairs attention produces in-context learning, we genuinely cannot say whether sparse attention breaks it. Those are different questions and the field is treating them like one.
Felix Ortiz: Oh — wait. That's — yeah. Because if it's tethered to the all-pairs computation specifically, pruning that kills the capability. But if it's just correlated with scale, then sparse variants might be totally fine and we've been worried about the wrong variable.
Tess Hollis: And nobody knows which one it is. That's not a dodge — that's the actual state of the field right now.
Felix Ortiz: Observed, not derived.
Tess Hollis: And then the analog hardware thing lands on top of that and it gets — actually, no, you tell me the fact first, because I want to react to the version I haven't already assembled in my head.
Felix Ortiz: Okay — so there are gain-cell memory chips, analog in-memory implementations of self-attention. Efficient, novel substrate, genuinely interesting hardware. And they cannot run pre-trained digital models directly. The hardware noise — the physical non-idealities of the analog medium — breaks the fidelity. You have to retrain from scratch on that substrate.
Tess Hollis: Wait — so the efficiency gain creates a new training cost.
Felix Ortiz: Every efficiency win on the substrate side erases the model you already built. The architecture that felt like pure math — turns out the math doesn't transfer cleanly across physical matter.
Tess Hollis: And that breaks something fundamental about the durability claim. Because the claim was always that the architecture is substrate-independent — it's just linear algebra, runs anywhere. But if the physical non-idealities of analog hardware mean GPT-3's weights are useless on those chips, then the 'pure math' framing was hiding what the math actually depends on.
Felix Ortiz: So you've got — okay, stack it: in-context learning, observed but not derived; relationship to all-pairs attention, unknown; and now the architecture that supposedly runs everywhere actually doesn't transfer across substrates without retraining. The durability story is built on empirical correlation the whole way down.
Tess Hollis: Not derivation. Correlation. And we keep calling it durability because it's worked — but 'it worked' and 'we understand why it worked' are sitting very far apart from each other right now.
Felix Ortiz: We built something — same query-key-value substrate, language, vision, time-series, molecular generation, medical imaging — it's everywhere. And we genuinely don't know if that's because the architecture found something true about how structure works, or because Google and OpenAI and Meta built so much around it that the ground shifted and now there's no other ground.
Tess Hollis: And you can't answer that from the inside.
Felix Ortiz: Right — you'd need to understand the mechanism. Which loops back to in-context learning. If we actually knew why all-pairs attention produces that, we could say something real about durability. But we don't have that. We have correlation at 175 billion parameters and a lot of very confident infrastructure spending.
Tess Hollis: And if in-context learning turns out to be genuinely tethered to the full n² operation — not just correlated with scale but structurally dependent on every token reaching every other token simultaneously — then sparse attention doesn't preserve the core. It cuts it. And the quadratic cost can't be solved without losing the thing. Which means we're not actually waiting for a better transformer.
Felix Ortiz: We're waiting for something we don't have a name for yet.
Tess Hollis: And we don't know what it looks like. That's — I mean, that's an honest place to sit. Vaswani's team solved translation in 2017 and accidentally handed the field its next decade of infrastructure. The analog substrate breaks the weights. The mechanism behind emergence is unresolved. The efficiency fixes might be trading away the capability they're trying to preserve. None of that resolves cleanly.
Felix Ortiz: No. It doesn't. And I think that's actually — yeah, that's where I'm landing. Not uneasy exactly, just. Sitting with it.
Tess Hollis: Good place to stop. Thanks for the n² number at the top — I needed something to hold.
