Lila Soto: Hugo, hey — I want to start with a scene, actually, because I think it names the strangeness better than any framing I could offer.
Lila Soto: A researcher in 2012 is running a neural network on a regular CPU. Results take — weeks. She borrows a gaming GPU from a colleague, kind of on a whim, probably felt a little silly about it. Runs the same code. And it's fifty times faster. Not a little faster. The factor of fifty is what changes what's even possible to ask.
Hugo Vance: Mm — fifty times. That scene is doing real historical work. Because what changed wasn't the algorithm. The algorithm was identical. The hardware was the variable.
Lila Soto: And that hardware existed because the video game industry had been pouring money into parallel architecture for thirty years — nobody in that industry was thinking about AI. They wanted better explosions, basically.
Hugo Vance: Well, more precisely — real-time 3D rendering through the nineties and two-thousands. Competitive commercial pressure. That is what funded the architecture.
Lila Soto: And then NVIDIA ships CUDA in 2007 — which is a software platform that lets researchers use GPU hardware for things that have nothing to do with graphics. And it just... waits. The deep learning scaling surge doesn't really arrive until the 2010s.
Hugo Vance: That interval — 2007 to the convolutional network moment, and then to transformers — that is what I find genuinely strange. Infrastructure preceding demand, and then demand arriving and finding the infrastructure already fully formed.
Lila Soto: Which raises the thing we're actually here to figure out — was this luck, or is there something underneath it that made this specific collision almost... structurally inevitable?
Hugo Vance: I'd be cautious about either answer given too quickly.
Lila Soto: And to get at what's underneath that — I keep wanting to ask what the hardware is actually doing when a neural network runs. Like, what is the math it's executing?
Hugo Vance: Yes. And the answer is almost offensively simple. Matrix multiplication. That's it. A neural network's forward pass, its backward pass — they reduce to sequences of large matrix multiplications. Thousands of them.
Lila Soto: Hm. And that's — that's the same operation that's happening when a GPU renders a pixel?
Hugo Vance: Exactly the same. 3D graphics rendering — transforming millions of vertices, coloring millions of pixels per frame — that is matrix and vector operations. Same arithmetic. The GPU does not know it has changed jobs.
Lila Soto: Wait. The hardware literally cannot tell the difference between 'what color is this pixel' and 'train a model to recognize a face'?
Hugo Vance: To the silicon, they are the same problem. Now — the analogy I'd reach for. Imagine painting a wall covered in a million tiny squares. A CPU paints one square, waits, paints the next. A GPU sends ten thousand painters in at once, each with a brush. Every square gets done simultaneously. That's rendering. And it turns out — training a neural network is also a wall of a million independent squares.
Lila Soto: Because each little dot product calculation doesn't need to know what the others are doing.
Hugo Vance: Precisely. This is what researchers call embarrassingly parallel — tasks where the sub-computations are fully independent, no coordination required. Pixel rendering is embarrassingly parallel. Matrix row-column dot products are embarrassingly parallel. You see, the GPU architecture was built for the first, and the second arrived to find — a perfectly shaped hole waiting for it.
Lila Soto: Oh, that's — yeah. That's the click. So it's not that someone was clever about repurposing the hardware. The math was the same math, the whole time.
Hugo Vance: Well — and here is where I'd add one layer. Because that's actually not the whole story, and I think it's worth sitting with. The RTX 4090, to take a concrete example, has sixteen thousand, three hundred and eighty-four cores. A standard CPU has eight to sixteen. That ratio — that's the throughput vs. latency split. CPUs minimize the time to finish one thing. GPUs maximize how many things finish per second. A CPU is a brilliant soloist. A GPU is a very large, very obedient chorus.
Lila Soto: And neural networks needed the chorus.
Hugo Vance: Needed the chorus. And the chorus existed because teenagers wanted smoother explosions in video games through the nineties. That is the actual causal chain. Which means — the fifty-times speed advantage your researcher found in 2012 wasn't luck in the sense of random. It was structural. The fit was built into the operations themselves.
Lila Soto: Though — I mean, knowing the math matched still doesn't explain why NVIDIA specifically, or why this locked in the way it did. That feels like a different layer entirely.
Hugo Vance: Well — and that's exactly where the surface explanation hides something. Because the ratio of cores is only half of it. The deeper question is why a CPU can't just run sixteen thousand simple cores itself. And the answer is — design philosophy at the level of transistors.
Lila Soto: Oh — meaning it's not just a quantity thing.
Hugo Vance: No. A CPU's transistor budget goes toward latency infrastructure — deep caches, branch prediction, out-of-order execution. These are all mechanisms for finishing one complex, branching task as fast as physically possible. That sophistication costs die area. Enormously.
Lila Soto: So each CPU core is kind of — expensive in silicon terms.
Hugo Vance: Exactly. A GPU core is comparatively simple — no branch predictor, shallow cache. It cannot handle branching logic gracefully. But it costs so little silicon that you can fit sixteen thousand of them, as in the RTX 4090, where a CPU fits eight or sixteen. The GPU trades per-core intelligence for sheer count.
Lila Soto: And neural network math never really branches — it just applies the same operation again and again across a huge array of numbers.
Hugo Vance: Which is precisely what the SIMT model formalizes. Single Instruction, Multiple Threads — the same instruction fires across thousands of threads simultaneously, each thread touching a different data element. The GPU is not doing something clever per element. It is doing one thing, in lockstep, at enormous width.
Lila Soto: Hm. And physically — I mean, what does that look like inside the chip? Like, where is SIMT actually happening?
Hugo Vance: Yes — that's the Streaming Multiprocessor. The SM. It's a cluster of cores, registers, shared memory — a self-contained scheduler. One SM handles thousands of threads. An NVIDIA GPU has dozens of SMs, all running simultaneously. So the parallelism isn't just across cores, it's organized in these structured execution units that schedule and dispatch the work.
Lila Soto: So the RTX 4090's sixteen thousand cores aren't just — loose cores floating around, they're organized into these clusters that are each doing coordinated lockstep work.
Hugo Vance: Structured parallelism, yes. And NVIDIA didn't stop there. With the Volta architecture they introduced Tensor Cores. Dedicated silicon units whose only job is matrix multiply-accumulate operations. Not general computation. Not rendering. Literally one mathematical primitive, baked into hardware.
Lila Soto: Wait — they took the thing that was already a structural fit and then made specialized hardware just for that specific operation?
Hugo Vance: Indeed. Which is — I mean, that's not accident anymore. That's a deliberate architectural commitment. The fit stopped being inherited and became designed. And that compounding is, I'd argue, what makes the CUDA software question the harder problem — which we'll get to, because the software lock-in is actually where the real durability lives.
Lila Soto: Yeah — the hardware story is almost the easier half, isn't it. The math matched, they deepened the match. But what I keep wondering is what's underneath the software side of that — like, what does it actually feel like to be AMD trying to catch up to that.
Hugo Vance: Well — AMD's position is actually the clearest demonstration of the whole problem. Their hardware has closed. Genuinely closed. The performance gap on raw silicon is not what it was. And yet — ROCm, their software stack, sits there and nobody is rewriting production codebases for it. Nobody.
Lila Soto: So the moat is inertia?
Hugo Vance: Not just inertia — that's actually the word I'd push back on. Inertia implies things are just... sitting still. What CUDA has is compounding. Every researcher who learns CUDA first writes libraries in CUDA, trains students on CUDA, publishes code in CUDA. The next researcher arrives and finds the entire knowledge infrastructure already there. cuDNN is optimized for neural network primitives specifically — not ported, not approximated, built from scratch for that purpose and iterated for over a decade. A competitor doesn't just need to match today's CUDA. They need to match ten years of accumulated tooling.
Lila Soto: That's — yeah. It's less like a wall and more like a... the wall keeps growing while you're trying to climb it.
Hugo Vance: Yes. And CUDA launched in 2007 — not for AI, not anticipating deep learning. NVIDIA built it so graphics programmers had more flexibility. The lock-in happened twice: once in hardware, when the silicon matched the math, and again in software, when a developer ecosystem formed around that match before anyone understood why it mattered.
Lila Soto: And that second lock-in is the one that's actually hard to break. The hardware you can — I mean, in principle, you can fab better silicon. But you can't just fab a community.
Hugo Vance: Which is where the hardware lottery question gets genuinely uncomfortable. Because if CUDA is the actual moat — not the silicon — then we have to ask: did deep learning win the 2010s because it was the best algorithm, or because it happened to fit the hardware that gaming had already funded?
Lila Soto: Oh. And the alternatives — sparse methods, symbolic reasoning — they didn't fail that test necessarily.
Hugo Vance: That's the part I find genuinely hard to dismiss. Those architectures require fundamentally different hardware profiles — not embarrassingly parallel, not dense matrix multiply. They may have been disadvantaged not because they were theoretically weaker but because the substrate simply wasn't there. The hardware lottery: you win not by being best, but by matching what's already built.
Lila Soto: Which is — kind of a vertiginous idea, actually. Like, what does it mean for the last decade of AI progress if the selection pressure was partly just... what ran fast on CUDA?
Hugo Vance: I'd be cautious about going too far there. But I'd not dismiss it either. What would have to be shown — and nobody has quite shown it — is that the sparse or symbolic alternatives were theoretically sound and computationally implausible on CPUs at the time. If that's true, the CUDA moat didn't just win a race. It picked the racers.
Lila Soto: And then Google TPUs, Cerebras wafer-scale chips — those feel like the first real structural challengers, not just hardware competition but a different bet entirely.
Hugo Vance: Indeed. Purpose-built for neural network primitives rather than inheriting a graphics rendering heritage. That's the contingency argument's strongest evidence — that the GPU's dominance is not inevitable, it's historical. But you see, they still have to dislodge CUDA's compounding. Google can build a better TPU. They cannot easily build a decade of libraries, tooling, and researchers who already know the ecosystem.
Lila Soto: So both things are true, maybe? The moat is structurally durable — and it's also genuinely contingent. Those aren't actually in contradiction.
Hugo Vance: That's the tension I'd leave open. CUDA's compounding is real and hard. But the hardware lottery also tells us that sufficiently large architectural shifts can reset the game — and purpose-built accelerators are exactly that kind of shift. Whether they're big enough is the question we genuinely don't have an answer to yet.
Lila Soto: The thing I'm still sitting with — and I don't know if it resolves — is the inference problem. Because GPUs are actually worse at inference in some cases. The same throughput optimization that makes training fast actively hurts latency. So we built the dominant paradigm around the problem we could see, and now the next problem has a different shape.
Hugo Vance: Yes. And that's where Google's TPUs and Cerebras become genuinely interesting rather than just competitive. They're not trying to out-GPU the GPU — they're asking whether the training-throughput frame is even the right one going forward. A Cerebras wafer-scale chip places the entire chip on a single wafer specifically to eliminate inter-chip communication bottlenecks. That's a different architectural bet at the root, not an incremental one.
Lila Soto: Right — but the part that doesn't fit is that neither of them inherited a graphics-rendering heritage. That's the whole point. They're purpose-built for neural network primitives. Which means... maybe the question isn't whether the GPU era lasts. Maybe it's whether it was always a fifteen-year window optimized for one specific phase of scaling.
Hugo Vance: I think that's the more honest frame, actually. And I'll go this far — if the hardware lottery shaped which algorithms became tractable in the 2010s, it's probably shaping something right now that we're not watching closely enough. Some architectural bet being laid down somewhere, not for AI, for something else entirely. And in fifteen years it fits a problem we don't have a name for yet.
Lila Soto: That's — yeah. That's the uneasy landing, isn't it. We can't see it because we're still inside the GPU era.
Hugo Vance: Quite. And I find I'm comfortable leaving it there — unresolved. It's the right size of uncertain for what it actually is. Thank you for pressing on the inference piece. That's the thread I kept almost dropping.
