Onpode
Cover art for Why Huang chose flexibility over efficiency — the CUDA philosophy that shaped AI infrastructure

Why Huang chose flexibility over efficiency — the CUDA philosophy that shaped AI infrastructure

July 24, 2026 · 13 min

Juniper Vale & Finn Brooks

Nvidia's CUDA dominance traces to a deliberate 2003 decision to add IEEE 32-bit floating point to GPU shader processors, sacrificing peak efficiency for programmability. That bet — whiteboard over printed poster — compounded over two decades into AI infrastructure lock-in, but specialized chips like Cerebras WSE-2 now run certain workloads 130x faster than optimized CUDA on an Nvidia A100.

In the early 2000s, Jensen Huang steered Nvidia away from fixed-function GPU designs toward a programmable, general-purpose computing architecture that would eventually be formalized as CUDA (Compute Unified Device Architecture). Fixed-function GPUs had their operations hardwired into silicon, offering developers little flexibility; any new algorithm required waiting for a new chip design.

0:0012:38
Get the next episode on Jensen Huang →

Follow it free — new episodes land in your feed.

Or make your own — any topic, in minutes

More Onpode episodes on Jensen Huang →

About this episode

In January 1993, Jensen Huang and two co-founders started a graphics chip company at a Denny's in San Jose. Thirty years later, Nvidia is the third most valuable company in the world — built on a technical philosophy that deliberately traded peak efficiency for programmability. This episode traces how that choice compounded. The hinge is a 2003 hardware decision: Nvidia added IEEE-compatible 32-bit floating point to its shader processors, letting scientific workloads run directly on GPU hardware for the first time. Three years later, CUDA followed. But the origin of CUDA isn't a corporate breakthrough — it's a Stanford PhD student named Ian Buck, running simulations on a gaming GPU because it was the only parallel compute he could afford. The episode sits honestly with the survivorship bias problem. AMD and Intel saw the same academic results. Nvidia moved. Whether that's vision or execution — or just faster reading — is genuinely contested here. And then there's the ceiling. A Cerebras WSE-2 runs a Monte Carlo particle transport kernel 130 times faster than an optimized CUDA implementation on Nvidia's A100. At that gap, 'we trade some efficiency for flexibility' stops being a satisfying answer. The moat was built when AI problems were wide and changing fast. The question the episode leaves open: does it hold when the problems have stopped changing that fast? Nobody rings a bell when that shift happens.

Frequently asked

Why did Nvidia choose programmability over efficiency in its GPU architecture?

Nvidia bet that researchers needed hardware they could reprogram weekly — a 'whiteboard' — more than a faster fixed-function chip optimized for one job. The 2003 addition of IEEE 32-bit floating point to Nvidia's shader processors let scientific workloads run directly on GPUs, without disguising math problems as graphics tasks.

Who invented CUDA and where did the idea come from?

CUDA was conceived around 2004 by Ian Buck and John Nickolls within Nvidia's orbit. Buck had earlier built Brook, a stream-computing framework, as a Stanford PhD student demonstrating that GPUs could handle non-graphics workloads. Nvidia recognized that academic work and built a commercial platform around it — the idea was not invented inside Nvidia's R&D.

How much faster is Cerebras WSE-2 than Nvidia A100 on AI workloads?

The Cerebras WSE-2 Wafer-Scale Engine ran a Monte Carlo particle transport kernel approximately 130 times faster than a highly optimized CUDA implementation on Nvidia's A100 GPU. That gap reflects the architectural cost of CUDA's programmability: a general-purpose platform loses badly to purpose-built silicon on narrow, mature, repeatable workloads.

Why can't AMD or Intel displace Nvidia CUDA despite competitive GPU hardware?

AMD and Intel face CUDA's software moat, not just its hardware. Twenty years of scientific and AI code is written in CUDA. Competitors have responded with transpilation — converting CUDA code to run on their chips — an approach that itself signals the lock-in is real and that building better hardware alone is insufficient to break Nvidia's ecosystem hold.

Did Jensen Huang predict CUDA's role in AI, or was Nvidia's dominance accidental?

Nvidia's CUDA dominance looks inevitable in hindsight but was not predicted even by major beneficiaries. Nvidia nearly went bankrupt multiple times before AlexNet's 2012 ImageNet win validated the programmability bet. The core 2003 architectural decision preceded any clear AI application by nearly a decade, and Stanford's Ian Buck, not Nvidia, first demonstrated the underlying insight.

Grounded in 9 sources
Efficient Algorithms for Monte Carlo Particle Transport on AI Accelerator Hardware ↗ · doi.org
HPCTransCompile: An AI Compiler Generated Dataset for High-Performance CUDA Transpilation and LLM Preliminary Exploration ↗ · doi.org
Benchmarking and Dissecting the Nvidia Hopper GPU Architecture ↗ · doi.org
How Jensen Huang’s Nvidia Is Powering the A.I. Revolution | The New Yorker ↗ · newyorker.com
GPU Wars: Nvidia, AMD, and Intel Competing in 2025 ↗ · medium.com
The Next Wave of AI Infrastructure Must Target NVIDIA's CUDA Moat ↗ · builtin.com
Why Nvidia Dominates AI: A History of CUDA and Parallel Computing ↗ · crvscience.com
The CUDA moat - why NVIDIA's software ecosystem defends the hardware monopoly long after AMD catches up · QuantAbundancia ↗ · quantabundancia.com
Nvidia: An Overnight Success Story 30 Years in the Making | Sequoia Capital ↗ · sequoiacap.com
Read transcript

Finn Brooks: Juniper, hey — okay, I have been in a weird headspace all week and I think today might explain why.

Juniper Vale: Weird how? Good weird or staring-at-the-ceiling weird?

Finn Brooks: Wait, no, I keep doing this — like, I've been thinking about my laptop, right, this thing that cost me two thousand dollars, and it has a chip in it that is genuinely worse at math than it could be, on purpose, and that was a deliberate design choice, and the company that made that choice is now the third most valuable company in the world. Like. What?

Juniper Vale: That is exactly where we're going today, yeah.

Finn Brooks: Because that should not work! Making your hardware worse — deliberately trading peak efficiency for programmability — that is the thing Jensen Huang and Nvidia did, and it sounds insane until you watch it compound over twenty years into the dominant AI infrastructure on the planet.

Juniper Vale: And the decision is even older than people realize. This traces back to a Denny's in San Jose, January 1993, where Huang and his two co-founders, Chris Malachowsky and Curtis Priem, started a graphics chip company. Consumer graphics. That's it. That's the whole plan at the time.

Finn Brooks: A Denny's. I mean — of course it was a Denny's.

Juniper Vale: And was steering Nvidia toward programmable, general-purpose architecture in the early 2000s actual foresight, or was it a bet that happened to land? Because if AI had moved literally any other direction, we'd be telling a completely different story about that company.

Finn Brooks: Right — and that is the survivorship bias problem, which, I want to sit with that for a second, because Nvidia almost went under. Multiple times. While waiting for the world to catch up to the bet they'd already placed.

Juniper Vale: So the puzzle isn't just 'how did they win.' It's — was this brilliant strategy, or the luckiest near-miss in semiconductor history?

Finn Brooks: Okay but — that luckiest near-miss framing breaks down a little, right, because there were actual technical decisions along the way that weren't accidents. Like the 2003 thing — do you want to explain what they actually changed in the hardware? Because I think that's the moment people skip over.

Juniper Vale: Yeah, let me try to make this land simply first. Think of it like this — a programmable GPU is a blank whiteboard. A researcher can rewrite it every week, different problem, different approach. A fixed-function chip is a printed poster. Faster to read, honestly, but you can never change what's on it. That's the whole trade.

Finn Brooks: Whiteboard versus printed poster. Okay I actually love that.

Juniper Vale: And the 2003 move is where Nvidia commits to the whiteboard — they add IEEE-compatible 32-bit floating point to their shader processors. FP32. Which sounds technical, but what it means is: scientific workloads could now just... run on this hardware. You didn't have to disguise your math problem as a graphics task anymore.

Finn Brooks: Wait — disguise it as a graphics task? That was a real thing people were doing?

Juniper Vale: That was genuinely how it worked before. If you wanted GPU compute power you had to trick the chip into thinking your physics simulation was, I don't know, shading a polygon. The 2003 decision ended that workaround. Nvidia chose not to freeze the hardware.

Finn Brooks: And then three years later — CUDA. 2006, 2007, alongside the GeForce 8800 GTX. Like, that's not — they didn't just flip a switch. That FP32 decision in 2003 is basically the foundation they built CUDA on top of.

Juniper Vale: Right, right. And suddenly any scientist or developer could access parallel processing without needing to speak graphics at all. That's when it opens up.

Finn Brooks: But — and this is the part I keep getting stuck on — the whiteboard is slower. That's not just a metaphor, that's a real cost. A fixed-function chip built for one job will beat a programmable GPU on that job. So Nvidia was shipping hardware that was, on paper, measurably less efficient than the alternative. And they knew that.

Juniper Vale: They absolutely knew that. I mean — that's the part that makes the decision interesting rather than obvious. You're not trading efficiency for flexibility by accident. You're choosing it. And the bet is that researchers would rather have a whiteboard they can rewrite on Tuesday and ship something Wednesday than wait for a perfect printed poster.

Finn Brooks: Iteration speed over peak optimization.

Juniper Vale: That's the actual bet. Not that programmable GPUs were faster — they weren't, not always — but that being able to change the whiteboard faster than anyone else could print a new poster would compound. And you know, the wild thing is they had to hold that bet for years, losing on benchmarks the whole time, before AlexNet even existed to prove them right.

Finn Brooks: And that's the part that kind of unravels the 'Jensen Huang genius' story — because where did CUDA actually come from? Like, not the PR version.

Juniper Vale: Ian Buck. Stanford. Early 2000s.

Finn Brooks: Right — so there's this PhD student, Ian Buck, at Stanford, and he builds this thing called Brook. A stream-computing framework. And what he's basically proving is that GPUs can crunch non-graphics workloads if you just — I mean, if you hand them the right kind of problem in the right format. That is not a corporate R&D team. That is one guy in a lab.

Juniper Vale: And the user he was building Brook for — think of it like this — it's a grad student at two in the morning running fluid-dynamics simulations on a gaming GPU. Not because it's the ideal hardware. Because it's the only parallel hardware she can actually afford.

Finn Brooks: On a GeForce. At 2 a.m. That is the whole origin.

Juniper Vale: That's exactly who Nvidia inherited when they scaled this. Buck moves into Nvidia's orbit, he and John Nickolls are the ones credited with conceiving CUDA around 2004 — and the insight they're codifying was already alive in that Stanford lab. Nvidia didn't invent it from nothing. They recognized it and built the platform around it.

Finn Brooks: Okay wait — so AMD saw those same Stanford papers. Intel saw them. These are published, open academic results. And they just... didn't move?

Juniper Vale: That's the uncomfortable part, yeah. The information wasn't secret. So at what point do we call Huang a visionary versus someone who just executed faster on something academia had already handed to the industry?

Finn Brooks: Because we don't call it visionary when — wait, actually I want to say this carefully — we don't hand someone credit for seeing the future if the future was already written down and published by a Stanford PhD student. That's not vision, that's reading speed.

Juniper Vale: I mean, maybe. Or the execution IS the vision? Like, knowing something is true and actually betting the company on it are two very different things.

Finn Brooks: No, that's fair — that's actually fair. AMD and Intel knew. They chose not to move. That gap between knowing and doing is real. It just feels different from 'one genius saw what no one else could see.'

Juniper Vale: And that distinction matters even more when you get to 2012 — AlexNet, the ImageNet competition — which is sort of where all of this gets locked in, and where a completely different question opens up about whether the moat this built actually has a ceiling. We'll get there.

Finn Brooks: And 2012 is exactly where the lock snaps shut — like, AlexNet walks into the ImageNet competition using CUDA-enabled Nvidia GPUs and just wins. Not barely. Wins in a way that everyone in that room understood meant something had changed permanently.

Juniper Vale: That's the hinge. A decade of quiet architectural work — the FP32 decision, Brook, CUDA — and then one competition result makes it destiny instead of a bet.

Finn Brooks: But the same logic that made CUDA irreplaceable in 2012 has a ceiling, and we can actually measure it now.

Juniper Vale: The Cerebras number.

Finn Brooks: One hundred and thirty times faster. The Cerebras WSE-2 — the Wafer-Scale Engine 2 — running a Monte Carlo particle transport kernel, compared to a highly optimized CUDA version on an Nvidia A100. Not a sloppy comparison, not apples to oranges. Optimized CUDA on their best chip. And the WSE-2 is still 130x faster on that workload.

Juniper Vale: Okay, I want to push on how we frame that — because 130x is not 'we trade some efficiency for flexibility.' That framing completely breaks at that number. That is 'we are dramatically slower on workloads that actually matter.'

Finn Brooks: Right, that's — wait, that's the thing nobody wants to say out loud.

Juniper Vale: And AMD and Intel — they have competitive hardware, genuinely. They haven't broken through. Not because their chips are bad, but because the code is CUDA. The ecosystem isn't the hardware anymore, it's the twenty years of software built on top of it. That's what they're actually fighting.

Finn Brooks: Which is why the competitive response has shifted — I mean, it's not 'build a better GPU.' It's transpilation. Converting CUDA code to run on other hardware entirely. You're not trying to beat the moat, you're trying to dissolve it from underneath.

Juniper Vale: That move tells you something. If you're spending resources converting someone else's code ecosystem to run on your chip — that's an admission that the lock-in is real, and that fighting the hardware alone doesn't work.

Finn Brooks: So at 130x, does developer inertia actually hold? Like, at what gap does convenience lose?

Juniper Vale: I think it depends on how narrow the workload gets. The whiteboard beats the poster when you need flexibility. But if your workload is mature, specific, repeatable — particle transport, certain simulation types — the poster wins by so much that the whiteboard argument just collapses.

Finn Brooks: And AI workloads are narrowing. Like, the more the field matures, the more the problems standardize, and the more that 130x actually represents what's at stake every single day.

Juniper Vale: Which is the real open question — the moat was built when the problems were wide and changing fast. Does it hold when the problems have stopped changing that fast? Nobody actually knows yet.

Finn Brooks: And that's the thing I keep — no, wait, I want to say this differently. The bet that built the moat and the thing that might eventually limit it are the same decision. The programmability bet, the whiteboard over the poster, that's what gave Nvidia twenty years of runway. And it's also exactly why a WSE-2 can be 130x faster on a single mature workload and Nvidia can't just... patch that.

Juniper Vale: They can't specialize without becoming something different.

Finn Brooks: Right. And the other part that I genuinely can't shake — Andrew Ng, who probably understood the AI revolution more clearly than almost anyone, is cited as a key figure in contextualizing what Nvidia's ecosystem actually enabled. And even he didn't predict the specific shape of CUDA's dominance. Nobody did. Not even the people who benefited most from it.

Juniper Vale: That's actually — yeah. That's the part that makes me uneasy about calling it foresight. The company nearly collapsed multiple times before 2012. The strategy only looks inevitable from where we're standing now.

Finn Brooks: It was obviously right in retrospect. That's not the same thing.

Juniper Vale: I think where I land — and I'm not sure this is comfortable — is that we're probably not watching the end of Nvidia's dominance right now. But we might be watching the moment where 'the platform everyone uses' starts shading into 'the platform everyone is stuck using despite better alternatives.' And the thing is, those feel identical from the inside. Until they don't.

Finn Brooks: Until they don't. Yeah.

Juniper Vale: And nobody rings a bell when that shift happens.

Finn Brooks: I mean — that's the honest place to stop, isn't it. I don't have a verdict. I just have the question sitting there.

Juniper Vale: That's enough. Good one to sit with.