Lila Soto: Iris, hey — I've been thinking about electricity all week, weirdly. Like how tiered access to power became invisible infrastructure and nobody named it while it was happening.
Iris Holm: That's a long way round to chips.
Lila Soto: It's not that long, actually. Today we're talking about who gets to run AI on hardware built specifically for AI — and who pays the tax for running it on hardware built for everything. Custom silicon versus general-purpose. And the story starts at Google in 2015.
Iris Holm: The TPU origin.
Lila Soto: Which is — okay, this is the thing I keep wanting people to sit with — it's not a vision story. Google introduces the Tensor Processing Unit in 2015, and the whole program starts because someone did the arithmetic on deploying neural networks at Google Search scale using general-purpose chips, and the number that came back was: you would need an economically prohibitive number of them. That's it. That's the founding myth. A spreadsheet said no.
Iris Holm: Hold on — they weren't ahead of anything, they were behind a cost curve.
Lila Soto: Reactive necessity, not visionary strategy. And Google is still the first hyperscaler to deploy custom AI silicon — which makes it even stranger, because the move that looks boldest in hindsight was just the only affordable exit from a bad number.
Iris Holm: So what does it mean that AWS and Meta and Microsoft all eventually ran the same arithmetic and got the same answer?
Lila Soto: It means the arithmetic is universal — and that's actually the thing that cracked it open for me. Because the question underneath all of it is: why does general-purpose hardware cost so much to run AI on in the first place? Like, what is the CPU even doing with all that energy?
Iris Holm: Okay. Think of it this way. A CPU is like a contractor who owns every tool ever made. The whole truck is packed — drill press, laser level, pipe wrench. Most of that truck is tools he'll never use on this job. You're paying to haul them anyway.
Lila Soto: And the ASIC is the worker who only owns a hammer.
Iris Holm: Because every job is a nail. The transistors that a CPU burns on branch prediction, instruction decoding, cache hierarchies — that's the pipe wrench in the truck. Gone. An ASIC doing the same task can be hundreds to a thousand times more energy-efficient. That's not a rounding error.
Lila Soto: So Google looked at their nail — inference at Search scale — and said, we only need the hammer.
Iris Holm: Exactly that. And the TPU's hammer is — I mean, this is the part that's actually elegant — it's called a systolic array. Data just flows through a grid of multipliers. Rhythmically. No memory traffic jams, no waiting for instructions, just numbers moving through a fixed lattice doing multiply-accumulate over and over. That's it. The whole design.
Lila Soto: Which only works because matrix multiplication is basically the entire job.
Iris Holm: The more precisely you can define the workload, the bigger the efficiency gain. That's the actual principle. And the corollary — the part nobody says out loud — is that NVIDIA's dominance isn't hardware superiority. It's CUDA. The software ecosystem. The chip is good enough. The lock-in is the product.
Lila Soto: And that lock-in is actually what makes the NRE number so brutal — because the door out of NVIDIA costs billions before you walk through it. Like, AWS doesn't build Trainium because they love chip engineering. They build it because at their query volume, paying NVIDIA's margins is the worse option.
Iris Holm: What's the actual number?
Lila Soto: Hundreds of millions to billions — that's the NRE range. Non-recurring engineering. You pay it once, upfront, before a single query runs. And it only becomes rational if you're amortizing it across an enormous, predictable workload. Which is — I mean, that's the whole gate. That's the thing that makes this structural and not just competitive.
Iris Holm: So what does a company do if they're not Google, not AWS, not Microsoft or Meta?
Lila Soto: They stay on NVIDIA GPUs. That's literally the answer. And — okay, imagine a founder, say she's pitching her third infrastructure meeting this quarter, her inference bill just crossed a million dollars a month, and someone asks why she's not on custom silicon. The answer is: because the entry fee is a billion dollars and she doesn't have a billion queries a day to spread it across. NVIDIA is not the best option. It's the only option that's financially coherent.
Iris Holm: Cerebras and Groq would object to that framing.
Lila Soto: Yeah — and they're genuinely interesting attempts. Both are inference-focused ASICs, purpose-built silicon outside the hyperscaler tier. But they face the exact same NRE math against a much smaller customer base. Groq isn't amortizing across billions of Google Search queries. So the efficiency gains are real and the economics are... harder to close.
Iris Holm: The moat replicates at every tier.
Lila Soto: Which is what MIT IDE is actually pointing at — this isn't just chip architecture, it's computing fragmenting into a fast lane and a slow lane. And the CUDA switching cost is what makes the slow lane sticky. We'll get to exactly how sticky in a minute, because the number there is worse than most people assume.
Iris Holm: Sticky is actually underselling it. Picture a team that's been writing CUDA kernels for three years. Not 'they use PyTorch on top of CUDA' — I mean hand-tuned, workload-specific CUDA code. Moving that to custom silicon isn't a migration, it's a rewrite. Months of engineering, minimum.
Lila Soto: And during those months you're still paying the GPU bill.
Iris Holm: Right — the meter doesn't stop while you rebuild the engine. And this is what the efficiency ratio misses entirely. Yes, custom silicon cuts cost-per-token. But the question is: compared to what? The H100 bill, or the H100 bill plus eighteen months of engineer salaries on a rewrite?
Lila Soto: Which is — I mean, that reframe is actually what makes the merchant GPU look rational again. CUDA and ROCm give you portability. You can move between NVIDIA and AMD without burning the software stack. Custom silicon, any of the four families — the hyperscaler XPU, the systolic TPU, an inference ASIC like Groq — each one is a different world. You're not just buying a chip. You're buying into a stack you can't easily leave.
Iris Holm: So the GPU tax is actually buying something real.
Lila Soto: Optionality. Yeah. Which is — huh — kind of a strange thing to pay a premium for, but it's not irrational.
Iris Holm: And this is exactly where the MIT IDE finding lands hard. Because they're not saying specialization wins a product cycle. They're saying Moore's Law slowing down plus deep learning dominating demand is structurally reversing the forces that made computing a general-purpose technology at all. Fast lane, slow lane — not a metaphor, an infrastructure split. Companies with enough scale compound efficiency gains on custom silicon. Everyone else is on legacy general-purpose hardware, permanently paying more per token than Google does.
Lila Soto: And they can't exit. Because the exit costs the thing they don't have — the scale to justify it.
Iris Holm: That's the detail that sticks. It's not 'startups pay more.' It's that the gap is structural and it compounds. Google's TPU efficiency widens every generation. The Series B company's CUDA bill just... stays.
Lila Soto: I keep thinking about the electricity thing — where I started. Because that's actually what this is. The same recommendation query, running on two different infrastructures, at two different costs, and it stays that way. Not because one company is smarter. Because one company could afford to build the hammer, and the other one is still hauling the truck.
Iris Holm: And the hammer is optimized for today's nail. That's the part I can't fully resolve. A chip tuned for transformer training — if the dominant architecture shifts, the silicon doesn't shift with it. The software ecosystem can't follow fast enough. So the thing that makes you 50x more efficient today is a liability the morning the workload changes.
Lila Soto: Yeah. And I don't think either of us knows when that morning is.
Iris Holm: No. That's the honest place to stop, I think. Thanks for walking through it.
