Otis Kim: Hey, settle in, this one's gentle. Picture a chef who's absurdly fast, can chop anything instantly, but every ingredient arrives through one narrow pantry door.
Iris Holm: Mm, hello. I'm guessing this isn't a cooking show.
Otis Kim: No, it's a chip. Nvidia's H200 has the same compute die as the H100, same paper compute speed, yet Nvidia claims up to 1.9 times faster Llama 2 70B inference. As far as the reporting suggests, that comes mostly from upgraded memory. I could be wrong, the sourcing's thin. But double the chef's speed and nothing changes. Widen the door and everything does.
Iris Holm: So the expensive chef is mostly idle. What's the question, exactly?
Otis Kim: Whether a chatbot's speed is set by the feed line rather than the calculator. And it's slipperier than it sounds, because the H200 also got more memory capacity, not only a wider door. I don't think we can separate those cleanly.
Iris Holm: Which means the door sets the pace, so let's put numbers on it. The roofline model says throughput is the minimum of two things: peak compute, or bandwidth times arithmetic intensity. Intensity is FLOPs per byte moved. Where does an H100 flip from one to the other?
Otis Kim: Roughly 295 FLOPs per byte. That's the ridge point. Below it, extra TFLOPS add nothing.
Iris Holm: And decoding sits where? Look, it makes one token at a time. Every step re-streams the weight matrices. Matrix-matrix work collapses into matrix-vector. At FP16, batch one, that's about one FLOP per byte.
Otis Kim: One against 295. Yeah, under one percent of the compute does anything. That's the low MFU people mean.
Iris Holm: Under one percent of the headline number. HBM is why the door is even that wide: DRAM dies stacked beside the processor, wide through-silicon vias. The Department of Energy's AMAIS report, February 2026, calls the compute-memory imbalance the central bottleneck. I've only seen a summary though, so hold that loosely.
Otis Kim: Okay, but the H200 doesn't fit that tidy version. Bandwidth goes from 3.35 to 4.8 terabytes per second. That's 1.43 times. Nvidia claims up to 1.9. I mean, 1.43 doesn't turn into 1.9, so that's not actually the whole story.
Iris Holm: Hold on. So the other half comes from capacity? 80 gigabytes to 141... Right. The cache is a different tax from the weights. It gets worse, and we'll get there.
Otis Kim: Yeah. But what is all that extra room actually buying?
Iris Holm: Buying room for the cache. The weights are fixed. The cache grows with every token, and every decoding step reloads all of it from HBM.
Otis Kim: Yeah, and that's where batching misleads. Batch a few hundred sequences and the weights amortize, because one read serves everybody. But every sequence drags its own cache. So 'memory-bound' is really a claim about batch size. At a few hundred it can flip to compute-bound.
Iris Holm: Then the cache number. Llama 2 70B, with grouped-query attention, the thrifty design. Still about 0.3 megabytes per token. At 32k context, roughly 10 gigabytes. Per user.
Otis Kim: Ten gigabytes for one person's chat? Okay, that's worse than I guessed. Picture her spreadsheet: a startup engineer, 70B support bot, weights 140 gigabytes in FP16. Customers paste 32k-token ticket histories. She skips the TFLOPS row. Leftover memory divided by that cache is her sessions, and that sets cost per conversation.
Iris Holm: So the H200 is mostly a bigger parking lot. Then why did the ratio get worse for years?
Otis Kim: It did. About 156 FLOPs per byte on the A100, 295 on the H100. The H200 pulled it back. I mean, is that escaping the wall or repainting it? And across multiple GPUs, NVLink becomes a third ceiling. Groq, Cerebras, Google's TPU all spend their silicon on data movement. I'd hedge the details, but the wall looks industry-wide.
Iris Holm: Now DeepSeek-V3. Trained on 2,048 H800s, published May 2025, designed around limited interconnect and memory. I read that as confirmation. The effort is the evidence. FastKV and the compression work say the same: nobody optimizes bytes moved unless bytes are the problem.
Otis Kim: Or erosion. Speculative decoding has a small draft model propose tokens, and the big model checks them in parallel. That raises arithmetic intensity without new silicon. It bends the ceiling. Doesn't remove it. I could be wrong, though. The sourcing's thin.
Iris Holm: Here's what I'm still turning over. Quantize to FP8, even FP4, and you halve or quarter the bytes. Great. But you've changed regimes, not escaped one, because the ridge point moves with the bytes. And cheaper bytes invite longer contexts. Sixteen thousand tokens becomes 32k, becomes more. The cache eats what the quantization saved. So if every saved byte gets spent, is the wall falling? Or just walking?
Otis Kim: Walking, I think. Yeah. Though I could be wrong, because I haven't seen anyone do that accounting end to end, and the sourcing is thin. What I'd bet on is that her Tuesday spreadsheet looks different in a few years. Maybe HBM bandwidth stays the row that matters. Maybe capacity takes over, or NVLink, once models shard wider. Or something we haven't named yet. And I wonder if she opens a spec sheet at all, or just compares quotes for tokens per dollar.
Iris Holm: Mm. Which is the chef again. Still absurdly fast, still waiting on the pantry door, except now the kitchen keeps ordering bigger meals every time we widen it. I don't have a verdict on that. Oddly, I'm fine with that.