Onpode
Cover art for The memory wall: how data movement, not TFLOPS, governs modern GPU design

The memory wall: how data movement, not TFLOPS, governs modern GPU design

November 21, 2026 · 7 min

Iris Holm & Otis Kim

At batch size one, an H100 GPU operates at roughly 1 FLOP per byte moved versus its ridge point of 295 — meaning under 1% of advertised compute does useful work during LLM token generation. Memory bandwidth, not TFLOPS, sets the speed limit for large-language-model inference.

Large language models (LLMs) and transformer architectures impose workload characteristics that differ fundamentally from classical high-performance computing. Traditional compute benchmarks prioritize floating-point operations per second (TFLOPS), but transformer inference—particularly autoregressive decoding—is dominated by repeated access to large weight matrices with limited data reuse.

0:006:51
Get the next episode on Nvidia →

Follow it free — new episodes land in your feed.

Or make your own — any topic, in minutes

More Onpode episodes on Nvidia →

About this episode

Everyone talks about TFLOPS. This episode argues that number is almost beside the point — at least during the phase of LLM inference that most people actually pay for. When a model generates tokens one at a time, the arithmetic collapses from matrix-matrix multiplication down to matrix-vector. At batch size one, that's roughly one FLOP per byte of memory moved. The H100's compute-to-bandwidth ratio sits around 295 FLOPs per byte. The math means well under 1% of advertised compute is doing anything useful. The episode works through why — using the roofline model as a guide — and then follows the numbers somewhere concrete: a startup engineer pricing out a 70B support bot discovers that at 32k-token contexts, the KV cache alone runs about 10 GB per user. Her spreadsheet never opens the TFLOPS row. The H200's story turns out to be more complicated than 'faster chip' — the jump in memory capacity matters as much as the wider bandwidth pipe, and the two are hard to disentangle. The episode also asks whether the fixes people reach for (quantization, speculative decoding, smarter batching) are escaping the wall or just redrawing it. No clean verdict, but the framing is sharp enough to change how you read a spec sheet.

Frequently asked

Why is LLM inference memory-bound instead of compute-bound?

During LLM token decoding, the GPU generates one token at a time, collapsing matrix-matrix operations into matrix-vector operations. At batch size one on an H100, arithmetic intensity falls to roughly 1 FLOP per byte moved — against the chip's roofline ridge point of 295 FLOPs per byte — leaving over 99% of peak compute idle.

What is the roofline model and how does it apply to GPU inference?

The roofline model defines throughput as the minimum of peak compute or memory bandwidth multiplied by arithmetic intensity (FLOPs per byte). For LLM inference on an H100, the ridge point is roughly 295 FLOPs per byte. Single-sequence decoding sits far below that, so adding more TFLOPS provides no benefit.

Why is the H200 faster than the H100 for LLM inference if the compute die is the same?

The H200 uses the same compute die as the H100 but upgrades memory: bandwidth rises from 3.35 to 4.8 TB/s and HBM capacity from 80 to 141 GB. Nvidia claims up to 1.9× faster Llama 2 70B inference; the bandwidth gain alone is roughly 1.43×, with extra capacity reducing KV-cache eviction pressure making up the difference.

How large is the KV cache for a long-context LLM conversation?

Llama 2 70B using grouped-query attention requires approximately 0.3 MB per token of KV cache. At 32,000-token context, that is roughly 10 GB per user session. Because every decoding step reloads the full KV cache from HBM, cache size directly caps how many concurrent sessions a GPU can serve.

Does quantization (FP8 or FP4) solve the GPU memory wall for LLM inference?

Quantization to FP8 or FP4 halves or quarters bytes moved, shifting the roofline ridge point, but does not escape the memory-bound regime — it relocates the boundary. In practice, saved memory tends to be consumed by longer contexts: 16k-token conversations become 32k, so the KV cache grows to absorb the gains.

Grounded in 12 sources
Speculative Decoding: Exploiting Speculative Execution for Accelerating Seq2seq Generation ↗ · arxiv.org
FastKV: Decoupling of Context Reduction and KV Cache Compression for Prefill-Decoding Acceleration ↗ · arxiv.org
AI Hardware Accelerators for Large Language Models: Architectures and the Memory Wall ↗ · arxiv.org
Hybrid Spark: Adaptive Speculative Decoding That Lets the Content Choose the Drafter ↗ · arxiv.org
PolyKV: A Shared Asymmetrically-Compressed KV Cache Pool for Multi-Agent LLM Inference ↗ · arxiv.org
Insights into DeepSeek-V3: Scaling Challenges and Reflections on Hardware for AI Architectures ↗ · arxiv.org
Dalorex: A Data-Local Program Execution and Architecture for Memory-bound Applications ↗ · arxiv.org
Evaluating MFU as a Proxy for GPU Power for Energy-Aware Simulation of LLM Training ↗ · arxiv.org
Prefill vs. Decode Bottlenecks: SRAM-Frequency Tradeoffs and the Memory-Bandwidth Ceiling ↗ · arxiv.org
BridgeKV: Large-Model Prefill and Small-Model Decoding for LLM Inference via KV Cache Translation ↗ · arxiv.org
AMAIS - Advanced Memory to Support AI for Science: Report February 2026 ↗ · doi.org
Hardware Acceleration for Neural Networks: A Comprehensive Survey ↗ · doi.org
Read transcript

Otis Kim: Hey, settle in, this one's gentle. Picture a chef who's absurdly fast, can chop anything instantly, but every ingredient arrives through one narrow pantry door.

Iris Holm: Mm, hello. I'm guessing this isn't a cooking show.

Otis Kim: No, it's a chip. Nvidia's H200 has the same compute die as the H100, same paper compute speed, yet Nvidia claims up to 1.9 times faster Llama 2 70B inference. As far as the reporting suggests, that comes mostly from upgraded memory. I could be wrong, the sourcing's thin. But double the chef's speed and nothing changes. Widen the door and everything does.

Iris Holm: So the expensive chef is mostly idle. What's the question, exactly?

Otis Kim: Whether a chatbot's speed is set by the feed line rather than the calculator. And it's slipperier than it sounds, because the H200 also got more memory capacity, not only a wider door. I don't think we can separate those cleanly.

Iris Holm: Which means the door sets the pace, so let's put numbers on it. The roofline model says throughput is the minimum of two things: peak compute, or bandwidth times arithmetic intensity. Intensity is FLOPs per byte moved. Where does an H100 flip from one to the other?

Otis Kim: Roughly 295 FLOPs per byte. That's the ridge point. Below it, extra TFLOPS add nothing.

Iris Holm: And decoding sits where? Look, it makes one token at a time. Every step re-streams the weight matrices. Matrix-matrix work collapses into matrix-vector. At FP16, batch one, that's about one FLOP per byte.

Otis Kim: One against 295. Yeah, under one percent of the compute does anything. That's the low MFU people mean.

Iris Holm: Under one percent of the headline number. HBM is why the door is even that wide: DRAM dies stacked beside the processor, wide through-silicon vias. The Department of Energy's AMAIS report, February 2026, calls the compute-memory imbalance the central bottleneck. I've only seen a summary though, so hold that loosely.

Otis Kim: Okay, but the H200 doesn't fit that tidy version. Bandwidth goes from 3.35 to 4.8 terabytes per second. That's 1.43 times. Nvidia claims up to 1.9. I mean, 1.43 doesn't turn into 1.9, so that's not actually the whole story.

Iris Holm: Hold on. So the other half comes from capacity? 80 gigabytes to 141... Right. The cache is a different tax from the weights. It gets worse, and we'll get there.

Otis Kim: Yeah. But what is all that extra room actually buying?

Iris Holm: Buying room for the cache. The weights are fixed. The cache grows with every token, and every decoding step reloads all of it from HBM.

Otis Kim: Yeah, and that's where batching misleads. Batch a few hundred sequences and the weights amortize, because one read serves everybody. But every sequence drags its own cache. So 'memory-bound' is really a claim about batch size. At a few hundred it can flip to compute-bound.

Iris Holm: Then the cache number. Llama 2 70B, with grouped-query attention, the thrifty design. Still about 0.3 megabytes per token. At 32k context, roughly 10 gigabytes. Per user.

Otis Kim: Ten gigabytes for one person's chat? Okay, that's worse than I guessed. Picture her spreadsheet: a startup engineer, 70B support bot, weights 140 gigabytes in FP16. Customers paste 32k-token ticket histories. She skips the TFLOPS row. Leftover memory divided by that cache is her sessions, and that sets cost per conversation.

Iris Holm: So the H200 is mostly a bigger parking lot. Then why did the ratio get worse for years?

Otis Kim: It did. About 156 FLOPs per byte on the A100, 295 on the H100. The H200 pulled it back. I mean, is that escaping the wall or repainting it? And across multiple GPUs, NVLink becomes a third ceiling. Groq, Cerebras, Google's TPU all spend their silicon on data movement. I'd hedge the details, but the wall looks industry-wide.

Iris Holm: Now DeepSeek-V3. Trained on 2,048 H800s, published May 2025, designed around limited interconnect and memory. I read that as confirmation. The effort is the evidence. FastKV and the compression work say the same: nobody optimizes bytes moved unless bytes are the problem.

Otis Kim: Or erosion. Speculative decoding has a small draft model propose tokens, and the big model checks them in parallel. That raises arithmetic intensity without new silicon. It bends the ceiling. Doesn't remove it. I could be wrong, though. The sourcing's thin.

Iris Holm: Here's what I'm still turning over. Quantize to FP8, even FP4, and you halve or quarter the bytes. Great. But you've changed regimes, not escaped one, because the ridge point moves with the bytes. And cheaper bytes invite longer contexts. Sixteen thousand tokens becomes 32k, becomes more. The cache eats what the quantization saved. So if every saved byte gets spent, is the wall falling? Or just walking?

Otis Kim: Walking, I think. Yeah. Though I could be wrong, because I haven't seen anyone do that accounting end to end, and the sourcing is thin. What I'd bet on is that her Tuesday spreadsheet looks different in a few years. Maybe HBM bandwidth stays the row that matters. Maybe capacity takes over, or NVLink, once models shard wider. Or something we haven't named yet. And I wonder if she opens a spec sheet at all, or just compares quotes for tokens per dollar.

Iris Holm: Mm. Which is the chef again. Still absurdly fast, still waiting on the pantry door, except now the kitchen keeps ordering bigger meals every time we widen it. I don't have a verdict on that. Oddly, I'm fine with that.