Onpode
Cover art for Why raw processing power matters less than moving data — the architecture constraint that persists

Why raw processing power matters less than moving data — the architecture constraint that persists

August 3, 2026 · 13 min

Eliza Ward & Brian Reed

GPU memory bandwidth, not raw compute power, is the binding constraint in LLM inference. An NVIDIA H100 delivers 1,000 trillion FLOPS but only 3.35 TB/s of memory bandwidth, producing a measured floor of 42 milliseconds per output token on high-bandwidth hardware — a structural limit William Wulf and Sally McKee formally described in 1995.

Modern processors — CPUs, GPUs, and TPUs — can execute arithmetic operations orders of magnitude faster than they can retrieve data from main memory. This gap, formally identified as the "memory wall" by William Wulf and Sally McKee in 1995, arises because processor performance (FLOPS) has historically scaled much faster than DRAM latency and bandwidth, which improve only a few percent per year.

0:0013:10
Get the next episode on Technology →

Follow it free — new episodes land in your feed.

Or make your own — any topic, in minutes

More Onpode episodes on Technology →

About this episode

An ML engineer installs a newer, more expensive GPU and watches it deliver the same sluggish output as the chip it replaced. That's not a setup — it's a diagnostic. The H100 can execute a thousand trillion floating-point operations per second. It spent most of Tuesday morning waiting for data. The binding constraint isn't compute. It's how fast memory can move data to the processor, and that number lags by orders of magnitude. This episode traces why. William Wulf and Sally McKee named the problem formally in 1995 — the memory wall — and the gap they described has only widened. A 2025 study measuring LLM inference latency found a hard floor of 42 milliseconds per output token, set purely by DRAM bandwidth saturation. That floor was measured on high-bandwidth HBM hardware. The ceiling moved; the floor held. The episode works through what the industry has actually built in response — HBM stacking, CXL memory pools, KV-cache optimization — and is honest about what each of those does and doesn't solve. Then it turns to where the architecture may eventually need to go: Processing-In-Memory, which eliminates data movement rather than accelerating it, and why the hardware-software stack required to get there doesn't exist at production scale yet. If you've ever approved a GPU purchase based on FLOPS, this episode is about what that number isn't telling you.

Frequently asked

Why doesn't upgrading to a faster GPU speed up LLM token generation?

LLM inference is memory-bandwidth-bound, not compute-bound. An NVIDIA H100 delivers roughly 1,000 trillion FLOPS but only 3.35 terabytes per second of memory bandwidth. The processor sits idle waiting for data, so swapping in a more powerful chip produces little improvement until bandwidth — not arithmetic throughput — increases.

What is the memory wall in AI and machine learning?

The memory wall is the growing gap between processor speed and memory bandwidth. Researchers William Wulf and Sally McKee formally named it in a 1995 paper, observing that CPU performance scales exponentially while DRAM bandwidth improves only incrementally. Thirty years later, this structural mismatch still sets the performance ceiling for LLM inference.

What is the fastest possible token generation speed on an H100 GPU?

A 2025 study measured a hard lower-bound floor of 42 milliseconds per output token on high-bandwidth GPUs, caused purely by DRAM bandwidth saturation. This floor was measured on HBM hardware — including H100-class GPUs with 3 TB/s bandwidth — meaning faster arithmetic units cannot reduce it further.

Does High Bandwidth Memory (HBM) solve the memory wall problem for AI inference?

HBM narrows the memory wall but does not close it. Stacking DRAM dies vertically via through-silicon vias raises GPU bandwidth from roughly 50–100 GB/s on standard DRAM to about 3 TB/s on an H100, yet compute capacity still far exceeds that bandwidth. The measured 42ms-per-token floor was recorded on HBM hardware.

What is Processing-In-Memory (PIM) and could it fix the GPU memory bottleneck?

Processing-In-Memory places compute logic inside or directly adjacent to DRAM, eliminating most of the data movement that causes bandwidth bottlenecks. Researcher Onur Mutlu at ETH Zurich has demonstrated significant performance and energy gains from PIM designs. Hardware like CORSAIR chiplets is approaching production, but no PIM solution currently operates at datacenter scale.

Grounded in 12 sources
Memory-Centric Computing: Recent Advances in Processing-in-DRAM ↗ · arxiv.org
AI and Memory Wall ↗ · arxiv.org
PIM-Opt: Demystifying Distributed Optimization Algorithms on a Real-World Processing-In-Memory System ↗ · arxiv.org
Amplifying Effective CXL Memory Bandwidth for LLM Inference via Transparent Near-Data Processing ↗ · arxiv.org
KV Cache Optimization Strategies for Scalable and ... ↗ · arxiv.org
Corsair: An In-Memory Computing Chiplet Architecture for Inference-Time Compute Acceleration ↗ · doi.org
Reconfigurable Computing and FPGAs for Adaptive AI Workloads ↗ · doi.org
PRISM: Photonic Reconfigurable In-Situ Memory for Next-Gen AI Workloads ↗ · doi.org
PIM or CXL-PIM? Understanding Architectural Trade-offs Through Large-Scale Benchmarking ↗ · doi.org
Mind the Memory Gap: Unveiling GPU Bottlenecks in Large-Batch LLM Inference ↗ · doi.org
Profiling Concurrent Vision Inference Workloads on NVIDIA Jetson ↗ · doi.org
Qualcomm's HBC Stacks Compute Beneath DRAM To ... ↗ · wccftech.com
Read transcript

Brian Reed: Long week for you, or — actually, you know what, I'm skipping that, I have a scene I need to set.

Eliza Ward: It was a long week, but fine, set the scene.

Brian Reed: Tuesday morning. There's an ML engineer — real job, real pressure — and they've just swapped a newer GPU into their inference stack for a customer-support language model. More expensive chip, should be faster. They hit run. And the output is... the same. Same slow crawl of tokens. One. Word. At. A. Time.

Eliza Ward: Hold on — same as what, exactly? Same as the old chip?

Brian Reed: Same as the old chip. The new chip is an NVIDIA H100 — it can do a thousand trillion floating-point operations per second. And it spent, let me see if I can say this right — roughly 99% of that Tuesday morning doing nothing. Not because it's bad hardware. Because the data pipeline feeding it moves at 3.35 trillion bytes per second, and the chip is so fast that it just... waits.

Eliza Ward: That's a staggering mismatch. A thousand trillion versus 3.35 trillion.

Brian Reed: Yeah — and the best analogy I have is a chef who can cook a thousand dishes a minute, but the kitchen door only fits one ingredient through at a time. You can hire a faster chef all you want. The door is still the door.

Eliza Ward: Okay, that actually lands. So the constraint isn't compute at all.

Brian Reed: Right — it's memory bandwidth. And a 2025 IEEE Cloud Computing study put a hard number on it: 42 milliseconds per output token, a measured lower-bound floor, coming purely from DRAM bandwidth saturation. Not a theoretical guess. Measured. That's the wall this engineer hit, and that's what this episode is actually about.

Eliza Ward: So when we say — wait, let me get this straight — they can't engineer around that 42 milliseconds. That's physics.

Brian Reed: That's physics. Which is maybe the thing that should have been on the spec sheet.

Eliza Ward: Oh — that's how it works. Yeah. Okay.

Brian Reed: And the part that stops me cold — this isn't a 2025 discovery. William Wulf and Sally McKee named it. Formally, in a paper, in 1995. They called it the memory wall.

Eliza Ward: Wait — 1995?

Brian Reed: 1995. They looked at the trajectory and said: processor speed is scaling exponentially, DRAM latency and bandwidth are improving incrementally. Those lines diverge. That's not a temporary lag — it's structural. And then thirty years went by.

Eliza Ward: So the engineer on Tuesday morning — they hit a wall that Wulf and McKee saw coming in 1995. That's not a hardware failure. That's, I mean — that's the architecture itself.

Brian Reed: The Von Neumann bottleneck, specifically. Data has to travel from memory across a bus to the processor. The bus grows slowly. The processor grows fast. That gap is baked into how we build computers.

Eliza Ward: Right — and HBM, the stacked memory on the A100 and H100, that's the industry's answer to this. Short, wide buses, dies stacked vertically. Gets you to 1–3 terabytes per second. Which sounds enormous until you put it next to the FLOPS number.

Brian Reed: And a standard high-end CPU isn't even close — 50 to 100 gigabytes per second. So HBM is a real improvement, but the compute capacity still laps it. The wall moved, it didn't disappear.

Eliza Ward: The wall follows you. Which — actually, that's the thing about NVIDIA Jetson. Edge device, smaller scale, but the profiling data shows GPU cores running at 15 to 30% capacity under heavy inference load. Heavy load. Still sitting idle most of the time.

Brian Reed: 15 to 30. On a device designed for edge inference.

Eliza Ward: Yeah. Because moving data the last mile — power budget, latency — you can't feed the chip fast enough. Same constraint, different zip code.

Brian Reed: The engineering community knows this is real — they built a whole formal tool around it. The Roofline model. You plot arithmetic intensity against hardware limits to figure out whether your workload is memory-bound or compute-bound. The fact that tool exists is, I mean, it's an admission that the distinction isn't obvious. You can't just eyeball it.

Eliza Ward: You need a formal model to see a thirty-year-old wall. That's the part that doesn't sit right with me.

Brian Reed: But the industry didn't just shrug at that wall — they built something real. HBM. You stack the DRAM dies vertically, connect them through-silicon vias, thousands of tiny short paths instead of one long bus. The A100 gets you to about a terabyte per second. The H100 pushes that to three.

Eliza Ward: And it works?

Brian Reed: Yes and no. The bandwidth is real — 3 TB/s on an H100 is genuinely impressive engineering. But compute is still underutilized. The gap narrows. It doesn't close. That 2025 study, the 42-millisecond floor — that was measured *on* high-bandwidth GPUs. On HBM hardware.

Eliza Ward: Hold on — so 3 TB/s and you're still hitting the floor.

Brian Reed: Still hitting the floor. You moved the ceiling, the arithmetic intensity of LLM inference just... follows you up. And there's a cost side to this that I don't think gets priced in enough — HBM dramatically increases the cost and power draw per GPU. So you've traded one problem, bandwidth, for two others.

Eliza Ward: That's — yeah, that's a real trade. More expensive hardware, higher energy bill, and you're still underutilized. Someone running an inference cluster is paying for all three of those at once.

Brian Reed: Picture a data center operator — say it's a Friday afternoon, they're reviewing power consumption reports for their H100 rack. The utilization numbers look fine on the FLOPS dashboard. But the watt-hours per token are brutal, and I mean — wait, actually the framing that matters here is: they bought bandwidth, and the bill is in energy.

Eliza Ward: What does CXL change? Because that's the other piece the industry is pushing.

Brian Reed: CXL — Compute Express Link — lets you attach large off-package memory pools to a processor with better bandwidth than standard DRAM interfaces. You're extending capacity without redesigning the chip. It's useful for inference workloads where the model is too big to fit in GPU memory.

Eliza Ward: So it's additive — more headroom, not a different architecture. And Qualcomm's variant, the HBC thing, is just another iteration of the same stacking logic applied underneath the compute die.

Brian Reed: Right — different geometry, same physics. You're still moving data to the processor. You're just shortening the trip.

Eliza Ward: HBM, CXL, HBC — all of it is buying time inside the same constraint that Wulf and McKee described. The ceiling isn't lifted. The position the industry holds is that this is the near-term answer, and I think that's probably right as a description. I'm less sure it's right as a solution.

Brian Reed: And there's a more fundamental direction — one that doesn't widen the bus but actually moves compute to where the data lives. That's where researchers like Onur Mutlu at ETH Zurich are pointing, and honestly the implications are bigger. We'll get there.

Eliza Ward: Right — but moving the city is harder than widening the highway. That's the honest version of the PIM pitch. Processing-In-Memory puts the computation logic inside or right next to the DRAM itself. Data barely moves at all. You're not speeding up the bus — you're eliminating the trip.

Brian Reed: Who's actually building that?

Eliza Ward: Onur Mutlu at ETH Zurich is the name that keeps coming up. His research group has run studies showing significant performance and energy gains — not marginal, significant — just from reducing how far data travels. The logic is almost insultingly simple once you see it: the movement itself is the cost.

Brian Reed: And there's hardware past the research stage?

Eliza Ward: CORSAIR is the closest — chiplet-based, combines compute and memory in the same package. It's cited as near-memory acceleration moving toward production. But, I mean — toward production isn't the same as in production. That distinction matters here.

Brian Reed: Right — and then there's PRISM.

Eliza Ward: Yeah. PRISM — Photonic Reconfigurable In-Situ Memory — is the frontier version. Uses photonic compute-in-memory to sidestep electronic bandwidth limits entirely. Not widen them. Sidestep. That's a different category of claim.

Brian Reed: Zero production citation uptake. I looked at that number and — hang on, what does that actually mean in practice? Like, nobody's running this in a real cluster anywhere?

Eliza Ward: Not at datacenter scale. The gap between prototype and production is unquantified — that's not me being vague, that's the honest state of the research. PRISM showed promise in a lab. That's confirmed. That it ships and scales? That's a bet.

Brian Reed: So PIM is more fundamental than HBM or CXL — eliminates the movement rather than just accelerating it — but the hardware-software stack it needs doesn't exist yet at scale. You'd have to rethink how memory is addressed, how compilers work, how models are even written.

Eliza Ward: That's exactly the tension. Imagine a chip architect — it's maybe two in the morning, she's staring at a CORSAIR spec and a procurement deadline. The physics point clearly toward PIM. Her software team would need six months minimum to rearchitect the stack. She buys the H100s. Not because PIM is wrong. Because 'theoretically compelling' isn't a 2025 procurement decision.

Brian Reed: So what does she actually do in the meantime — KV-cache optimization, smarter batching — that's the pragmatic middle ground people are using right now?

Eliza Ward: Yeah. KV-cache compression, eviction strategies, batching more requests together to squeeze utilization. Real gains. But they're working around the wall, not through it. The physics that Wulf and McKee named in 1995 is still the physics. PIM is probably where the architecture wants to go. The market just — it isn't there yet.

Brian Reed: And I mean this isn't a new thought at this point — it's her. The engineer from Tuesday morning. She didn't miss something. She bought what the spec sheet said was the right number. FLOPS. That's the metric vendors measure, that's what the benchmark showed, that's what the purchase order said. And none of that was wrong, exactly. It was just — it was the wrong question dressed up as the right answer.

Eliza Ward: And the physics isn't moving to meet her. That's the part that doesn't resolve. Signal propagation across copper has a ceiling. HBM pushed that ceiling — genuinely, 3 TB/s on an H100 is real — but the 42 millisecond floor was measured on that hardware. The ceiling moved, the floor stayed.

Brian Reed: Right — and somewhere right now, someone is approving a purchase order. Not in a hypothetical, like — today. The number on the page is FLOPS. The constraint is bandwidth. Those two facts are not touching each other in that room.

Eliza Ward: William Wulf and Sally McKee saw that room coming in 1995.

Brian Reed: They did. And I don't — I mean, I'm not saying the industry ignored the problem. HBM exists. CXL exists. Onur Mutlu's group is building real things. But the market communication piece, the part where the number on the spec sheet actually reflects the binding constraint? That hasn't caught up. And until it does, the purchasing decisions don't change.

Eliza Ward: That's the uncomfortable landing spot, yeah. It's not an engineering failure. It's a — the incentive structure just never got realigned to bandwidth.

Brian Reed: She hit run on Tuesday morning expecting faster tokens. She got the same crawl. And she probably thought it was a configuration problem.

Eliza Ward: It wasn't a configuration problem.

Brian Reed: No. It was 1995. Thanks for thinking through this one with me — I needed the pushback on the PIM stuff.