Jonathan Ingles: I want to start with a confession — I read the wrong thing into this story for about two hours before I figured out I was completely backwards.
Maya Chen: Oh good, tell me — what did you get wrong first?
Jonathan Ingles: I saw that per-token inference costs have dropped roughly a thousand times in three years and I thought — great, AI is getting cheap. Bill comes down, the economics clean up, normal business.
Maya Chen: Mm — and total AI spending over that same period went up three hundred and twenty percent.
Jonathan Ingles: Which is the opposite of what I expected, yeah. How?
Maya Chen: Okay, so — the core idea, just the plain version before anything else: cheaper tokens don't reduce bills, they generate more demand, and more demand means higher total cost. That's it. That's the whole inversion.
Jonathan Ingles: Give me the analogy that makes it stick.
Maya Chen: You build a factory — hundred million dollars, one time, done. But every unit the factory produces still costs money to make. Now you engineer a way to make each unit way cheaper. Do your costs go down? No, because... now you can afford to flood the factory with orders. Volume explodes. Total spend climbs. The efficiency gain doesn't save you, it accelerates you.
Jonathan Ingles: And the factory never stops — it's not a one-time build cost you pay off and move on.
Maya Chen: That's exactly where AI breaks the analogy in the worst way. OpenAI spent roughly two point three billion dollars on inference compute in 2024 alone. The training cost for GPT-4.5 — the one-time build — is estimated at around a hundred and fifty million. So the running costs are fifteen times the construction cost. In a single year. And per token is still getting cheaper, which means demand will keep climbing.
Jonathan Ingles: So the treadmill speeds up as the ticket gets cheaper.
Maya Chen: Yeah. And that's what we're actually trying to figure out today — whether there's any way off it, or whether this is just... the structure now.
Jonathan Ingles: But that framing — the treadmill — it only makes sense once you understand what software used to cost to run. Which is basically nothing.
Maya Chen: Right, and that's the part that's so easy to miss. Traditional software — you ship it, someone downloads it, the marginal cost of that download is... rounding error. Near zero. Deploy to a million users, your servers barely blink.
Jonathan Ingles: Which is why every tech IPO pitch from 2010 to 2020 was basically 'high fixed development cost, then scale infinitely for free.'
Maya Chen: Exactly — and AI just... snapped that assumption in half. Because every single query costs real money. Not rounding-error money. Compute, memory, electricity, latency infrastructure — all of it, per token, every time.
Jonathan Ingles: Okay but give me a concrete number. Not the treadmill — an actual person doing an actual thing.
Maya Chen: Okay, so — think about a graduate student, late night, running a legal research tool powered by GPT-4. She submits one long query, maybe a three-thousand token context window. That single request consumes real GPU cycles on OpenAI's infrastructure. Now multiply her by, I don't know, millions of similar users, and — wait, actually the number that makes it visceral is the training comparison. Stanford AI Index puts GPT-4 training above a hundred million dollars. Google's Gemini Ultra 1.0 training is estimated around a hundred and ninety-two million. That's the one-time build. The thing you cannot un-spend.
Jonathan Ingles: A hundred and ninety-two million. For one model.
Maya Chen: One model. And once that's done — it's done. The cost is sunk. You can't recover it by training again, you recover it by running the model. Every query that graduate student submits is a tiny installment on a hundred-and-ninety-two-million-dollar debt.
Jonathan Ingles: So the model becomes an asset — like a factory you built. But you still have to pay to run every shift.
Maya Chen: Yes — and here's where the factory analogy actually breaks down in the ugliest way. A factory has slow shifts, maintenance windows, off hours. The AI factory runs every time any user anywhere sends a message. There's no overnight shutdown. OpenAI's two-point-three billion in 2024 inference spend reflects that — it's not episodic, it's continuous. The fixed cost is sunk, but the running cost never stops accumulating.
Jonathan Ingles: And only Google and OpenAI are currently operating factories that size. That's the other part — the capital gate isn't just high, it's high enough that essentially two organizations can clear it.
Maya Chen: Which means the amortization math — spreading that sunk cost across as many queries as possible — isn't a strategy these companies chose. It's the only arithmetic that makes the numbers work at all.
Jonathan Ingles: But that's the part I keep tripping on — if the only arithmetic that works requires maximum scale, what actually happens when you drop the price?
Maya Chen: That's exactly the trap. Okay — picture a hospital administrator, Thursday afternoon, running patient-record summarizations through Anthropic's Claude API. Each query: cheap. Maybe a few cents. She does four hundred of them.
Jonathan Ingles: Four hundred in one afternoon.
Maya Chen: Right — and she only runs four hundred because per-token pricing made it affordable. The moment Anthropic cuts that price in half, her hospital's IT team doesn't think 'great, we saved money.' They think 'we can now afford eight hundred.' And then every administrator in the health system gets access. And now you've got thousands of staff running hundreds of queries each, and the total inference bill for that one health system explodes — even though the per-token cost is, you know, half what it was.
Jonathan Ingles: So efficiency progress is actually a margin destroyer.
Maya Chen: Yes — because the price drop removes the signal that was rationing demand. That's the Jevons dynamic. The cheaper the token, the more tokens get consumed. And the costs scale by input volume, output volume, context-window length — a summarization query with a long patient record is not one token, it's thousands. Multiply that by the query count and you see why OpenAI landed at two point three billion in inference spend for 2024 alone.
Jonathan Ingles: And token-based pricing — the way OpenAI and Anthropic both charge per input and output token — that makes the cost structure visible to the hospital, but it doesn't fix the math.
Maya Chen: It just — yeah, it surfaces the variable cost. It doesn't change what it costs to serve the request. The meter is visible. The meter is still running.
Jonathan Ingles: So per-token pricing is basically honest accounting, not a solution.
Maya Chen: Exactly. And what it reveals is — mm, wait, the number that gets me every time — per-token inference costs dropped roughly a thousand times over three years. A thousand. And total AI spending grew three hundred and twenty percent over the same window. Those two facts should be opposites. They're not. That's the inversion working in real time.
Jonathan Ingles: A thousand times cheaper, bill goes up. Frankly that should disturb more people than it does.
Maya Chen: And the part that comes next — why companies structurally cannot opt out of this, and what DeepSeek-V3 and Meta's open-weight moves actually do to the whole model — it's genuinely ugly.
Jonathan Ingles: And ugly is the right word — because the thing nobody says cleanly is that 'democratizing AI' and 'acquiring users to survive' are the same action. You cannot tell them apart from the outside.
Maya Chen: Oh — say that slower, because that's... yeah, that's the one that lands hard.
Jonathan Ingles: OpenAI needs volume to amortize a hundred and fifty million dollar training sunk cost. The mission says: get AI to everyone. The survival math says: get AI to everyone. Those are identical instructions. Which motive is real?
Maya Chen: And if you can't tell — structurally, from the outside — then the mission framing isn't dishonest exactly, it's just... it doesn't add information. It describes the same behavior the financial pressure already required.
Jonathan Ingles: Right — but the part that doesn't fit is what happens when someone breaks the cost structure from underneath. Because if someone can train a frontier model for five million instead of a hundred and fifty, the amortization math on the incumbent's sunk cost becomes... look, it becomes unrecoverable.
Jonathan Ingles: Give me the actual numbers.
Maya Chen: Wait, the architecture is the part that matters first. DeepSeek-V3 was trained on 2,048 NVIDIA H800 GPUs. Not ten thousand. Not a cluster that costs half a billion to assemble. Two thousand and forty-eight. And the reason that's possible is — mm, how to put this — they didn't just throw less compute at the same model shape. They redesigned what the model does per token. Multi-head Latent Attention compresses the key-value cache, so memory bandwidth stops being the wall. Mixture of Experts means for any given token, only a fraction of the model's parameters actually activate. You're not running the whole thing every time.
Jonathan Ingles: Hold on — only a subset activates? So the model is larger on paper than it is in practice, per query.
Maya Chen: Exactly — that's the MoE mechanism. Overall capacity stays high, but computation per token drops because most experts sit idle for any given input. And then FP8 mixed-precision means you're doing math at lower numerical resolution where it doesn't hurt output quality. The whole architecture is designed around what the H800 hardware can do efficiently. That's hardware-aware co-design — the attention mechanism, the expert routing, the precision format, all of it coordinated with the actual physical constraints of the chip.
Jonathan Ingles: And that makes the training cost somewhere between five and fifteen million. Against Google's Gemini Ultra at a hundred and ninety-two million.
Maya Chen: Which means — if a startup, or a government, or a research lab can replicate that architectural approach... the incumbent's two-point-three billion annual inference spend is funding a moat that someone just tunneled under. Because if users redirect to a cheaper capable alternative, that sunk training cost is just... gone. There's no recovering it.
Jonathan Ingles: And Meta is doing a version of this from a completely different angle — open-weight models, plus entering compute resale on pre-repricing GPU fleets. They're commoditizing the exact inference layer that OpenAI and Anthropic need to stay margin-positive on.
Maya Chen: Because Meta's incentive isn't to sell inference — it's to make sure nobody else can charge a premium for it. Different incentive, same structural effect: batching, quantization, distillation into smaller models, specialized inference chips — all of that optimization work that providers are doing to control per-query cost? Meta's making the floor of that market collapse.
Jonathan Ingles: And now GLM-5.2 is in the 2026-2030 scenario analyses as a frontier-capable open-weight model. Meaning capable inference, no incumbent provider, no per-token bill to OpenAI or Anthropic. The training cost just... never gets recovered.
Maya Chen: That's the one that keeps me up a little, honestly. Because — wait, let me think about what it actually means structurally. If you can run a frontier-capable model without paying anyone for inference, the amortization imperative just... it doesn't get to complete itself. The sunk cost stays sunk. And then you layer in the near-Shannon-limit KV-cache compression work, the lightweight runtimes — if those efficiency gains land in the next eighteen months, the 'inference dominates forever' thesis might just be a description of right now, not a permanent condition.
Jonathan Ingles: xAI and Meta entering compute resale is the early signal, isn't it. They're not waiting for the thesis to collapse — they're restructuring the inference market now, on pre-memory-repricing GPU fleets.
Maya Chen: Right — and the DRAM and HBM price projections for 2026 through 2030 add another layer, because if memory costs surge and China has its own domestic HBM supply chains, the cost curves just decouple. OpenAI is facing one price reality, a player with domestic memory access faces a completely different one. Solvency risk isn't evenly distributed.
Jonathan Ingles: So the next eighteen months is basically — the bet either pays or it doesn't. And what actually resolves it is whether customers keep paying. Not GPU capacity. Willingness to pay.
Maya Chen: Yeah. That's the scarcity that actually matters. Not compute — customer willingness to pay, when a capable alternative might be free.
Jonathan Ingles: Which brings me back to where I started — I read a thousand times cheaper and thought: bill comes down, economics clean up. I had it completely backwards. The cheaper the token, the bigger the trap. And now I'm not even sure the trap holds, because the open-weight floor might remove the reason to pay at all.
Maya Chen: It's a strange place to land, honestly. Two point three billion in annual inference spend — fifteen times the training cost, in a single year — and the honest answer to 'was that a genius bet or a very expensive mistake' is just... we don't know yet.
Jonathan Ingles: Eighteen months will answer it. Or at least make the answer undeniable.
Maya Chen: Good conversation. I mean that — genuinely didn't expect to land on 'willingness to pay' as the whole story, but here we are.
