Clara Bennett: You sent me that inference number at eleven pm and I just sat there.
Max Rivera: Right? I couldn't — I kept re-adding it. Two point three billion dollars. That's what OpenAI spent running their models in 2024. Not building them. Running them.
Clara Bennett: GPT-4 cost somewhere between seventy-eight and a hundred million dollars to train. One time.
Max Rivera: And they spent twenty-three times that just... serving it. In a single year. That's — wait, actually that's what we're getting into today, the whole training-versus-inference thing, why the cost of using these models has completely eclipsed the cost of making them.
Clara Bennett: The key is understanding what kind of cost each one is. Training: fixed, one-time, amortizable across every user who ever touches the model. Inference: marginal, recurring, hits you fresh on every single query. That distinction is what makes the numbers look impossible.
Max Rivera: And it's not like they're charging nothing — OpenAI had six billion in revenue last year. They still lost five billion more than they earned.
Clara Bennett: Which is why inference economics is worth pulling apart carefully, because the Microsoft filings that surface those numbers suggest this isn't a 2024 anomaly — it's where the cost curve lives.
Max Rivera: So the thing I want to figure out — is there actually a way out of this, or is running frontier AI just fundamentally expensive in a way that doesn't get solved by making the model smarter?
Clara Bennett: That way out question is exactly where the analogy helps. Training is like printing a textbook — you pay once to write it, once to print a million copies, and then every student who opens it costs you almost nothing. Inference is hiring a tutor who shows up fresh for every single session. Every query. Every time.
Max Rivera: And you can't — wait, you can't amortize the tutor.
Clara Bennett: Exactly. The textbook's cost gets spread across every student who ever reads it. Google DeepMind spent roughly a hundred and ninety-two million dollars training Gemini Ultra — one time. That fixed cost gets divided across millions of users, billions of queries. The per-query training contribution becomes a rounding error. But every inference call is the tutor showing up. That cost does not divide.
Max Rivera: What does that actually look like in, like, daily burn? Because I've seen the number but it still feels abstract.
Clara Bennett: OpenAI was burning over seven hundred thousand dollars a day just serving ChatGPT. Not building anything new. Not training. Just answering questions.
Max Rivera: Seven hundred thousand. Per day.
Clara Bennett: And NVIDIA has basically put a number on the structural shape of this — their framing is that roughly eighty percent of AI compute goes to inference, twenty percent to training. That's not a prediction. That's the industry describing its own cost distribution right now.
Max Rivera: So the thing we keep calling a one-time cost — the training, the hundred-million-dollar run — that's almost irrelevant to the actual operating reality. The tutor never stops showing up.
Clara Bennett: And the tutor just got cheaper to hire — which sounds like relief, except it isn't. Token prices fell roughly twelve hundred times from 2020 to 2025. A million tokens cost about sixty dollars in 2020. Today it's around five cents. And OpenAI still lost five billion dollars in 2024.
Max Rivera: Wait. Five cents per million tokens and they're still hemorrhaging money.
Clara Bennett: Because the price drop didn't shrink the bill — it grew the usage. Every time inference got cheaper, more people sent more queries. Volume outran the efficiency gain every single time.
Max Rivera: Okay — I mean, I want to make this concrete because I think it's easy to nod at this and not feel it. Say there's a startup founder. Forty thousand active users, each hitting the API maybe three times a day. She's at eighty thousand dollars a month. She reads that prices just dropped forty percent, so she's expecting — what, forty-eight thousand? And then the bill comes in at ninety-two thousand.
Clara Bennett: Because the latency improved when the price dropped, the product felt snappier, users engaged more — and she didn't change a line of her code.
Max Rivera: She's on a treadmill that speeds up every time it gets more comfortable.
Clara Bennett: And the labs are on the same treadmill, just bigger. Anthropic's gross margin in 2024 was negative ninety-four percent. That means before a single operating expense — before salaries, offices, anything — they lost money on every token they sold.
Max Rivera: Negative ninety-four — so why don't they just raise prices? And the answer is Meta and DeepSeek, right? LLaMA is free. DeepSeek is basically free. You can't price above open-weight models when the open-weight models are good enough for most things. The ceiling is gone.
Clara Bennett: That's the structural trap. And what's coming — the optimization toolkit, quantization, speculative decoding, all of it — sounds like the exit ramp. We should get into whether it actually is.
Max Rivera: The exit ramp — okay, so what is actually in the toolkit? Because I've heard quantization thrown around like it's magic and I want to know what it actually does.
Clara Bennett: Start with the most basic one. Every weight in the model has a numerical precision — how many bits you use to store each number. Standard training runs at sixteen-bit floats. Quantization compresses that down to eight-bit or four-bit integers. Smaller numbers, smaller memory footprint, more tokens per second. The production formats are GPTQ, AWQ, GGUF, FP8 — different implementations of the same compression idea.
Max Rivera: But near-lossless is the benchmark claim, right? Like — MMLU, GSM8K, those are controlled tests. What actually breaks?
Clara Bennett: That's the honest limit. Long-context reasoning, retrieval-augmented generation, domain-specific edge cases — those degrade in ways the standard benchmarks won't catch. 'Near-lossless' is a benchmark promise, not a production guarantee. Teams often find they needed the full-precision model after they'd already shipped the compressed one.
Max Rivera: Okay, that's — yeah, that's the thing that doesn't get said.
Clara Bennett: Now, speculative decoding is a different mechanism entirely. A small draft model proposes several tokens at once, and the large model verifies them in parallel instead of generating one token at a time. Latency drops without touching model quality — you're just changing how the work gets scheduled.
Max Rivera: So the big model is basically grading homework instead of writing it from scratch. Wait — is that actually faster even accounting for the draft model's compute?
Clara Bennett: It is, because rejection is cheap and the large model's forward pass covers multiple tokens at once. Then Mixture-of-Experts takes a structural approach — the model isn't fully activated for every token. Only a subset of expert sub-networks fires per token, so you get large effective model capacity at lower active compute. KV cache is where the memory pressure actually lives in production though — it stores intermediate attention computations across a sequence, and it scales directly with context length and batch size. That's usually your first GPU wall.
Max Rivera: And structured pruning — I mean, I've seen unstructured pruning mentioned and it sounds like it should be equivalent but apparently it's not actually useful in practice?
Clara Bennett: Right — zeroing individual weights doesn't map to hardware acceleration. Structured pruning removes whole neurons, attention heads, or layers, which gives the GPU something it can actually skip. That's what vLLM and TensorRT-LLM are packaging together — paged KV cache, quantization, speculative decoding, all integrated. These aren't academic tools. They're the actual production inference stacks. The battleground for profitability isn't the training run — it's what happens inside those frameworks on every single query.
Max Rivera: There's this same wall I keep hitting. The crossover point for self-hosting — five to ten million in annual inference spend — that's supposed to be the exit. Get big enough, pull it in-house, stop paying the API tax. But even if you clear that threshold, even if you've got the H100s and the vLLM stack running, the market below you just shifted. Meta dropped LLaMA, DeepSeek undercut everyone, and suddenly the commodity tier exists whether you wanted it or not. The whole API market stratified into three layers almost by accident — frontier models for hard reasoning, mid-tier for most real workloads, commodity open-weight at the bottom — and I'm not sure any individual company planned that.
Clara Bennett: The restructuring happened because the economics forced it. No one architected three tiers — inference cost reality did. And the genuinely open question, the one I don't have a clean answer for, is whether the next move is smaller models that need less inference to begin with. Smarter prompting, products designed to consume fewer tokens per outcome. Because if efficiency gains just summon more usage every time, and pricing can't rise because open-weight models set the floor, the only real escape is needing fewer tokens to get the same result.
Max Rivera: Nobody knows who figures that out first.
Clara Bennett: No. And that's where I'll actually sit with this — we started with a number that felt impossible. Two point three billion dollars, just running the thing. And the conversation didn't make it less impossible. It just made the shape of why clearer.
Max Rivera: Yeah. It's still — I mean, I re-add it and it still doesn't feel real. Thanks for helping me stare at it properly.
Clara Bennett: Worth staring at.
