Hana Field: Hey, you make it through the week okay — you looked a little haunted last time we talked.
Iris Holm: Surviving. I have something for you, though — a fact I want you to sit with before we even say what today's about.
Hana Field: Oh, that's ominous — go ahead.
Iris Holm: Anthropic — one of the most compute-rich AI labs on the planet — is still running Claude on a dense transformer. Every parameter, every token, every time. Meanwhile Gemini-1.5, Llama-4, DeepSeek-V3 have all moved to mixture-of-experts. All of them.
Hana Field: Wait — all three?
Iris Holm: All three. Production models. Not experiments.
Hana Field: And so Anthropic just... didn't. And the thing is, that's not a resource story — that's not 'we couldn't afford the routing infrastructure' — they absolutely could. So you end up with this weird picture where the lab most publicly associated with AI safety is also the lab that looks, on paper, like it's behind on architecture, and I keep wondering if those two things are actually the same choice.
Iris Holm: That's the puzzle. Not which architecture is faster. What does each lab believe they can afford to lose?
Hana Field: And to even get into that, we should probably say plainly what we're comparing — because mixture-of-experts sounds complicated and it's actually a pretty specific idea.
Iris Holm: Dense first: every parameter fires for every token. You send a sentence in, the whole model processes it. Consistent, auditable, expensive at scale.
Hana Field: And MoE — mixture-of-experts — you have a router, a learned gating function, that looks at each token and says 'you go to expert three and expert seven,' and the rest of the model just... sits there. So you can have an enormous model that only activates a small slice of itself for any given word.
Iris Holm: Right — and that gap, dense versus sparse activation, is where every consequential difference lives. That's what we're here to pull apart.
Hana Field: And that gap is — okay, I want to try an image for it, because I think the way most people hear 'sparse activation' and just... glaze over. Picture a hospital with a hundred specialists. A dense model is like sending every patient to every single doctor. Cardiologist, neurologist, all hundred. Every time. MoE has a triage nurse who looks at each patient and says 'you need two, maybe three of them' — and routes accordingly. The other ninety-seven specialists just sit in their offices.
Iris Holm: That's the right image. And now use that image to find the hidden cost.
Hana Field: You're still paying all hundred salaries.
Iris Holm: Exactly. All expert parameters live in memory whether they fire or not. You cut the floating-point operations per token — that's real — but the memory footprint doesn't shrink. Every specialist needs an office. That's the memory overhead problem, and it makes MoE genuinely difficult to deploy on anything memory-constrained.
Hana Field: So the efficiency gain is per-token compute, not total memory.
Iris Holm: Right — and there's a second cost the hospital image already contains, actually. When the triage nurse sends a patient to a specialist on a different floor, there's an elevator ride. In a distributed system, when tokens assigned to the same expert originate on different accelerators, you have to physically move that data across devices before the expert can run. That inter-device communication overhead — it adds latency. And benchmarks notoriously underweight it.
Hana Field: Wait — so the paper numbers don't capture the elevator ride?
Iris Holm: Often, no. The Switch Transformer paper — which is the paper that made MoE credible at scale, established the whole paradigm — it showed sparse models could scale efficiently. But 'efficiently' in that framing meant training compute. Not Tuesday afternoon production traffic with cross-device routing.
Hana Field: And that's the part that doesn't show up anywhere — the infrastructure team absorbs it and it never makes it into a results table.
Iris Holm: Mm. So the surface story is — MoE is more efficient. And it is. Per token, fewer operations. That's not wrong. But efficient in which dimension? Compute per token, yes. Memory, no. Latency under real routing load, unclear. Those are three different answers to the same question.
Hana Field: And Gemini-1.5, Llama-4, DeepSeek-V3 all bet that the compute-per-token win is the one that matters most at their scale — and maybe they're right — but I wonder if the labs running those models have fully priced in what happens when the router gets the routing wrong and tokens start getting dropped or re-routed under load.
Iris Holm: That's the load imbalance problem, and it's the one most people have never heard of — the model is silently degrading its own computation. Which is where this gets genuinely interesting, because Anthropic keeping Claude dense starts to look less like falling behind and more like a specific bet about which failure mode they can't tolerate.
Hana Field: And silently is the word that keeps catching me — because when I think about what token dropping actually looks like from the outside, there's nothing. No error message. The model just... quietly did less, and the output looks fine.
Iris Holm: That's the structural risk. The router gets lopsided — load imbalance — popular experts hit their capacity ceiling, tokens get discarded or shunted elsewhere. The model has silently degraded its own computation. And the output doesn't announce that.
Hana Field: Wait — how often is this actually happening? Like, is this a rare edge case or—
Iris Holm: Structural. Not a tail event. It requires active engineering countermeasures during training and inference to contain.
Hana Field: Which means there's a whole engineering arms race happening underneath these models that — and this is the part I find almost hard to hold — it doesn't show up in the benchmark number at all. The paper says 'our MoE model scores X' and somewhere in the infrastructure someone has built scaffolding to keep tokens from just vanishing, and that cost is... absorbed.
Iris Holm: And the countermeasures are genuinely interesting. Expert-choice routing flips the whole direction — instead of tokens selecting experts, experts select tokens. Which distributes load more evenly. Then there's ExpertFlow, which tries to pre-predict routing paths and pre-cache the experts you'll need before the token arrives.
Hana Field: Pre-cache which experts — okay, so it's guessing ahead.
Iris Holm: Predictive caching, yeah. Reduce memory pressure on constrained hardware. And then FlexMoE goes further — it takes a pretrained MoE model and compresses it into a family of smaller deployable subnetworks so you can match compute budget at inference time. Which tells you something: even after training, the inference costs require sustained engineering response.
Hana Field: So you build the model, and then you build another system on top of it just to run it affordably. And I keep thinking about — okay, so I have this image I want to put in front of you. A radiologist. She's using an AI diagnostic tool to flag potential tumors. Her workflow needs results in under 500 milliseconds or it breaks — she can't pause mid-read to wait. And if MoE routing overhead pushes that to 800ms, she either uses a smaller model that fits the window, or she uses the MoE system and her hospital's legal team now has a completely different problem on their hands if the model misses something.
Iris Holm: And that 300-millisecond gap isn't theoretical. That's the countermeasures not fully closing the distance.
Hana Field: And nobody has actually proven they close it, right? Like — whether expert-choice routing or ExpertFlow actually solve load imbalance fully, that's still an open research question.
Iris Holm: Open, yes. Which is the honest answer and also the uncomfortable one, because Gemini-1.5, DeepSeek-V3, Llama-4 are all running in production on architectures with an unresolved failure mode at their core.
Hana Field: And the part that I think reframes all of this — we haven't gotten there yet, but there's something about what MoE's dominance actually reflects about which metric the whole industry converged on — that changes how you read both the token dropping problem and Anthropic staying dense.
Iris Holm: The question is whether dominance means winning or just... all running the same race.
Hana Field: And that's the thing — 'running the same race' is actually exactly the right frame, because when you look at Gemini-1.5, Llama-4, DeepSeek-V3, they all converged on MoE, but they were all also optimizing for the same thing: training compute efficiency. That one dimension. And I wonder if that convergence looks like proof that MoE won, when really it's just... three labs with similar hardware stacks and similar cost pressures making the same bet.
Iris Holm: That's the reframe. Dominant doesn't mean correct. It means everyone measured the same thing.
Hana Field: And Anthropic just... measured something else.
Iris Holm: Behavioral predictability. The Switch Transformer paper — the one that established MoE as serious — that came out years before the recent Claude generations. Anthropic isn't uninformed. They read the paper. They made a choice anyway.
Hana Field: Which means Claude being dense is deliberate in a way that — okay, that actually changes how I read it.
Iris Holm: Now here's the fact that breaks the clean version of this story. MoE experts — the individual specialist subnetworks — are measurably less polysemantic than dense feed-forward neurons. That's an interpretability research finding. Sparsity forces specialization. Each expert encodes fewer distinct concepts. Dense neurons have many concepts superimposed on the same weights.
Hana Field: Wait — so MoE actually has a structural interpretability advantage?
Iris Holm: In principle. That's the exact tension — and I mean, it's almost perverse — Anthropic stays dense for predictability, but MoE's neurons are actually easier to characterize. One expert, fewer concepts, more legible. The architecture Anthropic rejected has a structural property that cuts in favor of the thing Anthropic cares about most.
Hana Field: So two different paths to explainability — dense gives you consistency, MoE gives you specialization — and neither one is actually proven legible to someone outside the lab.
Iris Holm: That's the unresolved question. Whether reduced polysemanticity in an MoE expert translates into something a regulator can use — or a compliance officer, or a plaintiff's lawyer — that's completely open. It's an internal research property right now. Not a courtroom property.
Hana Field: And so you have Anthropic arguing from behavioral predictability, and you have MoE research finding structural monosemanticity, and both of them are saying 'we're more explainable' but from completely opposite architectural directions, and neither one has actually closed that argument with a regulator in the room.
Iris Holm: Right. And the bet they're each making — it's really a prediction about when explainability becomes the load-bearing constraint. Not if. When.
Hana Field: And that's not a technical question anymore.
Iris Holm: It's a prediction about liability. About which failure mode regulators decide to price. Dense keeps Claude auditable at the behavior layer. MoE's monosemantic experts are legible at the neuron layer. Those are different levels of the stack — and right now nobody outside these labs actually knows which level courts are going to care about.
Hana Field: The question has completely shifted underneath us. It's not 'which architecture scales cheaper' anymore. It's 'which architecture lets you defend the model's behavior to a regulator, or an auditor, or someone who was actually harmed.' And that's a different question, and nobody's benchmarking it.
Iris Holm: Neither prediction has been tested. Anthropic's dense bet, MoE's monosemanticity advantage — both are claims about explainability that exist entirely inside research contexts. Not one has been stress-tested by a regulator with subpoena power.
Hana Field: Which means Gemini-1.5, DeepSeek-V3, Llama-4, Claude — all of them are making predictions about a future accountability regime that doesn't exist yet. And the architecture you chose today is the one you're stuck defending when that regime arrives.
Iris Holm: That's where I land. Uncomfortable, but honest.
Hana Field: Yeah. I think that's actually — and I don't mean this as a neat bow on it — but that's genuinely where I am too. Neither architecture wins. They're just bets. Different labs, different bets about which problems the next five years will surface. And we won't know who was right until it's too late to switch.
Iris Holm: Good place to stop. Thanks for going deep on this one.
