Brian Reed: Hey — glad you're here, because I have been staring at this leaderboard for twenty minutes and I need a second opinion.
Eliza Ward: The Liquid AI one?
Brian Reed: Yeah. So they post on November 16 — d1.2-ultra, 75 on the Artificial Analysis Decision Index, claiming the number one slot.
Eliza Ward: Okay, and — wait, actually, that's the tension right there. November 12, four days prior, Drex Max v1.1 from Nace AI and GLiDE v1.2 from Fastino Labs are tied at 73, independently verified. That's the last confirmed data point we have.
Brian Reed: So either d1.2-ultra genuinely jumped two points in four days, or—
Eliza Ward: Or no one's checked it yet.
Brian Reed: The thread had about 23 likes. That's — yeah, that's not a number that suggests a lot of independent eyes on it.
Eliza Ward: No high-engagement posts corroborating the 75 score, none pushing back on it either. So we're just sitting with the gap.
Brian Reed: But here's what I keep getting stuck on — what does d1.2-ultra actually do? Like, before we can even argue about the number, I want to know what the thing is.
Eliza Ward: Right. Okay. So — think of it like a referee. You show it a situation, it tells you which call to make. 'Approve this loan application.' 'Flag this transaction.' With a confidence score attached. That's the output. It's not writing you an essay.
Brian Reed: It doesn't generate anything.
Eliza Ward: Classifications, scores, choices. Structured probability outputs. That's the d1 series, that's what it ships. So when Liquid AI says 'omnimodal output — text, images, audio' — wait, that's not — I mean, that framing is doing a lot of work that the technical docs don't back up.
Brian Reed: So 'omnimodal' is describing the inputs, not the outputs.
Eliza Ward: That's the actual thing that's new with d1.2 and d1.2-ultra — you can hand it a photo, an audio clip, not just text. The referee can now watch the play instead of just reading a description of it. But the call it makes? Still 'approve' or 'flag.' Still a structured decision.
Brian Reed: No, let me get this right. Because d1-ultra only launched October 15. That's barely a month before d1.2-ultra. And d1.2 is the open-weight version sitting alongside it. That's a fast cadence for adding multimodal inputs and claiming a ten-point jump from the original 65.2.
Eliza Ward: Ten points in a month is — yeah, that's the number that needs scrutiny. The multimodal inputs are real, that's a genuine capability addition. The 'omnimodal output' in the marketing copy is the part that doesn't match what shipped.
Brian Reed: And nobody's condemned them for it — it's just a gap worth naming. The referee now has better cameras. It still calls 'approve' or 'flag.'
Eliza Ward: But here's what that framing actually breaks — the take I keep seeing is that the 75 is just a number you can look up. Like it's sitting on a leaderboard, it's verified, it's done. And that's the one I want to push back on, because Decision Index v2 shipped November 8. Eight days before the claim.
Brian Reed: Eight days. That's not a small gap.
Eliza Ward: No, and v2 isn't a minor patch. Private dataset weighting went from 25% to 45%. They added the AA-Paperclip corpus — 1,500 documents, resumes, receipts, invoices, research papers — that alone is 30% of the benchmark now. And calibrated-confidence scoring got added as a new component. So nearly every model's ranking reshuffled.
Brian Reed: So d1-ultra's 65.2 — that was under the old rules.
Eliza Ward: Completely different conditions. And there's an interim figure — d1-ultra at 70.4 on v2. Which is, I mean, it sits right between 65.2 and the claimed 75. So we don't actually know if that jump is a better model or just the new weighting inflating the score. We can't tell from the outside.
Brian Reed: And nobody re-ran Drex Max v1.1 or GLiDE v1.2 under v2 to get a comparable 75?
Eliza Ward: Not that I've seen. Which is — wait, that's actually the core problem. You can't claim #1 against scores that were set under a methodology that no longer exists. That's not a leaderboard fact, that's a snapshot under different rules.
Brian Reed: And then — hang on, because this is the part that stopped me cold — Claude Haiku 5.5 at 77 and GPT-6 Luna at 76. On the language-model side of the same index. Higher than any decision model.
Eliza Ward: Right. So what does 75 even mean if two general-purpose language models already clear it? That question gets much sharper once you get to what the closed-weight API strategy actually means for anyone trying to pick between d1.2-ultra, Drex Max, and GLiDE right now — but that's the next layer.
Brian Reed: And that closed-weight thing is — hang on, because that's not just a technical footnote. d1.2-ultra only exists as an API. OpenRouter, Vercel AI Gateway. That's it. So picture someone running procurement for a mid-size bank — she's got three tabs open, one for each model. d1.2-ultra, Drex Max v1.1, GLiDE v1.2. She's trying to pick a loan-flagging pipeline. She reads: Liquid AI, 75. Nace AI, 73. Fastino Labs, 73. Natural read is Liquid AI wins by two.
Eliza Ward: Except she can't pull the weights.
Brian Reed: She literally cannot. With Drex Max or GLiDE — open weights, you can re-run the eval yourself, check the methodology, stress-test it on your own documents. With d1.2-ultra, you send API calls and you get answers back. That's the whole surface area you have.
Eliza Ward: So she's not verifying the 75. She's trusting that Artificial Analysis tested it, or trusting Liquid AI's post, which — wait, those aren't the same thing. And Artificial Analysis is the only independent evaluator in this space right now. There's no second source.
Brian Reed: One evaluator, mid-overhaul.
Eliza Ward: Right — and we don't know if Drex Max v1.1 or GLiDE v1.2 have even been re-run under v2 rules. Their 73s are from November 12, before AA-Paperclip was weighted at 30%. So she's not actually comparing three scores under the same ruleset. She might be comparing one v2 number against two v1 numbers that nobody's updated.
Brian Reed: That's the part I actually don't know how to flag for her. Like, the leaderboard looks current. It has recent dates. Nothing on the page tells you those scores are pre- and post-methodology-change.
Eliza Ward: And the open-weight d1.2 — I mean, that exists, but it scores low enough that it doesn't appear in the benchmark conversation at all. So the version enterprises actually evaluate is closed, the version researchers can audit is weak. That's the split Liquid AI built in.
Brian Reed: So the concrete thing to watch is whether Artificial Analysis publishes a fresh run of Drex Max and GLiDE under the full v2 methodology — with AA-Paperclip at 30% — before anyone commits budget. Until that happens, that two-point gap isn't a confirmed lead. It's an open question.
Eliza Ward: And that's — wait, that's actually where I'd land right now. The one thing that would actually resolve this is an independent re-test of d1.2-ultra under Decision Index v2 rules, published by Artificial Analysis. Not Liquid AI's thread. Not an interim figure. Artificial Analysis runs the eval, posts the number. That's it. That's the only thing that closes the gap.
Brian Reed: And we don't have that.
Eliza Ward: We don't have that. And — I mean, honestly, we don't know if it's coming. Artificial Analysis could publish it next week. They could just leave the leaderboard as-is. There's no filing deadline, no regulatory clock. It happens when it happens, or it doesn't.
Brian Reed: So until then — the person choosing between d1.2-ultra at 75, Drex Max at 73, and GLiDE at 73 is working from a vendor's promotional claim on one side and two pre-updated scores on the other. That's not a comparison. That's just... three numbers that don't share a denominator.
Eliza Ward: That's where it sits. Story isn't over.