Megan Skiendel: Tell me you saw the SWE-ABS numbers, because I've been sitting on this since February and I needed someone to be confused with me.
David Sterling: I saw them. The seventy-eight-point-eight to sixty-two-point-two collapse.
Megan Skiendel: Right — and not some margin-of-error thing. Twenty percentage points on the same artifact, same task, just with a test harness that actually checked whether the patch was semantically correct. One in five solutions from top-30 agents on SWE-bench Verified were just... wrong. Passing because nobody looked closely enough.
David Sterling: The benchmark credibility event, I'd call it. SWE-bench Verified has been the dominant eval for coding agents and it's been inflating scores at scale.
Megan Skiendel: And here's the part that doesn't fit neatly — Claude, Anthropic's Claude, scores lower on the pure reasoning benchmarks. Competitors beat it there. But in actual agentic workflows, production deployments, the practitioners keep picking Claude. I've seen this pattern before where the metric and the outcome completely diverge, and it always means someone defined the metric wrong.
David Sterling: Or — and this is what I want to test today — the model is optimized for a different axis entirely. Capability versus competence. What you score in isolation versus what you accomplish in a live loop.
Megan Skiendel: Is that a design choice Anthropic made deliberately, or did the benchmark just fail to notice what they built?
David Sterling: That's the load-bearing question. And we can't answer it until we decide whether we trust any of these evals at all.
Megan Skiendel: But trusting the evals cuts both ways — so let me ask the dumb version of this, because I think it's actually not dumb. Were the scores just... made up?
David Sterling: Not fabricated. Inflated. There's a difference that actually matters here. Think of it like a driving test where the examiner only checks whether the car moved forward. You could roll five feet, run a red light, and pass. SWE-bench Verified was that examiner. The test suites weren't checking whether the patch was semantically correct — just whether it cleared a weak gate.
Megan Skiendel: So the grading was lenient, not the model being clever.
David Sterling: Exactly that. And SWE-ABS — the adversarial benchmark strengthening framework — what it actually did was tighten the examiner. Augmented test coverage, synthesized plausible-but-wrong patches to find the blind spots. And the top agent's score dropped from seventy-eight-point-eight to sixty-two-point-two. One number. That's your answer on how lenient the original grading was.
Megan Skiendel: Okay, wait — one in five patches from top-30 agents were semantically wrong and just... nobody caught it. For how long?
David Sterling: Well, that's the part I want to be careful about — the sourcing here is community analysis and adversarial research, so I don't want to overstate how settled this is. But the SWE-bench Pro numbers are harder to argue with. Same models that scored around seventy percent on SWE-bench Verified? They dropped to twenty-three percent on the harder successor. That's not noise. That's the benchmark measuring evaluator rigor, not model capability. The static benchmark may have been optimized against — data contamination, overfitting to specific instances — which is actually why SWE-bench Live switched to post-2024 GitHub issues. Fresh problems the models haven't seen.
Megan Skiendel: So the score was real — the task it was measuring just wasn't the task anyone thought it was measuring.
David Sterling: One sentence. Yes. And that's the contingent liability — enterprises allocated capital on those benchmarks. If the model ranked first on SWE-bench actually solves one in four fewer problems when the gate is tightened, someone's procurement decision is already wrong. They just don't know it yet.
Megan Skiendel: And that procurement gap is exactly where the architectural bet becomes visible — because someone at Anthropic looked at that liability and said, what if we stop chasing the benchmark and build for the loop instead. The Agent Loop. Act, observe the result from the environment, adjust, repeat. Not one long answer. An iteration.
David Sterling: The live feedback from the environment is doing work the model would otherwise have to reason through?
Megan Skiendel: That's precisely it. Imagine a developer at a payment startup — it's late Thursday, she's got a broken API integration, she tells her agent to audit it. The agent doesn't think the whole problem through in one shot. It calls a linting tool, reads the actual error output, patches the file, runs the tests. If they fail, it loops back. She doesn't touch it.
David Sterling: And Claude Code is that loop. Shipping.
Megan Skiendel: Claude Code is the loop in a live developer environment — reads files, executes commands, modifies code, runs tests. And then March into April 2026, Anthropic formalizes it: Advanced Tool Use on the Claude Developer Platform. Programmatic Tool Calling, Tool Search. That's the moment someone put a flag in the ground.
David Sterling: Explain PTC, because the mechanism actually matters here.
Megan Skiendel: PTC — Programmatic Tool Calling — Claude writes Python that invokes multiple tools inside a sandboxed container. Sequencing, branching logic, pre- and post-processing. No separate model round-trip per tool call. The intermediate results stay in the sandbox, so your context doesn't bloat across a twenty-step workflow. That's the engineering choice. That's Tool-Calling Reliability as the actual axis, not isolated reasoning scores.
David Sterling: Though I'd want to be careful calling that deliberate. There's no design document we can point to. We're inferring intent from product decisions — which, frankly, is how most strategy gets reconstructed after the fact.
Megan Skiendel: Fair. And honestly, the harder question — which we should get to — is whether any of this is even verifiable. If evals are broken going one direction, the claim that Claude actually wins in real agent tasks is also hard to confirm. That uncertainty may end up doing something structural to who benefits. But that's the next thread.
David Sterling: That uncertainty complicates the analysis and may actually entrench whoever's already inside the workflow. If you can't measure the gap, a challenger can't prove they've closed it.
Megan Skiendel: Oh, that's uncomfortable.
David Sterling: Think about what Thariq Shihipar actually said — he's on the Claude Code team at Anthropic — writing good evals for an AI agent is closer to art than science. That's not a throwaway line. That's the person building the tool telling you the measurement problem is unsolved.
Megan Skiendel: And if the measurement problem is unsolved, then the capability-versus-competence distinction — what a model does on a benchmark versus what it actually accomplishes in a messy real deployment — that gap is essentially unauditable right now.
David Sterling: Which collapses the signal in both directions. Terminal-Bench found that top SWE-bench scores don't transfer to live shell operation — different task, different result. So we've got a benchmark that inflated scores by twenty points, a harder successor where the same models fell to twenty-three percent, and a separate live-shell eval that tells a third story entirely. That's not a measurement system. That's three different things we're calling one thing.
Megan Skiendel: Okay but — I'd push back slightly. Practitioners didn't wait for the research to catch up. They migrated to private evals, adoption signals, internal benchmarks. The market self-corrected. Claude Code has tight developer feedback loops because developers are actually using it, filing issues, and that data feeds back to Anthropic in a way that a static benchmark just can't replicate.
David Sterling: I'd probably grant that — I mean, adoption as a signal is real. But here's the structural consequence: if private feedback loops are now the actual eval mechanism, incumbents with the most users generate the most signal. Claude Code's feedback advantage compounds. A new entrant can't buy that.
Megan Skiendel: Network effect on the eval itself. That's — honestly, that's the moat nobody's naming out loud.
David Sterling: We should hedge that — the sourcing on how much Anthropic actually benefits structurally is thin, it's inference from product position, not hard data. But the logic holds. Eval quality as the binding constraint, Thariq's own framing, and the fact that no single benchmark captures the full picture right now — that's not a temporary gap. That's the conditions under which an incumbent stays an incumbent.
Megan Skiendel: And maybe that's actually where we land. Not — Anthropic won. But that Claude Code is already inside enough developer workflows that the benchmark question almost doesn't matter anymore. Advanced Tool Use, PTC, the whole March–April 2026 formalization — that wasn't a product launch. That was the moment the race shifted to a different track entirely.
David Sterling: The benchmarks were covering fire. I'll buy that. Whether the architectural bet holds five years out — I mean, I genuinely don't know. Competitors close harness gaps. They do. Eighteen months, maybe less. But if the eval quality is the bottleneck and your incumbency generates the feedback signal — that's not a gap that closes just from shipping better reasoning scores.
Megan Skiendel: One thing I'm actually comfortable saying: Anthropic didn't need to win the benchmark war first.
David Sterling: No. They needed the workflow. Good conversation.
