Maya Chen: Nathan, quick thing before we get into it — have you ever gotten a performance review that felt like it was grading the wrong job?
Dr. Nathan Hayes: More than once. Where are you going with this?
Maya Chen: That's — mm, that's basically the entire episode. AI right now is getting evaluated on a very narrow slice of work, and companies are using those scores to justify enormous investment decisions. And in October 2026, Mercor published something that makes that gap visible in a way nobody had before — they called it the Benchmark Map.
Dr. Nathan Hayes: So the question is whether the tests match the actual job.
Maya Chen: And the answer is... mostly no. Mercor took twenty-nine AI agent evaluations and ran them against the full O*NET taxonomy — nine hundred and twenty-three occupations, nearly nineteen thousand tasks. The U.S. Department of Labor's official occupational ground truth. And only two hundred and eight of those nine hundred and twenty-three occupations have even one task with five or more benchmark prompts mapped to it. Six job families have zero.
Dr. Nathan Hayes: That's — hold on, that's not ninety percent uncovered, that's more like seventy-seven percent by occupation count alone.
Maya Chen: Right, and Mercor's own framing is explicit — the ninety percent refers to working hours, not just occupation headcount, and they label it provisional. They're identifying a measurement gap. Not proving AI can't do the work — that's a different claim, and they don't make it.
Dr. Nathan Hayes: Which is the scientifically honest position, and also the one the headline will ignore.
Maya Chen: And that's the part that actually keeps me up — because the headline ignores it, but so does the investment memo. Like, imagine a radiologist sitting down at 6 a.m., pulling up a chest X-ray, and someone hands her a report card that says 'AI: ninety percent not ready.' She reads that as capability. She should be reading it as 'we haven't built the test yet.'
Dr. Nathan Hayes: Now, that distinction — measurement gap versus capability gap — that's where I want to slow down. Because the benchmarks we *do* have are real. TheAgentCompany, published 2024, presented at NeurIPS, put LLM agents into a simulated software company doing actual workplace tasks. Best agent completed roughly thirty percent autonomously.
Maya Chen: Thirty percent. In software.
Dr. Nathan Hayes: In one narrow occupational slice. And HumanEval-V — same year — tested twenty-two multimodal models on diagram-based coding tasks. Claude 3.5 Sonnet, the top performer, hit thirty-six point eight percent pass@1. So even inside that one room, the gaps are genuine. My concern is — actually, wait — the concern isn't that the benchmarks are wrong. It's that we're using them to make claims about nine hundred and twenty-three occupations when they were designed for maybe a dozen.
Maya Chen: So we're testing it in one room of a very large house and writing the inspection report for the whole building.
Dr. Nathan Hayes: That's exactly Mercor's flag. And METR compounds it — they've documented a task-horizon doubling time of under three months for AI on longer software tasks. Which sounds like rapid progress until you notice: still software. That improvement curve doesn't generalize.
Maya Chen: Right — but here's what I keep getting stuck on. If the thirty percent figure from TheAgentCompany is real and rigorous, doesn't that mean the measurement gap argument actually *protects* a low number? Like, we might fill in the rest of the map and find... it's thirty percent everywhere.
Dr. Nathan Hayes: Or we find it's three percent for home health tasks and seventy percent for document summarization. BIG-Bench Hard gives us twenty-three task categories where models fall below average human-rater performance — that's a 2022 data point, already dated, but it's a shape. The problem is we can't extrapolate the shape beyond what we've measured.
Maya Chen: Which means the ninety percent figure is — mm — it's almost more alarming read charitably. It's not 'AI failed.' It's 'we have almost no idea.'
Dr. Nathan Hayes: And that 'almost no idea' is the part that has a direction to it. It's not random ignorance — it's structured. Benchmark coverage clusters around software engineering, financial analysis, law. The Mercor data makes that explicit.
Maya Chen: Structured toward who already has resources. Yeah.
Dr. Nathan Hayes: So here's what that means in practice. Picture a home health aide, Friday evening, reviewing a patient's overnight medication log — flagging that a dose looks off, calling the family member to confirm the morning routine, then looping in the supervising nurse before the shift handoff. That whole chain — relational, exception-heavy, time-pressured — appears nowhere in any of the 29 benchmarks Mercor mapped.
Maya Chen: Nowhere. Six entire O*NET job families with zero coverage — and home health is inside that gap.
Dr. Nathan Hayes: Now, is that because those tasks are hard to benchmark, or because nobody prioritized them? I think — honestly, both. But the effect is the same.
Maya Chen: And the usage data is where this gets — I mean, it stops being abstract. December 2024 to January 2025, Mercor pulled actual LLM usage figures. The top ten occupations with the highest worker desire for automation? They account for one point two six percent of total LLM usage. One point two six. The workers asking loudest for help are almost entirely absent from where the tools actually land.
Dr. Nathan Hayes: Wait — is that a supply problem? Access? Or R&D just pointing the wrong direction entirely?
Maya Chen: Mercor doesn't resolve which — and that's the honest answer. But the pattern is the story. If you never measure the work, investment never follows, tools never get built, and the person doing that Friday-evening medication check is not displaced by AI, not liberated by AI — they're just not in the conversation. That psychological dimension, the invisibility of it, that's what the one point two six percent actually represents. And healthcare is where this gets even sharper — though that's the part coming next that makes the whole measurement reform argument harder than it first looks.
Dr. Nathan Hayes: Right — because naming the gap and closing it are two completely different problems.
Maya Chen: And healthcare is where naming and closing that gap split the furthest apart. Because this isn't a field where we're flying blind — we have scores. Medical AI has aced standardized exams. And it still hasn't cracked actual clinical integration.
Dr. Nathan Hayes: That's the part that should unsettle people. High benchmark scores, real deployment failure. The ReXecution Framework — published 2025, Pacific Symposium on Biocomputing — tested chest X-ray AI assistants on a hundred expert-curated real-world radiology tasks. Not exam questions. Actual radiological workflow. And it found a persistent gap between what models can do and what radiologists actually need.
Maya Chen: A hundred tasks. That's — wait, who curated them?
Dr. Nathan Hayes: Clinicians. Which is exactly the point — the moment you let domain experts define the test, the test looks completely different from the leaderboard version.
Maya Chen: So PLOS Digital Health publishes a paper in 2026 — five principles for fixing healthcare AI evaluation specifically. Local, Task-specific, Agile, Reflective, Community-partnered. And I read that list and I think — mm — those are the right principles, but they're also an enormous amount of sector-specific labor before any of this moves.
Dr. Nathan Hayes: They don't port. That's my honest read. Those five principles work for radiology, maybe oncology — but you can't lift them and drop them on food service, social work, manufacturing. You'd need an entirely separate framework-building exercise for each sector. That's not a reason to abandon the project. It's just — actually, it's why the Mercor incentive loop is the real problem.
Maya Chen: Say more about that loop.
Dr. Nathan Hayes: Vendors take rankings from the same narrow benchmarks — the ones Mercor flagged as covering maybe ten percent of working hours — and use them to justify billion-dollar hiring and automation decisions. So the benchmarks are bad, the scores travel anyway, and the money follows the score. Fixing measurement sector by sector takes years. The capital doesn't wait.
Maya Chen: Which means a pulmonologist looking at a procurement decision next quarter is being handed a number that was never designed to answer her question. And the PLOS framework — even if it's exactly right — won't reach her in time.
Dr. Nathan Hayes: And that's — I keep wanting to find a technical fix, and I don't think there is one. The Mercor Benchmark Map is twenty-nine benchmarks against nine hundred and twenty-three occupations. It's honest about what it is. It's a mirror. It's not a repair.
Maya Chen: A mirror, yeah. The ninety percent figure — provisional, their word — it's not a verdict on capability. It's a verdict on what we decided was worth measuring. And that decision is where the accountability question actually lives, not the capability question.
Dr. Nathan Hayes: One point two six percent. That number is what I'm left with. Workers with the highest automation desire representing one point two six percent of actual LLM usage. That's not a measurement problem you solve with better benchmarks. That's a choice someone made.
