Walt Garner: Nina, you look like you've been ambushed by something — what happened?
Nina Park: I mean, kind of? I went down a rabbit hole this week reading about the reproducibility crisis and I hit one number that I literally had to close my laptop for a minute. Studies that fail to replicate get cited more than the ones that hold up. The broken science gets amplified.
Walt Garner: More citations. Not fewer.
Nina Park: More. And I keep turning it over because — okay, there's a version of this where it's just sad, right, it's a failure, things slipped through. But there's another version where it's not a slip at all. Where the system is working exactly as designed and that's the actual problem.
Walt Garner: Now that's where I want to start, because the design question matters enormously. The Royal Society introduced written referee reports in 1832 — that's the institutional founding of modern peer review — and the mandate was validity, originality, significance. Not replicability. Reproducibility was never in the job description.
Nina Park: So we gave this machine a job it was never built for.
Walt Garner: And kept all the original incentives intact while doing so. Journals favor novel, positive, counterintuitive results — publication bias is what that tendency is called — because novelty drives citations, which drive prestige. That dynamic was present in 1832 and it is present in 2025.
Nina Park: Which is — I keep wanting to call that a prestige economy. Like it's not so different from fashion weeks or film festivals, except it's wearing a lab coat.
Walt Garner: That's not a bad frame, actually — though I'd push on it slightly. A film festival celebrating a flawed film doesn't get anyone hurt. When the most exciting false findings in biomedicine get cited three hundred times before anyone checks them, the stakes are, you see, rather different.
Nina Park: No, fair — the stakes are what make this more than a vibes problem. The reproducibility crisis is documented across fields. It's not one bad study. It's a structural institutional output.
Walt Garner: And so the question that this whole thing hangs on — the one I find genuinely difficult to resolve — is whether what we're describing is a category error, us expecting something the system was never designed to deliver, or whether it's a failure that demands reform. Because both can be simultaneously true.
Nina Park: Yeah — and I think figuring out which one it is changes everything about what you do next. Because you don't fix a category error the same way you fix a broken machine.
Walt Garner: And that distinction — category error versus broken machine — actually gets sharper when you look at what a reviewer is physically doing with a manuscript. They're reading one document. One. They have no second dataset, no independent lab, no time-lapse of what happens when someone else runs the protocol. The format structurally cannot tell you whether the effect survives replication — that's not a reviewer failing at their job, that's the job description not including that task.
Nina Park: Wait — so the tool literally doesn't have that function built in.
Walt Garner: It never did. The Royal Society in 1832 was asking: is this valid, is it original, is it significant? Those are curatorial questions. And here's the analogy I keep returning to — a film festival jury. They pick the most interesting entries. That's the whole job. Nobody expects the Cannes jury to tell you which films historians will still be discussing in fifty years.
Nina Park: Oh — that actually clicks for me in a way the pure institutional-failure framing doesn't, because we don't walk out of Cannes angry that the Palme d'Or winner wasn't historically durable. We just — we knew what we were watching.
Walt Garner: Precisely. But with peer review we forgot we were watching a festival.
Nina Park: And the thing that makes it worse is that a single reviewer reading a single manuscript can't even assess replicability in principle. Like, it's not that they're being lazy. The information doesn't exist yet. The replication hasn't happened.
Walt Garner: You see, that's the uncomfortable layer under it. We've been blaming the jury for not doing the historian's job. Those are different people with different tools operating across completely different timescales.
Nina Park: And yet — okay, I want to push on something. Even if that's true, even if the design was never built for reproducibility, there's still a structural incentive bias toward novelty that the design actively rewards. Journals gain prestige from exciting findings. That's not neutral. That's the festival explicitly handing out trophies for surprise.
Walt Garner: No, that's right — and that's where the category error framing has a limit. The design didn't just fail to include reproducibility; it actively selected against it. Which, well, those are two different problems sitting on top of each other.
Nina Park: So it's both. The jury was never asked to pick durable films — and also, the scoring rubric gives extra points for the most surprising plot twist.
Walt Garner: And once you see those as separable problems, you start to see why reforms keep sliding off. You can fix the rubric. You cannot retroactively give a reviewer access to data that doesn't exist at the time of review.
Nina Park: Which — that's actually the part that makes me want to understand what people have tried. Because if the structural limit is real, what does a fix even look like?
Walt Garner: Well, the fixes tell you exactly what the problem is — and the one that keeps nagging at me is that every actor in this system is pointing the same direction without any coordination at all. Journals compete for prestige through landmark, heavily-cited papers, so editors favor novel, positive findings at the selection stage. That's layer one.
Nina Park: And layer two is researchers.
Walt Garner: Researchers advance careers and attract funding by publishing surprising, positive results — so they're under pressure to produce them. And then, now, here's the part people leave out — reviewers. Reviewers signal expertise by endorsing exciting work. A third independent incentive layer, pushing in exactly the same direction.
Nina Park: Wait — the reviewer gains something too? I hadn't actually followed that all the way through.
Walt Garner: It's inferred rather than directly measured, I'll grant that. But the structural logic is — if you're the reviewer who endorsed the landmark finding, that reflects on your curatorial judgment. You spotted it first. There's status in that.
Nina Park: Okay and here's what gets me — none of those three parties have to talk to each other for the bias to be total. The editor doesn't call the researcher, the researcher doesn't call the reviewer. They're all just — I mean, they're each making a rational local decision, and the aggregate is a system that systematically over-represents findings that won't replicate.
Walt Garner: No conspiracy required.
Nina Park: Which is actually scarier than a conspiracy, because you can't just find the bad actor and remove them. The incentive alignment is the problem. And — okay, I want to make this human for a second, because the abstraction is easy to shrug off. Picture a postdoc at eleven p.m., staring at three datasets. One is clean, replicable, solid. One is noisier but counterintuitive — unexpected interaction, surprising direction. And the third flatly contradicts her hypothesis. Her tenure clock is running. Her last three papers took two years each to publish. Which dataset gets the full analytical attention?
Walt Garner: She is not making an unethical choice.
Nina Park: She is making a rational one. That's the whole point. The structural incentive bias toward novelty doesn't need anyone to be villainous — it just needs the path of least resistance to run through the surprising result every single time.
Walt Garner: And what's remarkable — well, what's genuinely troubling — is that even reforming one layer doesn't interrupt the other two. You could change how journals select papers tomorrow, and the researcher's tenure pressure doesn't move. The reviewer's status incentive doesn't move. The machine recalibrates around the remaining gears.
Nina Park: Which is — yeah, that's exactly why I want to get into what people have actually tried to change it, because the reform graveyard here is specific and kind of devastating, and the statistical mechanics underneath are even stranger than the incentive story. That part gets worse before it gets better.
Walt Garner: Indeed. And the place to start, you see, is with what p-hacking and publication bias look like when you try to actually measure them — because that's where the picture becomes, well, almost counterintuitive in its own right.
Nina Park: Okay but measuring them is exactly where it breaks down — because Jonas Moss and Riccardo De Bin actually ran the selection models on this, and what they found is that p-hacking and publication bias don't just distort individual studies. They violate the assumptions that meta-analysis depends on. The tool we built to correct the bias is compromised by the same bias.
Walt Garner: So the corrective is downstream of the same problem.
Nina Park: It's turtles all the way down. And p-hacking is — I mean, the thing people miss is it doesn't require malice. You adjust your exclusion criteria, you shift your stopping rule, you try one more variable combination, and suddenly you have p less than point-oh-five. Whether you meant to or not.
Walt Garner: And null results simply — well, they simply vanish. Few journals allocate space to replication studies or null findings at all, structurally. PLOS ONE is the largest scientific journal in the world, and even studying its peer review reports to understand evolving practices is treated as noteworthy research. That absence is the literature.
Nina Park: Right — so the shelf is already curated before you even walk into the library.
Walt Garner: Now, PubMed Commons is where I think this becomes — you see, this is the reform failure that actually tells you something. The U.S. National Library of Medicine launched it in June 2013. A post-publication open commenting forum, part of a three-pronged initiative, specifically so researchers could flag problems after a paper was already in the literature. Gone by February 2018.
Nina Park: Five years. And no one really explained why.
Walt Garner: That silence is the interesting part. Was it unused? Did it make journals uncomfortable? The sources don't say. And that gap tells you more about institutional resistance than a full explanation would.
Nina Park: Oh — actually, registered reports are the other piece here, because that one is structurally elegant. Peer review happens before data collection, conditional acceptance based on method not results. You literally decouple the publication decision from whether the finding is exciting. And it still hasn't moved the needle.
Walt Garner: Marginal. Kevin J. Boudreau synthesized randomized experiments and audit studies on scientific evaluation through 2025 — decades of human-centered reforms — and the improvements are, well, marginal is the word he uses. Which means the problem is not ignorance of the solution.
Nina Park: We know what fixes it. We just haven't fixed it.
Walt Garner: And now there's AI-assisted peer review, which — the early evidence suggests quality comparable to human reviewers, and yes, faster. But faster doesn't change what gets submitted. It doesn't change what editors select. If the algorithm learns what counts as exciting from the existing literature, it may actually narrow the definition of novelty further.
Nina Park: So the reviewer burden collapses but the selection pressure intensifies. That's — huh. That's not a fix, that's acceleration.
Walt Garner: The machine runs faster toward the same destination. And the reviewer inconsistency problem — low inter-reviewer agreement, review work concentrated in a small minority of scholars — none of that changes the structural incentives that determine what arrives at the door in the first place.
Nina Park: And that's — I mean, that's where it stops being just a structural story for me. Because when a patient takes a drug whose efficacy rested on a p-hacked finding, 'the system is working as designed' is not a sentence you can finish out loud. The downstream cost lands on people who were never inside the incentive structure at all.
Walt Garner: No. And that's where 'structurally rational' and 'ethically sufficient' stop being the same sentence. The researcher made a locally rational choice. The reviewer made a locally rational choice. And somewhere down the line a clinician acted on a finding that couldn't survive an independent lab. Those are real choices by real people, even if each one was defensible within the rules of the system that produced it.
Nina Park: Right — but the part that doesn't fit is that nobody in that chain felt like they were doing something wrong.
Walt Garner: Which is, you see, precisely why the moral weight doesn't dissolve. The policymaker acting on that finding, the patient — they had no seat at the table when the incentives were arranged. That asymmetry is uncomfortable and I don't think it resolves.
Nina Park: The question is what happens if institutions start funding scientists on replication attempts and null results rather than publications and citations. Does the novelty bias actually leave the system? Or does it just find a new gatekeeper and relocate there?
Walt Garner: I won't pretend I know. The bias has survived every reform we've named today. That's not nothing.
Nina Park: Yeah. That's an honest place to sit. Thanks for thinking through this with me — the uncomfortable parts especially.