Onpode
Cover art for The structural reason most published findings don't replicate — and what that tells us

The structural reason most published findings don't replicate — and what that tells us

November 15, 2026 · 13 min

Sarah Lin & Dr. Nathan Hayes

Sarah, good to be back — I have to say, I've been sitting with something uncomfortable all morning and I think it's the right place to start. Oh, I want to hear this — what is it? So imagine a researcher — and this is not a hypothetical bad actor, this is a careful, methodologically…

The replication crisis refers to the widespread finding that many published scientific results cannot be reproduced by independent researchers. Three structural mechanisms are central to its emergence: p-hacking, selective reporting (publication bias), and underpowered studies.

0:0012:47
Get the next episode on Science →

Follow it free — new episodes land in your feed.

Or make your own — any topic, in minutes

More Onpode episodes on Science →

About this episode

The replication crisis refers to the widespread finding that many published scientific results cannot be reproduced by independent researchers. Three structural mechanisms are central to its emergence: p-hacking, selective reporting (publication bias), and underpowered studies.

Grounded in 12 sources
p-Hacking and Publication Bias Interact to Distort Meta-Analytic Effect Size Estimates ↗ · doi.org
Masquerading exploration as confirmation: Opening up Pandora’s Box of questionable research practices ↗ · doi.org
Power to Detect What? Considerations for Planning and Evaluating Sample Size ↗ · doi.org
Evidence that nonsignificant results are sometimes preferred: Reverse P-hacking or selective reporting? ↗ · doi.org
The past, present and future of Registered Reports ↗ · doi.org
Improving statistical reporting in psychology ↗ · doi.org
Opinion: Promoting open science ↗ · doi.org
Why Most Published Research Findings Are False | PLOS Medicine ↗ · journals.plos.org
How to Make More Published Research True | PLOS Medicine ↗ · journals.plos.org
A manifesto for reproducible science ↗ · nature.com
Degrees of Freedom in Planning, Running, Analyzing, and Reporting Psychological Studies: A Checklist to Avoid p -Hacking ↗ · ncbi.nlm.nih.gov
A framework for assessing the trustworthiness of scientific research findings ↗ · pmc.ncbi.nlm.nih.gov
Read transcript

Dr. Nathan Hayes: Sarah, good to be back — I have to say, I've been sitting with something uncomfortable all morning and I think it's the right place to start.

Sarah Lin: Oh, I want to hear this — what is it?

Dr. Nathan Hayes: So imagine a researcher — and this is not a hypothetical bad actor, this is a careful, methodologically literate person — runs twenty statistical tests on the same dataset. One of them hits p-less-than-0.05. She writes it up. Gets it published. Commits no fraud. Violates no written rule. And produces a finding that, in all likelihood, will not replicate. Now: what do we call that?

Sarah Lin: Mm. That's — it's not misconduct. It doesn't even feel like misconduct from the inside.

Dr. Nathan Hayes: No, it doesn't. And that's the question this episode is actually about — not fraud, not bad people, but a structure that makes this the rational thing to do.

Sarah Lin: Can I try to put a plain-language frame on the core mechanism? Because I think there's a simple version that makes everything else click. It's like... if you try twenty doors and only the one that opens gets written up, everyone reading the report concludes 'there's always a door here.' The locked doors are invisible.

Dr. Nathan Hayes: Yes. And that's essentially what John Ioannidis formalized in PLOS Medicine in 2005 — 'Why Most Published Research Findings Are False' — a probabilistic framework showing that when statistical power is low, researcher flexibility is high, and journals systematically favor positive results, the false positive can become the most common published outcome. Not the exception.

Sarah Lin: The most common. That stopped me.

Dr. Nathan Hayes: It should. Now, those three conditions Ioannidis names — low power, high flexibility, publication bias — they have specific clinical names. The flexibility piece is called p-hacking, and it's been operationalized: thirty-four distinct researcher degrees of freedom have been formally catalogued across design, analysis, and reporting stages. Varying your outcome measure, your subgroup, your exclusion criteria. Each choice is legitimate-seeming. The combination is not.

Sarah Lin: And the publication bias piece — that's journals only printing the doors that opened.

Dr. Nathan Hayes: Exactly — statistically significant, novel, positive results get published; null findings don't. So the two mechanisms aren't just parallel — they interact. And that interaction is, I'd argue, the under-discussed part of the replication crisis.

Sarah Lin: So what we're sitting with today is this: how does a careful person, working within normal field standards, generate a finding that probably isn't real — and what does the system have to do with that?

Dr. Nathan Hayes: But here's what I think we glossed over — thirty-four. That number. It's not a rough estimate. Simmons and colleagues formally catalogued thirty-four distinct decision points across design, data collection, analysis, and reporting. And the disturbing part isn't the count — it's that each individual choice looks, in isolation, completely defensible.

Sarah Lin: Wait — like what kind of choices?

Dr. Nathan Hayes: Switching from a two-tailed to a one-tailed test. Excluding two outliers. Adding a covariate you hadn't planned for. Any one of those — a reviewer would nod at it. Methodologically reasonable. But run enough of them opportunistically and you've, without committing a single act of fraud, produced a false positive that looks like a finding.

Sarah Lin: So the issue isn't any single door — it's that you have thirty-four keys and no one's tracking which ones you tried.

Dr. Nathan Hayes: Right — and this is where I want to complicate the word 'p-hacking,' because I think it conjures someone consciously gaming the system. What we're actually describing are questionable research practices — QRPs — which exist on a spectrum. Unplanned post hoc tests at one end. Selective outcome reporting further along. The intent differs. The downstream distortion to the literature is, in many cases, identical.

Sarah Lin: So where — mm, I want to sit with that for a second. Because that spectrum thing is uncomfortable. Like, at what point does 'I adapted to what my data was telling me' become something more troubling?

Dr. Nathan Hayes: There isn't a bright line. And Ioannidis's model — the PLOS Medicine paper — actually makes this point structurally: you don't need to invoke bad intent. The formal conditions he models are low power, researcher flexibility, publication bias. Under those conditions, false positives become modal. Most common outcome. That's without anyone deciding to cheat.

Sarah Lin: That's — yeah, that's the part that lands hardest. The incentive misalignment is doing the work that fraud would otherwise do.

Dr. Nathan Hayes: Exactly — career advancement, funding, publication all reward novelty and significance. Null results don't get you tenure. So QRPs become, and I think this word is precise, rational. Not ethical, necessarily, but rational given the system.

Sarah Lin: And that's — wait, actually that's what makes the 'teach better statistics' fix feel so thin. Because the researchers already know. It's not a knowledge gap.

Dr. Nathan Hayes: It's a choice architecture problem. The Nature Manifesto for Reproducible Science names preregistration and Registered Reports as structural interventions precisely because they constrain the thirty-four decision points before the data exist — not after. But here's what that reform conversation underweights: uptake depends on the same incentive structure we're trying to fix.

Sarah Lin: So the medicine requires the patient to already be healthy. Um — how do you actually break that loop?

Dr. Nathan Hayes: Breaking that loop — right, that's actually where Friese and Frankenbach's modeling work becomes uncomfortable to look at directly. Because they're not studying the loop. They're studying what happens inside it when two mechanisms run simultaneously.

Sarah Lin: Friese and Frankenbach — that's the non-linearity finding.

Dr. Nathan Hayes: Yes. So most people — including me, honestly, before sitting with the actual model — assume p-hacking plus publication bias compounds additively. More of both, proportionally more distortion. But that's not what they found. At medium levels of publication bias, p-hacking causes the most additional damage to meta-analytic effect size estimates. At high levels and low levels, p-hacking adds comparatively little.

Sarah Lin: Wait — at high publication bias, p-hacking adds less? That's... that's the counterintuitive part for me. I'd have expected the opposite.

Dr. Nathan Hayes: It's because at very high publication bias, the filter is already so aggressive that p-hacking is — I mean, the system is already selecting almost exclusively for the extreme results. P-hacking's marginal contribution shrinks. The publication bias is doing the heavy distortion work alone. But in the middle range? Both mechanisms are, sort of, partially engaged, and they amplify each other in ways the additive model doesn't predict.

Sarah Lin: So there's a perverse sweet spot.

Dr. Nathan Hayes: Exactly that. And the fields sitting in that sweet spot are specifically the ones studying small or near-zero true effects. Subtle phenomena. Which is — well, that's most of psychology. Most of mental health research.

Sarah Lin: Oh. So the distortion lands hardest where the effects are probably smallest to begin with. The people most likely to study something genuinely subtle are most likely to be reading a literature that's... kind of built on noise.

Dr. Nathan Hayes: And this is what troubles me about single-intervention reform narratives — the Manifesto for Reproducible Science in Nature Human Behaviour proposes preregistration, blinding, open materials as a coordinated suite. Which is right, as far as it goes. But if you only fix publication bias — say journals start accepting null results — and you don't simultaneously address p-hacking, the Friese and Frankenbach modeling suggests you may not move the needle much. The interaction structure means partial fixes leave most of the combined harm intact.

Sarah Lin: So 'more openness' as a slogan — um, it's not wrong, but it's aiming at one lever in a system where the levers are... entangled.

Dr. Nathan Hayes: Structurally insufficient. That's the precise problem. And it compounds in ways we haven't even touched — because the effect sizes coming out of this distorted literature are what researchers use to plan their next study's sample size. That inheritance is its own disaster, and we'll get to it.

Sarah Lin: Oh — yeah. The next layer. The number that gets handed down.

Dr. Nathan Hayes: That number that gets handed down — this is where the math becomes almost cruel. So a replication team reads the original paper. Sees d=0.78. Runs their power calculation off that. Designs a study accordingly. And they are structurally underpowered before a single participant walks in the door.

Sarah Lin: Wait — they don't know the true effect is d=0.2.

Dr. Nathan Hayes: They can't know. The simulation is stark on this — true effect d=0.2, n=20 per cell, and the studies that happen to clear the significance threshold report on average d=0.78. Nearly fourfold. That's the winner's curse. The original study wasn't lying, exactly. It was just the one that got lucky in the wrong direction.

Sarah Lin: Oh. So the replication is set up to fail by the very number it inherited.

Dr. Nathan Hayes: Inherited debt — I think that's actually the right frame. And what makes it worse is there's no consensus on how to fix the input. Do you use field-wide average effect sizes? Prior literature? Cost-effectiveness thresholds? Each one produces a different required sample size. Each is, in some sense, defensible. None is universally accepted. So power analysis — the tool that's supposed to prevent this — is itself running on contested assumptions.

Sarah Lin: So the fix has the same problem as the thing it's fixing.

Dr. Nathan Hayes: It's a choice architecture problem all the way down, yes.

Sarah Lin: Um — and this is what makes Registered Reports feel different to me, structurally. Because if peer review happens before data collection, the question of which effect size to use gets... examined before the results exist to rationalize it.

Dr. Nathan Hayes: That's exactly the mechanism. Registered Reports decouple the publication decision from the outcome. In-principle acceptance before you run a single participant. Which means the incentive to p-hack toward significance is — well, it's not eliminated, but it's substantially weakened. The Open Science Framework has built the preregistration infrastructure to support exactly this. Large-scale replication projects have run through OSF.

Sarah Lin: And yet — uptake in mental health research specifically is still limited. Which is the part that I keep, mm, circling back to. Because that's the field sitting in Friese and Frankenbach's distortion sweet spot.

Dr. Nathan Hayes: Right — and this is where I want to be honest about what we don't fully know. The adoption lag isn't — I mean, it's not because mental health researchers are unaware of Registered Reports. It's because a tenure committee that doesn't weight them equivalently to a high-impact publication makes adoption irrational. The structural remedy requires a structural receiving environment that doesn't yet exist.

Sarah Lin: So the researcher who does everything right — preregisters, publishes the null — is, in some real sense, penalized by the very system the reform is trying to change.

Dr. Nathan Hayes: Structurally, yes. And that's the argument the Nature Manifesto for Reproducible Science was making — culture-change rhetoric without structural backing doesn't scale. Incentives shape culture, not the reverse.

Sarah Lin: Which means the literature people in mental health are actually reading — the meta-analyses, the treatment guidelines — those are built on effect sizes that may be, like, deeply inflated. And the people building interventions off that literature don't know to discount them.

Dr. Nathan Hayes: That's the clinical consequence. And it's not abstract — underpowered original studies, inflated effect sizes, replications that inherit the inflation and fail, and a treatment evidence base that reflects the winner's curse more than the actual effect. That's the floor we're standing on.

Sarah Lin: And we haven't even asked what it would take to actually shift what counts as success inside the system — which feels like the question underneath all of this.

Dr. Nathan Hayes: And I don't think we can answer it. Not cleanly. Because the upstream question — what counts as scientific success — that's held in place by journals, funding agencies, and tenure systems all simultaneously rewarding novelty and statistical significance. You'd have to move all three at once. Changing one without the others just... shifts where the pressure goes.

Sarah Lin: That researcher on a Tuesday evening, running those extra tests — she wasn't failing the system. She was succeeding at it. Um. That's the thing I can't quite put down.

Dr. Nathan Hayes: And we keep building better tools — preregistration, Registered Reports, the OSF infrastructure — downstream of that upstream question. Which is not nothing. But it's... I mean, Ioannidis named the conditions in 2005. Twenty years on, the conditions are largely intact.

Sarah Lin: So the literature people are actually treating patients from — that's where we land. Genuinely unresolved.

Dr. Nathan Hayes: It is. And I think that's the honest place — not pessimism, just... the problem is correctly sized now. That's something.

Sarah Lin: Yeah. Thank you for thinking through it with me — I mean that. This one sat heavier than I expected.