Onpode
Cover art for The replication crisis isn't fraud — it's how incentives and design compound

The replication crisis isn't fraud — it's how incentives and design compound

November 22, 2026 · 12 min

Tess Hollis & Felix Ortiz

Standard psychology journals report 96% positive results; Registered Reports from the same researchers return only 44%. The gap isn't fraud — it's underpowered study designs and flexible analysis compounding publication bias, leaving more than half the scientific map hidden by incentive structures, not misconduct.

The replication crisis refers to a widespread and documented failure of published scientific findings — particularly in psychology, medicine, and economics — to hold up when independently repeated. The landmark empirical demonstration came from the Open Science Collaboration's 2015 Reproducibility Project: Psychology, which mobilized 270 researchers to replicate 100 published psychology studies.

0:0011:31
Get the next episode on Science →

Follow it free — new episodes land in your feed.

Or make your own — any topic, in minutes

More Onpode episodes on Science →

About this episode

Most people assume the replication crisis is a story about bad actors — data fabricators, p-hackers gaming the system for tenure. This episode makes a harder argument: the arithmetic produces it without any of that. When only 1 in 10 hypotheses is true, power is 50%, and analysts have even modest flexibility in how they define outcomes or handle outliers, roughly four in five published 'discoveries' can be wrong — with every researcher acting in complete good faith. The episode works through the numbers slowly and honestly, from Brian Nosek's Reproducibility Project (100 studies redone, effect sizes coming back at half) to John Ioannidis's probability model to a 2022 finding that upends the standard story: suppressing null results barely explains the replication gap. What actually accounts for it is underpowered study design — and underpowering is itself an incentive problem, since cheap, small studies are the ones that survive a funding landscape that rewards novelty and speed. There are real reforms — pre-registration, Registered Reports — and real evidence they shift the numbers. But the episode ends on an open question: pre-registration might just be paperwork if null results don't count on a CV, and the people sitting on tenure committees are largely the ones who got there on the lucky hits. Worth 12 minutes of your attention, especially if you've ever read a 'study finds' headline and wondered what happened to all the studies that didn't.

Frequently asked

What is the replication crisis and why does it happen?

The replication crisis is the finding that a large share of published scientific results fail when independently repeated. It is driven primarily by underpowered study designs, flexible analysis choices (p-hacking), and publication bias that buries null results — not by deliberate fraud from researchers.

What did the Reproducibility Project in psychology find?

The Reproducibility Project: Psychology, led by Brian Nosek's Center for Open Science and published in Science, retested 100 published studies. While 97% of originals were statistically significant, only 36–39% of replications were, and effect sizes came back at roughly half the original magnitude.

What is p-hacking and how much does it inflate false discoveries?

P-hacking is the practice of flexibly adjusting outlier rules, outcome measures, or subgroups until a result crosses p < .05. John Ioannidis's arithmetic shows that with 50% statistical power and a 10% prior rate of true hypotheses, honest analysis yields ~53% real findings; p-hacking drops that to roughly 22%.

Do Registered Reports actually reduce false positives in research?

Registered Reports, where journals peer-review and accept studies before data collection, return positive results about 44% of the time, compared to 96% for standard publications. The difference suggests most of the conventional literature's positive results reflect selection at the journal stage, not the true signal rate.

Is publication bias or low statistical power the bigger driver of failed replications?

A 2022 modeling study by Vu found that suppressing null results alone does not move the replication rate; underpowered designs explain all of the gap in experimental economics and about two-thirds in psychology. Power is itself an incentive problem — small, cheap studies are chosen because novelty, not rigor, drives publication.

Grounded in 12 sources
Large Scale Replication Projects in Contemporary Psychological Research ↗ · arxiv.org
Beyond the p < 0.05 trap: the adaptive integrity model for preventing and detecting P-hacking ↗ · doi.org
Are most published research findings false? Trends in statistical power, publication selection bias, and the false discovery rate in psychology (1975–2017) ↗ · doi.org
Early Experimental Psychology: How did Replication Work Before P-Hacking? ↗ · doi.org
Are lessons being learnt from the replication crisis or will the revolution devour its children? Open Q science from the editor's perspective ↗ · doi.org
Can the Replication Rate Tell Us About Publication Bias? ↗ · doi.org
Publication bias and the canonization of false facts ↗ · doi.org
Changing Practices and Priorities in Social Psychological Research Methods and Reporting ↗ · doi.org
Replication studies in economics—How many and which papers are chosen for replication, and why? ↗ · doi.org
ENTRUST-PE: An Integrated Framework for Trustworthy Pain Evidence. White Paper. ↗ · doi.org
The Uptake of Preregistration, and Its Impact on the Proportion of Supported Hypotheses, in Sports Science | Sports Medicine | Springer Nature Link ↗ · link.springer.com
John P.A. Ioannidis' Profile | Stanford Profiles ↗ · profiles.stanford.edu
Read transcript

Felix Ortiz: Hey, ready? I want to start with a number and just let it sit for a second, because I've been staring at it: 96%.

Tess Hollis: Ninety-six percent of what? Give me the denominator.

Felix Ortiz: Standard psychology papers that report positive results. And then Registered Reports, same sort of researchers, same questions, only 44%. Yeah, the only difference is when the journal commits, before the data or after.

Tess Hollis: Wait. That's the part that bothers me. Studies typically catch a real effect about half the time. Even if every idea tested were true, you'd see around 50%, never 96.

Felix Ortiz: Right, so, picture a wall of five-star reviews at a restaurant, and the unhappy diners never got a card to fill in. Perfect wall, and it tells you about who got to post, not about the meals.

Tess Hollis: So the literature is the wall.

Felix Ortiz: And every 'study finds' headline is a survivor. Nobody has to be lying for that. But how does a filter like that get built in the first place?

Tess Hollis: A filter like that leaves fingerprints, and in 2015 someone went looking. Brian Nosek's group, the Center for Open Science, ran the Reproducibility Project: Psychology, published in Science. 270 researchers redid 100 published studies.

Felix Ortiz: And the score?

Tess Hollis: 97% of the originals were significant. Only 36 to 39% of the replications were.

Felix Ortiz: Okay, that's brutal. But, I mean, the part I keep snagging on is the effect sizes, they came back at about half. Why does halving matter more than just, you know, fewer hits? Is that, wait, what does it actually tell us?

Tess Hollis: Noise would scatter. Some replications would come in bigger, some smaller, and it'd average out. Halving is one direction, every time.

Felix Ortiz: Oh. So the originals were inflated. Like they only got printed when they landed lucky-high.

Tess Hollis: Selection, yes. Publication bias, null results go in the drawer, significant ones fill the journals. That's the answer most people stop at.

Felix Ortiz: Yeah, and it's a good answer, I love it, it's tidy. But, okay, that's not the whole story. Say you fixed it tomorrow, every null result gets published. Honest researchers can still, I mean, manufacture false positives on the way in. Try three outlier rules, two outcome measures, and one of them hits p under .05. Nobody lied, and the null never even got written down.

Tess Hollis: So the drawer isn't only at the journal. It's inside the analysis.

Felix Ortiz: Right, and one more wrinkle: Ioannidis and others put pre-clinical and life-science reproducibility at only 10 to 25%.

Tess Hollis: Worse than psychology. But hold on, the crisis is best documented in psychology and pre-clinical medicine. Economics, physics, honestly unclear.

Felix Ortiz: Fair. So how bad can honest flexibility get? Is there actual math for that?

Tess Hollis: There is. But picture it first. Tuesday, 9 a.m. A third-year grad student has 20 participants per condition and p = .07. She wonders if two outliers were just inattentive, drops them, and gets p = .04. Her advisor says "great, write it up." Fellowship renewal is in March.

Felix Ortiz: Okay, but, where's the lie? Yeah, I'm looking for it and I can't find it. Inattentive people exist, right? Dropping them is, I mean, it's reasonable.

Tess Hollis: There isn't one. That's the point. Now add power. A study with 50% power has a coin-flip chance of catching a real effect. And in 2005 John Ioannidis published "Why Most Published Research Findings Are False" in PLoS Medicine. He did the arithmetic on exactly this.

Felix Ortiz: Walk me through it. Slowly. I'll get lost.

Tess Hollis: Say 1 in 10 tested hypotheses is true. Power is 50%, alpha is .05. Out of 1,000 ideas, 100 are true, so 50 get detected. Of the 900 false ones, 5% hit by chance. That's 45. So 50 real, 45 fake.

Felix Ortiz: So about 53% of the significant ones are real. Which is already, yeah, barely better than a coin. And that's the honest version?

Tess Hollis: Honest p-values. Now let flexible analysis in. Outlier rules, subgroups, how you define the variable. That's p-hacking, researcher degrees of freedom. Effective false-positive rate goes to 20%. Now 180 false hits, 50 real.

Felix Ortiz: Hold on. So only about 22% are real?

Tess Hollis: Roughly four in five "discoveries" wrong. Zero fraud.

Felix Ortiz: That's, wow. Okay, but, I love the no-villains thing, and what's the other side of it? Some of this has to be sloppy, or motivated. "Structural" can be a really comfortable way to dodge accountability, no?

Tess Hollis: Fair. Some people do know better. But look at the arithmetic: it needs none of them. Sincere people produce it. And why nobody fixes it, that's the part that gets stranger, and then a 2022 paper complicates the story again.

Felix Ortiz: So if the arithmetic needs no villains, then why hasn't anyone fixed it? Everybody knows. Yeah, I mean, the first person who runs a big, honest study that might come back null, they just lose the publication race, right?

Tess Hollis: They do. And there's a name for the mechanism. Smaldino and McElreath treated it like evolution. Labs that publish more survive, get grants, train students. Their methods get copied. Nobody intends it.

Felix Ortiz: Oh, so bad methods are like, a trait. Labs optimize for publication counts, journals for citations, and both pay for novelty, so the sloppy-but-fast lab just out-reproduces the careful one. Wait, no, out-reproduces is a terrible word here, but you know what I mean.

Tess Hollis: It's accurate, actually. And replication is the correction mechanism that should break the loop. Mueller-Langer and colleagues checked the top 50 economics journals from 1974 to 2014. Replications were 0.1% of publications.

Felix Ortiz: One in a thousand. Forty years. Nobody's career runs on that.

Tess Hollis: So the check barely exists. Now the twist.

Felix Ortiz: Okay, hold on, let me say the standard story first, so you can knock it. Publication bias is the main driver. Journals bury nulls, the literature fills with lucky hits, replications look bad. That's the account I'd have defended an hour ago, yeah.

Tess Hollis: Vu, 2022, modeled selective publication directly. Suppress the insignificant results and the replication rate doesn't move. What explains the gap is power. Miscalculated power accounts for all of it in experimental economics, and about two-thirds in psychology.

Felix Ortiz: Wait, what? The drawer doesn't matter?

Tess Hollis: For the replication rate, apparently not. The gap is between the power people intended and the rate they actually got. But okay, you just stepped around the thing. Who chose the sample size?

Felix Ortiz: Oh. Oh, the researcher did. And if novelty is what pays, you pick the cheapest design that might clear p under .05, and, yeah, cheap means small, and small means underpowered. So power isn't the rival to the incentive story, it's, it's the incentive showing up at design time. Before any journal sees anything.

Tess Hollis: Same cause, earlier in the pipeline. One caveat on the scoreboard, though. McShane and colleagues argue exact replications don't exist. Different sample, different lab, so pass or fail is shakier than it sounds.

Felix Ortiz: Yeah, and that makes me want to ask what actually moves the commitment point. Because if the filter is at design and at analysis, then fixing the journal at the end is, I mean, it's too late in the pipeline, right?

Tess Hollis: That's what the reforms try to do. Pre-registration: you publicly commit to hypotheses, sample size, and analysis plan before any data. Then the outlier rule she picked on Tuesday is visible as a choice after the fact. Registered Reports go further. Peer review and acceptance happen before data, on the question and the method.

Felix Ortiz: Which is the 44% against 96%. Oh, that's the whole gap, isn't it, the journal can't wait to see whether it likes the answer.

Tess Hollis: Yes. And the change is uneven. Roger Giner-Sorolla documents new methods and reporting standards in social psychology, real ones. But some corners have barely moved.

Felix Ortiz: I can't shake my surprise from earlier. If Registered Reports come back around 56% null, that's, wait, those are the well-designed studies. So nulls aren't the failure case. They're the normal state of science. And the literature has been hiding, what, more than half the map.

Tess Hollis: More than half the map. Huh.

Felix Ortiz: Yeah. Like the five-star wall from the start. We thought the empty spots were bad meals, and they were just cards nobody handed out.

Tess Hollis: Then stakes. ENTRUST-PE is a European Commission-funded network looking at how trustworthy pain research is. Unreliable findings feed low-value care. Real patients, real treatments built on survivors.

Felix Ortiz: Ugh. That makes the map thing less cute.

Tess Hollis: So here's my question. What would a tenure committee have to see for a null to count as a career asset?

Felix Ortiz: I've been trying to answer that while you talked. A pre-registered, well-powered null... counted like a paper? Like a real line on the CV? But then who sits on the committee, people who got there on the lucky hits. I don't have it.

Tess Hollis: Neither do I. Pre-registration might just be paperwork if nobody gets hired for it.

Felix Ortiz: Yeah. I keep picturing that wall of reviews, and the honest card, the three-star one, finally being allowed up there. And I still can't tell whether anyone will reward the unsurprising result. Maybe that's the thing I'll go home turning over.

The replication crisis isn't fraud — it's how incentives and design compound · Onpode