Felix Ortiz: Hey, ready? I want to start with a number and just let it sit for a second, because I've been staring at it: 96%.
Tess Hollis: Ninety-six percent of what? Give me the denominator.
Felix Ortiz: Standard psychology papers that report positive results. And then Registered Reports, same sort of researchers, same questions, only 44%. Yeah, the only difference is when the journal commits, before the data or after.
Tess Hollis: Wait. That's the part that bothers me. Studies typically catch a real effect about half the time. Even if every idea tested were true, you'd see around 50%, never 96.
Felix Ortiz: Right, so, picture a wall of five-star reviews at a restaurant, and the unhappy diners never got a card to fill in. Perfect wall, and it tells you about who got to post, not about the meals.
Tess Hollis: So the literature is the wall.
Felix Ortiz: And every 'study finds' headline is a survivor. Nobody has to be lying for that. But how does a filter like that get built in the first place?
Tess Hollis: A filter like that leaves fingerprints, and in 2015 someone went looking. Brian Nosek's group, the Center for Open Science, ran the Reproducibility Project: Psychology, published in Science. 270 researchers redid 100 published studies.
Felix Ortiz: And the score?
Tess Hollis: 97% of the originals were significant. Only 36 to 39% of the replications were.
Felix Ortiz: Okay, that's brutal. But, I mean, the part I keep snagging on is the effect sizes, they came back at about half. Why does halving matter more than just, you know, fewer hits? Is that, wait, what does it actually tell us?
Tess Hollis: Noise would scatter. Some replications would come in bigger, some smaller, and it'd average out. Halving is one direction, every time.
Felix Ortiz: Oh. So the originals were inflated. Like they only got printed when they landed lucky-high.
Tess Hollis: Selection, yes. Publication bias, null results go in the drawer, significant ones fill the journals. That's the answer most people stop at.
Felix Ortiz: Yeah, and it's a good answer, I love it, it's tidy. But, okay, that's not the whole story. Say you fixed it tomorrow, every null result gets published. Honest researchers can still, I mean, manufacture false positives on the way in. Try three outlier rules, two outcome measures, and one of them hits p under .05. Nobody lied, and the null never even got written down.
Tess Hollis: So the drawer isn't only at the journal. It's inside the analysis.
Felix Ortiz: Right, and one more wrinkle: Ioannidis and others put pre-clinical and life-science reproducibility at only 10 to 25%.
Tess Hollis: Worse than psychology. But hold on, the crisis is best documented in psychology and pre-clinical medicine. Economics, physics, honestly unclear.
Felix Ortiz: Fair. So how bad can honest flexibility get? Is there actual math for that?
Tess Hollis: There is. But picture it first. Tuesday, 9 a.m. A third-year grad student has 20 participants per condition and p = .07. She wonders if two outliers were just inattentive, drops them, and gets p = .04. Her advisor says "great, write it up." Fellowship renewal is in March.
Felix Ortiz: Okay, but, where's the lie? Yeah, I'm looking for it and I can't find it. Inattentive people exist, right? Dropping them is, I mean, it's reasonable.
Tess Hollis: There isn't one. That's the point. Now add power. A study with 50% power has a coin-flip chance of catching a real effect. And in 2005 John Ioannidis published "Why Most Published Research Findings Are False" in PLoS Medicine. He did the arithmetic on exactly this.
Felix Ortiz: Walk me through it. Slowly. I'll get lost.
Tess Hollis: Say 1 in 10 tested hypotheses is true. Power is 50%, alpha is .05. Out of 1,000 ideas, 100 are true, so 50 get detected. Of the 900 false ones, 5% hit by chance. That's 45. So 50 real, 45 fake.
Felix Ortiz: So about 53% of the significant ones are real. Which is already, yeah, barely better than a coin. And that's the honest version?
Tess Hollis: Honest p-values. Now let flexible analysis in. Outlier rules, subgroups, how you define the variable. That's p-hacking, researcher degrees of freedom. Effective false-positive rate goes to 20%. Now 180 false hits, 50 real.
Felix Ortiz: Hold on. So only about 22% are real?
Tess Hollis: Roughly four in five "discoveries" wrong. Zero fraud.
Felix Ortiz: That's, wow. Okay, but, I love the no-villains thing, and what's the other side of it? Some of this has to be sloppy, or motivated. "Structural" can be a really comfortable way to dodge accountability, no?
Tess Hollis: Fair. Some people do know better. But look at the arithmetic: it needs none of them. Sincere people produce it. And why nobody fixes it, that's the part that gets stranger, and then a 2022 paper complicates the story again.
Felix Ortiz: So if the arithmetic needs no villains, then why hasn't anyone fixed it? Everybody knows. Yeah, I mean, the first person who runs a big, honest study that might come back null, they just lose the publication race, right?
Tess Hollis: They do. And there's a name for the mechanism. Smaldino and McElreath treated it like evolution. Labs that publish more survive, get grants, train students. Their methods get copied. Nobody intends it.
Felix Ortiz: Oh, so bad methods are like, a trait. Labs optimize for publication counts, journals for citations, and both pay for novelty, so the sloppy-but-fast lab just out-reproduces the careful one. Wait, no, out-reproduces is a terrible word here, but you know what I mean.
Tess Hollis: It's accurate, actually. And replication is the correction mechanism that should break the loop. Mueller-Langer and colleagues checked the top 50 economics journals from 1974 to 2014. Replications were 0.1% of publications.
Felix Ortiz: One in a thousand. Forty years. Nobody's career runs on that.
Tess Hollis: So the check barely exists. Now the twist.
Felix Ortiz: Okay, hold on, let me say the standard story first, so you can knock it. Publication bias is the main driver. Journals bury nulls, the literature fills with lucky hits, replications look bad. That's the account I'd have defended an hour ago, yeah.
Tess Hollis: Vu, 2022, modeled selective publication directly. Suppress the insignificant results and the replication rate doesn't move. What explains the gap is power. Miscalculated power accounts for all of it in experimental economics, and about two-thirds in psychology.
Felix Ortiz: Wait, what? The drawer doesn't matter?
Tess Hollis: For the replication rate, apparently not. The gap is between the power people intended and the rate they actually got. But okay, you just stepped around the thing. Who chose the sample size?
Felix Ortiz: Oh. Oh, the researcher did. And if novelty is what pays, you pick the cheapest design that might clear p under .05, and, yeah, cheap means small, and small means underpowered. So power isn't the rival to the incentive story, it's, it's the incentive showing up at design time. Before any journal sees anything.
Tess Hollis: Same cause, earlier in the pipeline. One caveat on the scoreboard, though. McShane and colleagues argue exact replications don't exist. Different sample, different lab, so pass or fail is shakier than it sounds.
Felix Ortiz: Yeah, and that makes me want to ask what actually moves the commitment point. Because if the filter is at design and at analysis, then fixing the journal at the end is, I mean, it's too late in the pipeline, right?
Tess Hollis: That's what the reforms try to do. Pre-registration: you publicly commit to hypotheses, sample size, and analysis plan before any data. Then the outlier rule she picked on Tuesday is visible as a choice after the fact. Registered Reports go further. Peer review and acceptance happen before data, on the question and the method.
Felix Ortiz: Which is the 44% against 96%. Oh, that's the whole gap, isn't it, the journal can't wait to see whether it likes the answer.
Tess Hollis: Yes. And the change is uneven. Roger Giner-Sorolla documents new methods and reporting standards in social psychology, real ones. But some corners have barely moved.
Felix Ortiz: I can't shake my surprise from earlier. If Registered Reports come back around 56% null, that's, wait, those are the well-designed studies. So nulls aren't the failure case. They're the normal state of science. And the literature has been hiding, what, more than half the map.
Tess Hollis: More than half the map. Huh.
Felix Ortiz: Yeah. Like the five-star wall from the start. We thought the empty spots were bad meals, and they were just cards nobody handed out.
Tess Hollis: Then stakes. ENTRUST-PE is a European Commission-funded network looking at how trustworthy pain research is. Unreliable findings feed low-value care. Real patients, real treatments built on survivors.
Felix Ortiz: Ugh. That makes the map thing less cute.
Tess Hollis: So here's my question. What would a tenure committee have to see for a null to count as a career asset?
Felix Ortiz: I've been trying to answer that while you talked. A pre-registered, well-powered null... counted like a paper? Like a real line on the CV? But then who sits on the committee, people who got there on the lucky hits. I don't have it.
Tess Hollis: Neither do I. Pre-registration might just be paperwork if nobody gets hired for it.
Felix Ortiz: Yeah. I keep picturing that wall of reviews, and the honest card, the three-star one, finally being allowed up there. And I still can't tell whether anyone will reward the unsurprising result. Maybe that's the thing I'll go home turning over.