Nina Park: Walt, tell me you saw the thing about the replication numbers, because I've been weirdly quiet about it all week and I think I need to actually say it out loud to someone.
Walt Garner: Indeed. Go on.
Nina Park: In 2015 — and this is a real number, not a rhetorical number — researchers tried to replicate 100 published psychology findings. Original methods, original designs. Thirty-six percent replicated. And I've been sitting with that and what gets me is not the 36%, it's the explanation. Or the lack of one. Because the original researchers weren't doing anything wrong.
Walt Garner: Which is, you see, a more troubling fact than fraud would be.
Nina Park: Wait, say that again — why is it more troubling?
Walt Garner: Well, fraud has a remedy. You identify the bad actor, you retract the paper, you move on. But if the original researchers followed standard practices in psychology and the results still don't survive a second look — the problem is inside the standard practices themselves. There's nothing to retract. There's no one to blame. The results simply... are not where we thought they were.
Nina Park: And psychology is the field where we're seeing this most clearly — these large-scale replication initiatives, the policy changes at journals, the whole vocabulary that came out of it. But I keep wondering if that's because psychology is genuinely worse, or because it's the one discipline that actually looked.
Walt Garner: Mm, that distinction matters enormously — because if it's the latter, the 36% isn't a floor, it's a ceiling on what we know.
Nina Park: Which is a sentence I want to just let sit there for a second.
Walt Garner: Yes. So the real puzzle — and I think this is what this conversation is actually about — is: if everyone played by the rules, how do the results disappear?
Nina Park: Okay but that's actually the part I want to pull on — because when you say 'played by the rules,' the rules themselves have thirty-four decision points baked in. Like, someone actually catalogued them. Published a 34-item checklist — hypothesis formulation, study design, which statistical test, how to handle outliers, when to stop collecting participants — thirty-four.
Walt Garner: Thirty-four in psychology alone.
Nina Park: Right — but the part that doesn't fit neatly is that none of them are cheating. Not a single one. Imagine a graduate student — call her Maya — she finishes data collection on a Friday. Runs the analysis. Not significant. She removes two outliers she has a genuinely defensible reason to exclude. Still not quite there. She checks whether the effect is stronger in the female subsample. It is. She reports that finding. She hasn't lied once.
Walt Garner: And that — that is p-hacking. Not as a moral failing. As an architecture.
Nina Park: Which is the click, right? That's the thing.
Walt Garner: Now, the interesting part is what you lose when Maya writes the paper. Because she reports one analysis — one path. But the dataset permitted, legitimately, dozens of branching choices at every stage. This is what gets called forking paths — any dataset contains a garden of defensible analyses, and you only ever see the one that made it to print.
Nina Park: And the ones that didn't reach p = 0.05 just... don't exist, as far as the literature is concerned.
Walt Garner: Indeed. And if you look across an entire literature — thousands of published findings — you see suspicious spikes in the p-value distribution just below 0.05. Not one study behaving strangely. The whole field tilting.
Nina Park: Wait, you can see it in the distribution? Like statistically detectable?
Walt Garner: Graham Elliott, Nikolay Kudrin, and Kaspar Wüthrich actually worked out the theoretical power of exactly that test — detecting p-hacking through the shape of p-value distributions. The bump just under the threshold is not random noise.
Nina Park: So the cookbook looks clean, but if you line up a thousand cookbooks and count how many recipes end right at 'perfect,' the number is — wrong.
Walt Garner: That is, you see, precisely the image. And what it means is the result doesn't disappear from replication because someone lied. It disappears because the specific path Maya took through those thirty-four decision points was never disclosed — and no replicator can reconstruct it without knowing which fork she chose.
Nina Park: Which means replication isn't even testing the same thing. It's testing a different path through the same garden and calling it a check on the original.
Walt Garner: And that gap — between the path taken and the path tested — is where statistical power comes in, because power is actually the thing that determines whether any of this resolves honestly. What power means, specifically: it's the probability that your study detects a real effect if one actually exists. Not whether it finds significance. Whether it finds the real thing.
Nina Park: Which most studies are not doing well at.
Walt Garner: Many are chronically underpowered, yes. Small samples, because collecting data is expensive and slow. And here's what that produces — and this is the mechanism that I think is genuinely underappreciated — an underpowered study usually fails to reach significance. Most of the time it just goes nowhere. But occasionally, by chance, it tips over the threshold. And when it does, the effect size it reports is almost certainly inflated.
Nina Park: Wait — inflated how? Like it found the thing but made it bigger than it is?
Walt Garner: Precisely that. The sample was too small to find the true signal reliably, so the ones that did cross the line were the runs where noise happened to amplify the effect. You're selecting for lucky flukes. Which means the published effect size is not the real effect size — it's the real effect size plus a luck premium. This is what gets called the winner's curse.
Nina Park: Oh — that's a brutal name for it, actually. The winner's curse. You clear the bar and the prize is that you've overstated your finding.
Walt Garner: Now, hold that image, because the consequence for replication is direct. A replicator comes along, runs the same study — but properly powered, three times the participants. They find an effect, maybe, but smaller. Or nothing at all. The headline reads 'study fails to replicate.' But the original researcher wasn't fraudulent. The original finding may have pointed at something real. It just overstated how large it was, because the study was never powered to land accurately.
Nina Park: So the replication isn't failing the original study. The original study was already — I mean, it was honest, but it was optimistic in a way it couldn't even know it was being optimistic.
Walt Garner: Mm. Structurally optimistic.
Nina Park: And then — okay, this is the part that tips it from 'unfortunate' into 'actually a problem with the whole literature' — publication bias. Because the inflated finding gets published, the underpowered study that found nothing doesn't, and everyone reading the field is looking at a library that's been quietly filtered toward the most optimistic results. Annual Reviews actually documented this same distortion running through biomedical research, not just psychology. So it's not a psychology-specific glitch.
Walt Garner: Which is the uncomfortable implication of the Annual Reviews work, yes — the same incentive structures, the same filters, same inflation. Now, Maximilian Mandl and colleagues proposed a technical response to at least part of this — the minP adjustment — which tries to formally treat undisclosed researcher choices as a multiple-testing problem. Each hidden fork in the analysis is, in effect, an additional comparison, and the method attempts to correct for that. It is, you see, a narrow instrument. It doesn't fix the filter. It just tries to name what the filter costs.
Nina Park: A patch on the symptom, not — yeah. And what strikes me is that none of the technical corrections touch why the inflated findings were the ones that got through in the first place. Which is a story about journals and tenure committees and what counts as a career — and that's actually where this gets worse, because the institutional machinery around all of this is its own whole thing.
Walt Garner: Indeed — the selection pressure from the journals, the replicator's own degrees of freedom, what PNAS found about how misleading failures to replicate get produced. That's the layer we haven't fully opened yet.
Nina Park: But the PNAS finding is the one that actually broke something for me — because we've been talking as if the problem lives on the original researcher's side. And it doesn't. Only on that side.
Walt Garner: No, indeed. The replicator has thirty-four decision points of their own.
Nina Park: Wait — the same flexibility?
Walt Garner: Precisely the same class of choices — participant selection, procedural details, which analysis to run. PNAS published a study demonstrating that a replicator can design, analyze, and publish an attempt that looks methodologically airtight and still produce a misleading failure. Not by cheating. By choosing.
Nina Park: So neither result can be trusted at face value without seeing the full decision trail. The original or the replication.
Walt Garner: And adjudicating any single dispute becomes genuinely impossible — failure could mean a false positive in the original, or a poorly calibrated replication, and the published record almost never contains enough detail to tell which.
Nina Park: Which — okay, I mean, that's the vertigo of it. The audit tool has the same problem as the thing it's auditing.
Walt Garner: And Elliott, Kudrin, and Wüthrich showed that even the statistical tests built to detect p-hacking through p-value distributions — those audits — have low and variable power depending on which hacking strategy was used. The detection instruments are themselves unreliable.
Nina Park: That's — that's the floor dropping out.
Walt Garner: Now, what you have to ask is: what is the engine underneath all of this? And the answer — this is what I find the hardest to sit with — is publish-or-perish. University hiring, promotion, grant funding, prestige: all of it runs on publication counts and novel positive findings. A researcher who slows down to prioritize rigor over significance is, within that system, behaving irrationally. They are at a competitive disadvantage.
Nina Park: No individual dishonesty required. The aggregate literature just — warps. Because everyone's being sensible.
Walt Garner: Rational behavior within the existing reward system produces the distortion automatically. You see, that's the mechanism. Not bad actors breaking a good system — good actors, functioning exactly as the system trained them, are the mechanism.
Nina Park: That's — yeah. That's the hardest version of it. Because it means there's no intervention point that doesn't ask researchers to absorb a personal cost that the institution isn't absorbing with them.
Walt Garner: Pre-registration, registered reports — they're being asked to adopt those reforms while tenure committees are still counting positive publications. The incentive structure hasn't moved. Which is why the distortion is self-reproducing. Not self-correcting.
Nina Park: And that's where pre-registration is supposed to come in. You publicly commit — before you collect a single data point — here is my hypothesis, here is my design, here is exactly which analysis I will run. So Maya's thirty-four decision points get locked down in advance. There's even a stronger version, registered reports, where the journal peer-reviews your introduction and methods before you have results, and provisionally accepts the paper regardless of what you find. The outcome can't drive the story because no one knows the outcome yet.
Walt Garner: And the registered reports piece is the structurally interesting move, because it breaks publication bias at the source — the journal's own selection incentive.
Nina Park: Right — but the critique I can't shake is that pre-registration might just relocate the problem. If your flexibility gets cut off after registration, you still have all the flexibility in the world about how you design the study before you register it. The hidden choices move earlier. They don't disappear.
Walt Garner: The garden of forking paths, moved upstream.
Nina Park: Which — I mean, that's genuinely unsettling, because the reform is real, it does something, and still the conflict underneath it hasn't moved at all. Pre-register your study, sure, be credible, yes. And also publish positive findings if you want your career to survive. Those two things are both completely rational. They're both things I would want if I were that researcher. And they're just — they're not compatible. Not under the system as it currently exists.
Walt Garner: No. And that is not a methods question. What would have to change for a null result in a rigorously pre-registered study to be as career-building as a novel positive finding? That's an institutional question. A political one, almost.
Nina Park: And I don't have an answer. I've been turning it over and I genuinely don't know what that world looks like — what the tenure committee email reads like, what the grant review panel is optimizing for. I don't know how you get there from here.
Walt Garner: Nor do I, honestly. Which is, you see, a strange place to end — not with a solution in hand but with the shape of the problem finally clear. I'll take that, I think.
