Ben Okonkwo: Eleanor — okay, I have to tell you, I started reading for today and I got genuinely unsettled in a way I didn't expect. Like, not intellectually unsettled. Actually unsettled.
Eleanor Crane: That's a good sign, usually. What was the specific thing?
Ben Okonkwo: The ninety-seven percent. In the Open Science Collaboration's 2015 project — Brian Nosek, 250-plus researchers, a hundred studies from three major psychology journals — ninety-seven percent of the originals had reported statistically significant results. Every single paper was a hit. And then when they actually ran the replications, thirty-six percent held up.
Eleanor Crane: Ninety-seven percent originally. Thirty-six percent replicated.
Ben Okonkwo: And the unsettling part isn't the thirty-six — it's the ninety-seven. That's not a distribution you get from honest error. That's selection so thorough it looks like a design feature.
Eleanor Crane: It's a system that was working — just not at finding true things. Working perfectly at finding publishable things.
Ben Okonkwo: Exactly that. And the replication effect sizes averaged roughly half the magnitude of the originals — so even for studies where something real was probably there, the original result was inflated. You can't build reliably on a foundation that's systematically twice the actual height.
Eleanor Crane: The restaurant analogy I keep reaching for — someone got a glowing review years ago, word spread, the reputation stuck. Nobody checked that it's been closed. People are still recommending it. That's what a citation in 2024 to an unreplicated 2009 study looks like.
Ben Okonkwo: And ninety percent of researchers, when surveyed, say they know there's a replication crisis. Ninety percent. The restaurant is visibly, provably closed.
Eleanor Crane: So the question — the one I can't get out of my head — is not 'did scientists not know?' They knew. It's: why would anyone fix a system that professionally rewards you for not fixing it?
Ben Okonkwo: Right — and I want to push on whether that's the whole story, because there's also a colder, more mechanical version of this. The math of p-values under low statistical power just predicts this outcome. You don't even need perverse incentives. You just need underpowered studies and a publication filter. That's enough.
Eleanor Crane: And those two explanations — the institutional and the mathematical — they don't feel like the same problem to me. Which is maybe the thing we're actually trying to sort out.
Ben Okonkwo: Right — but separating those two is exactly what I want to do, because I think publication bias is where most people stop. Journals only publish the hits. So the record is skewed toward positive results. That's the surface explanation, and it's real, but it's not — wait, actually it's not enough to explain the decay.
Eleanor Crane: The effect size decay. The replication effects averaging half the original magnitude.
Ben Okonkwo: Exactly that. Because publication bias alone — if that's all it was — you'd expect a binary pattern. Either the effect is real and it shows up in replication, or it's noise and it doesn't. You wouldn't predict a systematic shrinkage. The effect doesn't become smaller just because a journal has a positivity filter.
Eleanor Crane: So the decay is pointing at something else.
Ben Okonkwo: Underpowered studies. Small samples. And this is the mechanism I find genuinely — I mean, it's almost elegant in how cruel it is. When your study is underpowered and it happens, by chance, to hit p-values below 0.05, the effect estimate you get is inflated. It has to be. The signal you caught was an outlier draw from the distribution. So the original finding isn't just possibly false — it's systematically too large. That's what gets called the winner's curse.
Eleanor Crane: The winner's curse. The studies that survive the filter are the ones that overstated.
Ben Okonkwo: And John Ioannidis laid this out theoretically before the OSC project even ran. His argument was structural — that the combination of low power, flexible design, and publication filtering mathematically predicts that most published findings are false positives. The data from Nosek's project didn't surprise him. It confirmed the prediction.
Eleanor Crane: Which brings in the third mechanism. And I want to put a concrete face on it, because I think this is where the abstraction collapses into something uncomfortable. Picture a postdoc — Tuesday afternoon, one dataset, five different analytical approaches. Different covariates, different transformations. Four of them don't cross significance. The fifth hits p equals 0.048. She writes up the fifth. She never mentions the other four. She submits it, it gets accepted.
Ben Okonkwo: That's p-hacking. And the thing is — it's not fraud.
Eleanor Crane: No. It's completely rational behavior under the circumstances. That's what makes it hard.
Ben Okonkwo: And the econometric work bears this out — Patrick Vu and Stefan Faridani modeled this directly and found that a finding at p equals 0.05 has an expected replication probability of only 0.10 to 0.25 across fields. Not fifty-fifty. Ten to twenty-five percent. The threshold that functions as the definition of success is mathematically predicting failure most of the time.
Eleanor Crane: So what we actually have is three mechanisms that are each doing damage independently — publication bias, p-hacking, underpowered studies with inflated effect sizes — but they're also feeding each other. The postdoc hacks because the journal filters. The journal filters because the field rewards significance. The underpowered study survives because it hit a threshold that was never equipped to catch it. Which one do you fix first?
Ben Okonkwo: Which one do you fix first — and that's actually where I want to complicate the p-hacking part, because I think we've been treating it as one thing and it isn't. Cai, Pernoud, and Xu modeled this in 2026 and they split p-hacking into two types. Fast p-hacking — big analytical leaps, dropping variables, swapping subgroups — that always makes bias worse. Always. But slow p-hacking, tiny tweaks, marginal changes to the model — under strong selection pressure, that can actually partially mitigate bias.
Eleanor Crane: Wait. Mitigate it?
Ben Okonkwo: I know. It runs against the intuition. The small tweaks — and we're talking genuinely small, like a slightly different covariate — when the publication filter is already extremely aggressive, nudging a borderline result over the threshold via a minor adjustment is... I mean, it's still bias-introducing behavior, but the modeling says it's not uniformly catastrophic. The villain is more specifically the fast version.
Eleanor Crane: That does something uncomfortable to the postdoc scenario. Because the distinction between fast and slow is partly about how aware she is of what she's doing.
Ben Okonkwo: Right — and it doesn't let the system off the hook, because either way she's hunting for p equals 0.05. Which brings me back to the number from Vu and Faridani: ten to twenty-five percent expected replication probability for a finding right at that threshold. That's the target everyone is aiming at. Not fifty percent. Not even a coin flip.
Eleanor Crane: And part of why is that p equals 0.05 is being asked to answer the wrong question. It tells you the probability of your data given the null hypothesis is true. Not the probability the hypothesis is true. Not whether it will replicate.
Ben Okonkwo: Completely different question. The p-value is — okay, it's a statement about the data under a hypothetical. People read it as a statement about the hypothesis under the data. Those are not the same thing and they are not even close to equivalent.
Eleanor Crane: Which means the decline effect — the way effect sizes shrink as replications pile up — that's almost predictable from the start.
Ben Okonkwo: It is. Documented across psychology, ecology, evolutionary biology. The first big striking result gets published. The replications are smaller. The average settles somewhere much lower. And it's consistent with exactly what underpowered studies with selective publication predict — the initial outlier draw gets enshrined, then reality regresses toward it.
Eleanor Crane: The math was always telling us. That's the part I keep returning to. Ioannidis wrote the theoretical prediction before the Open Science Collaboration ran a single replication. The numbers said this would happen.
Ben Okonkwo: And it kept happening anyway. Which is — I mean, that's not a statistical failure at that point.
Eleanor Crane: No. And the reason it kept happening anyway is the part that I think is actually the deepest layer here — and we haven't touched it yet, but it's where this gets genuinely hard to fix.
Ben Okonkwo: The incentive structure.
Eleanor Crane: And the incentive structure is the thing that makes all the math downstream of it. Because the math is just — it's the water finding its level. The system tells you what gets rewarded, and researchers, rationally, produce that thing.
Ben Okonkwo: Publish or perish. And I want to name that precisely because it gets used loosely — it means career advancement, tenure, prestige, all of it tied to placement in high-impact journals. And high-impact journals want novelty. They want significant results. A replication study almost never lands there.
Eleanor Crane: Almost never.
Ben Okonkwo: So if you spend two years carefully replicating someone else's finding and it holds — great, science worked — but you probably can't put that in a top-tier journal, it doesn't move your tenure case, and your department didn't hire you to confirm other people's work. That's the cost of choosing robustness.
Eleanor Crane: Which is exactly the choice Brian Nosek made. Not once — as a career. Building the Center for Open Science, the infrastructure for pre-registration, for open data, for verification. That's a structurally irrational thing to do inside the system he was sitting in.
Ben Okonkwo: Structurally irrational. Yeah. And — I mean, the fact that the Open Science Collaboration even happened, that Nosek got 250-plus researchers to spend time replicating rather than discovering — that itself is a kind of miracle given what the incentives predict.
Eleanor Crane: So what does it mean that the people who knew the most about the crisis were the least able to fix it professionally? That's not a personal failure. That's — well, that's the system working.
Ben Okonkwo: And this is where Klaus Nielsen's framing lands hard for me. Because Nielsen's argument isn't 'the statistics are broken.' It's that the crisis reflects epistemological and political-institutional failures — the problem is embedded in how psychology defines what counts as knowledge at all. Pre-register everything you want, that doesn't change what a department considers a successful career.
Eleanor Crane: Wait — so Nielsen is saying the fix can't be purely methodological.
Ben Okonkwo: Right. Pre-registration is real progress, but — okay, if the reward structure doesn't shift, what stops a researcher from pre-registering and then running additional exploratory analyses anyway and just reporting those? The incentive to find something significant doesn't disappear because you filed a document first.
Eleanor Crane: And this is the inversion that matters. The 97% original significance rate — that's not the system failing. That's the system working perfectly. It was never designed to find true things. It was designed to find publishable things, and it did that with extraordinary efficiency.
Ben Okonkwo: That framing is — I mean, it's uncomfortable because it removes the villain. These aren't bad scientists. They're rational actors reading the environment correctly and responding to it. The fraud isn't in the behavior. It's baked into the architecture.
Eleanor Crane: Which is maybe the hardest thing to fix. You can rewrite a statistical threshold. You can't as easily rewrite what makes a career.
Ben Okonkwo: And pre-registration is the thing that's supposed to interrupt that. You file your hypotheses, your methods, your analysis plan — before you collect a single data point. So when the postdoc runs those five approaches, there's a record. The departure from the plan is visible. The Center for Open Science built the infrastructure for exactly this, TOP guidelines, registered reports — it's real. It's not nothing.
Eleanor Crane: No, it's genuinely real. But — and this is the thing that gets me — who's adopting it?
Ben Okonkwo: The researchers who already cared. I mean, that's the honest answer. The journals and labs most likely to pre-register are the ones already oriented toward reform. The majority of published literature is still operating under the same incentives that produced the 36% replication rate in the first place.
Eleanor Crane: Which means — and I don't know how to resolve this — if replication rates improve over the next decade, we learn two very different things depending on where the improvement shows up. If it's everywhere, the crisis was methodological. A technical problem, technical fix. But if it's only in the corner that adopted open science practices, while the rest of the literature carries on unchanged... then what we'll have learned is that the problem was never really about p-values. It was about what the institution chose to reward. And a patch on one corner doesn't cure that.
Ben Okonkwo: That's — yeah. I can't close that. I genuinely don't know which one it is yet. The experiment is still running.
Eleanor Crane: The experiment is still running. That's kind of where I am too. The thing I can't settle is whether Brian Nosek built a bridge or just proved a bridge was possible. Those are very different outcomes for the same amount of work.
Ben Okonkwo: Hm. That's going to sit with me for a while, actually.
Eleanor Crane: Good. I think it should.
