Ben Okonkwo: Eleanor, I have to tell you — I found something in the prep for this that I'm still a little unsettled by, honestly.
Eleanor Crane: That tracks. What was it?
Ben Okonkwo: The Royal Society introduced written referee reports in 1832. Same basic structure as today — expert assessment before publication. And we've been doing this for nearly two centuries. But the widespread standardized version, the one every journal now runs, only hardened after World War II — and it hardened for reasons that had nothing to do with finding true things.
Eleanor Crane: Prestige. Page space.
Ben Okonkwo: Journal reputation management — yeah. And I think that's the thing I keep sitting with, because we treat peer review like it's science's immune system. But if it was engineered to protect journals, not to track truth, then the immune-system metaphor is — it might be flattering the wrong organ entirely.
Eleanor Crane: And it has a very long founding story to give it the appearance of deep legitimacy — Oldenburg's Philosophical Transactions in 1665, the earliest precedent most people will cite. So there's this sense of: this thing has been working for centuries. We just — we haven't examined what it was working toward.
Ben Okonkwo: And the answer to that, mechanically, is: the system rewards novelty. A surprising result that clears a p-value threshold gets published. A result that confirms something reliable — reviewers call it uninteresting, it doesn't move.
Eleanor Crane: Which means a surprising false positive is, structurally, more publishable than a boring true one. That's not an accident of the culture — that's the design feature nobody wanted to name out loud.
Ben Okonkwo: Interesting — and the question I want us to actually work out today is whether peer review can be reformed inside that structure, or whether the structure itself is the problem.
Eleanor Crane: Because those are very different diagnoses.
Ben Okonkwo: Completely different. And the reforms — registered reports, open review, post-publication review — they answer one diagnosis. I'm not sure they answer the other.
Eleanor Crane: But that gap — between what the system was built to do and what we think it does — that's where I want to slow down, because I don't think most people have actually looked at what a reviewer is literally being asked to evaluate.
Ben Okonkwo: Right — and this is the analogy I use. A reviewer reads a manuscript the way a panel of expert editors judges a dish for a restaurant menu. They taste it. They assess the technique. They decide: is this interesting, does it fit our style, does the presentation hold up? But they never go back to the kitchen and cook it again to see if it reproduces.
Eleanor Crane: They're judging the dish. Not the recipe.
Ben Okonkwo: Exactly that. And the criteria they're actually using — novelty, methodological soundness, statistical analysis, significance, fit with the journal — none of those criteria test whether independent researchers can reproduce the result. Not one.
Eleanor Crane: Which means peer review can catch errors, it can improve how a finding is written up — but it cannot, by design, tell you whether the finding is true.
Ben Okonkwo: And I think — okay, this is the part that still gets me. Because the post-WWII standardization, the moment Science and journals like it started leaning heavily on outside referees — that happened because editors were drowning in manuscripts. Volume management. That's the actual driver.
Eleanor Crane: Wait — so the system that the entire scientific community now treats as a truth-verification mechanism was originally a capacity solution?
Ben Okonkwo: A capacity and reputation solution, yeah. Not an epistemological one. Nobody sat down and asked: what's the best way to establish whether a finding is reliable? They asked: how do we handle the volume and protect the journal's name?
Eleanor Crane: And that's — well, that's not a neutral administrative fact. If you build the gate to protect prestige, then prestige is what the gate is optimized for. Not truth.
Ben Okonkwo: Which means the thing we call quality control is actually — hm, it's more like curation. Sophisticated curation, but curation.
Eleanor Crane: And the word 'review' does a lot of heavy lifting there. It sounds like scrutiny. Like verification. Most people hear 'peer-reviewed' and they hear something like: tested, confirmed, reliable.
Ben Okonkwo: When what it actually means is: a small number of specialists found the manuscript interesting enough, and technically coherent enough, to put on the menu. That's the whole guarantee. And once you see that gap — between what it means and what people assume it means — the replication crisis stops looking like a surprise.
Eleanor Crane: And that gap — between 'peer-reviewed' and 'reliable' — is actually where novelty bias does its worst work, because the criteria aren't neutral. A reviewer sees a surprising result and something happens. It clears a bar that a confirmatory finding literally cannot clear, not because it's better science, but because it's more interesting to look at.
Ben Okonkwo: Right, and 'interesting' is doing real mechanical work there. Not metaphorical — the actual decision point.
Eleanor Crane: So walk me through what a reviewer is literally seeing. In the manuscript. What makes them vote yes?
Ben Okonkwo: Actually, the statistical test looks clean. The effect is surprising, counterintuitive even. The p-value clears the threshold, typically p less than 0.05. And from inside the manuscript, the reviewer has no way to know whether the researcher ran one version of that analysis or forty versions until one worked. They cannot see the file drawer. They're voting on the dish, not the kitchen.
Eleanor Crane: Which is exactly what makes p-hacking so structurally rational. It's not — I mean, we want to call it misconduct, but from inside the incentive, it's just... a researcher doing what the system rewards.
Ben Okonkwo: And the thing that's hard to sit with is that null results face the same wall from the other direction. There's a junior researcher — call her Dr. Yemi Adisa — she runs the experiment carefully, gets a null result, and her lab's funding review is in six weeks. She knows what happens to null results when they go to journals. So she doesn't write it up. The file drawer closes. The literature never sees that data, and the published record looks like the effect is real.
Eleanor Crane: That's not a failure of courage. That's a completely rational calculation.
Ben Okonkwo: Completely rational. She's not doing anything wrong by the system's logic. Publish or perish means you need a continuous output of novel, significant papers — that's not a preference, that's an existential career requirement. A null result doesn't feed that.
Eleanor Crane: And p-hacking is almost the mirror image of Dr. Adisa's decision. She's choosing not to submit because the result won't clear the bar. Someone else is analyzing the same data twelve different ways until one version does clear it. Both moves come from the same place.
Ben Okonkwo: Same incentive, two expressions of it. And both of them push the published literature in the same direction — toward positive, significant, surprising findings, regardless of whether those findings are actually reproducible. Statistical significance is not the same thing as truth, but the gate treats them as equivalent.
Eleanor Crane: And what I want to know — what I think makes this worse than it sounds — is whether we even have a reliable way to measure how distorted the literature has become. Because if the correction tools are also broken, that's a different level of problem.
Ben Okonkwo: That's exactly where this goes next, and honestly it's the part I find most unsettling — what happens when you look at the Fail Safe File Drawer method, the tool people actually use to correct for this bias, and what Scargle showed about what it's actually measuring. That's — we need to pull that apart.
Eleanor Crane: Pull that apart, because I think most people hear 'there's a statistical correction for that' and they exhale. Like, fine — the bias exists, but we have a tool.
Ben Okonkwo: And the tool is the Fail Safe File Drawer method. The idea is — you take a published positive finding, and you calculate how many unpublished null results would have to be sitting in file drawers to overturn it. If that number is huge, the finding feels safe. If it's small, you're worried.
Eleanor Crane: Which sounds reasonable.
Ben Okonkwo: It sounds reasonable. And then Jeffrey D. Scargle — a physicist — actually checked the math. And the problem he identified is, I mean, it's almost elegant in how fundamental it is. The method assumes the unpublished studies are unbiased. It treats the file drawer as a random sample of all the research that got run.
Eleanor Crane: But the whole point is that it isn't.
Ben Okonkwo: The whole point — the entire reason we need the correction — is that the file drawer is not a random sample. It's specifically full of nulls that didn't get published because they were null. So the method's core assumption is the thing it's supposed to be correcting for. Scargle showed it consistently underestimates how bad the bias actually is.
Eleanor Crane: Wait. The correction tool fails because it assumes the problem doesn't exist.
Ben Okonkwo: Essentially, yes. And the part that's hard to sit with — this critique was published. Scargle put it in the literature. And meta-analyses across social science, medicine, even psychic research keep citing the Fail Safe File Drawer method anyway. Because there's no universally accepted replacement. So we know it's broken and we're still using it.
Eleanor Crane: That's — I mean, what does that do to your confidence in the literature? If the adjustment tool we're applying to correct for distortion is itself producing distorted estimates—
Ben Okonkwo: It means we've been stacking uncertainty on top of uncertainty and calling it a correction. And the replication crisis is, in a way, what happens when you finally go back and check. Actually run the experiments again. Large-scale replication projects in psychology and biomedicine did exactly that — and a substantial proportion of published findings just didn't reproduce.
Eleanor Crane: And that wasn't a sudden breakdown.
Ben Okonkwo: No — that's the thing. The replication crisis didn't create the problem. It made the problem visible. The false positives were always there, accumulating, because science never had a reliable way to estimate the false-positive rate in the published literature until someone actually went back and ran the experiments. The failure was structural and chronic. We just — we hadn't looked.
Eleanor Crane: And when you look at what drove the failure rate — it's not fraud. It's not individual bad actors. It's a system that was optimized for publishing surprise, and surprise is exactly what a false positive delivers.
Ben Okonkwo: Predictable output of a system built around the wrong target. The reviewers were doing their job. The researchers were responding to real incentives. Nobody had to cheat for this to happen at scale.
Eleanor Crane: Which is what makes the reform question so difficult — because if the problem is structural, then the question isn't who to hold accountable. It's whether registered reports, open peer review, any of it, can actually realign the structure. Or whether they're patching the visible symptoms of something that runs deeper.
Ben Okonkwo: And that's — I mean, the reform inventory from the Research on Research Institute is actually quite thorough on this. They've mapped the landscape. Registered reports provisionally accept a study on its question and methodology before data are collected, which does partially decouple publication from outcome. Open peer review, as studied at F1000Research — mixed evidence on whether it reduces geographic or social bias, and there's a real possibility transparency introduces new social-pressure dynamics rather than eliminating bias overall. Each fix targets something real. A specific node. But the structural misalignment — novelty rewarded over reliability — that's still running underneath all of it.
Eleanor Crane: And the FT50, journal impact factors — those haven't moved.
Ben Okonkwo: Right — the prestige metrics are intact. The incentive engine keeps running even when individual practices improve.
Eleanor Crane: Which means the thing I keep — well, what I land on is that peer review was never actually asked to verify truth. It was asked to protect reputation. And at some point we just... started assuming it was doing both. And that might be the category error the whole system rests on. Not a failure of execution. A failure of the original brief.
Ben Okonkwo: And that brings us back to 1832, doesn't it. The Royal Society's written referee reports — same expert-before-publication structure. It looked like verification then too.
Eleanor Crane: Same gate. Different assumption about what the gate was for. Nearly two centuries of that assumption accumulating before anyone went back and ran the experiments.
Ben Okonkwo: Hm. That's an uncomfortable place to land.
Eleanor Crane: It is. And I think that discomfort is the honest place — not the reforms, not the crisis, but just sitting with the question of whether we ever asked the system to do the thing we needed it to do. I'm glad we didn't tidy that up.
