Eleanor Crane: I've been thinking about Tuesday mornings — not any specific one, just the general shape of them. A postdoc, coffee going cold, data on the screen. And I think that's actually where this episode lives.
Ben Okonkwo: The decision point.
Eleanor Crane: A 2026 survey of experimental philosophers found that most null results — studies where the expected effect didn't appear — stall during early data analysis or presentation. Before submission. Before any editor sees them. The file drawer problem, it turns out, is not a filing problem. It's a quitting problem, and it happens in private.
Ben Okonkwo: Wait — so publication bias and the file drawer problem, we've been picturing them as downstream, as what journals do to researchers' work. But the data says researchers are doing it to themselves first.
Eleanor Crane: And it makes a certain painful sense, doesn't it, when you consider what publish-or-perish actually means — that hiring, tenure, grants all run through publication count and impact factor. A null result doesn't move any of those levers. So the rational move is to not write it up.
Ben Okonkwo: Right — and the downstream cost of that shows up in something like the Open Science Collaboration's 2015 replication study, published in Science. A hundred psychology studies replicated, sampled from journals including Psychological Science. Only roughly 36% reproduced a statistically significant effect. Now, here's what bothers me about that number: it assumes the 100 original studies represent what was actually found. But if most null results never made it to those journals in the first place, then the 36% is measuring the replication rate of an already-optimistic literature.
Eleanor Crane: Which means the problem we can see is probably not the full problem.
Ben Okonkwo: And that's the part that doesn't get named directly — novelty bias, publication bias, all of it, they're usually described as journal-level distortions. But if the distortion starts when a researcher decides what's worth finishing, we're looking at something that's genuinely upstream of the editorial system entirely.
Eleanor Crane: And that upstream distortion — that's actually the part that needs the plain-language version, because I think most people picture one corrupt editor somewhere. But there isn't one.
Ben Okonkwo: No — and this is the click I want to land. Think of a music chart. Radio stations play what listeners request. Listeners request what they've already heard. No one is conspiring. But only certain songs get airtime. The whole system self-selects for the familiar and the attention-grabbing, and it does that without a single bad actor.
Eleanor Crane: And in publishing, the signal everyone's chasing is — what, exactly?
Ben Okonkwo: Impact factor. Introduced in the 1970s, measures average citations per paper over two years. That's it. That's the metric. And it got adopted — I mean, it became the proxy for scientific quality, which, okay, it measures attention. Not rigor. Those are genuinely different things and the system doesn't distinguish them.
Eleanor Crane: So an editor chasing a high impact factor needs cited papers. Cited papers tend to be novel, surprising, counterintuitive. Which means — and this is where publish-or-perish closes the loop — the postdoc who wants tenure also needs publications in those journals. The incentive points the same direction at every level.
Ben Okonkwo: Every rational actor — the editor, the postdoc, the grant committee scanning impact factors — is responding to the same signal. Publication bias isn't a journal pathology. It's an emergent property of the whole structure operating normally.
Eleanor Crane: Which is — I mean, that's actually worse, isn't it. You can't shame it away.
Ben Okonkwo: Right — and what that means concretely is that novelty bias, publication bias, the file drawer problem — they're not three separate failures. They're one feedback loop. And the Reproducibility Project: Cancer Biology found more than half of high-impact studies from journals like Nature, Science, Cell couldn't be replicated. The literature kept building on those papers. Because the incentive to cite them didn't stop just because the underlying result was shaky.
Eleanor Crane: And that's the part that keeps pulling at me — the building on shaky foundations. Because the file drawer problem doesn't just mean the literature is optimistic. It means the distortion is invisible. A postdoc finishes three clean experiments, no significant effect, and on a Friday afternoon she just... doesn't open a new document. Not because an editor said no. Because she already ran the calculation in her head.
Ben Okonkwo: The decision happens before the system can even see it.
Eleanor Crane: Which is what makes positivity bias genuinely strange — it's not a filter at the gate. It runs all the way back through study design, through how you frame the analysis, through whether you finish writing at all.
Ben Okonkwo: Right — and the 2026 Science Advances paper on this actually names that explicitly. The bias operates at every layer of the pipeline simultaneously. Which means reforms aimed only at journals, preregistration, open peer review, they can't reach the decision that was never recorded anywhere.
Eleanor Crane: How do you reform a document that was never started?
Ben Okonkwo: Exactly — and, wait, actually this is the piece I keep wanting to flag. Preregistration genuinely reduces p-hacking. That's real. But it's still voluntary, and researchers who adopt it don't see faster promotion. So we've built a tool that's epistemically sound and career-neutral at best. The rational actor looks at that and... keeps filing the drawer.
Eleanor Crane: A reform that works in theory and stalls at the Tuesday morning calculation.
Ben Okonkwo: And the downstream cost of all those unfiled drawers — we'll get to that when we look at the cancer biology numbers closely, because that's where you can't blame methodology or execution anymore.
Eleanor Crane: What I find strange is that the scientific record looks continuous. Confident. And beneath it are all these decisions that were never formally made — just quietly not made.
Ben Okonkwo: But the part that actually breaks the clean version — the RP:CB numbers. Initiated in 2013, co-organized through Brian Nosek's Center for Open Science and Science Exchange, funded by Arnold Ventures. They went after high-impact cancer biology studies, published in Science, Nature, Cell, 2010 to 2012. And more than half couldn't be replicated.
Eleanor Crane: More than half.
Ben Okonkwo: And what makes it hard to argue away — eLife peer-reviewed every replication protocol before a single experiment ran. Methodological fidelity, confirmed in advance. So when the results didn't hold, you can't point at the replication team. The failure points back at the originals.
Eleanor Crane: Which is — I mean, that's the design doing real work there. You've closed the escape hatch.
Ben Okonkwo: And what you're left with is psychology at 36%, cancer biology at worse than half, different methods, different funding cultures, different journals — same result. Which, okay, actually that's the load-bearing observation. It's not a field-specific pathology. The mechanism is the same incentive logic running through every field.
Eleanor Crane: And what does that mean for the literature that kept building on those papers? Because it did. Researchers cited them, designed studies around them — the scaffold went up.
Ben Okonkwo: The citation pressure didn't stop when the underlying result turned out to be shaky. Nobody sent a memo. And — wait, this is the part I keep stumbling on — a replication study that fails doesn't generate novelty. It has no citation advantage. So it's career-neutral to publish it and actively costly to have run it in the first place.
Eleanor Crane: So the person who did the careful work, confirmed the fidelity, ran the experiment, found nothing held — that person is worse off for having done it.
Ben Okonkwo: That's not an irony. That's the mechanism. The system isn't broken — it's working exactly as the incentives designed it to work.
Eleanor Crane: And that's — I keep returning to that postdoc. Tuesday morning, coffee going cold. She's done everything right. The data is clean. And the system's answer is: that's not enough, and also, we'd prefer you didn't mention it.
Ben Okonkwo: The 2026 Science Advances paper names why that stays stuck — procedural reform, cultural change, stakeholder coordination, all three have to shift at once. Preregistration and Registered Reports work epistemically, genuinely, but if hiring committees and funding bodies still run on impact factor, voluntary adoption is career-risky. You can't ask one rational actor to go first into a game where going first costs them.
Eleanor Crane: Who changes first.
Ben Okonkwo: That's the one I don't have.
Eleanor Crane: I think that's actually where we have to leave it — not as a failure to conclude, but because the honest answer really is that nobody in the system is waiting for permission to be irrational. The postdoc on Tuesday morning is going to make the sensible call.
Ben Okonkwo: Yeah. The coffee gets cold either way.
