Megan Skiendel: Quick question before we get into anything — do you actually trust published science?
David Sterling: That's a strange way to start a Tuesday.
Megan Skiendel: I'm serious though. Amgen — the pharmaceutical company — sent their scientists back through preclinical cancer studies that had been published in major journals. Peer-reviewed, prestigious, the signal was as strong as it gets. Eleven percent replicated. Eleven.
David Sterling: Hold on. That's not a slight underperformance. That's the entire credentialing system of academic publishing producing output that fails eighty-nine percent of the time.
Megan Skiendel: And the part that I can't get past — it's not that peer review missed some errors. It's that peer review checked methodology, not whether the finding would hold. Those are completely different things.
David Sterling: Frankly — and this is the thing — the Open Science Collaboration's Reproducibility Project found thirty-six percent replication across a hundred psychology studies. So you've got eleven percent in biomedicine and thirty-six in psychology. The replication crisis isn't one field's problem.
Megan Skiendel: Right — but thirty-six versus eleven, that's not the same problem at the same scale.
David Sterling: Correct. The mechanism might differ. But the structure is the same: journals favor novel positive results, researchers respond to what journals reward, and null results — the ones that say 'this didn't work' — disappear into file drawers. Publication bias is the name for it and it compounds over time.
Megan Skiendel: And nobody upstream — not the tenure committee, not the grant panel — is asking what percentage of last year's published findings actually held up.
David Sterling: The metric they're using is impact factor. Which measures citations, not accuracy. It's a price signal that's measuring the wrong thing.
Megan Skiendel: So the whole incentive structure — perverse from top to bottom. Which is what we're actually here to pull apart.
David Sterling: And the question that doesn't have a clean answer yet: if peer review is functioning correctly, why is eighty-nine percent of preclinical cancer research not surviving contact with a second lab?
Megan Skiendel: Think about it like a restaurant guide that only publishes reviews of meals that were surprisingly delicious. Not good — surprisingly delicious. Every mediocre meal, every bad meal, just never gets written up. The guide isn't lying. But the picture it paints is completely unreal.
David Sterling: That's the file drawer problem in one sentence. The missing reviews don't disappear — they just never get written.
Megan Skiendel: And the part that breaks the analogy — with restaurants, you can go eat there yourself. With a cell biology study, most people can't run the experiment. You're completely dependent on the published record being representative.
David Sterling: Which it isn't. Structurally cannot be. Because peer review — and I want to be precise here — peer review evaluates the methodology. Did you control for the right variables, did you run the right statistical test. It does not evaluate whether the finding will survive a second lab. Those are genuinely different questions.
Megan Skiendel: So a well-designed study that finds nothing is, what — good science, bad news?
David Sterling: Exactly that. And 'bad news' means it doesn't get submitted. Or it gets submitted and rejected. The filter isn't catching bad science — it's filtering out inconvenient science.
Megan Skiendel: Listen, I want to make this really concrete. Tuesday morning, nine a.m. Junior researcher — postdoc, let's say — finishes an experiment. Her result quietly contradicts a celebrated paper from five years ago. She goes to her PI.
David Sterling: And the PI says drop it.
Megan Skiendel: The PI says — and I mean this is almost verbatim what happens — 'replication studies can't compete for grants, and if you publish it you'll spend two years defending against angry citations instead of building your own research line.' So she buries the null result. Footnote in a methods paper. Gone.
David Sterling: And three other labs did the same thing that same week. That's not a conjecture — that's what the file drawer problem actually describes at scale.
Megan Skiendel: And nobody made a corrupt decision. The PI's math is right. That's the part that I — honestly, that's what makes this hard to fix with exhortation. The individuals are being rational.
David Sterling: Which is the load-bearing thing. Publication bias isn't a failure of individual integrity — it's a structural filter that turns individually rational choices into a collectively distorted record. The question is what you'd have to change upstream to make the postdoc's null result worth keeping visible. And that answer isn't peer review.
Megan Skiendel: And the scale of that distortion — I mean, we've been treating thirty-six percent like it's a bad number. But the Open Science Collaboration's finding is actually worse when you look at what happened to the studies that did replicate.
David Sterling: The effect sizes.
Megan Skiendel: The effect sizes. Even in the successes — the thirty-six percent that did come through — the replicated effects were roughly half the size of what the original paper reported. Half. So the finding holds, but the claim attached to it was twice as big as the reality.
David Sterling: So the replication rate understates the problem. Because the replications that count as wins are still describing a world that's systematically inflated.
Megan Skiendel: Exactly — and nobody celebrates the replication paper for catching that. The original author gets the citation stack. The person who ran it again and said 'actually, the effect is half the size' gets, what, a footnote somewhere.
David Sterling: Well, that's the asymmetry that matters. Prestige flows to the discoverer. The auditor gets nothing — or worse, she spends two years in the crossfire with the original lab.
Megan Skiendel: And here's the timing piece that I think is the actual trap — a replication paper takes two years, minimum. By the time it lands, the original finding has already seeded three derivative papers, shaped someone's entire research program, probably been cited in grant applications. The correction arrives late. The cascade is already running.
David Sterling: That's not a publishing problem. That's a compounding problem. The false discovery earns interest before anyone audits the principal.
Megan Skiendel: Which is why the replication crisis isn't a scandal about cheating. Nobody's cheating. It's the predictable output of a filter that was never designed to catch this.
David Sterling: The filter approved the methodology. Not the truth of the claim. Those have always been different questions and we — I think we built seventy years of career infrastructure pretending they were the same thing.
Megan Skiendel: And honestly the part that comes later in all this — why even structural reforms like Registered Reports might not be enough until the tenure committee stops reading impact factor as a quality signal — it gets really uncomfortable.
David Sterling: Right, but — the load-bearing question before we get there: if the published record is systematically inflated, what's the actual cost? Not reputationally. Financially. How many grant dollars chased findings that were already dead?
Megan Skiendel: And the grant money question is almost the easier version of it. The harder version is — who controls the number that tells everyone what good science looks like? Because that number is impact factor. Calculated through Journal Citation Reports, rooted in the Science Citation Index. Average citations per article over a defined window. That's it. That's the whole signal tenure committees are reading as quality.
David Sterling: Citations per article. Not replication rate. Not accuracy. Just — how many people cited you.
Megan Skiendel: Which a wrong paper can absolutely generate. A splashy false positive gets cited everywhere while people try to build on it.
David Sterling: That's the principal-agent problem in one line. Funders want accurate, reproducible knowledge. Researchers are paid — structurally, through career advancement — for novelty and journal prestige. Those are not the same objective. They are not even correlated.
Megan Skiendel: And look, business schools crystallized this completely — the FT50, the Financial Times 50 journal list, it's a binary. You're on it or you're not. Hiring, promotion, accreditation, all reading that one list.
David Sterling: Wait — is the list actually tracking influence, or is it just — what, incumbency?
Megan Skiendel: That's the citation impact heterogeneity finding. Journals inside the FT50 — same prestige tier, same list — show striking variation in actual scholarly influence, policy influence, patent citations. The binary label doesn't reliably sort for quality. It sorts for being on the list.
David Sterling: So the price signal is noisy even within the prestige tier. Which means the people using it as a quality proxy are — frankly, they're doing something worse than guessing. They're guessing with false precision.
Megan Skiendel: And here's where the trap actually closes — I mean, you might think, well, journals could just decide to value replications. Shift the culture. But a journal that starts publishing replications instead of novel findings watches its impact factor drop. Lower impact factor, harder to recruit top authors, harder to attract good reviewers. The journal that tries to fix this unilaterally punishes itself.
David Sterling: The trap is collective, not individual. No single journal can defect from the system without eating the cost alone.
Megan Skiendel: Royal Society Open Science explicitly accepts replication studies. And it matters that they exist. But one journal absorbing that cost doesn't move the impact factor calculus that tenure committees are running everywhere else.
David Sterling: The incentive misalignment is upstream of the journals. Until tenure committees stop using impact factor as a quality proxy — until grant panels do the same — journal-level reforms are self-defeating. The postdoc's PI already knows this. That's why the null result goes in the footnote.
Megan Skiendel: The individuals are doing the math correctly. The math is just built on the wrong foundation.
David Sterling: Which leaves the reform question. Registered Reports — peer review happens before data collection, publication guaranteed regardless of outcome. That's the structural fix that actually addresses the mechanism. And early evidence suggests it does reduce publication bias. So — why isn't that enough?
Megan Skiendel: Because a Registered Report publication doesn't count the same way on a tenure file. Not yet. Until the hiring committee treats it as equivalent to a Nature paper — and they don't — the postdoc is still doing the math and losing.
David Sterling: So the mechanism fix is real, but it's — I mean, it's a rounding error until the demand side moves. Supply-side reform, demand-side inertia.
Megan Skiendel: And the Biden-Harris administration declared 2023 the Year of Open Science. Federal agencies adopting open science regulations. Which sounds like demand-side pressure, right? That's governmental. But — who's enforcing it in a department meeting when a committee is weighing two candidates and one has the higher impact factor?
David Sterling: Nobody. That's the gap.
Megan Skiendel: And honestly the thing I can't resolve — the people who could restructure tenure criteria, rewrite what grant panels score for, reframe accreditation standards — those are the people who built their careers under the old system. They got hired because their impact factor was high. They got grants because they published novel findings in prestigious journals. Why would they vote to devalue the currency they're holding?
David Sterling: They wouldn't. Not voluntarily. The question is whether the fix can move faster than the natural turnover — generational turnover — of who's actually sitting in those chairs. And I don't know the answer to that.
Megan Skiendel: Neither do I. And I've been turning it over and I keep landing in the same uncomfortable place — the system isn't broken in a way that produces obvious losers with enough power to demand change. The losers are junior researchers and, eventually, patients. Neither group controls the lever.
David Sterling: That's where I'll leave it, too. Genuinely unresolved. Rare for me to say that and mean it.
