Eleanor Crane: I want to try something — I'm going to give you a number and I want your first instinct, no setup.
Eleanor Crane: Twenty percent.
Ben Okonkwo: Twenty percent of what?
Eleanor Crane: Computational reproducibility in bioinformatics. That's — some reviews of published analyses in the field landed around there. One in five.
Ben Okonkwo: In bioinformatics. In genomics.
Eleanor Crane: In genomics — the field that, along with astronomy, gets held up as proof that open science works. That the infrastructure exists. That sharing data and code and protocols converts a scientific claim into something inspectable rather than something you just have to trust.
Ben Okonkwo: Interesting — because the 20% is actually the tighter, easier test. Computational reproducibility means: take the same data, the same code, run it, do you get the same answer. It's not asking whether a new experiment confirms the result. It's just asking whether the analysis reruns. And still.
Eleanor Crane: And inaccessible data is still being linked to failed replication attempts even among geneticists within genomics. So part of it is the materials aren't even available — even in the flagship.
Ben Okonkwo: Which is — okay, that's the thing I can't square. Because the whole argument for open data and code sharing is that it redesigns the reward landscape. Structural incentive alignment — the claim that transparency becomes durable not because researchers are suddenly more virtuous, but because hiding errors gets costly. That's the mechanism. And you're telling me the mechanism is running in the field built to demonstrate it, and the output is 20%.
Eleanor Crane: And meanwhile the 2015 psychology replication study — 36% of findings holding up — that's the wound that sent everyone toward open science in the first place. Toward genomics as the model.
Ben Okonkwo: So the model has a 20% rate on its own narrower standard, and we're asking psychology to follow it. The question sitting underneath all of this is — what is the structural fix actually fixing, if this is what success looks like?
Eleanor Crane: But before we sit too long with that failure number — I want to back up one step, because I think there's a listener somewhere who's still asking: what does the mechanism even look like when it works?
Ben Okonkwo: Right — okay, so the cleanest way I can put it: imagine a chef hands you the recipe, the raw ingredients, and a video of every single step. You don't have to trust the dish is good. You can cook it yourself and find out. That's what Open Data, Code, and Protocol Sharing does to a published paper. The claim stops being 'trust me' and becomes an inspectable workflow.
Eleanor Crane: And the reason that matters practically is the Cost of Verification — the time, the money, the expertise it takes to check someone else's work — that collapses when the materials are already on the table.
Ben Okonkwo: Exactly, and — wait, here's the part that I think actually makes it click. P-hacking. A researcher runs thirty different cuts of the data, finds the one that gives a significant p-value, and publishes that one. Without open code, you see the result and you have no idea about the other twenty-nine attempts. With open code, a reader can see every path taken. The fishing expedition is visible.
Eleanor Crane: And preregistration is the companion fix to that — you publicly commit your hypothesis and your analysis plan before you collect a single data point. So you can't retroactively declare the fishing trip a planned expedition.
Ben Okonkwo: PLOS — the Public Library of Science — is the organization that actually institutionalized this at scale. They built open data requirements into journal policy, not just aspiration.
Eleanor Crane: So structurally, the mechanism is real.
Ben Okonkwo: The mechanism is real. But — okay, here's where I have to flag the assumption the whole thing rests on. The mechanism assumes someone actually downloads and checks the files. And that's... I mean, how often does that actually happen at a rate high enough to create deterrence?
Eleanor Crane: Is the error-catching happening, or are we building elaborate filing systems?
Ben Okonkwo: That's — yeah, that's the question. A Harvard Data Science Review reproducibility challenge study from 2022 found most researcher teams were building their own reproducibility protocols from scratch. No off-the-shelf tools. Which means the infrastructure isn't actually shared infrastructure — it's everyone reinventing the same wheel in a different lab.
Eleanor Crane: Wait — so even the people trying to do this right are starting from zero each time?
Ben Okonkwo: Which is why the HDSR study argues universities have to play a central infrastructure role — you can't just mandate openness and expect individual labs to solve the tooling problem independently. Actionable reproducibility, they call it. Not theoretically available — practically adoptable. And that distinction is, I think, where the 20% actually comes from.
Eleanor Crane: And that's — I mean, that's the part that doesn't fit the clean story. Because the Budapest Open Access Initiative gave the whole field its legal-political scaffolding. The European Commission literally tied funding conditions to open science compliance — not a suggestion, a structural requirement. And still you get labs reinventing the wheel and a 20% reproducibility rate. So what exactly did the structure change?
Ben Okonkwo: The structure changed what's on paper. Not necessarily what's on the hard drive.
Eleanor Crane: Which is — right, but is that a policy failure or an adoption failure? Because those are different problems.
Ben Okonkwo: Okay, so — picture a computational biologist, Tuesday afternoon, wants to verify a published genomics analysis. She downloads the data file. Opens the code. And the code calls a library version that no longer exists. The raw data is missing two preprocessing steps that were apparently obvious to the original lab and to no one else. The repository is open. The result is inaccessible. That's not a policy gap — that's a usability gap. The Budapest Open Access Initiative didn't specify that your code has to actually run.
Eleanor Crane: A deprecated library version. That's — that's what five years of open science infrastructure reduces to on a Tuesday.
Ben Okonkwo: And it's not bad faith. The original researcher wasn't hiding anything. The workflow was genuinely open. It just became illegible the moment the software environment drifted.
Eleanor Crane: Which is part of why the Journal of Memory and Language's open data mandate is being watched so closely — it's a live experiment. Does requiring data submission at publication actually narrow the gap, or does it produce a lot of files that technically exist and functionally don't work?
Ben Okonkwo: And that distinction — computational reproducibility versus replicability — matters enormously here. Computational reproducibility is the narrow test: same data, same code, same result. Replicability is the harder one: go collect new data, does the finding hold? Open science mandates address the first. The Journal of Memory and Language mandate is a test of the first. We're not even at the second yet.
Eleanor Crane: And we're failing the easier one.
Ben Okonkwo: At roughly 20%, in the field optimized for it. Now — and this is the part that I think gets more troubling the closer you look — adoption is uneven not just across disciplines but within genomics itself. Some labs share everything. Some share nothing. Same journal policies, same European Commission funding conditions. Which means the structural lever isn't reaching uniformly even where the structure is most mature.
Eleanor Crane: And what that suggests is that the compliance that does happen might be the part that actually obscures the problem. Because partial compliance looks like openness from the outside. What that does to error detection is something we need to get into.
Ben Okonkwo: Yeah — and that's connected to something the HDSR reproducibility challenge flagged that I don't want to skip past. The question isn't just whether researchers share — it's whether sharing is actionable. Theoretically available and practically adoptable are not the same standard. And that gap, I think, is where most of this is actually breaking down.
Eleanor Crane: And the partial compliance thing — that's actually the counterintuitive piece, because it's worse than nothing in a specific way. If there's no data file at all, a reader knows they can't verify the work. But if there's a data file and no code, they think they can. They start trying. And then they hit the wall and they don't know if the problem is their understanding or the original analysis.
Ben Okonkwo: It creates a false floor of confidence.
Eleanor Crane: Exactly that. The appearance of openness without the function of it.
Ben Okonkwo: And the HDSR 2022 challenge showed this in a really specific way — it wasn't that teams were refusing to share. They were trying. They just had no common tooling. Every team was building its own reproducibility stack from scratch. Which means the 'full-stack transparency' the structural incentive argument assumes — data, code, protocols, all linked and runnable — that stack doesn't actually exist as an off-the-shelf thing. The researchers have to construct it manually each time.
Eleanor Crane: Wait — so the infrastructure argument assumes infrastructure that doesn't exist yet.
Ben Okonkwo: Not at the usable layer, no. And that's — I mean, actionable reproducibility is the HDSR's term for what's missing. Not theoretical availability. Whether a researcher on a Tuesday afternoon can actually run the thing without a PhD in that specific software stack.
Eleanor Crane: And then you add Copyright Transfer Agreements into this and it gets — well, I think most people don't realize this is even a problem. Researchers sign away IP rights to journals, and that can legally prevent them from self-archiving code. Not unwillingness. A contract they signed at submission, probably without reading it closely.
Ben Okonkwo: So a researcher wants to comply, the legal structure blocks them.
Eleanor Crane: Which is a structural barrier, not a behavioral one — and it's the kind of thing the Budapest Open Access Initiative framework didn't anticipate fully, because BOAI was built around access to papers, not code repositories.
Ben Okonkwo: And Diamond Open Access — the model the European Commission was pushing as a priority around 2020, no fees for authors, no fees for readers — that addresses the reading layer. Whether someone without an institutional subscription can even see the paper. But if they can read it and still can't run the analysis because the code's behind a Copyright Transfer Agreement, the access problem and the reproducibility problem are running in parallel and neither solution touches the other.
Eleanor Crane: Two different walls, and we've only half-dismantled one of them.
Ben Okonkwo: Which is — okay, I want to be precise about what this means for the structural incentive argument. The argument is: make transparency the path of least resistance and errors become costly to hide. That's the claim. But if compliance is partial, errors are still hideable. Not by intention — just by incompleteness. A reader who hits a missing protocol step can't distinguish 'the researcher forgot to include this' from 'the researcher didn't want this visible.' The deterrence effect collapses even without any bad faith.
Eleanor Crane: And that's the human thread sitting inside all this machinery — the researcher who shared genuinely and still produced something unverifiable. Who's covered by the appearance of openness without actually being checked. It doesn't protect them from a replication failure. It doesn't protect the field from acting on their result.
Ben Okonkwo: So the question that actually follows from all of this — and I don't think we've answered it yet — is whether the fix is more infrastructure, or whether it's a different kind of requirement entirely. Because mandating openness without mandating usability is how you get a 20% rate in the field that's supposed to be the model.
Eleanor Crane: The structural argument is so elegant, though. That's the thing I keep bumping into. You don't have to make scientists more honest. You just make honesty the easier path. And everything we've traced today suggests the path isn't actually easier yet.
Ben Okonkwo: Psychology and cancer biology are the next live test of that. Mandatory data-sharing policies rolling out — not suggestions, requirements. And the question isn't whether open science is a good idea. It's whether mandates without the cultural shift produce real compliance or compliance theater.
Eleanor Crane: And if it's theater — if patchy compliance persists even after the mandates — then the constraint was never infrastructure. It was social trust. Disciplinary norms. The stuff that's genuinely hard to engineer.
Ben Okonkwo: Which structural incentive alignment, by definition, cannot fix.
Eleanor Crane: We started with a number. Twenty percent. I threw it at you cold. And I think it lands differently now — not as evidence that open science failed, but as evidence of how much distance there still is between the mechanism existing and the mechanism working.
Ben Okonkwo: One in five. In the flagship. That's — I mean, that's where we are.
Eleanor Crane: It's an honest place to stop, I think. Not a resolved one.
Ben Okonkwo: Unresolved is accurate. Thanks for pushing on the mechanism — I needed the pull back to the person behind the compliance gap.
