Eleanor Crane: Ben, tell me something — when you read a study that says 'based on a sample of sixteen thousand participants,' does the number feel small to you?
Ben Okonkwo: Interesting — it depends entirely on the design. Sixteen thousand in a well-run randomized controlled trial is a very different thing from sixteen thousand in an observational cohort.
Eleanor Crane: That's almost exactly where I wanted to start. Because the HRT story runs directly into that intuition and breaks it. Observational studies suggesting hormone replacement therapy was safe — or even cardioprotective — involved millions of women over decades. Clinical practice adopted HRT widely on that basis. And then a single randomized controlled trial, sixteen thousand women, reversed the whole thing. Heart disease risk went up, not down. And the smaller study won.
Ben Okonkwo: That's — yeah, sixteen thousand against millions, and the sixteen thousand carries more evidential weight. Most people's intuition says that shouldn't be possible.
Eleanor Crane: And their intuition is wrong. Which is what today is actually about.
Ben Okonkwo: So the mechanism — the reason it's possible — is confounding. The women in those observational cohorts who chose HRT were not a random sample of women. They were, on average, healthier. More engaged with preventive care. More health-conscious in ways that are genuinely difficult to quantify. And because you can't measure that variable, it doesn't get controlled for, and it makes the treatment group look better than it is. The association you're seeing isn't a treatment effect — it's a baseline difference.
Eleanor Crane: An unmeasured third variable influencing both who gets treatment and what outcome they experience.
Ben Okonkwo: Exactly — that's the definition of confounding. And what randomization does is break that link. Random assignment means the health-conscious women get distributed across both arms by chance, so the treatment and control groups are exchangeable — identical in expectation across every characteristic, including the ones you never measured. That's the precise thing an RCT buys that an observational study cannot.
Eleanor Crane: In expectation — across large samples, with people who actually comply with their assigned treatment. Now that phrase is going to do a lot of work later. But the place I want to sit for a moment is: the U.S. National Academies designates RCTs as the gold standard for efficacy claims precisely because of that randomization property. And yet the HRT case shows us that the gap between the gold standard and the observational evidence wasn't just a small statistical discrepancy. It was a directional error that shaped medical practice for a generation.
Ben Okonkwo: And the question I can't stop sitting with is — why did we have such high confidence in the observational signal? The sample sizes were massive. The studies were careful. What does it tell us that unmeasured confounding can override all of that?
Eleanor Crane: That's what this episode is trying to work out.
Ben Okonkwo: Think about it this way. A hospital notices that patients who take a certain vitamin tend to live longer. Big pattern, consistent across years of records. But those same patients — they exercise, they don't smoke, they show up for every checkup. The vitamin looks like it's working. It isn't. It's a marker. A flag for people who already take care of themselves.
Eleanor Crane: The vitamin gets the credit for the healthy lifestyle.
Ben Okonkwo: Exactly. And that hidden difference — that's confounding. One clean sentence: a third variable you didn't measure is pulling both the treatment assignment and the outcome in the same direction, and it looks like causation.
Eleanor Crane: Which is what happened with HRT. Women choosing hormone replacement therapy were, on average, more health-conscious. You can't put 'general health conscientiousness' in a spreadsheet column.
Ben Okonkwo: Right — and that's the mechanical reason the signal looked so clean for so long. Confounding doesn't just add noise. That's the part people miss. It bends the estimate consistently in one direction. So a larger sample doesn't help you — you're just measuring the bias more precisely.
Eleanor Crane: Wait — more data makes you more confidently wrong?
Ben Okonkwo: In the presence of systematic confounding, yes. Millions of women, decades of data — and the direction of the error didn't change. The signal just got louder.
Eleanor Crane: That's — I mean, that reframes what sample size is actually protecting you against.
Ben Okonkwo: It protects against random noise. It doesn't touch systematic bias. And there's a specific version of this that's worth naming — confounding by indication, where sicker patients are more likely to receive a treatment. So now your treated group is already worse off at baseline. You compare their outcomes to healthier untreated patients, and the treatment looks harmful even if it isn't. Or reverse the severity, and it looks beneficial. You're comparing groups that were never comparable to begin with.
Eleanor Crane: Apples and oranges that look identical in your data.
Ben Okonkwo: And — this is where I want to push slightly further than the analogy — the reason this is so hard to fix in an observational study is that you can only adjust for variables you actually measured. 'Health conscientiousness' was never in the HRT dataset. So no statistical adjustment, however clever, reaches it.
Eleanor Crane: Which is why the question isn't whether a study was careful. It's whether the unmeasured thing is large enough to explain what you're seeing. And in the HRT case, it was.
Ben Okonkwo: And that's exactly the place randomization enters — because it doesn't fix the unmeasured variable, it renders it irrelevant. You pull a slip from a random number table, and suddenly whether a patient is health-conscious or not doesn't predict which arm they land in. That's the mechanism. It's not measurement. It's scrambling.
Eleanor Crane: Picture it concretely — a clinical trial coordinator in 1995, sealing envelopes. Each one has a slip inside: treatment or placebo, pulled from a random number table. A patient with three chronic conditions opens one envelope. A healthy volunteer opens another. And who gets treatment is an accident of chance.
Ben Okonkwo: That accident is the entire point.
Eleanor Crane: Because now the sicker patient isn't systematically more likely to end up in one arm. The health-conscious patient isn't either. The two groups become — what's the technical term you'd reach for?
Ben Okonkwo: Exchangeable. That's the word. The treatment and control groups are, in expectation, identical across every baseline characteristic — the ones you measured, and the ones you never thought to measure. So when the outcomes diverge, confounding isn't a plausible explanation anymore. The only systematic explanation left is the intervention.
Eleanor Crane: Which is why the U.S. National Academies puts RCTs at the top for efficacy claims. That formal designation rests on exactly this — exchangeability from randomization, not just bigger samples or better data collection.
Ben Okonkwo: And the 20th century recognition that drove RCTs to dominance was precisely that — observational comparisons of treatment-choosers to non-choosers were, structurally, always tangled with who chose. You can't separate the choice from the chooser.
Eleanor Crane: Or the antiretroviral therapy case — HIV treatment decisions that followed RCT evidence rather than observational associations. That's the version where the causal inference wasn't just academically cleaner. It was the thing that guided treatment and saved lives.
Ben Okonkwo: Right — and that contrast with HRT is worth holding. Both cases, observational data existed. In HIV, following the RCT evidence rather than associations was the load-bearing decision.
Eleanor Crane: So then — once you've achieved exchangeability, how do you protect it? Because randomization is the design decision, but the trial runs for years after those envelopes are opened.
Ben Okonkwo: Intention-to-Treat analysis. Participants stay in the groups they were randomized to regardless of whether they actually took the drug. Someone drops out, switches arms, stops complying — they're still analyzed where they started. It's the least biased approach precisely because it preserves the exchangeability the randomization created.
Eleanor Crane: Preserves it — but there's a cost there that I think is worth flagging, because where that cost leads is actually something we haven't fully resolved yet.
Ben Okonkwo: Noncompliance, ITT diluting the signal, what happens to internal versus external validity — that's the part that actually puts pressure on the gold-standard label in practice. We'll get there.
Eleanor Crane: But preserving exchangeability at the cost of what you're actually measuring — that's the part that doesn't get said out loud enough. ITT keeps the groups balanced, but it measures the effect of being *assigned* a treatment, not of actually *receiving* one. And those are not the same thing.
Ben Okonkwo: Right — and that gap opens exactly when noncompliance is high. Forty percent of your trial doesn't take the drug. You analyze them in the treatment arm anyway. Now your effect estimate is diluted by forty percent of people who got nothing.
Eleanor Crane: So you've traded one problem for another.
Ben Okonkwo: Precisely — and not in a neutral way. The people who don't comply aren't random. They're different in ways you can't measure. Maybe they stopped because of side effects, or because they felt worse. So now you're back in observational territory — the compliers versus the non-compliers are not exchangeable. Randomization bought you something at the start that noncompliance is quietly returning.
Eleanor Crane: Wait — so the very problem the design was built to eliminate re-enters through behavior during the trial itself?
Ben Okonkwo: Through behavior, yes. And there are other analytic approaches — per-protocol, as-treated, instrumental variable methods — but each one carries its own limitation. Per-protocol excludes non-compliers, which reintroduces selection bias. As-treated compares by actual intake, which is observational again. None of them fully solve it. They're each accepting a different kind of error.
Eleanor Crane: And then there's the population problem sitting underneath all of this — even if the trial runs perfectly, who agreed to be in it?
Ben Okonkwo: Internal versus external validity. Strong causal credibility inside the trial, but the people who volunteer for trials — they're healthier, more compliant, often less diverse than the patients who'll eventually receive the treatment. An RCT can be internally pristine and still not generalize.
Eleanor Crane: And the feasibility constraints compound this — pediatric populations, surgical interventions, rare diseases. There are entire domains where running an RCT is ethically or practically impossible.
Ben Okonkwo: Which lands us on the Cochrane and JAMA tension, and I find this genuinely hard to resolve. The 2014 Cochrane review — updated again in 2024 — found little evidence of systematic effect differences between well-conducted observational studies and RCTs. Same question, same outcomes. The designs largely agree.
Eleanor Crane: And JAMA 2001 says the opposite.
Ben Okonkwo: Found that discrepancies beyond chance are common. Not noise — systematic divergence. Same literature, same century, methodologists looking at the same evidence base and arriving at incompatible answers. That's not a small gap to paper over.
Eleanor Crane: Which is where Target Trial Emulation enters — the attempt to recover RCT-like structure from real-world data. Electronic health records, registries, observational cohorts. You design the observational study to mirror a hypothetical trial you couldn't run.
Ben Okonkwo: And the irony is — I mean, it's almost vertiginous when you sit with it — TTE's validity depends entirely on adequate confounding adjustment. The very assumption RCTs exist to make unnecessary. You've substituted a structural guarantee with an analytic claim you can never fully verify. The ghost of unmeasured confounding is still there.
Eleanor Crane: So the gold-standard label — the one the National Academies attaches to RCTs — it holds conditionally. High compliance, representative population, ethically feasible. Strip any one of those, and what remains is a design that's better than the alternative, not a guarantee.
Ben Okonkwo: And conditional is the word I keep wanting to press on. Because the Cochrane finding and the JAMA finding — they're not answering the same question, exactly. Cochrane is asking whether well-designed observational studies diverge systematically from RCTs. JAMA is asking whether divergence beyond chance is common. Those are different thresholds. But neither paper resolves which design to trust for which question. And that gap is still open in the literature right now.
Eleanor Crane: Which means the gold-standard label is doing more rhetorical work than evidentiary work. That's the thing I can't quite let go of. If methodologists can't agree on whether RCTs systematically outperform well-designed observational studies — and they genuinely can't — then calling one the gold standard is more institutional consensus than settled science.
Ben Okonkwo: Maybe the frame has to shift. RCTs as the right tool for a specific question, a specific population — not universally superior. Target Trial Emulation exists because for a lot of questions, you can't run the trial. Elderly patients with five comorbidities weren't in the RCT that generated the guideline. So the observational data is what you have, and TTE is an attempt to make it as clean as possible — knowing that the unmeasured confounding problem never fully disappears.
Eleanor Crane: That's an uncomfortable place to land. Because what it says is — we've moved the uncertainty, we haven't resolved it.
Ben Okonkwo: Yeah. And the HRT case is the evidence for why that matters outside a methods seminar. Millions of women, decades of practice, built on a signal that was confounded from the start. The question of which design to trust — that had consequences that were measured in cardiovascular events.
Eleanor Crane: And I'm not sure we've actually learned how to know, in advance, which design to trust. We've learned to describe the conditions under which each one holds. That's real. But the honest place is — we've named the uncertainty more precisely. We haven't dissolved it.
Ben Okonkwo: That's probably the right note for this one. Naming uncertainty precisely is not nothing — but it's not a guarantee either. Good thinking through it with you.
