Zara Reyes: Hey — you look like you've already been arguing with someone this morning.
Megan Skiendel: Frankly, only with a paper. I pulled the original Women's Health Initiative coverage and I cannot stop thinking about one number. Forty-plus observational studies. A 1991 synthesis. About fifty percent estimated cardiovascular risk reduction from hormone therapy — fifty — and then one trial comes along and buries all of it.
Zara Reyes: And that's framed as science correcting itself.
Megan Skiendel: Exactly how it's framed. Except — look, the Women's Health Initiative enrolled women averaging sixty-three years old, over a decade past menopause. The observational studies were following women who started hormone therapy right at menopause. Same treatment. Completely different biological window. And we let the RCT overwrite the whole file.
Zara Reyes: So it's not that the randomized trial was wrong — it's that it was asking a different question about a different population and got treated like it answered the original one.
Megan Skiendel: Which is the whole thing we're here to pull apart. Because that move — where the higher-ranked study type claims authority over a question it didn't quite study — that's not a one-time error. That's what happens when a credibility structure becomes a rule. The hierarchy was built to get us out of hidden bias in observational work. But what if the bias just moved up a level and got insulated by the pyramid's own logic?
Zara Reyes: It doesn't eliminate the bias. It promotes it.
Megan Skiendel: And gives it a methodological credential. That's the problem.
Zara Reyes: But before we can even get to that — the reason the pyramid exists at all is kind of beautiful, actually. Like, the intuition is so simple. Imagine you notice that everyone carrying an umbrella seems to get wet. You might conclude umbrellas cause rain. That's confounding. A hidden variable — actual rain — makes two things look causally linked when they're not. The whole pyramid is just a machine for catching that.
Megan Skiendel: And that is a genuinely good machine to build.
Zara Reyes: Right — it was a revolution. Archie Cochrane, 1972, Effectiveness and Efficiency — his whole argument was that medicine had never actually tested itself. Doctors were just pattern-matching from whatever they'd seen in their own practice. Cochrane said: that's the umbrella problem at scale.
Megan Skiendel: And it took another two decades before Sackett and that whole cohort actually embedded it — into education, into clinical guidelines. The pyramid didn't come pre-installed. Someone had to wire it into the institution.
Zara Reyes: Which is — wait, that's the part I want to sit on. Because random allocation, the thing that puts RCTs at the top — that's the only design that actually eliminates selection bias at the source. Not controls for it. Eliminates it. That's real.
Megan Skiendel: Genuinely real. But — honestly, here's what that wiring cost. Once the pyramid got into liability frameworks, into which drugs a hospital would cover — it stopped being a thinking tool. It became a shield. And a shield doesn't care if it's blocking the wrong thing.
Zara Reyes: So Cochrane builds a flashlight and Sackett et al. turns it into a wall.
Megan Skiendel: That's — yeah. And the Women's Health Initiative is what happens when the wall holds even after the flashlight would've said look over there instead.
Zara Reyes: But the wall held in a specific direction — like, it held against the observational studies, not against the WHI's own design problem. That asymmetry is what gets me.
Megan Skiendel: Walk me through the healthy-user piece first, because I think people hear 'confounding' and they think noise. This was structural.
Zara Reyes: Right — women who chose hormone therapy at menopause were systematically healthier to begin with. Wealthier, better access to care, more likely to exercise. The observational studies were measuring that whole profile, not the estrogen. So the fifty percent cardiovascular benefit — it was picking up 'healthy person behavior,' not drug effect.
Megan Skiendel: That part the WHI genuinely fixed. Randomization scrambles the healthy-user sorting. That's real.
Zara Reyes: It fixed that. And then — wait, it introduced its own version. Because enrolling women who are sixty-three, over a decade past menopause, to study a therapy most people start at fifty? That's not a minor sampling wrinkle. The biology is different.
Megan Skiendel: Honestly, the Danish Osteoporosis Prevention Study is the thing that makes this undeniable. DOPS actually enrolled recently menopausal women — the population the original observational cohorts were tracking — and it found cardiovascular benefit. So the WHI didn't settle HRT. It settled HRT for a specific age window the observational studies were never primarily about.
Zara Reyes: The bias didn't disappear. It just got a different address.
Megan Skiendel: Now — the beta-carotene story is the contrast that makes this legible. The CARET trial found that beta-carotene supplements actually increased lung cancer risk in high-risk populations, flipping prior observational optimism. No population mismatch wrinkle. Clean correction. The hierarchy just worked. Which means the problem isn't RCTs per se — it's what question you chose to ask with yours.
Zara Reyes: And if that design choice is invisible enough — the part that keeps me up — is what it means when you get to meta-analyses, the supposed apex. One evaluation, 163 of them, wildly different conclusions depending on analytical choices. That's where we're going next and frankly it makes this whole arc worse.
Megan Skiendel: And that's exactly where it collapses in the most embarrassing way. Because if you're a cardiologist, eleven p.m., you pull up a meta-analysis to settle a treatment question for a fifty-eight-year-old woman — and you find two meta-analyses on the same intervention pointing opposite directions. That's not a hypothetical.
Zara Reyes: It's documented. 163 meta-analyses — one evaluation — wildly different estimated outcomes depending on analytical strategy.
Zara Reyes: Same apex. Same study type sitting at the top of the pyramid. And the conclusion changes based on how the researcher chose to slice the data. So we built this whole structure to get bias out of the lower levels — and the top level turns out to be, like, maximally sensitive to methodological choices that nobody has to disclose.
Megan Skiendel: It's not aggregation. It's curation dressed up as aggregation.
Zara Reyes: And the people who figured this out first weren't outsiders. The GRADE Working Group — early 2000s, built from inside the EBM community that made the pyramid — they formalized it. They said: RCT evidence can be downgraded for bias or inconsistency. Observational evidence can actually be upgraded for large effect sizes or dose-response patterns. That's the original hierarchy's own architects saying the ranks aren't fixed.
Megan Skiendel: Honestly, BMJ Evidence-Based Medicine published a whole 'new evidence pyramid' proposal making that explicit — the hierarchy is a contextual default, not a law. And that landed, what, quietly?
Zara Reyes: So quietly. Because the institutional version — the one in the guidelines, in liability law — that one didn't update. The thinking tool updated. The shield didn't.
Megan Skiendel: Which means the bias didn't just relocate from the bottom of the pyramid to the top. It got tenure.
Zara Reyes: And that's — I mean, that's the thing that actually stays with me. Not that the pyramid is wrong. It does exactly what it was designed to do. It ranks how well a study design controls for confounding. That's it. That's the whole job. But a clinician isn't asking 'which study design is least confounded.' They're asking 'does this finding apply to the person in front of me right now.'
Megan Skiendel: And those are genuinely different questions. The pyramid has no answer to the second one. It was never built to.
Zara Reyes: The WHI is — honestly, it's the clearest example of that gap. Internally valid. Randomization worked. The confounding got scrambled. And it was still the wrong answer for a fifty-two-year-old just hitting menopause, because RCTs produce population-level estimates, and that population wasn't her.
Megan Skiendel: So the question I keep turning over — not whether the hierarchy is broken, but whether we've been asking it to answer something it was never designed to answer. External validity. Generalizability. The specific patient. The pyramid is silent on all of that.
Zara Reyes: And I don't think either of us has a clean answer to that.
Megan Skiendel: No. Frankly, I think that's the right place to stop — sitting with the discomfort of it. This was a good one.