Onpode
Cover art for The evidence pyramid: why anecdotes lose to randomized trials structurally

The evidence pyramid: why anecdotes lose to randomized trials structurally

November 22, 2026 · 12 min

Juniper Vale & Hope Sterling

Randomized trials outrank anecdotes structurally because a coin flip balances unmeasured confounders that statistical adjustment cannot touch. The 2002 Women's Health Initiative showed hormone therapy raised coronary events by ~29%, reversing observational studies that looked convincing — because healthier women had simply chosen hormones more often.

Evidence-based medicine (EBM) organizes research study designs into a hierarchy ranked by their resistance to systematic error — commonly visualized as a pyramid. At the base sit anecdotes, case reports, and expert opinion, which carry the highest risk of bias.

0:0011:42
Get the next episode on Health →

Follow it free — new episodes land in your feed.

Or make your own — any topic, in minutes

More Onpode episodes on Health →

About this episode

Most of us know anecdotes aren't great evidence. Fewer of us know exactly why — or why fixing them isn't as simple as getting more of them. This episode works through the structure of the evidence pyramid from the ground up, using a real case that still unsettles researchers: hormone therapy and heart disease. For years, large careful observational studies suggested hormone users had roughly half the heart disease risk. Then a 2002 randomized trial found the opposite. Nobody lied. The problem was confounding — the women who chose hormones were already healthier in ways no spreadsheet fully captured. The episode explains why statistical adjustment can only go so far, what randomization actually does (and why it's almost embarrassingly simple), and why blinding addresses a completely different human mistake than randomization does. It also pushes back on the idea that the top of the pyramid is clean: a 2023 Cochrane review and a 2024 narrative review looked at the same face-mask data and issued contradictory verdicts, depending on what kinds of evidence each allowed in. The episode doesn't resolve that tension — it sits with it honestly. What it does offer are two questions worth carrying into any health claim, and a clearer sense of what the pyramid is actually for.

Frequently asked

Why do randomized controlled trials rank higher than observational studies in the evidence hierarchy?

Randomized controlled trials rank higher because random assignment balances both measured and unmeasured confounders between groups. Statistical adjustment in observational studies can only correct for variables researchers recorded. Unmeasured factors — called residual confounding — still push the result in one direction, producing precise but systematically wrong estimates.

What was wrong with the observational evidence on hormone therapy and heart disease?

Observational studies on hormone therapy and heart disease were distorted by confounding: women who chose hormone therapy were healthier, wealthier, and saw doctors more often. When the 2002 Women's Health Initiative randomly assigned hormones, the estrogen-plus-progestin group had approximately 29% more coronary events — the opposite of what earlier studies suggested.

What is confounding in medical research?

Confounding in medical research occurs when a third variable influences both who receives a treatment and what outcome they experience, creating a false appearance of cause and effect. Residual confounding persists even after statistical adjustment because researchers can only correct for factors they measured and recorded in the first place.

Are systematic reviews and meta-analyses always the most reliable evidence?

Systematic reviews and meta-analyses sit at the top of the evidence pyramid but are not automatically reliable. A review of 30 years of RCTs in The American Journal of Sports Medicine found persistent reporting-quality problems. The conclusions also depend on which study types researchers allow in — two 2023–2024 face-mask reviews reached opposite verdicts using the same underlying data.

Do anecdotes and case reports have any value in evidence-based medicine?

Anecdotes and case reports cannot control for confounders, but they generate hypotheses and are often the first signal of rare adverse events. Observational studies remain essential when randomization would be unethical, such as studying the harms of smoking. The evidence pyramid ranks study designs by reliability, not by whether a design has any value.

Grounded in 10 sources
What is the best evidence and how to find it | BMJ Best Practice ↗ · bestpractice.bmj.com
Assessment of 30 Years of Randomized Controlled Trials in The American Journal of Sports Medicine: 1990-2020 ↗ · doi.org
The Effectiveness of Mind–Body Exercise on Health‐Related Quality of Life and Mental Health During and After Breast Cancer Treatment: An Umbrella Review of Meta‐Analyses for Randomized Controlled Tria ↗ · doi.org
Protocol: a systematic review+ (SR+) to combine associative and mechanistic evidence on the efficacy of face masks in reducing transmission of respiratory diseases ↗ · doi.org
4 M's to make sense of evidence – Avoiding the propagation of mistakes, misinterpretation, misrepresentation and misinformation ↗ · doi.org
Understanding the Levels of Evidence in Medical Research ↗ · doi.org
New evidence pyramid | BMJ Evidence-Based Medicine ↗ · ebm.bmj.com
Healthy User and Related Biases in Observational Studies of Preventive ... ↗ · pmc.ncbi.nlm.nih.gov
Understanding the Levels of Evidence in Medical Research - PMC ↗ · pmc.ncbi.nlm.nih.gov
FDA and Clinical Drug Trials: A Short History ↗ · fda.gov
Read transcript

Hope Sterling: Hey, you made it — okay, I have to tell you, my friend texted me at like 8 a.m. about this mushroom powder, and she has SO much energy now, and I literally spent ten minutes going, but did the mushrooms do it?

Juniper Vale: Hi, you. Oh, I know that text. Did she change anything else that week?

Hope Sterling: Sleeping eight hours, quit drinking, and it's finally spring. Like, three other suspects!

Juniper Vale: Then you've already got the whole episode in a mug. Let me back up to a Tuesday in 1999. A 54-year-old woman sits in her doctor's office, and the doctor says, the research shows this hormone therapy protects your heart.

Juniper Vale: She isn't being careless. That really was the best evidence at the time. Then in 2002 the Women's Health Initiative randomly gave thousands of women either hormones or a placebo, and the estrogen-plus-progestin group had about 29 percent more coronary events.

Hope Sterling: Stop. More heart problems? When the earlier studies said hormone users had way fewer? I cannot with this. Who lied?

Juniper Vale: Nobody. Think of it like this: you can't tell if fertilizer works if you only gave it to the gardens that already had the best soil. The women who chose hormones were healthier, wealthier, and saw doctors more often. A coin flip scrambles the soil.

Hope Sterling: Oh, the healthy girlies got the credit, and the pills just stood next to them! Okay but, like, these were careful researchers. So what does being fooled even mean here?

Juniper Vale: Being fooled doesn't mean being sloppy. The Nurses' Health Study was huge and careful, and it suggested hormone users had roughly half the heart disease. The problem is a thing called confounding. That's a variable that shapes who gets the treatment and also shapes the outcome, so it fakes a link.

Hope Sterling: That's the mushroom text! Sleep, no drinking, spring. Those are confounders, like, literally confounding in a mug. I finally have the word, I'm living for this.

Juniper Vale: Same structure, bigger scale. Patients who choose pricier therapies often have more resources, earlier diagnoses, better follow-up. So the improvement can come from who they already were.

Hope Sterling: Wait, so why not just, like, fix it on paper? Compare rich hormone ladies to rich non-hormone ladies? Is that not a thing?

Juniper Vale: It is, it's called adjusting. They did it. But you can only adjust for what you measured. I mean, income, sure. But how much she cares about her health, how she eats, whether she walks every morning? Whatever's left over is called residual confounding.

Hope Sterling: Ohh. So it's the stuff nobody wrote down. Like if my friend's mushroom-powder energy is secretly, I don't know, she's just a person who's finally, like, in a good place, and there's no checkbox for that. Ugh.

Juniper Vale: Right, right, and that's the part that matters. A bigger sample doesn't help. It just gives you a more precise wrong answer, because the error pushes the same direction every time. That's bias in the technical sense: systematic error, not random noise.

Hope Sterling: Systematic, not random. Hm, so random error is like a wobbly scale, and this is a scale that's always, like, five pounds light?

Juniper Vale: Yes, a scale that's always five pounds light. And that's why you need something sturdier than adjusting. So here's the fix, and it's almost embarrassingly simple: randomization. Assign people to treatment by chance, like a coin flip.

Hope Sterling: A coin flip? That's it? Like, that's the big fancy thing?

Juniper Vale: That's it. Because a coin doesn't know who walks every morning. It can't favor the healthy girlies. So both groups end up with about the same mix of walkers, worriers, wealthy people, all of it.

Hope Sterling: Wait, wait. Even the stuff nobody wrote down? The no-checkbox stuff? Like, the "she's just in a good place" stuff?

Juniper Vale: Even that. Statistical adjustment can't balance a variable nobody measured, but chance balances it anyway, without anyone naming it. It also kills selection bias, meaning nobody gets to pick who gets the pill.

Hope Sterling: Oh my gosh. Okay, that's my chills moment, like, literally. It fixes things we don't even know exist? That's kind of unfair to every other study design.

Juniper Vale: Mm-hm. But there's a second human mistake, and randomizing doesn't touch it. Once people are in their groups, everyone starts watching. Doctors know who got the real drug. Patients know too.

Hope Sterling: Ohh, so like, if I know I'm on the good stuff, I'm gonna feel better? Or my doctor's gonna, like, squint at me and go, you look great?

Juniper Vale: Both. That's observer bias: clinicians unconsciously seeing the improvement they expect. And there's a sneakier one. Doctors might give the treated group extra attention, extra advice. That's called unequal co-interventions. Double-blinding means neither patient nor clinician knows who got what.

Hope Sterling: So randomization is who gets what, and blinding is what the observer sees.

Juniper Vale: That's it exactly. Each one fixes a specific human mistake.

Hope Sterling: Okay, but then, like, a hundred friends all texting me about the mushroom powder. Doesn't a hundred beat one? Math says more is more, right? Wait, no, you said bigger samples don't fix it. Ugh, I'm losing the thread.

Juniper Vale: You found it, though. A hundred anecdotes share the same uncontrolled biases. Same spring, same sleep, same hopeful friends. So it's one anecdote's flaw repeated a hundred times, and one well-run RCT outweighs them.

Hope Sterling: Stop, that's so clean. Okay, so who decided all this was the ranking? Like, who made the pyramid?

Juniper Vale: In the early 1990s, Gordon Guyatt and David Sackett at McMaster University formalized it, building on Archie Cochrane's earlier push for rigorous trial evidence. The idea: decisions follow study design quality, not just expert opinion. At the top sit systematic reviews and meta-analyses, which pool all the RCTs using pre-specified criteria. The FDA bakes it into drug approval, BMJ Best Practice carries it to clinicians, and Cochrane produces the reviews. But the top of the pyramid has a catch, and we'll get to it next.

Hope Sterling: Okay but wait, if the top is systematic reviews of RCTs, then that's the answer, right? Like, case closed? Because I read that two reviews on face masks looked at the same data and said opposite things. Wait, what?

Juniper Vale: They did. A 2023 Cochrane review only let in RCTs, and it said the evidence for masks was weak and non-definitive. Then a 2024 narrative review added mechanistic evidence, and it called efficacy strong and definitive.

Hope Sterling: Same data. Opposite verdicts. I cannot with this.

Juniper Vale: Mechanistic evidence just means how something works physically or biologically. Like, what a mask does to droplets. The strict camp says keep it out. You get a clear, reproducible verdict, and lower-tier evidence dilutes the rigor. Trisha Greenhalgh argues the other way: use synthesis methods that value multiple kinds of evidence.

Hope Sterling: Okay, so who's right? Like, literally, pick one!

Juniper Vale: I'm not going to pick. What I'll say is the pyramid is a tool, not a verdict. Which evidence you let in changes the answer.

Hope Sterling: Hmm. But mechanism feels so convincing, like, masks block stuff, obviously? And then I remember the hormone thing, and that made sense too, right? Like, estrogen and hearts, it sounded so logical, and, okay, it was wrong. So plausibility is, what, a trap?

Juniper Vale: Not a trap. Mechanism supplies causal logic that associations can't. But on its own it's a hypothesis. Hormone therapy had a lovely biological story and still missed on hearts.

Hope Sterling: Fine, so RCTs at the top are safe, though.

Juniper Vale: Not automatically. A systematic review of thirty years of RCTs in The American Journal of Sports Medicine, 1990 to 2020, found reporting quality problems that persisted the whole time. The label doesn't guarantee a well-run trial.

Hope Sterling: Thirty years?! Okay so, like, the crown doesn't make the king. And then, mmm, this is why I get twitchy calling anecdotes the bottom, because isn't a weird rare side effect usually first spotted by one person going, hey, this happened to me?

Juniper Vale: Yes. Case reports and anecdotes can't control for confounders, but they generate hypotheses and flag rare adverse events. They're starting questions. And observational studies are essential when randomizing would be unethical, like assigning people to smoke.

Hope Sterling: Oh, that's good news, honestly. Nobody gets thrown out.

Juniper Vale: Here's the hardest one, though. The Women's Health Initiative participants averaged about 63, more than a decade past menopause. Many observational users started near menopause. The timing hypothesis says the trial may not have overturned the earlier studies for younger women. A good RCT answers a specific question about a specific population.

Hope Sterling: Okay, so I keep thinking about my friend and the mushroom powder. Like, I can't text back "where's your randomized trial," I'd lose her. How do I ask how we know without sounding like a jerk?

Juniper Vale: Ask it as curiosity, not a challenge. Something like, what else changed that week? You did it in your first text, and nobody got hurt. Then there are two questions I'd carry around. What kind of study did this claim come from? And could anything besides the treatment explain the result?

Hope Sterling: Two questions. Okay, that fits in a group chat. Wait, but now I'm thinking about the newer stuff. Like, real-world data, wearables, AI summarizing a million patient records. Where does that go on the pyramid?

Juniper Vale: I don't think it maps cleanly. Real-world data is observational, so confounding follows it in. But it's huge and fast, and sometimes it catches what trials can't. And AI-generated evidence isn't even a study design yet. It's more like a new way of producing claims.

Hope Sterling: So a dynamic pyramid sounds great, but who decides what's allowed in? Because, like, hormone therapy was plausible AND well-measured, and it still got in the door.

Juniper Vale: That's what I'm still turning over. A flexible hierarchy could absorb new data and keep its honesty, if it keeps asking the bias question of every tier. But I can picture the version where flexible just means whatever sounds convincing. I don't know which one we'd get, and I don't think data types are going to wait for trials to catch up.

Hope Sterling: Mm. Honestly, I'm okay not knowing yet. I'll just ask my friend what else changed, and ask nicely.

Juniper Vale: And tell her the spring part was your favorite suspect. Go drink some coffee, and let me know if the mushrooms win.