Onpode
Cover art for How control groups and blinding eliminate the observer effect from results

How control groups and blinding eliminate the observer effect from results

November 15, 2026 · 14 min

Tess Hollis & Felix Ortiz

Tess, good — you're here, I've been sitting with the strangest historical detail and I need to say it out loud to someone. That's why I show up. What is it? Benjamin Franklin, 1784, Paris — he is not doing diplomacy, he is not doing electricity, he is putting a blindfold on a patient who…

Experimental design is a methodological framework used across empirical sciences to isolate causation from confounding factors. Its two foundational tools are control groups and blinding.

0:0013:35
Get the next episode on Science →

Follow it free — new episodes land in your feed.

Or make your own — any topic, in minutes

More Onpode episodes on Science →

About this episode

Experimental design is a methodological framework used across empirical sciences to isolate causation from confounding factors. Its two foundational tools are control groups and blinding.

Grounded in 12 sources
Enhancing Longitudinal Clinical Trial Efficiency with Digital Twins and Prognostic Covariate-Adjusted Mixed Models for Repeated Measures (PROCOVA-MMRM) ↗ · arxiv.org
Intentional Ignorance: A History of Blind Assessment and Placebo Controls in Medicine ↗ · doi.org
Inventing the randomized double-blind trial: the Nuremberg salt test of 1835 ↗ · ncbi.nlm.nih.gov
Blinding in Clinical Trials: Seeing the Big Picture ↗ · pmc.ncbi.nlm.nih.gov
Franklin, Lavoisier, and Mesmer: origin of the controlled clinical trial ↗ · pubmed.ncbi.nlm.nih.gov
The origins of modern clinical research - PubMed ↗ · pubmed.ncbi.nlm.nih.gov
Blinding (masking) | NC3Rs EDA ↗ · eda.nc3rs.org.uk
Royal Commission on Animal Magnetism - Wikipedia ↗ · en.wikipedia.org
Blinded experiment - Wikipedia ↗ · en.wikipedia.org
6 historic experiments that helped pave the way for modern clinical research ↗ · gavi.org
[PDF] Benjamin Franklin and the Debunking of Mesmerism ↗ · ijuh.org
Blinding ↗ · iwh.on.ca
Read transcript

Felix Ortiz: Tess, good — you're here, I've been sitting with the strangest historical detail and I need to say it out loud to someone.

Tess Hollis: That's why I show up. What is it?

Felix Ortiz: Benjamin Franklin, 1784, Paris — he is not doing diplomacy, he is not doing electricity, he is putting a blindfold on a patient who thinks they are about to be magnetized by Franz Anton Mesmer's animal magnetism fluid. And Antoine Lavoisier is standing next to him helping design the experiment.

Tess Hollis: Wait — Lavoisier, as in the oxygen guy.

Felix Ortiz: The same person, yeah — and Joseph-Ignace Guillotin is also in that room, so that commission has a real 'this is a lot of historical weight for one meeting' energy. But the reason I can't stop thinking about it is: they were trying to prove Mesmer was a fraud. They succeeded — the Royal Commission on Animal Magnetism concluded animal magnetism had no physical basis. That part worked. Except —

Tess Hollis: Except the patients got better anyway.

Felix Ortiz: Even with the sham intervention. Even when the treatment was fake. That's the placebo response, surfacing for the first time in a systematic way, and it shows up as a byproduct of a debunking exercise. I find that — yeah, that's genuinely strange.

Tess Hollis: And the deeper move — the one I want to hold onto — is that Franklin's commission didn't just blind the patients. They blinded Mesmer's own practitioners. Which is a different kind of claim entirely.

Felix Ortiz: Oh — right, because if you only blind the patient, you're guarding against patient expectation. But if you blind the practitioner too, you're saying the person delivering this treatment will unconsciously tilt the result if they know what they're delivering. That's observer bias as a design problem, not a character flaw.

Tess Hollis: That's the mechanical insight the whole thing is built on. And it's the same mechanism we use today — the double-blind trial, blinding from both ends. Invented to catch one specific fraud, 240 years ago.

Felix Ortiz: So the question that actually runs through today is — okay, we've had this tool since 1784, it works, and yet we're still arguing about how to use it. What does the method actually do, and is there something it structurally cannot see?

Tess Hollis: And that question — what can the method see versus not see — actually starts one layer earlier. Before blinding even enters it. Because you need to explain why you need a comparison group at all.

Felix Ortiz: Okay yeah — start there.

Tess Hollis: Picture a soccer coach. His team just had a brutal match — everyone played badly, they lost badly. So that week he gives them a new recovery shake. Next match, they bounce back, play well. And he thinks: the shake did that.

Felix Ortiz: But they were probably always going to bounce back.

Tess Hollis: Exactly that. When you pick people — or teams — at their worst, they tend to move toward average on their own. That's regression to the mean. It's not magic, it's just how variation works. So without a second group that didn't get the shake, playing the same schedule, with the same players — you cannot tell the shake apart from time passing.

Felix Ortiz: That's — wait, that's the whole thing, right? The control group is just asking: what would have happened anyway?

Tess Hollis: That's what it is. That's the sentence. The control group is the counterfactual — identical in every measurable way, except it didn't get the intervention.

Felix Ortiz: And confounding factors are just — anything that breaks that 'identical' assumption. Like, if the shake group also happened to get more sleep that week, now you don't know what did the work. It's not the shake versus nothing, it's the shake-plus-sleep versus nothing, and you can't untangle them.

Tess Hollis: Right — and this is where the randomized controlled trial is doing something specific. Random assignment to treatment or control spreads those confounding factors across both groups by chance. So on average they cancel. That's the actual mechanism, it's not just procedural tidiness.

Felix Ortiz: So the RCT isn't three separate inventions bolted together — randomization, control group, blinding — it's more like, each one is patching a different hole in the same argument.

Tess Hollis: That's the design. Control group answers 'what happens anyway.' Randomization answers 'are your two groups actually comparable.' Blinding answers 'are you seeing what's there or what you expect.' Pull any one out and you get a different kind of wrong answer.

Felix Ortiz: Okay and — wait, the part that actually breaks my brain a little is regression to the mean, because it means researchers can run a study, see real improvement, and be genuinely deceived. Not dishonest. Just wrong in a systematic, predictable way. That's why regulatory agencies mandate the full RCT design now — it's not bureaucratic caution, it's that every piece is load-bearing.

Tess Hollis: But that 'each piece is load-bearing' framing — it actually hides something. Because most people, when they say 'blinding,' mean one thing. And it's two.

Felix Ortiz: Two separate problems, right — wait, say that slower.

Tess Hollis: The patient's belief produces the placebo response. The researcher's expectation produces observer bias. Those are not the same mechanism.

Felix Ortiz: Oh — okay, that actually reframes the whole Franklin thing, because when the Royal Commission blinded Mesmer's own practitioners, that wasn't about the patients at all. That was purely about observer bias. They were saying: your magnetizers will unconsciously see improvement if they know they just delivered the treatment.

Tess Hollis: And those are two different targets requiring one shared solution. Which is why collapsing them is such a problem.

Felix Ortiz: Because if you only blind the patient — single-blind — you've shut one channel and left the other completely open. The researcher still knows. Every marginal call, every ambiguous data point, every 'does this count as improvement' judgment — that all runs through someone who knows which group got the drug.

Tess Hollis: And observer bias is operating totally independently of anything happening inside the participant.

Felix Ortiz: That's the part that — yeah, researchers literally cannot replicate their own findings without blinding. Not other people's work. Their own. That's not theoretical, that's: I ran this experiment, I know what should happen, and that knowledge is the confound.

Tess Hollis: Which brings us to 1835.

Felix Ortiz: The Nuremberg salt test — yeah, okay, this is the one that gets me. A 'society of truth-loving men' — that's the actual name — evaluating homeopathy. Randomization and double-blinding, 1835. Not the mid-twentieth century, not some modern regulatory invention. 190 years ago.

Tess Hollis: And it was adversarial. Same as Franklin. The method wasn't invented as a neutral truth-seeking tool — it was invented twice, against specific practitioners, whose claims were already politically inconvenient.

Felix Ortiz: Right — wait, no, I want to sit with that for a second because it's actually uncomfortable. If the double-blind trial was forged as a weapon against mesmerism and homeopathy, does that change how we read its authority? Or is the method just — good, regardless of the motive?

Tess Hollis: That's the tension I don't think we should resolve too fast. And it gets stranger — because the placebo response those same experiments exposed turns out to be a real physiological event, not a perception trick, which means we're designing experiments to control for something that's genuinely doing biological work. That's the part we need to get to.

Felix Ortiz: And that's — yeah, that's the thing. Because 'real physiological event' is doing a lot of work in that sentence. What does that actually mean, mechanically?

Tess Hollis: It means your body responds to belief. Not metaphorically. Measurably. Opioid release, blood pressure changes, documented symptom reduction — driven purely by the participant's expectation of receiving an active treatment. No pharmacological action required.

Felix Ortiz: Wait — opioid release from a sugar pill?

Tess Hollis: From the belief that you got the real thing. Which is why a sham intervention isn't just about fooling people for the sake of it. The control group has to receive something indistinguishable from the real treatment — so the placebo response is present in both arms equally. Then you can measure it and subtract it.

Felix Ortiz: Oh — okay, so you're not eliminating the placebo response. You're holding it constant across both groups so it cancels.

Tess Hollis: Right. And whatever's left — that gap between groups — that's the treatment effect. That's what you're actually measuring. The sham intervention is load-bearing.

Felix Ortiz: And Franklin's commission actually proved this in both directions at once — because Mesmer's patients genuinely felt better. The Royal Commission's finding didn't make that improvement go away. It just showed animal magnetism wasn't necessary to explain it.

Tess Hollis: That's the part nobody sits with long enough. The improvement was real. Full stop.

Felix Ortiz: Which means — wait, actually this is the uncomfortable part — if you then design an experiment to control for placebo, to subtract it out, you might be eliminating the very mechanism by which that treatment was actually working in practice. Like, for some patients, the belief component isn't noise. It's — it's load-bearing too.

Tess Hollis: That's the paradox. Maximizing internal validity — controlling for placebo — can reduce external validity in settings where placebo is part of how treatment works in the real world.

Felix Ortiz: So you're trading a clean answer in the lab for a less accurate picture of what happens in the clinic.

Tess Hollis: And the Rubin causal model formalizes exactly why the trade happens. Randomization makes treatment assignment independent of potential outcomes — that's the epistemological guarantee. Mean differences between groups become unbiased estimators of the true treatment effect. It's airtight inside the experiment. But the experiment is not the clinic.

Felix Ortiz: So the RCT gives you the clearest possible signal — stripped of belief, stripped of expectation — and that stripping is exactly what makes the signal hard to translate back.

Tess Hollis: That's the bet you're making every time you run one. What you gain in confidence, you trade in context. And nobody puts that in the methods section.

Felix Ortiz: And that's — yeah, that's exactly where I keep snagging on something. Because we now have tools like PROCOVA-MMRM, right — AI-assisted prognostic covariate adjustment, designed to make trials smaller, faster, preserve the RCT logic nominally. And the pitch is basically: we can statistically account for confounders instead of relying purely on randomization to distribute them. Which sounds like progress, except — wait, no, the whole point of randomization was that it handled confounders you don't know to measure.

Tess Hollis: The model only catches what it's looking for.

Felix Ortiz: Right — and that's the thing I genuinely cannot resolve. Because covariate adjustment introduces dependence on model assumptions that randomization was invented specifically to make unnecessary. You're trading an epistemological guarantee for a statistical bet. And that bet is only as good as your list of measured confounders.

Tess Hollis: Which is exactly the structure of the problem Franklin was solving in 1784. Mesmer's practitioners thought they knew what they were measuring. They had a whole theory about animal magnetism — they were tracking the right variables, by their own account. They just couldn't see the bias because they were inside it.

Felix Ortiz: So the question is — and I genuinely don't know the answer — are we about to repeat that? Except now the thing we can't see isn't inside a magnetizer's hands. It's inside the statistical model. The unmeasured confounders are still there. We just stopped making them irrelevant by design and started hoping we named them all.

Tess Hollis: And I can't answer that either. What I can say is: anecdotes fail for the same set of reasons Mesmer's pre-1784 case reports failed. Natural variation, regression to the mean, placebo response, observer bias — all active, all invisible at once, without controls. PROCOVA-MMRM isn't anecdote, it's not that bad. But the gap between 'this is more efficient' and 'this preserves the guarantee' — I don't think we know how wide that gap is yet.

Felix Ortiz: That's an uncomfortable place to sit for two people who just spent an hour on why the method works.

Tess Hollis: Maybe that's where it should end, though. Not with a verdict. The depth is kind of the point.