Jordan Hale: Alex, I want to start with a confession — I went into this thinking I understood what Anthropic was doing with Claude, and then I actually read the 2022 Bai et al. paper and realized I had been confidently wrong for like a year.
Alex Mercer: What were you wrong about specifically?
Jordan Hale: I thought Constitutional AI meant — no reward model. Like, full stop. The whole pitch felt like: we're done with RLHF, we're done with humans rating outputs, the model just... learns from principles. Which sounds almost philosophical, you know? But stage two — I did not clock stage two.
Alex Mercer: Stage two being RLAIF. Reinforcement learning from AI feedback. There's still a reward model — trained on AI-generated pairwise comparisons rather than human ones, but a reward model.
Jordan Hale: Which means the paradigm didn't disappear, it just — the thing generating the preference data changed. And I keep wondering if that's a genuinely meaningful difference or if we're essentially calling the same structure by a new name.
Alex Mercer: To some extent it's both. Stage one is a real departure — the self-critique loop, critique and revise against the constitution, fine-tuning on those revised outputs — that's not how RLHF works at all. No human labels. But then stage two pulls you back toward familiar territory.
Jordan Hale: So it's not a clean break. It's more like — half a break.
Alex Mercer: Which is interesting, because Anthropic built the Claude model family on this. And the question of whether that half-break is a breakthrough depends entirely on what you think was actually broken about RLHF in the first place.
Jordan Hale: Okay, and that's — yeah, that's exactly where I want to dig in. Because if the problem with RLHF was scalability, human annotation costs, inconsistency across raters — maybe CAI solves that. But if the problem was something deeper, like whose values get baked in, then moving from human preference labels to AI-generated ones... does that actually help?
Alex Mercer: And now you're at the constitution. Which someone wrote. At Anthropic.
Jordan Hale: And whoever wrote that constitution — like, that's the whole load-bearing wall, right? But wait, can we back up one second, because I don't think we've actually explained *how* stage one works. Like the mechanism. Because when I finally got it, it was almost embarrassingly simple.
Jordan Hale: No wait, imagine you write an essay, and then your teacher hands you a rubric and says: grade your own draft, find what's wrong, rewrite it. The teacher never reads the original. That rewrite is what gets submitted. That's stage one. The model writes something, gets handed the constitutional principles as a rubric, critiques its own output, revises it — and Anthropic fine-tunes on *those revised outputs*. Zero human preference labels. That's the whole trick.
Alex Mercer: That's basically it. The supervised learning phase is genuinely annotation-free — the self-critique loop produces the training data. Which is a real structural difference from RLHF.
Jordan Hale: It's clean. Like, almost too clean.
Alex Mercer: That's the problem. Because the clean story stops at stage one. Stage two — RLAIF — an AI evaluator compares pairs of responses according to the constitution, generates preference data, and you use that to train a reward model. Which then drives reinforcement learning. The reward model didn't go away.
Jordan Hale: Wait — so Bai et al. in 2022 are saying 'we reduced reliance on human preference data,' but there's still a reward model sitting in stage two, just seeded by AI comparisons instead of human ones?
Alex Mercer: Exactly. And that distinction matters because a reward model can be gamed, overfit, misspecified — all the failure modes don't vanish. They just shift upstream to whoever drafted the constitution.
Jordan Hale: Which is — I mean, that's a human. Writing principles. With blind spots. And if a principle is poorly phrased, you don't get a misaligned AI that *knows* it's confused, you get one that's just... confidently executing a bad rubric.
Alex Mercer: Harvard researchers actually documented this in an evaluation of Inverse Constitutional AI — models overfitting to specific phrasings within principles, or misinterpreting them outright. The phrase that stands out to me is 'faithful but erroneous alignment.' The model did exactly what the principle said. The principle was just wrong.
Jordan Hale: That's — okay, that's genuinely unsettling. Because stage one felt like the breakthrough, right? No annotators, the model teaches itself from principles. But stage two means there's still a reward model shaping Claude's behavior, and its quality is completely bounded by how well Anthropic wrote the constitution.
Alex Mercer: And Bai et al. reported CAI-trained models are more harmless than RLHF-trained alternatives — that finding is real. I'm not dismissing it. But 'more harmless' according to whose evaluation, in which deployment contexts? The scalability win from removing human annotators is genuine. The question is whether you've just relocated the governance problem.
Jordan Hale: Relocated it to one document. That one organization controls. With no public audit trail.
Alex Mercer: Right — and that's where stage one being genuinely clean actually makes stage two *harder* to scrutinize, not easier. The annotation-free part is impressive. But it creates this feeling that the reward model problem is solved when it's really just been... repackaged.
Jordan Hale: And that repackaging is — that's what I'm stuck on. Because if stage one is where the alignment really gets baked in, if the self-critique loop is the structural heart of the whole thing, then the constitution isn't an input to the system. It *is* the system. Like, the two things are the same thing.
Alex Mercer: That's the exact right way to put it.
Jordan Hale: And then Harvard researchers are evaluating Inverse Constitutional AI and finding that models overfit to specific phrasings — not the intent of a principle, the *wording* of it. Which means... you could write a completely reasonable principle, use slightly the wrong words, and the model trains itself to follow a phrase rather than an idea?
Alex Mercer: That's essentially what the overfitting finding says, yeah. And misinterpretation is the other documented failure mode — the model reads the principle and arrives at a confident but wrong understanding of it. No alarm goes off.
Jordan Hale: No alarm. That's the terrifying part — because the model isn't confused. It's certain.
Alex Mercer: Think about what 'faithful-but-erroneous alignment' actually means in practice. A nurse practitioner, rural clinic, eleven p.m., asks Claude about a medication dosage edge case — no one else to call. If the constitution's 'be cautious with medical advice' principle is over-specified, or just phrased too broadly, Claude refuses. Not because it evaluated that situation. Because it pattern-matched to a phrase the authors wrote and never tested in that context.
Jordan Hale: Wait — so the refusal isn't a judgment call. It's more like... a false positive from a phrase detector.
Alex Mercer: Basically. And the nurse practitioner has no way to know that. The response looks principled. It probably *sounds* careful and responsible. The model is doing exactly what it was trained to do.
Jordan Hale: Which is the problem. I mean — that's the thing that makes 'faithful but erroneous' such a brutal phrase, right? Because faithful sounds like a virtue. In this context it's, no wait, it's almost the opposite. It means the model is too faithful. To the wrong thing.
Alex Mercer: And the scalability argument cuts both ways here. CAI's advantage is that it applies the same principles uniformly across every training example — unlike human raters who vary across cultures and contexts. Consistency is real. But uniform application of a badly-phrased principle isn't better than inconsistent human judgment. It might be worse, because it's systematic.
Jordan Hale: Oh. Scale makes bad phrasing more dangerous, not less.
Alex Mercer: Consistently wrong, across millions of interactions. There's no inconsistency to average out.
Jordan Hale: And we haven't even gotten to who's writing this document — like who Anthropic actually is in this picture, and whether 'the company wrote a constitution' is really that different from 'the company made the call.' That part gets even more uncomfortable when you pull on it.
Alex Mercer: I think that's where the principled alignment versus preference-based alignment debate stops being abstract. It becomes a question about governance. Which we should get into.
Jordan Hale: But 'the company made the call' is — I mean, that's almost too easy, right? Because Anthropic isn't just making a call, they're writing a document that makes the call look like a principle. And those are structurally different things in a way that I think is actually kind of disturbing.
Alex Mercer: That's the counterintuitive part. We replaced thousands of human annotators — distributed, inconsistent, representing genuinely different perspectives — with a single document authored by one team at one company.
Jordan Hale: Wait — so the diversity actually went down?
Alex Mercer: Structurally, yes. Human annotators disagreed with each other. That's usually framed as the problem. But disagreement across annotators is also — to some extent — a signal. It reflects the fact that people actually hold different values. A constitution smooths that out. It enforces one interpretation.
Jordan Hale: Which is — no, wait, that's the US Constitution problem exactly. Like, the founders thought they were writing universal principles. But 'all men are created equal' did not mean what we'd now call universal. The drafters' blind spots got locked in on day one, and then the system applied them uniformly for decades. That's not a metaphor. That's the mechanism.
Alex Mercer: And with an actual founding document, you at least have public debate, amendment processes, court challenges. Anthropic's constitution has none of that. No external audit. No public version we can inspect. If a principle is wrong, there's no check before it propagates through training.
Jordan Hale: No alarm. Again.
Alex Mercer: And this is where I think CAI is genuinely interesting as a governance question rather than an engineering one. Because principled alignment — what CAI does — encodes what a drafting team believes people should value. Preference-based alignment, what RLHF and DPO do, captures what people actually choose. Those are different theories of human intent. And neither side of that debate has settled it.
Jordan Hale: DPO being — that's Direct Preference Optimization, the other alternative where you're still working from human pairwise comparisons, you're just not building a separate reward model on top.
Alex Mercer: Right. And the philosophical split between DPO-style approaches and CAI is basically: do you trust revealed preference, or do you trust normative principle? And the honest answer is — each one captures something real and loses something real.
Jordan Hale: Preferences include all the inconsistencies and biases people actually have. Principles include all the blind spots of whoever drafted them. You're not escaping human judgment, you're just... choosing which humans and which moment in time.
Alex Mercer: That's basically the load-bearing question. And there's a tradeoff the research flags that I don't think gets enough attention — rule-based models trained on a constitution tend to be less flexible than preference-based ones. More consistent, yes. But that consistency costs you adaptive helpfulness in edge cases the drafters never anticipated.
Jordan Hale: So you get a model that's principled and rigid, versus one that's flexible and maybe... reflecting whoever happened to be doing the ratings that week.
Alex Mercer: Neither of which is obviously better. And I think that's the honest place to leave it — CAI concentrated the governance problem, made it less visible, and called it a technical solution. Whether that trade was worth it depends entirely on how much you trust the authors of the document.
Jordan Hale: And I don't know how you fix that without rebuilding the exact problem CAI was supposed to solve. If you want to democratize constitution-writing, open it up, get more voices in the room — you're back to aggregating human input at scale. You're back to something that looks a lot like a labeling problem.
Alex Mercer: And Bai et al. showed real gains — the CAI-trained models were genuinely more harmless than the RLHF-trained alternatives. That's not nothing. But how far does that generalize? Across which model sizes, which deployment contexts? The paper doesn't fully establish that, and I don't think we know yet.
Jordan Hale: So even the headline result — the thing the whole method is resting on — has an asterisk.
Alex Mercer: A meaningful one. I'm not saying it doesn't work. I'm saying we don't know the shape of where it works.
Jordan Hale: I guess what I can't settle is — who audits what Claude actually understood? Not what the constitution says. What the model, you know, took from it. Because those are different things, and right now there's no mechanism for that. No amendment process, no external review. It's just... Anthropic, and the document, and then Claude.
Alex Mercer: Yeah. That's genuinely open. I don't have an answer to that one.
