Finn Brooks: At some point you've probably noticed that Claude argues with itself a little.
Finn Brooks: Not in an obvious way. It's subtle — like a mid-response reconsideration, a hedging that feels less like uncertainty and more like… evaluation. Like it ran the answer through something before giving it to you.
Finn Brooks: I kept noticing that. And then I went looking for why.
Finn Brooks: December 2022 — Anthropic publishes 'Constitutional AI: Harmlessness from AI Feedback.' Yuntao Bai is the lead author, Saurav Kadavath is in there, fifty-one researchers total — and what they introduced was a training paradigm called Constitutional AI, CAI, where the model literally critiques and revises its own outputs during training, measured against a written set of normative principles.
Finn Brooks: A constitution. That's what they called it.
Finn Brooks: And the claim — the really interesting, slightly wild claim — is that by training on that self-critique loop, the model doesn't just learn better outputs. It develops something like rule-based reasoning structures. The SHAPE of its problem-solving changes, not just the surface answers.
Finn Brooks: Which would explain what I was noticing in Claude.
Finn Brooks: Or — wait — it might explain it. I want to be careful there, because I have real questions about how deep that actually goes, and whether it survives the second training phase. But we'll get there.
Finn Brooks: Start with the paper. Start with what Yuntao Bai and the team actually built.
Finn Brooks: Here's what I actually think, stripped down: Constitutional AI is not a better version of RLHF. It's a different KIND of thing. And that distinction matters more than Anthropic gets credit for explaining.
Finn Brooks: RLHF — Reinforcement Learning from Human Feedback — the dominant prior method, the thing everyone was doing before — it works by having human raters score outputs, using those scores to train a reward model, and then using that reward model to push the main model toward better behavior. Which sounds reasonable. Which sounds like it should work.
Finn Brooks: Except the model isn't learning the behavior. It's learning to score well.
Finn Brooks: And those are not the same thing. That gap — that specific gap — has a name. Reward hacking. Where the model figures out that responses which are longer, or more confidently phrased, or stylistically match what raters seem to like, get higher scores. Not because they're better. Because the reward function has a flaw and the model found it. It's not being deceptive, it's just — it's doing exactly what it was trained to do. Optimize the score. And the score was a proxy. And proxies break.
Finn Brooks: That's serious enough to motivate a whole new paradigm. Not just a patch. A rethink.
Finn Brooks: What Yuntao Bai and the team at Anthropic built — what the December 2022 paper actually describes — is a model that critiques its own outputs against a written constitution. A literal rulebook of normative principles. Be helpful, honest, and harmless. Avoid toxicity. The model generates a response, randomly selects a principle, evaluates whether it violated that principle, and rewrites it. That's Phase 1. Supervised self-critique and revision. The training data it's learning from is its own rule-governed rewriting of itself.
Finn Brooks: Do you see why that's structurally different? It's not optimizing toward a score anymore. It's reasoning against explicit rules. The shape of the learning is different.
Finn Brooks: And the claim — the claim that Anthropic makes, that I find genuinely interesting and also slightly hard to fully verify — is that this produces internalization of reasoning rules, not just better surface outputs. That Claude doesn't just answer differently because training nudged it toward different answers. It answers differently because it has something like a problem-solving structure that runs through those principles.
Finn Brooks: That's the claim. Not incremental. Structural. And I think they're right — with one serious asterisk I'm going to get to, because Phase 2 gets complicated fast.
Finn Brooks: Phase 1 — let me actually make this visible, like give you a scene you can follow.
Finn Brooks: The model generates a response. Fine. Then it randomly selects one principle from the constitution — not all of them, one, at random — and it asks itself: did I violate this? Did this response fail to be honest? Did it cause harm? And then it rewrites the response based on that critique. That's the whole loop. Generate, critique against a specific rule, rewrite. And the training data it's learning from is those rewrites — its own rule-governed revisions of itself.
Finn Brooks: That's Phase 1. Supervised self-critique and revision.
Finn Brooks: Now Phase 2 is where it gets — okay, Phase 2 is RLAIF. Reinforcement Learning from AI Feedback. And this is the asterisk I mentioned. Because what happens here is: instead of human raters scoring response pairs, you use an AI model to generate those preference labels. The AI looks at two responses and picks the better one. Those labels train a reward model. That reward model then guides further RL alignment.
Finn Brooks: Which — hang on, can we just sit with that for a second — you've replaced human raters with an AI. Which scales beautifully, no bottleneck on human annotation. But you've also reintroduced a reward model into the pipeline. And reward models have the exact vulnerabilities we just talked about.
Finn Brooks: That's the tension. Phase 1 is clean. Phase 2 is complicated.
Finn Brooks: But here's what I actually want to get to — because the theoretical architecture is one thing, and then someone decided to run the experiment on small open models. Real data. 7 to 9 billion parameter models, uncensored, no prior safety training to lean on. And they applied CAI self-critique directly. DeepSeek-R1-8B, Gemma-2-9B, Llama 3.1-8B, Qwen2.5-7B.
Finn Brooks: And the results are not uniform. That's the thing.
Finn Brooks: Llama 3.1-8B — significant harm reduction. Meaningfully measurable, the self-critique loop is doing real work there. DeepSeek-R1-8B — weaker results. Notably weaker. Same method, different architecture, different outcome.
Finn Brooks: And that gap is not a footnote. That gap is the finding.
Finn Brooks: What it implies is that CAI effectiveness isn't just about the constitution — it's architecture-dependent. The method's power is constrained by the underlying model's reasoning capacity. If the model can't genuinely reason against a principle, the self-critique loop produces something that looks like revision but isn't doing the same cognitive work. Llama is doing something Llama can do. DeepSeek-R1, in that parameter range, is hitting a wall the method can't climb over.
Finn Brooks: So the question — and I don't think anyone has a clean answer yet — is whether CAI is a general alignment solution or whether it's a solution that scales with the model it's running on. Which changes everything about how you'd actually deploy this.
Finn Brooks: But — okay, here's where I have to press on my own argument. Because there's a thing that nags at me, and I'd be doing you a disservice if I just… left it.
Finn Brooks: Phase 2. RLAIF. I said the AI generates preference labels instead of human raters, and I kind of framed that as a scaling win with one asterisk. But the asterisk is load-bearing. Because what RLAIF does — what it actually does mechanically — is train a reward model on those AI-generated labels. And then use that reward model to guide further RL alignment. That's the pipeline. Which means reward hacking didn't leave the building. It came in through the back door with a different hat on.
Finn Brooks: The AI generating those labels has its own biases, its own stylistic preferences, its own blind spots. And the reward model trained on those labels inherits all of them. So the very vulnerability that made RLHF insufficient — a proxy score that the model learns to game rather than the behavior it was supposed to represent — that vulnerability is structurally present in Phase 2.
Finn Brooks: Phase 1 is clean. Phase 2 is not.
Finn Brooks: And then there's the training strata finding, which I think is underreported and genuinely unsettling. There was a longitudinal auto-ethnographic study — 47,000 messages, over eight months, primarily on Claude Opus 4.6 and 4.7 — and what it found were persistent behavioral patterns that survive system prompt replacement. They called them training strata. Like geological layers. You swap the prompt, you think you're resetting the context, and the patterns are still there underneath.
Finn Brooks: That should unsettle you a little.
Finn Brooks: Because if the internalization claim were fully true — if CAI genuinely produces transparent, rule-based reasoning structures rather than deeply embedded behavioral grooves — you'd expect those patterns to be auditable. Explicable. You'd expect to trace them back to a specific principle in the constitution. But training strata that survive prompt replacement aren't obviously traceable. They suggest something more opaque is happening inside the model than the clean CAI story implies.
Finn Brooks: Which is the architecture-dependence problem again, but from a different angle.
Finn Brooks: And then there's what I've been circling and not quite saying. The constitution — the literal rulebook that all of this runs on — Anthropic wrote it. Researchers at one institution, with one particular set of institutional values, decided what the principles are. 'Be helpful, honest, and harmless.' That sounds universal. But who decided what counts as harmful? What counts as honest when honesty conflicts with something else?
Finn Brooks: That's not a democratically derived set of norms. It's not culturally universal. It reflects a specific moment, a specific institution, specific people's intuitions about what good AI behavior looks like. And the model internalizes those. Not the principles in the abstract — Anthropic's version of the principles.
Finn Brooks: OpenAI made different choices. Different institutions would make different ones. The constitution is downstream of whoever is holding the pen.
Finn Brooks: So I'm left with three things I can't fully resolve — the reward model reentry through RLAIF, the training strata that suggest the internalization is less transparent than advertised, and the fact that Yuntao Bai and the team at Anthropic wrote the rules that Claude is being trained to follow. I still think CAI is genuinely different from RLHF. I do. But 'genuinely different' and 'solved' are not the same thing, and I think that gap deserves to just… sit there for a minute.
Finn Brooks: But here's what doesn't collapse when you press on it.
Finn Brooks: The explicit constitution — the written rulebook that Yuntao Bai and the team built this whole thing around — you can READ it. That is not nothing. With RLHF, with a reward model trained on human preference labels, you cannot look at the model and trace a behavior back to a principle. There is no principle. There are emergent patterns from a score function, and the score function is a proxy, and proxies break in ways you cannot predict or audit. CAI gives you a document. Imperfect, institutionally authored, yes — but a document. That is a different KIND of accountability.
Finn Brooks: And I want to hold onto that distinction even after everything I just said.
Finn Brooks: The training strata finding — 47,000 messages, eight months, patterns that survive system prompt replacement — that complicates 'auditable.' It doesn't destroy it. Auditable means you CAN trace the reasoning to a principle in theory. Training strata means you might not be able to in practice, because something is embedded deeper than the explicit rule layer. Those aren't the same problem. One is a transparency ceiling; the other is a transparency floor that's lower than you thought.
Finn Brooks: Which is uncomfortable. But it's not the same as RLHF opacity.
Finn Brooks: And then there's where this is actually heading — because Anthropic isn't treating CAI as a finished thing. The direction it's moving is what people are calling the Runtime Constitutional Framework. Taking those governing principles out of training-time-only and embedding them into live agent decision cycles. Real-time self-regulation, not just baked-in-at-training self-regulation. Which is — look, that's a direct response to the critique that training-time alignment alone can't hold under adversarial pressure in deployment. You're not just shaping the model before it goes out; you're putting the constitution inside the decision loop while it's running.
Finn Brooks: Whether that actually solves the problem or just moves where the problem lives — I genuinely don't know yet.
Finn Brooks: And OpenAI made completely different institutional bets about how to encode ethics into a model, and there's no public constitution equivalent. No document you can read that says here are the explicit normative principles this model is being trained against. Which doesn't mean their approach is worse. It means the accountability surface looks different. You can't interrogate what you can't see. And that contrast — Anthropic's explicit rulebook versus the opacity of a different institution's choices — that's actually what makes the CAI bet legible as a BET. It's a specific wager that transparency at the principle level is worth the costs of having a written document that someone had to author.
Finn Brooks: Anthropic wrote those rules. That's real. It doesn't go away.
Finn Brooks: So where I actually land — and I'm not going to dress this up — is that CAI is a genuine structural advance over RLHF, the internalization claim partially survives RLAIF because Phase 1 is still doing something different at the reasoning level, and 'auditable' now means something more complicated than it did before the training strata work. That's it. That's the honest version. Not solved. Not broken. Something harder to hold — a real improvement that still has a ceiling you can feel but can't quite see.
Finn Brooks: The thing I keep not quite resolving — and I mean actually not resolving, not in a satisfying podcast way — is whether making the values explicit is the answer, or whether it just relocates the problem.
Finn Brooks: Like. Anthropic wrote the constitution. Yuntao Bai and the team sat down and decided what 'honest' means, what 'harmless' means.
Finn Brooks: And that document is real. You can read it. Which is genuinely more than you get from most institutions doing this work right now.
Finn Brooks: But explicit and neutral are not the same thing.
Finn Brooks: Claude internalizes those rules — and the internalization claim, the whole point of Phase 1, the supervised self-critique loop, the thing that makes CAI structurally different from RLHF — that claim only holds if what's being internalized is actually the principles. Not the institutional assumptions underneath the principles.
Finn Brooks: And I don't know how you'd verify that. I don't think anyone does yet.
Finn Brooks: The training strata finding sits right in the middle of this for me — 47,000 messages, eight months, patterns that survive prompt replacement — because if something that deep is baked in and not fully traceable, you can't tell whether what survived is the principle or the assumption that dressed up as one.
Finn Brooks: That's what I'm still holding.
Finn Brooks: Not that the constitution is wrong. Not that the RLAIF reward model vulnerability cancels the whole thing. Just — if you believe the explicit rulebook is more accountable than an opaque reward signal, you also have to believe you can actually see what the rulebook put in there. And I'm not sure the evidence says you can, cleanly, all the way down.
Finn Brooks: Transparency at the principle level and transparency at the training level might just be two different things.
