Finn Brooks: There's this image I can't stop thinking about.
Finn Brooks: A language model — some version of the thing you've probably talked to, probably argued with, probably asked to write your emails — sitting there being asked to read its own output and decide whether it said something harmful.
Finn Brooks: Not a human reading it. The model. Reading itself.
Finn Brooks: And I kept sitting with that, going… wait — is that genius, or is that just a very sophisticated way of hoping for the best?
Finn Brooks: Because the reason we even got to that weird image in the first place is that the old answer, the dominant answer, the thing everyone was doing, started hitting a wall.
Finn Brooks: That answer was Reinforcement Learning from Human Feedback. RLHF.
Finn Brooks: The idea is actually pretty elegant — you take a model, you have human raters compare two outputs, pick the better one, and you train a reward model on those preferences. The model learns what humans like. Iteratively. Over thousands of comparisons.
Finn Brooks: And for a while, that worked.
Finn Brooks: But then the models got better. Faster than the humans rating them could keep up with. And this is where the scalability problem bites — like, genuinely bites — because if a model is producing novel reasoning, specialized outputs, things that require real expertise to evaluate, you suddenly need an army of expensive, slow, and — honestly — inconsistent human annotators just to keep the signal clean.
Finn Brooks: Inconsistent is the word I want to sit on for a second.
Finn Brooks: Because it's not just slow or expensive — though it's both of those things — it's that humans don't always agree. On what's harmful. On what's helpful. On what counts as a good response versus a technically accurate but kind of awful one.
Finn Brooks: That disagreement gets baked into the reward model.
Finn Brooks: And you're building something that's supposed to be aligned with human values, but the signal you're training it on is… noisy. And getting noisier as the capability gap widens.
Finn Brooks: So there's a group — Anthropic — that looked at that wall and said, okay, what if the feedback didn't have to come entirely from humans?
Finn Brooks: And in 2022 they published something called 'Constitutional AI: Harmlessness from AI Feedback.' A paper that — I don't want to oversell it before I've even explained it — but genuinely changed how people think about this problem.
Finn Brooks: We're going to get into all of it. But I want the tension to sit for a second — the human feedback bottleneck is real, it's structural, and the solution Anthropic proposed is strange enough that it deserves a slow look.
Finn Brooks: Here's what I actually think — and I want to say this plainly before I start hedging, because I will start hedging — Constitutional AI solves a real problem.
Finn Brooks: Not a temporary problem. Not a 'we just need more annotators' problem. The thing that Yuntao Bai and the team at Anthropic were responding to in 2022 was structural — the velocity of human feedback cannot keep pace with the velocity of model capability.
Finn Brooks: That gap doesn't close.
Finn Brooks: And what the paper — 'Constitutional AI: Harmlessness from AI Feedback' — what it actually proposes is, instead of pouring more humans into that gap, you write down your principles. Explicitly. In natural language. You give the model a constitution — a literal curated set of principles defining how it should behave — and then you train it to critique its own outputs against that constitution and revise them.
Finn Brooks: The model reads itself. That's the image from the top.
Finn Brooks: And Anthropic reported that this approach was substantially less resource-intensive than collecting tens of thousands of human harmlessness labels. I mean — that's not a rounding error. That's a different order of magnitude of effort.
Finn Brooks: The researchers on this — Bai, but also Amanda Askell, Jared Kaplan, the broader team — they framed the goal around what Anthropic calls HHH. Helpful, Harmless, and Honest. That's the behavioral target. And the specific thing CAI was trying to crack was how you get harmlessness at scale without just trading it for helplessness — without the model becoming so over-cautious it stops being useful.
Finn Brooks: Which is a real failure mode. Worth naming.
Finn Brooks: But I think CAI swaps one bottleneck for a quieter one. And I think the quieter one is actually more dangerous, because it's less visible.
Finn Brooks: With RLHF, the limitation is obvious. You can see it. You can count the annotators, measure the disagreement, notice the lag. The seams show.
Finn Brooks: With CAI, the bottleneck moves upstream — into the constitution itself. Someone has to write those principles. Someone decides what goes in, what gets left out, how they're framed. And once that document exists, the model treats it as ground truth and iterates against it at machine speed.
Finn Brooks: Think about that for a second.
Finn Brooks: The values of whoever wrote the constitution get encoded — quietly, cleanly, at scale — into a system that will then go apply them to millions of outputs, treating them not as one team's considered judgment but as objective correctness. That's not a solved problem. That's the same problem wearing a neater coat.
Finn Brooks: And I don't think that means CAI is wrong. I think it means CAI is real progress with a hidden cost attached — and we're going to have to look directly at that cost, because a system as capable as Claude, trained this way, is not a prototype anymore.
Finn Brooks: Let me actually walk through the mechanism — because I keep saying 'the model reads itself' like that's self-explanatory, and it's not.
Finn Brooks: There are two phases. Distinct. And they matter differently.
Finn Brooks: Phase one is what the paper calls supervised learning — the self-critique and revision loop. The model gets a prompt, generates a response, then gets handed a constitutional principle and asked: does this response violate that principle? Critique it. And then — revise it.
Finn Brooks: No human in that loop. Not at each step.
Finn Brooks: And you can iterate that — generate, critique, revise, critique again — until the output clears the bar the constitution sets. Which is wild to even say out loud, because what you're describing is a model using language to evaluate language, without a human having to sign off on every single exchange.
Finn Brooks: That's phase one. Phase two is where RLAIF comes in — Reinforcement Learning from AI Feedback.
Finn Brooks: Instead of humans comparing two outputs and picking the better one — which is how RLHF works, right, that's the whole engine — you have the model do the comparing. Apply the constitutional principles, generate a preference judgment, and use THAT as the training signal. AI-generated labels, at machine speed, at machine scale.
Finn Brooks: And look — Anthropic reported that this whole setup was substantially less resource-intensive than collecting tens of thousands of human harmlessness labels. That's from the 2022 paper, Yuntao Bai and the team. Tens of thousands. That's not a marginal improvement. That's a different category of effort.
Finn Brooks: Think about what that actually means for Claude — the model they were training — at scale.
Finn Brooks: Now here's where I want to slow down, because the mechanism is elegant but the inputs are — they're everything. The constitution itself. What's IN it. How it's written.
Finn Brooks: There's this research framework called C3AI — Kyrychenko and colleagues, 2025 — that looked specifically at how constitutional principles should be constructed to actually work.
Finn Brooks: And the finding is — okay, this is the part that got me — it's not just what the principle says. It's HOW it's framed.
Finn Brooks: Positively framed, behavior-based principles — 'do this' rather than 'don't do that', focused on concrete behavior rather than abstract traits — those align more closely with human preferences than negatively framed ones or trait-based ones.
Finn Brooks: Which means the person writing the constitution isn't just picking values — they're picking syntax. They're making editorial choices that have measurable downstream effects on what the model actually does.
Finn Brooks: That's the quiet bottleneck I mentioned. Right there.
Finn Brooks: And then — dude — there's the architecture question, which I did not expect to care about this much until I read the Chacón Menke and Phan study from 2025.
Finn Brooks: They took four small uncensored models — 7 to 9 billion parameters — DeepSeek-R1-8B, Gemma-2-9B, Llama 3.1-8B, Qwen2.5-7B — and applied the CAI self-critique mechanism to all of them. Same basic approach. Variable results.
Finn Brooks: Llama-based models — strong improvement. The self-critique loop did what it was supposed to do, harm reduction was real.
Finn Brooks: The others? Weaker. Inconsistent. And not in a 'we need to tune this more' way — in a 'the architecture itself seems to respond differently to the mechanism' way.
Finn Brooks: So the method that Anthropic built — that Bai and Askell and Kaplan put together to solve the scalability problem — it's not a universal fix. It's architecture-dependent. Principle-framing-dependent. The variables go deep.
Finn Brooks: Does that mean it's not progress? No — it absolutely is. The resource reduction alone is a real thing, the self-critique loop is genuinely novel, RLAIF as a concept changes what's even possible.
Finn Brooks: But if you're applying this to something as widely used as Claude, and the outcomes shift depending on what architecture you're running and how whoever drafted the constitution chose to word a sentence — that's not a footnote. That's the whole question.
Finn Brooks: But here's what nags at me about my own argument.
Finn Brooks: And I want to say it plainly because I think I've been — look, I've been making the case that CAI is real progress, that Anthropic and Bai and the team solved something structural, and I believe that. I do. But there's a version of this that I can't argue my way around, and I'd rather just say it than slide past it.
Finn Brooks: The self-critique loop only works if the model can correctly interpret the constitution in the first place.
Finn Brooks: That's it. That's the load-bearing assumption the whole thing sits on.
Finn Brooks: And if the model's baseline reasoning is already bent — already carrying some systematic bias, some blind spot, some failure mode baked in from pretraining — then what happens when you ask it to evaluate its own outputs against principles it's interpreting through that same bent reasoning?
Finn Brooks: It doesn't catch the error. It confirms it.
Finn Brooks: The self-critique loop — the thing that makes CAI elegant, the thing that makes it scalable — it COMPOUNDS the problem if the reasoning underneath is off. You iterate toward a more polished version of the same mistake. At machine speed.
Finn Brooks: And the Chacón Menke and Phan findings from 2025 — the ones where they ran CAI self-critique across those four small models, the 7 to 9 billion parameter ones — Llama 3.1-8B, DeepSeek-R1-8B, Gemma-2-9B, Qwen2.5-7B — that's actually evidence for this concern, not against it.
Finn Brooks: Because Llama-based models showed strong harm reduction. The others? Weaker. Inconsistent.
Finn Brooks: And the interpretation there isn't just 'different architectures need tuning.' It's that the same self-critique mechanism — the same constitutional principles applied the same way — produces different outcomes depending on something structural to the model itself. Which means the method isn't reading the principles cleanly and neutrally. It's reading them through whatever that architecture is.
Finn Brooks: The method is not universal.
Finn Brooks: And if you're sitting there thinking — okay, but Claude is Claude, it's not a 7B parameter uncensored test model — yeah, fair. But the point isn't Claude specifically. The point is that the mechanism depends on what's underneath it in ways that the technical literature on CAI has not fully addressed.
Finn Brooks: Then there's the harder one.
Finn Brooks: The C3AI framework work — Kyrychenko and colleagues — showed that how you frame a principle changes what the model does. Positive framing, behavior-based rather than trait-based, those things have measurable effects on alignment outcomes. Which means whoever wrote the constitution isn't just expressing values. They're engineering behavior through word choice.
Finn Brooks: And then the system trains on that. Claude trains on that. At scale, treating those editorially constructed principles as if they're just… correct.
Finn Brooks: That's not the elimination of human power over the system. That's the concentration of it — into whoever drafted the document, before the training even starts.
Finn Brooks: Whose values are those? The technical literature — the 2022 paper, Bai, Askell, Kaplan — it does not answer that question. It mostly doesn't ask it.
Finn Brooks: And then — dude — the 'substantially less resource-intensive' claim. I've been repeating that because it's in the paper and it's real. But it needs scrutiny, because the fine-tuning phases, the formal reinforcement learning, training a preference model on AI-generated labels — that still requires serious compute and careful engineering. It's not free. The human annotation bottleneck shrinks; the infrastructure cost doesn't disappear.
Finn Brooks: That's why efforts to formalize CAI mathematically — to model the constitutional rule set as something rigorous, constraint-based, not just natural language sitting there hoping to be interpreted correctly — those efforts exist precisely because natural-language principles carry no formal alignment guarantees. None.
Finn Brooks: So the thing I can't dismiss is this: CAI moves the bottleneck, it doesn't remove it. And the new bottleneck is quieter, harder to see, and concentrated in fewer hands. I don't know how to resolve that. Not yet.
Finn Brooks: And yet the objection doesn't defeat the core thing. I think it clarifies it.
Finn Brooks: The scaling argument is real. Full stop.
Finn Brooks: RLHF's velocity problem — the part where human feedback just cannot keep pace with model capability, where you need increasingly expert annotators to evaluate increasingly sophisticated outputs — that is structural. That gap does not close by hiring more people. The RLHF-Blender research, the work on just how complicated and layered human preference data actually is, underscores that. The complexity of aggregating diverse human feedback isn't a solvable ops problem. It's a fundamental mismatch between the speed of human judgment and the speed of model development.
Finn Brooks: And what Anthropic did — what Bai and Askell and Kaplan actually published in 2022 — was answer THAT problem. Substantially less resource-intensive than collecting tens of thousands of human harmlessness labels. That's not a marginal win. That's a category shift.
Finn Brooks: The self-critique loop, RLAIF replacing human preference comparisons — that's a genuine structural advance. I'm not walking that back.
Finn Brooks: What the objection DOES defeat — and I want to be honest about this — is the cleaner story. The story where Constitutional AI closes the loop. Where you write the constitution, the model critiques itself, and you've removed the problem of human authority baked into the system.
Finn Brooks: That story doesn't survive.
Finn Brooks: Because what CAI actually does is relocate the authority. It doesn't eliminate it. Whoever drafts the constitution — whoever decides the framing, the syntax, whether a principle says 'do this' versus 'don't do that' — that person is engineering Claude's behavior through word choice, at a level the C3AI framework work from Kyrychenko and colleagues makes really concrete. And the technical literature, the 2022 paper itself, mostly doesn't ask whose values those are. It assumes the constitution is a solved input and focuses on the mechanism downstream.
Finn Brooks: That's not a footnote. That's the question.
Finn Brooks: And then — look, if you've been tracking this with me — the architecture-dependence finding from Chacón Menke and Phan sits right next to that. Same constitution, same self-critique mechanism, variable results across Llama 3.1-8B versus DeepSeek-R1-8B versus the others. That tells you the method isn't reading constitutional principles neutrally. It's reading them through whatever that underlying architecture is. Which means the effectiveness of the whole approach is conditional in two directions simultaneously — on how the principles are written AND on what model is doing the reading.
Finn Brooks: Conditional doesn't mean broken.
Finn Brooks: But it does mean that CAI is not a universal fix that you can drop into any system and trust. It's a promising, genuinely novel methodology — the self-critique and revision loop changed what was even possible — and it works variably, depending on inputs that the method itself cannot control.
Finn Brooks: So where I actually land — and I think this is honest — is here: Anthropic solved a real problem. The human feedback scalability problem was going to bottleneck alignment work regardless. CAI addresses that structurally, not temporarily. The HHH target, the Helpful Harmless Honest frame, getting harmlessness at scale without collapsing into over-caution — that's what the 2022 paper actually delivered on, at least as a methodology.
Finn Brooks: And it carries a real new problem forward. The bottleneck moved upstream, into whoever controls the constitution, and that concentration of authority is quieter and harder to see than a room full of annotators disagreeing with each other. At least you can see that room.
Finn Brooks: I don't know how you resolve that from inside the method. I genuinely don't. The formal mathematical approaches — trying to model the constitutional rule set as something rigorous rather than natural language hoping to be interpreted correctly — those exist precisely because natural-language principles carry no formal alignment guarantees. Someone's working on it. But it's not resolved.
Finn Brooks: Real advance. Real open problem. That's where this sits — and I think that's the only honest place to stand.
Finn Brooks: But I don't have a clean answer for this part, and I want to be upfront about that.
Finn Brooks: If Constitutional AI actually works — if it scales the way the 2022 paper suggested, if the self-critique loop holds, if RLAIF keeps delivering — then the most consequential editorial decisions in the history of AI alignment were made by whoever sat down and wrote the first constitution. Yuntao Bai. Amanda Askell. Jared Kaplan. Real people. Specific people. People who made real choices about what principles go in, what gets left out, how a sentence gets framed. And the C3AI work from Kyrychenko and colleagues makes concrete how much that framing matters — positive versus negative, behavior-based versus trait-based, measurably different outcomes downstream. So those weren't neutral choices. They couldn't have been.
Finn Brooks: And then the system trains on that. Claude trains on that. At scale. Applying those choices to millions of outputs, infinitely, without a democratic input mechanism, without a transparent audit trail that a person outside Anthropic can actually examine.
Finn Brooks: That's the part that sits heavy.
Finn Brooks: With RLHF the room full of disagreeing annotators was at least — visible. You could see the seams. You could study the noise. With a constitution, the authority is upstream, embedded before the training even starts, and by the time anyone's interacting with Claude it's just… Claude. Not 'Claude whose harmlessness behavior was shaped by specific editorial choices made by a small team in 2022.' Just Claude. Helpful. Harmless. Honest.
Finn Brooks: I don't know how you audit that later. I genuinely don't know how you go back in and say — wait, whose values are these exactly, and did we agree to that.
