Hugo Vance: Lila, long week — I've been turning something over and I want your read on it before we record.
Lila Soto: Oh yeah, same energy — what is it?
Hugo Vance: In November 2023, Anthropic ships Claude 2.1. 200,000-token context window. And the pitch — the actual framing — is that this eliminates the need for chunking and retrieval pipelines entirely. RAG, meaning Retrieval-Augmented Generation, which has been the dominant architectural pattern for getting models to work with large document sets — RAG is, in this telling, the workaround. And now the constraint that made it necessary is gone.
Lila Soto: That's a pretty clean story.
Hugo Vance: Yes. Maybe too clean. Because a token is roughly three-quarters of a word — so 200,000 tokens is about 150,000 words, 500-plus pages. That's real. I'm not disputing the number. What I keep returning to is whether the model actually *uses* all of that equally. And the research — I'd flag the sourcing here is thinner than I'd prefer — hints strongly that it does not.
Lila Soto: Huh — so you can fit the whole book in the window, but the model kind of... loses the chapters in the middle?
Hugo Vance: That is the shape of it, yes. Which means the question underneath all of this is: if long context windows are large enough to replace RAG, why do models still seem to miss things that are sitting right there inside the context?
Lila Soto: And that's — yeah, that's the thing I want to pull on. Because that gap between what's promised and what's actually recoverable inside the window feels like it changes everything about how you'd actually build on top of this.
Hugo Vance: And that gap is exactly where the benchmark data gets uncomfortable. Let me try to make it concrete — imagine you hire a researcher, hand them a stack of 500 pages, and say: find what matters. A large context window means they can read the whole stack themselves, rather than you pre-selecting the pages for them. That's the genuine promise. The complication is what happens around page 200.
Lila Soto: They start skimming.
Hugo Vance: They start skimming. And there's a name for it now — context rot. Accuracy and recall degrade as the token count grows. Not uniformly. The middle suffers most.
Lila Soto: Mm, I want to make sure I understand what that actually means in scale terms, because this is the part that genuinely surprised me. The RULER benchmark tested 17 long-context models. And the needle-in-a-haystack tests — hide a sentence, ask the model to find it — almost all of them passed that. Fine. But once you make the task more complex, something breaks. Only half of those 17 models could maintain satisfactory performance at 32,000 tokens. Which is — I mean, these are models claiming windows of 100,000 tokens, 200,000 tokens.
Hugo Vance: Half. At 32K. That's the number that stands out.
Lila Soto: And the needle-in-a-haystack test — I kind of think that's where the marketing lives, honestly. It's the simplest possible version of the task. Can you find one specific thing I told you is in there? But that's not how anyone actually uses a model on a codebase or a legal document. You're asking it to synthesize, to cross-reference.
Hugo Vance: Yes, and I'd flag — the sourcing on the very largest window claims, the million-token figures, is thinner than what we have on Claude 2.1's 200,000. That November 2023 number is documented. The claims above it, I'd hold more loosely.
Lila Soto: Right — but even accepting the 200K number at face value, the RULER data suggests the usable window is a different question than the claimed window. Those aren't the same thing.
Hugo Vance: Which leaves us with something genuinely strange. The window is large — that's real. But something is happening inside it. The space exists; the attention doesn't distribute evenly across the space. And that distinction is where the engineering problem actually lives now.
Lila Soto: And that unevenness — that's not random, right? Liu et al. actually mapped the shape of it. The degradation isn't just 'more text, worse performance.' It's positional. Start of the context, end of the context — those get real attention. The middle is where things go quiet.
Hugo Vance: Lost in the middle. That's the actual name for it. And it matters structurally because — well, think about the codebase-in-context workflow specifically. A security engineer loads a 90,000-line fintech codebase into Claude on a Friday afternoon for a dependency audit. The files at the top of the context, the files at the bottom — those get read. The middleware sitting around token 60,000? That's the dead zone.
Lila Soto: The middleware. Which is — I mean, in a fintech system, that's often where the actual risk lives.
Hugo Vance: Yes. And no one told the model where the dangerous part is. You've handed it everything and trusted it to find the thing.
Lila Soto: Okay, so — mm, this is the part that's been sitting with me, and I want to just put the numbers on it. A production codebase runs 80,000 to 150,000 lines. One line of code is roughly five to ten tokens. So a one-million-token window — that sounds enormous — but you're actually holding maybe 100,000 to 200,000 lines before you're hitting context rot. That's basically the size of the problem you were trying to solve.
Hugo Vance: You've just described a window that fits the codebase and fails it simultaneously.
Lila Soto: And the architectural gain is real — I don't want to dismiss that. Loading everything into context instead of chunking it, that's not nothing. But bounded. It's real and bounded in a way the marketing doesn't quite say.
Hugo Vance: You see, what unsettles me is that with RAG, when retrieval failed, you could instrument it — you could see which chunks weren't being surfaced. Here, the problem has moved into the attention mechanism, and we have considerably fewer tools to observe it from the outside.
Lila Soto: Right — but the part that actually complicates this further is when chunking itself was destroying something the query needed in the first place, and that's where I think the 'RAG is dead' framing starts to crack in a different direction than we've gone.
Hugo Vance: And that crack runs in an interesting direction — because the October 2026 EBG study actually isolated two specific failure modes. Not context rot. RAG itself failing. Similarity retrieval missing all the required chunks for multi-fact queries. And chunking destroying document metadata entirely — the information that tells you *which* document you're even in.
Lila Soto: Wait — chunking destroys the metadata? Like, the document loses its own identity in the process?
Hugo Vance: Yes. You split a contract into forty chunks, each chunk loses the header. Now the model doesn't know which clause belongs to which party. The retrieval surfaces the right text and the wrong context.
Lila Soto: Okay, and that's — mm, that's actually where multi-hop reasoning breaks down, isn't it? Because you're asking the model to synthesize across five files simultaneously, and chunked retrieval gives you five individually relevant chunks, but it misses the *combination*. The relationship between them.
Hugo Vance: That's the canonical case. And it's where long context actually wins — not universally, but specifically there. When the bottleneck was retrieval quality, not the model's ability to read.
Lila Soto: Which means — I guess the routing question becomes: what shape is your query? Because Qdrant's position is basically that long context and RAG aren't competing, they're... complementary. And the cost math is part of why.
Hugo Vance: Well, yes — full-context scan incurs higher latency and cost per query. But a reusable dense retrieval index, the cost advantage accumulates as queries multiply over the same corpus. The tenth query against a RAG index is cheaper than the tenth full scan. That's not nothing at scale.
Lila Soto: So 'RAG is dead' is just — empirically wrong. It's a routing question. Which bottleneck does your specific query hit. And I'd hedge that — the EBG study is October 2026, the evidence is still accumulating.
Hugo Vance: I'd hold that conclusion loosely, yes. What we can say is: there are queries where chunking destroys the thing you needed. Long context exists precisely for those. The rest? The index still earns its keep.
Lila Soto: And I think what's sitting with me — at the end of all of this — is that we started with Anthropic shipping Claude 2.1 and the framing being 'the workaround is over.' And now I'm not sure the workaround ever fully ends. It just moves.
Hugo Vance: Yes. RAG emerged because context windows were small. That constraint is dissolving — genuinely dissolving. But the habits, the infrastructure, the teams who built retrieval as a core competency — those don't dissolve at the same rate. And now the deeper problem is that we've moved chunking from an explicit engineering step into implicit model behavior, inside the attention mechanism, and we have fewer observability tools there than we ever had in the retrieval layer. The constraint changed. The uncertainty didn't.
Lila Soto: The window is large enough. We just don't know yet whether the model can attend to all of it.
