Brian Reed: Hey. Good to be here.
Eliza Ward: Yeah, likewise. You read the Futurism piece this morning?
Brian Reed: I did, and I keep rereading the same paragraph.
Brian Reed: The one where Google AI Overviews is just — telling people that Flock Safety cameras contain one to five grams of gold and two to twenty-three pounds of copper. As a fact. At the top of search results. And the source is an anonymous Substack called do.not.obey.do.not.comply.
Eliza Ward: Wait — and the entire camera housing weighs three pounds.
Brian Reed: The whole thing. Three pounds. So the copper alone — let me see, if you took the AI Overview at face value, the copper outweighs the camera it's supposedly inside.
Eliza Ward: That's not a fact-check problem. That's a scale problem. A literal one.
Brian Reed: But hang on — isn't the obvious reframe just 'bad source problem'? Like, Google grabbed a bad source. Fix the source ranking. Done.
Eliza Ward: That's actually the wrong frame, and the three-pound thing is exactly why. A source-quality fix would help with stuff that's at least physically plausible. This isn't. Two to twenty-three pounds of copper inside a three-pound housing — that's not a credibility judgment. That's arithmetic. And nothing in the system flagged it.
Brian Reed: So there's no sanity check layer at all.
Eliza Ward: None. And here's how I'd put it plainly: Google AI Overviews is basically a friend who read everything on the internet and summarized it for you — but has no idea whether something was a joke, so if enough people shared the joke, it sounds like the truth. The Substack account that seeded this is called do.not.obey.do.not.comply. That's not a subtle source. The system absorbed it anyway because it circulated.
Brian Reed: The name alone — I mean, do.not.obey.do.not.comply — that's not even trying to sound credible.
Eliza Ward: Right, and this connects to the Reddit piece. Anti-Flock meme culture was already circulating — Futurism flagged rising vandalism as of July 16th, weeks before August 5th. So those claims had volume. Enough volume that Gemini treated them as signal. That's the meme-to-misinformation pipeline in one sentence: circulation without verification becomes the default search answer.
Brian Reed: And Google updated its spam policies in May 2026 — specifically to cover this kind of manipulation. So the policy existed. It just... didn't catch it.
Eliza Ward: Which means 'we're working on it' was already the answer in May, and August still happened. TechSpot called it a clean example of how a widely used AI tool turns a joke into something that looks like fact. That's not a content moderation gap. That's structural — the architecture treats a meme and a measurement the same way.
Brian Reed: But that framing — 'Google fixed it' — that's the take I keep seeing, and it's wrong in a specific way. The correction came from Futurism on August 5th. TechSpot on August 6th. Not from Gemini. Not from any internal flag. Journalists did the work.
Eliza Ward: Right — and there's no evidence anywhere in the record that Google's own systems detected this before the press did.
Brian Reed: So 'the system worked' — that's the clean version people are running with. But what actually happened is a publication caught it, wrote it up, and then Google issued a correction that said, I mean the language is striking, it says 'viral internet rumors and AI hallucinations claiming a single unit holds two to twenty-three pounds of copper are completely false.' Which is — okay, true, but that's Google describing its own output as a hallucination, after the fact.
Eliza Ward: After journalists named it.
Brian Reed: Which is not a safety mechanism. That's reactive cleanup. And the part that makes this harder — the May 2026 spam policy update. Google specifically updated those policies to cover manipulation of AI-generated responses. BBC had already shown it was possible. The policy existed. August still happened.
Eliza Ward: Wait — and Sundar Pichai already had a version of this conversation internally. After the 2024 Gemini image debacle — generating people of color in Nazi uniforms, the whole thing — he put out a memo saying some responses had offended users and shown bias. That's a documented pattern. Not a one-off.
Brian Reed: So there's a memo, there's a policy update, and then — do.not.obey.do.not.comply gets surfaced as a factual source at the top of search. I keep wondering what exactly those policies were scoped to catch, because this got through.
Eliza Ward: That's actually the missing piece — whether the May update was designed for paid manipulation specifically, or whether it ever covered organic meme drift. Those are different threat models and we don't have the answer.
Brian Reed: And the dollar figure that came out of all this — we haven't even gotten there yet. Up to $650 of recoverable material per unit, delivered by search to anyone who typed the question. It's harder to sit with.
Eliza Ward: And that's exactly the pivot — because $650 isn't abstract. That's a number someone sees on their phone and runs a quick mental calculation on. Picture someone who just watched a Flock camera go up on their block, late at night, has a truck payment due. They search 'are Flock cameras valuable' and the AI Overview serves them that figure before Google has touched a line of that response.
Brian Reed: And the vandalism wave was already live. Futurism had that July 16th report — Reddit memes, people celebrating cut-down Flock towers. The AI didn't create those people. It handed them a receipt.
Eliza Ward: A receipt. That's the right word, actually.
Brian Reed: Because there's a difference between 'I want to knock that camera down' as a cultural sentiment and 'I want to knock that camera down, and my phone just told me it's worth $650 in recoverable gold and copper.' The second one has math behind it. The motivation didn't come from the AI — but the AI made it feel economically rational.
Eliza Ward: And we can't draw a straight line from the Overview to any specific act. That causal link isn't confirmed. But — legitimation at search-engine scale is still a real consequence, even without a proven chain.
Brian Reed: Right — but the part that doesn't fit the 'bad source' cleanup story is the BBC finding. May 2026. A journalist seeded claims — including, I mean, that they were a world-champion hot-dog eater — into Google and ChatGPT. Within twenty minutes those claims were surfaced as fact.
Eliza Ward: Twenty minutes.
Brian Reed: Twenty minutes. So this isn't one rogue Substack. Anyone can do this, and the window between seeding and surfacing is, wait — that's not a policy problem you close with an update. That's the architecture.
Eliza Ward: Which is what the $650 figure actually proves. The thing to watch now isn't whether Google patches this specific claim — they already did. It's whether Gemini, sitting inside products used by billions of people, has any structural limit on how fast false economic claims can move from a meme to a default answer. And right now, nothing in the record says it does.
Brian Reed: And that's the thing I can't resolve. The anonymous Substack — do.not.obey.do.not.comply — we don't actually know if that account intended to manipulate Google's AI or if it was just... venting online. Both possibilities end in the same place: millions of search results, one number, $650. The intent is almost beside the point.
Eliza Ward: That's — yeah. Identical effect either way.
Brian Reed: So the question I actually can't answer — and I don't think the evidence answers it — is whether Google could fix this with better spam policies and faster corrections, or whether any system that synthesizes web content at that scale will just... do this. Inevitably. Because the architecture treats volume as signal, and the internet produces volume for everything, true and false, satirical and sincere. I mean, I want to believe it's a company problem. That's the fixable version. But the BBC journalist seeded a claim in twenty minutes. That's not a policy gap. That's — I don't know what that is, actually.
Eliza Ward: No confirmed answer in anything we have. Google updated spam policies in May. August still happened. That's the only data point we've got on whether the fix works. And it's not encouraging.
