Onpode
Cover art for Major websites are closing doors to AI agents—despite OpenAI, Meta, and Google's agentic ambitions

Major websites are closing doors to AI agents—despite OpenAI, Meta, and Google's agentic ambitions

October 18, 2026 · 6 min

Iris Holm & Lila Soto

Amazon blocked Meta's Muse shopping agent on September 20, 2026, and a federal court provisionally banned Perplexity's Comet browser agent from Amazon while a lawsuit proceeds. Major platforms including eBay, Delta, and LinkedIn are blocking or ignoring third-party AI agents — not purely for security, but to protect transaction revenue and customer data ownership.

AI agents—software systems capable of interpreting goals, browsing websites, filling forms, and completing multi-step tasks autonomously—have become a major competitive focus for leading technology companies including OpenAI, Meta, Google, and xAI.

0:006:05
Get the next episode on Artificial Intelligence →

Follow it free — new episodes land in your feed.

Or make your own — any topic, in minutes

More Onpode episodes on Artificial Intelligence →

About this episode

AI shopping agents were supposed to browse, compare, and buy on your behalf. The reality in late 2026 is messier: Amazon blocks third-party agents while running Rufus, Alexa for Shopping, and Buy for Me internally. eBay named the threat explicitly in its updated user agreement. Yelp turned access into a paid licensing product. Delta Air Lines hasn't signed a single agent deal. The episode works through why this pattern keeps repeating — and why calling it a 'security' decision often obscures what's really being protected: transaction revenue, customer data, and control over the buying relationship. There's a deeper infrastructure problem underneath the business one. Cloudflare flipped AI crawler blocking on by default for all new sites in July 2025. The tools to say no are mature and widely deployed. The tools to say yes — to credentialize a trusted agent and let it through cleanly — are still early-stage pilots at best. Web Bot Auth, announced by Cloudflare, Visa, and Mastercard in October 2025, hasn't seen broad adoption. So even a platform that genuinely wants to open its doors to agents has no reliable way to do it yet. The episode compares this moment to the early days of robots.txt — the informal standard that eventually produced rough norms for web crawling. But the stakes here are orders of magnitude higher. This is transaction revenue and customer ownership, not just indexing. That commercial pressure may be exactly why the patchwork sticks.

Frequently asked

Why is Amazon blocking AI shopping agents like Meta Muse and Perplexity Comet?

Amazon blocked Meta Muse on September 20, 2026, and sued Perplexity over its Comet browser agent, winning a provisional court order banning Comet from Amazon. Analysts note Amazon simultaneously runs its own first-party agents — Rufus, Alexa for Shopping, and Buy for Me — suggesting the motive is protecting transaction revenue, not just security.

Did a court rule that platforms can block AI agents acting on behalf of users?

Yes. In the Amazon vs. Perplexity lawsuit over the Comet browser agent, a federal court provisionally ordered Comet off Amazon while the case proceeds. This is considered the first major legal test of whether a platform can enforce its terms of service against an AI agent acting on a user's behalf — and the court said yes, at least provisionally.

Which major websites block AI agents in 2026?

As of 2026, Amazon, eBay, LinkedIn, Facebook, JPMorgan Chase, Charles Schwab, and Delta Air Lines all block or exclude third-party AI agents. eBay explicitly banned 'buy-for-me agents' by name in a February 2026 user-agreement update. Delta had zero signed agent-access deals as of October 2026.

Is there a technical standard that lets AI agents prove their identity to websites?

Web Bot Auth, announced by Cloudflare, Visa, and Mastercard in October 2025, is designed to let AI agents authenticate themselves to platforms. As of late 2026 it remains a pilot with no confirmed broad adoption. Meanwhile, Cloudflare made AI crawler blocking the default for all new sites in July 2025 and began selling per-page crawler access as a paid tier.

Do any major platforms allow third-party AI shopping agents?

Shopify has opened its storefronts to third-party AI agents, showing the permissive approach is technically feasible. Yelp takes a different route, offering paid data-licensing deals that grant agent access as a revenue product. The pattern across most large platforms, however, is blocking or absence of any agent-access infrastructure.

Grounded in 7 sources
Meta may use sweeping new commercial TOS to avoid millions in AI licensing fees - Ars Technica ↗ · arstechnica.com
Websites are closing their doors to AI agents ↗ · theverge.com
Meta removes language that may have allowed it to scrape news sites - The Verge ↗ · theverge.com
Amazon Blocks Meta's Muse AI Shopping Agent Over Terms of Service - Adsroid ↗ · adsroid.com
Cloudflare AI Crawl Control vs robots.txt: Check Both Layers ↗ · agentready.to
Scaling cloud migrations with agentic AI on Amazon Bedrock AgentCore | Artificial Intelligence ↗ · aws.amazon.com
Websites Block AI Agents: The New Web Access Battleground ↗ · emergent.sh
Read transcript

Iris Holm: Good to be back — how'd your week treat you?

Lila Soto: Oh, weird, actually — I saw something that I keep thinking about. There's a Meta product called Muse, a shopping agent, and Meta and Walmart publicly partnered so it could work on Walmart's site. And users are reporting it just gets blocked anyway. Walmart's own anti-bot systems flag it.

Iris Holm: Right. A contract doesn't touch infrastructure.

Lila Soto: Which, yeah — but also, Amazon didn't even try the partnership route. September 20, 2026, they just blocked Meta Muse. And before that, November 2025, they sued Perplexity over its Comet browser agent. A federal court ordered Comet off Amazon while the lawsuit proceeds. So we have actual legal precedent forming in real time.

Iris Holm: First major legal test of whether a platform can enforce its terms of service against an agent acting on a user's behalf. And the court said yes — at least provisionally.

Lila Soto: So the thing I want to figure out — if platforms can block agents even when users want them, and even when there's a deal on paper, what does the promise of AI shopping agents actually mean?

Iris Holm: Think about a mall that lets shoppers in freely but bans the personal shopper you hired yourself — while running its own personal-shopping service in every store. That's Amazon. They block Meta Muse, they sue Perplexity over Comet, and meanwhile they're running Rufus, Alexa for Shopping, Buy for Me — all first-party agents, all inside the walls.

Lila Soto: Wait, so Yelp is literally charging agents to get in?

Iris Holm: Explicitly. Paid data-licensing program. Access is a revenue line. Not a risk to manage — a product to sell. That's not a security posture. That's a toll booth.

Lila Soto: And eBay made it even more literal — they updated their user agreement in February 2026 to ban buy-for-me agents by name. Not 'unauthorized bots.' The actual category of thing.

Iris Holm: Named the threat. Which tells you they knew exactly what they were protecting. JPMorgan Chase and Charles Schwab say security — and I mean, fine, maybe — but that rationale also happens to lock a third party out of their customer relationships and transaction data. Delta Air Lines has zero agent deals as of October 2026. Zero. Not one. LinkedIn and Facebook block nearly every bot they haven't approved themselves. The pattern isn't security. The pattern is: we monetize the access.

Lila Soto: Huh — and even when a platform wants to let a specific agent in, there's apparently no real infrastructure to do that cleanly yet. Which sounds like it gets worse before it gets better.

Iris Holm: Much worse. But that's the next layer. The question right now is simpler: Shopify opened its storefronts to agents, so the permissive move is possible. Amazon chose not to. That's a business decision wearing a security badge.

Lila Soto: But Shopify *can* open the door, okay, so the permissive move is real. But picture someone on a Friday night, on their smart TV, asking a voice agent to book a last-minute Delta flight. The agent just... hits a wall. Not because Delta put out a press release saying no. Because Delta never built a pathway for agent-based booking at all. There's nothing to knock on.

Iris Holm: Delta has zero agent deals as of October 2026. Zero. Not one signed agreement. So the agent isn't being rejected — there's no handshake protocol to even attempt.

Lila Soto: Which is — I mean, that's a different problem than blocking, right? That's just absence.

Iris Holm: Absence that gets enforced by mature tools. CAPTCHA, rate limiting, IP-blocking of known AI infrastructure, advanced fingerprinting — those aren't new. They're already deployed widely. The blocking side is built. The credentialing side — the 'yes, this is a trusted agent, let it through' side — that's not.

Lila Soto: And Web Bot Auth was supposed to be that, kind of? Cloudflare, Visa, Mastercard announced it October 2025 — a standard for agents to prove their identity to platforms. But I guess... it's still a pilot? I haven't seen any confirmation it's actually been adopted broadly.

Iris Holm: Early-stage. And look — Cloudflare flipped to blocking AI crawlers by default for all new sites in July 2025. Three months later they're selling per-page crawler charging. So the company announcing the credentialing standard is also the company that just turned blocking into a baseline and access into a pricing tier. That's not a contradiction — that's a strategy.

Lila Soto: So even if Delta wanted to let a trusted agent in tomorrow — like, genuinely wanted to — there's no reliable way to do that yet. The tools to say yes don't exist at scale. Only the tools to say no do.

Iris Holm: That's actually where I land, yeah. The patchwork right now — Amazon blocks outright, Yelp charges per license, Shopify opens — that's probably not temporary chaos. That's probably the template. robots.txt eventually produced rough norms across the early web, but it took years and the stakes were just indexing. Here the stakes are transaction revenue, liability, customer ownership. The commercial pressure to *keep* the patchwork is orders of magnitude higher.

Lila Soto: So the question isn't whether agents will exist. It's whether your agent will work where you actually need it to — or only where someone already cut a deal.

Iris Holm: Exactly that.

Lila Soto: Kind of a strange place to end up, given how the whole thing was sold. Anyway — this was good. Genuinely.