What happens between you and an AI chatbot does not always stay between you and a chatbot. Just ask Google.
Over the weekend, people were surprised to discover that some Anthropic Claude chats could be easily found via web search. The issue, which appears to have been first flagged by a redditor, exposed chats that included people asking for advice about what political party they should join, whether attorneys in Kansas are required to self-report when they believe they’ve committed an ethical violation, and erotic role play.
Claude allows users to share with other people “snapshots” of chats by creating a public URL to a specific chatbot thread. The reasons some of these URLs were indexed by major search engines comes down to the basic functions of websites, search engines, and the collision of the two when generative AI gets in the mix.
Basically, Anthropic instructs web crawlers, like those used by search engines like Google and Bing, not to index chats a user decides to share with other people. The company does this via something called a robots.txt file, which has long been considered the standard way to tell web scrapers what parts of a site are appropriate to access. Anthropic’s robots.txt has made “shared” chats off limits to web scrapers since at least September 2025, according to a snapshot on the Wayback Machine.
But it’s harder to prevent pages from being included in search engine results than you might expect.
Bing, which still shows “about 612 results” if you search “site:claude.ai/share” at the time of this writing, says in its technical documentation that developers can block the search engine’s web crawlers using robots.txt—but that web developers should also include a “noindex” tag on individual pages as well.
Some of the chats that showed up in search results flagged in the Reddit post were deleted by the time WIRED viewed them, and results no longer show up in Google when you search the “share” query that still worked on Bing. In a developer guide, Google says that it ignores robots.txt instructions if that page is linked to from elsewhere on the internet and the page owner doesn’t also include a special “noindex” html tag on the page or a “x-robots-tag” in the page’s response header.
WIRED reviewed a sample of the exposed Claude chat pages and found that they did not include the “noindex” tag that both Bing and Google say they take into consideration when deciding whether or not to index a page.
Microsoft, which owns Bing, did not provide comment ahead of publication. Anthropic did not respond to multiple requests for comment.
Google spokesperson Ned Adriance tells WIRED that the indexing of shared Claude chats is Anthropic’s responsibility. “Neither Google nor any other search engine controls what pages are made public on the web, and these pages were indexed across many search engines,” Adriance says. “We give site owners clear controls to decide whether pages can be crawled or indexed, and we always respect those directives.”
Anthropic didn’t respond to questions about why it didn’t include the “noindex” tag on the shared chat pages.
Last September, Anthropic got heat for the same issue, and told Forbes that it uses robots.txt to let crawlers know they shouldn’t access the shared chats. But as the Forbes report points out, there’s no guarantee that will stop search engines from indexing specific web pages.
Even if robots.txt doesn’t always work to prevent pages from being indexed on search engines, AI labs are still making use of it for other purposes. Many labs promise creators that their websites won’t be used as AI training fodder so long as the developers make sure to “disallow” certain crawlers in their robots.txt files, and the labs themselves take that advice as well.
Anthropic, Meta, and OpenAI all include instructions in their chatbots’ robots.txt files that “disallow” their competitors’ web crawlers from accessing any part of the website where the chatbots are hosted. OpenAI and Meta did not respond to a request for comment about the practice. Google did not address WIRED’s questions about the fact that its competitors are blocking its AI training crawler, Google-Extended, from their chatbots’ sites.
Though shared chats are no longer turning up in Google results, the pages WIRED reviewed still don’t have a “noindex” tag, meaning that they could potentially show up in search engines again. Claude users can go to Settings > Privacy > Shared chats to manage access—and keep their secret conversations private.
Additional reporting by Andrew Couts.
Comments
Back to top