Major AI Companies Back One Web Signal to Block Training Scrapes

Four of the biggest names in artificial intelligence are backing a common way for websites to say their content should not be collected for model training. Google, OpenAI, Microsoft and Anthropic have committed to honoring a shared instruction placed in a site’s robots.txt file.

One signal instead of a list of crawlers

Today, publishers often have to identify and block separate crawlers operated by individual AI companies. The proposed standard would replace much of that provider-by-provider work with a single usage signal. A website could disallow AI training across its pages while still opening selected areas, such as a press kit.

The proposal was written by Kevin Kelley, Anthropic’s technical lead for standards, and is being considered through the Internet Engineering Task Force. The familiar robots.txt file already tells automated crawlers which parts of a website they may access. This extension gives site owners a more specific way to distinguish AI-training collection from other uses.

Publishers gain a simpler control, not a legal shield

The participating companies say their training crawlers will respect the shared signal. Microsoft has also described more granular options for separating model training from “grounding,” where an AI system points to current information to improve an answer. Search visibility can remain separate from those preferences, meaning a site does not necessarily have to disappear from traditional search results to reject training use.

The move addresses a practical complaint from publishers: maintaining an expanding block list becomes harder as new crawlers appear. It also arrives while media companies, creators and platform operators debate whether automated access should require permission, payment or licensing.

The new signal does not settle those larger disputes. Robots.txt instructions have historically depended on crawlers choosing to comply, and bad actors can ignore them. Still, support from Google, OpenAI, Microsoft and Anthropic gives the proposal immediate weight. Meta has expressed support for the idea, though its public position stopped short of a clear commitment to honor the setting.

For website owners, the value is straightforward: one readable preference could reach several major AI providers at once. The next test is whether the proposal becomes a durable open standard and whether companies apply it consistently in real-world crawling.