OpenAI, Google, Microsoft, and Anthropic have jointly announced their adoption of a new standard that will enable content publishers to uniformly opt out of AI scraping of their website content. This standard allows publishers to declare their content usage preferences for AI model training to multiple AI companies simultaneously, eliminating the need to set rules individually for each company. The standard, proposed by Anthropic technical lead Kevin Kelley to the Internet Engineering Task Force, will be integrated into websites' robots.txt files. Previously, publishers had to separately declare their preferences to each AI company and were overwhelmed by various AI crawlers continuously scraping content for model training. The new standard aims to streamline this process. Currently, Anthropic has taken the lead in adopting it, OpenAI plans to implement it within the next few months, Google and Microsoft are expected to support it next year, and Meta says it supports establishing such a universal signal but has not specified an implementation plan.
