Google and Microsoft among those signing up to OpenAI's plan for a new website crawling standard — including opting-out of internet scraping

One robots.txt line could soon tell Google, Microsoft, OpenAI, and Anthropic not to train on your site, but it won't stop their bots from crawling it.