OpenAI, Google and Microsoft back a shared IETF robots.txt token for AI-training opt-outs; Bing also splits training from grounding, effective 2027

Why it matters

On 2026-10-30, OpenAI and Google announced support for a shared robots.txt AI-training preference signal based on the IETF AI Preferences (AIPREF) working group, and Microsoft Bing announced matching training and grounding controls; Search Engine Land reports Anthropic also joined. The IETF draft uses a policy token in the User-Agent line (__usage_ai-training) with ordinary Disallow/Allow rules; wildcard groups do not match it. OpenAI will implement it in the coming months alongside its existing crawler controls. Google plans support in 2027 and will treat an opt-out as disallowing Google-Extended unless Google-Extended is set explicitly. Bing separates AI Training from AI Grounding, adds page-level controls and a DisallowExtended option, and starts honoring the new preferences on Feb 1, 2027. The signal is voluntary and covers foundation training only, not retrieval or grounding.