Google plans shared robots.txt control for AI training

Google said it plans to support a shared AI training control in robots.txt, a change meant to let website owners express training preferences across multiple AI providers in one place. The announcement came from the Google Search Central Blog, which said the approach is based on work from the AIPREF working group at the IETF.

Google also said this new option will sit alongside its existing Google-Extended control. The company said it aims to support the new control with other AI providers in 2027 and will share progress updates.

What changed

Under the proposed setup, site owners would be able to place a shared policy token in robots.txt instead of managing separate instructions for each AI company. Google said that if a site uses that token to opt out of training, and Google-Extended is not explicitly named, it plans to treat that as if Google-Extended had been disallowed.

Google said existing company-specific controls will keep working. That means a publisher could use the shared signal for a broad preference while still setting different rules for individual AI providers. Google also noted that websites that specifically allow Google-Extended are eligible to appear in responses and receive clicks via grounding in Gemini Apps.

What it means for your website

For business websites, this does not change normal Google Search crawling rules. It is about AI training preferences, and Google is presenting it as a simpler way to state those preferences across participating providers.

If your site already uses Google-Extended, there is no sign that the current control is going away. The main difference is that a shared robots.txt signal may reduce the need to manage separate instructions one by one, while still letting you make exceptions for Google or other AI providers.

What to do now

• Do not rush to change robots.txt today. Google said it aims to support this new control in 2027, so this is a plan, not a live requirement.
• Review whether your site currently uses Google-Extended or any other AI-related robots.txt rules, so you know what is already in place.
• If AI training preferences matter to your business, monitor Google’s updates on how the shared policy token will be implemented and interpreted.
• If you want different treatment for different AI providers, keep in mind that Google said company-specific controls will continue to function alongside the shared signal.
• If your site relies on visibility in Gemini Apps, note Google’s statement that websites that explicitly allow Google-Extended remain eligible for responses and clicks through grounding.