Google's Coming robots.txt Switch for AI Training

Today, telling AI companies what they may do with your content means a list. One robots.txt rule for Google's training crawler, another for the next company, another for the one after that. Nobody on a small-business site is keeping that list current.

Google just said it wants to replace the list with one line. Not yet, though. The date it gave is 2027.

What Google announced

The proposal is a single shared signal in your robots.txt file. A site sets a broad preference once, and every participating AI provider reads the same line instead of each inventing its own rule.

Three details from Google's post matter for a site owner:

• It maps onto what exists. If your robots.txt opts out of training with the shared token and doesn't name Google-Extended, Google plans to treat it as a Google-Extended disallow.
• The old controls stay. Google says existing company-specific controls "will continue to function alongside this new joint signal." If you've already written a Google-Extended rule, it keeps working.
• It's a plan, not a switch. Google says it aims to support this in 2027 and will post updates. The post links to an IETF draft for details and says the rest of the AIPREF vocabulary is still under discussion. It doesn't give the final syntax, so don't copy a line from a blog post that claims to know it.

Training and appearing are two separate dials

This is where owners get hurt. "Block AI" sounds like one decision. It's at least two.

Training is whether a company may use your pages to build or improve its models. Appearing is whether an AI product can quote you and send people your way. Google's post says sites that specifically allow Google-Extended "are eligible to appear in responses and receive clicks via grounding in Gemini Apps." So opting out of training with the new token is not a neutral act: if it counts as a Google-Extended disallow, it may change how your site is treated in that Gemini grounding.

Google's post doesn't say what the token does to ordinary Search ranking or AI Overviews, so there's nothing sourced to claim either way. Treat anyone who tells you with confidence as selling something.

What to do before 2027

Nothing in this announcement is urgent, and nothing needs rewriting this week. A few boring checks are worth an afternoon:

1. Open your robots.txt. Visit yourdomain.com/robots.txt. Know what's in it. Plenty of sites carry rules a plugin or a former freelancer added years ago.
2. Decide the actual question. Do you want to be quoted by AI answers? For most local businesses, being named in answers is the point, and our AEO pillar post covers why. Training opt-out is a separate preference, and it's yours to make.
3. Check the other gate. robots.txt isn't the only thing that can shut crawlers out. A host or CDN setting can block AI crawlers before robots.txt is ever read, which is the problem in Cloudflare's block-by-default change.
4. Watch who's actually visiting. If you want to see which AI crawlers hit your site, tracking AI agent traffic shows how.
5. Write down the decision and the date. A one-line note beats rediscovering it in 2027.

Robots.txt moves to skip for now

• Pasting an unverified directive. Until the syntax is published, a made-up line either does nothing or blocks something you didn't mean to.
• Blocking everything to be safe. A blanket disallow can also keep you out of the places customers now ask questions.
• Treating robots.txt as a lock. It's a request that well-behaved crawlers honor. Google's own post frames the new signal as a preference, not enforcement.
• Panic-editing now. Google says existing controls keep working. The cost of waiting for the final syntax is near zero.

If you'd rather have someone read your robots.txt, your host settings and your AI visibility together, that's part of getting found on Google and AI.

FAQ

What is the new Google robots.txt AI training control?

It's a shared "policy token" Google plans to support in robots.txt so a site can set one AI training preference for participating providers. Google announced the plan on October 30, 2026, and says it aims to support it in 2027.

Can I use it today?

No. Google says support is planned for 2027, and the final syntax wasn't given in the announcement. Google-Extended is the control that works now.

Will opting out of AI training hurt my Google rankings?

Google's announcement doesn't address Search ranking or AI Overviews. Check Google's documentation when the syntax is published.

What is Google-Extended?

It's the robots.txt control Google has offered since 2023 for AI training. Under the new plan, a shared opt-out is treated as a Google-Extended disallow when you haven't named it yourself.

Will my existing Google-Extended rule stop working?

No. Google says existing company-specific controls will continue to function alongside the new joint signal.

Does this apply to ChatGPT, Claude and Perplexity too?

Only if those companies adopt it. Google says other providers can handle the token in a similar way, but its post doesn't name any.

Should a small business opt out of AI training?

It depends on what you want. Opting out of training and being quoted in AI answers are separate decisions, and the second is usually what a local business wants. Decide each on purpose rather than by blanket rule.