Cloudflare's New Default Could Hide You From AI Search

Mark September 15, 2026 on your calendar, next to a setting you never touched. That's the day Cloudflare — a company that sits quietly in front of a huge share of the web, possibly including your site, set up by a host or a plugin you never configured yourself — starts blocking AI bots by default on any page that carries an ad. You didn't ask for this. Whoever built your site three years ago couldn't have warned you, because the policy didn't exist yet.

Here's the part that makes it worth an afternoon of your attention instead of a shrug: this has nothing to do with your robots.txt file. You can have that file set up perfectly and still get quietly caught by this, because it operates at a different layer entirely — the firewall in front of your site, not the file on it.

What's actually happening here

Cloudflare runs a feature called AI Crawl Control that sorts incoming AI bot traffic into three buckets: Search (bots building a search index), Agent (bots fetching a page in response to something a real person just asked, in real time), and Training (bots harvesting content to train future models). Historically, site owners who wanted any control over this had to go set it up themselves.

That changes September 15, 2026. Per Cloudflare's own announcement, new domains joining Cloudflare, new customer accounts, and existing accounts on the free plan will get a new default: bots classified as Training or Agent are blocked on any page that displays ads, while Search stays allowed. If you're an existing paying customer on an existing domain, your current settings aren't automatically rewritten — but plenty of small business owners have no idea which of those buckets they're even in, which is the actual problem here.

The part that makes this worth double-checking even if you think none of that applies to you: a large share of AI bot traffic doesn't fit neatly into one category. Cloudflare's own numbers put mixed-use crawlers — ones that do both search indexing and model training — at more than a third of all the AI bot traffic it sees. Its rule for those is to apply whichever restriction is stricter. Multiple outlets covering the rollout flagged that Cloudflare's classification treats Googlebot itself as one of these mixed-use crawlers, which means a site that blocks Training bots on its ad-supported pages can end up blocking ordinary Google Search crawling on those same pages too, not just the AI-specific bots — unless the setting is explicitly overridden.

Why this actually matters for your site

If you've put any work into showing up in AI search — a redesign, review generation, structured data, the works — none of it shows up as a visible bug when a firewall setting quietly starts turning away the tools that would cite you. Getting an AI system to notice you exist in the first place is the whole reason Answer Engine Optimization exists as its own discipline, separate from traditional search ranking, and a bot block undoes that at the most basic level there is: the tool never even gets in the door. Your site loads fine. A customer clicking through it has no idea anything changed. The only symptom is an answer you should have shown up in, where you just don't, and there's no error message telling you why.

I ran a dog grooming salon for years before any of this existed, and I can promise you nobody running a small local business is checking their CDN's bot-classification defaults on a Tuesday between appointments. That's not a knock — there was never a reason for a business owner to know this was a lever that existed, let alone one that could flip itself without anyone touching it. This is exactly the kind of thing that falls through the crack between "my website guy set it up" and "nobody's actually watching it."

The distinction that actually matters here — which bots you'd want blocked versus allowed — is a separate, deeper topic than this post, and we've already covered the specific bot names and what each one does in the do-it-yourself audit for your own robots.txt file. What's new is that Cloudflare's defaults now make that decision for you, at the network level, if you never made it yourself.

The mechanics: what to actually check

1. Find out if you're on Cloudflare at all. If you don't know, ask whoever manages your hosting or DNS — it's a fair question and a fast one to answer. A lot of hosts, page builders, and security plugins put a site behind Cloudflare as part of their default setup, with no announcement to the business owner.

2. If you are, log into the Cloudflare dashboard and open AI Crawl Control. There are three relevant tabs: Crawlers, which shows which named bots are actually requesting your content and lets you set allow or block per bot; Robots.txt, which shows how bots are interacting with your existing robots.txt directives; and Metrics, for volume and traffic patterns.

3. Decide your policy deliberately, not by default. Blocking Training-only bots is a reasonable, low-cost choice if you'd rather your content not feed future model training. Blocking Agent or Search bots is a different decision entirely — those are the ones fetching your page because a real customer just asked a real question, which is the exact moment you want to be readable.

4. Recheck this after September 15, even if you're sure you didn't change anything. "I didn't touch it" isn't the same as "nothing changed" when the default itself is what's moving.

5. Treat it as a recurring check, not a one-time fix. This is the same lesson as checking whether an AI agent can actually parse your page once it gets there: platforms revise their defaults, plugins update, hosts change configurations, and a setting that was fine in August isn't guaranteed to still be fine in December.

What won't help

Blocking every AI bot "to be safe." This is the instinct that backfires hardest here. Blanket-blocking Training bots on ad-supported pages is exactly what risks catching regular Googlebot too, per Cloudflare's own mixed-use classification — you can accidentally hurt your ordinary Google visibility while trying to protect yourself from something else entirely.

Assuming this doesn't apply because you "don't run ads." Maybe it doesn't. But the only way to know your current Cloudflare bot settings is to check them — not to reason your way to an assumption about a dashboard you've never opened.

Waiting for your web person to mention it. If your site was built or last touched before July 2026, whoever built it had no reason to know this policy would exist. This isn't a gap in their work. It's a new rule from a company neither of you works for.

Treating "I checked once" as done. Same as the robots.txt audit, same as the accessibility-tree check — this is maintenance, not a one-time project. A managed website is built around catching exactly this kind of silent, platform-level shift before it costs you visibility for months.

FAQ

What is Cloudflare, and how do I know if my site uses it?

Cloudflare is a company that sits in front of a huge number of websites, handling things like DNS, security, and content delivery — often set up by a host or developer without the business owner ever seeing it directly. If you're not sure, ask whoever manages your hosting or domain; it's a quick question with a definite answer.

What exactly changes on September 15, 2026?

For new domains joining Cloudflare, new customer accounts, and existing free-tier accounts, AI bots that Cloudflare classifies as "Training" or "Agent" will be blocked by default on any page that carries ads, while bots doing search indexing remain allowed.

Does this affect my Google ranking?

Not directly for most pages — but on ad-supported pages specifically, Cloudflare's own mixed-use classification has reportedly caught regular Googlebot traffic when Training bots are blocked, which is a genuine risk worth checking rather than assuming away.

I don't run ads on my site — do I need to do anything?

Probably lower urgency, but you still won't know your actual current settings until you check the Cloudflare dashboard (or ask whoever manages it for you). "Probably fine" isn't the same as "confirmed fine."

What's the difference between a "Training" bot and an "Agent" or "Search" bot?

Training bots harvest content to help build future AI models. Agent and Search bots are the ones doing something in real time — indexing your site or fetching a page because a live person just asked a question an AI tool is trying to answer. Blocking the first is a low-stakes content-licensing decision; blocking the second is what actually makes you invisible in AI answers.

Do I need a developer to fix this?

Not necessarily. If you're comfortable logging into a dashboard and flipping a few toggles, you can likely handle the Cloudflare side yourself in a few minutes. Where a developer helps is if you're not sure whether you're on Cloudflare at all, or what your site's ad setup actually looks like from the outside.

What if I'm not on Cloudflare at all?

Then this specific change doesn't apply to you directly — but the underlying lesson does: bot access to your site is increasingly controlled at layers you didn't build and don't automatically monitor, whether that's a CDN, a security plugin, or a hosting default. Worth a periodic check regardless of which one you're behind.

---

Catching a policy change like this before it quietly costs you real AI visibility is exactly the kind of unglamorous, ongoing work getting found on Google and AI is built around. Nobody emails you when a default flips. Something has to be watching for it anyway.