DB
Blog

We build your website FREE

On a build + Care package with a 12-month minimum — from $199/mo, switch tiers anytime. No separate charge for the build.

See packages →

BLOG

Should Your Website Block AI Agents? Big Brands Just Did

·10 min read·
ai-agentsaeoai-search-visibilitycare-plan
Share
Add The Digital Back Office as a preferred source on Google

Surfaces our work more for you across Google Search and AI answers.

Picture a customer a few miles from you telling their phone to find a groomer who can take a nervous cat on Saturday and book it. The assistant goes looking. It reaches your site, hits a bot wall your host turned on last spring, and moves to the next name. You never see the visit. Nothing in your analytics moves. The appointment lands somewhere else.

Meanwhile, two of the biggest companies on the internet have made the opposite bet, loudly. It's worth understanding why their decision is the wrong one to copy.

Quick read: On November 11, 2026, United Airlines updated its website terms so that "No Automated Agent shall be permitted to access United's site or app without prior written permission from United," while exempting search engines that index its public pages; the same day it published a release citing customer data, unintended purchases, and Federal Aviation Administration (FAA) and Department of Transportation (DOT) rules as the reasons (Search Engine Journal; United). Two days later OpenAI began rolling out ChatGPT Wallet, which lets its "dots" agents pay on a merchant's own checkout with a network-issued stand-in card number after the user approves each purchase with a biometric check; it is limited to US ChatGPT Pro subscribers aged 18 and over (Help Net Security). OpenAI's merchant documentation warns that "your security settings may unintentionally block these visits" and describes allowlisting its signed agent requests in Cloudflare, Akamai, HUMAN, and Vercel (OpenAI). A robots.txt file reaches only crawlers that honor it: OpenAI says its rules "may not apply" to user-initiated ChatGPT-User fetches, and Google says its user-triggered fetchers "generally ignore robots.txt rules" (OpenAI; Google). Amazon blocked Meta's Muse agent in September 2026 under its Conditions of Use rather than robots.txt (Search Engine Journal).

What United and Amazon did, and with which lever

United's move is a terms-of-service change. Search engines may keep indexing its public pages. A person with a browser may keep booking. Anything United classes as an "Automated Agent" needs written permission first, and United says it may block one "at any time, regardless of who operates it." Its stated reasons are the ones you'd expect from an airline: customers handing account passwords to a third-party assistant, miles spent or tickets bought by mistake, and the chance that an automated action trips a federal aviation rule.

Amazon got there first. In September it cut off Meta's Muse shopping agent and showed its users a page saying continued access "violates Amazon's Conditions of Use." Amazon's robots.txt had nothing to do with it. Meta had never published a name for Muse that a robots file could address, and Meta's own launch materials said Muse browsing "will appear as your activity" (Search Engine Journal).

Notice the lever both companies reached for. A contract the customer agreed to, enforced by whoever runs the company's bot defenses. Neither one touched the file most owners think of as "the bot settings."

Why a destination brand can afford that and you can't

If an assistant can't book a United flight, the customer still flies United. The agent reports back, the person opens a browser, and the sale happens anyway, minus the middleman the airline didn't want. The brand is the destination. Locking the side door costs it very little.

Will the agent come back for you after it bounces? That's the whole question, and for most local businesses the honest answer is no. The person asked for a groomer who can take a cat on Saturday, and there are four other groomers on the list. The assistant doesn't carry a grudge. It carries on.

The three risks United named don't translate either. Nobody is sharing a MileagePlus password to request a quote from a plumber. There are no federal rules against booking a Tuesday drain cleaning by software. If an agent asks for an estimate you didn't want to give, you have a phone number and a cancel button.

So the decision in front of a small business is simpler than the one in front of an airline, and it points the other way. The AI tools your customers use are moving from answering questions to completing the task. OpenAI's own merchant guide now says a sale through its agent happens on your site, with your checkout or booking flow, and that the merchant keeps the customer. A business that can't be reached that way is, in those sessions, a business that doesn't exist.

The three switches, and the one that does anything

"Block AI" sounds like one decision. When we covered Google's robots.txt switch for AI training we split it into two: whether a company may train on your pages, and whether an AI product may quote you. Agents add a third: whether software acting for a customer may use your site. Three different switches, in three different places.

robots.txt. This is a note left on the door. Well-behaved crawlers read it. OpenAI's search crawler, OAI-SearchBot, honors it, and blocking that one keeps you out of ChatGPT's search answers. But OpenAI is explicit that when a user asks ChatGPT to open a page, "robots.txt rules may not apply." Google says the same about its user-triggered fetchers, including the one it calls Google-Agent. Meta's Muse and OpenAI's dots don't appear in either company's published crawler lists at all, so there is no name to write in the file. For agents, robots.txt is the wrong tool whether you want them in or out.

Bot protection. This is the actual lock, and it probably isn't yours. Cloudflare, your host's firewall, a WordPress security plugin: whichever one sits in front of your pages decides, request by request, who looks like a person and who gets a challenge page. This is where OpenAI is pointing merchants. Its agent signs requests with a cryptographic header (the scheme is called Web Bot Auth) so a firewall can verify the visit came from ChatGPT, and OpenAI's guide walks through allowing it by name in Cloudflare's "Personal Agents" policy, in Akamai, in HUMAN, and notes Vercel recognizes it on its own. Its warning to merchants is aimed at exactly this layer: settings that "unintentionally block."

Terms of service. This is a statement of intent with a lawyer behind it. For Amazon and United it's a real lever, because they have the lawyers, the engineering to detect the agent, and customers who will come back regardless. For a small business it's mostly a sentence on a page nobody reads, and nothing on this site is legal advice. Decide what you want to happen first. The terms can describe it afterward.

Where to leave the door open and where to put the lock

A small business doesn't need a policy. It needs five checks, and most of them are questions for whoever runs your hosting.

  1. Find out what stands in front of your site. Ask whoever manages hosting: are we on Cloudflare, is there a firewall or security plugin, and what is its bot setting? If your site was last touched before this summer, nobody set that on purpose. It came with the host.
  2. Separate the three switches in that dashboard. Training bots are your call either way. Search crawlers such as OAI-SearchBot and Googlebot stay allowed unless you want to disappear. Verified agents, the Cloudflare "Personal Agents" category and its equivalents, are the new line, and the default may have been chosen for you. We walked through Cloudflare's own screens when its defaults changed in September; the agent policy lives one tab over.
  3. Check what the agent sees once it's inside. Letting it through the door is wasted if the page it reaches hides your hours in an image and your booking button behind a script. What an agent can and can't read on a page is a separate check, and it's the one most redesigns skip.
  4. Decide what no software should do without you. Maybe that's booking without a deposit, or quoting a price you'd rather give by phone. Put that gate inside the flow (a confirmation step, a deposit, a "we'll call to confirm" message) rather than at the front door. OpenAI's own agent requires "human review before any consequential action," so a confirmation step is the one friction it expects.
  5. Expect the visits to look human. Agents using a real browser on a customer's behalf show up in analytics as a person did. If calls or bookings start arriving with no visible source, that's one explanation, and the number to watch is bookings, not sessions.

Here's the thing about this whole list: none of it is a project. It's a set of settings that drift. A host updates its defaults, a plugin adds a rule, and a door that was open in November is shut by February with nobody told. I ran a grooming salon for years, and the back-of-house checks were the first thing to go when the day got long. This is a back-of-house check.

Panic moves copied from an airline

Pasting an "AI agents need written permission" clause into your terms. It changes nothing on its own, you have no way to enforce it, and it tells a customer who notices that you'd rather they did it the hard way.

Setting the security plugin to block every bot "to be safe." The safest-sounding setting is the one that catches the agent carrying a booking, and sometimes the search crawler too. Block training bots if you like. Blocking by category is a decision; blocking everything is a shrug.

Adding a CAPTCHA to the quote form. It stops the visitor you wanted and slows the one who was going to call anyway. We removed ours after the third-party script it depended on kept failing and took the form down with it.

Fixing it in robots.txt. You'd be writing rules for crawlers that read them, about agents that don't.

Buying an "agent-readiness" audit. The five steps above are the audit. If someone charges you four figures for a score, ask which firewall setting it changed.

Doing nothing because the agents are still rare. They are. ChatGPT Wallet is a US-only, Pro-tier feature in its first week. The settings that decide whether you're reachable are already live, though, and they were set without you. Checking them now costs an hour. Finding out in a slow March costs more.

FAQ

Does blocking AI agents hurt my Google ranking?

Not directly. Google's ranking uses Googlebot, which is a crawler and reads robots.txt. The risk is indirect: a bot-protection rule written by behavior rather than by name, broad enough to challenge an AI agent, can challenge a search crawler too. Check the specific rule rather than assuming.

Can I block AI agents in robots.txt?

Mostly no. OpenAI says robots.txt "may not apply" to its user-initiated fetches, Google says its user-triggered fetchers "generally ignore" the file, and Meta's Muse and OpenAI's dots have no published crawler name to address. robots.txt is for crawlers such as OAI-SearchBot and Googlebot. Agent decisions happen in your firewall or bot-protection settings.

What is Web Bot Auth?

A way for an automated visitor to prove who it is by cryptographically signing each request, so a site can verify the signature against a published key instead of trusting a user-agent string. OpenAI signs its ChatGPT agent's browser requests this way, and Cloudflare, Akamai, HUMAN, and Vercel can verify them. For you it means an "allow verified agents" setting is possible without opening the door to every bot.

Should a local service business let AI agents book appointments?

If you'd take the booking from a person on your website, there is little reason to refuse it from software acting for that person. Put any condition you care about, a deposit, a confirmation call, inside the booking flow so it applies to everyone. The agent expects a confirmation step; OpenAI's requires human review before consequential actions.

Why did United ban AI agents from its website?

United's November 11, 2026 release cites three reasons: customers sharing account credentials with third-party assistants, unintended purchases or use of MileagePlus miles, and the possibility that automated actions violate FAA and DOT regulations. Its terms still allow search engines to index public pages. Those reasons fit an airline with a loyalty program and federal oversight; most of them don't apply to a small business site.

Will I see AI agent visits in Google Analytics?

Usually not as a separate line. Agents that use a real browser for a customer appear as that customer's visit, and Meta has said Muse browsing "will appear as your activity." OpenAI's agent identifies itself through signed headers your firewall can read, but analytics tools don't surface that. Watch bookings and calls rather than sessions.

Is ChatGPT Wallet available to my customers?

At launch, only to US ChatGPT Pro subscribers aged 18 and over with eligible US-issued cards, and only through OpenAI's dots agents. The purchase happens on the merchant's own checkout with a stand-in card number, approved by the user each time. It's early and narrow, which is the right time to check the settings rather than the right time to redesign anything.

Being the business an AI answer names, and the one an assistant can then reach, is what we mean by Answer Engine Optimization: the second search. If you'd rather someone checked the firewall, the robots file, and the booking flow every month and told you in plain English what changed, that's the ongoing work behind being findable by Google and by the AI tools people now ask.

Want a back office for your site?

The build is free on any Care plan with a 12-month minimum. Care starts at $199/mo, and you can switch tiers anytime.

Apply for a build slot →