Your Site Has AI Bot Traffic Now. Should You Track It?

For as long as small business websites have existed, there's been one honest answer to "who's looking at my site besides customers?": search engines, full stop, and nobody bothered to check past that. That answer just got more complicated. On August 13, a company called OtterlyAI shipped a feature that does one specific thing — it reads a site's own server logs and reports, bot by bot, which AI systems showed up that month. Not a ranking. Not a score. Actual visits, actual pages, actual bot names.

I'll be honest about why this caught my attention: it's not that the underlying activity is new. AI bots have been crawling small business sites for a while now. What's new is that "how much of that is happening to you, specifically" just turned into a number a vendor will sell you, which means it's about to show up in more sales pitches aimed at business owners who've never had a reason to think about it.

What "AI agent traffic" actually is

Every time an AI tool touches your website, something is doing the fetching, and it identifies itself with a user-agent string the same way a browser does. OtterlyAI's own breakdown sorts that traffic into three buckets: a search crawler building an index (comparable to how Googlebot works), an on-demand agent fetching your page right now because a real person just asked a real question, and a training scraper harvesting content to help build a future model. The company names specific bots it tracks in each category, including OpenAI's ChatGPT-User, Anthropic's Claude-User, Perplexity-User, and Google's own AI-agent crawler.

That three-way split matters more than the traffic count itself. A pile of training-bot visits tells you your content might end up shaping how a future model talks about your industry in general — mildly interesting, not urgent. A pile of on-demand agent visits tells you someone, this week, asked an AI tool a question your site helped answer in real time. Those are wildly different signals wearing the same label of "AI bot traffic," and a tool that lumps them together is telling you less than it looks like.

None of this bypasses robots.txt. Reputable crawlers — the ones from OpenAI, Anthropic, Google, and Perplexity among them — still read and generally follow the rules a site publishes there. But it's a request, not a lock: the file has no technical way to force a crawler to comply, and it's on the bot's owner to build in that respect voluntarily. Worth knowing too: allowing one of a company's bots doesn't automatically allow its others. OpenAI's GPTBot (training) and OAI-SearchBot (search indexing) are governed by separate lines in the same file, and a site can permit one while blocking the other.

Why this matters for your specific situation

Crawl traffic is the floor everything else in Answer Engine Optimization (AEO) sits on — a bot has to reach and read your page before any of the citation work matters. If you run a small business, the honest answer to "should I buy a tool like this" is usually: not yet, and maybe never. What actually matters is the question underneath the product — is anything showing up in your server logs at all, and if it is, which bucket is it in.

That's worth knowing regardless of whether you pay to track it monthly, because the answer tells you which problem you have. Zero agent-category visits after months of otherwise-normal traffic usually means something structural is keeping AI tools out — a robots.txt rule blocking more than you meant to, or a technical issue keeping your actual content from ever loading for anything reading the page. Agent visits with no resulting mentions when you ask the same AI tools directly is a different, narrower problem: the bots can reach you, they're just not finding a reason to cite you yet.

I ran a dog grooming salon for years before any of this existed, and I can tell you exactly how much bandwidth a business owner has left at the end of a day to parse a new analytics dashboard: not much. That's the actual argument against most small businesses buying a dedicated AI-traffic tool right now — not that the data is fake, but that a $29-and-up monthly subscription measuring a problem you haven't diagnosed yet is solving the wrong step first.

The mechanics: what to actually check

1. Find out if you can see your own server logs. Most standard hosting — cPanel, Plesk, most managed WordPress hosts — gives you raw access logs, or a summary tool like AWStats, at no extra cost. This is the same data a paid AI-traffic tool is packaging for you, just less organized.

2. Search those logs for the bot names that matter. ChatGPT-User (OpenAI), Perplexity-User (Perplexity), and Claude-User (Anthropic) are the ones tied to a real-time, on-demand question — start there. If your host's log tool supports a text search, searching each name one at a time takes a few minutes.

3. Check what's actually in your robots.txt before assuming a block. Visit yourdomain.com/robots.txt directly. We've covered the specific bot-by-bot breakdown of what to look for there before — a blanket Disallow: / with no exceptions is the most common way a site accidentally shuts every AI crawler out at once.

4. Only consider a paid tracking product once you know your baseline. OtterlyAI's entry-level plan starts around $29 a month; its higher tiers with broader agent coverage and API access run well into the hundreds. That's a real cost for a real business to weigh against what the data actually changes about what you'd do next — worth it if you're actively working AI visibility and want the trend line; overkill if you haven't checked your logs even once yet.

5. Recheck after any hosting, CDN, or firewall change. A new host, a security plugin, or a CDN's own default settings can quietly change what reaches your server in the first place, independent of anything in your own robots.txt.

What won't help

Buying the biggest tracking tier before you've looked at your own logs once. The free check — searching your existing access logs for bot names — answers the "am I getting crawled at all" question a $489-a-month enterprise plan is also answering, just less prettily.

Treating any bot visit as proof you're being recommended. A training-scraper visit means your content might inform a future model someday. An on-demand agent visit means a live question got asked and your page got fetched — closer to the outcome you actually want, but still not the same thing as a citation, and not the same thing as a recommendation. Traffic in the logs is a leading indicator, not a result.

Confusing crawl visibility with a vendor "visibility score." We've made this case before about proprietary AI visibility scores generally — a number without a published methodology can't tell you which page or setting is the problem. Bot-traffic data is more concrete than a black-box score, but it's still a diagnostic, not a grade.

Panicking about training-bot traffic specifically. Blocking GPTBot or similar training crawlers is a reasonable, low-stakes content-licensing decision if you'd rather your pages not train a future model. It has essentially no effect on whether the same company's search or agent bots can still find and cite you — those run on separate rules in the same file.

FAQ

What is "AI agent traffic"?

Visits to your website from AI systems fetching a page in response to a real, live user question — as opposed to a search-indexing crawl or a bulk content scrape used to train a future model. It's one of three categories AI bot traffic generally falls into.

How is this different from Google crawling my site?

The mechanism is similar — a bot identifies itself with a user-agent string and requests your pages — but the traffic is split further. AI companies run separate bots for search indexing, real-time question-answering, and model training, and a site can allow or block each independently.

Do I need a tool like OtterlyAI to check this, or can I do it myself?

You can check the basics yourself for free if your host gives you server log access, which most standard hosting does. A paid tool organizes that data, tracks it over time, and ties it to citation trends — useful once you're actively working AI visibility, unnecessary before you've looked at your logs even once.

What's the difference between a search bot, an agent bot, and a training bot?

A search bot builds an index, similar to how Googlebot works. An agent bot fetches a page right now because someone just asked an AI tool a live question. A training bot harvests content to help build a future model. All three can visit the same site and mean very different things.

If I have zero AI bot traffic, does that mean I'm blocked?

Not necessarily, but it's the first thing worth ruling out. Check your robots.txt file directly — a blanket disallow rule is a common accidental cause. If nothing there is blocking access, zero traffic can also just mean AI tools haven't yet found a reason to fetch your specific pages.

Does more AI bot traffic mean more customers?

Not directly, and not automatically. A visit from an on-demand agent bot means your page got fetched in response to a live question — it doesn't guarantee your business was the one actually named in the answer, or that the person asking became a customer. It's a signal worth watching, not a result to report on its own.

How often should I check this?

Treat it the same as any other technical setting that can drift without an announcement — a periodic check, not a one-time project. Hosting changes, CDN defaults, and firewall rules can all quietly change what reaches your server between checks.

---

Knowing whether AI tools can even reach your site — and what they find when they do — is exactly the kind of ongoing, unglamorous work that getting found on Google and AI is built around. Nobody emails you when a crawler starts or stops showing up. Something has to be watching for it anyway.