Not every AI bot deserves the same answer from your website.
Source Note
This post uses Cloudflare's July 1 traffic taxonomy and its crawl-versus-referral framing, with TechRadar's invisible-traffic piece as supporting commentary. The argument is not to name one “good” bot list forever. Bot names and behaviours change too fast for that. It is to set a policy logic that still holds when the names change.
Three categories of bot
Before you decide anything, sort the traffic into three buckets. They look similar in a log file and mean completely different things for your business:
- Search and discovery crawlers. They index your pages so you can appear in search and AI answers later. This is the traffic that earns you future visibility. Googlebot is the classic example; AI-search crawlers increasingly sit here too.
- Real-time agent fetchers. They retrieve a specific page because a user just asked an assistant about you. This is a live, high-intent visit by proxy: someone is deciding something right now and your page is the evidence. Blocking it can remove you from the exact moment you wanted to win.
- Bulk training collectors. They gather content at scale to train models, with no direct path back to you. GPTBot and CCBot largely live here. This is the category where "what do I get in return?" is the honest question.
The mistake most guides make is treating these as one thing called "AI bots" and reaching for a single allow-or-block switch. The value gap between a real-time fetcher and a bulk training crawler is enormous, and a blanket rule taxes the useful behaviour to punish the extractive one.
Three questions to ask
- Does this behavior support discovery?
- Does it support a real user action?
- Does it mostly extract value without return?
Those questions are better than "is it AI?" because one bot may index for future search, another may help a user act in real time, and another may only scrape. The word "AI" tells you nothing about the tradeoff. The behaviour does.
A simple small-brand policy
You do not need a per-bot spreadsheet. A workable default for most small sites:
- Public, commercially important pages stay discoverable. Service pages, proof, pricing, the content you want cited. Allow search and agent access here without hesitation; this is where visibility converts.
- Sensitive, monetized, or transactional pages get stricter. Account areas, checkout, gated resources. There is no discovery upside and real downside to exposing them.
- Extractive behaviour with no traceable return gets questioned first. Bulk training crawlers are the honest place to draw a line if you want to draw one, because they are the category that takes without sending anything back.
The useful policy is not universal openness or universal blocking. It is selective access with visible business reasons, reviewed as the landscape shifts. Openness maximises reach and gives your work away; blocking protects the work and makes you invisible. The right point sits between them, per page type, and it moves over time.
Your crawl policy is now part of your brand strategy. Set it with the same discipline you use for pricing pages, proof assets, and conversion flows, because it decides whether the AI systems shaping discovery can see you at all. It is one of the first things I review when preparing a service site for agentic discovery, before any content or schema work, because a page a model cannot read cannot be cited no matter how good it is.