Not every AI bot deserves the same answer from your website.
Source Note
This post uses Cloudflare's July 1 traffic taxonomy and its crawl-versus-referral framing, with TechRadar's invisible-traffic piece as supporting commentary. The argument is not to name one “good” bot list forever. It is to set a policy logic.
Three questions to ask
- Does this behavior support discovery?
- Does it support a real user action?
- Does it mostly extract value without return?
Those questions are better than “is it AI?” because one bot may index for future search, another may help a user act in real time, and another may only scrape.
A simple small-brand policy
If the page is public and commercially important, keep it discoverable. If the page is sensitive, monetized, or transactional, get stricter. If the behavior is extractive and you cannot trace value back, question why it should keep access.
Analysis: the useful policy is not universal openness or universal blocking. It is selective access with visible business reasons.
CTA: Your crawl policy is now part of your brand strategy. Set it with the same discipline you use for pricing pages, proof assets, and conversion flows.