Verified bot traffic, impersonated-crawler detection, and a bot-traffic allowance
Bot traffic is now checked against each operator's published IP ranges, impersonated crawlers are flagged separately, and bot traffic gets its own per-cycle allowance with a self-serve add-on.
Why verify bot traffic
Every major crawler operator publishes the exact IP ranges its bots use — OpenAI, Google, Microsoft (Bing) and Perplexity all ship a machine-readable list. ShonyLabs now fetches those lists automatically and, for every incoming request that claims to be one of those crawlers, checks the connecting IP against them.
A request whose user-agent says "GPTBot" but which comes from an IP that isn't OpenAI's is almost always a scraper wearing a costume — to dodge blocklists, or to make its traffic look legitimate in your logs. Those hits are now labelled "Impersonated crawlers" on your site dashboard, kept out of the normal bot-traffic breakdowns, and never counted against your bot-traffic allowance.
The bot-traffic allowance
Bot and crawler hits have always been excluded from your billed event quota. They still are — but processing and storing them isn't free, and a small number of sites get an enormous amount of crawler traffic. Each account now gets a per-billing-cycle allowance of accepted bot requests across all its sites.
If you reach it, nothing about your real analytics changes: your dashboards stay available and human traffic keeps being tracked exactly as before. Only the recording of new bot traffic pauses until your cycle resets. If you want it to keep recording, you can add capacity from the Billing page — $9/month for each additional 1,000,000 bot requests, applied immediately.
What shipped
- NewBot traffic is now verified. When a request says it's GPTBot, Googlebot, Bingbot, PerplexityBot (and more), we check its IP against that operator's own published range list. Anything claiming to be a known crawler from an IP outside those ranges is shown separately under "Impersonated crawlers" and kept out of your real bot numbers.
- NewBot / crawler traffic now has its own allowance, separate from your event quota. If you reach it, your dashboard stays live and your normal analytics keeps tracking — only new bot traffic pauses until your cycle resets. You can add capacity any time from the Billing page ($9/mo per extra 1,000,000 requests).
- NewSettings → Exclusions has a new "Exclude crawlers" list — switch off any specific bot (for example a scraper hitting you hard) and it stops being recorded and stops counting toward your bot allowance. You can also delete anything it already recorded.
- ImprovedThe bot catalog gained more crawlers (PetalBot, SeznamBot, ImageSiftBot, Meta-ExternalFetcher, Timpibot) and each entry now tracks how it's verified.