AI crawler & bot directory
Every crawler below is recognized automatically the moment a site installs the ShonyLabs tracking snippet — no configuration needed. Hits are classified by purpose, kept out of your human visitor counts, and surfaced separately on the dashboard's bot & AI traffic cards. Any single crawler can also be excluded per-site from Settings → Exclusions.
38 crawlers across 6 categories
AI answers
Live, user-triggered fetches while an AI assistant answers someone's question in real time - e.g. ChatGPT or Claude opening a page mid-conversation.
ChatGPT-UserOpenAI
Fetches a page on behalf of a user during a ChatGPT conversation.
Claude-UserAnthropic
Fetches a page on behalf of a user during a Claude conversation.
Meta-ExternalFetcherMeta
Live page fetch by Meta AI when a user's prompt references your URL directly.
Perplexity-UserPerplexity
Fetches a page on behalf of a user asking Perplexity.
AI search & indexing
An AI product's own retrieval crawler, building the index it searches to answer questions - separate from a traditional search engine.
Claude-SearchBotAnthropic
Powers search results shown to Claude users.
OAI-SearchBotOpenAI
Powers search results shown inside ChatGPT.
PerplexityBotPerplexity
Crawls and indexes content for Perplexity's answer engine.
YouBotYou.com
Crawls and indexes content for the You.com AI search engine.
AI training
Crawlers collecting public content that may be used to train future AI models.
AmazonbotAmazon
Crawls content for Amazon's Alexa and AI products.
Applebot-ExtendedApple
Controls use of content for Apple's generative AI features.
BytespiderByteDance
Crawls content, reportedly for ByteDance/TikTok AI model training.
CCBotCommon Crawl
Builds the open Common Crawl dataset, widely used to train LLMs.
ClaudeBotAnthropic
Crawls content to train Anthropic's Claude models.
cohere-aiCohere
Crawls content to train Cohere's language models.
DiffbotDiffbot
Extracts structured data for Diffbot's knowledge graph and AI products.
Google-ExtendedControls use of content to train Google's Gemini/Vertex AI models.
ImageSiftBotImageSift
Crawls images for ImageSift's visual-search / dataset index (operated by The Hive).
Meta-ExternalAgentMeta
Crawls content to train Meta's AI models.
TimpibotTimpi
Crawler for Timpi's decentralised web index used for AI training data.
Search engines
Traditional web crawlers that index pages for search results.
BaiduspiderBaidu
Baidu's primary web crawler.
GooglebotGoogle's primary web crawler for Search.
PetalBotHuawei
Crawler for Huawei's Petal Search engine.
SeznamBotSeznam
Crawler for Seznam.cz, the Czech search engine.
YandexBotYandex
Yandex's primary web crawler.
SEO tools
Third-party crawlers powering backlink and rank-tracking tools - not a search engine or AI product.
AhrefsBotAhrefs
Crawls the web to power Ahrefs' backlink/SEO index.
MJ12botMajestic
Crawls the web to build Majestic's backlink index.
SemrushBotSemrush
Crawls the web to power Semrush's SEO/marketing index.
Social & link previews
Fetches a page once to render a link-preview card when it's shared on a chat app or social platform.
DiscordbotDiscord
Generates link previews (embeds) when a URL is shared on Discord.
facebookexternalhitMeta
Generates link previews when a URL is shared on Facebook/Instagram.
LinkedInBotGenerates link previews when a URL is shared on LinkedIn.
TelegramBotTelegram
Generates link previews when a URL is shared on Telegram.
TwitterbotX (Twitter)
Generates link previews (cards) when a URL is shared on X.
WhatsAppMeta
Generates link previews when a URL is shared on WhatsApp.
Seeing a crawler that isn't listed?
Unrecognized bot traffic still gets flagged and tracked separately under "Other bot" — it just isn't broken out by name yet. Email [email protected] with the user agent and we'll add it.