Websites record requests from automated programs as well as visits from people. These programs include crawlers that build search indexes, bots that collect information for AI answers, and tools that act on a user’s request. Which bots access content, for what purpose, and what do their requests mean?
247COMPASS’s Cloudflare data, connected to Microsoft Clarity, recorded 72,578 bot requests from November 4–10, 2026. The observed activity included search crawling, content collection that may support model development, user-requested page fetching, and advertising-page checks.
This article retains Clarity’s classifications and compares the recorded counts with each bot’s officially documented role. It examines request distribution and differences in function, distinguishing observations supported by the logs from citations, visits, and conversions that require additional evidence.
Scope: The data covers November 4–10, 2026, and the official reference review date is November 11, 2026. The denominator for the percentages below is 72,578 bot requests during this period, rather than all site traffic including people.
Jump to the classification and role table for all 93 bots →
First, what do these numbers count?#
Clarity’s Bot Activity uses a different measurement unit from session recordings and heatmaps. It counts bot requests using logs supplied by a CDN or server . With the Cloudflare integration, LogPush sends HTTP request logs to Clarity .
One request does not equal one person or one visit. A bot can request the same page repeatedly or fetch images, JSON, XML, and other resources alongside the page text. A request alone does not establish successful content delivery. Clarity distinguishes successful, redirected, and failed requests , but this dataset does not include counts by response status.
The direct meaning of 72,578 is therefore the volume of recorded bot access attempts. It cannot be converted into the number of AI-answer citations or human visits referred by AI services.
Seven days of bot requests in Clarity’s categories#
AI Crawler was the largest category, with 32,924 requests and a 45.36% share. AI Search followed with 14,925 requests, and AI Assistant with 12,637. Together, the three categories accounted for 60,486 requests, or 83.34% of all bot requests.
The charts and tables retain Clarity’s category names and membership. GoogleBot and BingBot are included in AI Crawler, while OAI-SearchBot is in Search Engine Crawler. Official role descriptions are compared separately and are not used to recalculate the category totals.
| Clarity category | Requests | Share of all bot requests |
|---|---|---|
| AI Crawler | 32,924 | 45.36% |
| AI Search | 14,925 | 20.56% |
| AI Assistant | 12,637 | 17.41% |
| Search Engine Crawler | 7,011 | 9.66% |
| Search Engine Optimization | 2,550 | 3.51% |
| Advertising & Marketing | 2,070 | 2.85% |
| Automation & Tooling | 295 | 0.41% |
| Monitoring & Analytics | 83 | 0.11% |
| Page Preview | 54 | 0.07% |
| Security | 21 | 0.03% |
| Academic Research | 5 | Under 0.01% |
| Aggregator | 3 | Under 0.01% |
| Total | 72,578 | 100% |
83.34% is the combined share of the three categories with AI in their names. It does not mean that AI accounted for 83.34% of site visits or that 83.34% of the site’s content was used for AI training. A category groups records; the ultimate use of an individual request requires separate verification.
Different roles within one operator: the OpenAI example#
OpenAI was the largest operator label in the dataset, with 8,285 requests across four bots. OpenAI’s official crawler documentation distinguishes search, model development, user requests, and advertising checks.
| Original bot name | Requests | Clarity category | Officially documented role |
|---|---|---|---|
| OAI-SearchBot | 2,125 | Search Engine Crawler | Collection for ChatGPT search results |
| ChatGPT-User | 2,101 | AI Assistant | Fetching a page for a specific user-requested action |
| OAI-AdsBot | 2,067 | Advertising & Marketing | Checking the safety and relevance of pages submitted for ChatGPT advertising |
| GPTBot | 1,992 | AI Crawler | Collecting content that may be used to train foundation models |
The role determines what needs checking. OAI-SearchBot relates to the search-crawling route, ChatGPT-User to access on a user’s behalf, and GPTBot to policies governing training-related content use. OpenAI describes independent controls for its search and training bots.
OAI-AdsBot’s stated purpose is to check pages submitted as advertisements; its collected data is not used to train foundation models. Its 2,067 requests should be distinguished from search-visibility outcomes. The counts alone cannot establish whether 247COMPASS ran advertising. Submission history, request paths, and verification status would need to be compared.
OpenAI’s total request count cannot explain these four functions. Adding bot-specific roles to an operator total makes it possible to distinguish which function the recorded access relates to.
Role distinctions observed at Anthropic and Perplexity#
Anthropic recorded 6,225 requests, and Perplexity 4,194. Anthropic distinguishes ClaudeBot, Claude-SearchBot, and Claude-User. Perplexity distinguishes its search crawler PerplexityBot from the user-requested Perplexity-User.
| Original bot name | Requests | Clarity category | Role description |
|---|---|---|---|
| Claude | 2,130 | AI Crawler | Official ClaudeBot collects material for potential model training. The source display name requires comparison with the full User-Agent |
| Claude-SearchBot | 2,010 | AI Search | Collection to improve Claude search quality |
| Claude-User | 2,085 | AI Assistant | Fetching pages in response to user requests |
| PerplexityBot | 2,182 | AI Search | Collection for Perplexity search; official documentation distinguishes it from foundation-model training collection |
| Perplexity-User | 2,012 | AI Assistant | Fetching pages in response to user actions |
ChatGPT-User, Claude-User, and Perplexity-User together accounted for 6,198 requests, or 8.54% of the total. This records access by bots associated with user requests. A person can generate multiple requests, and operator testing may also be included. The total does not represent 6,198 readers or an equivalent number of unsolicited recommendations.
Observed activity from search, AI search, and preview bots#
Among search crawlers, GoogleBot recorded 5,047 requests, the most of any individual bot, and BingBot recorded 2,792. Their combined 7,839 requests represented 10.80% of the total.
Clarity places both bots in AI Crawler. Separately, Google documents Googlebot’s search-crawling role , and Bing describes Bingbot’s index-crawling role . Access for a search index should be distinguished from evidence of a citation in a particular AI answer.
NAVER’s Yeti recorded 351 requests, classified by Clarity as Search Engine Crawler. NAVER requests appeared only in this category during the period. NAVER’s official guidance describes Yeti as a robot that collects web documents for search.
The following table compares the roles of major bots observed across search, AI, and public-data collection routes.
| Original bot name | Requests | Clarity category | Officially described role |
|---|---|---|---|
| GoogleOther | 988 | AI Crawler | May support general-purpose crawling and research and development by Google product teams |
| Amazonbot | 2,419 | AI Crawler | Broad purposes including Amazon service improvement and model training |
| ShapBot | 2,378 | AI Search | Building a search index for Parallel web APIs |
| Applebot | 2,317 | AI Search | Supporting Apple search experiences; collected data may also be used for AI training and answer context |
| DuckAssistbot | 2,074 | AI Assistant | Collecting information for AI-assisted search answers; official documentation states it is not used for model training |
| CCBot | 2,032 | AI Search | Collection for the Common Crawl public web-crawl archive |
| KimiBot | 2,124 | AI Crawler | Collecting content that may be used to train Kimi foundation models |
| Kimi-SearchBot | 1,941 | AI Search | Collection for the Kimi search index |
| Yeti (NAVER) | 351 | Search Engine Crawler | Collecting web documents for NAVER search; recorded only as Search Engine Crawler during this period |
Sources: Google , Amazon , Parallel , Apple , DuckDuckGo , Common Crawl , Kimi , NAVER .
AI Search is Clarity’s classification for CCBot; collecting material for a public web-crawl archive is the operator’s stated role. These request records do not reveal which models or services later used the collected material. Some official descriptions, such as Applebot’s, cover multiple uses, so one bot should not automatically be assigned a single purpose.
Official documentation describes Google-Extended and Applebot-Extended as controls over the use of collected data through robots.txt. They are not separate web crawlers. Their absence from the observed list does not establish whether the related data uses occurred.
Meta: collection, user requests, search, and previews#
Meta’s official documentation distinguishes collection, user requests, AI search, and link previews.
| Original bot name | Requests | Clarity category | Role described by Meta |
|---|---|---|---|
| Meta-ExternalAgent | 2,510 | AI Crawler | Collection for foundation-model training, product indexing, and other uses |
| Meta-ExternalFetcher | 2,007 | AI Assistant | Fetching individual links requested by users |
| FacebookExternalHit | 37 | Page Preview | Fetching titles, descriptions, and thumbnails for shared links |
| Meta-WebIndexer | 16 | Page Preview | Supporting Meta AI search quality, citations, and links |
Meta-WebIndexer retains its original Page Preview classification. Its officially described AI-search role appears in a separate column. Category totals preserve the supplied data, while functional descriptions are compared against operator documentation.
What 247COMPASS’s seven-day dataset establishes#
Request distribution and concentration by operator label#
OpenAI, Google, Anthropic, Meta, and Perplexity were the top five operator labels. Their combined 29,599 requests represented 40.78% of all bot requests; the other 58 operator labels accounted for 59.22%.
The source labels include general-purpose tools and libraries alongside company names. Related services can also appear under different labels, as with Moonshot AI and Kimi. The 63 labels should therefore not be counted as 63 independent companies or AI services.
| Operator label | Requests | Share of all bot requests |
|---|---|---|
| OpenAI | 8,285 | 11.42% |
| 6,325 | 8.71% | |
| Anthropic | 6,225 | 8.58% |
| Meta | 4,570 | 6.30% |
| Perplexity | 4,194 | 5.78% |
| Other 58 operator labels | 42,979 | 59.22% |
Recorded requests are distributed across many operator labels, rather than concentrated entirely in one or two. Operational checks should compare requested URLs and response status across major search and AI routes. These shares do not measure platform market share or preference for 247COMPASS.
Distinguishing training, search, and user-requested access#
Different bot roles call for different operational checks:
- Search access: whether public pages can be crawled, and error responses on important pages.
- Training-related access: content-use policies and operator-specific controls.
- User-requested access: availability of the requested information and the behavior of access controls.
- Preview, SEO audit, and automation tools: accuracy of shared information and the operational impact of repeated requests.
ClarityBot/, classified as Search Engine Optimization, recorded 850 requests. seoClarity’s documentation describes a bot that performs customer-requested site audits; it is separate from Microsoft Clarity’s analytics tool. Semrush bots support backlink graphs, site audits, and page checks, while Majestic’s MJ12Bot collects links between sites. These records do not identify who commissioned an analysis.
General-purpose tool names such as curl, python-requests, and Go-http-client do not identify the actual operator or purpose. A name alone also cannot establish whether a request is useful or malicious. Where sender-verification methods are available, such as Google’s crawler verification guidance
or Bing’s verification tool
, they can be compared with the actual logs.
A single-period distribution and a trend are different findings#
The dataset contains one seven-day aggregate. Without a previous-week comparison or daily counts, it cannot verify claims that AI bots surged, that crawling increased after a new article, or that AI bots replaced search crawlers.
Without requested URLs or language breakdowns, collection volume cannot be attributed to individual articles. Without status codes and response sizes, collection success rates and server load cannot be calculated. Similar request counts across bots do not establish equal demand. Observed access distribution should be distinguished from trends and outcomes that require additional observation.
After the request: responses, citations, and visits#
247COMPASS’s seven-day record observes content access associated with operator and service functions. It includes automated collection and bot access that fulfills user requests. Whether those requests resulted in content delivery, AI-answer citations, or human visits must be checked using separate evidence.
Bot logs can help investigate errors in the content access and delivery process relevant to GEO. Analytics data classified as user traffic observes the content visitors reach and the choices and actions that follow. Treating direct verification of AI-answer citations as a separate step clarifies what each dataset measures.
1. Content access by a service
Bot traffic · Cloudflare → Clarity
Which bot requested the content, and was delivery successful?
- Operator purposes: collection, search, and handling user requests
- Requesting bot, URL, and timestamp
- Response status, blocks, and repeated requests
Check errors along the content delivery route
2. Citations in AI answers
Direct observation of actual answers and sources
Which content appeared as a source in an actual answer?
- Source URL displayed in the answer
- Query, service, and observation date
- Citation context and link format
Verify actual citations and their context
3. User selection and visits
User traffic · Analytics
Which content did visitors reach, and what did they do next?
- Visited pages and identifiable referral sources
- Sessions and configured behavior events
- Measured conversions such as inquiries and downloads
Observe selected content and behavior after arrival
Bot requests, citations, and user visits are different metrics. Bot requests alone do not establish citations, and Analytics visits alone do not verify an actual citation. Compare identifiable AI-service referrals with the sources shown in actual answers.
Referral sources and landing pages in Analytics are distinguished using the definitions in Google’s traffic acquisition report and landing page report . Actual citations require direct observation and comparison of the answer and its sources.
Requested URLs and response status would allow checks of where major bots encountered errors or access restrictions. These detailed logs are absent from the seven-day aggregate, so this article does not calculate an error rate or assert that delivery failed on a particular page. It likewise does not estimate separate Analytics visit or conversion counts.
This also relates to the difference between AI-search visibility and actual referrals . Bot statistics describe routes of content access. Evaluating what information reached readers and what they did requires additional observation of cited pages, referral sources, and behavior after arrival.
Data and interpretation criteria#
- Source: bot operator and activity data from 247COMPASS’s Cloudflare integration with Microsoft Clarity.
- Period: November 4–10, 2026.
- Official reference review date: November 11, 2026.
- Coverage: 93 bot names, 63 operator labels, and 72,578 requests.
- Percentage denominator: all 72,578 bot requests. Operator and individual-bot totals are not double-counted; percentages are rounded to two decimal places.
- Role evidence: official operator documentation. Uncertain name mappings and roles are identified separately in the appendix.
- Outside the dataset: individual response outcomes, actual AI-answer citations, and user visits or conversions.
Appendix: observed bots, classifications, and roles#
The directory compares bot request counts, Clarity categories, official roles, and the supporting evidence.
Classifications and roles of all 93 botsRequests by bot · Clarity category · Official roles and sources Open the full directoryClose the directory
| Bot name / Operator label | Requests | Clarity category | Role description and evidence |
|---|---|---|---|
| OAI-SearchBot OpenAI | 2,125 | Search Engine Crawler | Collection for ChatGPT search results. Requests alone do not establish citations or visibility. OpenAI Crawlers · Official description and display-name comparison |
| ChatGPT-User OpenAI | 2,101 | AI Assistant | Page fetching for a specific user request. It is not for automated web crawling and does not count users. OpenAI Crawlers · Official description and display-name comparison |
| OAI-AdsBot OpenAI | 2,067 | Advertising & Marketing | Safety and relevance checks on pages submitted for ChatGPT advertising. Not for foundation-model training. Submission history, IPs, and paths require comparison. OpenAI Crawlers · Official description and display-name comparison |
| GPTBot OpenAI | 1,992 | AI Crawler | Collecting content that may be used for foundation-model training. Inclusion of any individual request in training is unverified. OpenAI Crawlers · Official description and display-name comparison |
| GoogleBot | 5,047 | AI Crawler | Crawler for Google Search. The original AI Crawler classification is retained; these requests are not established as training-related. Google common crawlers · Official description and display-name comparison |
| GoogleOther | 988 | AI Crawler | May support general-purpose collection and internal research and development by Google product teams. The purpose of individual requests is unverified. Google common crawlers · Official description and display-name comparison |
| Chrome-Lighthouse | 177 | Search Engine Optimization | Classified as SEO in the source. Bot-specific audit, backlink, and other operations have not yet been checked against operator documentation in this research. Classification in supplied data · Detailed official role not yet verified |
| Google-Agent | 78 | AI Assistant | Classified as AI Assistant in the source. The name alone does not establish an individual user request or a person count. Classification in supplied data · Detailed official role not yet verified |
| Google Images | 22 | Search Engine Crawler | Display name associated with Google image search. Mapping to the official Googlebot-Image identifier requires comparison of the full User-Agent. Google common crawlers · Official description and display-name comparison |
| HeadlessChrome | 10 | Automation & Tooling | Classified as automation and tooling in the source. A general-purpose client name alone cannot identify the actual operator or purpose. Classification in supplied data · Detailed official role not yet verified |
| Google-Apps-Script | 1 | Search Engine Optimization | Classified as SEO in the source. Bot-specific audit, backlink, and other operations have not yet been checked against operator documentation in this research. Classification in supplied data · Detailed official role not yet verified |
| Google Site Verification | 1 | Security | Classified as security in the source. The record does not establish a vulnerability or an attack. Classification in supplied data · Detailed official role not yet verified |
| Google Image Proxy | 1 | Page Preview | Classified as page preview in the source. Exact behavior requires further comparison with official documentation. Classification in supplied data · Detailed official role not yet verified |
| Claude Anthropic | 2,130 | AI Crawler | Official documentation describes ClaudeBot as collecting for potential model training. Mapping the source label Claude to the exact User-Agent needs verification. Anthropic crawler controls · Official description and display-name comparison |
| Claude-User Anthropic | 2,085 | AI Assistant | Page fetching in response to user requests. Cannot be converted into numbers of people or recommendations. Anthropic crawler controls · Official description and display-name comparison |
| Claude-SearchBot Anthropic | 2,010 | AI Search | Collection to improve Claude search quality. Anthropic crawler controls · Official description and display-name comparison |
| Meta-ExternalAgent Meta | 2,510 | AI Crawler | Collection for foundation-model training, product indexing, and other purposes. Not restricted to a single use. Meta Web Crawlers · Official description and display-name comparison |
| Meta-ExternalFetcher Meta | 2,007 | AI Assistant | Fetches individual user-requested links to support product features. Meta Web Crawlers · Official description and display-name comparison |
| FacebookExternalHit Meta | 37 | Page Preview | Preview fetching of titles, descriptions, and thumbnails for links shared in Meta apps. Meta Web Crawlers · Official description and display-name comparison |
| Meta-WebIndexer Meta | 16 | Page Preview | Supports Meta AI search quality, citations, and links. The original Page Preview category is not reassigned to AI Search. Meta Web Crawlers · Official description and display-name comparison |
| PerplexityBot Perplexity | 2,182 | AI Search | Collection for Perplexity search, distinguished from collection for foundation-model training. Perplexity crawlers · Official description and display-name comparison |
| Perplexity-User Perplexity | 2,012 | AI Assistant | Fetching triggered by user actions. Not described as automated crawling or training collection. Perplexity crawlers · Official description and display-name comparison |
| KimiBot Moonshot AI | 2,124 | AI Crawler | Collecting content that may be used to train Kimi foundation models. Kimi Crawlers · Official description and display-name comparison |
| Kimi-SearchBot Moonshot AI | 1,941 | AI Search | Collection for the Kimi search index. Kimi Crawlers · Official description and display-name comparison |
| BingBot Microsoft | 2,792 | AI Crawler | Collection of new and updated pages for the Bing index. Records alone do not establish Copilot citations. Bing crawlers · Official description and display-name comparison |
| Amazonbot Amazon | 2,419 | AI Crawler | May support Amazon service improvement, accurate information, and model training. Not to be confused with the separate Amzn-SearchBot or Amzn-User. Amazon web search crawlers · Official description and display-name comparison |
| ShapBot Parallel | 2,378 | AI Search | Builds a search index for Parallel web APIs. Signature and IP verification methods are published. Parallel ShapBot · Official description and display-name comparison |
| Applebot Apple | 2,317 | AI Search | Supports search experiences including Spotlight, Siri, and Safari. Data may be used for AI training and answer context; individual request purposes remain unverified. About Applebot · Official description and display-name comparison |
| ChatGLM-Spider Other Operators | 2,027 | AI Crawler | Operator and category labels are retained from the source platform. This research did not find first-party role and control documentation for this exact name. Inclusion in training cannot be established. Cloudflare bot reference · Official description and display-name comparison |
| Operator Other Operators | 203 | AI Assistant | Classified as AI Assistant in the source. The name alone does not establish an individual user request or a person count. Classification in supplied data · Detailed official role not yet verified |
| ExaUser Other Operators | 39 | AI Assistant | Classified as AI Assistant in the source. The name alone does not establish an individual user request or a person count. Classification in supplied data · Detailed official role not yet verified |
| Google-NotebookLM Other Operators | 20 | AI Assistant | Classified as AI Assistant in the source. The name alone does not establish an individual user request or a person count. Classification in supplied data · Detailed official role not yet verified |
| Baiduspider Baidu | 2,097 | Search Engine Crawler | Classified as search collection in the source. The service-specific official purpose and User-Agent verification require further checking. Classification in supplied data · Detailed official role not yet verified |
| DuckAssistbot DuckDuckGo | 2,074 | AI Assistant | Real-time information collection for AI-assisted search answers, not model training. The original AI Assistant category is retained. DuckAssistBot · Official description and display-name comparison |
| DuckDuckBot DuckDuckGo | 14 | Search Engine Crawler | Collection for DuckDuckGo search, distinct from DuckAssistBot for AI answers. DuckDuckBot · Official description and display-name comparison |
| Hunyuan Tencent | 2,076 | AI Crawler | Official HySpider is described as potential training collection, but its mapping to the source label Hunyuan has not been verified. Tencent Hunyuan crawler · Official description and display-name comparison |
| YouBot You.com | 2,074 | Search Engine Crawler | Classified as search collection in the source. The service-specific official purpose and User-Agent verification require further checking. Classification in supplied data · Detailed official role not yet verified |
| MoonshotBot Kimi | 2,049 | AI Crawler | The source operator label is Kimi. The official Kimi crawler documentation did not verify the role or mapping of this exact name. It is not merged with KimiBot. Kimi Crawlers · Official description and display-name comparison |
| CCBot Common Crawl Foundation | 2,032 | AI Search | Collection for the Common Crawl public web-crawl archive. The original AI Search category is retained. This does not prove specific model training or direct citations. Common Crawl CCBot · Official description and display-name comparison |
| QwenBot Alibaba | 2,024 | AI Crawler | Operator and category labels are retained from the source platform. This research did not find first-party role and control documentation for this exact name. Inclusion in training cannot be established. Cloudflare bot reference · Official description and display-name comparison |
| cohere-ai Cohere | 2,017 | AI Assistant | Current official documentation states that Cohere does not crawl with Cohere bots for foundation-model training. The display name alone does not establish the actual role. Cohere web crawlers · Official description and display-name comparison |
| DeepSeekBot DeepSeek | 2,006 | AI Crawler | Official material describes DeepseekBot collection for model training and source cross-checking. Full User-Agent and IP authentication need further verification. DeepSeek training-content summary · Official description and display-name comparison |
| Bravebot Brave Software, Inc. | 2,002 | AI Search | Official documentation states that the search crawler does not use a dedicated identifying User-Agent. The actual sender and role behind this label need verification. Brave Search crawler · Official description and display-name comparison |
| Bytespider ByteDance | 2,000 | AI Crawler | Operator and category labels are retained from the source platform. This research did not find first-party role and control documentation for this exact name. Inclusion in training cannot be established. Cloudflare bot reference · Official description and display-name comparison |
| ClarityBot/ Seoclarity | 850 | Search Engine Optimization | Page and site audits requested or scheduled by seoClarity customers, separate from Microsoft Clarity analytics. This table does not identify the requesting party. seoClarity ClarityBot · Official description and display-name comparison |
| SemrushBotBacklinks Semrush | 774 | Search Engine Optimization | Official SemrushBot collects a link graph for backlink analysis. The source combined display name needs comparison with the exact User-Agent. Semrush Bot · Official description and display-name comparison |
| SemrushBotBacklinkAudit Semrush | 8 | Search Engine Optimization | Official SemrushBot-BA serves the Backlink Audit tool. The source display name and exact User-Agent need comparison. Semrush Bot · Official description and display-name comparison |
| SiteAuditBot Semrush | 3 | Search Engine Optimization | Site audit bot for identifying technical and SEO issues. Semrush Bot · Official description and display-name comparison |
| SemrushBotSI Semrush | 3 | Search Engine Optimization | Official SemrushBot-SI supports tools including On Page SEO Checker. Mapping to the source display name needs comparison. Semrush Bot · Official description and display-name comparison |
| PetalBot Huawei | 635 | AI Crawler | Official search excerpts describe Petal Search indexing, Huawei recommendations, and AI Search support. The complete official text was not accessible. The original AI Crawler category is retained. PetalBot (Official search excerpt only) · Official description and display-name comparison |
| Go-http-client Golang | 219 | Automation & Tooling | Classified as automation and tooling in the source. A general-purpose client name alone cannot identify the actual operator or purpose. Classification in supplied data · Detailed official role not yet verified |
| YandexBot Yandex | 216 | Search Engine Crawler | Crawler for the main Yandex search index. Yandex robots verification · Official description and display-name comparison |
| SERankingBacklinksBot SE Ranking | 133 | Search Engine Optimization | Collects public links to support link-structure analysis. SE Ranking Backlinks crawler · Official description and display-name comparison |
| SE Ranking Bot SE Ranking | 18 | Monitoring & Analytics | Official SEBot-WA supports site audits and On-Page SEO checks. Mapping to this display name needs verification. The original Monitoring & Analytics category is retained. SE Ranking crawler FAQ · Official description and display-name comparison |
| DataForSEO Bot DataForSEO | 139 | Search Engine Optimization | Stores discovered links for backlink analysis. The official DataForSeoBot User-Agent needs comparison with the source display name. DataForSEO Link Bot · Official description and display-name comparison |
| DataForSEO DataForSEO | 3 | Search Engine Optimization | Official documentation describes DataForSEO backlink collection, but the exact User-Agent and role of this standalone display name need verification. DataForSEO Link Bot · Official description and display-name comparison |
| MJ12Bot Majestic | 133 | Search Engine Optimization | Collects links between sites to build the Majestic backlink index. Majestic MJ12Bot · Official description and display-name comparison |
| AhrefsBot Ahrefs | 102 | Search Engine Optimization | Official search excerpts describe recording external links in a database and recrawling periodically. Only excerpts were accessible; the official full text was restricted. AhrefsBot (Official search excerpt only) · Official description and display-name comparison |
| Barkrowler Babbar | 96 | Search Engine Optimization | Classified as SEO in the source. Bot-specific audit, backlink, and other operations have not yet been checked against operator documentation in this research. Classification in supplied data · Detailed official role not yet verified |
| Chrome Privacy Preserving Prefetch Proxy Datadome | 88 | Search Engine Optimization | Classified as SEO in the source. Bot-specific audit, backlink, and other operations have not yet been checked against operator documentation in this research. Classification in supplied data · Detailed official role not yet verified |
| NetAPI Datadome | 4 | Search Engine Optimization | Classified as SEO in the source. Bot-specific audit, backlink, and other operations have not yet been checked against operator documentation in this research. Classification in supplied data · Detailed official role not yet verified |
| HappyWing Datadome | 3 | Search Engine Optimization | Classified as SEO in the source. Bot-specific audit, backlink, and other operations have not yet been checked against operator documentation in this research. Classification in supplied data · Detailed official role not yet verified |
| Lightpanda Grok-adjacent | 85 | AI Crawler | Classified as AI Crawler in the source. The exact bot-specific role and ultimate use require further official documentation and original-request verification. Classification in supplied data · Detailed official role not yet verified |
| nbot NBot | 63 | AI Search | Classified as AI Search in the source. Exact indexing and answer-use behavior remain unverified. Classification in supplied data · Detailed official role not yet verified |
| KeenableBot Keenable | 41 | Search Engine Crawler | Classified as search collection in the source. The service-specific official purpose and User-Agent verification require further checking. Classification in supplied data · Detailed official role not yet verified |
| Keenable-User Keenable | 1 | AI Assistant | Classified as AI Assistant in the source. The name alone does not establish an individual user request or a person count. Classification in supplied data · Detailed official role not yet verified |
| HubSpot Crawler HubSpot | 37 | Monitoring & Analytics | Classified as monitoring and analytics in the source. The detailed request purpose remains unverified. Classification in supplied data · Detailed official role not yet verified |
| curl Haxx | 34 | Automation & Tooling | Classified as automation and tooling in the source. A general-purpose client name alone cannot identify the actual operator or purpose. Classification in supplied data · Detailed official role not yet verified |
| ibou-search ibou | 27 | Search Engine Crawler | Classified as search collection in the source. The service-specific official purpose and User-Agent verification require further checking. Classification in supplied data · Detailed official role not yet verified |
| HaloBot Haloscan | 26 | Monitoring & Analytics | Classified as monitoring and analytics in the source. The detailed request purpose remains unverified. Classification in supplied data · Detailed official role not yet verified |
| Moz dotbot Moz | 24 | Search Engine Optimization | Classified as SEO in the source. Bot-specific audit, backlink, and other operations have not yet been checked against operator documentation in this research. Classification in supplied data · Detailed official role not yet verified |
| CensysInspect/ Censys | 20 | Security | Classified as security in the source. The record does not establish a vulnerability or an attack. Classification in supplied data · Detailed official role not yet verified |
| Reflectionbot Reflection AI | 20 | AI Crawler | Classified as AI Crawler in the source. The exact bot-specific role and ultimate use require further official documentation and original-request verification. Classification in supplied data · Detailed official role not yet verified |
| python-requests python-requests | 13 | Automation & Tooling | Classified as automation and tooling in the source. A general-purpose client name alone cannot identify the actual operator or purpose. Classification in supplied data · Detailed official role not yet verified |
| Cốc Cốc Coccoc | 12 | Search Engine Crawler | Classified as search collection in the source. The service-specific official purpose and User-Agent verification require further checking. Classification in supplied data · Detailed official role not yet verified |
| Qwantbot Qwant | 11 | Search Engine Crawler | Classified as search collection in the source. The service-specific official purpose and User-Agent verification require further checking. Classification in supplied data · Detailed official role not yet verified |
| SofyaBot Sofya | 9 | Search Engine Crawler | Classified as search collection in the source. The service-specific official purpose and User-Agent verification require further checking. Classification in supplied data · Detailed official role not yet verified |
| ALittle Client Udger | 9 | Automation & Tooling | Classified as automation and tooling in the source. A general-purpose client name alone cannot identify the actual operator or purpose. Classification in supplied data · Detailed official role not yet verified |
| ZoomInfo ZoomInfo | 8 | Search Engine Crawler | Classified as search collection in the source. The service-specific official purpose and User-Agent verification require further checking. Classification in supplied data · Detailed official role not yet verified |
| ICBot/ Ideasandcode | 5 | Search Engine Optimization | Classified as SEO in the source. Bot-specific audit, backlink, and other operations have not yet been checked against operator documentation in this research. Classification in supplied data · Detailed official role not yet verified |
| TurnitinBot Turnitin | 4 | Academic Research | Classified as academic research in the source. The specific research project and actual use remain unverified. Classification in supplied data · Detailed official role not yet verified |
| YisouSpider Yisou | 4 | Search Engine Crawler | Classified as search collection in the source. The service-specific official purpose and User-Agent verification require further checking. Classification in supplied data · Detailed official role not yet verified |
| httpx Python-httpx | 4 | Automation & Tooling | Classified as automation and tooling in the source. A general-purpose client name alone cannot identify the actual operator or purpose. Classification in supplied data · Detailed official role not yet verified |
| GeedoShopProductFinder Geedo SIA | 3 | Aggregator | Classified as an aggregator in the source. The exact collection purpose remains unverified. Classification in supplied data · Detailed official role not yet verified |
| BrightEdge Bot BrightEdge | 3 | Advertising & Marketing | Classified as advertising and marketing in the source. This does not establish an advertising campaign or its performance. Classification in supplied data · Detailed official role not yet verified |
| Python-urllib Python-urllib | 3 | Automation & Tooling | Classified as automation and tooling in the source. A general-purpose client name alone cannot identify the actual operator or purpose. Classification in supplied data · Detailed official role not yet verified |
| okhttp okhttp | 2 | Automation & Tooling | Classified as automation and tooling in the source. A general-purpose client name alone cannot identify the actual operator or purpose. Classification in supplied data · Detailed official role not yet verified |
| Wordpress Namecheap Namecheap | 2 | Monitoring & Analytics | Classified as monitoring and analytics in the source. The detailed request purpose remains unverified. Classification in supplied data · Detailed official role not yet verified |
| Observer Obsrvr | 2 | Search Engine Optimization | Classified as SEO in the source. Bot-specific audit, backlink, and other operations have not yet been checked against operator documentation in this research. Classification in supplied data · Detailed official role not yet verified |
| SeobilityBot Seobility | 2 | Search Engine Optimization | Classified as SEO in the source. Bot-specific audit, backlink, and other operations have not yet been checked against operator documentation in this research. Classification in supplied data · Detailed official role not yet verified |
| trafilatura Github | 1 | Automation & Tooling | Classified as automation and tooling in the source. A general-purpose client name alone cannot identify the actual operator or purpose. Classification in supplied data · Detailed official role not yet verified |
| ResearchBot Kaust | 1 | Academic Research | Classified as academic research in the source. The specific research project and actual use remain unverified. Classification in supplied data · Detailed official role not yet verified |
| Yeti NAVER | 351 | Search Engine Crawler | Collection of web documents for NAVER search. All 351 NAVER requests in this period were recorded only in Search Engine Crawler. Clarity counts · Official NAVER guidance |
The 247COMPASS perspective#
AI citations and search visibility start with a technical foundation that allows bots to access and collect our content. After a bot’s activity, actual visibility and citations still depend on each platform’s indexing, algorithms, and policies.
This is why Technical SEO matters. Content sits on that technical foundation. Think of a pizza: Technical is the dough, and content is the topping.
Is your dough ready for great toppings?



