Google’s automated web crawler used to discover and retrieve web documents. Review the concept, business applications and implementation considerations.
Googlebot uses desktop and smartphone user agents. Under mobile-first indexing, the smartphone crawler handles most indexing-related crawling. A user-agent string alone does not prove a request is genuine: verify requests through Google’s documented IP or reverse-and-forward DNS checks, including relevant googlebot.com hostnames. This helps distinguish genuine crawling from scrapers that impersonate Googlebot. Server-log analysis reveals actual visit frequency and URL patterns and is important for diagnosing crawl-budget issues on large sites.
Some websites are seeing sharp increases in site: operator queries in Search Console performance reports. Explore possible causes, measurement effects and diagnostic responses.
