Crawling
크롤링
The process through which search crawlers follow links, discover URLs and retrieve new or updated content.
robots.txt guides crawler access, not indexing or data security. Learn practical checks, rendering-resource access and its relationship with noindex.
robots.txt
robots.txt guides URL crawling. It does not replace indexing controls or protect confidential information. A blocked URL can still appear in search results when discovered through links.
Before restricting repetitive parameter URLs, distinguish unnecessary crawl requests from useful pages and resources. Protect administrative screens with server-side access controls such as authentication.
For search exclusion, consider a crawlable noindex directive or another suitable method. Avoid broadly blocking rendering resources. Check crawl logs and indexing separately; a rule does not guarantee crawl-budget savings or better rankings.
크롤링
The process through which search crawlers follow links, discover URLs and retrieve new or updated content.
색인
The process of analyzing retrieved web pages and storing their information in a search engine's index.
노인덱스
A meta tag or HTTP header directing supporting search engines to exclude a resource after crawling it.
robots meta tag
A page-level HTML meta tag controlling search indexing, link following and preview behavior.
크롤 버짓
The crawl capacity and demand that determine how many pages search engines fetch over time.