Technical SEO & Web Infrastructure

robots.txt

Reviewed on

robots.txt guides URL crawling. It does not replace indexing controls or protect confidential information. A blocked URL can still appear in search results when discovered through links.

Practical example#

Before restricting repetitive parameter URLs, distinguish unnecessary crawl requests from useful pages and resources. Protect administrative screens with server-side access controls such as authentication.

Implementation guidance and limitations#

For search exclusion, consider a crawlable noindex directive or another suitable method. Avoid broadly blocking rendering resources. Check crawl logs and indexing separately; a rule does not guarantee crawl-budget savings or better rankings.

Crawling

크롤링

The process through which search crawlers follow links, discover URLs and retrieve new or updated content.

Indexing

색인

The process of analyzing retrieved web pages and storing their information in a search engine's index.

noindex Directive

노인덱스

A meta tag or HTTP header directing supporting search engines to exclude a resource after crawling it.

Robots Meta Tag

robots meta tag

A page-level HTML meta tag controlling search indexing, link following and preview behavior.

Crawl Budget

크롤 버짓

The crawl capacity and demand that determine how many pages search engines fetch over time.