Robots Exclusion Standard
robots.txt
A root-level text file defining which website paths compliant crawlers may access.
A directive instructing search engines to exclude a page from their index. Review the concept, business applications and implementation considerations.
노인덱스
A noindex page remains available to visitors but is excluded from search results once the directive is processed. Common uses include thank-you pages, internal search results and low-value tag or filter combinations. It is separate from nofollow, so links on the page can still be followed. Blocking the page in robots.txt prevents crawlers from reading noindex and may leave the URL indexed contrary to the site’s intention.
Keeps duplicate, test and low-value internal-search pages out of the index.
Allow crawling so search engines can discover and process noindex.
robots.txt
A root-level text file defining which website paths compliant crawlers may access.
색인
The process of analyzing retrieved web pages and storing their information in a search engine's index.
크롤링
The process through which search crawlers follow links, discover URLs and retrieve new or updated content.
내부 링크
A hyperlink connecting one webpage to another within the same domain.
301 리디렉션
An HTTP status code indicating a permanent move to a new URL and supporting the transfer of link equity.
302 리다이렉트
An HTTP response code indicating that a resource is temporarily available at another URL.