Robots Exclusion Standard
robots.txt
A way to guide cooperating crawlers with URL-access rules in a site-root robots.txt file.
noindex directs supporting search engines to exclude a crawled resource. Review examples, robots.txt conflicts, processing delays and access controls.
노인덱스
A noindex directive can be sent through a robots meta tag or X-Robots-Tag response header. Supporting search engines must read the response, so removal is not necessarily immediate.
For a public test page that does not need search visibility, check the directive and crawl access. Protect personal information and private documents with authentication and authorization.
Blocking the same URL in robots.txt can prevent detection of noindex. Inspect the actual HTML or headers and Search Console status. noindex and nofollow serve different purposes; explaining the directive does not require applying it to the explanation page.
robots.txt
A way to guide cooperating crawlers with URL-access rules in a site-root robots.txt file.
robots meta tag
A page-level HTML meta tag controlling search indexing, link following and preview behavior.
색인
The process of analyzing retrieved web pages and storing their information in a search engine's index.
크롤링
The process through which search crawlers follow links, discover URLs and retrieve new or updated content.
캐노니컬 태그
A rel=canonical annotation that signals the preferred URL among duplicate or very similar pages.