Robots Exclusion Standard
robots.txt
A root-level text file defining which website paths compliant crawlers may access.
A Markdown-based proposal that helps LLMs and AI agents quickly identify a website’s essential information and documentation. Explore its practical uses.
llms.txt
llms.txt is an emerging proposed convention that places summaries and links to important pages in a Markdown file at the site root. It is intended to help AI crawlers or agents understand key information without exploring an entire site indiscriminately. Official support from Google or OpenAI has not been confirmed in the source material, although adoption by some tools and communities has encouraged early implementation. Unlike robots.txt, which governs access permissions, llms.txt provides a complementary summary of what matters on a site.
Can support more efficient consumption of a site’s most important pages with fewer tokens.
Publish summaries and links to important pages in Markdown at /llms.txt.
robots.txt
A root-level text file defining which website paths compliant crawlers may access.
크롤링
The process through which search crawlers follow links, discover URLs and retrieve new or updated content.
AI 크롤러
A dedicated bot that collects web content for generative AI training data or real-time search-based answers.
LLM
A deep-learning neural network pretrained on extensive text data to understand and generate language in a humanlike form.
토큰
The smallest unit of text processed by an AI model, which may be a word, character or subword fragment.
AEO
A strategy for optimizing content so that brands and their content can be cited in the single answers generated by AI search systems or voice assistants.