Robots Exclusion Standard
robots.txt
A way to guide cooperating crawlers with URL-access rules in a site-root robots.txt file.
A Markdown-based proposal that helps LLMs and AI agents quickly identify a website’s essential information and documentation. Explore its practical uses.
llms.txt
llms.txt is an emerging proposed convention that places summaries and links to important pages in a Markdown file at the site root. It is intended to help AI crawlers or agents understand key information without exploring an entire site indiscriminately. Official support from Google or OpenAI has not been confirmed in the source material, although adoption by some tools and communities has encouraged early implementation. Unlike robots.txt, which governs access permissions, llms.txt provides a complementary summary of what matters on a site.
Can support more efficient consumption of a site’s most important pages with fewer tokens.
Publish summaries and links to important pages in Markdown at /llms.txt.
robots.txt
A way to guide cooperating crawlers with URL-access rules in a site-root robots.txt file.
크롤링
The process through which search crawlers follow links, discover URLs and retrieve new or updated content.
AI 크롤러
A dedicated bot that collects web content for generative AI training data or real-time search-based answers.
LLM
A deep-learning neural network pretrained on extensive text data to understand and generate language in a humanlike form.
토큰
The smallest unit of text processed by an AI model, which may be a word, character or subword fragment.
AEO
An approach to organizing clear, supported content that search, AI and voice services can understand and use when answering a question directly.