The llms.txt file is a standard proposed by Answer.AI and fast.ai in 2024. While robots.txt tells a search engine bot 'do not crawl this page', llms.txt tells an artificial intelligence system 'this is who I am, I produce the following content, I have expertise in these topics'. The technical complexity of the file is zero — it is a text file written in standard Markdown format.
Basic Structure of the File
llms.txt is a Markdown file uploaded to your site's root directory: it must be accessible from yourcompany.com/llms.txt. The file has mandatory and optional sections.
- # Site Name → First line, site name or brand name as an H1 heading. Mandatory.
- > Short description → Single-sentence site definition in blockquote format. Mandatory.
- Introduction paragraph → 2-3 sentences describing what the site does. Optional but recommended.
- ## Sections → Categories like Blog, Services, Documents and page links below them.
- ## Restrictions → Types of content you do not want AI bots to use. Optional.
Example for an E-commerce Site
Example for a SaaS Company
Which Platforms Support It?
Perplexity actively supports the llms.txt standard and uses the file in real-time indexing. Claude has developed bot versions that read llms.txt during web searches. While full compliance for ChatGPT and Gemini is still uncertain, the file's existence contributes to the correct categorization of content during these platforms' training data crawling. In short, it makes sense to create the file today to benefit from currently supported platforms and to be prepared for future support.