llms.txt: what it is, what it's for and how to check yours

In short

llms.txt is a Markdown file at the root of a website that tells language models who you are, what you offer and which pages to read; robots.txt, by contrast, says which bots may come in. It is a proposal from llmstxt.org, not a mandatory standard, and having one does not guarantee an AI will cite you. This guide explains how to write it on WordPress, PrestaShop or Shopify and which AI bots to block or allow in robots.txt.

llms.txt is a Markdown text file that summarizes your website for language models and AI agents: who you are, what you offer and which pages are worth reading. It's useful and cheap to set up, but it's worth knowing what it is and what it isn't before you spend time on it.

By Jose SorianoLast updated: October 24, 20266 min read

What llms.txt is

It's a proposal published by Jeremy Howard in September 2024 at llmstxt.org. The idea: AI assistants have a limited context window, and web pages are full of menus, scripts and ads. A short Markdown file with a summary and links to what matters saves them work and reduces mistakes.

The format the proposal defines, in this order:

An H1 title with the name of the site or project. It's the only required part.
A summary in a blockquote (a line starting with >) with what's needed to understand the rest.
Optional free text (paragraphs or lists, no headings) with more context.
H2 sections with lists of links, each in the form [name](url), optionally followed by a colon and a note.
An "Optional" section by convention, with secondary links an agent can skip if it's short on space.

It usually lives at the root (https://your-site.com/llms.txt). The proposal also suggests offering a Markdown version of every important page (for example /pricing.md) and announcing it with a rel="alternate" type="text/markdown" link.

llms.txt vs robots.txt

People mix them up all the time, but they do different things:

robots.txt tells crawlers what they may visit. It's a standard (RFC 9309) that serious crawlers respect, and it's where you block a bot.
llms.txt tells models what matters and where it is. It neither allows nor forbids anything: a Disallow written in llms.txt has no effect at all.

If you don't want an AI bot to read your site, that goes in robots.txt. If you want what it reads to be understood better, that goes in llms.txt. You can have both, and it usually makes sense to.

A real example: this site's

This is the start of WebIntacta's llms.txt (translated here; the live file is in Spanish). It's generated automatically from the site's content so it never goes stale:

# WebIntacta

> Web security service for online stores and business websites: free
> external scan, security report with a read-only connector (no FTP),
> hacked website cleanup with a 90-day guarantee and maintenance
> with no lock-in. WordPress, WooCommerce and PrestaShop. Run by
> Jose Soriano (sorijose.dev).

Every page has a Markdown version: request the URL with
Accept: text/markdown or prefix the path with /md.

## Main

- [Home](https://webintacta.com): what WebIntacta does and the free scan
- [Pricing](https://webintacta.com/precios): every published price

On top of that, every page of the site responds with a Link header pointing to its Markdown version and to llms.txt (rel="describedby"), which is how the proposal suggests agents should find them. What matters in the example: the summary says in two lines what I do and for whom, and every link carries a note explaining why it's worth reading.

On WordPress, PrestaShop and Shopify

WordPress

The two most popular SEO plugins already generate it:

Yoast SEO: in Yoast SEO → Settings → Site features, under the AI tools, turn on llms.txt and save. You can pick by hand which pages appear. According to Yoast, it's not available on multisite, and if the file isn't generated you need to check the server's write permissions.
Rank Math: turn on the LLMS Txt module in its dashboard and configure it in Rank Math SEO → General Settings → Edit llms.txt: post types, taxonomies and texts. Pages set to noindex are left out.

Without a plugin, a llms.txt file uploaded by hand to the root (next to wp-config.php) also works. The problem is that it goes stale as soon as you change the site.

PrestaShop

PrestaShop has no built-in option. The simplest route is to write the file and upload it by FTP to the store's root. There are third-party modules that generate it from the catalog; if you use one, check that it doesn't publish discontinued products or internal pages, and that the /llms.txt URL returns text, not the store's template.

Shopify

Shopify doesn't let you upload loose files to the root of the domain, so the usual route is an app. Yoast SEO for Shopify has the option under Settings → Advanced → llms.txt, and other apps in the Shopify App Store generate it too. After installing any of them, open your-store.com/llms.txt and check what actually comes out.

How to check yours

Open https://your-site.com/llms.txt in the browser. You should see plain text starting with # and the name of your site.
Make sure it isn't your home page or a "not found" page served with a 200 status, a common mistake with WordPress themes and online stores.
Look at the headers: it should return 200 with a text content type.
Read the summary as if you were a stranger: is it clear what you do, for whom and where?
Try four or five random links. An llms.txt full of broken links is worse than none at all.

curl -sI https://your-site.com/llms.txt

The free scan includes an AI-agent readiness score: it reads your robots.txt and tells you which AI bots you're blocking without knowing it, checks whether llms.txt exists and really responds, and whether your pages offer a Markdown version. If you want me to review it and get it right, that's the AI-agent ready website service.

Which AI bots to block and which to allow

This is decided in robots.txt, and the key is that each company uses different bots for training its models and for search. You can say no to the first without disappearing from the second. According to each company's documentation:

GPTBot (OpenAI): collects content that may be used to train its models. Blocking it doesn't remove you from ChatGPT search.
OAI-SearchBot (OpenAI): the ChatGPT search bot. If you block it, you don't show up in its search answers.
ChatGPT-User (OpenAI): visits a page when a user asks it to. OpenAI warns that, since it's a user action, robots.txt may not apply.
ClaudeBot (Anthropic): collects content for training.
Claude-SearchBot (Anthropic): indexes content to improve Claude's search.
Claude-User (Anthropic): visits pages when a Claude user asks a question. Blocking it reduces your visibility in those answers.
Google-Extended: not a crawler but a signal. Google crawls with its usual bots, and this rule decides whether the content may be used to train Gemini and to ground its answers. Google says explicitly that it doesn't affect your inclusion or ranking in Search.
PerplexityBot: the one behind Perplexity's results; according to Perplexity, it isn't used to train models.
Perplexity-User: visits requested by a user; Perplexity says it generally ignores robots.txt.

My recommendation for an online store or a service business: let the search bots and the user bots in (you want to be recommended), and decide separately whether you accept training. If you don't, a robots.txt like this makes it clear without shutting you out of search:

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: *
Allow: /

Sitemap: https://your-site.com/sitemap.xml

Sources

Frequently asked questions

What is the llms.txt file?

It's a Markdown file, usually at the root of a website, that summarizes the site and links to its important pages so language models can understand it without reading all the HTML. It's a proposal from llmstxt.org, not a mandatory standard.

Does llms.txt help with SEO on Google?

Not directly. Google doesn't use it for ranking, and having one doesn't change your position in Search. It can help the assistants that do read it understand your site better.

Can I block ChatGPT with llms.txt?

No. llms.txt neither allows nor forbids anything. To block a bot you use robots.txt with its name, for example GPTBot for OpenAI's training or OAI-SearchBot for its search.

How do I create llms.txt on WordPress?

With Yoast SEO (Settings → Site features → llms.txt) or with Rank Math's LLMS Txt module. You can also upload an llms.txt file to the root by hand, although you'll have to keep it up to date yourself.

If I block GPTBot, do I disappear from ChatGPT?

No. GPTBot is the training bot. ChatGPT search uses OAI-SearchBot; as long as you don't block it, your site can keep appearing in its answers.