# llms.txt: what it is, what it's for and how to check yours

> What llms.txt is, how it differs from robots.txt, how to write it on WordPress, PrestaShop or Shopify, and which AI bots to block in robots.txt or let in.

URL: https://webintacta.com/en/guides/llms-txt
Language: English (Spanish version: see the hreflang alternate of the HTML page)
Last updated: 2026-10-24

**In short:** llms.txt is a Markdown file at the root of a website that tells language models who you are, what you offer and which pages to read; robots.txt, by contrast, says which bots may come in. It is a proposal from llmstxt.org, not a mandatory standard, and having one does not guarantee an AI will cite you. This guide explains how to write it on WordPress, PrestaShop or Shopify and which AI bots to block or allow in robots.txt.


By Jose Soriano (WebIntacta) · published 2026-10-24

llms.txt is a Markdown text file that summarizes your website for language models and AI agents: who you are, what you offer and which pages are worth reading. It's useful and cheap to set up, but it's worth knowing what it is and what it isn't before you spend time on it.

## What llms.txt is

It's a **proposal** published by Jeremy Howard in September 2024 at [llmstxt.org](https://llmstxt.org/). The idea: AI assistants have a limited context window, and web pages are full of menus, scripts and ads. A short Markdown file with a summary and links to what matters saves them work and reduces mistakes.

The format the proposal defines, in this order:

1. **An H1 title** with the name of the site or project. It's the only required part.
2. **A summary in a blockquote** (a line starting with `>`) with what's needed to understand the rest.
3. **Optional free text** (paragraphs or lists, no headings) with more context.
4. **H2 sections with lists of links**, each in the form `[name](url)`, optionally followed by a colon and a note.
5. **An "Optional" section** by convention, with secondary links an agent can skip if it's short on space.

It usually lives at the root (`https://your-site.com/llms.txt`). The proposal also suggests offering a Markdown version of every important page (for example `/pricing.md`) and announcing it with a `rel="alternate" type="text/markdown"` link.

> llms.txt is a proposal, not a standard search engines have to follow. Having one won't lift you in Google or guarantee that ChatGPT or Claude cite you. It's a help for the agents that read it; treat it that way.

## llms.txt vs robots.txt

People mix them up all the time, but they do different things:

- **robots.txt** tells crawlers **what they may visit**. It's a standard (RFC 9309) that serious crawlers respect, and it's where you block a bot.
- **llms.txt** tells models **what matters and where it is**. It neither allows nor forbids anything: a `Disallow` written in llms.txt has no effect at all.

If you don't want an AI bot to read your site, that goes in robots.txt. If you want what it reads to be understood better, that goes in llms.txt. You can have both, and it usually makes sense to.

## A real example: this site's

This is the start of [WebIntacta's llms.txt](https://webintacta.com/llms.txt) (translated here; the live file is in Spanish). It's generated automatically from the site's content so it never goes stale:

```
# WebIntacta

> Web security service for online stores and business websites: free
> external scan, security report with a read-only connector (no FTP),
> hacked website cleanup with a 90-day guarantee and maintenance
> with no lock-in. WordPress, WooCommerce and PrestaShop. Run by
> Jose Soriano (sorijose.dev).

Every page has a Markdown version: request the URL with
Accept: text/markdown or prefix the path with /md.

## Main

- [Home](https://webintacta.com): what WebIntacta does and the free scan
- [Pricing](https://webintacta.com/precios): every published price
```

On top of that, every page of the site responds with a `Link` header pointing to its Markdown version and to llms.txt (`rel="describedby"`), which is how the proposal suggests agents should find them. What matters in the example: the summary says in two lines what I do and for whom, and every link carries a note explaining why it's worth reading.

## On WordPress, PrestaShop and Shopify

### WordPress

The two most popular SEO plugins already generate it:

- **Yoast SEO:** in **Yoast SEO → Settings → Site features**, under the AI tools, turn on **llms.txt** and save. You can pick by hand which pages appear. According to Yoast, it's not available on multisite, and if the file isn't generated you need to check the server's write permissions.
- **Rank Math:** turn on the **LLMS Txt** module in its dashboard and configure it in **Rank Math SEO → General Settings → Edit llms.txt**: post types, taxonomies and texts. Pages set to `noindex` are left out.

Without a plugin, a `llms.txt` file uploaded by hand to the root (next to `wp-config.php`) also works. The problem is that it goes stale as soon as you change the site.

### PrestaShop

PrestaShop has no built-in option. The simplest route is to write the file and upload it by FTP to the store's root. There are third-party modules that generate it from the catalog; if you use one, check that it doesn't publish discontinued products or internal pages, and that the `/llms.txt` URL returns text, not the store's template.

### Shopify

Shopify doesn't let you upload loose files to the root of the domain, so the usual route is an app. Yoast SEO for Shopify has the option under **Settings → Advanced → llms.txt**, and other apps in the Shopify App Store generate it too. After installing any of them, open `your-store.com/llms.txt` and check what actually comes out.

## How to check yours

1. Open `https://your-site.com/llms.txt` in the browser. You should see plain text starting with `# ` and the name of your site.
2. Make sure it isn't your home page or a "not found" page served with a 200 status, a common mistake with WordPress themes and online stores.
3. Look at the headers: it should return 200 with a text content type.
4. Read the summary as if you were a stranger: is it clear what you do, for whom and where?
5. Try four or five random links. An llms.txt full of broken links is worse than none at all.

```
curl -sI https://your-site.com/llms.txt
```

The [free scan](https://webintacta.com/en/free-website-security-scan) includes an AI-agent readiness score: it reads your robots.txt and tells you which AI bots you're blocking without knowing it, checks whether llms.txt exists and really responds, and whether your pages offer a Markdown version. If you want me to review it and get it right, that's the [AI-agent ready website](https://webintacta.com/en/ai-agent-ready-website) service.

## Which AI bots to block and which to allow

This is decided in robots.txt, and the key is that each company uses **different bots for training its models and for search**. You can say no to the first without disappearing from the second. According to each company's documentation:

- **GPTBot** (OpenAI): collects content that may be used to train its models. Blocking it doesn't remove you from ChatGPT search.
- **OAI-SearchBot** (OpenAI): the ChatGPT search bot. If you block it, you don't show up in its search answers.
- **ChatGPT-User** (OpenAI): visits a page when a user asks it to. OpenAI warns that, since it's a user action, robots.txt may not apply.
- **ClaudeBot** (Anthropic): collects content for training.
- **Claude-SearchBot** (Anthropic): indexes content to improve Claude's search.
- **Claude-User** (Anthropic): visits pages when a Claude user asks a question. Blocking it reduces your visibility in those answers.
- **Google-Extended**: not a crawler but a signal. Google crawls with its usual bots, and this rule decides whether the content may be used to train Gemini and to ground its answers. Google says explicitly that it doesn't affect your inclusion or ranking in Search.
- **PerplexityBot**: the one behind Perplexity's results; according to Perplexity, it isn't used to train models.
- **Perplexity-User**: visits requested by a user; Perplexity says it generally ignores robots.txt.

My recommendation for an online store or a service business: let the search bots and the user bots in (you want to be recommended), and decide separately whether you accept training. If you don't, a robots.txt like this makes it clear without shutting you out of search:

```
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: *
Allow: /

Sitemap: https://your-site.com/sitemap.xml
```

> What I find most often isn't a chosen block but an accidental one: a security plugin or a hosting firewall rule that blocks every AI bot, search bots included. Check your robots.txt and, if you use a CDN, its bot rules.

## Sources

- [The llms.txt proposal](https://llmstxt.org/)
- [OpenAI: bots and user agents](https://developers.openai.com/api/docs/bots)
- [Anthropic: how Claude crawls and how to block it](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler)
- [Google: common crawlers and Google-Extended](https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers)
- [Perplexity: its crawlers](https://docs.perplexity.ai/guides/bots)
- [Yoast: enable llms.txt](https://yoast.com/help/enable-llmstxt/) and [on Shopify](https://yoast.com/help/enable-llms-txt-yoast-seo-shopify/)
- [Rank Math: llms.txt](https://rankmath.com/kb/llms-txt/)

## Frequently asked questions

### What is the llms.txt file?

It's a Markdown file, usually at the root of a website, that summarizes the site and links to its important pages so language models can understand it without reading all the HTML. It's a proposal from llmstxt.org, not a mandatory standard.

### Does llms.txt help with SEO on Google?

Not directly. Google doesn't use it for ranking, and having one doesn't change your position in Search. It can help the assistants that do read it understand your site better.

### Can I block ChatGPT with llms.txt?

No. llms.txt neither allows nor forbids anything. To block a bot you use robots.txt with its name, for example GPTBot for OpenAI's training or OAI-SearchBot for its search.

### How do I create llms.txt on WordPress?

With Yoast SEO (Settings → Site features → llms.txt) or with Rank Math's LLMS Txt module. You can also upload an llms.txt file to the root by hand, although you'll have to keep it up to date yourself.

### If I block GPTBot, do I disappear from ChatGPT?

No. GPTBot is the training bot. ChatGPT search uses OAI-SearchBot; as long as you don't block it, your site can keep appearing in its answers.

## About the author

[Jose Soriano](https://webintacta.com/en/about): Full-stack developer specialized in PrestaShop, WordPress and WooCommerce, who has passed the official PrestaShop course and exam. He has been building online stores and modules since 2023 and is the founder of WebIntacta, the website security service through which he scans, cleans and maintains small-business websites and online stores himself, remotely from Ibi (Alicante, Spain). WebIntacta facts: https://webintacta.com/en/about#datos