# Your website, ready for AI agents: they read it if you want them to. > Free check of whether your website is ready for AI agents like ChatGPT: robots.txt, which AI bots to block and which not, llms.txt and structured data. URL: https://webintacta.com/en/ai-agent-ready-website Language: English (Spanish version: see the hreflang alternate of the HTML page) Last updated: 2026-10-24 **In short:** A website is ready for AI agents when assistants like ChatGPT, Claude or Perplexity can read and understand it, if the owner wants them to: a robots.txt that tells training bots apart from search bots, an llms.txt file and structured data. The free WebIntacta scan checks this with a score kept separate from the security grade. Blocking training (GPTBot, for example) does not remove you from ChatGPT search answers. ChatGPT, Claude and Perplexity already visit websites to answer their users, and some agents are starting to buy. Your site can let them in, block them or say nothing at all. The scanner tells you where you stand, with a score that is separate from the security grade. ## What it checks - `robots.txt`: whether it names AI bots and whether it blocks search bots by mistake - Content Signals: which uses of your content you accept (search, training) - `llms.txt` and a Markdown version of the pages - Sitemap and discovery headers - Structured data (schema.org) for the business and, on online stores, for the products - Agent capabilities: API catalog, MCP, Agent Skills ## How I do it 1. **Scan your website.** It is part of the free scan: you see the score next to the security grade. 2. **Decide what you want.** Showing up in assistants' answers while not handing your content over for training is a perfectly valid choice. What matters is deciding. 3. **Apply it.** If you want, I set it up: `robots.txt`, `llms.txt` and structured data, without touching your security. ## Which AI bots to block and which not Each company now separates the bot that collects content for training from the one that searches to answer questions. Blocking the first does not take you out of the answers; blocking the second does. The names, according to each company's documentation: - **OpenAI:** `GPTBot` collects content for training; `OAI-SearchBot` decides whether you appear in ChatGPT search; `ChatGPT-User` is the visit ChatGPT makes when a user asks it to, and OpenAI notes that `robots.txt` may not apply to those visits. - **Anthropic:** `ClaudeBot` is the training bot; `Claude-SearchBot` the search bot; `Claude-User` the one that visits at the request of a Claude user. - **Google:** `Google-Extended` is not a bot but a token in `robots.txt` that says whether your content can be used to train Gemini. It does not affect Google Search or your rankings. - **Perplexity:** `PerplexityBot` indexes for its answers and `Perplexity-User` visits at a user's request. A common setup: allow the search and user bots, and decide on the training bots separately. What `llms.txt` is and how to write one is in the [llms.txt guide](https://webintacta.com/en/guides/llms-txt). ## How this website does it I practice what I preach: this site publishes [llms.txt](https://webintacta.com/llms.txt), answers in Markdown when asked with `Accept: text/markdown`, declares Content Signals in [robots.txt](https://webintacta.com/robots.txt) and describes the business, the services and the prices in schema.org. ## Price The AI agent readiness check is included in the free scan and in the security report. ### External scan — Free Security grade and the problems visible from the internet, in a few minutes and with no sign-up. - Platform and version (WordPress, PrestaShop, WooCommerce, Shopify…) - Visible plugins and modules with published flaws - Files that should not be public (backups, .git, configuration files) - HTTPS, certificate and security headers - Domain email (SPF, DKIM, DMARC) and blocklists - AI agent readiness (scored separately) ### Security report · business website — €129 (excluding VAT) An inside analysis with the read-only connector and a PDF with an action plan. $129 outside the EU. - Core integrity: every file compared with the original of your version - Malware, backdoors and PHP where it does not belong - Inventory of plugins or modules with versions and known flaws - Administrator accounts, configuration and permissions - Full PDF with fixes, plus a diagnosis-only version - Q&A by email or chat to go through it together > Fully credited if you hire a cleanup or maintenance within 30 days. The same applies to websites and online stores outside the EU: the check is free. ## Frequently asked questions ### Should I block GPTBot? GPTBot is the bot that collects content to train OpenAI's models. Blocking it does not remove you from ChatGPT's answers, which use OAI-SearchBot and ChatGPT-User. Many sites block training and allow search. ### Does blocking Google-Extended take me out of Google? No. Google-Extended only controls whether your content is used to train Gemini; Google states that it does not affect inclusion in Google Search or rankings. ### What is llms.txt? A text file at the root of the website that sums up, for language models, what the site is and which pages matter. It is a recent proposal (llmstxt.org) and simple to add. More in the [llms.txt guide](https://webintacta.com/en/guides/llms-txt). ### How do I optimize my website for ChatGPT and other AI assistants? Make sure the search bots (OAI-SearchBot, Claude-SearchBot, PerplexityBot) are not blocked in `robots.txt`, publish an `llms.txt`, and describe your business and products with schema.org structured data. The free scan tells you which of these you already have. ### Does it affect my security grade? No. AI agent readiness has its own score and never adds to or subtracts from the security grade. ### Is it useful for online stores? Especially for online stores: shopping agents need price, availability and product identifiers (SKU, GTIN) in the structured data. Hire or ask (email and chat): https://webintacta.com/soporte?lang=en