SEO PlaysGPTBot and friends: let AI crawlers in, then verify they visit
An assistant can only cite a page its crawler was allowed to fetch. That single dependency now has several parts, because each vendor runs more than one crawler with a different job each: one that builds a search index, one that gathers training data, one that fetches a page in the moment because a user asked for it.
Every one of those carries its own user-agent token, so robots.txt has stopped being one decision. Allowing the search crawlers is a choice to be citable. Allowing the training crawlers is a separate choice, and a revocable one.
Know what the trade is before you open the gates. These crawlers read enormous amounts and send back almost nobody; what you are buying is presence in answers, not visits. And an allowed token is worth verifying afterwards, because a user agent is just a string that anything can send.
What you get
A robots.txt that admits exactly the AI crawlers you want, and log-level proof of which ones actually visit your site.
Steps
- List the bots and what each one feeds. OpenAI documents four: OAI-SearchBot (ChatGPT search results), GPTBot (model training), ChatGPT-User (fetches when a user clicks), OAI-AdsBot. Anthropic documents ClaudeBot (training), Claude-SearchBot and Claude-User. Perplexity documents PerplexityBot and Perplexity-User. Google-Extended controls AI training use, not Search.
- Decide by category, not by fear: allow the search bots (OAI-SearchBot, Claude-SearchBot, PerplexityBot) if you want AI answers citing you; the training bots (GPTBot, ClaudeBot, Google-Extended) are a separate, revocable choice.
- Write the robots.txt blocks, one per user agent, for example: "User-agent: OAI-SearchBot" then "Allow: /". The exact tokens are in each vendor's bot documentation (developers.openai.com, support.claude.com, docs.perplexity.ai — all reviewed 2026-09-01).
- Know the limits: both OpenAI and Perplexity state that user-triggered fetchers (ChatGPT-User, Perplexity-User) may ignore robots.txt, because a human asked for the page.
- Wait a day; OpenAI and Perplexity both state robots.txt changes take up to 24 hours to be honored.
- Verify visitors are genuine: the vendors publish IP lists (openai.com/gptbot.json and searchbot.json, perplexity.com/perplexitybot.json) to check log entries against.
- Grep your access logs for the UA tokens (GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot); behind Cloudflare, the AI Crawl Control dashboard (formerly AI Audit, on all plans) shows the same per-crawler picture without log work.
- Set expectations before you count on traffic: Cloudflare's 2025 measurements put the AI crawlers orders of magnitude above Googlebot in pages read per referral sent.
How to verify it worked
- Access logs (or CDN analytics) show hits from the allowed AI user agents, from published vendor IPs, within a week or two.
- robots.txt fetched from yourdomain.com/robots.txt shows the exact per-bot blocks you intended, with no typos in the tokens.
Why it works
Nothing downstream works if the fetch never happened. robots.txt is the one part of the assistant pipeline entirely under your control, and now that every vendor documents its agents separately, a blanket allow or block is still a decision — just one you did not make on purpose.
Evidence
Vendor bot documentation reviewed 2026-09-01: OpenAI (four agents, ~24-hour robots.txt propagation, published IP lists), Anthropic (ClaudeBot, Claude-SearchBot, Claude-User), Perplexity (two agents, user fetches exempt from robots.txt).
Cloudflare 2025 crawl-to-referral ratios: Anthropic ~73,000 pages crawled per referral, OpenAI ~1,700:1, Googlebot ~14:1 — AI search reads everything and sends little.
Last reviewed: 2026-09-01. A play whose evidence stops holding up is rewritten or retired.