AI Crawlers, robots.txt, and llms.txt: What Actually Matters

Which AI bots visit your site, what blocking them really costs, and an honest read on whether llms.txt does anything yet.

AI Cited · · 2 min read Technical AEO

Three kinds of AI-related bots hit your site, and they have very different jobs. Conflating them is the most common technical AEO mistake we see.

The three bot types

Training crawlers (GPTBot, ClaudeBot, Google-Extended, Applebot-Extended) collect data for future model training. Blocking them is a legitimate editorial choice for publishers who don’t want their writing in training sets — but for a brand that wants AI visibility, it’s shooting yourself in the foot in slow motion: today’s crawl is next year’s model’s memory of you.

Search/index crawlers (OAI-SearchBot, PerplexityBot) build the retrieval indexes assistants search at answer time. Blocking these removes you from AI answers now, not next year.

On-demand fetchers (ChatGPT-User, Perplexity-User, Claude-User) fetch a specific page because an assistant is answering a question with it right this second. This traffic is the good kind — it usually means you’re about to be cited. Blocking it turns citations into dead ends.

The practical default for anyone doing AEO: allow all three classes explicitly in robots.txt, and check your WAF and bot-protection rules too — Cloudflare and friends block several of these bots by default, and a robots.txt allow doesn’t override a firewall challenge page. If your “AI traffic” graph is flat, check the firewall before the content.

Does llms.txt matter?

llms.txt is a proposed convention: a markdown file at your site root that gives language models a curated map of your most important pages. It costs twenty minutes to add and it might help — but be honest about the state of it: as of mid-2026, no major assistant has committed to fetching it, and we’ve seen no reproducible evidence of it changing answers. Treat it as a cheap hedge, not a strategy. The same twenty minutes spent tightening your sitemap or fixing a blocked bot has a much higher expected return.

What to actually check this week

  1. Grep your CDN or server logs for GPTBot, ClaudeBot, OAI-SearchBot, PerplexityBot, and -User fetchers. Presence and frequency is your baseline.
  2. Fetch your key pages with curl (no JavaScript). If the answer-bearing content isn’t in the HTML, assistants aren’t reading it either.
  3. Confirm your robots.txt and firewall agree with your intentions.
  4. Keep your sitemap current — index crawlers lean on it harder than classic search ever did.

None of this is glamorous. All of it is upstream of every content tactic on this site: an answer engine can’t cite what it never managed to read.

Keep reading