What are AI crawlers?
AI crawlers are the bots that read the web for AI systems. OpenAI runs GPTBot and OAI-SearchBot, Anthropic runs ClaudeBot and Claude-SearchBot, Perplexity runs PerplexityBot, and Google-Extended is a robots.txt token for Gemini apps and Vertex AI, not for AI Overviews. Whether these bots can read your site, controlled by a few lines in your robots.txt, decides whether the AI systems behind them can know, describe and recommend your business. Most site owners have never checked which ones they allow.
- The bots that feed AI systems: each engine has its readers.
- Two jobs: training data collection and live answer retrieval.
- robots.txt controls access, bot by bot, and defaults surprise people.
- Blocked search crawler = absent from that engine's answers.
The cast, name by name
The ones that matter for business visibility. GPTBot: OpenAI's main crawler, gathering content that shapes what its models know. OAI-SearchBot and ChatGPT-User: OpenAI's search and browsing agents, the ones behind live ChatGPT answers that cite the web. PerplexityBot: Perplexity's crawler, aggressive and fast, powering an engine that cites sources openly. ClaudeBot (training), Claude-SearchBot (search) and Claude-User (user fetches): Anthropic's readers; the older Claude-Web and anthropic-ai tokens are retired. Perplexity-User fetches pages when a user asks and generally ignores robots.txt. Google-Extended: not a separate crawler but a control token for Gemini apps and Vertex AI training and grounding. AI Overviews and AI Mode follow Googlebot and snippet controls instead. And Bingbot: a classic search crawler that also feeds Copilot and Bing AI answers, which Bing Webmaster Tools has reported as citations since 10 Feb 2026. Also worth knowing: OAI-AdsBot, Applebot-Extended (an Apple training opt-out, not a crawler), meta-externalagent, Meta-WebIndexer, Amazonbot, Amzn-SearchBot and Amzn-User.
Training crawlers shape what models know long-term. Search crawlers feed live answers today. The distinction matters because you can welcome one job and decline the other, per bot, in robots.txt.
How control works
Your robots.txt, a plain file at yourdomain.com/robots.txt, grants or denies access per user-agent. A Disallow under GPTBot shuts OpenAI's training crawler out; a Disallow under the wildcard shuts everyone out, including bots you wanted. The traps in practice: old templates and security defaults that blocked AI bots before you ever decided anything, wildcard rules with unintended reach, and CDN-level bot protection silently rejecting crawlers your robots.txt allows. The fix starts with knowing your current state, my free scan reads your robots.txt and reports access bot by bot, and mine names the major AI crawlers one by one, which you can verify at /robots.txt.
Should you allow them? The short version
I wrote the full both-sides argument separately; the compressed rule: if your content is the product you sell, paywalled publishing, blocking or licensing is a coherent strategy. If your content is marketing for what you sell, services, products, expertise, blocking is self-erasure: the pages exist to be read, and the readers that matter increasingly include machines that recommend. For that second group, which is most businesses, the crawlers are distribution, not theft.
The advanced layer: being read well
Access is binary; comprehension is not. Once crawlers can reach you, what they extract depends on your legibility: clean indexable HTML rather than script-locked content, schema that verifies facts, an llms.txt giving models your canonical summary, one consistent entity across pages. Allowing the bots is the door; the house they find inside decides whether the visit produces citations. Both halves are checkable, and both halves are the technical core of GEO.
Common questions
How do I know if AI crawlers can access my site?
Read yourdomain.com/robots.txt and look for Disallow rules under agents like GPTBot, PerplexityBot, ClaudeBot, OAI-SearchBot and Google-Extended, and check for a wildcard catching everyone. Also check CDN bot protection, which blocks silently. Or run my free scan, which reports access bot by bot.
Do AI crawlers respect robots.txt?
The training and search crawlers from OpenAI, Google, Anthropic, Microsoft and Perplexity document honoring robots.txt. User-triggered fetchers are the exception: OpenAI says ChatGPT-User may not follow it, and Perplexity says Perplexity-User generally ignores it. The practical stance: robots.txt is the standard control and the majors respect it; it is a fence, not a vault, and for marketing content a fence is the right tool.
Does allowing AI crawlers slow down my website?
Crawler traffic is a real load but the named AI bots crawl politely compared to the scraper noise every public site already absorbs. If load is a concern, rate limiting handles it without the visibility cost of blocking. For a typical business site, the performance impact of welcoming AI crawlers is negligible against the cost of being unreadable.
Do you actually know who can read your site?
Most owners are guessing. Scan your site free and get the bot-by-bot answer in ten seconds.
Run my visibility check