AI crawlers are the bots that read the web for AI systems. GPTBot and OAI-SearchBot read for OpenAI, PerplexityBot for Perplexity, ClaudeBot for Anthropic, Google-Extended governs Google's AI training use, and Bingbot's index feeds ChatGPT's web search. Whether these bots can read your site, controlled by a few lines in your robots.txt, decides whether the AI systems behind them can know, describe and recommend your business. Most site owners have never checked which ones they allow.
- The bots that feed AI systems: each engine has its readers.
- Two jobs: training data collection and live answer retrieval.
- robots.txt controls access, bot by bot, and defaults surprise people.
- Blocked crawler = invisible business, for that engine.
The cast, name by name
The ones that matter for business visibility. GPTBot: OpenAI's main crawler, gathering content that shapes what its models know. OAI-SearchBot and ChatGPT-User: OpenAI's search and browsing agents, the ones behind live ChatGPT answers that cite the web. PerplexityBot: Perplexity's crawler, aggressive and fast, powering an engine that cites sources openly. ClaudeBot and Claude-Web: Anthropic's readers. Google-Extended: not a separate crawler but a control token, it governs whether Google may use your content for AI training, without affecting Google Search. And Bingbot: technically a classic search crawler, strategically an AI one, because Bing's index is what ChatGPT's web search leans on. Block Bingbot and you cut ChatGPT's supply line without touching an "AI bot" at all.
Training crawlers shape what models know long-term. Search crawlers feed live answers today. The distinction matters because you can welcome one job and decline the other, per bot, in robots.txt.
How control works
Your robots.txt, a plain file at yourdomain.com/robots.txt, grants or denies access per user-agent. A Disallow under GPTBot shuts OpenAI's training crawler out; a Disallow under the wildcard shuts everyone out, including bots you wanted. The traps in practice: old templates and security defaults that blocked AI bots before you ever decided anything, wildcard rules with unintended reach, and CDN-level bot protection silently rejecting crawlers your robots.txt allows. The fix starts with knowing your current state, my free scan reads your robots.txt and reports access bot by bot, and mine welcomes every AI crawler by name, which you can verify at /robots.txt.
Should you allow them? The short version
I wrote the full both-sides argument separately; the compressed rule: if your content is the product you sell, paywalled publishing, blocking or licensing is a coherent strategy. If your content is marketing for what you sell, services, products, expertise, blocking is self-erasure: the pages exist to be read, and the readers that matter increasingly include machines that recommend. For that second group, which is most businesses, the crawlers are distribution, not theft.
The advanced layer: being read well
Access is binary; comprehension is not. Once crawlers can reach you, what they extract depends on your legibility: clean indexable HTML rather than script-locked content, schema that verifies facts, an llms.txt giving models your canonical summary, one consistent entity across pages. Allowing the bots is the door; the house they find inside decides whether the visit produces citations. Both halves are checkable, and both halves are the technical core of GEO.
Common questions
How do I know if AI crawlers can access my site?
Read yourdomain.com/robots.txt and look for Disallow rules under agents like GPTBot, PerplexityBot, ClaudeBot, OAI-SearchBot and Google-Extended, and check for a wildcard catching everyone. Also check CDN bot protection, which blocks silently. Or run my free scan, which reports access bot by bot.
Do AI crawlers respect robots.txt?
The major named crawlers from OpenAI, Google, Anthropic and Microsoft publicly commit to honoring robots.txt, and observably do. Perplexity has faced disputes over access behavior. The practical stance: robots.txt is the standard control and the majors respect it; it is a fence, not a vault, and for marketing content a fence is the right tool.
Does allowing AI crawlers slow down my website?
Crawler traffic is a real load but the named AI bots crawl politely compared to the scraper noise every public site already absorbs. If load is a concern, rate limiting handles it without the visibility cost of blocking. For a typical business site, the performance impact of welcoming AI crawlers is negligible against the cost of being unreadable.
Do you actually know who can read your site?
Most owners are guessing. Scan your site free and get the bot-by-bot answer in ten seconds.
Run my visibility check