Should my business block AI crawlers?

For most businesses that sell something, no: blocking AI crawlers makes you invisible to the exact systems your buyers now ask for recommendations. For publishers whose content IS the product, the calculus genuinely differs, and blocking can be rational. The mistake is treating this as one decision. It is several: different bots do different jobs, and robots.txt lets you choose per bot. Here is the honest trade, both sides, and a framework instead of a slogan.

Why so many sites blocked, and what they were buying

When AI crawlers appeared, thousands of sites disallowed them on principle: content scraped for training, no permission, no payment, no clicks back. For news organizations and paywalled publishers, that position is coherent, their content is the product, and a system that ingests it and answers readers directly is a competitor wearing their work. If that is you, blocking, or negotiating licensing, is a legitimate strategy, and nothing in this article argues otherwise.

What blocking costs everyone else

Now the other side, the one that applies if you sell services, products, or expertise rather than content itself. Your buyers ask ChatGPT and Perplexity who to hire. Those systems answer from what their crawlers can read. Block GPTBot and OAI-SearchBot and you have not protected anything a service business needs protecting, your marketing pages were written to be read, but you have removed yourself from the answer. Your competitor who stayed readable inherits your citations. It is the only marketing decision I know where the downside is total, invisible, and free for your rival.

Deciding per bot, the part everyone skips

Robots.txt is per-agent, and the agents differ in what your blocking actually withholds:

• Search and citation bots, OAI-SearchBot, Claude-SearchBot and PerplexityBot, plus the user-fetch agents ChatGPT-User, Claude-User and Perplexity-User: power live answers that name and link sources. User-fetch agents may not follow robots.txt. Blocking the search bots removes you from recommendations, the core GEO surface. For any business that wants buyers, leave them open.
• Training-oriented crawling, GPTBot, ClaudeBot, and Google-Extended (which controls Gemini apps and Vertex AI training and grounding, not Google Search, AI Overviews or AI Mode, per Google's crawler documentation): mainly affects future model knowledge. Blocking them is the philosophically cleaner protest, and even here a business wanting AI visibility benefits from being in the models' baseline picture of its category.
• The nuance that bites: some access paths overlap, and the ecosystem shifts. A middle position exists, open the citation bots, restrict training-only agents, and it is the reasonable compromise for the genuinely conflicted.

My position, stated plainly

For service businesses, consultancies, local businesses, product companies, anyone whose site exists to win customers: open them all. The training question is real but abstract for you; the recommendation question is concrete and immediate, and being absent from AI answers costs actual pipeline today. My own robots.txt allows every crawler and names the main AI agents one by one, which you can verify at the scan or by reading the file. I practice the position because the trade is not close for a business like mine, or, probably, like yours. And whichever way you decide, decide on purpose: check what the AI currently says about you first, because plenty of sites blocked years ago in a template default and never knew.