Skip to main content

Knowledge base

Should we block AI crawlers?

Only if you are protecting proprietary content you do not want reproduced. For most businesses, blocking AI crawlers removes you from AI answers entirely — which means removal from the shortlist buyers form before they contact anyone.

Only if you are protecting proprietary content you do not want reproduced. For most businesses, blocking AI crawlers removes you from AI answers entirely — which means removal from the shortlist buyers form before they contact anyone.

Blocking is often accidental — security plugins and CDN rules block unknown user agents by default.

Check your robots.txt and firewall logs for GPTBot, ClaudeBot, PerplexityBot, and Google-Extended.

An explicit allowlist is more durable than relying on a blanket allow rule.

See where you currently appear in AI answers.

Start the conversation