HTTP bot accessibility

Why it matters

AI crawlers and agents must be able to fetch your pages via HTTP. If bots are blocked, AI systems cannot index or interact with your content.

How to fix it

Return HTTP 200 to legitimate bots and do not block them in the WAF or CDN.

1. Request the home page with a bot user agent and record the HTTP status.

2. When the status is 401 or 403, add a WAF or CDN rule that allows known AI user agents.

   map $http_user_agent $is_ai_bot {
     default       0;
     "~*GPTBot"    1;
     "~*ClaudeBot" 1;
   }

3. Keep rate limits, but do not answer verified AI crawlers with a challenge page.

Verify

Run a new scan for this checkpoint: