Skip to content
Methodology

AI crawler directives

ia.ai-crawler-directives · 1.2

Find MUST Scanner data AR-FIND-01

Why it matters

robots.txt determines which paths crawlers may fetch. This checkpoint measures effective crawl access for common AI user agents; it does not infer training consent.

How to fix it

Let the common AI user agents fetch your public pages in robots.txt.

  1. Find the rules in robots.txt that block AI user agents such as GPTBot, ClaudeBot and PerplexityBot.

  2. Add an explicit Allow group for each AI user agent that may fetch your public pages.

    User-agent: GPTBot
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: PerplexityBot
Allow: /

  3. Keep Disallow rules only for private paths, for example /account.

Test it with the free tool

Verify

curl -sf https://{domain}/robots.txt

Expected: No Disallow: / rule applies to GPTBot, ClaudeBot or the * group

Run a new scan for this checkpoint:

isagentready results {domain} --checkpoint ia.ai-crawler-directives
Details Mapping, lifecycle and references
AR mapping
Full match
Score role
Core
Introduced
2026-08-24
Revised
2026-09-02
Methodology version
2026-10-22

Checkpoint data (JSON)