Skip to content
Methodology

Training crawler blocking

ia.training-crawler-blocking · te.2

Find Info Scanner data AR-FIND-01

Why it matters

Some crawlers collect data for model training. This check reports which training crawlers robots.txt blocks. It is policy information and does not change the score.

How to fix it

Set explicit rules for training crawlers that match your policy.

  1. List the training crawlers that you want to allow or block, for example GPTBot, CCBot and Google-Extended.

  2. Add a User-agent group for each crawler with Allow or Disallow.

    User-agent: Google-Extended
Disallow: /

User-agent: GPTBot
Allow: /

  3. Keep search and user-triggered agents allowed, for example OAI-SearchBot and Claude-User.

Verify

curl -sf https://{domain}/robots.txt

Expected: User-agent groups for the training crawlers that match your policy

Run a new scan for this checkpoint:

isagentready results {domain} --checkpoint ia.training-crawler-blocking
Details Mapping, lifecycle and references
AR mapping
Partial match
Score role
Informational
Introduced
2026-08-24
Methodology version
2026-10-22

Checkpoint data (JSON)