Skip to content
Methodology

Common Crawl presence

ia.common-crawl-presence · te.1

Find Info Scanner data

Why it matters

Common Crawl is a public web archive that many AI training sets use. This check reports if the domain is in the latest index. It is information only and does not change the score.

How to fix it

Decide if you want your content in public training archives, then set CCBot rules in robots.txt.

  1. Decide if the content of the site may appear in public training archives.

  2. To opt out, add a Disallow rule for CCBot to robots.txt.

    User-agent: CCBot
Disallow: /

Verify

curl -sf 'https://index.commoncrawl.org/collinfo.json' | jq -r '.[0].id'

Expected: The latest index id. Query that index for url={domain} to see the presence.

Run a new scan for this checkpoint:

isagentready results {domain} --checkpoint ia.common-crawl-presence
Details Mapping, lifecycle and references
AR mapping
No AR practice
Evidence source
IsAgentReady benchmark
Score role
Informational
Introduced
2026-08-24
Methodology version
2026-10-22

Checkpoint data (JSON)