Skip to content
Methodology

robots.txt present

ia.robots-txt · 1.1

Find MUST Scanner data AR-FIND-01

Why it matters

robots.txt is the universal standard for telling crawlers what they can access. AI crawlers like GPTBot, ClaudeBot and Googlebot-Extended check this file first before crawling your site.

How to fix it

Serve a plain-text /robots.txt with at least one valid User-agent group.

  1. Create a file at the site root with the name robots.txt.

  2. Add a User-agent group with Allow or Disallow rules that follow RFC 9309.

    User-agent: *
Allow: /

Sitemap: https://example.com/sitemap.xml

  3. Serve the file with HTTP 200 and the Content-Type text/plain.

Test it with the free tool

Verify

curl -sf https://{domain}/robots.txt

Expected: HTTP 200 with a text body that contains a User-agent line

Run a new scan for this checkpoint:

isagentready results {domain} --checkpoint ia.robots-txt
Details Mapping, lifecycle and references
AR mapping
Full match
Score role
Core
Introduced
2026-08-24
Revised
2026-09-02
Methodology version
2026-10-22

Checkpoint data (JSON)