Skip to content
Methodology

Meta robots / X-Robots-Tag

ia.meta-robots · 1.5

Find MUST Scanner data AR-FIND-01

Why it matters

Meta robots tags and X-Robots-Tag headers control per-page indexing. Restrictive directives like noindex or noai prevent AI systems from using your content.

How to fix it

Remove restrictive robots directives from pages that agents must use.

  1. Find noindex, nofollow, nosnippet, noai or noimageai in meta robots tags and in X-Robots-Tag headers.

  2. Remove the directives from public pages, or set index and follow.

    <meta name="robots" content="index, follow">

Verify

curl -sI https://{domain}/

Expected: No X-Robots-Tag header with noindex, noai or nosnippet

Run a new scan for this checkpoint:

isagentready results {domain} --checkpoint ia.meta-robots
Details Mapping, lifecycle and references
AR mapping
Partial match
Score role
Core
Introduced
2026-08-24
Methodology version
2026-10-22

Checkpoint data (JSON)