How Claude Search Selects Sources to Cite
Claude's web search relies almost entirely on one search backend: Brave Search. BrightEdge found an 86.7% alignment between citations and Brave's organic results. Most of what Claude cites also ranks in Brave.
This article draws on Anthropic's crawler documentation, API specs, third-party studies, and technical disclosures. It covers how Claude picks its sources and how that differs from ChatGPT and Google AI Overviews.
The Brave Search Backbone
In March 2025, TechCrunch confirmed that Claude's web search runs on Brave Search. That shapes how you optimize for visibility in Claude.
The BrightEdge analysis put numbers on it: 86.7% of Claude's cited results overlap with Brave's top non-sponsored organic results. For comparison, ChatGPT shows 26.7% alignment with Bing's top results. Claude's citations follow its search backend far more closely than ChatGPT's follow Bing.
Most of Claude's cited results also appear in Brave's top results. Brave's index has a threshold: content needs visits from at least 20 unique Brave browser users with data-sharing enabled before becoming eligible for indexing. Established domains with diverse traffic reach that threshold more easily.
Three Crawlers, Three Purposes
Anthropic operates three separate crawlers , each with its own purpose. The documentation was last updated February 20, 2026, when the newest crawler was added.
CLAUDE-SEARCHBOT
Indexes and evaluates content quality for search results. Blocking this reduces your visibility and accuracy in Claude-powered search.
CLAUDEBOT
Crawls content for AI model training data. Can be blocked independently from search without affecting visibility.
CLAUDE-USER
Fetches pages when users explicitly ask Claude to read a specific URL. Still honors robots.txt, unlike OpenAI's equivalent.
Unlike OpenAI, all three of Anthropic's crawlers still honor robots.txt, including Claude-User. OpenAI stopped respecting robots.txt for its ChatGPT-User bot in December 2025. Anthropic's crawlers also support the non-standard
Crawl-delay
directive and do not try to bypass CAPTCHAs.
As Search Engine Journal reported , this three-bot system gives site owners finer-grained control than any other AI platform. You can allow search indexing while blocking training, or allow user browsing while restricting automated crawling.
A
robots.txt
that allows search and browsing and blocks training:
# Allow Claude search indexing
User-agent: Claude-SearchBot
Allow: /
# Allow Claude user-initiated browsing
User-agent: Claude-User
Allow: /
# Block AI model training (optional)
User-agent: ClaudeBot
Disallow: /How Claude Actually Selects Sources
Claude's source selection is a multi-step process, documented in Anthropic's API documentation :
- Decision to search: Claude autonomously decides whether to search based on three criteria: freshness (does the query need current info?), specificity (how targeted is the question?), and intent (what's the underlying purpose?)
- Query execution: The Brave Search API returns the top organic results
- Content evaluation: Claude filters and evaluates results based on relevance, clarity, and extractability
- Iteration: This cycle can repeat up to ten times in a single conversation turn, refining the search as Claude learns more
The Groundy analysis reports that Claude favors content that is "concise, current, and aligns closely with the user's phrasing and intent." That favors pages that match conversational query patterns and content written in a natural, question-answering style over keyword-stuffed copy. It fits the shift from traditional SEO toward AI Engine Optimization (AEO) , where writing for how people ask questions matters more than keyword density.
Dynamic Filtering: Why Clean HTML Matters
In February 2026, Anthropic released dynamic filtering . Claude can now write and execute Python code to post-process raw HTML before it reaches the context window.
In practice, Claude strips away:
- Navigation menus and sidebars
- Footer content and boilerplate
- Advertising and tracking markup
- Irrelevant metadata
Anthropic's dynamic filtering announcement reports these results:
Dynamic filtering is currently available only on Opus 4.6 and Sonnet 4.6 via the Claude API and Azure (not Vertex AI), and requires the code execution tool to be enabled alongside web search.
How Claude Cites Sources
Claude uses inline citations with clickable source links, like ChatGPT and unlike Perplexity's footnote-heavy approach. Every web-sourced claim includes:
- URL: The source page URL
- Title: The source page title
- Cited text: Up to 150 characters of the specific content being cited
- Encrypted index: A reference for maintaining citations in multi-turn conversations
The API docs state that citation metadata (cited_text, title, url) does not count toward input or output token usage. That lowers the token cost of applications built on Claude's web search.
Claude "only cites what it can verify" and avoids hallucinated citations. When it cannot verify a claim against search results, it omits the citation or qualifies the statement instead of inventing a plausible-looking reference.
The Citations API: A Separate Feature
Claude's web search citations (discussed above) are a different feature from the separate Citations API launched in January 2025. That API lets developers ground Claude's responses in user-provided documents (PDFs, plain text, custom content) with precise character-level references.
Anthropic's internal evaluations show the Citations API increases recall accuracy by up to 15% compared to custom prompt-based implementations. But it is a developer tool for supplied documents, and it does not affect how Claude's web search discovers or cites your website.
No Publisher Licensing Deals
OpenAI has formal licensing deals with AP, Conde Nast, Financial Times, News Corp, The Atlantic, Springer, and Washington Post. Anthropic has announced no publisher licensing partnerships.
Anthropic did reach a $1.5 billion copyright settlement (September 2025), the largest in U.S. history, covering roughly 500,000 copyrighted works. The settlement covers only past use (before August 25, 2025) and is explicitly not a licensing deal for future use.
For website owners, this means Anthropic has announced no "preferred publisher" list for Claude citations. Visibility runs through Brave Search rankings and content quality.
LEVEL PLAYING FIELD
Without announced publisher licensing deals, no publisher has a documented advantage. Visibility in Claude follows Brave Search rankings and content quality.
How Claude Compares to ChatGPT and Google
| Dimension | Claude | ChatGPT | Google AI Overviews |
|---|---|---|---|
| Search backend | Brave Search | Bing (+ Google for paid) | Google Search |
| Backend alignment | 86.7% with Brave | 26.7% with Bing | Native integration |
| User-browsing bot | Honors robots.txt | Ignores robots.txt (since Dec 2025) | N/A |
| Content processing | Dynamic filtering (strips boilerplate) | Direct content ingestion | Full index processing |
| Publisher deals | None | AP, Conde Nast, FT, News Corp | Various licensing agreements |
| Citation style | Inline clickable links | Inline links in text | Source cards with URLs |
| Crawl-delay support | Yes | Not documented | No |
What You Can Do Today
These four steps follow from how Claude searches and filters content:
1. ALLOW CLAUDE-SEARCHBOT
Claude-SearchBot indexes content for Claude's search results. Blocking it reduces your presence in Claude's search answers.
2. OPTIMIZE FOR BRAVE
With 86.7% alignment, most of Claude's cited results come from Brave's top results. Make sure Brave can index your content.
3. CLEAN YOUR HTML
Claude's dynamic filtering strips boilerplate. Clean semantic HTML with a content-first structure leaves it less to strip.
4. WRITE CONVERSATIONALLY
Claude favors content that matches conversational query patterns. Write in natural language and avoid keyword stuffing.
-
Use semantic HTML
(
<article>,<main>,<section>) to help Claude's filtering understand your content structure - Server-side render your content. Claude's crawlers cannot execute client-side JavaScript
- Keep content concise and current. Claude filters for relevance and freshness
- Add structured data. Well-structured pages tend to rank and get cited, so structured data correlates with stronger Brave rankings which can carry through to Claude. In Google AI Overviews, Wellows observed that schema correlates with a +73% selection rate , and on ChatGPT-visibility data, FAQPage schema appeared on 6.2% of cited sites versus 0.8% of non-cited ones (Insightland), a correlation, not a guarantee of citation . A 2026 Ahrefs study of 1,885 pages found that adding schema did not measurably change AI-search citations, so treat this as a correlation (cited pages tend to be well-structured) rather than schema directly driving citations ( Ahrefs )
- Maintain an XML sitemap. Helps all crawlers, including Claude-SearchBot, discover your content
The Bottom Line
Claude's source selection is easy to trace: 86.7% of its cited results overlap with Brave's top organic results. The practical steps are to rank well in Brave, allow Claude-SearchBot, and write clean, well-structured content that matches how people ask questions.
Anthropic's crawlers respect robots.txt, including for user-initiated browsing. The three-crawler setup gives site owners fine-grained control, and Anthropic has announced no publisher licensing deals. Visibility depends mainly on Brave rankings, content quality, and technical execution.
For the full picture across AI platforms, read our analyses of how ChatGPT chooses which websites to cite and how Google AI Overviews selects sources . For broader trends, see key insights from Vercel's 2026 AEO report .
Sources
- Anthropic: Crawler Documentation -Three-crawler system and robots.txt guidance
- Anthropic: Web Search Tool API Documentation -Official search tool specifications
- Anthropic: Web Search Now Available Globally -Official launch announcement
- Anthropic: Improved Web Search with Dynamic Filtering -Dynamic filtering technical details
- Anthropic: Introducing Citations API -Document-level citations feature
- TechCrunch: Anthropic Uses Brave for Web Search -Brave Search backend confirmation
- BrightEdge: The Ultimate Guide to Claude Search -86.7% Brave Search alignment analysis
- Groundy: Claude's Web Search Changes Everything -Content selection behavioral analysis
- Search Engine Journal: Anthropic's Claude Bots -Three-bot system analysis
- Search Engine Land: Anthropic Clarifies Claude Bots -Crawler documentation updates
- Ropes & Gray: Anthropic Copyright Settlement -$1.5B settlement analysis