AI Crawlers Ignore llms.txt, But AI Agents Don't
Dries Buytaert, founder of Drupal, recently published a data-driven analysis of llms.txt and markdown adoption by AI crawlers. He found that zero AI crawlers accessed his llms.txt file, that markdown pages increased total crawl traffic by 7%, and that no crawler used HTTP content negotiation. He called llms.txt "a solution looking for a problem."
The data is solid, but it covers crawlers, and his conclusion is about a standard built for agents.
What the Data Shows
Dries analyzed his Cloudflare logs after making all his pages available as markdown files. The findings:
Across Acquia's entire hosting infrastructure, one of the largest Drupal hosting platforms, llms.txt accounted for 0.001% of 400 million requests. All 52 requests to llms.txt came from SEO audit tools, not AI systems.
Leon Furze ran a similar experiment on his WordPress blog. The result was the same. Markdown and HTML pages were crawled at roughly the same rate, traffic did not change measurably, and llms.txt had no visible effect on crawler behavior.
The data is clear: AI crawlers don't use llms.txt. That says little about agents, just as counting the trucks in a bike lane says little about whether cyclists use it.
Crawlers and Agents Have Different Jobs
Dries' analysis covers crawlers only. Crawling for training is not the only way AI systems reach web content, and the two groups behave differently:
| AI Crawlers | AI Agents | |
|---|---|---|
| Purpose | Scrape content for training data | Complete a task for a specific user |
| Behavior | Mass crawl, grab everything | Targeted fetch, get what's needed |
| Token efficiency | Irrelevant, data is preprocessed offline | Critical, every token costs time and money |
| Content format | HTML is fine, they strip it anyway | Markdown saves 80% of tokens |
| Discovery | Sitemaps, link crawling | llms.txt, content negotiation, tool manifests |
| Examples | GPTBot, ClaudeBot, Google-Extended | Claude Code, Cursor, Windsurf, Bun |
AI crawlers collect as much of the web as they can. Their pipelines were built years ago for HTML scraping, and a few sites offering raw markdown give them little reason to change that setup.
AI agents fetch specific pages to solve a specific task, and every token counts. A blog post that is 20% content and 80% navigation HTML wastes most of those tokens. Markdown and llms.txt address that directly.
Coding Agents Are Already Using These Standards
Outside crawler logs, several agent tools already use these standards:
Claude Code
Anthropic's coding agent sends Accept headers that prefer markdown when fetching documentation. It also looks for llms.txt to discover relevant content on a site.
Bun
The JavaScript runtime started sending content negotiation headers when fetching documentation pages, preferring markdown when available.
Cursor & Windsurf
AI-powered code editors fetch documentation to help developers. They benefit directly from markdown versions that preserve structure without HTML noise.
Cloudflare
Now offers content negotiation and markdown transformation in its paid tiers, which suggests that platform providers see demand from agents.
Some documentation platforms already put "agent directives" on pages that point agents to llms.txt for content discovery. Adoption of content negotiation and llms.txt comes from developer tooling for agents, not from training pipelines.
Adoption Is Industry-Specific
Adoption of llms.txt and markdown also skews toward developer documentation. Dries runs a personal blog, not a docs site, so his data does not cover that use case.
Coding agents fetch documentation pages often. When Claude Code needs to understand a library API, or Cursor needs a framework's configuration options, they read documentation. On those pages:
- Markdown versions save the most tokens, because docs pages carry a lot of navigation and sidebars
- llms.txt gives agents a curated entry point to the most relevant pages
- Content negotiation lets agents get clean content without the UI chrome
Vercel, Cloudflare, Stripe, and other developer-facing companies have already implemented these standards. The Vercel State of AEO report recommends llms.txt as part of its AI visibility advice. Vercel even built AEO tracking for coding agents to measure this adoption.
Why Crawlers Will Probably Never Use llms.txt
Four reasons explain why crawlers ignore llms.txt:
- Scale economics. Crawlers process billions of pages. A curated discovery step per domain adds complexity for little gain, and they already have sitemaps and link graphs
- Training incentives. Training benefits from more data. A curated llms.txt that points to 20 key pages runs against that goal
- Existing infrastructure. HTML scraping pipelines are mature, so rebuilding them for markdown has no clear business case
- Content control concerns. A crawler gets more context if it takes everything, so a curated list gives it no advantage
So the missing crawler traffic does not show that llms.txt fails. It shows that llms.txt was not built for crawlers.
Readiness Is Not About Today's ROI
Dries' article ends with practical advice: focus on "clear writing, authoritative content, and timely publishing" rather than llms.txt. That advice is sound for a personal blog, but it leaves out agent traffic.
A site can prepare for agent traffic before that traffic shows up in logs. Adding llms.txt and markdown responses costs little, and several agent tools already request them.
Coding agents now read documentation for many developers, and AI-powered browsing agents like ChatGPT Search and Claude Search are still maturing. Those agents can already read sites that serve markdown and llms.txt.
What You Should Implement
These four steps follow agent adoption, not crawler adoption:
1. llms.txt
Create a curated entry point for agents. List your most important pages with brief descriptions. It takes little effort and helps any agent that looks for the file.
2. Content Negotiation
Serve markdown when agents request it in the Accept header. Cloudflare offers this out of the box. Markdown saves agents 80% of token overhead.
3. Structured Data
JSON-LD, Schema.org types, and FAQPage schema make your content machine-readable for both crawlers and agents. Cited sites are far more likely to carry FAQPage schema (an 8x correlation), though a 2026 Ahrefs study found adding schema alone does not directly move citations.
4. Crawler Access
Allow AI crawlers in robots.txt. Block training bots if you want, but keep search bots open. Without access, crawlers cannot see your content.
The first two are agent-specific. The last two help both crawlers and agents.
The Bottom Line
Dries' data is accurate: AI crawlers don't use llms.txt. But crawler behavior says little about llms.txt adoption, in the same way that browser traffic says little about how well an API works. The audience is different.
Coding assistants, browsing agents and task automation tools are the intended users of llms.txt and content negotiation. Their traffic is smaller than crawler traffic, but they are the audience these standards serve.
"Do AI crawlers use llms.txt today?" is the wrong question to ask. A better one is whether your site serves agents that already fetch its pages.
Sources
- Dries Buytaert: Markdown, llms.txt, and AI Crawlers (Original analysis of crawler behavior with Cloudflare log data)
- Leon Furze: Letting the Robots In (Independent replication on WordPress with similar findings)
- llms.txt Specification (The original proposal by Jeremy Howard)
- How Vercel Built AEO Tracking for Coding Agents (Vercel's approach to measuring agent adoption)
- Cloudflare: Markdown for Agents (Content negotiation and markdown transformation feature)
- IsAgentReady: The State of AEO: Key Insights from Vercel's 2026 Report