BLOG
Articles about AI agent readiness, web standards, and preparing for autonomous agents.
Content Negotiation for AI Agents: Why Sentry Serves Markdown Over HTML
Sentry co-founder David Cramer shows how content negotiation — a 25-year-old HTTP standard — saves AI agents 80% of tokens. We break down the implementation: Accept headers, markdown delivery, authenticated page redirects, and what this means for every website preparing for agent traffic.
Cloudflare /crawl Endpoint: One API Call to Crawl Any Website
Cloudflare launched a /crawl endpoint that crawls entire websites with one API call — returning HTML, Markdown, or AI-extracted JSON. We break down what this means for AI agent readiness: why your robots.txt, sitemap, semantic HTML, and server-side rendering now matter more than ever.
AI Crawlers Ignore llms.txt — But AI Agents Don't
Dries Buytaert's data shows zero AI crawlers use llms.txt. But he measured the wrong thing. Crawlers scrape for training data — agents complete tasks. We break down why the crawler vs agent distinction matters, which coding agents already use llms.txt and content negotiation, and what you should implement today.
Anthropic's AI Exposure Index: What Real-World Usage Data Means for Your Website
Anthropic's new 'observed exposure' metric reveals a 61-point gap between theoretical AI capability and actual usage. We break down the data — from 75% task coverage for programmers to 14% hiring slowdowns for young workers — and explain why this adoption gap is a countdown for website AI agent readiness.
The State of AEO: Key Insights from Vercel's 2026 Report
Vercel's State of AEO report lays out the shift from the attention economy to the answer economy. We break down the key statistics, platform landscape, content architecture recommendations, and what it all means for your website's AI visibility.
SEO vs AEO: How Traditional Search Optimization Differs from AI Optimization
SEO optimizes for rankings. AEO optimizes for citations. We break down the key differences between traditional search optimization and the new discipline of Answer Engine Optimization, covering the Princeton GEO study, zero-click statistics, and practical strategies.
How ChatGPT Search Chooses Which Websites to Cite
We analyzed 485,000+ citations and OpenAI's official docs to reveal how ChatGPT selects sources. Domain authority, Q&A headings, FAQPage schema, and content freshness, ranked by evidence strength.
How Google AI Overviews Selects Sources to Cite
Google says 'no special requirements' for AI Overviews. The data says otherwise. We analyzed 500M+ keywords to reveal what actually gets cited, from query fan-out to E-E-A-T signals and the 68% opportunity.
How Claude Search Selects Sources to Cite
Claude's 86.7% alignment with Brave Search makes it the most transparent AI citation system. We break down three crawlers, dynamic filtering, and what makes Claude's source selection unique.
EXPLORE MORE
Most websites score under 45. Find out where you stand.