Cloudflare /crawl Endpoint: One API Call to Crawl Any Website

Cloudflare just launched a new /crawl endpoint for its Browser Rendering service. You send one POST request with one URL, and Cloudflare crawls the site and returns HTML, Markdown, or structured JSON extracted by AI. The endpoint entered open beta on March 10, 2026, and is available on both free and paid Workers plans.

For AI agent readiness, this matters because anyone with a Cloudflare account can now build a crawler for whole sites with little code. That makes it easier to test how machine-readable your site is.

How the /crawl Endpoint Works

The endpoint is asynchronous. You submit a starting URL, get back a job ID, and poll for results as pages complete. Cloudflare designed it for full-site crawls, not single-page fetches.

# 1. Start a crawl
curl -X POST \
  https://api.cloudflare.com/client/v4/accounts/{account_id}/browser-rendering/crawl \
  -H "Authorization: Bearer {token}" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://example.com", "limit": 100, "formats": ["markdown"]}'

# Response: {"success": true, "result": "c7f8s2d9-a8e7-..." }

# 2. Poll for results
curl https://api.cloudflare.com/client/v4/accounts/{account_id}/browser-rendering/crawl/{job_id}

The main numbers:

Three Output Formats and Why Markdown Matters

The /crawl endpoint supports three output formats, each for a different use case:

The Markdown option matters most for AI agent readiness. When an agent crawls your site with formats: ["markdown"], it gets clean content that fits into the context window of an LLM. Sites with semantic HTML, a proper heading hierarchy, and meaningful content structure should produce better markdown than sites built from JavaScript-rendered widgets and nested divs.

The JSON format uses Workers AI to extract structured data with a custom prompt and schema. In effect, it is structured data extraction at crawl scale:

{
  "url": "https://example.com/products",
  "limit": 500,
  "formats": ["json"],
  "jsonOptions": {
    "prompt": "Extract product name, price, and availability",
    "response_format": {
      "type": "json_schema",
      "json_schema": {
        "name": "product",
        "properties": {
          "name": "string",
          "price": "number",
          "inStock": "boolean"
        }
      }
    }
  }
}

Crawl Scope and Discovery

You control which pages the endpoint crawls and how it discovers URLs:

The source parameter shows how discovery works. Setting it to sitemaps means the crawler finds pages through your XML sitemaps only, the way search engine crawlers do. A site with a complete, current sitemap gets crawled more fully. A site without a sitemap falls back to link discovery, which may miss orphaned pages.

The render toggle changes what the crawler sees. Setting render: false skips the headless browser and fetches static HTML. This is faster and cheaper, but JavaScript-rendered content is invisible. A site that renders its main content on the client returns empty pages in static mode. A site that renders on the server returns its full content.

robots.txt Controls What the Crawler Can Reach

The /crawl endpoint respects robots.txt fully, including crawl-delay directives. URLs blocked by robots.txt appear in results with "status": "disallowed". In practice:

• If you block AI crawlers, the /crawl endpoint does not access those pages
• If your robots.txt is misconfigured, you may block legitimate agent access without knowing it. Some sites block all bots to stop training data scraping and lose agent access as a side effect
• If you set crawl-delay, the endpoint follows it, so you can limit the request rate

The endpoint operates as a signed bot. It identifies itself as automated and cannot bypass bot detection, CAPTCHAs, or Cloudflare protection. This is the kind of crawler that robots.txt was designed for.

What This Means for AI Agent Readiness

Cloudflare's /crawl endpoint is infrastructure, not an agent. It lowers the effort needed to build agent systems that read entire websites. Four points follow from that:

Agent Readiness Checklist for the /crawl Era

Because this crawling infrastructure is easy to use, prioritize these steps:

1. Audit your robots.txt. Check that you do not block compliant AI crawlers by accident. You can block training bots and still keep agent access open
2. Maintain your sitemap. The source: "sitemaps" option makes your sitemap the input that decides how completely the crawler covers your site
3. Use server-side rendering. The render: false option is faster and cheaper. A site that works without JavaScript is crawled more efficiently
4. Improve semantic HTML. A clean heading hierarchy, proper landmarks, and descriptive link text all lead to better markdown output when the crawler visits your site
5. Add structured data. JSON-LD and Schema.org types can help both the HTML-to-markdown conversion and the AI-powered JSON extraction return accurate results
6. Serve llms.txt. The /crawl endpoint uses sitemaps and links for discovery, but agents that consume the crawled content may start with llms.txt to learn what a site offers

Where /crawl Fits

Cloudflare already offers MCP integration through its Workers AI platform, and supports Playwright MCP for browser automation. The /crawl endpoint adds scalable, compliant, full-site crawling with output formats aimed at AI use.

Crawlers used to scrape HTML for search indexes. Now agents also crawl to build knowledge bases, RAG pipelines, and structured datasets. Sites that are already machine-readable should produce better results with these tools.

The practical question is whether the output of an AI-powered crawl represents what your site offers.

Sources

• Cloudflare Changelog: Browser Rendering /crawl Endpoint (Open Beta) (official announcement, March 10, 2026)
• Cloudflare Docs: /crawl Endpoint Technical Documentation (full API reference with configuration options)
• Cloudflare Browser Rendering Overview (product overview and use cases)
• IsAgentReady: AI Crawlers Ignore llms.txt, But AI Agents Don't (why the crawler vs agent distinction matters)