Content Negotiation for AI Agents: Why Sentry Serves Markdown Over HTML

David Cramer, co-founder and CPO of Sentry, recently published a practical guide to optimizing content for AI agents . His argument: "you should be optimizing content for agents, just as you optimize things for people." His method needs no new protocol. It uses HTTP content negotiation, a standard that has been in the specification since 1999.

When a request includes Accept: text/markdown, the server returns clean markdown instead of a full HTML page. Navigation, JavaScript and sidebar widgets are stripped, and the agent receives only the content.

What Is Content Negotiation?

Content negotiation is an HTTP mechanism. The client tells the server which format it prefers, and the server responds in that format. It is defined in RFC 9110 and has been part of the web since HTTP/1.1. Browsers have always used it. When your browser sends Accept: text/html, the server knows to return HTML.

AI agents can use the same mechanism. When an agent sends Accept: text/markdown, the server can return a stripped-down markdown version of the page. The URL and the content stay the same, and only the representation changes. The server needs no separate API and no special endpoints.

GET /docs/getting-started HTTP/1.1
Host: docs.example.com
Accept: text/markdown, text/html;q=0.9

---

HTTP/1.1 200 OK
Content-Type: text/markdown; charset=utf-8
Vary: Accept

# Getting Started

Install the SDK with your package manager:

```bash
npm install @example/sdk
```

The Three Optimization Areas

Cramer names three areas where content optimization matters most for agents. He bases them on how frontier models process web content:

Together, these explain why serving HTML to an agent wastes tokens. A documentation page might be 15,000 tokens as HTML and 3,000 tokens as markdown. The markdown keeps the headings, paragraphs, code blocks and links, and drops the structural overhead.

How Sentry Implements Content Negotiation

Sentry's approach has three layers. Each one solves a different problem.

1. Documentation: Serve True Markdown

When an agent requests Sentry's documentation with Accept: text/markdown, the server returns the raw MDX source. This is the content that generates the HTML page, without the rendered navigation, JavaScript bundles or UI chrome. Sentry also restructures the link hierarchy, so agents find related pages through a flat sitemap and do not have to crawl nested navigation trees.

Of the three layers, this one applies to the most sites. Any site with documentation, help articles or structured content can detect the Accept header and return a stripped representation.

2. Authenticated Pages: Redirect to Programmatic Alternatives

When an agent requests an authenticated page in the Sentry web UI, such as a dashboard, an issue detail or project settings, Sentry does not return a login wall. It redirects the agent to a programmatic alternative: the MCP server, the CLI tool or the REST API. The agent learns where it can access the data.

3. Bootstrap Discovery: Let Agents Self-Configure

For Sentry's CLI tools and integrations, an agent can fetch a markdown document that describes the purpose, capabilities and quick-start steps of the tool. The agent learns what the tool does and how to use it without parsing HTML documentation pages.

This is the same principle behind llms.txt . That file gives agents a curated, machine-readable entry point. Content negotiation controls the format of a page, and llms.txt controls which pages the agent reads first.

Who's Already Doing This?

Other companies also use content negotiation with agents:

Both sides are moving. Some content providers now serve markdown, and some AI agents now request it.

How to Implement Content Negotiation

The implementation takes three steps.

Step 1: Detect Agent Requests

Check the Accept header for text/markdown. Compare the q-values instead of searching for the string, so that a browser that sends */* still gets HTML. Also set Vary: Accept on every response:

defmodule MyAppWeb.Plugs.ContentNegotiation do
  import Plug.Conn

  def init(opts), do: opts

  def call(conn, _opts) do
    conn
    |> put_resp_header("vary", "accept")
    |> assign(:prefer_markdown, prefers_markdown?(get_req_header(conn, "accept")))
  end

  # Markdown wins only when it has the highest q-value.
  # A browser that sends */* keeps getting HTML.
  defp prefers_markdown?(accept_headers) do
    preferred =
      accept_headers
      |> Enum.flat_map(&String.split(&1, ","))
      |> Enum.map(&parse_media_range/1)
      |> Enum.max_by(fn {_type, q} -> q end, fn -> {"*/*", 1.0} end)

    match?({"text/markdown", q} when q > 0, preferred)
  end

  defp parse_media_range(range) do
    [type | params] = range |> String.split(";") |> Enum.map(&String.trim/1)

    q =
      Enum.find_value(params, 1.0, fn
        "q=" <> value ->
          case Float.parse(value) do
            {q, _rest} -> q
            :error -> 0.0
          end

        _param ->
          nil
      end)

    {String.downcase(type), q}
  end
end

// List text/html first: for Accept: */* Express picks the first type,
// so browsers and plain curl keep getting HTML.
function contentNegotiation(req, res, next) {
  res.vary('Accept');
  req.prefersMarkdown = req.accepts(['text/html', 'text/markdown'])
    === 'text/markdown';
  next();
}

app.get('/docs/:slug', contentNegotiation, (req, res) => {
  if (req.prefersMarkdown) {
    res.type('text/markdown').send(getMarkdownContent(req.params.slug));
  } else {
    res.render('docs', { slug: req.params.slug });
  }
});

Step 2: Serve Clean Markdown

Strip everything that isn't content: navigation, headers, footers, JavaScript, tracking pixels, cookie banners. What remains should be the same content that appears in the rendered page, but as clean markdown with proper heading hierarchy.

• Do include: headings, paragraphs, code blocks, links, lists, images with alt text, tables
• Don't include: navigation HTML, script tags, style blocks, tracking pixels, UI framework markup
• Do set: Content-Type: text/markdown and Vary: Accept headers

The Vary: Accept header tells caches and CDNs that the response depends on the Accept header. Without it, a cache can serve a cached markdown response to a browser.

Test both representations with curl. The first request must return Content-Type: text/markdown, the second must return HTML, and both must include Vary: Accept:

curl -sI -H "Accept: text/markdown" https://docs.example.com/docs/getting-started
curl -sI https://docs.example.com/docs/getting-started

Step 3: Front-Load Key Information

Agents read sequentially and may truncate long content, so put the most important information first:

• Start with a clear title and one-paragraph summary
• Put installation/quick-start instructions near the top
• Move prerequisites, background context, and changelog to the bottom
• Use descriptive headings that make sense without reading the content below them

Beyond Developer Docs

Content negotiation started in developer documentation, but the same approach fits other content-heavy websites. Some examples:

As AI agents take on browsing, shopping and research tasks, more websites will receive agent requests. Those sites can choose to serve the content as markdown or as full HTML.

How This Fits into AI Agent Readiness

Content negotiation is one part of agent readiness. It works alongside the other signals agents look for:

The Bottom Line

David Cramer's post describes agent optimization with an existing tool: content negotiation, a 25-year-old HTTP standard, applied to agent requests.

To implement it, detect Accept: text/markdown, return clean markdown and put your key content first. For authenticated pages, redirect agents to APIs and CLIs instead of login walls. For discovery, provide a curated entry point with llms.txt .

Sentry, Vercel and Cloudflare build for agents with standards that already exist. The main obstacle is that few teams know about the mechanism.

Sources

• David Cramer: Optimizing Content for Agents (the original post on Sentry's content negotiation approach)
• RFC 9110: HTTP Semantics, Content Negotiation (the HTTP specification for content negotiation)
• How Vercel Built AEO Tracking for Coding Agents (Vercel's approach to measuring agent traffic)
• Cloudflare: Introducing Markdown for Agents (platform-level content negotiation and markdown transformation)
• IsAgentReady: AI Crawlers Ignore llms.txt, But AI Agents Don't (why llms.txt matters for agents, not crawlers)
• Checkly: The Current State of Content Negotiation for AI Agents (survey of which coding agents send Accept: text/markdown headers)
• Cloudflare: Introducing Markdown for Agents (benchmark showing 80% token reduction, from 16,180 HTML tokens to 3,150 markdown)
• llms.txt Specification (the original proposal by Jeremy Howard)