Content Negotiation for AI Agents: Why Sentry Serves Markdown Over HTML

David Cramer, co-founder and CPO of Sentry, recently published a practical guide to optimizing content for AI agents . His argument is simple: "you should be optimizing content for agents, just as you optimize things for people." Not with new protocols or complex infrastructure — with HTTP content negotiation, a standard that's been in the spec since 1999.

The technique is straightforward. When a request includes Accept: text/markdown, the server returns clean markdown instead of a full HTML page. Navigation, JavaScript, sidebar widgets — all stripped. Just the content the agent actually needs.

What Is Content Negotiation?

Content negotiation is an HTTP mechanism where a client tells the server what format it prefers, and the server delivers accordingly. It's defined in RFC 9110 and has been part of the web since HTTP/1.1. Browsers have always used it — when your browser sends Accept: text/html, the server knows to return HTML.

The insight is that AI agents can use the same mechanism. When an agent sends Accept: text/markdown, the server can return a stripped-down markdown version of the page. Same URL, same content, different representation. No separate API, no special endpoints, no new protocols needed.

GET /docs/getting-started HTTP/1.1
Host: docs.example.com
Accept: text/markdown, text/html;q=0.9

---

HTTP/1.1 200 OK
Content-Type: text/markdown; charset=utf-8
Vary: Accept

# Getting Started

Install the SDK with your package manager:

```bash
npm install @example/sdk
```

The Three Optimization Areas

Cramer identifies three areas where content optimization matters most for agents. These aren't theoretical — they're based on how frontier models actually process web content:

Together, these explain why serving HTML to an agent is wasteful. A typical documentation page might be 15,000 tokens as HTML but just 3,000 tokens as markdown. The agent gets the same information — headings, paragraphs, code blocks, links — without the structural overhead.

How Sentry Implements Content Negotiation

Sentry's approach has three layers, each solving a different problem:

1. Documentation: Serve True Markdown

When an agent requests Sentry's documentation with Accept: text/markdown, the server returns the raw MDX source — the same content that generates the HTML page, but without the rendered navigation, JavaScript bundles, or UI chrome. The link hierarchy is restructured so agents can discover related pages through a flat sitemap rather than crawling nested navigation trees.

This is the most broadly applicable pattern. Any site with documentation, help articles, or structured content can implement the same approach: detect the Accept header and return a stripped representation.

2. Authenticated Pages: Redirect to Programmatic Alternatives

When an agent hits an authenticated page in the Sentry web UI — a dashboard, issue detail, or project settings — Sentry doesn't just return a login wall. Instead, it redirects the agent toward programmatic alternatives: the MCP server, the CLI tool, or the REST API. The agent gets pointed to where it can access the data, rather than being stuck at a page it can't.

3. Bootstrap Discovery: Let Agents Self-Configure

For Sentry's CLI tools and integrations, agents can fetch a markdown document that describes the tool's purpose, capabilities, and quick-start instructions. This lets an agent bootstrap itself — understanding what a tool does and how to use it — without parsing HTML documentation pages.

This is the same principle behind llms.txt — providing a curated, machine-readable entry point. Content negotiation is the transport layer; the curated content is the value layer.

Who's Already Doing This?

Sentry isn't alone. Content negotiation for agents is becoming a pattern across the developer tools ecosystem:

The pattern is emerging on both sides: content providers are starting to serve markdown, and AI agents are starting to request it. Early movers have a compounding advantage — agents that get better results from your content will return more often.

How to Implement Content Negotiation

The implementation is simpler than you might think. Here's a practical approach:

Step 1: Detect Agent Requests

Check the Accept header for text/markdown. Most web frameworks make this straightforward:

defmodule MyAppWeb.Plugs.ContentNegotiation do
  import Plug.Conn

  def init(opts), do: opts

  def call(conn, _opts) do
    accepts = get_req_header(conn, "accept")

    if Enum.any?(accepts, &String.contains?(&1, "text/markdown")) do
      assign(conn, :prefer_markdown, true)
    else
      assign(conn, :prefer_markdown, false)
    end
  end
end

function contentNegotiation(req, res, next) {
  req.prefersMarkdown = req.accepts(['text/markdown', 'text/html'])
    === 'text/markdown';
  next();
}

app.get('/docs/:slug', contentNegotiation, (req, res) => {
  if (req.prefersMarkdown) {
    res.type('text/markdown').send(getMarkdownContent(req.params.slug));
  } else {
    res.render('docs', { slug: req.params.slug });
  }
});

Step 2: Serve Clean Markdown

Strip everything that isn't content: navigation, headers, footers, JavaScript, tracking pixels, cookie banners. What remains should be the same content that appears in the rendered page, but as clean markdown with proper heading hierarchy.

• Do include: headings, paragraphs, code blocks, links, lists, images with alt text, tables
• Don't include: navigation HTML, script tags, style blocks, tracking pixels, UI framework markup
• Do set: Content-Type: text/markdown and Vary: Accept headers

The Vary: Accept header is important — it tells caches and CDNs that the response varies based on the Accept header, preventing a cached markdown response from being served to browsers.

Step 3: Front-Load Key Information

Since agents read sequentially and may truncate long content, put the most important information first:

• Start with a clear title and one-paragraph summary
• Put installation/quick-start instructions near the top
• Move prerequisites, background context, and changelog to the bottom
• Use descriptive headings that make sense without reading the content below them

Beyond Developer Docs

Content negotiation started in developer documentation, but the pattern applies to any content-heavy website. Consider:

As AI agents expand beyond coding into browsing, shopping, and research tasks, any website with valuable content stands to benefit. The question isn't whether agents will consume your content — it's whether they'll do it efficiently or wastefully.

How This Fits into AI Agent Readiness

Content negotiation is one piece of a broader agent readiness strategy. It works best in combination with the other signals agents look for:

The Bottom Line

David Cramer's post cuts through the noise around AI agent optimization. You don't need new protocols or complex infrastructure. You need content negotiation — a 25-year-old HTTP standard — applied to a new audience.

The implementation is simple: detect Accept: text/markdown, return clean markdown, front-load your key content. For authenticated pages, redirect agents to APIs and CLIs instead of login walls. For discovery, provide a curated entry point via llms.txt .

The companies building for agents today — Sentry, Vercel, Cloudflare — aren't using exotic new technology. They're using standards that already exist. The gap isn't technical. It's awareness. And that gap is closing fast.

Sources

• David Cramer: Optimizing Content for Agents — The original post on Sentry's content negotiation approach
• RFC 9110: HTTP Semantics — Content Negotiation — The HTTP specification for content negotiation
• How Vercel Built AEO Tracking for Coding Agents — Vercel's approach to measuring agent traffic
• Cloudflare: Introducing Markdown for Agents — Platform-level content negotiation and markdown transformation
• IsAgentReady: AI Crawlers Ignore llms.txt — But AI Agents Don't — Why llms.txt matters for agents, not crawlers
• Checkly: The Current State of Content Negotiation for AI Agents — Survey of which coding agents actually send Accept: text/markdown headers
• Cloudflare: Introducing Markdown for Agents — Benchmark showing 80% token reduction (16,180 HTML tokens to 3,150 markdown)
• llms.txt Specification — The original proposal by Jeremy Howard