Content negotiation for AI agents serves one page as two different formats
Content negotiation is the decades-old HTTP mechanism that lets a single URL return different representations of the same page depending on who is asking. Applied to AI agents, it means a server reads the request's Accept header and serves a browser the full HTML page while serving an agent that explicitly asks for text/markdown a cleaner, lighter version of the same content. It is a separate layer from llms.txt, which only lists which pages exist rather than changing what any one of them returns.
By Rish Sadh, founderUpdated
Definition
Content negotiation for AI agents is the practice of using the HTTP Accept header on an incoming request to decide what format a server sends back for the same URL: full HTML with navigation, styling and scripts for a browser, and a stripped-down text/markdown response for an agent that asks for it directly. The response carries a Vary: Accept header so caches know the body depends on the request, and nothing about the URL itself has to change for this to work.
What actually happens on the request
Content negotiation is not a new idea invented for AI. The underlying HTTP mechanism, a client stating what representation it prefers and a server honouring that preference, has existed since the earliest HTTP specifications and is normally invisible because almost every request simply asks for HTML.
What changed is who is sending the request. An AI agent fetching a page to answer a question or complete a task can send an Accept header asking specifically for text/markdown instead of text/html. A server set up to honour that strips out the layout, the navigation chrome and the client-side scripting, and returns the same information as plain, well-structured markdown: headings, lists and links, with none of the rendering weight an agent has no use for.
Request from an AI agent:
GET /docs/getting-started HTTP/1.1
Accept: text/markdown
Response:
HTTP/1.1 200 OK
Content-Type: text/markdown; charset=utf-8
Vary: Accept
# Getting started
...plain markdown body...Why this is not the same thing as llms.txt
llms.txt is a plaintext index file at a site's root that lists its key pages, so an agent knows where to look before it fetches anything. Content negotiation operates one layer deeper: it decides what a single one of those URLs actually returns once the agent arrives, based on what that agent asked for in the request itself.
A site can have one without the other. An llms.txt file with no content negotiation behind it still points an agent at the right pages, which then arrive as ordinary HTML. Content negotiation with no llms.txt still serves markdown to any agent that asks for it correctly, even if nothing told that agent the page existed in the first place. The two solve different problems and are usually built together rather than as alternatives. Read what llms.txt actually does for the discovery half of this.
97%
of llms.txt files examined in server logs received zero requests in May 2026, which is one reason a discovery file alone is not the same guarantee as a server actually changing what it serves.
Ahrefs, llms.txt server-log analysis
Nothing about the address changes. What changes is what the server decides to hand back, and to whom.
The fallback paths when an agent sends no header at all
Most AI agents today do not reliably send an Accept header that requests markdown, which is the practical gap in relying on content negotiation alone. Coverage of the practice from early 2026 describes adoption on both the serving and the requesting side as still low and uneven, so a site that wants agents to receive the lighter version usually builds more than one path to it.
Three fallbacks are in common use. A server can detect a known AI agent by its user agent string and serve markdown to it directly, without waiting for the right Accept header. A page can carry a <link rel="alternate" type="text/markdown"> tag in its head, so an agent that inspects the page's own markup can discover the alternate version for itself. Or a site can simply let any URL that ends in .md return the markdown copy directly, which needs no header negotiation at all because the format is stated in the address.
None of these fallbacks make the Accept-header mechanism unnecessary; they exist precisely because the mechanism depends on an agent doing something most agents do not yet reliably do. That is a statement about today's adoption, not about whether the underlying HTTP mechanism works. Checking whether a crawler can read a page at all is still the more basic question most sites should answer first.
Common questions
What is content negotiation for AI agents?
It is a server reading the Accept header on an incoming request and returning a different version of the same page depending on what was asked for: full HTML for a browser, and a plainer text/markdown document for an agent that explicitly requests that format.
Is this the same thing as llms.txt?
No. llms.txt is a discovery file that lists a site's key pages so an agent knows where to look. Content negotiation controls what a single URL actually returns once an agent gets there. The two solve different problems and work together rather than competing.
Do most AI agents actually send the header needed for this to work?
Not reliably yet. Coverage of the practice from early 2026 describes adoption as still low and uneven, with most crawlers and assistants not consistently sending an Accept header that requests markdown, which is why sites that bother with this also build a fallback.
What are the fallback options if an agent never sends the header?
Three common ones: detect known AI user agents and serve markdown to them directly, add a <link rel="alternate" type="text/markdown"> tag in the page head so an agent can discover the alternate version, or simply let a URL ending in .md return the markdown copy.
Does a typical business site need to build this?
Usually not yet. It matters most for documentation-heavy sites that AI coding agents and assistants are built to parse directly. For a normal business page, clean semantic HTML and accurate structured data do more for AI readability than adding a parallel markdown endpoint most agents will never request.
AI-readable websites
The hub this discovery-and-format layer sits inside.
What is llms.txt
The discovery file content negotiation is often paired with.
What is a mixed-use crawler
Why the agent on the other end of the request is not always what it seems.
AI crawler user agents
The named bots a user-agent fallback actually has to detect.
See whether your own site already serves anything differently to an AI agent than it does to a browser.