On Serving Markdown to AI Agents

Recently the idea of serving Markdown to AI agents has gained traction. Cloudflare wrote about it and built a dashboard toggle to convert responses to Markdown automatically, using content negotiation to detect whether the requesting client wants Markdown or HTML.

Spatie then released laravel-markdown-response, a Laravel package that does much the same thing - but goes a step further by detecting AI agents via User Agent and serving them Markdown by default, even without an explicit Accept: text/markdown header.

Why this is interesting

AI is increasingly how people interact with the web, and AI responses are increasingly citing sources - meaning click-throughs from AI are becoming a meaningful traffic source, even if lower volume than traditional search.

This creates a new SEO-adjacent consideration: if your content is easier for AI to parse, does it get cited more often? Hugely speculative, but then so was a lot of early SEO thinking.

There’s also a practical bandwidth argument. AI crawlers can hammer a server, and if serving raw Markdown is cheaper to generate and transfer than a full HTML page - especially if Markdown is already your source format - that’s a worthwhile optimisation.

But ChatGPT can’t handle text/markdown

I tried implementing a similar mechanism in a project I’m working on: serve raw Markdown if the client sent Accept: text/markdown, or if the User Agent matched a known AI bot. Otherwise, serve normal HTML.

I was pleased with the results: If nothing else it’s cool and fun to do. However, within a day the editorial team complained. They had asked ChatGPT to summarise an article on the site and got:

I couldn’t load the text of the exact article you linked directly, but based on the parts that are visible in search results and summaries of that piece […]

I reverted the change, re-tested in a fresh conversation (ChatGPT caches responses and won’t re-fetch the same URL within a conversation), and it worked fine.

Clearly ChatGPT just can’t handle markdown responses. Not what I wanted.

I asked ChatGPT to explain the failure. The relevant part of its response (unbelievably it gave me a 5.5k character novel in response to a short question):

The browsing system I use does MIME-type gating before parsing. It expects:

• text/html
• application/xhtml+xml
• sometimes text/plain

When it sees Content-Type: text/markdown it typically […] rejects the response as non-renderable, aborts parsing [and] surfaces a generic “failed to fetch” error.

This is not a limitation of “AI” in general. It’s a limitation of the controlled browsing sandbox’s MIME policy.

If you pasted the Markdown directly into chat, I could read and parse it instantly. But when retrieving via HTTP, strict MIME validation applies.

Conclusion

The idea has merit - reduced bandwidth, simpler parsing, potential AI visibility benefits. But don’t rely on User Agent sniffing to decide who gets Markdown without first verifying that each bot on your list can actually handle a text/markdown response. ChatGPT, at least for now, cannot.