llms.txt: what it is, how to write one, and how agents use it

By Agent Observatory · Updated 2026-09-23 · Reading time about 6 minutes

llms.txt is a plain Markdown file served at the root of your site (https://example.com/llms.txt) that gives language-model agents a short, curated map of your product. The convention was proposed by Jeremy Howard in 2024 at llmstxt.org. Where robots.txt says what crawlers may fetch and sitemap.xml lists everything, llms.txt says what matters and where to start.

Why it matters

• Agent frameworks and research agents fetch it first because it fits in a prompt. A good file saves an agent ten page loads.
• AI search crawlers use it to summarize what you do without guessing from navigation menus.
• It is the cheapest agent-readiness fix: one static file, no code changes.

Format

The convention is simple: an H1 with the product name, a blockquote summary, then sections of links with one-line descriptions. Optional sections can be marked so smaller agents skip them.

# Acme Enrichment API

> Company and contact enrichment for sales and research agents. REST and MCP. Free tier of 500 lookups a month.

## Docs
- [Quickstart](https://acme.example/docs/quickstart): first call in five minutes
- [API reference](https://acme.example/docs/api): endpoints, parameters, errors
- [Authentication](https://acme.example/docs/auth): API keys, rate limits (60 requests a minute)

## For agents
- [MCP server](https://mcp.acme.example/mcp): streamable HTTP, tools: enrich_company, find_contacts
- [OpenAPI](https://acme.example/openapi.json)
- [Pricing](https://acme.example/pricing): free 500/month, then $0.02 per lookup

## Optional
- [Changelog](https://acme.example/changelog)
- [Status](https://status.acme.example)

What to include

• One sentence on what the product does and for whom. Agents match tasks to tools; say the task.
• The canonical docs, not the marketing pages.
• Machine endpoints: MCP server URL, OpenAPI file, agent card, webhooks.
• Pricing, limits and auth, so an agent can tell its operator what signing up involves.
• A contact or handoff link for the human behind the agent.

What to leave out

• Marketing copy, testimonials, hero slogans.
• Dumps of every page. Use llms-full.txt for the long version and keep llms.txt to one screen.
• Anything behind a login, and anything you would not want quoted.

Serve it correctly

• Content type text/plain or text/markdown, UTF-8, HTTP 200. An HTML error page at /llms.txt is worse than no file.
• Allow it in robots.txt. Some sites block everything under a wildcard rule and then wonder why agents never read the file.
• Link it from the homepage: <link rel="alternate" type="text/plain" href="/llms.txt">.
• Keep it current. A stale pricing line is a support ticket waiting to happen.

Check it

Our free readiness check requests /robots.txt, your homepage, /sitemap.xml and /llms.txt with an identified crawler and reports whether each one is present, permitted and readable. It is a point-in-time sample, not an audit, and it never launches a language-model agent against your site.