Agent readiness checklist: 9 signals an AI agent reads on your website
An AI agent visiting your site is a fast, literal reader with a small window. It cannot run your JavaScript app, wait for a modal, or infer what you do from a logo. These are the nine signals our readiness check looks at, in the order an agent meets them.
Access
1. Crawl permissions (robots.txt)
Why: polite agents and every AI provider crawler read robots.txt first. A blanket Disallow: / or a rule that blocks unknown user agents stops them before your homepage. Fix: publish an explicit policy. Allow identified crawlers on public pages, disallow private paths. If you want to block specific AI training crawlers, do it by name rather than blocking everything.
2. Homepage access
Why: the homepage must return HTTP 200 with HTML over valid HTTPS. Bot challenges ("checking your browser"), rate-limit pages and login walls all read as "no content". Fix: exempt identified crawlers from challenge pages, or serve a static fallback for them.
Discovery
3. Indexing directives
Why: a noindex meta tag or X-Robots-Tag header tells crawlers to forget the page. Staging settings left on production are the usual cause. Fix: remove noindex from public pages.
4. Page identity (title and meta description)
Why: the title and description are the two strings most agents quote when they summarize you. Fix: write a specific title ("Company enrichment API for sales agents") and a one-sentence description that says the task you solve.
5. Navigable links
Why: agents follow ordinary <a href> links to docs and pricing. Links that exist only as JavaScript click handlers are invisible. Fix: real links to docs, pricing, API reference and contact.
6. Sitemap
Why: /sitemap.xml is how crawlers find pages nobody links to. Fix: generate one and reference it in robots.txt.
7. llms.txt
Why: the curated map agents read first. See the llms.txt guide. Fix: one Markdown file at the root.
Extraction
8. Server-readable content
Why: if the initial HTML has fewer than a couple of hundred characters of text, everything you sell lives in a JavaScript bundle the agent will not execute. Fix: server-render the product summary, headings and the first paragraph of the docs. A static pre-render is enough.
9. Structured data (JSON-LD)
Why: a SoftwareApplication, Product or Organization block gives agents typed facts: name, category, price, URL. Malformed JSON-LD is worse than none. Fix: one valid block in the head, validated.
Beyond the checklist
Passing all nine means an agent can read you. It does not mean an agent will choose you. For that, list a real tool on the surfaces agents search (see how to list on MCP registries) and measure who arrives, separating crawlers from identified AI providers from unknown automation.