Is Your Site Agent-Ready? The 5-Category Framework Every Developer Needs to Check

Paste your URL into Is It Agent Ready . In thirty seconds, you'll know how invisible your site is to the AI agents already browsing it. Most sites fail every single category. Not because they blocked agents — but because they never declared themselves.

The tool runs a structured scan across five categories — discoverability, content accessibility, bot access control, protocol discovery, and commerce — and returns a score against a growing set of emerging agent standards. The checklist is blunt and useful: it measures the gap between where your site is today and where the ecosystem is already heading.

That gap is not a cosmetic problem. As AI agents become the primary interface through which people and automated systems access content, make purchases, and trigger workflows, the sites that are not agent-ready will simply be invisible to them. This is the SEO story of the agentic web — except the consequences are architectural, not just ranking-based.

Why Agent-Readiness Is the New Technical Debt

The shift from browser-first to agent-first consumption is not a future scenario — it is already happening. According to BrightEdge's 2026 data , AI agent HTTP requests now represent 88% as much traffic as human organic search, and Human Security's State of AI Traffic report found that automated traffic grew nearly 8,000% in 2025 alone. Cloudflare's CEO Matthew Prince projected that by 2027, AI bots will surpass human traffic entirely. The transition is happening the same way mobile overtook desktop: quietly, then all at once.

The web was not designed for this. HTML was designed for humans. Robots.txt was designed for crawlers. Neither was designed for agents that need to authenticate, negotiate content formats, discover available tools, and initiate transactions — all before they can do useful work. The standards gap is real, and the sites that close it early will have a structural advantage that compounds over the next two to three years.

Agent-readiness is not an optimization. It is a new baseline — and most of the web is not there yet.

The Five Categories That Actually Matter

The Is It Agent Ready scanner maps agent-readiness across five structured layers. Each one represents a different dimension of the agent interaction model. Missing any single layer does not break everything — but missing several makes your site opaque to the agents that matter most.

Discoverability

The first check is the most forgiving — and the most commonly failed. A valid robots.txt with explicit AI bot directives, a sitemap link, and useful Link response headers are the minimum that tell an agent where it is, where it can go, and what rules apply. Many sites have a robots.txt written for the GoogleBot era that says nothing useful about AI clients. The scanner checks specifically for AI bot rules — GPTBot, ClaudeBot, PerplexityBot — not just the legacy search crawler list. Explicit is better than inherited.

Content Accessibility

AI agents do not want to parse your HTML. They want your content in a format that maps cleanly to language model input — ideally Markdown. The scanner checks whether your server supports content negotiation for text/markdown — meaning an agent can send Accept: text/markdown in its request headers and receive a clean Markdown response instead of a full HTML document with navigation, ads, cookie banners, and JavaScript payloads. This is not yet widely implemented, but it is the right direction: separating content from presentation at the protocol level, for the benefit of non-human consumers.

Bot Access Control

This category is where the ethical and commercial dimensions of agent-readiness intersect. The scanner checks not just whether your robots.txt mentions AI crawlers, but whether your site implements Web Bot Auth — an emerging pattern that lets site owners require authentication tokens from AI bots before granting access. It also looks for content signals: structured declarations about licensing, usage rights, and synthetic content warnings. The sites getting this right are building a foundation for a world where AI access to content is a negotiated relationship, not an assumed entitlement.

Most sites either block all AI access indiscriminately or allow it unconditionally. Neither is the right answer for a web where AI agents are becoming paying customers.

Protocol Discovery

This is the most technically interesting category — and the one most developers have not yet encountered. It checks for the presence of an MCP Server Card, an Agent Skills manifest, a WebMCP endpoint, an API catalog, and OAuth discovery documents. These signals tell agents what capabilities your site exposes — not just what content it has. An MCP Server Card, for instance, is a structured declaration that your site operates a Model Context Protocol server: agents can discover it, authenticate against it, and invoke its tools programmatically. If your site exposes useful actions — search, booking, data retrieval — and you have not published an MCP Server Card, agents cannot find them.

Agent Skills — the filesystem-based capability modules introduced by Anthropic in October 2025 — represent a similar discovery layer. A site that publishes an Agent Skills manifest is telling the agent ecosystem: here is a structured description of how to interact with us. The agent does not need to guess, infer, or hallucinate the interface. The interface is declared.

Commerce

This is the frontier category — the one where most sites score zero and where the stakes are highest. The scanner checks for x402 (a HTTP payment protocol that lets agents pay for access at the request level), MPP (Monetization Policy Protocol), UCP (Usage and Copyright Policy), and ACP (Agent Commerce Protocol). Together, these protocols represent an emerging infrastructure for a web where AI agents can autonomously initiate and complete transactions — purchasing content licenses, paying API access fees, or triggering subscriptions — without a human in the loop. None of these standards is fully mature, but the direction is clear: the agentic web will need a payment layer that operates at machine speed.

x402 is not a prediction. Coinbase and Cloudflare announced the x402 Foundation in September 2025, and Vercel shipped x402-mcp — bringing native payment support into MCP servers — the same month. Cloudflare's Pay Per Crawl beta already uses it to charge AI crawlers for content access. The question is when — not whether — it reaches mainstream adoption.

Start With the Easy Wins

Not everything on the checklist requires architectural investment. Several high-impact improvements can be shipped in an afternoon — and together they cover the most commonly failed checks. Here is the priority order:

Update your robots.txt with explicit rules for GPTBot, ClaudeBot, PerplexityBot, and other major AI crawlers. Add a sitemap reference if it is missing. Effort: 15 minutes. Impact: high.
Add Link headers to your HTTP responses — canonical URL, sitemap, API docs. A Cloudflare Worker or a few lines of middleware can handle this without touching your application logic. Effort: 1 hour. Impact: medium-high.
Publish an MCP Server Card at /.well-known/mcp.json — a JSON document declaring your MCP server endpoint, its available tools, and its authentication requirements. Once published, any agent that reads MCP Server Cards (Claude, Cursor, and a growing list of others) can discover and invoke your tools automatically. Effort: half a day. Impact: high for tool-exposing sites.
Add content licensing declarations — a structured statement in your page metadata about usage rights for AI training and inference. This is increasingly required by content licensing frameworks and is a one-line addition to your HTML head. Effort: 30 minutes. Impact: medium.

The sites that will be invisible to agents are not the ones that actively block them. They are the ones that simply never declared themselves.

Or Let a Coding Agent Handle It

There is something quietly ironic about manually implementing standards designed for automated agents. You do not have to. Every item on this checklist is mechanical enough that a coding agent — Claude, Cursor, GitHub Copilot Workspace, or any capable tool you already use — can implement it from a single well-formed prompt. The hard part is knowing what to ask for. Here is the translation layer.

The prompts below are written to be dropped into a coding agent with minimal context. They assume a generic web stack and ask for complete, production-ready output. Adjust the stack references as needed — the intent and the spec references are what matter.

Prompt: Discoverability

Generate a complete robots.txt file for a public web application that wants to be fully accessible to AI agents. Include explicit Allow directives for GPTBot, ClaudeBot, PerplexityBot, Applebot-Extended, and GoogleExtended. Add a Sitemap reference pointing to /sitemap.xml. Then write a middleware snippet (for [your framework]) that injects the following HTTP Link headers on every response: rel="canonical" pointing to the current URL, rel="sitemap" pointing to /sitemap.xml, and rel="describedby" pointing to /api-docs (if an API exists). Follow the emerging best practices for agent discoverability as documented at isitagentready.com.

Prompt: Content Accessibility

Add content negotiation support to my [framework] application so that any route serving HTML content also supports returning clean Markdown when the request includes the header Accept: text/markdown. The Markdown response should strip navigation, ads, footers, and any non-content elements. Preserve headings, body text, code blocks, and links. Return the correct Content-Type: text/markdown; charset=utf-8 header. If the page is rendered server-side, extract the main content element before converting. Use a battle-tested HTML-to-Markdown library appropriate for my stack.

Prompt: Bot Access Control

Implement Web Bot Auth for my web application. I want to require AI crawlers to present a valid bearer token before accessing content. Generate: (1) middleware that detects known AI bot user agents (GPTBot, ClaudeBot, PerplexityBot, etc.) and checks for a Bearer token in the Authorization header, returning 401 with a WWW-Authenticate header pointing to my token registration endpoint if missing; (2) a /.well-known/bot-auth.json endpoint that describes my authentication requirements following the Web Bot Auth draft spec; (3) meta tags for the HTML head that declare content licensing as ai-disallowed-training, ai-allowed-inference following the proposed standard at isitagentready.com/specs/ucp.

Prompt: Protocol Discovery

Generate a complete MCP Server Card for my web application and serve it at /.well-known/mcp.json. The card should follow the MCP Server Card specification at modelcontextprotocol.io. Include: the server name and description, the MCP endpoint URL (at /mcp), the supported protocol version, a list of available tools with their names and descriptions (I will fill in the actual tools), and OAuth 2.0 discovery pointing to /.well-known/oauth-authorization-server. Also add a Link: </.well-known/mcp.json>; rel="mcp-server-card" header to all HTTP responses via middleware.

Prompt: Commerce

Add x402 payment support to my API so that premium endpoints return HTTP 402 with the correct x402 payment details when accessed without a valid payment credential. Follow the x402 specification at x402.org. Generate: (1) middleware that intercepts requests to protected routes and checks for a valid X-Payment header; (2) a 402 response body in the x402 JSON format specifying the price in USDC, the payment network, and the facilitator URL; (3) a /.well-known/monetization.json file following the Monetization Policy Protocol (MPP) spec that describes my site's payment requirements for AI agent access. Make the implementation framework-agnostic and clearly mark where I need to plug in my payment facilitator credentials.

The checklist exists to be automated. Run these prompts, review the output, ship the files. The agents will find you.

The Bigger Picture

The Is It Agent Ready project is useful precisely because it turns an abstract concern — "is the agentic web coming for my site?" — into a concrete, actionable score. That concreteness is what makes it worth taking seriously. It is not predicting the future; it is measuring the present gap between where your site is and where the ecosystem is heading.

The deeper point is architectural. The web was built in layers: HTML on top of HTTP on top of TCP/IP. Each layer solved a different problem and created a platform for the next. The agent-readiness stack — MCP, OAuth protected resources, Agent Skills, x402, WebMCP — is a new layer being built on top of HTTP. It is happening faster than most people realize, largely because the primary beneficiaries (AI agents) can adopt new standards at software speed, not at human adoption speed.

The developers building agent-native services today are not ahead of the curve in a speculative sense. They are solving real problems that real agents encounter right now — and building infrastructure that will be table stakes within 18 months. If you want to understand the shape of what is coming, scan your own site. The gap between your current score and a full pass is a direct measure of the work ahead.