“Agent readiness” is a useful phrase only if we say which kind of agent we mean.
An automatic crawler, a fetch triggered by a ChatGPT user, and an agent operating a browser are three different clients. They can follow different access rules and have different JavaScript capabilities. A single score cannot prove that all three can read, understand, or cite a page.
My Agent Readiness Checker provides a technical baseline. It checks access controls, content available without a browser, and common discovery signals. The score is my diagnostic rubric, not an industry standard or a promise of AI visibility.
TL;DR
- Name the agent: Automatic crawlers, user-triggered fetches, and browser agents are different clients with different access rules and JavaScript capabilities, so one score cannot cover all three.
- Check two layers: Robots.txt expresses a preference to crawlers, but a CDN, firewall, or bot-management rule can still deny the request before content is returned.
- JavaScript varies: A raw fetch may only see the server response while a browser agent can render the page, so key content should be present in the initial HTML when practical.
- Access is not visibility: Technical readiness removes obvious barriers but does not make a page useful, indexed, recommended, or cited by any AI product.
- A scored baseline: The Agent Readiness Checker weights crawler access at 45 percent, content accessibility at 35 percent, and discovery signals at 20 percent, as a site-authored rubric rather than an industry standard.
Cloudflare’s Agent Readiness score
Cloudflare introduced its Agent Readiness score in April 2026. Its current framework looks beyond ordinary crawlability. It checks areas such as discoverability, content formats, bot access control, authentication, agent capabilities, and emerging commerce standards.
My checker has a narrower job. It helps a site owner spot common technical problems on a public page. It does not reproduce Cloudflare’s score, and the weights should not be cited as Cloudflare’s methodology.
That distinction matters because “ready for an agent” can mean several things. A content crawler may only need a fetchable page. A purchasing agent may need an API, authentication flow, and a transaction protocol.
Three kinds of access
Automatic crawlers
Automatic crawlers visit pages on a provider’s schedule. Examples include OAI-SearchBot for ChatGPT search and GPTBot for content that may be used in OpenAI model training. Providers publish separate user-agent and IP information, and their purposes are not interchangeable.
Robots.txt can express a preference to these crawlers. A CDN, firewall, or bot-management rule can still deny the request before useful content is returned.
User-triggered fetches
Some visits happen because a person asked an AI product to open a page. OpenAI documents ChatGPT-User for certain user actions in ChatGPT and Custom GPTs. It is not an automatic crawler and is not used to determine whether a site may appear in ChatGPT search. OpenAI also notes that robots.txt rules may not apply because the action comes from a user.
This is why a robots.txt matrix cannot describe every possible ChatGPT visit.
Browser agents
A browser agent can load a page in a browser, run JavaScript, click controls, and navigate a workflow when the product and permissions allow it. That is different from fetching raw HTML with a crawler user agent.
Browser agents can still fail because of authentication, consent screens, inaccessible controls, bot defenses, or unclear page structure. But it is inaccurate to assume that every AI agent stops at the initial HTML response.
A special case: Google-Extended
Google-Extended is not a separate HTTP user agent. Google calls it a robots.txt product token that controls whether content already crawled by Google may be used for Gemini model training and certain grounding uses.
Google also states that Google-Extended does not affect inclusion or ranking in Google Search. The official crawler reference is clear on both points.
I include the token in the checker’s access-policy view because site owners should know how their robots.txt file treats it. It should not be described as a bot that the checker fetches with, or as a gate for Google Search visibility.
Why technical access still matters
I once reviewed a site whose firewall rejected requests carrying several AI-crawler user agents even though its robots.txt file allowed them. That was a real access problem worth fixing.
It did not, by itself, explain every stale answer about the business. A model can answer from older learned patterns, a search index, a third-party source, or a user-triggered page visit. Fixing one blocked crawler changes access for that crawler. It does not rewrite model knowledge or guarantee a citation.
Technical readiness should therefore answer a limited question: what obvious barriers can we find and fix? It should not be used to claim that a score determines whether a brand appears in an AI answer.
The two access checks people often miss
Start with robots.txt, but do not stop there.
First, inspect the rule for each documented provider token. A broad disallow rule can block an automatic crawler even when nobody intended it.
Then inspect the infrastructure response. A CDN or firewall may return a denial, challenge page, or different content to a crawler user agent. Compare the status, redirects, and response body with a normal request. Check server logs when you can, especially if the provider publishes IP ranges that your security layer verifies.
A request that merely copies a user-agent string is useful for finding obvious differences. It cannot prove how every verified crawler will be treated in production.
JavaScript is not a yes-or-no test
JavaScript support varies by provider and product. A raw fetch may only receive the server response. A browser agent can render the page and interact with it. Public documentation does not support the blanket statement that most AI agents never execute JavaScript.
The practical test is still worthwhile: fetch the page without running JavaScript and inspect what remains. If the title, main text, links, and basic structure disappear, you have made the page harder for any client that relies on the initial response.
Server rendering or static rendering gives those clients a useful baseline. Treat that as resilient web design, not proof that the page will be indexed or cited.
What my checker scores
The checker groups its scored tests into three areas:
- Crawler access and response behavior account for 45 percent.
- Content accessibility in the returned page accounts for 35 percent.
- Common discovery signals account for 20 percent.
Those weights are my site-authored rubric. They do not come from OpenAI, Google, Anthropic, Cloudflare, or a standards body. The tool also reports newer agent standards separately as informational checks so they do not silently change the core score.
The score bands are labels for this rubric:
- 80 or higher means the page passed most of the weighted checks.
- 60 to 79 means it passed many checks but has material gaps.
- 40 to 59 means several important checks need attention.
- Below 40 means the page failed many checks in this diagnostic.
A score below 40 does not prove that an AI agent cannot read the site. A score of 100 does not prove that an AI system understands, indexes, recommends, or cites it.
How to use the result
Run the homepage and one important deep page through the Agent Readiness Checker. Review the individual checks before looking at the total.
Fix denied requests, broken redirects, missing primary text, and contradictory crawler policies first. Then decide whether optional discovery files or emerging agent standards serve a real use case for your site.
For products you care about, read the provider’s current documentation and compare it with your server logs. OpenAI’s crawler documentation is a good example because it separates OAI-SearchBot, GPTBot, and ChatGPT-User instead of treating them as one agent.
Once access is sound, evaluate the content separately. A readable page can still be vague, unsupported, outdated, or irrelevant to the question. Technical readiness gets the page through one door. It does not decide what happens after that.
If you want a second set of eyes on your technical access findings, Book a free 30-minute call and bring the page you tested.
