Agent Readiness Checker
Agent readiness is whether an AI agent, the crawlers behind ChatGPT, Claude, and Perplexity, can actually access, parse, and cite your website. In 2026 that means passing a live fetch test with a bot user agent, rendering real text without JavaScript, and publishing the files agents look for first. Enter your URL and I will run that test now, no AI model, no guesswork.
Your agent readiness score, a per-bot access matrix, and exactly what an AI agent sees on your page will appear here.
Can AI crawlers read your website?
Most AI crawlers cannot read your website unless robots.txt explicitly allows them and your server returns the same response to a bot user agent that it gives a browser. That crawler access is the foundation of Generative Engine Optimization (GEO): if GPTBot, ClaudeBot, or PerplexityBot get blocked or served a different response than a normal visitor, your content cannot be indexed, quoted, or surfaced in AI Overviews and other generative answers, no matter how good the writing is underneath.
I built this checker because I kept finding sites with strong content that no AI agent could ever reach: a WAF rule silently blocking bot user agents, a robots.txt written for Googlebot years before GPTBot existed, or a client-rendered app shell that ships an empty page to anything that does not run JavaScript. This tool fetches your page twice, once as a browser and once as GPTBot, then checks robots.txt against all 8 major AI crawlers by name.
What does this checker test?
This tool runs three groups of deterministic checks with no AI model involved: crawler access (robots.txt rules for all 8 tracked bots, bot-versus-browser response parity, HTTPS, response time), content parseability (visible text without JavaScript, a single H1, a main content landmark, title and meta tags), and AI signals (structured data, an llms.txt file, a sitemap, canonical and Open Graph tags). Every result is deterministic: run it twice on an unchanged page and you get the same score.
If you also want an AI model to judge how citable your writing itself is, entity clarity, answer-first structure, topical depth, run the AI Search Readiness Scorecard next. That tool grades the content; this one grades whether an agent can reach the content at all.
How to make your website AI friendly
- Explicitly allow AI crawlers in robots.txt instead of leaving the file silent or blocking everything with a wildcard rule.
- Server-render or statically render your primary content so it exists in the initial HTML response, not just after JavaScript runs.
- Publish an llms.txt file at your domain root. Build one free with the llms.txt Generator in under a minute.
- Add JSON-LD structured data, a canonical tag, and a working sitemap so agents can confirm what a page is and find the rest of your site.
- Keep response time fast and redirect chains short. Crawlers with limited budgets abandon slow or redirect-heavy requests.
Want me to walk through the fixes with you instead of reading a report alone? Book a free 30-minute call or reach me here.
Common questions
Short answers on what agent readiness means, how this checker works, and how it differs from the Scorecard.
What is agent readiness?
Agent readiness is how well an AI agent, the crawlers and retrieval systems behind tools like ChatGPT, Claude, and Perplexity, can access, parse, and cite your website. It covers three things: whether your robots.txt and server let AI crawlers in, whether your content is readable without running JavaScript, and whether you publish the signals, llms.txt, structured data, a sitemap, that agents look for first.
Can AI read my website if it depends on JavaScript?
Only partially, and often not at all. Most AI crawlers, including GPTBot and ClaudeBot, do not execute JavaScript, so they see whatever HTML the server sends on the first request. If your content only appears after client-side rendering, an AI agent sees an empty page. Server-render or statically render your primary content so it is present in the initial HTML response.
How do I unblock GPTBot and ClaudeBot?
Add explicit Allow rules for each AI crawler in your robots.txt file at your domain root. A minimal example:
User-agent: GPTBot Allow: / User-agent: ClaudeBot Allow: / User-agent: PerplexityBot Allow: /
Does having an llms.txt file make my site agent-ready?
No, not by itself. llms.txt is a helpful signal, a curated index of your best pages for AI systems, but it does not override a robots.txt block, fix JavaScript-only rendering, or replace structured data. Treat it as one of several signals this checker measures, not a fix-all.
Is this the same as the AI Search Readiness Scorecard?
No. The Agent Readiness Checker runs pure, deterministic technical checks: crawler access, JavaScript dependence, and the files agents look for, with no AI model in the loop. The AI Search Readiness Scorecard adds a qualitative AI grade on top of similar technical checks, judging things like entity clarity and answer-first structure that only a language model can assess. Run this one first for a fast technical pass, then the scorecard for a deeper citability grade.