Can AI agents read your site?
Enter your URL. I fetch it as a browser and as GPTBot, then run 14 deterministic checks: per-bot robots.txt access, WAF bot blocking, JavaScript dependence, llms.txt, and what an agent actually sees. No AI model, no guesswork.
Your agent readiness score, a per-bot access matrix, and exactly what an AI agent sees on your page will appear here.
Can AI crawlers read your website?
Most AI crawlers cannot read your website unless robots.txt explicitly allows them and your server returns the same response to a bot user agent that it gives a browser. That crawler access is the foundation of Generative Engine Optimization (GEO): if OAI-SearchBot, GPTBot, ClaudeBot, or PerplexityBot get blocked or served a different response than a normal visitor, your content may be unavailable to that platform, no matter how good the writing is underneath.
I built this checker because I kept finding sites with strong content that no AI agent could ever reach: a WAF rule silently blocking bot user agents, a robots.txt written for Googlebot years before GPTBot existed, or a client-rendered app shell that ships an empty page to anything that does not run JavaScript. This tool fetches your page twice, once as a browser and once as GPTBot, then checks robots.txt against all 8 major AI crawlers by name.
What does this checker test?
This tool runs three scored groups plus an informational advanced section covering DNS-AID, MCP Server Cards, Agent Skills, OAuth discovery, and WebMCP with no AI model involved: crawler access (robots.txt rules for all 8 tracked bots, Content Signals, bot-versus-browser response parity, HTTPS, response time), content accessibility (visible text without JavaScript, Markdown negotiation, title and meta tags), and discoverability signals (structured data, Link headers, an llms.txt file, a sitemap, canonical and Open Graph tags). Every result is deterministic: run it twice on an unchanged page and you get the same score.
If you also want an AI model to judge how citable your writing itself is, entity clarity, answer-first structure, topical depth, run the AI Search Readiness Scorecard next. That tool grades the content; this one grades whether an agent can reach the content at all.
How to make your website AI friendly
- Explicitly allow AI crawlers in robots.txt instead of leaving the file silent or blocking everything with a wildcard rule.
- Server-render or statically render your primary content so it exists in the initial HTML response, not just after JavaScript runs.
- Publish an llms.txt file at your domain root. Build one free with the llms.txt Generator in under a minute.
- Add JSON-LD structured data, a canonical tag, and a working sitemap so agents can confirm what a page is and find the rest of your site.
- Keep response time fast and redirect chains short. Crawlers with limited budgets abandon slow or redirect-heavy requests.
Want me to walk through the fixes with you instead of reading a report alone? Book a free 30-minute call or reach me here.
Common questions
Short answers on what agent readiness means, how this checker works, and how it differs from the Scorecard.
What is agent readiness?
Agent readiness is how well an AI agent, the crawlers and retrieval systems behind tools like ChatGPT, Claude, and Perplexity, can access, parse, and cite your website. It covers three things: whether your robots.txt and server let AI crawlers in, whether your content is readable without running JavaScript, and whether you publish the signals, llms.txt, structured data, a sitemap, that agents look for first.
Can AI read my website if it depends on JavaScript?
Only partially, and often not at all. Most AI crawlers, including GPTBot and ClaudeBot, do not execute JavaScript, so they see whatever HTML the server sends on the first request. If your content only appears after client-side rendering, an AI agent sees an empty page. Server-render or statically render your primary content so it is present in the initial HTML response.
How do I unblock AI crawlers for search visibility?
Add explicit Allow rules for the crawlers you want to support in your robots.txt file at your domain root. OAI-SearchBot controls ChatGPT search visibility; GPTBot is a separate training-data choice. A minimal example:
User-agent: OAI-SearchBot Allow: / User-agent: GPTBot Allow: / User-agent: ClaudeBot Allow: / User-agent: PerplexityBot Allow: /
Does having an llms.txt file make my site agent-ready?
No, not by itself. llms.txt is a helpful signal, a curated index of your best pages for AI systems, but it does not override a robots.txt block, fix JavaScript-only rendering, or replace structured data. Treat it as one of several signals this checker measures, not a fix-all.
Is this the same as the AI Search Readiness Scorecard?
No. The Agent Readiness Checker runs pure, deterministic technical checks: crawler access, JavaScript dependence, and the files agents look for, with no AI model in the loop. The AI Search Readiness Scorecard adds a qualitative AI grade on top of similar technical checks, judging things like entity clarity and answer-first structure that only a language model can assess. Run this one first for a fast technical pass, then the scorecard for a deeper citability grade.