Free tool · no signup
AI crawler & robots.txt checker
Enter a website and we check whether search and AI crawlers can actually reach it: robots.txt rules for GPTBot, ClaudeBot, PerplexityBot, Googlebot and Bingbot, live reachability, and whether your sitemap is valid and declared.
Why this matters
AI engines pick their sources from live search results, so a blocked crawler makes everything downstream pointless — no ranking, no citation, no mention. The failure is usually invisible: robots.txt reads as fully open while a CDN rule quietly returns 403 to anything that is not a browser.
Full context in the AI SEO operating guide and the free playbook.
Frequently asked questions
- What does this tool check?
- Three things: which crawlers your robots.txt allows or blocks (including AI-specific agents like GPTBot and OAI-SearchBot), whether your server actually answers non-browser requests, and whether your sitemap exists, is valid, and is declared in robots.txt.
- Why test with a bot user agent at all?
- Because robots.txt only states intentions. Around 27% of sites block a major AI crawler at the CDN or firewall layer while robots.txt looks perfectly open — a file check alone would report those sites as healthy.
- Do you pretend to be GPTBot?
- No. We request your homepage twice: once with a normal browser user agent and once with our own clearly identified ConnectingDotsBot. If a browser gets through and an honest bot does not, your server blocks non-browser traffic generically — which is the rule that usually catches GPTBot too.
- Is blocking AI crawlers always wrong?
- No. Publishers who license their content often block training crawlers deliberately. The distinction that matters: blocking GPTBot stops training use, while blocking OAI-SearchBot removes you from ChatGPT search results. Many sites block the second by accident while meaning only the first.
- Why does Bingbot matter for AI visibility?
- ChatGPT search draws on Bing’s index. Blocking Bingbot can make you invisible inside ChatGPT even when every AI-specific crawler is allowed.