Bot identification
User-agent + reverse-DNS verification for GPTBot, ClaudeBot, PerplexityBot, Google-Extended, CCBot, Bytespider, and more.
Loading
Search for a command to run...
Intelligence
GPTBot, ClaudeBot, PerplexityBot, and Google-Extended crawl your site differently than Googlebot. Crawler Analytics shows you exactly what each bot fetched, when, and how often.
Log files hide which AI bots hit your site
You cannot confirm that your robots.txt / llms.txt is being honored
You do not know which pages AI engines have actually read
Competitors may be indexed while you are being skipped
User-agent + reverse-DNS verification for GPTBot, ClaudeBot, PerplexityBot, Google-Extended, CCBot, Bytespider, and more.
Time-series view of each bot's visits to your site, with outliers and spikes highlighted.
Heatmap showing which of your pages each bot actually fetched — and which it ignored.
Verifies your llms.txt and robots.txt against observed bot behavior; flags mismatches.
Detects when bots receive 4xx/5xx, rate-limit headers, or hostile responses so you can fix them.
Cloudflare, Vercel, Fastly, AWS CloudFront log formats supported out of the box.
Upload a CDN log sample, or connect live (Cloudflare Logpush, Vercel, custom HTTP endpoint).
We verify user-agents with reverse DNS and match to known AI crawlers.
We compute per-bot, per-page crawl frequency and status-code distribution.
Your llms.txt and robots.txt are compared against what bots actually did.
Spikes, drops, and new bot IPs are flagged in real time.
Prove AI bots are crawling the pages you want them to.
Detect impersonating user-agents that pretend to be AI bots.
Enforce llms.txt decisions with evidence of bot compliance.
Start with a free account — no credit card required. Upgrade when you need more scans, profiles, or seats.