Live diagnostic / no result storage

Your policy says one thing. Does your edge agree?

Review 37 prominent AI crawler, search, user-fetcher, dataset, policy-only, and legacy identities in one ordered report.

Exposure scanner

37 identities / prominence order

GPTBot

Awaiting test

ClaudeBot

Awaiting test

Google-Extended

Awaiting test

OAI-SearchBot

Awaiting test

PerplexityBot

Awaiting test

Meta-ExternalAgent

Awaiting test

Plus 31 additional crawler, search, user-fetcher, dataset, policy-only, and legacy identities.

What it checks

The effective site’s robots.txt policy and limited homepage requests using fixed crawler identities, with controlled concurrency.

Policy-only tokens

Google-Extended and Applebot-Extended control how other crawls may be used; they are shown but not misrepresented as standalone HTTP crawlers.

Limits and privacy

The test does not verify source IPs or signatures. Responses are analyzed in memory; content and results are not stored.