How to verify that AI crawlers actually reach your site

Why a faked user agent proves little

The quick test everyone tries is curl -A "GPTBot" https://yoursite.com/. It tells you less than it seems:

• CDNs check who is really asking. Cloudflare identifies verified bots by a cryptographic signature, a published IP list or reverse DNS. Its bot score ignores the user agent, and on Pro plans and above, its fake bot rules flag requests that claim to be Googlebot or Bingbot but fail that check. Hostinger’s CDN and Vercel also verify crawlers by IP or reverse DNS.
• So the two results can disagree. Your fake GPTBot may be blocked while the real one gets in. And a rule that blocks a verified crawler category will not match your fake request, so you may see 200 while the real crawler gets 403.
• Logs can be faked the same way. A user agent is whatever the client says it is. Bing says plainly that user agent strings are easy to spoof.

The steps below use evidence the real crawlers leave behind, from your own settings outward to the search engines.

1. Read the robots.txt your site actually serves

Open https://yoursite.com/robots.txt in a browser, not the file in your code or on your server. What visitors get can differ:

• Cloudflare can add its own rules in front of your file (Managed robots.txt, now replaced by Bot Preference Sync).
• Some site builders rewrite it. On Squarespace, for example, ticking “Block known artificial intelligence crawlers” adds a list of AI crawlers, including GPTBot, ClaudeBot and Google-Extended, to robots.txt. The box is unticked by default.

Look for Disallow lines that name GPTBot, OAI-SearchBot, ClaudeBot, Claude-SearchBot, PerplexityBot, Googlebot or bingbot, and for Content-Signal lines. Remember that robots.txt is a request: it shows what you asked for, not who actually came. See AI crawlers: the list, and how to allow or block them for what each rule means.

2. Check your CDN and hosting bot settings

These can block crawlers no matter what robots.txt says:

If you use another CDN, firewall or security plugin, look for the same things: bot or AI crawler rules, challenge pages, and rate limits.

3. Look at your CDN’s AI crawler data

On Cloudflare, open AI Crawl Control, available on all plans:

• Crawlers lists each AI crawler with its request count and your Allow or Block setting.
• Metrics separates “All requests”, including blocked and failed ones, from “Allowed requests”, which got a 2xx response. A big gap means something is stopping them.
• Status codes show 403 for blocked requests and 402 where payment was required.
• Directives lists AI crawlers that requested paths your robots.txt disallows.

Two limits on the Free plan: crawlers are identified by user agent only, so a fake GPTBot counts as GPTBot, and metrics cover only the past 24 hours. Cloudflare’s docs describe fuller detection on paid plans.

4. Read your server logs

Nginx and Apache’s standard “combined” log format records the client IP and the user agent. Two things to fix first:

• Behind Cloudflare, the logged IP is Cloudflare’s, not the crawler’s. The real IP is in the CF-Connecting-IP header. Use nginx’s realip module (set_real_ip_from plus real_ip_header CF-Connecting-IP) or Apache’s mod_remoteip, as Cloudflare’s guide describes.
• Cached pages never reach your server. If your CDN caches HTML, crawler visits served from the cache will not appear in these logs.

A first look, with the log in the usual place:

grep -E 'GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|Claude-SearchBot|Claude-User|PerplexityBot|Perplexity-User|Googlebot|bingbot' /var/log/nginx/access.log \
  | awk '{print $1, $9}' | sort | uniq -c | sort -rn | head -20

This counts requests by IP and status code. Status 200 is what you want; 403, 429 and 503 mean something turned the crawler away.

5. Verify the IPs

A user agent alone is not evidence. Check each IP from step 4 against the company’s own list:

Reverse DNS, then forward DNS, for an IP from your log:

host 66.249.66.1
host crawl-66-249-66-1.googlebot.com

The first command should return a googlebot.com name, and the second should return the IP you started with.

To check an IP against one of the JSON lists, a few lines of Python are enough. These files list ranges under prefixes, as ipv4Prefix or ipv6Prefix:

import ipaddress, json, sys

ranges = json.load(open(sys.argv[1]))["prefixes"]
ip = ipaddress.ip_address(sys.argv[2])
nets = [ipaddress.ip_network(r.get("ipv4Prefix") or r.get("ipv6Prefix")) for r in ranges]
print("listed" if any(ip in n for n in nets) else "NOT listed")

Save it as check_ip.py, download the list, and run python3 check_ip.py gptbot.json 203.0.113.7. A request that says GPTBot from an IP that is not listed should be treated as someone else using its name.

You will never see Google-Extended in your logs: it is a robots.txt token, not a separate crawler.

6. Google Search Console

• URL Inspection shows the last crawl, whether robots.txt allowed it (“Crawl allowed?”), whether the fetch succeeded (“Page fetch”), and the HTML Google received. One trap: if robots.txt blocks the page, “Indexing allowed?” still says Yes, because Google cannot see a noindex it was not allowed to fetch. The live test uses a different crawler, Google-InspectionTool, so passing it does not guarantee Googlebot gets in.
• Crawl Stats (Settings) shows Googlebot’s requests by response code and any problems fetching robots.txt or reaching your server. It is only available for root-level properties.
• Settings > Search generative AI must be set to include your site if you want to appear in AI Overviews and AI Mode.
• The Search generative AI performance report, available to all sites since 31 August 2026, shows impressions in Google’s AI features. It does not show clicks.

7. Bing Webmaster Tools

• URL Inspection shows when Bing discovered, crawled and indexed a page. Its Live URL test shows what Bingbot sees when it downloads the page.
• AI Performance shows how often your pages are cited in Microsoft Copilot and Bing’s AI summaries, and the “grounding queries” behind those citations. It is a sample of citation activity, not a complete record, and citations are not clicks. You can export it as CSV or Excel.

What each step proves

Citeloop’s free GEO audit, opening soon, will check step 1 and spot the CDN in step 2 from the outside, and tell you which of the remaining checks need your access.