• AI crawlers: the list, and how to allow or block themWhat GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot and Google-Extended do, what blocking each one changes, and how to set robots.txt and Cloudflare.
• What is llms.txt? Format, examples and who reads itllms.txt points AI agents to the pages that matter. The v2 format, a copyable example, where to put it, and which AI companies say they read it.
• What is generative engine optimization (GEO)?GEO makes your content easy for AI search to find and cite: where the term comes from, what research found, what Google and Bing say, and what is manipulation.
• How to verify that AI crawlers actually reach your siteSeven checks: your live robots.txt, CDN bot settings, AI crawler logs, IP and reverse DNS checks, Google Search Console and Bing Webmaster Tools.