GlossaryAI search
AI crawler
Also called: AI bot · GPTBot
An AI crawler is a bot that fetches web pages to train AI models or to answer user questions. You can allow or block most through robots.txt.
An AI crawler is an automated program run by an AI company that visits websites. Some collect content to train models, some fetch pages live to answer a user's question, and some do both. Examples include GPTBot, ClaudeBot, PerplexityBot and Google-Extended.
You choose whether to allow them. Most reputable crawlers follow robots.txt rules, and many companies use separate bots for training and for live answers, so you can treat them differently. Blocking training doesn't always mean blocking visibility, but the details differ by company and change over time.
Practical point: decide what you want. If you want AI tools to be able to cite you, don't block their search bots by accident. Check your robots.txt, your firewall and CDN settings, which sometimes block bots by default. Review each company's published bot names and update the file now and then.
Links you can check.
Every link we place comes with a receipt. The free link plan shows the pages worth pushing and what the links would cost.