robots.txt

robots.txt is a plain-text file at a site's root telling crawlers which paths they may fetch, addressed per user agent. It governs crawling, not indexing — a blocked URL can still appear in results if linked elsewhere. The modern decision is the AI-bot matrix: which training crawlers (GPTBot, ClaudeBot, CCBot) and which search crawlers (OAI-SearchBot, PerplexityBot) to allow. They are separate entries.