Crawl rules with per-bot AI crawler control (GPTBot, ClaudeBot, PerplexityBot…).
# robots.txt generated with ExitFounder tools User-agent: * Allow: / Disallow: /admin Disallow: /api/private Disallow: /dashboard # AI crawlers that may not use this site User-agent: GPTBot User-agent: ClaudeBot User-agent: CCBot User-agent: Bytespider User-agent: Amazonbot Disallow: / # AI crawlers welcome (answers, citations, live browsing) User-agent: OAI-SearchBot User-agent: ChatGPT-User User-agent: Claude-User User-agent: Claude-SearchBot User-agent: PerplexityBot User-agent: Perplexity-User User-agent: Google-Extended User-agent: Applebot-Extended User-agent: meta-externalagent Allow: / Disallow: /admin Disallow: /api/private Disallow: /dashboard Sitemap: https://acme.example/sitemap.xml # Agent guide: https://acme.example/llms.txt
Tip: block training crawlers but keep search/answer bots (OAI-SearchBot, Claude-SearchBot, PerplexityBot) allowed if you want to be cited in AI answers.