Does Anthropic crawl data from the web, and how can site owners block the crawler?
When individuals ask questions to Claude, it may access websites using a Claude-User agent… An example of this might be:
User-agent: ClaudeBot
Crawl-delay: 1
To block a Bot from your… An example of this is:
User-agent: ClaudeBot
Disallow: /
Opting out of being crawled by Anthropic