Does Anthropic crawl data from the web, and how can site owners block the crawler?
ClaudeBot
Crawl-delay: 1
To block a Bot from your entire website, add this to the robots.txt file… Disallow: /
Opting out of being crawled by Anthropic Bots requires modifying the robots.txt file… correctly or persistently guarantee an opt-out, as doing so impedes our ability to read your robots.txt file