Does Anthropic crawl data from the web, and how can site owners block the crawler?
As per industry standard, Anthropic uses a variety of robots to gather data from the public web for model… Anthropic’s Bots respect “do not crawl” signals by honoring industry standard directives in robots.txt… To limit crawling activity, we support the non-standard Crawl-delay extension to robots.txt