AI Crawl Control
Visibility and control for AI content scraping
AI crawlers may scrape webpages thousands of times for every referral they send. AI Crawl Control lets you control when and how they can access your content.
Benefits of Cloudflare’s AI Crawl Control
Better visibility
Understand which AI crawlers are scraping your site, how often they do it, and whether they’re following your rules.
Stronger control
Set a crawler-by-crawler strategy for who should be blocked and who should be allowed through.
Accurate detection
Use machine learning, behavioral analysis, and fingerprinting based on Cloudflare’s visibility into 20% of all Internet traffic.
Monetization
Charge the AI crawlers of your choice for scraping your data [in private beta].
HOW IT WORKS
Block, allow, or charge AI crawlers on your website
Cloudflare sees and fingerprints more AI bots than any other provider, thanks to our broad visibility into global Internet traffic.
AI Crawl Control uses this visibility to give you granular insight into AI crawler activity on your domains. It also lets you:
- Block or allow specific crawlers
- Charge certain crawlers for access to your content [in private beta]
What our customers are saying
“While bots, data scraping, credential harvesting, and DDoS attacks continue to evolve, Cloudflare allows us to mitigate incoming threats, protecting millions of storefronts, and ensuring customers can complete their transactions without interruption.”
Duncan Davidson
VP Developer Productivity, Shopify
Top AI Crawl Control use cases
Protect your intellectual property
Decide which AI crawlers can access which domains across your digital environments
Improve web performance
Prevent repeated AI crawls from overwhelming your origin servers
Clean up marketing analytics
Keep AI crawlers from skewing web tracking with repeated scraping
Learn more about Cloudflare’s bot management capabilities
Resources
Press
Cloudflare just changed how AI crawlers scrape the Internet-at-Large; Permission-based approach makes way for a new business model.
Blog
The crawl before the fall… of referrals: understanding AI’s impact on content providers.