AI Crawl Control
Visibility and control for AI content scraping
AI crawlers may scrape webpages thousands of times for every referral they send. AI Crawl Control lets you control when and how they can access your content.
Benefits of Cloudflare’s AI Crawl Control
Better visibility
Understand which AI crawlers are scraping your site, how often they do it, and whether they’re following your rules.
Stronger control
Set a crawler-by-crawler strategy for who should be blocked and who should be allowed through.
Accurate detection
Use machine learning, behavioral analysis, and fingerprinting based on Cloudflare’s visibility into 20% of all Internet traffic.
Monetization
Charge the AI crawlers of your choice for scraping your data [in private beta].
HOW IT WORKS
Block, allow, or charge AI crawlers on your website
Cloudflare sees and fingerprints more AI bots than any other provider, thanks to our broad visibility into global Internet traffic.
AI Crawl Control uses this visibility to give you granular insight into AI crawler activity on your domains. It also lets you:
- Block or allow specific crawlers
- Charge certain crawlers for access to your content [in private beta]
What our customers are saying
The great thing about Cloudflare Bot Management solutions is I don't need to spend time fine-tuning. The machine learning algorithms just work because Cloudflare has such great data. Our lives are 1000 times easier while still ensuring our sites are both safe and fast for our customers.
Manager, Security Engineering — SoFi
Top AI Crawl Control use cases
Protect your intellectual property
Decide which AI crawlers can access which domains across your digital environments
Improve web performance
Prevent repeated AI crawls from overwhelming your origin servers
Clean up marketing analytics
Keep AI crawlers from skewing web tracking with repeated scraping
Learn more about Cloudflare’s bot management capabilities
Resources
Press
Cloudflare just changed how AI crawlers scrape the Internet-at-Large; Permission-based approach makes way for a new business model.
Blog
The crawl before the fall… of referrals: understanding AI’s impact on content providers.