Cloudflare accuses AI startup Perplexity of stealth crawling websites
Cloudflare has accused Perplexity, a rising AI search startup, of bypassing website restrictions by disguising its bots to access content that was explicitly blocked through tools like robots.txt and firewalls. According to Cloudflare, Perplexity used tactics such as changing user agents and rotating IP addresses to crawl tens of thousands of domains, violating transparency norms. The company has since removed Perplexity from its verified bots list and introduced new methods to block such stealth crawling.
This raises serious concerns about AI firms ignoring data protection norms and undermining website owners’ control over their content.
About the Author
