How does it differentiate an “AI crawler”, from any other crawler?
Search engine crawler?
Someone monitoring data to offer statistics?
Archiving?
This is not good. They are most likely doing the crawling themselves and them selling the data to the best bidder. That bidder could obviously be openAI for all we know.
They just know that introducing the sentence “this is anti AI” a lot of people is not going to question anything.
Well, they have access to logs showing who connects to 24 million websites, how they use those websites, and for how long. So if there’s anyone who knows what traffic is crawlers, and which crawlers are AI, it’s Cloudflare. There’s no way they wouldn’t know, they have all the data they would ever need to figure it out. In fact, there’s nobody on the internet who is better positioned to be able to identify AI crawlers than Cloudflare.
How does it differentiate an “AI crawler”, from any other crawler? Search engine crawler? Someone monitoring data to offer statistics? Archiving?
This is not good. They are most likely doing the crawling themselves and them selling the data to the best bidder. That bidder could obviously be openAI for all we know.
They just know that introducing the sentence “this is anti AI” a lot of people is not going to question anything.
Well, they have access to logs showing who connects to 24 million websites, how they use those websites, and for how long. So if there’s anyone who knows what traffic is crawlers, and which crawlers are AI, it’s Cloudflare. There’s no way they wouldn’t know, they have all the data they would ever need to figure it out. In fact, there’s nobody on the internet who is better positioned to be able to identify AI crawlers than Cloudflare.