4 hoursDetect whether the user is a human, but instead of blocking the request (which is going to be obvious to the scraper operator and will just cause the bot developers to go work on better human emulation until they get the data), poison the response. Just as blocking scrapers is hard for website operators, so is separating useful data from not-useful data for people building AI training corpuses.
https://www.cloudflare.com/learning/ai/data-poisoning/
Data poisoning involves injecting malicious information into training datasets to manipulate an AI model’s behavior, compromising its accuracy, reliability, and the overall integrity of machine learning results.
- ag10n@lemmy.worldEnglish1 hour
This is the way. Computer use agents are common and can easily ‘browse’ to a page and grab the content.
The really exasperating thing is that Wikipedia makes all of their data available in compressed database form, and I suspect that other MediaWiki wikis could do so too. I checked one of the wikis being complained about, the Minecraft wiki, and it’s a MediaWiki wiki.
This isn’t even a case where the server operators have any problem with the data being available to the bots. In fact, people are going out of their way to package up all the data in a form that is optimal for computer processing (better for the bots) and which doesn’t create additional load on the wiki servers (better for the wiki operators).
But the bots aren’t using them! And on top of that, they aren’t even written to follow traditional conventions for being polite in scraping. We’ve had lots of software spider the Web — that’s not new — but normally that spidering software has followed basic politeness to try to avoid excessive disruption to the servers being spidered.
It’s as if someone has set up a table out front of their house with a big sign reading “Free cookies!” with boxes of pre-packaged cookies. Instead of taking the damn cookies, the bots are obtaining the largest trucks possible and then ramming holes in the house with a truck and scavenging through the ruins, desperately trying to find cookies.
- go_go_gadget@lemmy.worldEnglish3 hours
It’s a weird phenomenon I encountered as a data engineer inside corporations. Teams wanted to run endless queries off our API instead of using the snapshots we provided. Most of the apps didn’t have a justifiable reason for wanting up to the minute information. 🤷♂️
- CosmoNova@lemmy.worldEnglish8 hours
Ai scrapers are making it kinda suck to run anything on the internet to be honest.
I wonder if it would be possible to filter traffic by limiting the rate at which links can be followed after the initial connection.
I suppose one problem with that is it would also limit the crawlers you want, like search engine indexing. Maybe if enough sites were doing this it would generate some pressure on the AI companies to behave better.
- 6 hours
If they have 10k URLs and in parallel find out all links via different IPs, they get new links.
When they do that again with the new links, there is no connection between visits, because the IPs differ. It looks like someone else is requesting what is behind the link.
- Jason2357@lemmy.caEnglish4 hours
The issue with wiki’s and source forges is that there is a maze of links to all past versions of everything, each generated on demand from a cpu-expensive database query. You basically have to limit the pages anonymous users can spider into. Forgejo has a setting to block expensive pages from non-logged in users for example.
I could see maybe caching that and providing it to a not-clearly-human user if it is in cache. That lets someone do something like link to a particular version of a file in a discussion here on the Threadiverse. The first user loading it will cause it to be cached.
