I code and do art things. Check https://cloudy.horse64.org/ for the person behind this content. For my projects, https://codeberg.org/ell1e has many of them.

  • 1 Post
  • 7 Comments
Joined 3 months ago
cake
Cake day: July 16th, 2025

help-circle







  • ell1e@leminal.spacetoTechnology@lemmy.worldI was wrong about robots.txt
    link
    fedilink
    English
    arrow-up
    1
    arrow-down
    1
    ·
    edit-2
    3 months ago

    Often it is respected, but the resulting problem is platforms conflate things with the questionable AI scraping crawlers to blackmail websites into participating in feeding AI.

    For example, Googlebot if enabled won’t just list you for search, but will also scrape your contents for Google’s AI. I imagine LinkedinBot, given it’s microsoft, will feed some other AI of theirs as well on top of the previews.

    Until regulation steps in to require AI bots to separately ask for crawling permission, or to actually get a proper license for reuse of the contents, this situation isn’t going to improve.