New York is set to become the first state to impose guardrails on “stealth crawlers,” or unauthorized software that trawls news sources to scrape content.
  • Zarxrax@lemmy.worldEnglish
    1 day

    Inaccurate headline. The bill doesn’t ban web scraping, it just requires that bots accurately identify themselves through the user agent string, and maybe some additional requirements to disclose the purpose of scraping the data.

    • deliriousdreams@fedia.io
      22 hours

      And there’s a fine if the company doesn’t comply which is basically now gonna be considered the cost of doing business

      • partofthevoice@lemmy.zipEnglish
        44 minutes

        That’s still a win. Make the act less lucrative. Fewer entremenureial types to take after it.

        • deliriousdreams@fedia.io
          27 minutes

          Oh I’m not saying it’s not better than doing nothing. I’m just saying I’m still disatisfied with the results because of the socio economic implications of businesses being treated legally like people.

          I agree with you though.

    • TryingToBeGood@reddthat.comEnglish
      1 day

      The New York measure defines a stealth crawler as any software that retrieves, scrapes or otherwise accesses a website, including AI agents. Under the bill, the attorney general’s office would be able to sue companies that fail to disclose such activity. Violations could net civil penalties of up to $15,000 per day.

      🤔

      • Pika@sh.itjust.worksEnglish
        1 day

        “This website or search engine is not designed for the state of new york” I assume is going to be a disclaimer we will be seeing soon.

        • XLE@piefed.socialEnglish
          1 day

          I hope those tools are exempt because, just like a browser, they respond only to specific commands issued by a human user. They don’t “crawl” pages in the way we describe bots that jump from page to page.

  • AceFuzzLord@lemmy.zipEnglish
    1 day

    The only bots I hope gets exempt are Internet Archive bots. Only bots I support.

    • Rob T Firefly@lemmy.worldEnglish
      23 hours

      The article is about “stealth bots” that don’t identify themselves as such. The Internet Archive bots have always been clearly identifiable.

  • Pika@sh.itjust.worksEnglish
    1 day

    Honestly, I would love if forced ident was required. but archival services need a hard exemption from being blocked as well.

  • rob200@retrofed.comEnglish
    1 day

    Realistically what good would it do once you already had scraped the pattern of news sites it’s already over. All this is doing in actuality is preventing new start ups from competing in the ai space. so really this is the fastest enshitification world record of a medium. Whether you like or hate ai this is actually an enshitification of it. ( I hate ai.)

    • fonix232@fedia.io
      4 hours

      What are you on about?

      For AI purposes the really useful part of a news site is the actual news - you know, the stuff that changes practically every minute - not the “structure” of the site.

      These news sites aren’t being scraped for training data anymore but to provide near-realtime up to date information to the models.

      Meaning e.g. Gemini can scan your news article, extract the useful information for the user, and deliver it to the user, without them ever going to your news site and providing the interaction that at the end of the day is converted to money - money your site needs to run.

    • nullspace@lemmy.worldEnglish
      24 hours

      I’m guessing it’s to eliminate the issue of a site not getting clicks because the article you were about to read is already summarized for you. It also opens the door for revenue negotiations for allowing their content to be scraped for that purpose, as the scraper bots would now be identified.

  • tangeli@piefed.socialEnglish
    1 day

    Why only news sites?

    Any bot that ignores robots.txt should be banned.