• sobchak@programming.dev
      link
      fedilink
      English
      arrow-up
      1
      ·
      edit-2
      4 hours ago

      Rsync, syncthing, backups, mp3s, photos, json files; idk, a lot of tasks involve large amounts of small files. I personally ran into this problem training models on millions of photos. My GPUs would only get up to 25% utilization with mirrored HDDs, so I had to switch to SSDs.

      Edit: the difference is also significant when compiling large projects or just using git. I imagine some game servers need a lot of random accesses too.

      • Gladaed@feddit.org
        link
        fedilink
        English
        arrow-up
        1
        ·
        edit-2
        3 hours ago

        Why are you doing that on a network storage as opposed to on device?

        Also who got millions of photos at home?

        • sobchak@programming.dev
          link
          fedilink
          English
          arrow-up
          1
          ·
          3 hours ago

          Not enough room in the GPU machine for all the HDDs I needed.

          Also who got millions of photos at home?

          People working on biological datasets.

          • Gladaed@feddit.org
            link
            fedilink
            English
            arrow-up
            1
            ·
            1 minute ago

            Why are you doing that recreationally? How are you different from a researcher?