I’m looking for experiences and opinions on kubernetes storage.

I want to create a highly available homelab that spans 3 locations where the pods have a preferred locations but can move if necessary.

I’ve looked at linstore or seaweedfs/garage with juicefs but I’m not sure how well the performance of those options is across the internet and how well they last in long term operation. Is anyone else hosting k3s across the internet in their homelab?

Edit: fixed wording

  • F04118F@feddit.nlEnglish
    1·
    4 months ago

    I tried Longhorn, and ended up concluding that it would not work reliably with Volsync. Volsync (for automatic volume restore on cluster rebuild) is a must for me.

    I plan on installing Rook-Ceph. I’m also on 1Gb/s network, so it won’t be fast, but many fellow K8s home opsers are confident it will work.

    Rook-ceph does need SSDs with Power Loss Protection (PLP), or it will get extremelly slow (latency). Bandwidth is not as much of an issue. Find some used Samsung PM or SM models, they aren’t expensive.

  • ChaosMonkey@lemmy.dbzer0.comEnglish
    1·
    4 months ago

    Longhorn is pretty easy to use. Garage works well too. Ceph is harder to use but provides both block and object storage (s3).

    • InnerScientist@lemmy.worldOPEnglish
      0·
      4 months ago

      I mean storage backends as in the provisioner, I will use local storage on the nodes with either lvm or just storage on a filesystem.

      I already set up a cluster and tried linstore, I’m searching for experiences with the options because I don’t want to test them all.

      I currently manage all the servers with a NixOS repository but am looking for better failover.

      • just_another_person@lemmy.worldEnglish
        1·
        4 months ago

        You won’t find any benchmarks for distributed filesystems, because they don’t apply to any one setup. Nobody knows your network situation, disk speed, availability across clusters…etc.