If you want to have a single node hosting your storage, ZFS either directly or through TrueNAS is the way to go. ECC is not really necessary.
If you do want to have a higher availability and as you have 3 potential nodes already I can recommend Ceph. It’s pretty much THE open source network storage provider and has been battle tested a lot. With Ceph you can define how you want redundancy configured, but I haven’t gone into that enough to tell you what to do.
As long as 2/3 nodes are up you won’t have problems, but less nodes will lead to either a broken cluster or split brain scenario if you’re not careful.
Not sure how Ceph handles nodes with mismatched drives, I only tested using the same number and size of drives. Maybe you’d have to get two more drives for your other nodes.
Re RAID: At the moment RAID 6 does not offer any advantage over RAID 10, so you could go with that instead. RAID 10 would provide higher performance, but future upgrades need to be done in pairs of 2 disks.
RAID 5 is okish, but with 20TB you’re looking at very long restore times and the likelyhood of failure is very high during the restore and it’s very likely you’ll have a half finished restore when a second drive fails.
See this article for an explaination. (The same issue also applies to RAID 6, but the drives can be bigger as you have 2 redundant drives. Regardless, RAID 10 doesn’t have this problem due to being so simple)
If you’ve got >=3 machines and >=3 devices, I’d suggest at least strongly considering Rook. It should allow for future growth and will let you tolerate the loss of one node at the storage level too, assuming you have replication configured. Which (replication params) you can set per StorageClass in case you want to squeeze every last byte out for cases where you don’t need storage-level replication.
I’ve run my own k8s cluster for years now, and solid storage from rook really made it take off with respect to how many applications I can build and/or run on it.
As for backup, there’s velero. Though I haven’t gotten it to work on bare metal. My ideal would be to just use it to store backups in Backblaze B2 given the ridiculously low cost. Presumably I could get there with restic, since that’s my outside-k8s backup solution, but I still haven’t gotten that set up since it’s much more cloud-provider friendly.
I’m also running arch. Unfortunately I’ve been running mine long enough that it’s just my own bespoke Ansible playbooks for configs that have morphed only as required by breaking changes or features/security I want to add. I think the best way to start from scratch these days is kubeadm, and I think it should be fairly straightforward on arch or whatever distro you like.
Fundamentally my setup is just kubelet and kubeproxy on every node, the oci runtime (CRIO for me), etcd (set up manually but certs are automated now) and then some k8s manifests templated and dropped into the k8s manifest folder for the control plane on 3 nodes for HA. The more I think about it, the more I remember how complicated it is unless you want a private CA. Which I have and love the convenience and privacy it affords me (no CTL exposing domain names unless I need public certs and they’re public anyway).
I have expanded to 6 nodes (5 of which remain, RIP laptop SSD) and just run arch on all of them because it kinda just works and I like the consistency. I also got quite good at the arch install in the process.
That’s rad… I have a set of Ansible playbooks/roles/collections already for most system-wide settings. I have a love-hate relationship with Ansible though, but it gets the job done. I may try for cloud-init first until I reach its limitations. I’ve gotten pretty good at the Arch install too, although setting up the disks with LUKS was the most challenging part. Fortunately, the few times I’ve broke things I’ve been able to boot the installer ISO and mount my LUKS volumes from memory, but I couldn’t tell you how I set them up in the first place. 🤣 However I do it, I really just want to automate the process so that I can add new nodes and expand should I decide to rent out colocation space someday.
If you’re planning to go BSD, or buy all the drives you’re ever gonna have in the cluster up-front, then ZFS is great. Otherwise, be mindful of the hidden cost of ZFS. Personally, for my home server, because I’m gradually adding more drives still, I’m using mdraid on RAID6 with 8 x 8TB WD Reds/HGST Ultrastars, and I’m loving the room for activities.
Having said that, regardless of the solution you go with, since you’ve got only 4 drives, higher RAID level (and equivalent of thereof such as RAIDZ2) might be out of reach as you’d be “wasting” a lot of space for the extra piece of mind. If I were in your situation, I’d probably use RAID5 (despite RAID 5 is dead in 2009, or have they continued chugging on after 2013) for less important data (so sustain 1 drive failure) or RAID 10 if I need more performance (and depending on luck of draw, potentially sustain 2 drive failures depending on which drive fails).
Thanks. I’ve got Gigabit Fiber so I guess I’ll try Hetzner as a remote backup, or see how much it will cost to upgrade my Google Workspace account since they started enforcing their storage quotas.
I don’t have any friends really 😥 and the unlimited storage with Google Workspace was $25/mo. I think it will cost me about $125/mo. now to get enough pooled storage with Google, but it is doable at least in the short-term. I guess I need to make some friends with fiber connections.
You are not logged in. However you can subscribe from another Fediverse account, for example Lemmy or Mastodon. To do this, paste the following into the search field of your instance: !selfhosted@lemmy.world
A place to share alternatives to popular online services that can be self-hosted without giving up privacy or locking you into a service you don’t control.
Rules:
Be civil: we’re here to support and learn from one another. Insults won’t be tolerated. Flame wars are frowned upon.
No spam posting.
Posts have to be centered around self-hosting. There are other communities for discussing hardware or home computing. If it’s not obvious why your post topic revolves around selfhosting, please include details to make it clear.
Don’t duplicate the full text of your blog or github here. Just post the link for folks to click.
Submission headline should match the article title (don’t cherry-pick information from the title to fit your agenda).
Would suggest raid0 for maximum read speed /s
If you want to have a single node hosting your storage, ZFS either directly or through TrueNAS is the way to go. ECC is not really necessary.
If you do want to have a higher availability and as you have 3 potential nodes already I can recommend Ceph. It’s pretty much THE open source network storage provider and has been battle tested a lot. With Ceph you can define how you want redundancy configured, but I haven’t gone into that enough to tell you what to do.
As long as 2/3 nodes are up you won’t have problems, but less nodes will lead to either a broken cluster or split brain scenario if you’re not careful.
Not sure how Ceph handles nodes with mismatched drives, I only tested using the same number and size of drives. Maybe you’d have to get two more drives for your other nodes.
Re RAID: At the moment RAID 6 does not offer any advantage over RAID 10, so you could go with that instead. RAID 10 would provide higher performance, but future upgrades need to be done in pairs of 2 disks.
RAID 5 is okish, but with 20TB you’re looking at very long restore times and the likelyhood of failure is very high during the restore and it’s very likely you’ll have a half finished restore when a second drive fails.
See this article for an explaination. (The same issue also applies to RAID 6, but the drives can be bigger as you have 2 redundant drives. Regardless, RAID 10 doesn’t have this problem due to being so simple)
If you’ve got >=3 machines and >=3 devices, I’d suggest at least strongly considering Rook. It should allow for future growth and will let you tolerate the loss of one node at the storage level too, assuming you have replication configured. Which (replication params) you can set per StorageClass in case you want to squeeze every last byte out for cases where you don’t need storage-level replication.
I’ve run my own k8s cluster for years now, and solid storage from rook really made it take off with respect to how many applications I can build and/or run on it.
As for backup, there’s velero. Though I haven’t gotten it to work on bare metal. My ideal would be to just use it to store backups in Backblaze B2 given the ridiculously low cost. Presumably I could get there with restic, since that’s my outside-k8s backup solution, but I still haven’t gotten that set up since it’s much more cloud-provider friendly.
Thanks. That is what I’m leaning towards. Do you have any suggestions for a particular distro for your K8S nodes? I’m running Arch on my desktop.
The idea of being able to setup different storage classes is very appealing, as well as learning how to build my on K8S cloud.
I’m also running arch. Unfortunately I’ve been running mine long enough that it’s just my own bespoke Ansible playbooks for configs that have morphed only as required by breaking changes or features/security I want to add. I think the best way to start from scratch these days is kubeadm, and I think it should be fairly straightforward on arch or whatever distro you like.
Fundamentally my setup is just kubelet and kubeproxy on every node, the oci runtime (CRIO for me), etcd (set up manually but certs are automated now) and then some k8s manifests templated and dropped into the k8s manifest folder for the control plane on 3 nodes for HA. The more I think about it, the more I remember how complicated it is unless you want a private CA. Which I have and love the convenience and privacy it affords me (no CTL exposing domain names unless I need public certs and they’re public anyway).
I have expanded to 6 nodes (5 of which remain, RIP laptop SSD) and just run arch on all of them because it kinda just works and I like the consistency. I also got quite good at the arch install in the process.
That’s rad… I have a set of Ansible playbooks/roles/collections already for most system-wide settings. I have a love-hate relationship with Ansible though, but it gets the job done. I may try for cloud-init first until I reach its limitations. I’ve gotten pretty good at the Arch install too, although setting up the disks with LUKS was the most challenging part. Fortunately, the few times I’ve broke things I’ve been able to boot the installer ISO and mount my LUKS volumes from memory, but I couldn’t tell you how I set them up in the first place. 🤣 However I do it, I really just want to automate the process so that I can add new nodes and expand should I decide to rent out colocation space someday.
If you’re planning to go BSD, or buy all the drives you’re ever gonna have in the cluster up-front, then ZFS is great. Otherwise, be mindful of the hidden cost of ZFS. Personally, for my home server, because I’m gradually adding more drives still, I’m using mdraid on RAID6 with 8 x 8TB WD Reds/HGST Ultrastars, and I’m loving the room for activities.
Having said that, regardless of the solution you go with, since you’ve got only 4 drives, higher RAID level (and equivalent of thereof such as RAIDZ2) might be out of reach as you’d be “wasting” a lot of space for the extra piece of mind. If I were in your situation, I’d probably use RAID5 (despite RAID 5 is dead in 2009, or have they continued chugging on after 2013) for less important data (so sustain 1 drive failure) or RAID 10 if I need more performance (and depending on luck of draw, potentially sustain 2 drive failures depending on which drive fails).
Thanks. I believe TrueNAS does ZFS as well… maybe by default. If I want to keep it simple this will probably be the route I go.
if the data is mission critical have multiple backups for it, and test the restore process of said backups.
As for your raid vs software raid, just install truenas and throw it on there, use a LSI in IT mode and create a ZFS cluster. Don’t over think it.
Thanks. I’ve got Gigabit Fiber so I guess I’ll try Hetzner as a remote backup, or see how much it will cost to upgrade my Google Workspace account since they started enforcing their storage quotas.
It’s a lot cheaper in the long run to do an off site backup rig and just put it at a friend’s house if you can. Google will be expensive.
I don’t have any friends really 😥 and the unlimited storage with Google Workspace was $25/mo. I think it will cost me about $125/mo. now to get enough pooled storage with Google, but it is doable at least in the short-term. I guess I need to make some friends with fiber connections.