Lenovo M920q mini pc with Proxmox installed (this doesn’t have IPMI, only vPRO and it’s annoying me)
Fujitsu TX1320 M3 with TrueNAS Core installed - ZFS + RAID1 (this is a low-end “enterprise grade” server, and best thing - it has IPMI).
The Proxmox PC keeps all its CTs and 1 VM on the TrueNAS using iSCSI.
The idea behind my setup was that it felt nice that the TrueNAS would handle all the storage heavy lifting - ZFS, RAID etc., while the Proxmox mini PC would be a “compute-only” node that has a naked Proxmox install with some config.
The problem with that is if the TrueNAS machine loses power or is restarted, the Proxmox CTs/VMs switch their filesystem to read-only and stop responding to requests. This is because the iSCSI connection is interrupted. When the TrueNAS is back online, Proxmox doesn’t make any attempt to restart the VMs/CTs - they’d still be broken.
It’s annoying to me to have to VPN to the Proxmox web ui and wait 15 minutes until all the CTs/VMs are restarted and now again functioning on the “alive” iSCSI connection.
I was wondering what are my options here to remove the dependency chain?
I’m really into the idea of decomissioning the Proxmox node because I’m scared I won’t be able to (over VPN) change the power state of the machine if something goes wrong, since it only has vPro and not iSCSI like the TrueNAS machine. By doing that, I’d consolidate the storage and the compute into the TrueNAS machine.
Options I can think of:
Decomission the Proxmox node and move all Debian VMs/CTs to TrueNAS BSD jails. Is that even possible? Will all my Debian VMs work in BSD?
Decomission the Proxmox node, switch TrueNAS Core to TrueNAS Scale and move CTs/VMs to TrueNAS Scale’s Linux VMs
Keep the Proxmox node and somehow figure out how to get Proxmox to refresh the CTs/VMs on iSCSI connection loss.
Keep the Proxmox PC, but switch it to iESXI hoping that it handles the iSCSI failure more gracefully
EDIT: I didn’t make it clear at first - TrueNAS stores more data than just VMs - documents, Linux ISOs ™, photos, Syncthing
I edited my post to clarify that TrueNAS keeps more than just VMs. It has photos, documents etc. as well.
Generally when people run two different servers at home, they keep the VM drives on the hypervisor and just use the NAS for storing bigger things like media files
This is simple and makes sense as well. My TrueNAS is only 2 HDDs, which is not ideal for VMs. I could get a larger drive SSD/M.2 drive for the hypervisor, though the Lenovo M920q supports 1xM.2 and 1x2.5" drive.
Hosting VM drives over iSCSI works in an enterprise environment, but if you can’t guarantee uptime for your storage solution then all you’re doing is adding failure modes.
Well, my whole setup comes from the fact that I wanted to cosplay as an enterprise environment (famous last words for a homelabber). I’ve been powering the TrueNAS up and down a lot due to some electricity-related construction in my apartament, and it brought out this flaw in my setup. I guess an UPS would be in order, as another poster pointed out.
In general if you lose your iscsi storage you are hosed.
The way around this is replication where you write one byte to two locations and pseudo load balancing where you have an active and inactive link. When power on one storage fabric goes down you flip to the other. Iscsi isn’t really good for this use case
You are not logged in. However you can subscribe from another Fediverse account, for example Lemmy or Mastodon. To do this, paste the following into the search field of your instance: !selfhosted@lemmy.world
A place to share alternatives to popular online services that can be self-hosted without giving up privacy or locking you into a service you don’t control.
Rules:
Be civil: we’re here to support and learn from one another. Insults won’t be tolerated. Flame wars are frowned upon.
No spam posting.
Posts have to be centered around self-hosting. There are other communities for discussing hardware or home computing. If it’s not obvious why your post topic revolves around selfhosting, please include details to make it clear.
Don’t duplicate the full text of your blog or github here. Just post the link for folks to click.
Submission headline should match the article title (don’t cherry-pick information from the title to fit your agenda).
No trolling.
Resources:
selfh.st Newsletter and index of selfhosted software and apps
I edited my post to clarify that TrueNAS keeps more than just VMs. It has photos, documents etc. as well.
This is simple and makes sense as well. My TrueNAS is only 2 HDDs, which is not ideal for VMs. I could get a larger drive SSD/M.2 drive for the hypervisor, though the Lenovo M920q supports 1xM.2 and 1x2.5" drive.
Well, my whole setup comes from the fact that I wanted to cosplay as an enterprise environment (famous last words for a homelabber). I’ve been powering the TrueNAS up and down a lot due to some electricity-related construction in my apartament, and it brought out this flaw in my setup. I guess an UPS would be in order, as another poster pointed out.
I feel that. Experimenting I get, but I’d never have it be my primary vm backing storage. Esp. not on a 1Gb network, no ups, no redundancy.
Someone else suggested local vm storage and a PBS VM on the TrueNas box. I think that’s a solid solution to consider.
In general if you lose your iscsi storage you are hosed.
The way around this is replication where you write one byte to two locations and pseudo load balancing where you have an active and inactive link. When power on one storage fabric goes down you flip to the other. Iscsi isn’t really good for this use case