I’m thinking about starting a self hosting setup, and my first thought was to install k8s (k3s probably) and containerise everything.
But I see most people on here seem to recommend virtualizing everything with proxmox.
What are the benefits of using VMs/proxmox over containers/k8s?
Or really I’m more interested in the reverse, are there reasons not to just run everything with k8s as the base layer? Since it’s more relevant to my actual job, I’d lean towards ramping up on k8s unless there’s a compelling reason not to.
So first, kubernetes is a different ball of wax than containers, and if you want to run it on one machine you can, but it’s really for running containers across a cluster of machines. I’m guessing you just generally mean containers so I’ll go with that.
Containers are essentially just apps running on a virtual os. Virtual machines are an OS running on virtual hardware. You can abstract both layers and have virtual hardware running an os that runs a virtual os for your containers, and nothing will really mind - in fact that’s kind of the way to do it if you have one big machine you need to run a bunch of services on. You might cut up a server into a Linux VM, a Windows VM, and a BSD VM, and run containers on each one. Or you might run 3 Linux VMs and have the containers for 3 different services split between them.
It really depends on what you’re hosting and trying to do for how exactly to go about it. Take for instance a pretty common self hosted stack:
Plex
Radarr
Prowlarr
Deluge
TrueNAS
Now you could install TrueNAS scale and run all of those as containers on it, and it would work ok, but TrueNAS scale isn’t really meant for managing a ton of containers right now. You could make a vm on it for each service and have them all talk to each other but then you’re probably wasting resources by duplicating the OS 5 times. Also, what if you want to run TrueNAS core instead of scale? Can you get everything else working in jails – maybe? – but it’ll probably be a pain.
Instead, you might install proxmox and pass through the drive controller, and set up one VM for TrueNAS core. Then you might make another VM for the arrs containers, and a third for Plex itself.
It gets you the best of both worlds. TrueNAS can run on BSD instead of Linux, your arrs are easy to deploy and update in containers that keep everything separated, and Plex is sequestered in a hardened os with read only access to everything else since it gets a port forwarded and is more of a security risk. Again that’s just one option though.
VMs get you a ton of really handy things like snapshots and for simple VMs, very easy portability between relatively similar hardware. I’ll probably get ruined for saying this but they’re also a security tool that you should probably keep in your belt. If someone manages to break out of a container and your files are just sitting there for the taking that’s not great. If someone manages to break into your VM and “the good stuff” is on another VM that’s another layer of security they have to break through.
Containers on the other hand use way fewer resources, especially ram - and are much easier to wrangle than many OSes for updates and config.
There’s really a lot of self hosted stuff that assumes you’re running docker and treats regular install as a kind of weird edge case, so you’ll probably run docker even if you don’t want to.
Kubernetes on the other hand I would argue isn’t really meant for self hosting where you probably have a one or two servers that you own. Its meant to deploy containers across various cloud servers in a way that’s more automated to manage. If you need storage in a kubernetes cluster you’ll probably use something like s3 buckets, not a hard drive.
If you want to learn it you can totally deploy it on a computer running a few VMs as nodes or with a few laptops / SBCs as a cluster, but if you just want the services to run on your server in the closet it’s a bit like using a sledgehammer to nail a chair back together. That’s why you don’t tend to see it talked about as much - it’s a bit of a different rabbit hole.
VMs are often imperative and can be quite easy and familiar to setup for most people, but can be harder or more time-consuming to reproduce, depending on the type of update or error to be fixed. They have their own kernel and can have window managers and graphical interfaces, and can therefore also be a bit resource heavy.
Containers are declarative and are quite easy to reproduce, but can be harder to setup, as you’ll have to work by trial-and-error from the CLI. They also run on your computers kernel and can be extremely slimmed down.
They are both powerful, depends how you want to maintain and interface with them, how resource efficient you want them to be, and how much you’re willing to learn if necessary.
I generally tend to try to use containers for everything and only branch out to VMs if it doesn’t work or I need more separation.
This is my general recommendation as containers are easier to set up and in my opinion individual software packages are easier to maintain with things like compose. I have limited time for my self hosted instance and that took away a lot of work, especially when updating.
I generally tend to try to use containers for everything and only branch out to VMs if it doesn’t work or I need more separation.
This is my general recommendation as containers are easier to set up and in my opinion individual software packages are easier to maintain with things like compose. I have limited time for my self hosted instance and that took away a lot of work, especially when updating.
I’m running a 3 pi cluster with k3s at the moment. The main benefit I’ve found is that all my pis run exactly the same software setup as a base so it’s easy to add new ones or replace/update one. I use a deployment management application to push my deployments too which means it’s super easy to redeploy everything if something goes funky.
That can be fun. The benefit of kubernetes is flexibility in the orchestration and (sometimes) scaling. Also the tooling in Kubernetes is more sofisticated compared to plain containers or manual services.
Kubernetes is basically just a finite-state machine that is able to manage a certain number of nodes as a pool of resources. This has added complexity compared to you managing the scheduling (I.e. I install this service on this box and this on this other box), but it also allows for much easier automation.
The basics can be useful there. The whole idea with k8s is to be able to run applications across multiple hosts in a given fleet. Your cluster can be that fleet! :)
I get why too. I’m a full stack (including devops) software engineer, and docker/k8s is just completely opaque to me. I’m not sure why, but I really just can’t wrap my head around it. Thankfully my current company has a devops team that takes care of it, but jeez
Tbh those stuff aren’t really intuitive. But, as was my case for instance, that’s something that can be “easily” learnt as a hobbyist like us. And when you understand those concepts, at least from an abstract point, my stance is that you can become a better dev/ops/sys :) I strongly advice anyone in the field to at least play a little with Docker/containers to grasp what it is.
I, personally, haven’t done a whole lot of VM work but I do run a metric ass-ton of containers. I can spool up servers in docker compose on absolutely dogshit hardware and have it run serviceably. Also, the immutability of the container OS is really nice for moving things around and/or getting them set up quickly.
Just get started somewhere. I ran traditional VMs for most things before and I would never go back unless it was necessary for something.
Easiest way is just to start using Docker for some service you’re hosting that has a public image available and go from there. If you want a more visual approach there’s stuff like Portainer you can use too.
Also get started early on with docker compose, it makes it much easier to organize your container configs.
Not a proxmox pro by any means, but it can do both VMs and containers. I have a few VMs for various Linux distros to play around with. I also have one dedicated VM for all my security related tools.
Stuff like PI hole, jellyfin, logstash, etc. dont really have any need for a full OS, so a container works perfectly. Plus having a full OS with several things running on it makes it more difficult if you just need to restart one service
I started doing everything in VMs but over time realized some things were better to maintain as containers
It depends on your use case and what you are trying to achieve.
You do not need k8s (or k3s…) to use containers though. Plain old containers could also suffice, or Docker Swarm if you need some container orchestration functionality.
Trying to learn k8s would be a good reason to use k8s though :)
If it’s relevant for your job, go for k8s. The more you tinker with it, the more knowledge you’ll accumulate. Is it the optimal solution for a self hosting setup? Well, it depends but most probably not.
Debian hypervisor with raidz2 hosting vms, the main ones being 1 main freebsd host with 20 jails containing 1-2 apps each, and 1 main debian vm hosting things that are too much of a pain in the ass to get running on freebsd, so it hosts 5 docker containers.
Like many others here, I went with Proxmox as the base host. But most of my services are Docker containers , running in a “dockerVM” on top of Proxmox.
Having Proxmox as the base is just so flexible, which is very handy for a homelab.
For instance I set up a VM with Wireguard back when Wireguard had only just been merged in to the mainline kernel, without affecting the other
You can have separate VM for docker testing, and docker production
You can run multiple VMs for multiple Kubernetes hosts, to try it out and get your feet wet without affecting the “production” containers
If you get additional servers, you can just migrate those Kubernetes VMs
You can run Windows VM should you need, and BSD (and thus pfSense/opensense or TRUE AS)
You can run a full graphical environment if you want
Proxmox has easy setup for firewalls for each VM
I have a VM running a legacy bare metal system (from the same server now running proxmox) that I’ve been slowly de-commissioning piece by piece
What is your system backup solution like? Having it separated seems convenient for that since you can just back up the vm storage somewhere I’m guessing?
Not OP, but similar setup (Proxmox with docker on a VM). The VM (plus a few LXCs) are backed up daily using the backup built into Proxmox, and those backups are mirrored to the cloud with rclone.
This is exactly what I do for my personal servers (except with ESXi instead of proxmox).
You will probably want both VMs and containers, there are some things that are not well supported in containers (e.g. gitlab).
I run a couple k8s clusters for work and the complexity is beyond what most people starting out would want, I would imagine.
Unless you need something that has a helm chart but not docker support (e.g. gitlab) or you are really keen on learning, it can be quite a jump…
(For gitlab I still would recommend a VM with the omnibus installer over k8s unless you are big enough to have a separate team managing your k8s clusters. It would suck to have a PV issue and lose all your data.)
Just to add my two cents: When I started out I thought I’d need a datacenter, with 10 Gig connectivity and a lot of storage. Turns out, a Raspberry Pi 4 8GB would’ve been sufficient for the things I actually use.
My recommendation would be therefore to start minimalistic and build up according to your needs from there. Start with a Raspberry PI and Docker or use a used Micro SFF and go up from there, this advice would’ve saved me a lot of money and electricity.
I personally really, really like (Docker) containers and I host most of my stuff with it, on a Raspberry Pi and on (free tier) Oracle Cloud VPS’s. I also plan to (re)install Proxmox on a spare old laptop and run some stuff in VMs on that (namely Home Assistant) and might try a NixOS server too.
So really, use both. Use the right tool for the job. And you can also run containers in VMs and even use Ansible to configure everything with playbooks, allowing you to re-run said playbooks when things go wrong.
You are not logged in. However you can subscribe from another Fediverse account, for example Lemmy or Mastodon. To do this, paste the following into the search field of your instance: !selfhosted@lemmy.world
A place to share alternatives to popular online services that can be self-hosted without giving up privacy or locking you into a service you don’t control.
Rules:
Be civil: we’re here to support and learn from one another. Insults won’t be tolerated. Flame wars are frowned upon.
No spam posting.
Posts have to be centered around self-hosting. There are other communities for discussing hardware or home computing. If it’s not obvious why your post topic revolves around selfhosting, please include details to make it clear.
Don’t duplicate the full text of your blog or github here. Just post the link for folks to click.
Submission headline should match the article title (don’t cherry-pick information from the title to fit your agenda).
No trolling.
Resources:
selfh.st Newsletter and index of selfhosted software and apps
Basically, it’s “why not both?”
So first, kubernetes is a different ball of wax than containers, and if you want to run it on one machine you can, but it’s really for running containers across a cluster of machines. I’m guessing you just generally mean containers so I’ll go with that.
Containers are essentially just apps running on a virtual os. Virtual machines are an OS running on virtual hardware. You can abstract both layers and have virtual hardware running an os that runs a virtual os for your containers, and nothing will really mind - in fact that’s kind of the way to do it if you have one big machine you need to run a bunch of services on. You might cut up a server into a Linux VM, a Windows VM, and a BSD VM, and run containers on each one. Or you might run 3 Linux VMs and have the containers for 3 different services split between them.
It really depends on what you’re hosting and trying to do for how exactly to go about it. Take for instance a pretty common self hosted stack:
Plex Radarr Prowlarr Deluge TrueNAS
Now you could install TrueNAS scale and run all of those as containers on it, and it would work ok, but TrueNAS scale isn’t really meant for managing a ton of containers right now. You could make a vm on it for each service and have them all talk to each other but then you’re probably wasting resources by duplicating the OS 5 times. Also, what if you want to run TrueNAS core instead of scale? Can you get everything else working in jails – maybe? – but it’ll probably be a pain.
Instead, you might install proxmox and pass through the drive controller, and set up one VM for TrueNAS core. Then you might make another VM for the arrs containers, and a third for Plex itself.
It gets you the best of both worlds. TrueNAS can run on BSD instead of Linux, your arrs are easy to deploy and update in containers that keep everything separated, and Plex is sequestered in a hardened os with read only access to everything else since it gets a port forwarded and is more of a security risk. Again that’s just one option though.
VMs get you a ton of really handy things like snapshots and for simple VMs, very easy portability between relatively similar hardware. I’ll probably get ruined for saying this but they’re also a security tool that you should probably keep in your belt. If someone manages to break out of a container and your files are just sitting there for the taking that’s not great. If someone manages to break into your VM and “the good stuff” is on another VM that’s another layer of security they have to break through.
Containers on the other hand use way fewer resources, especially ram - and are much easier to wrangle than many OSes for updates and config.
There’s really a lot of self hosted stuff that assumes you’re running docker and treats regular install as a kind of weird edge case, so you’ll probably run docker even if you don’t want to.
Kubernetes on the other hand I would argue isn’t really meant for self hosting where you probably have a one or two servers that you own. Its meant to deploy containers across various cloud servers in a way that’s more automated to manage. If you need storage in a kubernetes cluster you’ll probably use something like s3 buckets, not a hard drive.
If you want to learn it you can totally deploy it on a computer running a few VMs as nodes or with a few laptops / SBCs as a cluster, but if you just want the services to run on your server in the closet it’s a bit like using a sledgehammer to nail a chair back together. That’s why you don’t tend to see it talked about as much - it’s a bit of a different rabbit hole.
also curious abt this
VMs are often imperative and can be quite easy and familiar to setup for most people, but can be harder or more time-consuming to reproduce, depending on the type of update or error to be fixed. They have their own kernel and can have window managers and graphical interfaces, and can therefore also be a bit resource heavy.
Containers are declarative and are quite easy to reproduce, but can be harder to setup, as you’ll have to work by trial-and-error from the CLI. They also run on your computers kernel and can be extremely slimmed down.
They are both powerful, depends how you want to maintain and interface with them, how resource efficient you want them to be, and how much you’re willing to learn if necessary.
That sums it up really well.
I generally tend to try to use containers for everything and only branch out to VMs if it doesn’t work or I need more separation.
This is my general recommendation as containers are easier to set up and in my opinion individual software packages are easier to maintain with things like compose. I have limited time for my self hosted instance and that took away a lot of work, especially when updating.
That sums it up really well.
I generally tend to try to use containers for everything and only branch out to VMs if it doesn’t work or I need more separation.
This is my general recommendation as containers are easier to set up and in my opinion individual software packages are easier to maintain with things like compose. I have limited time for my self hosted instance and that took away a lot of work, especially when updating.
Unless you have multiple systems, I don’t think k8s will yield much benefit over plain docker.
A multitude of things are far easier to do on Kubernetes. If you combine it with an immutable OS, then less effort too.
So, if I plan to build a pi cluster I should get familiar with k8s?
I’m running a 3 pi cluster with k3s at the moment. The main benefit I’ve found is that all my pis run exactly the same software setup as a base so it’s easy to add new ones or replace/update one. I use a deployment management application to push my deployments too which means it’s super easy to redeploy everything if something goes funky.
That can be fun. The benefit of kubernetes is flexibility in the orchestration and (sometimes) scaling. Also the tooling in Kubernetes is more sofisticated compared to plain containers or manual services.
Kubernetes is basically just a finite-state machine that is able to manage a certain number of nodes as a pool of resources. This has added complexity compared to you managing the scheduling (I.e. I install this service on this box and this on this other box), but it also allows for much easier automation.
The basics can be useful there. The whole idea with k8s is to be able to run applications across multiple hosts in a given fleet. Your cluster can be that fleet! :)
Also k8s is in high demand in the sector, so those are good skills that could be turned into $$
I get why too. I’m a full stack (including devops) software engineer, and docker/k8s is just completely opaque to me. I’m not sure why, but I really just can’t wrap my head around it. Thankfully my current company has a devops team that takes care of it, but jeez
Tbh those stuff aren’t really intuitive. But, as was my case for instance, that’s something that can be “easily” learnt as a hobbyist like us. And when you understand those concepts, at least from an abstract point, my stance is that you can become a better dev/ops/sys :) I strongly advice anyone in the field to at least play a little with Docker/containers to grasp what it is.
I can’t even get my head around Traefik, let alone Kubernetes (and k3 vs k8???)
I, personally, haven’t done a whole lot of VM work but I do run a metric ass-ton of containers. I can spool up servers in docker compose on absolutely dogshit hardware and have it run serviceably. Also, the immutability of the container OS is really nice for moving things around and/or getting them set up quickly.
Where did you learn so much about Docker? Having a server at home, I’m more inclined to spin up a VM. I would like to learn more about Docker.
Just get started somewhere. I ran traditional VMs for most things before and I would never go back unless it was necessary for something.
Easiest way is just to start using Docker for some service you’re hosting that has a public image available and go from there. If you want a more visual approach there’s stuff like Portainer you can use too.
Also get started early on with docker compose, it makes it much easier to organize your container configs.
If I’m honest, I’ve stumbled nose-first through pretty much everything I know. I am never afraid to break things as long as I learn from it.
I think it depends on your scale. If homelab stuff docker is awesome IMO.
Not a proxmox pro by any means, but it can do both VMs and containers. I have a few VMs for various Linux distros to play around with. I also have one dedicated VM for all my security related tools.
Stuff like PI hole, jellyfin, logstash, etc. dont really have any need for a full OS, so a container works perfectly. Plus having a full OS with several things running on it makes it more difficult if you just need to restart one service
I started doing everything in VMs but over time realized some things were better to maintain as containers
K8s are more complex than containers using proxmox. If you are up for the challenge sure go crazy.
It depends on your use case and what you are trying to achieve.
You do not need k8s (or k3s…) to use containers though. Plain old containers could also suffice, or Docker Swarm if you need some container orchestration functionality.
Trying to learn k8s would be a good reason to use k8s though :)
If it’s relevant for your job, go for k8s. The more you tinker with it, the more knowledge you’ll accumulate. Is it the optimal solution for a self hosting setup? Well, it depends but most probably not.
Por que los dos?
Debian hypervisor with raidz2 hosting vms, the main ones being 1 main freebsd host with 20 jails containing 1-2 apps each, and 1 main debian vm hosting things that are too much of a pain in the ass to get running on freebsd, so it hosts 5 docker containers.
Rocks my world.
This is the way
Why not both?
Like many others here, I went with Proxmox as the base host. But most of my services are Docker containers , running in a “dockerVM” on top of Proxmox.
Having Proxmox as the base is just so flexible, which is very handy for a homelab.
What is your system backup solution like? Having it separated seems convenient for that since you can just back up the vm storage somewhere I’m guessing?
Not OP, but similar setup (Proxmox with docker on a VM). The VM (plus a few LXCs) are backed up daily using the backup built into Proxmox, and those backups are mirrored to the cloud with rclone.
Proxmox Backup Server: Incremental de-duplicateed image backups of the whole VM, with possibility of individual file restore. It’s like magic
For the legacy bare metal system I have rsnapshots of the data folder (set it up ages ago, and never changed it)
An nginx LXC container has a single static backup of the container, with the nginx config file stored in a git repo
I use proxmox for the sole benefit of just spinning up a VM of choice without having to deal with usb-sticks, etc.
From there I just run everything with Docker containers, via Portainer.
This is exactly what I do for my personal servers (except with ESXi instead of proxmox).
You will probably want both VMs and containers, there are some things that are not well supported in containers (e.g. gitlab).
I run a couple k8s clusters for work and the complexity is beyond what most people starting out would want, I would imagine.
Unless you need something that has a helm chart but not docker support (e.g. gitlab) or you are really keen on learning, it can be quite a jump…
(For gitlab I still would recommend a VM with the omnibus installer over k8s unless you are big enough to have a separate team managing your k8s clusters. It would suck to have a PV issue and lose all your data.)
Just to add my two cents: When I started out I thought I’d need a datacenter, with 10 Gig connectivity and a lot of storage. Turns out, a Raspberry Pi 4 8GB would’ve been sufficient for the things I actually use.
My recommendation would be therefore to start minimalistic and build up according to your needs from there. Start with a Raspberry PI and Docker or use a used Micro SFF and go up from there, this advice would’ve saved me a lot of money and electricity.
I personally really, really like (Docker) containers and I host most of my stuff with it, on a Raspberry Pi and on (free tier) Oracle Cloud VPS’s. I also plan to (re)install Proxmox on a spare old laptop and run some stuff in VMs on that (namely Home Assistant) and might try a NixOS server too.
So really, use both. Use the right tool for the job. And you can also run containers in VMs and even use Ansible to configure everything with playbooks, allowing you to re-run said playbooks when things go wrong.