Many of the posts I read here are about Docker. Is anybody using Kubernetes to manage their self hosted stuff? For those who’ve tried it and went back to Docker, why?
I’m doing my 3rd rebuild of a K8s cluster after learning things that I’ve done wrong and wanted to start fresh, but when enhancing my Docker setup and deciding between K8s and Docker Swarm, I decided on K8s for the learning opportunities and how it could help me at work.
Love is a strong word, but kubernetes is definitely interesting. I’m finishing up a migration of my homelab from a docker host running in a VM managed with Portainer to one smaller VM and three refurbished lenovo mini PCs running Rancher. It hasn’t been an easy road, but I chose to go with Rancher and k3s since it seemed to handle my usecase better than Portainer and Docker Swarm could. I can’t pass up those cheap mini PCs
Does rancher connect the pcs together? I have like 3 mini pcs sitting around, and I’ve always wanted to kinda combine them somehow
Like being able to combine cpu power or something. Idk if this is possible without getting a mobo with multiple cpu slots, but if it is. I’d love to learn!
Yeah, Kubernetes is designed to run in a cluster so you can pool processing power and memory from multiple devices. I banged my head against the wall for hours trying to figure out how to set up a cluster by hand, but then discovered if you install Rancher in a regular docker container it can handle all that for you
No shit. So you’re saying I can hook up like three mini pcs and make a mega at home server!? I gotta look into this. Did you follow a guide or anything you think is good enough or is as easy as a Google?
My recommendation is to look into k3sup and Rancher. I had a lot of trouble trying to install rancher in a docker container and migrating to a cluster after, and k3sup makes it really easy to set up a k3s cluster without having to configure everything manually
You can accomplish the same task with docker swarm, but I figured it would be better to learn something that wasn’t abandonware
I haven’t dug into the storage side yet since I have a separate NAS, but it will probably be beneficial to set up something like Ceph, GlusterFS, or Longhorn if you don’t have one
I have a K3OS cluster built out of a bunch of raspberry pis, it works well.
The big reason I like kubernetes is that once it is up and running with git ops style management, adding another service becomes trivial.
I just copy paste one if my e is ting services, tweak the names/namespaces, and then change the specific for the pods to match what their docker configuration needs, ie what folders need mounting and any other secrets or configs.
I then just commit the changes to github and apply them to the cluster.
The process of being able to roll back changes via git is awesome
I’d love to hear more about your GitHub to K8s setup. I’ve been thinking about doing something similar, but I’m not sure how to keep my public stuff public while injecting my personalized (private) configuration during deployment.
Running an RKE cluster as VMs on my ceph+proxmox cluster. Using Rook and external ceph as my storage backend and loving it. I haven’t fully migrated all of my services, but thus far it’s working well enough for me!
It’s a damn shame it’s going not free open source, I Just switched my lab over to nomad and consul last year and it has been incredibly smooth sailing.
I’ve been reading into k3s out of curiosity, which as I understand is supposed to be one of the simpler ones, and even as someone who works as a developer and maintains a small homelab, it just makes me feel utterly clueless lol. Which is to say, I’ll definitely be giving Nomad a good look.
Oh and if you do happen to have any other more newbie friendly suggestions, I’d love to hear about them!
Seriously though I changed to nomad/consul/gluster and it’s been wonderful. I still have some other things running on my nas software like Jellyfin and audiobookshelf, but that’s just for performance and simplicity.
I was a bit put off by Hashicorps license change, but I don’t think I’m changing back to k3s anytime soon. Nomad is just so nice and easy.
I was looking into converting my docker services into a cluster to get high availability and to learn it for work, but while investigating it, I read that kubernetes is actually meant for scalability and just a single service per cluster.
Also read that docker swarm is actually what is recommended for my homelab use case. So I’m right now on my way to convert everything to docker stacks. What do you think?
Of course high availability always requires multiple nodes.
Its just that while choosing how to set up my cluster I looked up several options (proxmox, swarm, kubernetes…) and I noticed that kubernetes is generally meant for bigger deployments.
I only need a single replica for each of my containers and they can all run on a single node, so kubernetes is overkill just to get high availability For my use case
I’ve spent the last two weeks on getting a k3s cluster working and I’ve had nothing but problems but it has been a great catalysts for learning new tools like ansible and load balancers. I finally got the cluster working last night. If anyone else is having wierd issues with the cluster timing out ETCD needs fast storage. Moving my VMs from my spinning rust to a cheap SSD fixed all my problems.
Usually though it’s because I run most stuff bare metal anyway so LXC is for temporary or random cases where I need a weird dependency or I want to run a niche service.
Only use docker for when I actually want faster setup like docker-osx which does all the vm stuff for running a virtual Mac for you.
I don’t really mind docker, but for homelab I just find myself rewriting dockerfile anytime I want to change something which I don’t really need to do if I’m not publishing it or even reusing it.
Kubernates is really more effective for actual load services, which you never need in homelab lol. It’s great to use to learn k8s cluster, but the resources get eaten fast.
I find mine useful as both a learning process and as a thing need. I don’t like using cloud services where possible so I can set things up to replace having to rely on those such as next loud for storage, plex and some *arr servers for media etc. And I think once you put the hardware and power costs vs what I’d pay for all the subs (particularly cloud storage costs) it comes out cheaper at least with hardware I’m using.
I was a big proponent of k3s in the homelab, but I’m starting to think otherwise these days. I still expel choice words towards Docker’s networking, but it starts becoming more of a philosophical issue with what the company is doing and whoever decided this kind of networking is nice.
Is the networking on Podman any better? I understand using k8s at home to learn, but what if you don’t care about learning? I have never seen a point to k8s in homelabs other than in home-datacentres, and I’m starting to veer away from k3s too, since I don’t need extreme HA over 3 machines for my services (I would have used Proxmox if I wanted that).
Yeah, could someone give me a primer on how Podman is better than Docker? I’m adamant that I don’t want to use anything with the name “Docker” in my lab.
A lot of people thought this was the case for VMs and docker as well, and now it seems to be the norm.
Yes, but docker does provide features that are useful at the level of a hobbyist self-hosting a few services for personal use (e.g. reproducibility).
I like using docker and ansible to set up my systems, as I can painlessly reproduce everything or migrate to a different VPS in a few minutes.
But kubernetes seems overkill. None of my services have enough traffic to justify replicas, I’m the only user.
Besides learning (which is a valid reason), I don’t see why one would bother setting it up at home. Unless there’s a very specific use-case I’m missing.
For me, I find that I learn more effectively when I have a goal. Sure, it’s great to follow somebody’s “Hello World” web site tutorial, but the real learning comes when I start to extend it to include CI/CD for example.
As far as a use case, I’d say that learning IS the use case.
AKS is a shame. Most of azure, actually. I do my best to find ways around the insanity but it always seems to leak back in with something insane they chose to do for whatever Microsoft reason they have.
I use k8s at work and have built a k8s cluster in my homelab… but I did not like it. I tore it down, and currently using podman, and don’t think I would go back to k8s (though I would definitely use docker as an alternative to podman and would probably even recommend it over podman for beginners even though I’ve settled on podman for myself).
K8s itself is quite resource-consuming, especially on ram. My homelab is built on old/junk hardware from retired workstations. I don’t want the kubelet itself sucking up half my ram. Things like k3s help with this considerably, but that’s not quite precisely k8s either. If I’m going to start trimming off the parts of k8s I don’t need, I end up going all the way to single-node podman/docker… not the halfway point that is k3s.
If you don’t use hostNetworking, the k8s model of traffic routes only with the cluster except for egress is all pure overhead. It’s totally necessary with you have a thousand engineers slinging services around your cluster, but there’s no benefit to this level fo rigor in service management in a homelab. Here again, the networking in podman/docker is more straightforward and maps better to the stuff I want to do in my homelab.
Podman accepts a subset of k8s resource-yaml as a docker-compose-like config interface. This lets me use my familiarity with k8s configs iny podman setup.
Overall, the simplicity and lightweight resource consumption of podman/docker are are what I value at home. The extra layers of abstraction and constraints k8s employs are valuable at work, where we have a lot of machines and alot of people that must coordinate effectively… but I don’t have those problems at home and the overhead (compute overhead, conceptual overhead, and config-overhesd) of k8s’ solutions to them is annoying there.
I’d suggest Podman over docker if someone is starting fresh. I like Podman running as rootless, but moving an existing docker to Podman was a pain. Since the initial docker setup was also a pain, I’d rather have only done it once :/
For me the use case of K8s only makes sense with large use cases (in terms of volume of traffic and users). Docker / Podman is sufficient to self-host something small.
I’m not very familiar with kubernetes or k3s but I thought it was a way to manage docker containers. Is that not the case? I’m considering deploying a k3s cluster in my proxmox environment to test it out.
Kubernetes is abbreviated K8s (because there’s 8 letters between the “k” and the “s”. K3s is a “lite” version. Generally speaking, kubernetes manages your containers. You basicaly tell K8s what the state should be and it does what it needs to do to get the environment as you’ve declared. It’ll check and start or restart services, start containers on a node that can run them (like ensuring enough RAM is available). There’s a lot more, but that’s the general idea.
Oh it is not that much, I run adguard DNS with adblocking, searxng as my search engine, vaultwarden as my password manager. All combined with Argo CD as GitOps engine, nginx ingress with cert-manager for lets encrypt certificates, longhorn as storage layer and metallb as loadbalancer solution. I am planning to completely replace my current setup (which is an old sandy bridge powered HP microserver) with a turing pi 2 clusterboard with 4 RPi4 CMs as soon as they get cheaper.
You are not logged in. However you can subscribe from another Fediverse account, for example Lemmy or Mastodon. To do this, paste the following into the search field of your instance: !selfhosted@lemmy.world
A place to share alternatives to popular online services that can be self-hosted without giving up privacy or locking you into a service you don’t control.
Rules:
Be civil: we’re here to support and learn from one another. Insults won’t be tolerated. Flame wars are frowned upon.
No spam posting.
Posts have to be centered around self-hosting. There are other communities for discussing hardware or home computing. If it’s not obvious why your post topic revolves around selfhosting, please include details to make it clear.
Don’t duplicate the full text of your blog or github here. Just post the link for folks to click.
Submission headline should match the article title (don’t cherry-pick information from the title to fit your agenda).
No “love” from my side, I have thousands of users, not thousands of servers so that’s not a solution for any of my problems :)
Love is a strong word, but kubernetes is definitely interesting. I’m finishing up a migration of my homelab from a docker host running in a VM managed with Portainer to one smaller VM and three refurbished lenovo mini PCs running Rancher. It hasn’t been an easy road, but I chose to go with Rancher and k3s since it seemed to handle my usecase better than Portainer and Docker Swarm could. I can’t pass up those cheap mini PCs
Does rancher connect the pcs together? I have like 3 mini pcs sitting around, and I’ve always wanted to kinda combine them somehow
Like being able to combine cpu power or something. Idk if this is possible without getting a mobo with multiple cpu slots, but if it is. I’d love to learn!
Yeah, Kubernetes is designed to run in a cluster so you can pool processing power and memory from multiple devices. I banged my head against the wall for hours trying to figure out how to set up a cluster by hand, but then discovered if you install Rancher in a regular docker container it can handle all that for you
No shit. So you’re saying I can hook up like three mini pcs and make a mega at home server!? I gotta look into this. Did you follow a guide or anything you think is good enough or is as easy as a Google?
My recommendation is to look into k3sup and Rancher. I had a lot of trouble trying to install rancher in a docker container and migrating to a cluster after, and k3sup makes it really easy to set up a k3s cluster without having to configure everything manually
You can accomplish the same task with docker swarm, but I figured it would be better to learn something that wasn’t abandonware
I haven’t dug into the storage side yet since I have a separate NAS, but it will probably be beneficial to set up something like Ceph, GlusterFS, or Longhorn if you don’t have one
Oh I just realized this is for kubernates. Unraid is all dockers. Can a docker swarm also pool resources?
Yep, similar concept. Not sure how well unraid will handle the swarm behavior but I can imagine there’s someone out there who has tried it before
I have a K3OS cluster built out of a bunch of raspberry pis, it works well.
The big reason I like kubernetes is that once it is up and running with git ops style management, adding another service becomes trivial.
I just copy paste one if my e is ting services, tweak the names/namespaces, and then change the specific for the pods to match what their docker configuration needs, ie what folders need mounting and any other secrets or configs.
I then just commit the changes to github and apply them to the cluster.
The process of being able to roll back changes via git is awesome
I’d love to hear more about your GitHub to K8s setup. I’ve been thinking about doing something similar, but I’m not sure how to keep my public stuff public while injecting my personalized (private) configuration during deployment.
Running an RKE cluster as VMs on my ceph+proxmox cluster. Using Rook and external ceph as my storage backend and loving it. I haven’t fully migrated all of my services, but thus far it’s working well enough for me!
Docker with or without Compose and systemd is good enough for most of my use cases. SaltStack is good enough for config-as-code.
K3s seems pretty cool
K3s? What did you do to the other 5 letters??
https://k3s.io/
Its k3s because its fast and light. It can run on a single core with 512mb of ram
Nomad all the way. K8s is so bloated. Docker swarm can only do docker. Nomad can do basically anything.
It’s a damn shame it’s going not free open source, I Just switched my lab over to nomad and consul last year and it has been incredibly smooth sailing.
Nomad is a breath of fresh air after working with k8s professionally.
Don’t get me wrong, love k8s, but it’s a bit much (until you need it)
I’ve been reading into k3s out of curiosity, which as I understand is supposed to be one of the simpler ones, and even as someone who works as a developer and maintains a small homelab, it just makes me feel utterly clueless lol. Which is to say, I’ll definitely be giving Nomad a good look.
Oh and if you do happen to have any other more newbie friendly suggestions, I’d love to hear about them!
There are dozens of us!
Seriously though I changed to nomad/consul/gluster and it’s been wonderful. I still have some other things running on my nas software like Jellyfin and audiobookshelf, but that’s just for performance and simplicity.
I was a bit put off by Hashicorps license change, but I don’t think I’m changing back to k3s anytime soon. Nomad is just so nice and easy.
I was looking into converting my docker services into a cluster to get high availability and to learn it for work, but while investigating it, I read that kubernetes is actually meant for scalability and just a single service per cluster.
Also read that docker swarm is actually what is recommended for my homelab use case. So I’m right now on my way to convert everything to docker stacks. What do you think?
I’m not sure what you mean by that.
It provides high availability if you have multiple nodes and pods.
Also what do you mean by single service per cluster? Because that’s not the idea at all.
Of course high availability always requires multiple nodes.
Its just that while choosing how to set up my cluster I looked up several options (proxmox, swarm, kubernetes…) and I noticed that kubernetes is generally meant for bigger deployments.
I only need a single replica for each of my containers and they can all run on a single node, so kubernetes is overkill just to get high availability For my use case
I’ve spent the last two weeks on getting a k3s cluster working and I’ve had nothing but problems but it has been a great catalysts for learning new tools like ansible and load balancers. I finally got the cluster working last night. If anyone else is having wierd issues with the cluster timing out ETCD needs fast storage. Moving my VMs from my spinning rust to a cheap SSD fixed all my problems.
I am insane and use bare bone LXC.
Stupid ramblings you can probably ignore:
spoiler
Usually though it’s because I run most stuff bare metal anyway so LXC is for temporary or random cases where I need a weird dependency or I want to run a niche service.
Only use docker for when I actually want faster setup like docker-osx which does all the vm stuff for running a virtual Mac for you.
I don’t really mind docker, but for homelab I just find myself rewriting dockerfile anytime I want to change something which I don’t really need to do if I’m not publishing it or even reusing it.
Kubernates is really more effective for actual load services, which you never need in homelab lol. It’s great to use to learn k8s cluster, but the resources get eaten fast.
Seems a bit overkill for a personal use selfhosting set-up.
Personally, I don’t need anything that requires multiple replicas and loadbalencers.
Do people who have homelabs actually need them? Or is it just for learning?
I find mine useful as both a learning process and as a thing need. I don’t like using cloud services where possible so I can set things up to replace having to rely on those such as next loud for storage, plex and some *arr servers for media etc. And I think once you put the hardware and power costs vs what I’d pay for all the subs (particularly cloud storage costs) it comes out cheaper at least with hardware I’m using.
Yes, those are all great uses of it. But could all still be achieved with docker containers running on some machines at home, right?
Have you ever had a situation where features provided by kubernetes (like replicas, load balancers, etc) came in handy?
I’m not criticizing, I’m genuinely curious if there’s a use-case for kubernetes for personal self-hosting (besides learning).
I was a big proponent of k3s in the homelab, but I’m starting to think otherwise these days. I still expel choice words towards Docker’s networking, but it starts becoming more of a philosophical issue with what the company is doing and whoever decided this kind of networking is nice.
Is the networking on Podman any better? I understand using k8s at home to learn, but what if you don’t care about learning? I have never seen a point to k8s in homelabs other than in home-datacentres, and I’m starting to veer away from k3s too, since I don’t need extreme HA over 3 machines for my services (I would have used Proxmox if I wanted that).
Yeah, could someone give me a primer on how Podman is better than Docker? I’m adamant that I don’t want to use anything with the name “Docker” in my lab.
Not sure if it’s exactly what you’re looking for, but here’s a “primer” on Podman for Docker users: https://lemmy.world/post/213870
Thank you!
A lot of people thought this was the case for VMs and docker as well, and now it seems to be the norm.
Yes, but docker does provide features that are useful at the level of a hobbyist self-hosting a few services for personal use (e.g. reproducibility). I like using docker and ansible to set up my systems, as I can painlessly reproduce everything or migrate to a different VPS in a few minutes.
But kubernetes seems overkill. None of my services have enough traffic to justify replicas, I’m the only user.
Besides learning (which is a valid reason), I don’t see why one would bother setting it up at home. Unless there’s a very specific use-case I’m missing.
For me, I find that I learn more effectively when I have a goal. Sure, it’s great to follow somebody’s “Hello World” web site tutorial, but the real learning comes when I start to extend it to include CI/CD for example.
As far as a use case, I’d say that learning IS the use case.
I do aks. I can’t say love is the right word for it. Lol
AKS is a shame. Most of azure, actually. I do my best to find ways around the insanity but it always seems to leak back in with something insane they chose to do for whatever Microsoft reason they have.
I use k8s at work and have built a k8s cluster in my homelab… but I did not like it. I tore it down, and currently using podman, and don’t think I would go back to k8s (though I would definitely use docker as an alternative to podman and would probably even recommend it over podman for beginners even though I’ve settled on podman for myself).
Overall, the simplicity and lightweight resource consumption of podman/docker are are what I value at home. The extra layers of abstraction and constraints k8s employs are valuable at work, where we have a lot of machines and alot of people that must coordinate effectively… but I don’t have those problems at home and the overhead (compute overhead, conceptual overhead, and config-overhesd) of k8s’ solutions to them is annoying there.
I’d suggest Podman over docker if someone is starting fresh. I like Podman running as rootless, but moving an existing docker to Podman was a pain. Since the initial docker setup was also a pain, I’d rather have only done it once :/
For me the use case of K8s only makes sense with large use cases (in terms of volume of traffic and users). Docker / Podman is sufficient to self-host something small.
Used k3s to manage my single instance. Lots of gotcha moments to learn! Will add Flux for CD after I decide on how to self-host the Git server
I run k3s and all my stuff runs in it no need to deal with docker anymore.
How did you write your templates? Did you use Kompose to translate from Docker compose files, or did you write them from scratch?
I’m not very familiar with kubernetes or k3s but I thought it was a way to manage docker containers. Is that not the case? I’m considering deploying a k3s cluster in my proxmox environment to test it out.
Kubernetes is abbreviated K8s (because there’s 8 letters between the “k” and the “s”. K3s is a “lite” version. Generally speaking, kubernetes manages your containers. You basicaly tell K8s what the state should be and it does what it needs to do to get the environment as you’ve declared. It’ll check and start or restart services, start containers on a node that can run them (like ensuring enough RAM is available). There’s a lot more, but that’s the general idea.
You can use kubernetes on any OCI container deployment.
So if you don’t want/need to install the docker program, you can go with containerd.
deleted by creator
Could you list some of your “stuffs” that you run on your k3s? I’m curious.
Oh it is not that much, I run adguard DNS with adblocking, searxng as my search engine, vaultwarden as my password manager. All combined with Argo CD as GitOps engine, nginx ingress with cert-manager for lets encrypt certificates, longhorn as storage layer and metallb as loadbalancer solution. I am planning to completely replace my current setup (which is an old sandy bridge powered HP microserver) with a turing pi 2 clusterboard with 4 RPi4 CMs as soon as they get cheaper.
Wow you’re self-hosting a password manager! Don’t you feel scared if something went wrong?
I’m also running Adguard as my DNS-level adblocker on my Pi 3. Feels way more content than Pihole.