I want to reset my server soon and I’m toying with the idea of using a different operating system. I am currently using Ubuntu Server LTS. However, I have been toying with the idea of using Fedora Server (I use Fedora on my laptop and made good experiences with it) or even Fedora CoreOS. I also recently installed NixOS on my desktop computer and find the declarativeness pretty cool (but I’m still a complete beginner) and could imagine that it would fit well into a server setup.
I have quite a few services running on my server, such as Nextcloud, Conduit (Matrix), Jellyfin, etc. and all in containers. I would also rather not install programs without containers, because 1. compose is super easy to maintain and set up, 2. it remains very clear with containers (and compose) and 3. I believe that containers are more secure. But since I also want to make the services inside the containers available, I currently have Nginx installed as a reverse proxy (not in the container, but on the system) and always create certificates with certbot so that I can use HTTPS encryption.
In the paragraph above I actually described exactly the use-case of Fedora CoreOS, but I have no experience with the system and how it works. That’s why I’m still a bit hesitant at considering the OS at the moment.
I can imagine that NixOS with its declarative nature seems well suited, since, as I have heard, you can configure containers as well as Nginx and with Nginx also https certificates declaratively.
But I could also use a base system like before (Fedora Server or Ubuntu Server) and simply install podman, nginx and certbot and manage everything that way.
Have you had any experience with Fedora Server, Fedora CoreOS, NixOS or a completely different operating system for servers and what are/were your impressions with this setup? Or do you just want to share your knowledge here? I would be delighted.
You can also use container within NixOS and AFAICT even declare the containers which should be running. Also NixOS is sad to be stable, or am I missing something?
Yeh, but I already have compose files and ansible things to set up a server.
And I’d have to figure out how health checks and depends-on works for that.
I’m sure it would give me an amazing experience, but I have all the tools and I can run them in isolation (ie I can install docker on any os I can SSH into)
Nixos will use/download cached binaries that are available in its repo. It has one of the biggest repositories of any Linux distro. It’s on par with Arch with around 90 thousand packages.
Unless you are doing something custom or niche, your nixos won’t have to compile anything.
Are all those packages available in binary format? Not familiar with Nix but that’s certainly not the case for Arch. Arch has 85k packages in the AUR as source recipes but not as binaries.
I still think Debian makes a better use case for a server since it provides everything as binaries.
If you’re going to use binaries what’s the point of using Nix anyway? The declarative aspect is nice in an abstract sort of way but you can achieve a system deploy or restore just as fast by installing a vanilla system and a few config files.
I have used all three! I started with Server then went to CoreOS running Kubernetes and settled on NixOS which I have been very happy with for about a year now. I run about 25-30 services all using built in modules.
Regarding security, if you are using well crafted modules on NixOS, there should be good systemd hardening in place. That being said there is no reason you can’t just use containers on NixOS.
I also find deploying NixOS far superior to butane/ignition used by CoreOS/Fedora. I use nixos-anywhere and can deploy my entire server in a few minutes without manual intervention.
I’m a NixOS convert too. I actually still run everything in docker but the idea of not having to remeber or document how I set everything up was too appealing…a year later and my desktop and laptop are now on nix too
I am also thinking about installing nix on my laptop, but I need a proprietary library for work, which is kinda hard to install/ not working on NixOS. But we might be able to just use docker for development. Well, that is currently preventing me from installing nix on my laptop, I am still looking for a way to fix that issue.
I use Debian at home on my homeserver and a mix of Debian and Arch for my workstations. Most of my stuff is managed with Ansible to make rebuilding easier and most workloads in podman containers.
Personally I don’t overthink the distro thing. I recently started using Arch and quite like it. I’ve noticed packages that are available in Debian but not Arch and vice-versa. Debian Stable is nice because it’s just, well, stable.
Fedora has an annoying release cadence IMO. I have experienced desktop bugs in the early GA releases before which put me off. If I wanted instability I would sooner go with Arch (and I am yet to have many issues with Arch yet).
If I were to go with a BSD for a home server it would probably be OpenBSD or FreeBSD. OpenBSD has vmm and a bunch of tooling around it, and FreeBSD has bhyve and jails. I haven’t taken the plunge because Linux works and it’s what I know.
These days I hear about people using proxmox on their homeserver with LXC containers and/or VMs.
Fedora has an annoying release cadence IMO. I have experienced desktop bugs in the early GA releases before which put me off. If I wanted instability I would sooner go with Arch (and I am yet to have many issues with Arch yet).
Do you mean they are too frequent, or what do you mean?
I went with Fedora on my VPS because I was also planning to use rootless Podman. Quadlets and running everything through systemd with SELinux enabled is working pretty well for me.
If you mainly use containers perhaps OpenSUSE Micro OS is of interest to you. Other then that pretty much any distro will do. I use rocky Linux my self for a few different things.
If you want to try out many different distros virtualization is also a option. KVM or something like XCP-NG with XO or proxmox are great options.
I really really like Fedora Server, but any RHEL derivative is my go to for servers. I use Rocky Linux when I need something closer to RHEL, and Fedora server for pretty much everything else. I highly recommend Cockpit as well (main reason I like Fedora server) as it has allowed me to so easily manage all of my servers from a single point.
Portage has supported binary packages since forever, back in 2012 I had some binary packages on my system, I clearly remember because it was a pain in the ass to compile certain things, for those I installed the binary version. It’s like Debian supporting source packages, it’s been there since forever but people don’t know about it.
The point is that they have recently focused on better binary package availability. Sure they always had support for binary packages but most software needed to be compiled.
I’m probably the odd one out, but my home server is running Arch Linux. And it’s been really smooth. I do weekly maintenance in the form of updates and cleanup and it’s been reliable since I set it up a couple of years ago.
I’ve been running Arch on my servers for over a decade now and never had issues. I think people have a perception that it’s not stable or it randomly breaks but that’s not been the case for me.
I’m probably the odd one out, but my home server is running Arch Linux. And it’s been really smooth. I do weekly maintenance in the form of updates and cleanup and it’s been reliable since I set it up a couple of years ago.
I am basically doing the same right now, all by hand. It’s just that I am not doing the system and container updates regularly. I also often forget which services I have running and some of the Services I am not even using anymore. I just wanted to give them a try and now they are sitting there, wasting (barely any, but it’s nagging me) resources.
I’ve been swapping between Arch with OpenZFS and FreeNAS/TrueNAS for probably 5-7 years now. In fact, I’m doing that right now! I think SCALE is finally stable enough to my liking…but we’ll see.
ZFS becomes a pain to manage via the CLI when you have more than a few disks, a nice web GUI takes the pain away.
I use Fedora CoreOS on my homeserver and a bunch of VPSs. Migrated the homeserver just recently, but I’ve migrated the first VPSs a bit more than a year ago. So far, I had no problems with it. There’s a low-traffic mailing list where the devs inform about security issues and breaking changes to the whole container stack.
I used debian before for some years, but at some point became tired of manually updating the system (which is probably one of the biggest benefit of FCOS). It takes, however, quite some time to put your first Ignition config together, and debugging is tedious as you have to redeploy to see if a bug / error is now gone (I’ve used a VM for that).
I use podman on some, Docker on other servers (you can’t use both at the same time). Both have been working well so far.
I’d recommend it, but would also recommend taking a look at Flatcar Linux which is more or less the same without the IBM dependency (which makes my stomach hurt sometimes).
I’d recommend it, but would also recommend taking a look at Flatcar Linux which is more or less the same without the IBM dependency (which makes my stomach hurt sometimes).
Why exactly are the IBM dependencies a problem for you?
I used debian before for some years, but at some point became tired of manually updating the system (which is probably one of the biggest benefit of FCOS). It takes, however, quite some time to put your first Ignition config together, and debugging is tedious as you have to redeploy to see if a bug / error is now gone (I’ve used a VM for that).
I can’t really find good resources on how FCOS is working and what are the benefits. Is it updating the system/kernel automatically as well as the containers? And what are generally, in your opinion, the advantages of FCOS?
Why exactly are the IBM dependencies a problem for you?
I guess I just like independent, community-driven distros, since there’s less space for financially motivated enshittification. Just shortly after I decided to go with FCOS, RedHat / IBM decided to close down CentOS, for example.
I can’t really find good resources on how FCOS is working and what are the benefits. Is it updating the system/kernel automatically as well as the containers?
The system & kernel yes. The whole system is basically a read-only system “image” for which the devs make sure all the packages play nicely together. Packages are not updated individually, but whole system “image” are released periodically, which the system then downloads automatically and reboots (you decide when it actually reboots through the config). If anything goes wrong, the system is rolled back to the previous “image”.
When you go with podman, there’s a systemd service you can enable which will update the containers (i.e. pull the specified image tag). I’m not aware of a similar mechanism for Docker, which is why I use watchtower for that which has been working smoothly so far.
Edit:
And what are generally, in your opinion, the advantages of FCOS?
For me, it’s the (quite safely designed) auto-updates of the base system (I just feel like having to do less repetitive work), infrastructure-as-code aspect, and the container mindset (as I containerize everything anyways). Also I just have a weakness for new, fancy stuff.
You might want to consider just Dockerising everything. That way, the underlying OS really doesn’t matter to the applications running.
I’ve got a few Raspberry Pi’s running Debian, and on top of that, they’re running a kubernetes cluster with K3s. I host a bunch of different services, all in their own containers (effectively their own OS) and I don’t have to care. If I want to change the underlying OS, the containers don’t know either. It’s pretty great.
Slackware is a great, simple OS that does what it does and does it well. There will be some getting used to, but when it clicks, it makes sense and doesn’t do anything you wouldn’t expect. It is great if you want to use containers as it provides you with the stable, simple base to run all your containers on top of.
You are not logged in. However you can subscribe from another Fediverse account, for example Lemmy or Mastodon. To do this, paste the following into the search field of your instance: !selfhosted@lemmy.world
A place to share alternatives to popular online services that can be self-hosted without giving up privacy or locking you into a service you don’t control.
Rules:
Be civil: we’re here to support and learn from one another. Insults won’t be tolerated. Flame wars are frowned upon.
No spam posting.
Posts have to be centered around self-hosting. There are other communities for discussing hardware or home computing. If it’s not obvious why your post topic revolves around selfhosting, please include details to make it clear.
Don’t duplicate the full text of your blog or github here. Just post the link for folks to click.
Submission headline should match the article title (don’t cherry-pick information from the title to fit your agenda).
Acronyms, initialisms, abbreviations, contractions, and other phrases which expand to something larger, that I’ve seen in this thread:
[Thread #672 for this sub, first seen 11th Apr 2024, 15:35] [FAQ] [Full list] [Contact] [Source code]
Good bot
I always think about using nixos. But considering I dockerise everything, I always end up using Debian.
Good old stable Debian
You can also use container within NixOS and AFAICT even declare the containers which should be running. Also NixOS is sad to be stable, or am I missing something?
Yeh, but I already have compose files and ansible things to set up a server.
And I’d have to figure out how health checks and depends-on works for that.
I’m sure it would give me an amazing experience, but I have all the tools and I can run them in isolation (ie I can install docker on any os I can SSH into)
Just that compiling packages on a server is not ideal.
Nixos will use/download cached binaries that are available in its repo. It has one of the biggest repositories of any Linux distro. It’s on par with Arch with around 90 thousand packages.
Unless you are doing something custom or niche, your nixos won’t have to compile anything.
Are all those packages available in binary format? Not familiar with Nix but that’s certainly not the case for Arch. Arch has 85k packages in the AUR as source recipes but not as binaries.
I still think Debian makes a better use case for a server since it provides everything as binaries.
If you’re going to use binaries what’s the point of using Nix anyway? The declarative aspect is nice in an abstract sort of way but you can achieve a system deploy or restore just as fast by installing a vanilla system and a few config files.
Yes, all packages in nixos are available as binaries to download.
The comparison with Arch was just in terms of number of packages. Not the binary availability.
At the bottom of this page, they say that binary cache is currently at 120TB. https://nixos.org/community/index.html
If packages being available as binaries is the main criteria, nix has you covered there.
The biggest issue for most people with Nixos is the learning curve just because it’s so different.
I have used all three! I started with Server then went to CoreOS running Kubernetes and settled on NixOS which I have been very happy with for about a year now. I run about 25-30 services all using built in modules.
Regarding security, if you are using well crafted modules on NixOS, there should be good systemd hardening in place. That being said there is no reason you can’t just use containers on NixOS.
I also find deploying NixOS far superior to butane/ignition used by CoreOS/Fedora. I use nixos-anywhere and can deploy my entire server in a few minutes without manual intervention.
I’m a NixOS convert too. I actually still run everything in docker but the idea of not having to remeber or document how I set everything up was too appealing…a year later and my desktop and laptop are now on nix too
I am also thinking about installing nix on my laptop, but I need a proprietary library for work, which is kinda hard to install/ not working on NixOS. But we might be able to just use docker for development. Well, that is currently preventing me from installing nix on my laptop, I am still looking for a way to fix that issue.
I use Debian at home on my homeserver and a mix of Debian and Arch for my workstations. Most of my stuff is managed with Ansible to make rebuilding easier and most workloads in podman containers.
Personally I don’t overthink the distro thing. I recently started using Arch and quite like it. I’ve noticed packages that are available in Debian but not Arch and vice-versa. Debian Stable is nice because it’s just, well, stable.
Fedora has an annoying release cadence IMO. I have experienced desktop bugs in the early GA releases before which put me off. If I wanted instability I would sooner go with Arch (and I am yet to have many issues with Arch yet).
If I were to go with a BSD for a home server it would probably be OpenBSD or FreeBSD. OpenBSD has vmm and a bunch of tooling around it, and FreeBSD has bhyve and jails. I haven’t taken the plunge because Linux works and it’s what I know.
These days I hear about people using proxmox on their homeserver with LXC containers and/or VMs.
Do you mean they are too frequent, or what do you mean?
Yeah, too frequent and too buggy. It got annoying having to do upgrades every six months and have to deal with all the new bugs that came with it.
Basically give me Debian-style biannual releases or Arch-style rolling releases.
Fedora Server works well, and the Podman integration is great.
I guess it is the boring option, but probably the best when coming from Ubuntu.
I went with Fedora on my VPS because I was also planning to use rootless Podman. Quadlets and running everything through systemd with SELinux enabled is working pretty well for me.
If you mainly use containers perhaps OpenSUSE Micro OS is of interest to you. Other then that pretty much any distro will do. I use rocky Linux my self for a few different things.
If you want to try out many different distros virtualization is also a option. KVM or something like XCP-NG with XO or proxmox are great options.
What is the difference/benefit to Fedora CoreOS?
See you back on Debian in a few months
Or Proxmox
Proxmox is just Debian 12 with additional software preinstalled
Kind of, yeah. That’s why I replied with it.
Or, better yet, LXD/Incus.
Can’t be hit by new backdoors when your packages haven’t had updates for years 😉
In all seriousness Debian makes solid choices that makes everything as low maintenance as it can get for self hosting.
For someone who recently lost a bunch of their free time, that is amazing to not have to mess with stuff.
I really really like Fedora Server, but any RHEL derivative is my go to for servers. I use Rocky Linux when I need something closer to RHEL, and Fedora server for pretty much everything else. I highly recommend Cockpit as well (main reason I like Fedora server) as it has allowed me to so easily manage all of my servers from a single point.
Debian. Server? Debian.
Debian, Debian and maybe Debian
Give Gentoo a shot. It’s super stable and you will understand everything in your system. Also it now supports binary packages
And by now you mean for the past decade at least.
Huh?
Portage has supported binary packages since forever, back in 2012 I had some binary packages on my system, I clearly remember because it was a pain in the ass to compile certain things, for those I installed the binary version. It’s like Debian supporting source packages, it’s been there since forever but people don’t know about it.
I mean it’s had -k/-K since mid 2000s from what I remember but it’s changed
https://lwn.net/Articles/956428/
The point is that they have recently focused on better binary package availability. Sure they always had support for binary packages but most software needed to be compiled.
I’m probably the odd one out, but my home server is running Arch Linux. And it’s been really smooth. I do weekly maintenance in the form of updates and cleanup and it’s been reliable since I set it up a couple of years ago.
I’ve been running Arch on my servers for over a decade now and never had issues. I think people have a perception that it’s not stable or it randomly breaks but that’s not been the case for me.
My home server also runs arch, mostly because it’s a computer I was using for myself before and I’m lazy and just left what was already there.
I am basically doing the same right now, all by hand. It’s just that I am not doing the system and container updates regularly. I also often forget which services I have running and some of the Services I am not even using anymore. I just wanted to give them a try and now they are sitting there, wasting (barely any, but it’s nagging me) resources.
I’ve been swapping between Arch with OpenZFS and FreeNAS/TrueNAS for probably 5-7 years now. In fact, I’m doing that right now! I think SCALE is finally stable enough to my liking…but we’ll see.
ZFS becomes a pain to manage via the CLI when you have more than a few disks, a nice web GUI takes the pain away.
I use Fedora CoreOS on my homeserver and a bunch of VPSs. Migrated the homeserver just recently, but I’ve migrated the first VPSs a bit more than a year ago. So far, I had no problems with it. There’s a low-traffic mailing list where the devs inform about security issues and breaking changes to the whole container stack.
I used debian before for some years, but at some point became tired of manually updating the system (which is probably one of the biggest benefit of FCOS). It takes, however, quite some time to put your first Ignition config together, and debugging is tedious as you have to redeploy to see if a bug / error is now gone (I’ve used a VM for that).
I use podman on some, Docker on other servers (you can’t use both at the same time). Both have been working well so far.
I’d recommend it, but would also recommend taking a look at Flatcar Linux which is more or less the same without the IBM dependency (which makes my stomach hurt sometimes).
Why exactly are the IBM dependencies a problem for you?
I can’t really find good resources on how FCOS is working and what are the benefits. Is it updating the system/kernel automatically as well as the containers? And what are generally, in your opinion, the advantages of FCOS?
I guess I just like independent, community-driven distros, since there’s less space for financially motivated enshittification. Just shortly after I decided to go with FCOS, RedHat / IBM decided to close down CentOS, for example.
The system & kernel yes. The whole system is basically a read-only system “image” for which the devs make sure all the packages play nicely together. Packages are not updated individually, but whole system “image” are released periodically, which the system then downloads automatically and reboots (you decide when it actually reboots through the config). If anything goes wrong, the system is rolled back to the previous “image”.
When you go with podman, there’s a systemd service you can enable which will update the containers (i.e. pull the specified image tag). I’m not aware of a similar mechanism for Docker, which is why I use watchtower for that which has been working smoothly so far.
Edit:
For me, it’s the (quite safely designed) auto-updates of the base system (I just feel like having to do less repetitive work), infrastructure-as-code aspect, and the container mindset (as I containerize everything anyways). Also I just have a weakness for new, fancy stuff.
Thanks, I might give it a try. I am not settled yet but FCOS sounds very promising. We will see.
Debian
You might want to consider just Dockerising everything. That way, the underlying OS really doesn’t matter to the applications running.
I’ve got a few Raspberry Pi’s running Debian, and on top of that, they’re running a kubernetes cluster with K3s. I host a bunch of different services, all in their own containers (effectively their own OS) and I don’t have to care. If I want to change the underlying OS, the containers don’t know either. It’s pretty great.
Slackware is a great, simple OS that does what it does and does it well. There will be some getting used to, but when it clicks, it makes sense and doesn’t do anything you wouldn’t expect. It is great if you want to use containers as it provides you with the stable, simple base to run all your containers on top of.