For this new year, I’d like to learn the skills necessary to self host. Specifically, I would like to eventually be able to self host Nextcloud, Jellyfin and possibly my email server too.
I’ve have a basic level understanding of Python and Kotlin. Now I’m in the process of learning Linux through a virtual machine because I know Linux is better suited for self hosting.
Should I stick with Python? Or is JavaScript (or maybe Ruby) better suited for that purpose? I’m more than happy to learn a new language, but I’m unsure on which is better suited.
And if you could start again in your self hosting journey, what would you do differently? :)
EDIT: I wasn’t expecting all these wonderful replies. You’re all very kind people to share so much with me :)
The consensus seems to be that hosting your own email server might be a lot, so I might leave that as future project. But for Nextcloud and Jellyfin I saw a lot of great tips! I forgot to mention that ideally I would like to have Nextcloud available for multiple users (ie. family memebers) so indeed learning some basic networking/firewalling seems the bare minimum.
I also promise that I will carefully read the manuals!
These 1000% eventually your gonna run into a problem / situation that does not have much documentation. Powering through step by step logically can test the best of us. You can spend 56 hours in a day on one problem. Give up. The next morning figure it out in 10 minutes. It’s a marathon not a sprint.
Honestly, you just need to pick a video on follow along these days. There’s a load of YouTube videos out there that take you step by step.
Lewis rosman recently put out a 14-hour mega video of doing everything, well he might have made some controversial choices, The outcome is quite comprehensive.
It really depends. I actually needed to learn a bit about networking to be able to host multiple things on nginx on the same port. Internally they run on different ports, but they can get routed by the host name
Setting up jellyfin, I used docker on debian, and an old Quadro card. What could possibly go wrong?
Turns out that week the Nvidia drivers got a faulty update pushed to debian stable and caused an error with getting the GPU to work in any container. I could either wait a week or pull the simple fix from testing. So impatiently I pulled it from testing.
There’s no need to learn Python or any programming language to self host stuff, you just need to be able to follow blog posts and run some Docker commands.
I’m a software dev and haven’t touched a single line of code on my NAS. Everything is docker compose and other config files.
You don’t really need to know a specific language to self-host anything. But things like YAML, JSON, Docker, and some networking basic will go a long way.
If I could do anything different though, it would definitely be to write more documentation. Document the steps taking setting things up, log notes on when you have to fix something, archive webpages and videos that you used along the way. Currently doing that myself now after some time self-hosting.
I have friends in the SMB space, one thing they do is a regular backup verification (quarterly). At that frequency, restoring even a few files (especially to a new VM), is very indicative, especially if it’s a large dataset (e.g. Quickbooks).
In Enterprise, we do all sorts of validation, depending on the system. Some is performed as part of Data Center operations, some is by IT (those are separate things), some by Business Unit management and their IT counterparts.
Unfortunately, that wouldn’t have done anything. Because I did that in December and they stopped running like 2 weeks after my verification. I would have caught it on my next scheduled validation, but that doesn’t help me now 😕
Learning any coding language will help you understand a bit more about the programs will work, however there isn’t much need to actually learn a specific language unless you plan to add custom programs or scripts.
The general advice for email is don’t. It’s very risky to host and it’s a big target for spam. Plus there’s challenges getting the big companies to trust your domain.
However hosting things behind a VPN (or locally on your home network) can let you learn a lot about networking and firewalls without exposing yourself to much risk.
I have no direct experience with next cloud but I understand it can be hosted on Linux, you can buy a Synology NAS and run it in that, or use something like TrueNAS.
Personally my setup is on one physical server so I use Proxmox which lets me run 2 different Linux servers and trueNAS on one single computer through virtual machines. I like it because it lets me tinker with different stuff like home assistant and it won’t affect say my adblocker/VPN/reverse proxy. I also use Docker to run multiple services on one virtual machine without compatibility issues.
If I started again, I’d probably have gotten bigger drives or invested in SSDs. My NAS is hard drives because of cost but it’s definitely hitting a limit when I need to pull a bunch of files.
Super happy with wireguard-easy for VPN. I started with a proprietary version of openVPN on Oracle Linux and that was a mistake.
I feel like objecting to the “General advice about email is don’t” thing but I don’t know if I understand the objections well enough to refute them. I self host email for mspencer.net (meaning all requests including DNS are served from hardware in my living space) and I have literally zero spam and can’t remember the last time I had to intervene on my mail server.
On one hand: My emails are received without issue by major providers (outlook, gmail, etc) and I get nearly zero spam. (Two spam senders were using legitimate email services, I reported them, and got human-seeming replies from administrators saying they would take care of it.) And I get amusing pflogsumm (summarizes postfix logs) emails daily showing like 5 emails delivered, 45 rejected, with all of the things that were tried but didn’t work.
On the other: most of the spam prevention comes from greylist, making all new senders retry after a few minutes (because generally a legit MTA will retry while a spammer will not) and that delays most emails by a few minutes. And it was a bear to set up. I used a like 18 step walkthrough on linuxbabe dot com I think, but added some difficulty by storing some use and alias databases on OpenLDAP / slapd instead of in flat files.
But hey, unlimited mail aliases, and I’m thinking of configuring things so emails bounce if they seem to contain just a notification that terms and conditions are updated somewhere. I don’t know, cause some chaos I guess.
And I have no idea if my situation is persuasive for anyone because I don’t know what the general advice means. And I worry it’ll have the unfortunate side effect of making self hosting type nerds like me start forgetting how to run their own email, causing control of email to become more centralized. And I strongly dislike that.
Is there a good way to not self host email yet maintain good control? Like storing it on a local device. I know that addresses are portable with a domain, but still.
I self host many services without any coding languages under my belt. I use Unraid because I found it user friendly for a newb like me. The most important skills I needed were good data backup habits just in case I messed something up, a willingness to read and learn, and the persistence to try more than once.
To self-host, you do not need to know how to code.
I agree but also say that learning enough to be able to write simple bash scripts is maybe required.
There’s always going to be stuff you want to automate and knowing enough bash to bang out a script that does what you want that you can drop into cron or systemd timers is probably a useful time investment.
As others have said, you don’t need to know how to code, but you do need to be comfortable editing structured documents, so knowing a little programming does help.
Unfortunately, Nextcloud and email are two of the most difficult things to self-host. This is by reputation, I haven’t tried myself. Email is supposed to be particularly difficult and the usual advice is to not bother.
Jellyfin is pretty straight-forward as long as you don’t have a weird hardware decoding setup and as long as you don’t want remote access. If you do want remote access you need to use third party tools to do it securely. If it’s just for your own use then Tailscale makes it really easy. If you want to share with non-technical users it gets messy.
Performance may be an issue. It’s not specifically designed for streaming performance, and being a software VPN, it will depend a great deal on the devices used at each end.
I believe the issue is only with Tailscale Funnels. With Funnels, the data runs through TS’s infrastructure so it’s subject to whatever kind of bandwidth limitation they feel like enacting.
Documentation has been mentioned already, what I’d add to that is planning.
Start with a list of high-level objectives, as in “Need a way to save notes, ideas, documents, between multiple systems, including mobile devices”.
Then break that down to high-level requirements such as “Implement Joplin, and a sync solution”.
Those high-level requirements then spawn system requirements, such as Joplin needs X disk space, user accounts, etc.
Each of those branches out to technical requirements, which are single-line, single-task descriptions (you can skip this, it’s a nice-to-have):
“Create folder Joplin on server A”
“Set folder permissions XYZ on Joplin folder”
Think of it all as a tree, starting from your objectives. If you document it like this first, you won’t go doing something as you build that you won’t remember why you’re doing it, or make decisions on the fly that conflict with other objectives.
If you want to program something, the closest you’re gonna get to programming is Ansible and Bash scripts.
You might want to get self hosting hardware like Synology or the like if you’re not ready to dig.
Otherwise here’s some things you need to know:
Docker
Easy, consistent deployment of services in their own environments. Think a VM but with almost no overhead.
Docker Compose
Run docker containers with consistent configuration in files.
Connect various containers to each other on the same or different networks.
Get multiple containers to start together and talk to each other.
Systemd
Manage any service on Linux. If anything needs to start on boot, restart when crashed, start on timer, you want Systemd.
You can manage your docker compose containers lifecycle via Systemd.
NGINX/Apache/Caddy
A web server for reverse proxy. You’d probably need one at some point, especially if you want HTTPS. Your services get hidden behind it.
ZFS
Reliable redundant storage. You’ll need storage. Use ZFS with 2-disk redundancy.
Supports automatic snapshots for recovering from oopsies. E.g. deleted something or some software shat on your data.
Can use recertified disks from serverpartsdeals.
Can use USB disks or USB box with multiple disks. If you end up going the USB route, ask me for tested hardware.
Backup system
Something to do backup. There are many options.
Ansible
If you want to write code that describes your services and make them happen, you want Ansible. You write code (well YAML) and Ansible installs things, writes config files, sets up Systemd services, restarts things. It can be convenient especially if you have a lot of stuff and you want to be able to see all of your infrastructure in code in one place and be able to version it.
Prometheus
Monitoring your stuff. Is my backup service running? If not send me an email.
Ansible is nice but I’ll repeat (as I said in another thread) it’s kind of advanced and gives a much better return on investment if you manage several hosts, plan to switch hosts regularly, or plan to do regular rebuilds of the environment.
X570 is especially bad, though I’ve had some success with B350, when using the chipset ports. The CPU ports are all bad. Small form factor PCs often only expose CPU USB ports. They work with single disk per port but if you peg a port with a multi-disk box, they crap out regularly.
Post AM5
Have only tested USB4 on X870 and it’s solid.
Client side
WD Elements / MyBook
If you get disconnects under load and you’re not on a shit AMD USB host, the USB-SATA controller is overheating. Open them and ahere a heatsink on it. Drill a hole in the case above it for better ventilation. Disconnections will stop. If you don’t want to deal with any of that buy the item below.
OWC Mercury Elite Pro Quad
Well built, solid controllers, no issues over a year of testing. I have 2, hosting an 8-disk RAIDz2 and 2 hosting a 5-disk RAIDz2.
Terramaster
A friend bought a 6-bay and tore it down for me. It has the same controllers as the OWC in a similar topology. If it’s cheaper it might be OK. I can vouch for the OWC though.
Cables
Get name brand cables, ideally higher spec than what you’d need! They aren’t important for a single USB disk but running a 4-disk box can max out the port bandwidth. If the cable can’t handle it… errors. Casually transmitting 10Gbps via easily detachable cables and ports isn’t trivial.
Much appreciated 🙏
Gnarly stuff with the WD’s huh? Unfortunately I think that’s what I’ll end up having to put up with since I can’t really find the other options for a decent price around here.
Funny enough I was half-considering just using a bunch of WD Elements. You think the MyBooks might fare any better?
I used a mix of Elements and MyBook for years. Upon opening to heatsink, I didn’t see any significant differences between them. They use ASMedia or Jmicron, mostly ASMedia. The overheating issue depends on ambient temp and load. I’ve had one machine in a basement never experience them. Either way the solution is pretty straightforward and cheap. Once heatsinked, I haven’t had a problem.
Why Debian or Ubuntu? (I have my own thoughts, but it would be useful to show even high-level reasons why they’re preferred).
Re: Backup - Backblaze has a great writeup on backup approach today. I’m a fan of cloud being part of the mix (I use a combo of local replication and cloud, to mitigate different risks). Getting people to include backup from the start will help them long-term, so great you included it!
Predictable cadence, stable operation, timely updates, huge community and therefore documentation. You can get up to 5 years from an LTS release of Debian or Ubuntu. With Ubuntu LTS and Ubuntu Pro (free) you could theoretically run a machine without upgrading for 10 years. If you run workloads in containers, it doesn’t matter how old the host OS is. As long as it’s security patches, you can keep on trucking.
Why are you wanting to use python for self hosting? Python is a programming and scripting language.
There are two big things I would focus on. The first thing is networking, the OSI model and http basics. The other thing I would look into is Linux containers. If you can get both of these you are golden. Learn how to use and write docker compose files and then looking into building your own containers with Dockerfiles. You don’t really need to build your own containers but it is good for learning
Programming is generally not needed when self-hosting. At best you might learn Ansible, Puppet, Salt, or Terraform, but that’s for advanced scenarios (e.g. easily shifting the workloads between machines or into the cloud).
Learning the ins-and-outs of containers will get you the biggest return on investment. They’re not strictly necessary but most tools will expect that is the common use-case and the community won’t be as much help. Until you know more about containers I would also recommend Docker over Podman. It has a few more “conveniences” than Podman and orgs like LinuxServer will target Docker as the engine.
You are not logged in. However you can subscribe from another Fediverse account, for example Lemmy or Mastodon. To do this, paste the following into the search field of your instance: !selfhosted@lemmy.world
A place to share alternatives to popular online services that can be self-hosted without giving up privacy or locking you into a service you don’t control.
Rules:
Be civil: we’re here to support and learn from one another. Insults won’t be tolerated. Flame wars are frowned upon.
No spam posting.
Posts have to be centered around self-hosting. There are other communities for discussing hardware or home computing. If it’s not obvious why your post topic revolves around selfhosting, please include details to make it clear.
Don’t duplicate the full text of your blog or github here. Just post the link for folks to click.
Submission headline should match the article title (don’t cherry-pick information from the title to fit your agenda).
No trolling.
Resources:
selfh.st Newsletter and index of selfhosted software and apps
the patience to read lots of documentation.
And maybe patience to power through a lack of documentation.
These 1000% eventually your gonna run into a problem / situation that does not have much documentation. Powering through step by step logically can test the best of us. You can spend 56 hours in a day on one problem. Give up. The next morning figure it out in 10 minutes. It’s a marathon not a sprint.
Honestly, you just need to pick a video on follow along these days. There’s a load of YouTube videos out there that take you step by step.
Lewis rosman recently put out a 14-hour mega video of doing everything, well he might have made some controversial choices, The outcome is quite comprehensive.
No special knowledge needed except the very basic ability to understand and run commands from documentation.
Until you run into some kind of problem :D
It really depends. I actually needed to learn a bit about networking to be able to host multiple things on nginx on the same port. Internally they run on different ports, but they can get routed by the host name
Networking isn’t specific to Linux. It’s just networking. Nginx configs work the same on every OS.
Who said anything about linux
Setting up jellyfin, I used docker on debian, and an old Quadro card. What could possibly go wrong?
Turns out that week the Nvidia drivers got a faulty update pushed to debian stable and caused an error with getting the GPU to work in any container. I could either wait a week or pull the simple fix from testing. So impatiently I pulled it from testing.
Why didn’t you do a rollback?
Persistence and reading comprehension.
There’s no need to learn Python or any programming language to self host stuff, you just need to be able to follow blog posts and run some Docker commands.
I’m a software dev and haven’t touched a single line of code on my NAS. Everything is docker compose and other config files.
Working hands, ability to type characters into keyboard.
Docker really. If something goes bad, trash the container and start again without loosing your actual data.
Mostly Docker.
Portainer and plugging Docker Compose XML into Portainer stacks makes Docker stupid-simple. (personally speaking as a stupid person that does this)
Cloudflare tunnels for stuff people other than you might want to access.
Tailscale if it’s only you.
Reverse proxy & port forwarding for sharing media over Jellyfin without violating the Cloudflare Tunnel ToS.
Dokploy is a pretty easy web gui and is itself a docker container.
Makes it dead simple to manage multiple containers and domains. (Not for power users that need kubernetes level flexibility)
You don’t really need to know a specific language to self-host anything. But things like YAML, JSON, Docker, and some networking basic will go a long way.
If I could do anything different though, it would definitely be to write more documentation. Document the steps taking setting things up, log notes on when you have to fix something, archive webpages and videos that you used along the way. Currently doing that myself now after some time self-hosting.
One under-appreciated aspect of Docker is that it forces you to document all your setup steps in your dockerfile and docker-config files.
Learn how to properly backup your data in case you nuke something you shouldn’t
And regularly check them. I just found out the hard way this last week that my backups haven’t been running for a few weeks …
Yep.
I have friends in the SMB space, one thing they do is a regular backup verification (quarterly). At that frequency, restoring even a few files (especially to a new VM), is very indicative, especially if it’s a large dataset (e.g. Quickbooks).
In Enterprise, we do all sorts of validation, depending on the system. Some is performed as part of Data Center operations, some is by IT (those are separate things), some by Business Unit management and their IT counterparts.
Unfortunately, that wouldn’t have done anything. Because I did that in December and they stopped running like 2 weeks after my verification. I would have caught it on my next scheduled validation, but that doesn’t help me now 😕
I mean, it still helps right? It limits your losses to X weeks instead of X months or, I hate to say it, X years.
Learning Linux is a great start.
Learning any coding language will help you understand a bit more about the programs will work, however there isn’t much need to actually learn a specific language unless you plan to add custom programs or scripts.
The general advice for email is don’t. It’s very risky to host and it’s a big target for spam. Plus there’s challenges getting the big companies to trust your domain.
However hosting things behind a VPN (or locally on your home network) can let you learn a lot about networking and firewalls without exposing yourself to much risk.
I have no direct experience with next cloud but I understand it can be hosted on Linux, you can buy a Synology NAS and run it in that, or use something like TrueNAS.
Personally my setup is on one physical server so I use Proxmox which lets me run 2 different Linux servers and trueNAS on one single computer through virtual machines. I like it because it lets me tinker with different stuff like home assistant and it won’t affect say my adblocker/VPN/reverse proxy. I also use Docker to run multiple services on one virtual machine without compatibility issues. If I started again, I’d probably have gotten bigger drives or invested in SSDs. My NAS is hard drives because of cost but it’s definitely hitting a limit when I need to pull a bunch of files. Super happy with wireguard-easy for VPN. I started with a proprietary version of openVPN on Oracle Linux and that was a mistake.
I feel like objecting to the “General advice about email is don’t” thing but I don’t know if I understand the objections well enough to refute them. I self host email for mspencer.net (meaning all requests including DNS are served from hardware in my living space) and I have literally zero spam and can’t remember the last time I had to intervene on my mail server.
On one hand: My emails are received without issue by major providers (outlook, gmail, etc) and I get nearly zero spam. (Two spam senders were using legitimate email services, I reported them, and got human-seeming replies from administrators saying they would take care of it.) And I get amusing pflogsumm (summarizes postfix logs) emails daily showing like 5 emails delivered, 45 rejected, with all of the things that were tried but didn’t work.
On the other: most of the spam prevention comes from greylist, making all new senders retry after a few minutes (because generally a legit MTA will retry while a spammer will not) and that delays most emails by a few minutes. And it was a bear to set up. I used a like 18 step walkthrough on linuxbabe dot com I think, but added some difficulty by storing some use and alias databases on OpenLDAP / slapd instead of in flat files.
But hey, unlimited mail aliases, and I’m thinking of configuring things so emails bounce if they seem to contain just a notification that terms and conditions are updated somewhere. I don’t know, cause some chaos I guess.
And I have no idea if my situation is persuasive for anyone because I don’t know what the general advice means. And I worry it’ll have the unfortunate side effect of making self hosting type nerds like me start forgetting how to run their own email, causing control of email to become more centralized. And I strongly dislike that.
Is there a good way to not self host email yet maintain good control? Like storing it on a local device. I know that addresses are portable with a domain, but still.
I personally haven’t explored self hosting mail. This thread is a year old but might give you insight from people who have.
I’ve heard about using
mailbox.orgto do what you’re talking about. It seems the general consensus is getting a clean IP mentioned in the thread linked above is the biggest challenge.Edit: mailbox isn’t the what I was thinking of. I’ve definitely heard of services that let you self host half of it and just do the send receive part.
I self host many services without any coding languages under my belt. I use Unraid because I found it user friendly for a newb like me. The most important skills I needed were good data backup habits just in case I messed something up, a willingness to read and learn, and the persistence to try more than once.
To self-host, you do not need to know how to code.
I agree but also say that learning enough to be able to write simple bash scripts is maybe required.
There’s always going to be stuff you want to automate and knowing enough bash to bang out a script that does what you want that you can drop into cron or systemd timers is probably a useful time investment.
As others have said, you don’t need to know how to code, but you do need to be comfortable editing structured documents, so knowing a little programming does help.
Unfortunately, Nextcloud and email are two of the most difficult things to self-host. This is by reputation, I haven’t tried myself. Email is supposed to be particularly difficult and the usual advice is to not bother.
Jellyfin is pretty straight-forward as long as you don’t have a weird hardware decoding setup and as long as you don’t want remote access. If you do want remote access you need to use third party tools to do it securely. If it’s just for your own use then Tailscale makes it really easy. If you want to share with non-technical users it gets messy.
I remember reading that tailscale can’t be used for sharing media, was that wrong?
Performance may be an issue. It’s not specifically designed for streaming performance, and being a software VPN, it will depend a great deal on the devices used at each end.
I believe the issue is only with Tailscale Funnels. With Funnels, the data runs through TS’s infrastructure so it’s subject to whatever kind of bandwidth limitation they feel like enacting.
Documentation has been mentioned already, what I’d add to that is planning.
Start with a list of high-level objectives, as in “Need a way to save notes, ideas, documents, between multiple systems, including mobile devices”.
Then break that down to high-level requirements such as “Implement Joplin, and a sync solution”.
Those high-level requirements then spawn system requirements, such as Joplin needs X disk space, user accounts, etc.
Each of those branches out to technical requirements, which are single-line, single-task descriptions (you can skip this, it’s a nice-to-have):
“Create folder Joplin on server A”
“Set folder permissions XYZ on Joplin folder”
Think of it all as a tree, starting from your objectives. If you document it like this first, you won’t go doing something as you build that you won’t remember why you’re doing it, or make decisions on the fly that conflict with other objectives.
This is really smart actually
If you want to program something, the closest you’re gonna get to programming is Ansible and Bash scripts.
You might want to get self hosting hardware like Synology or the like if you’re not ready to dig.
Otherwise here’s some things you need to know:
Oh and use Debian or Ubuntu LTS.
Ansible is nice but I’ll repeat (as I said in another thread) it’s kind of advanced and gives a much better return on investment if you manage several hosts, plan to switch hosts regularly, or plan to do regular rebuilds of the environment.
Send these my way chief
As briefly as possible:
Much appreciated 🙏
Gnarly stuff with the WD’s huh? Unfortunately I think that’s what I’ll end up having to put up with since I can’t really find the other options for a decent price around here.
Funny enough I was half-considering just using a bunch of WD Elements. You think the MyBooks might fare any better?
I used a mix of Elements and MyBook for years. Upon opening to heatsink, I didn’t see any significant differences between them. They use ASMedia or Jmicron, mostly ASMedia. The overheating issue depends on ambient temp and load. I’ve had one machine in a basement never experience them. Either way the solution is pretty straightforward and cheap. Once heatsinked, I haven’t had a problem.
The cables they come with are good.
Great summary!
Why Debian or Ubuntu? (I have my own thoughts, but it would be useful to show even high-level reasons why they’re preferred).
Re: Backup - Backblaze has a great writeup on backup approach today. I’m a fan of cloud being part of the mix (I use a combo of local replication and cloud, to mitigate different risks). Getting people to include backup from the start will help them long-term, so great you included it!
Predictable cadence, stable operation, timely updates, huge community and therefore documentation. You can get up to 5 years from an LTS release of Debian or Ubuntu. With Ubuntu LTS and Ubuntu Pro (free) you could theoretically run a machine without upgrading for 10 years. If you run workloads in containers, it doesn’t matter how old the host OS is. As long as it’s security patches, you can keep on trucking.
Damn, 5 years from LTS? That’s impressive
Why are you wanting to use python for self hosting? Python is a programming and scripting language.
There are two big things I would focus on. The first thing is networking, the OSI model and http basics. The other thing I would look into is Linux containers. If you can get both of these you are golden. Learn how to use and write docker compose files and then looking into building your own containers with Dockerfiles. You don’t really need to build your own containers but it is good for learning
Programming is generally not needed when self-hosting. At best you might learn Ansible, Puppet, Salt, or Terraform, but that’s for advanced scenarios (e.g. easily shifting the workloads between machines or into the cloud).
Learning the ins-and-outs of containers will get you the biggest return on investment. They’re not strictly necessary but most tools will expect that is the common use-case and the community won’t be as much help. Until you know more about containers I would also recommend Docker over Podman. It has a few more “conveniences” than Podman and orgs like LinuxServer will target Docker as the engine.