I am running a Kubernetes cluster for this domain, and I’m looking at more services to run (right now I have Mastodon and Lemmy).
I was considering WriteFreely and PixelFed, but they don’t seem to have an easy solution for running on Kubernetes (WriteFreely doesn’t even have a production-ready docker image).
Is anyone else running federated services in their lab? Do you run any of them on Kubernetes?
I run plenty of services in my lab on k3s. Most of them I had to manually build charts for and add them to my cluster. I doubt anyone has built charts for Lemmy, maybe mastodon. Anything that’s dockerized like Lemmy can be put into kubernetes, but it’s going to take some doing. Good luck out there, and of course if you get it working then I think the maintainers would be happy to get a helm chart merged into their repo.
Yeah, that’s the pain point - building and maintaining the charts.
Also, I know the charts likely wouldn’t have to be super complex, but I’m used to working with Bitnami’s charts that are massively complex - I just don’t have the time to go that in-depth.
I start simple with mine. Start with the deployment, just getting the pod up and running, completely stateless. Get that stabilized, once that’s happy then start moving onto the service, connecting to the UI. Then I finish anything else I can before adding in state - environment variables (first just in the deployment, then once that’s done worry about a config map and secrets). (Adding volumes in just complicates it because then you have to reset those volumes if you want to go back to a clean state).
As a list:
Deployment (stateless) - get to not crash
Service - connect to it
Environment variables, extracting out what can be
Secrets, adding those to the chart
Volumes, adding in state
Move to a nice clean setup with a clean values.yaml.
Finally, once your happy, then make it conform to other standards. Getting it stood up in k8s is the hard part, then customizing can come after.
This all makes sense to me since we deal with it at work. I would maybe add a service vs route point to differentiate things like UI that need external exposure. The main difference is we use kustomize instead of helm. Out of curiosity if you had any experience with both and why did you settle in helm?
That’s actually super helpful! I haven’t done much custom Helm chart-ing, and was kinda lost where to start. That really helps break the process down, and the tip about skipping state to start is very wise.
I’ve done… an annoying amount of them now. I hope my trials help you.
Be ready, it’s a very annoying and slow process, watching logs, figuring out why things are failing, debugging, githubs, everything. I just did one last week that was saying that it couldn’t write to /etc/passXXXXX, and it took 2 hours to track down that there was an optional command I could pass in that would change the running user of the container I was running (separate from the kube user). It’s a slog, but when you get it running it’s a rush of endorphins.
Biggest thing - kubernetes is a read-only file system compared to docker. So, for good devs they minimize writes to the filesystem unless they have to, and keep them localized. For bad devs, they write everywhere - and that gets sticky fast. So, if you’re getting write errors, know that there’s probably another volume you need to attach. The kicker is knowing which ones can be an emptyDir scratch directory and which ones actually need to persist. If you have a docker-compose file it’s a great place to start, just set all the volumes to emptyDir to start off with.
I think both of the ones I mentioned have docker-compose files, which I think I can convert with kompose convert? I guess from there I would follow your steps and then start parameterizing it once it’s running properly.
Thanks! I think I’ll start trying out PixelFed tomorrow.
I don’t like helm, so I use nix to maintain my fediverse deployments in kubernetes. Typically that’d just autoupdate itself to new releases, but for lemmy specifically I upgrade by hand nowadays since one release some time ago broke my deployment and its schema change was incompatible with the automated rollback.
You are not logged in. However you can subscribe from another Fediverse account, for example Lemmy or Mastodon. To do this, paste the following into the search field of your instance: !selfhosted@lemmy.world
A place to share alternatives to popular online services that can be self-hosted without giving up privacy or locking you into a service you don’t control.
Rules:
Be civil: we’re here to support and learn from one another. Insults won’t be tolerated. Flame wars are frowned upon.
No spam posting.
Posts have to be centered around self-hosting. There are other communities for discussing hardware or home computing. If it’s not obvious why your post topic revolves around selfhosting, please include details to make it clear.
Don’t duplicate the full text of your blog or github here. Just post the link for folks to click.
Submission headline should match the article title (don’t cherry-pick information from the title to fit your agenda).
No trolling.
Resources:
selfh.st Newsletter and index of selfhosted software and apps
I run plenty of services in my lab on k3s. Most of them I had to manually build charts for and add them to my cluster. I doubt anyone has built charts for Lemmy, maybe mastodon. Anything that’s dockerized like Lemmy can be put into kubernetes, but it’s going to take some doing. Good luck out there, and of course if you get it working then I think the maintainers would be happy to get a helm chart merged into their repo.
Yeah, that’s the pain point - building and maintaining the charts.
Also, I know the charts likely wouldn’t have to be super complex, but I’m used to working with Bitnami’s charts that are massively complex - I just don’t have the time to go that in-depth.
I start simple with mine. Start with the deployment, just getting the pod up and running, completely stateless. Get that stabilized, once that’s happy then start moving onto the service, connecting to the UI. Then I finish anything else I can before adding in state - environment variables (first just in the deployment, then once that’s done worry about a config map and secrets). (Adding volumes in just complicates it because then you have to reset those volumes if you want to go back to a clean state).
As a list:
Finally, once your happy, then make it conform to other standards. Getting it stood up in k8s is the hard part, then customizing can come after.
This all makes sense to me since we deal with it at work. I would maybe add a service vs route point to differentiate things like UI that need external exposure. The main difference is we use kustomize instead of helm. Out of curiosity if you had any experience with both and why did you settle in helm?
That’s actually super helpful! I haven’t done much custom Helm chart-ing, and was kinda lost where to start. That really helps break the process down, and the tip about skipping state to start is very wise.
I’ve done… an annoying amount of them now. I hope my trials help you.
Be ready, it’s a very annoying and slow process, watching logs, figuring out why things are failing, debugging, githubs, everything. I just did one last week that was saying that it couldn’t write to /etc/passXXXXX, and it took 2 hours to track down that there was an optional command I could pass in that would change the running user of the container I was running (separate from the kube user). It’s a slog, but when you get it running it’s a rush of endorphins.
Biggest thing - kubernetes is a read-only file system compared to docker. So, for good devs they minimize writes to the filesystem unless they have to, and keep them localized. For bad devs, they write everywhere - and that gets sticky fast. So, if you’re getting write errors, know that there’s probably another volume you need to attach. The kicker is knowing which ones can be an
emptyDirscratch directory and which ones actually need to persist. If you have adocker-composefile it’s a great place to start, just set all the volumes toemptyDirto start off with.Good luck!
I think both of the ones I mentioned have
docker-composefiles, which I think I can convert withkompose convert? I guess from there I would follow your steps and then start parameterizing it once it’s running properly.Thanks! I think I’ll start trying out PixelFed tomorrow.
I don’t like helm, so I use nix to maintain my fediverse deployments in kubernetes. Typically that’d just autoupdate itself to new releases, but for lemmy specifically I upgrade by hand nowadays since one release some time ago broke my deployment and its schema change was incompatible with the automated rollback.
My setup is a combination of https://github.com/farcaller/nixdockertag (auto-updated docker imagesfor things where I fully own the deployments) and https://github.com/farcaller/nixhelm (for helm charts that I either consume verbatim PR have local patches on). Both just auto update nightly thanks to github.