How do you monitor your server containers, disks, load…?
Do you use an easy-to-use web interface? Do you do everything via SSH? Or maybe you’ve got a more complicated setup?
I want to change my setup and I’m looking for new ideas, I’ve been using Cockpit for some years and some of the plugins are really outdated (ZFS for example) and others are completely broken (docker-compose).
docker-compose doesn’t scale well and if you run it natively it is a little less secure.
Virtualization adds 1-2% of overhead at most and gives you way more control of how the hardware is used.
If you setup is small docker-compose might be easier to manage but as soon as you get more hardware it becomes the limiting factor. I still use docker-compose but now I run it in a VM
I switched from docker compose to pure Ansible for deploying my containers. Makes managing config and starting containers across multiple hosts super easy. I considered virtualizing too but decided it didn’t offer me enough advantages. If I ever have an issue with the host OS I just reinstall using a preseed file and then rerun my playbooks and it’s ready to go.
Grafana set up to run on the server locally, then I connect to it via SSH forwarding. Then I can view all kinds of metrics in my browser in a neat interface.
This is the first time I’ve heard of Victoria Metrics. It looks like it has a similar use case as Prometheus, is that correct? If so, what made you or your team choose one over the other?
Now I am intrigued to develop one that is called YOLO.
But just in case: no, I don’t monitor my server. If I notice something not working, I ssh into the machine and check what’s up. I don’t want to deal with another zoo of services for the monitoring part.
I’m a huge fan of Netdata, very configurable and monitors just about anything you could want. Great interface and alerts too - https://www.netdata.cloud/
I love how easy to use NetData is, but when running it on my home servers it destroys their performance lol. Every once in awhile I check in to see if it runs better.
That’s strange, I’ve run it fine on some very underpowered hardware. Are you adding a specific monitoring integration with it, or just out of the box settings?
Just out of the box. I am usually running it as a container on UnRAID on an x86 machine. It seems primarily to just be a big memory hog when I’ve tried to use it.
Weird! For reference one VM I run on only has 1 GB of memory, and Netdata uses 100-200 MB. Could be something going on with UnRAID though. Definitely some sort of bug I’d think, since normally resource usage should be very low across the board.
As others stated, you can run and access the interface locally (or setup your own reverse proxy) for free. Their Cloud dashboard is also free for up to 5 nodes. They recently added a flat-rate “Homelab” plan as well, if you want to remove the limit. It’s all quite usable for $0 otherwise though!
Adding my vote for Zabbix. It was a bit of a bear to set up and I had to write custom scripts to install the agents with TLS settings that were secure enough for me, but once it’s all set up it’s amazingly easy and intuitive to use and incredibly customizable.
Node exporter on hosts, OpenTelemetry collector to scrape metrics and collect logs, shipping them to Prometheus and Loki, visualising with Grafana.
Day job is for an observability platform where we heavily encourage the use of (and also contribute) to the OpenTelemetry collector project, hence my use of it.
Try VictoriaMetrics. Basically the same feature set as Prometheus, but so much more resource friendly for homelab scale. I store some metrics for 12 months now, because it’s easy.
Similar setup here with additional exporters like cadvisor for container metrics and other components.
OpenTelemetry is awesome, but still a very fast moving project. Expect therefore more frequent updates and changes compared to more older and established projects.
You are not logged in. However you can subscribe from another Fediverse account, for example Lemmy or Mastodon. To do this, paste the following into the search field of your instance: !selfhosted@lemmy.world
A place to share alternatives to popular online services that can be self-hosted without giving up privacy or locking you into a service you don’t control.
Rules:
Be civil: we’re here to support and learn from one another. Insults won’t be tolerated. Flame wars are frowned upon.
No spam posting.
Posts have to be centered around self-hosting. There are other communities for discussing hardware or home computing. If it’s not obvious why your post topic revolves around selfhosting, please include details to make it clear.
Don’t duplicate the full text of your blog or github here. Just post the link for folks to click.
Submission headline should match the article title (don’t cherry-pick information from the title to fit your agenda).
No trolling.
Resources:
selfh.st Newsletter and index of selfhosted software and apps
At home, libreNMS. Just SNMP everything.
For work, whatever the tool of the day is from management.
Um, Proxmox?
I am running everything in docker compose so I’ve never found a use to it that justifies the waste of power
docker-compose doesn’t scale well and if you run it natively it is a little less secure.
Virtualization adds 1-2% of overhead at most and gives you way more control of how the hardware is used.
If you setup is small docker-compose might be easier to manage but as soon as you get more hardware it becomes the limiting factor. I still use docker-compose but now I run it in a VM
I switched from docker compose to pure Ansible for deploying my containers. Makes managing config and starting containers across multiple hosts super easy. I considered virtualizing too but decided it didn’t offer me enough advantages. If I ever have an issue with the host OS I just reinstall using a preseed file and then rerun my playbooks and it’s ready to go.
“Huh weird, I tried to use <insert service here> and it’s not working. Welp, guess I better fix it…”
i just have top running through ssh on an xterm window.
At home, nagios, at work colleagues. (I finally escaped the admin rat race)
Grafana set up to run on the server locally, then I connect to it via SSH forwarding. Then I can view all kinds of metrics in my browser in a neat interface.
I liked Grafana a lot, but I can’t monitor things like zfs pools with it right?
I don’t know as I don’t use zfs pools, but a simple search led me to this https://grafana.com/grafana/dashboards/15362-zfs-pool-metrics/
Nevermind then! Will take a look at it ^^
Zabbix for agent / snmp based statistics.
Uptime Kuma for up/down states with a webhook notification into Discord so I get instant alerts on my phone when one goes down.
My own server? YOLO
At work? Grafana, KOBS, Victoria Metrics, Jaeger, OpsGenie, …
This is the first time I’ve heard of Victoria Metrics. It looks like it has a similar use case as Prometheus, is that correct? If so, what made you or your team choose one over the other?
IIRC it had better performance than Prometheus. We also ditched Elasticsearch in favor of ClickHouse to keep up with log ingestion.
Thanks for the info! Looks pretty cool I’ll have to check it out
I can second that. We had some really good experiences with ClickHouse and its performance. If it fits the bill, it’s a very nice piece of software.
I can’t figure out whether there’s a monitoring tool called YOLO or you don’t monitor anything.
Now I am intrigued to develop one that is called YOLO.
But just in case: no, I don’t monitor my server. If I notice something not working, I ssh into the machine and check what’s up. I don’t want to deal with another zoo of services for the monitoring part.
You are me
Yes.
Btop and logwatch with logrotate. I use healthchecks to check if the server is unreachable and it notifies me.
I’m a huge fan of Netdata, very configurable and monitors just about anything you could want. Great interface and alerts too - https://www.netdata.cloud/
I love how easy to use NetData is, but when running it on my home servers it destroys their performance lol. Every once in awhile I check in to see if it runs better.
That’s strange, I’ve run it fine on some very underpowered hardware. Are you adding a specific monitoring integration with it, or just out of the box settings?
Just out of the box. I am usually running it as a container on UnRAID on an x86 machine. It seems primarily to just be a big memory hog when I’ve tried to use it.
Weird! For reference one VM I run on only has 1 GB of memory, and Netdata uses 100-200 MB. Could be something going on with UnRAID though. Definitely some sort of bug I’d think, since normally resource usage should be very low across the board.
Same been running netdata for years. They’re monetizing now where it used to just be free. Good for them, it’s a great product. And it’s foss
I was looking for something free that I could host on my machine but thanks, I didn’t know about it
As others stated, you can run and access the interface locally (or setup your own reverse proxy) for free. Their Cloud dashboard is also free for up to 5 nodes. They recently added a flat-rate “Homelab” plan as well, if you want to remove the limit. It’s all quite usable for $0 otherwise though!
Netdata is free and can be run standalone. Just install it and do not configure the cloud integration. You can see your dashboard on localhost:19999
Oh that’s neat, will take a look! Can you run it on docker?
https://learn.netdata.cloud/docs/installing/docker
Netdata 100%
It feeds my itch for more data than I know what to do with and it’s presented in one of the cleanest ways I’ve ever seen for so much info.
Zabbix
Adding my vote for Zabbix. It was a bit of a bear to set up and I had to write custom scripts to install the agents with TLS settings that were secure enough for me, but once it’s all set up it’s amazingly easy and intuitive to use and incredibly customizable.
Second Zabbix. Been using it for years and it just works.
My clients when they text me the server is down.
This has the same energy as my spouse yelling at me because jellyfin went down
Or my partners greeting me in the morning “Home assistant went down again, so the lights are all manual”
Thankfully that one is mostly solved.
So damn accurate ahhaha
Node exporter on hosts, OpenTelemetry collector to scrape metrics and collect logs, shipping them to Prometheus and Loki, visualising with Grafana.
Day job is for an observability platform where we heavily encourage the use of (and also contribute) to the OpenTelemetry collector project, hence my use of it.
Try VictoriaMetrics. Basically the same feature set as Prometheus, but so much more resource friendly for homelab scale. I store some metrics for 12 months now, because it’s easy.
Do you have a name for the opentelemetry collector? I’m interested.
Use the Contrib version of the collector, it has many more receivers, processors and exporters
GitHub - OTEL collector contrib
Similar setup here with additional exporters like cadvisor for container metrics and other components.
OpenTelemetry is awesome, but still a very fast moving project. Expect therefore more frequent updates and changes compared to more older and established projects.
Monitorix or Netdata.
Prometheus and Altertmanager