We all have bare-metal servres, VPS:es, containers and other things running. Some of them may be exposed openly to the internet, which is populated by autonomous malicious actors, and some may reside on a closed-off network since they contain sensitive data.
And there is a lot of solutions to monitor your servers, since none of us want our resources to be part of a botnet, or mine bitcoins for APTs, or simply have confidential data fall into the wrong hands.
Some of the tools I’ve looked at for this task are check_mk, netmonitor, monit: all of there monitor metrics such as CPU, RAM and network activity. Other tools such as Snort or Falco are designed to particularly detect suspicious activity. And there also are solutions that are hobbled together, like fail2ban actions together with pushover to get notified of intrusion attempts.
So my question to you is - how do you monitor your servers and with what tools? I need some inspiration to know what tooling to settle on to be able that detect unwanted external activity on my resources.
Sometimes I just sit and stare at my apache access logs because I’m bored
GoAccess is pretty nice for a broad overview of Apache logs, also.
For other services I generally just look at them every now and then and if something looks off I investigate. I found a cryptominer on my network once because it was spamming DNS and that shows up in DNS logs.
I used to use some logging script made in Go where you could filter your logs and they would update in real time. Was great for catching stuck processes, leave it running on a different desktop, mousewheel over to it (i miss openbox so so much) and check my logs. I just have nothing facing outwards now so i ignore everything.
I used zabbix at some point, but I never looked at the data so I stopped. Zabbix shows all kind of stuff.
I have cockpit on my bare-metal that has some stats, and netdata on my firewall, I do not track any of my VM’s (except vnstat that runs on everything device).
Reduce your threat profile.
Run sslh, 443 handles both SSL and ssh.
Adjust your host based firewall to just 443
Attack yourself on that port, identify the logs
Add the new profiles to fail2ban
Enable fail2ban email
If you don’t like email, use a service that translates email to notification.
Ntfy.sh is free notifications
Or…
Use something like tailscale and don’t offer a remote login to the general Internet.
I submitted your post to got here’s what it thought
Icinga2 works reasonably well for us. It is easy to write new checks as small shell scripts (or any other binary that can print and set and exit status code).
It’s open source, it’s easy to setup, its agents are available for nearly anything including OpenWrt, it can serve the simplest use case of “is it down” as well as much more complicated ones that stem from its ability to collect data over time.
Personally I’m monitoring:
Is it up?
Is the storage array healthy?
Are the services I care about running?
I used to run it ephemerallly - wiping data on restart. Recently started persisting its data so I can see data over the longer run.
Netdata (agent only/not the cloud-based features), and a bunch of scanners running from cron/systemd timers, rsyslog for logs (and graylog for larger setups)
Since your question is also related to securing your setup, inspect and harden the configuration of all running services and the OS itself. Here is my common ansible role for basic stuff. Find (prefereably official) hardening guides for your distribution and implement hardening guidelines such as DISA STIG, CIS benchmarks, ANSSI guides, etc.
I used to pass all the data through to Home Assistant and show it on some dashboards, but I decided to move over to Zabbix.
Works well but is quite full-featured, maybe moreso than necessary for a self hoster. Made a mediatype integration for my announciator system so I hear issues happening with the servers, as well as updates on things, so I don’t really need to check manually. Also a custom SMART template that populates the disk’s physical location/bay (as the built in one only reports SMART data).
It’s notified me of a few hardware issues that would have gone unnoticed on my previous system, and helped with diagnosing others. A lot of the sensors may seem useless, but trust me, once they flag up you should 100% check on your hardware. Hard drives losing power during high activity because of loose connections, and a CPU fan failure to name two.
It has a really high learning curve though so not sure how much I can recommend it over something like Grafana+Prometheus - something I haven’t used but the combo looks equally as comprehensive as long as you check your dashboard regularly.
I use node exporter for host metrics (Proxmox/VMs/SFFs/RaspPis/Router) and a number of other *exporters:
exportarr
plex-exporter
unifi-exporter
bitcoin node exporter
I use the OpenTelemetry collector to collect some of the above metrics, rather than Prometheus itself, as well as docker logs and other log files before shipping them to Prometheus/Loki.
Oh, I also scrape metrics from my Traefik containers using OTEL as well.
What does having OpenTelemetry improve? I have a setup similar to yours but data goes from Prometheus to Grafana and I never thought I would need anything else.
Not a whole lot to be honest. But I work with OpenTelemetry everyday for my day job, so it was a little exercise for me.
Though, OTEL does have some advantages in that It is a vendor agnostic collection tool. allowing you to use multiple different collection methods and switch out your backend easily if you wish.
You are not logged in. However you can subscribe from another Fediverse account, for example Lemmy or Mastodon. To do this, paste the following into the search field of your instance: !selfhosted@lemmy.world
A place to share alternatives to popular online services that can be self-hosted without giving up privacy or locking you into a service you don’t control.
Rules:
Be civil: we’re here to support and learn from one another. Insults won’t be tolerated. Flame wars are frowned upon.
No spam posting.
Posts have to be centered around self-hosting. There are other communities for discussing hardware or home computing. If it’s not obvious why your post topic revolves around selfhosting, please include details to make it clear.
Don’t duplicate the full text of your blog or github here. Just post the link for folks to click.
Submission headline should match the article title (don’t cherry-pick information from the title to fit your agenda).
I’m pretty old school, but as I only have 1 server, I just use
ssh,df,duandtop.Not even htop? That is old school.
Not even btop? That’s middle school.
Not even bottom? That’s elementary school.
Okay priest.
Emoji reactions are missing 😂😂😂
Sometimes I just sit and stare at my apache access logs because I’m bored
GoAccess is pretty nice for a broad overview of Apache logs, also.
For other services I generally just look at them every now and then and if something looks off I investigate. I found a cryptominer on my network once because it was spamming DNS and that shows up in DNS logs.
I used to use some logging script made in Go where you could filter your logs and they would update in real time. Was great for catching stuck processes, leave it running on a different desktop, mousewheel over to it (i miss openbox so so much) and check my logs. I just have nothing facing outwards now so i ignore everything.
I used zabbix at some point, but I never looked at the data so I stopped. Zabbix shows all kind of stuff.
I have cockpit on my bare-metal that has some stats, and netdata on my firewall, I do not track any of my VM’s (except vnstat that runs on everything device).
Reduce your threat profile. Run sslh, 443 handles both SSL and ssh. Adjust your host based firewall to just 443 Attack yourself on that port, identify the logs Add the new profiles to fail2ban Enable fail2ban email If you don’t like email, use a service that translates email to notification. Ntfy.sh is free notifications Or… Use something like tailscale and don’t offer a remote login to the general Internet.
I submitted your post to got here’s what it thought
https://shareg.pt/Tz0El4k
Icinga2 works reasonably well for us. It is easy to write new checks as small shell scripts (or any other binary that can print and set and exit status code).
It cannot notify you, you have to check it manually, but: I use DaRemote on my phone to periodically check my bare metal.
Prometheus.
It’s open source, it’s easy to setup, its agents are available for nearly anything including OpenWrt, it can serve the simplest use case of “is it down” as well as much more complicated ones that stem from its ability to collect data over time.
Personally I’m monitoring:
I used to run it ephemerallly - wiping data on restart. Recently started persisting its data so I can see data over the longer run.
What do you use to see the data? Prometheus itself is easy to set up, but getting to the data seemed complicated.
You can use grafana to visualise the data.
Grafana isn’t too hard to use.
The Prometheus built-in web UI. I find it pretty simple.
Uptime-Kuma
Netdata (agent only/not the cloud-based features), and a bunch of scanners running from cron/systemd timers, rsyslog for logs (and graylog for larger setups)
My base ansible role for monitoring.
Since your question is also related to securing your setup, inspect and harden the configuration of all running services and the OS itself. Here is my common ansible role for basic stuff. Find (prefereably official) hardening guides for your distribution and implement hardening guidelines such as DISA STIG, CIS benchmarks, ANSSI guides, etc.
I used to pass all the data through to Home Assistant and show it on some dashboards, but I decided to move over to Zabbix.
Works well but is quite full-featured, maybe moreso than necessary for a self hoster. Made a mediatype integration for my announciator system so I hear issues happening with the servers, as well as updates on things, so I don’t really need to check manually. Also a custom SMART template that populates the disk’s physical location/bay (as the built in one only reports SMART data).
It’s notified me of a few hardware issues that would have gone unnoticed on my previous system, and helped with diagnosing others. A lot of the sensors may seem useless, but trust me, once they flag up you should 100% check on your hardware. Hard drives losing power during high activity because of loose connections, and a CPU fan failure to name two.
It has a really high learning curve though so not sure how much I can recommend it over something like Grafana+Prometheus - something I haven’t used but the combo looks equally as comprehensive as long as you check your dashboard regularly.
Just wish there were more android apps
deleted by creator
PRTG has a community edition Elastiflow for netflow has free/community edition Grafana and influxdb open source
UptimeKuma is great, I use it for the simple “are my services up?” and is what I pay most attention to.
I still use zabbix for finer grained monitors though like checking raid status, smartctl, disk space, temperatures, etc.
I’ve been trying out librenms with more custom snmp checks too and am considering going that route instead of zabbix in the future
Prometheus for metrics
Loki for logs
Grafana for dashboards.
I use node exporter for host metrics (Proxmox/VMs/SFFs/RaspPis/Router) and a number of other *exporters:
I use the OpenTelemetry collector to collect some of the above metrics, rather than Prometheus itself, as well as docker logs and other log files before shipping them to Prometheus/Loki.
Oh, I also scrape metrics from my Traefik containers using OTEL as well.
What does having OpenTelemetry improve? I have a setup similar to yours but data goes from Prometheus to Grafana and I never thought I would need anything else.
Not a whole lot to be honest. But I work with OpenTelemetry everyday for my day job, so it was a little exercise for me.
Though, OTEL does have some advantages in that It is a vendor agnostic collection tool. allowing you to use multiple different collection methods and switch out your backend easily if you wish.
Have you tried the proxmox exporter? I have tried it briefly for a grafana lab and it seemed pretty good.
https://github.com/prometheus-pve/prometheus-pve-exporter
I haven’t, but it looks like I’ve got another exporter to install and dashboard to create 😁
If you want to run the exporter without docker (like I did) and you get problems with installing the exporter try using this guide: https://github.com/prometheus-pve/prometheus-pve-exporter/wiki/PVE-Exporter-on-Proxmox-VE-Node-in-a-venv
Uptime Kuma and ntfy.
I don’t do much in the way of monitoring. I guess I should do that.