Hi, recently (ironically, right after sharing some of my posts here on Lemmy) I had a higher (than usual, not high in general) number of “attacks” to my website (I am talking about dumb bots, vulnerability scanners and similar stuff). While all of these are not really critical for my site (which is static and minimal), I decided to take some time and implement some generic measures using (mostly) Crowdsec (fail2ban alternative?) and I made a post about that to help someone who might be in a similar situation.
The whole thing is basic, in the sense that is just a way to reduce noise and filter out the simplest attacks, which is what I argue most of people hosting websites should be mostly concerned with.
That is basically the essence of this post too! Except crowdsec is used to do what fail2ban does + some light form of WAF (without spinning another machine - which is not strictly needed for a WAF, you can use owasp modsecurity-ready proxies).
I would just like to add some additional information that favors changing your SSH port to something other than the default. When crawlers are going around the internet looking for vulnerable SSH servers, they’re more than likely going to have an IP range and specifically look for port 22.
Now can they go through and scan your IP and all of its ports to look for the SSH service? Yes. But you will statistically have less interactions with bad actors this way since they might specifically be looking for port 22.
As a side note, don’t be cute and pick port 221 or 2222 or 22022 or whatever that’s got “22” in it.
I know that sounds silly but the slightly less stupid bots are written by people who understand people do things like that and account for them, and thus port 2222 isn’t actually better than 22, or whatever.
imho - never expose that shit anyways, and VPN into your local network first. Only thing I ever expose to the internet is 80/443.
At the very least, if you’re going to expose an SSH session to the internet, set up some sort of port-knocking. It’s security by obscurity, sure - but it will keep all but the most ardent intruders out.
Agreed, but sometimes you have to expose things you’d rather not; I just figured I’d mention that almost everyone’s immediate urge to just go ‘port 2222! that’ll do it!’ isn’t exactly better than doing nothing anymore.
There is still not a reason anyone has been able to give for 99% of self-hosters to expose SSH.
If you need to access your machine via ssh while on the go. Wireguard to your local network, use SSH. Done. Unless you are running an always-up public facing site, the amount of times you have to access your machine that can’t wait until after work is very low anyway.
Bots will scan all ports. That is just how it works. Less than 22, but you will still get spammed. Why force your computer to go through the fail2ban loop and take up resources when it is simply not needed at all and you can block it on another machine?
This blog is specifically for websites that are public facing. Sure, you can wireguard into your local network, but you can also SSH into your local network. Either way you have to poke a hole.
Comfort is the main reason, I suppose. If I mess up Wireguard config, even to debug the tunnel I need to go to the KVM console. It also means that if I go to a different place and I have to SSH into the box I can’t plug my Yubikey and SSH from there. It’s a rare occurrence, but still…
Ultimately I do understand both point of view. The thing is, SSH bots pose no threats after the bare minimum hardening for SSH has been done. The resource consumption is negligible, so it has no real impact.
To me the tradeoff is slight inconvenience vs slightly bigger attack surface (in case of CVEs). Ultimately everyone can decide which compromise is acceptable for them, but I would say that the choice is not really a big one.
Thanks! I did mention this briefly, although I belong to the school that “since I am anyway banning IPs that fail authentication a few times, it’s not worth changing the port”. I think that it’s a valid thing especially if you ingest logs somewhere, but if you do don’t choose 2222! I have added a link to shodan in the post, which shows that almost everybody who changes port, changes to 2222!
Yeah, I just left my SSH port as 22 since I only use key-based authentication so there’s really no security risk. Plus, it’s funny going through the logs and looking at all the login attempts.
Yep I agree. Especially looking at all the usernames that are tried. I do the same and the only risk come from SSH vulnerabilities. Since nobody would burn a 0-day for SSH (priceless) on my server, unattended upgrades solve this problem too for the most part.
Yeah I know (I mentioned it myself in the post), but realistically there is no much you can do besides upgrading. Unattended upgrades kick in once a day and you will install the security patches ASAP. There are also virtual patches (crowdsec has a virtual patch for that CVE), but they might not be very effective.
I argue that VPN software is a smaller attack surface, but the problem still exists (CVEs) for everything you expose.
Unrelated to your actual post (plan to read later), but is your RSS busted? The rss link on the webpage gives a 404 and my RSS reader is erroring on it as well…
Oh, it looks like! Something went wrong with Zola build and I must have not noticed. Thanks a lot for pinging me about that, I will fix it today!
EDIT:
Fixed! That’s what you get when you forget to bump the Docker image version after you upgrade zola version locally with a breaking change in the config! :)
Thanks for letting me know, it would have taken me a long time to see it was broken!
Can you use CrowdSec to track logs from a k8s pod? Say I have my website and some other services hosted on a k3s cluster, do I need to spin up a new pod for CrowdSec or should it be installed on the host?
First, you have the problem of sourcing the data. In essence, Crowdsec won’t be able to go and fetch those logs for you dynamically, but can go and take those logs from a file (you can do a dirty solution like a sidecar deployment) or from a stream. You can deploy crowdsec in multiple modes, and you can have many instances that talk to each other. You can also simply have some process tailing the pod logs and sending them to a file crowdsec has access to or serving them as a stream (see https://doc.crowdsec.net/docs/data_sources/intro).
The above means that it doesn’t really matter whether you run Crowdsec inside your cluster (it does have a Helm chart) or on the host. Ultimately all it matters is that crowdsec has access to your pods logs (for example, the logs of your ingress controller).
The next piece is the remediation component. What do you want crowdsec to do, once it is able to detect bad IPs? If you want to just add IPs to the firewall, then it might make more sense running it on the host(s) you use in ingress, if you want to add the IPs to network policies you can do it, but you need to develop your own remediation components. I am planning to write a remediation component that will add the IPs to Hetzner firewall, some other systems are already supported, but this would be a way to basically block the IPs outside your cluster. For nginx ingress controller there is already a pre-made remediation component
.
In practice I personally would choose a simple setup where the interesting logs are just forwarded (in Syslog format for example) to a single crowdsec instance. If you have ingress from a single node, I’d go for running it on the host and banning via firewall, if you have multiple ingress nodes, then I would run it inside the cluster and ban via a loadBalancer/cloud firewall/whatever you have in front.
In essence, I would spend some time to think about your preferences, and it might take a little bit to make the setup clean, but I think you have plenty of flexibility to do what you prefer.
Let me know if you want to bounce some more ideas!
Fair question. What I meant is that suggesting that would have made the whole post 10 lines long and not worth doing.
So I avoided such suggestions that completely change the threat model.
It’s not useless to avoid a good security posture (although you might have concerns of a monopoly gatekeeping the internet, TLS traffic inspection privacy concerns etc.), on the contrary makes everything I have written about here redundant (+ provide more, like DDoS protection) as you are outsourcing the security controls.
Some people also swear by other measures, like changing the SSH port to something else. Most people end up using 2222 to easily remember. This is borderline useless, as you can see for yourself.
While being useless against a sophisticated attacker, there hasn’t been any bot activity in my sshd logs since changing my ssh port to a different one.
Yeah, what I mean is that it’s useless using ports like 2222, that’s like the unofficial SSH port!
Bots are generally harmless (once you move to key auth), and you get functional the same result with the automatic IP ban on failed auth, minus the bother to change client configurations to your custom port. Anyway, if someone does want cleaner logs, changing port works :)
AFAIK I know that SSH has MaxAuthTries and LoginGraceTime, but all it does is terminating the SSH session (I.e. slow down at most), it won’t block the IP via firewall or configuration.
Not sure if there is a recent feature that does the same.
sshd(8) will now penalise client
addresses that, for various reasons, do not successfully complete
authentication. This feature is controlled by a new sshd_config(5)
PerSourcePenalties option and is on by default.
sshd(8) will now identify situations where the session did not
authenticate as expected. These conditions include when the client
repeatedly attempted authentication unsucessfully (possibly
indicating an attack against one or more accounts, e.g. password
guessing), or when client behaviour caused sshd to crash (possibly
indicating attempts to exploit bugs in sshd).
When such a condition is observed, sshd will record a penalty of
some duration (e.g. 30 seconds) against the client’s address. If
this time is above a minimum configurable threshold, then all
connections from the client address will be refused (along with any
others in the same PerSourceNetBlockSize CIDR range) until the
penalty expire.
Repeated offenses by the same client address will accrue greater
penalties, up to a configurable maximum. Address ranges may be
fully exempted from penalties, e.g. to guarantee access from a set
of trusted management addresses, using the new sshd_config(5)
PerSourcePenaltyExemptList option.
You are not logged in. However you can subscribe from another Fediverse account, for example Lemmy or Mastodon. To do this, paste the following into the search field of your instance: !selfhosted@lemmy.world
A place to share alternatives to popular online services that can be self-hosted without giving up privacy or locking you into a service you don’t control.
Rules:
Be civil: we’re here to support and learn from one another. Insults won’t be tolerated. Flame wars are frowned upon.
No spam posting.
Posts have to be centered around self-hosting. There are other communities for discussing hardware or home computing. If it’s not obvious why your post topic revolves around selfhosting, please include details to make it clear.
Don’t duplicate the full text of your blog or github here. Just post the link for folks to click.
Submission headline should match the article title (don’t cherry-pick information from the title to fit your agenda).
Fail2ban + key-based SSH + self-hosted WAF if you can spin up another machine == 80% of your Web hosting problems gone
That is basically the essence of this post too! Except crowdsec is used to do what fail2ban does + some light form of WAF (without spinning another machine - which is not strictly needed for a WAF, you can use owasp modsecurity-ready proxies).
Good read.
I would just like to add some additional information that favors changing your SSH port to something other than the default. When crawlers are going around the internet looking for vulnerable SSH servers, they’re more than likely going to have an IP range and specifically look for port 22.
Now can they go through and scan your IP and all of its ports to look for the SSH service? Yes. But you will statistically have less interactions with bad actors this way since they might specifically be looking for port 22.
As a side note, don’t be cute and pick port 221 or 2222 or 22022 or whatever that’s got “22” in it.
I know that sounds silly but the slightly less stupid bots are written by people who understand people do things like that and account for them, and thus port 2222 isn’t actually better than 22, or whatever.
imho - never expose that shit anyways, and VPN into your local network first. Only thing I ever expose to the internet is 80/443.
At the very least, if you’re going to expose an SSH session to the internet, set up some sort of port-knocking. It’s security by obscurity, sure - but it will keep all but the most ardent intruders out.
Agreed, but sometimes you have to expose things you’d rather not; I just figured I’d mention that almost everyone’s immediate urge to just go ‘port 2222! that’ll do it!’ isn’t exactly better than doing nothing anymore.
There is one neat trick: don’t expose SSH.
There is still not a reason anyone has been able to give for 99% of self-hosters to expose SSH.
If you need to access your machine via ssh while on the go. Wireguard to your local network, use SSH. Done. Unless you are running an always-up public facing site, the amount of times you have to access your machine that can’t wait until after work is very low anyway.
Bots will scan all ports. That is just how it works. Less than 22, but you will still get spammed. Why force your computer to go through the fail2ban loop and take up resources when it is simply not needed at all and you can block it on another machine?
This blog is specifically for websites that are public facing. Sure, you can wireguard into your local network, but you can also SSH into your local network. Either way you have to poke a hole.
Comfort is the main reason, I suppose. If I mess up Wireguard config, even to debug the tunnel I need to go to the KVM console. It also means that if I go to a different place and I have to SSH into the box I can’t plug my Yubikey and SSH from there. It’s a rare occurrence, but still…
Ultimately I do understand both point of view. The thing is, SSH bots pose no threats after the bare minimum hardening for SSH has been done. The resource consumption is negligible, so it has no real impact.
To me the tradeoff is slight inconvenience vs slightly bigger attack surface (in case of CVEs). Ultimately everyone can decide which compromise is acceptable for them, but I would say that the choice is not really a big one.
Thanks! I did mention this briefly, although I belong to the school that “since I am anyway banning IPs that fail authentication a few times, it’s not worth changing the port”. I think that it’s a valid thing especially if you ingest logs somewhere, but if you do don’t choose 2222! I have added a link to shodan in the post, which shows that almost everybody who changes port, changes to 2222!
Yeah, I just left my SSH port as 22 since I only use key-based authentication so there’s really no security risk. Plus, it’s funny going through the logs and looking at all the login attempts.
Yep I agree. Especially looking at all the usernames that are tried. I do the same and the only risk come from SSH vulnerabilities. Since nobody would burn a 0-day for SSH (priceless) on my server, unattended upgrades solve this problem too for the most part.
I mean we just had https://nvd.nist.gov/vuln/detail/CVE-2024-6387 – so my guess is that you’re updating quite often to be so confident in your unattended upgrades.
Yeah I know (I mentioned it myself in the post), but realistically there is no much you can do besides upgrading. Unattended upgrades kick in once a day and you will install the security patches ASAP. There are also virtual patches (crowdsec has a virtual patch for that CVE), but they might not be very effective.
I argue that VPN software is a smaller attack surface, but the problem still exists (CVEs) for everything you expose.
deleted by creator
Unrelated to your actual post (plan to read later), but is your RSS busted? The rss link on the webpage gives a 404 and my RSS reader is erroring on it as well…
Oh, it looks like! Something went wrong with Zola build and I must have not noticed. Thanks a lot for pinging me about that, I will fix it today!
EDIT: Fixed! That’s what you get when you forget to bump the Docker image version after you upgrade zola version locally with a breaking change in the config! :) Thanks for letting me know, it would have taken me a long time to see it was broken!
Can you use CrowdSec to track logs from a k8s pod? Say I have my website and some other services hosted on a k3s cluster, do I need to spin up a new pod for CrowdSec or should it be installed on the host?
Hey, the short answer is yes, you can.
I would elaborate a little more:
In practice I personally would choose a simple setup where the interesting logs are just forwarded (in Syslog format for example) to a single crowdsec instance. If you have ingress from a single node, I’d go for running it on the host and banning via firewall, if you have multiple ingress nodes, then I would run it inside the cluster and ban via a loadBalancer/cloud firewall/whatever you have in front.
In essence, I would spend some time to think about your preferences, and it might take a little bit to make the setup clean, but I think you have plenty of flexibility to do what you prefer. Let me know if you want to bounce some more ideas!
Acronyms, initialisms, abbreviations, contractions, and other phrases which expand to something larger, that I’ve seen in this thread:
6 acronyms in this thread; the most compressed thread commented on today has 7 acronyms.
[Thread #892 for this sub, first seen 27th Jul 2024, 23:25] [FAQ] [Full list] [Contact] [Source code]
Genuine question from someone with a single page static site - why is Cloudflare a useless suggestion?
It is bad for the open web and constantly asks you to verify you are human
Fair question. What I meant is that suggesting that would have made the whole post 10 lines long and not worth doing. So I avoided such suggestions that completely change the threat model.
It’s not useless to avoid a good security posture (although you might have concerns of a monopoly gatekeeping the internet, TLS traffic inspection privacy concerns etc.), on the contrary makes everything I have written about here redundant (+ provide more, like DDoS protection) as you are outsourcing the security controls.
Thanks, I was more worried that there was something completely wrong security-wise with that approach!
While being useless against a sophisticated attacker, there hasn’t been any bot activity in my sshd logs since changing my ssh port to a different one.
Yeah, what I mean is that it’s useless using ports like 2222, that’s like the unofficial SSH port! Bots are generally harmless (once you move to key auth), and you get functional the same result with the automatic IP ban on failed auth, minus the bother to change client configurations to your custom port. Anyway, if someone does want cleaner logs, changing port works :)
I recall hearing that openssh has something like fail2ban built-in now. I forget the name of the feature.
AFAIK I know that SSH has MaxAuthTries and LoginGraceTime, but all it does is terminating the SSH session (I.e. slow down at most), it won’t block the IP via firewall or configuration.
Not sure if there is a recent feature that does the same.
It’s this (excuse formatting): https://www.openssh.com/releasenotes.html
sshd(8) will now penalise client addresses that, for various reasons, do not successfully complete authentication. This feature is controlled by a new sshd_config(5) PerSourcePenalties option and is on by default.
sshd(8) will now identify situations where the session did not authenticate as expected. These conditions include when the client repeatedly attempted authentication unsucessfully (possibly indicating an attack against one or more accounts, e.g. password guessing), or when client behaviour caused sshd to crash (possibly indicating attempts to exploit bugs in sshd).
When such a condition is observed, sshd will record a penalty of some duration (e.g. 30 seconds) against the client’s address. If this time is above a minimum configurable threshold, then all connections from the client address will be refused (along with any others in the same PerSourceNetBlockSize CIDR range) until the penalty expire.
Repeated offenses by the same client address will accrue greater penalties, up to a configurable maximum. Address ranges may be fully exempted from penalties, e.g. to guarantee access from a set of trusted management addresses, using the new sshd_config(5) PerSourcePenaltyExemptList option.
Nice! I didn’t know this. Thanks!