Every time I try to access this community, ther’s some kind of problem with the server. If you have a look at the status page, it’s almost all orange/red.
The problem aren’t DDoS attack since the server is behind Cloudflare protection.
Admin/mods, why don’t you move this community to a different server instance?
I’m not accusing anybody, I know that maintain a server can be a challenging sometimes, I just want to enjoy this community!
Cloudflare DDos protection isn’t a silver bullet; the attacks are distributed and come from shifting source IPs, and are sophisticated in that they exploit resource intensive queries specifically designed to overload a Lemmy instance. If lemmy.world were to pivot to some other instance, who’s to say the culprits wouldn’t just resume their efforts pointed at the new location? There are theories these may be carried out by the recently-defederated fringe hate communities
Yep Cloudflare protects against classic DDoS (like many clients doing a lot of small requests). Here attacks are performed presumibly by users that know very well how the Lemmy’s backend works and where bottlenecks are, so that with a small number of well made requests they are able to mess up the backend and Cloudflare doesn’t notice it
From what I understand, Cloudflare can block some DDoS attacks, but not all of them.
The attacks on Lemmy have to do with poorly optimized SQL requests; these are requests that shouldn’t take long to execute, but do due to some oversight. By spamming these requests, the attackers can bring Lemmy on it’s knees.
Actually, wouldn’t this attack better be categorized as a DoS attack ? What’s so distributed about it ?
Right, but it’s possible to execute those API requests to trigger those expensive database requests in a way that wouldn’t necessarily trigger cloudflare’s DDoS protection.
Cloudflare has DDoS protection but it can’t stop everything 100% of the time. According to the admins, the attackers are very familiar with how lemmy works and are using this knowledge to overwhelm resources. This isn’t just a simple script kiddy or bonnet for hire but likely points to someone that has worked within the lemmy community.
It can help, but is not perfect. There’s kind of been an arms race between services like Cloudflare and script kiddies/hackers DDOSing: Their methods became a little more sophisticated to the point that they keep the traffic shifting from address to address so it’s harder to track/block.
They’ll do other things to mess with the host too like spamming “white noise” pictures to fill up server storage space, so while DDOS attacks play a role, there are other issues at play on top of that.
For what it’s worth, I’ve been using Kbin.social and sh.itjust.works as well and they have also had some issues here and there. No host will be perfect and invulnerable from every attack, and when an instance becomes more popular, it becomes increasingly likely to be targeted by attackers.
You are not logged in. However you can subscribe from another Fediverse account, for example Lemmy or Mastodon. To do this, paste the following into the search field of your instance: !selfhosted@lemmy.world
A place to share alternatives to popular online services that can be self-hosted without giving up privacy or locking you into a service you don’t control.
Rules:
Be civil: we’re here to support and learn from one another. Insults won’t be tolerated. Flame wars are frowned upon.
No spam posting.
Posts have to be centered around self-hosting. There are other communities for discussing hardware or home computing. If it’s not obvious why your post topic revolves around selfhosting, please include details to make it clear.
Don’t duplicate the full text of your blog or github here. Just post the link for folks to click.
Submission headline should match the article title (don’t cherry-pick information from the title to fit your agenda).
No trolling.
Resources:
selfh.st Newsletter and index of selfhosted software and apps
The problem is DDoS attacks. See: https://lemmy.world/post/2923697
But isn’t Cloudflare supposed to block those attacks?
Cloudflare DDos protection isn’t a silver bullet; the attacks are distributed and come from shifting source IPs, and are sophisticated in that they exploit resource intensive queries specifically designed to overload a Lemmy instance. If lemmy.world were to pivot to some other instance, who’s to say the culprits wouldn’t just resume their efforts pointed at the new location? There are theories these may be carried out by the recently-defederated fringe hate communities
Yep Cloudflare protects against classic DDoS (like many clients doing a lot of small requests). Here attacks are performed presumibly by users that know very well how the Lemmy’s backend works and where bottlenecks are, so that with a small number of well made requests they are able to mess up the backend and Cloudflare doesn’t notice it
From what I understand, Cloudflare can block some DDoS attacks, but not all of them.
The attacks on Lemmy have to do with poorly optimized SQL requests; these are requests that shouldn’t take long to execute, but do due to some oversight. By spamming these requests, the attackers can bring Lemmy on it’s knees.
Actually, wouldn’t this attack better be categorized as a DoS attack ? What’s so distributed about it ?
They explained that the attacks are in the form of requests that tax the database server, not the website itself.
The database can’t be accessed directly, all the requests needs to be done from the website or API.
Right, but it’s possible to execute those API requests to trigger those expensive database requests in a way that wouldn’t necessarily trigger cloudflare’s DDoS protection.
Cloudflare has DDoS protection but it can’t stop everything 100% of the time. According to the admins, the attackers are very familiar with how lemmy works and are using this knowledge to overwhelm resources. This isn’t just a simple script kiddy or bonnet for hire but likely points to someone that has worked within the lemmy community.
https://www.cloudflare.com/learning/ddos/what-is-a-ddos-attack/
https://www.cloudflare.com/learning/ddos/glossary/web-application-firewall-waf/
It can help, but is not perfect. There’s kind of been an arms race between services like Cloudflare and script kiddies/hackers DDOSing: Their methods became a little more sophisticated to the point that they keep the traffic shifting from address to address so it’s harder to track/block.
They’ll do other things to mess with the host too like spamming “white noise” pictures to fill up server storage space, so while DDOS attacks play a role, there are other issues at play on top of that.
For what it’s worth, I’ve been using Kbin.social and sh.itjust.works as well and they have also had some issues here and there. No host will be perfect and invulnerable from every attack, and when an instance becomes more popular, it becomes increasingly likely to be targeted by attackers.