So I selfhost a number of servers in various locations and utilize a DigitalOcean VPS as a hub/gateway to transmit data between these nodes.
I have a consistent issue when running large backups or transfers in which DigitalOcean flags my server for a DDOS attack and sends traffic to a black hole for 3-4 hours.
Customer support has been the absolute worst and does fuck all to help remedy the situation in any way. These events have been consistent over the past 8 months.
Does anybody have recommendations for a solid VPS provider?
Price isn’t too much of a factor. I was running a $8/month server but I don’t mind something more expensive if the company has a good reputation for reliability and privacy.
As mentioned, I primarily use a VPS as a gateway/hub for file transfer services. I’m also hoping to spin up another VPS for static websites.
Try using ICE instead of proxying all your traffic through a VPS. If you’re just using the VPS for session establishment you won’t be using a lot of bandwidth and won’t get blocked or go over quota. Try searching for things like “wireguard mesh stun”.
I would try Vultr, if DO is being weird. They are awesome and have very similar pricing to DO.
Hetzner is ok as well, but they have annoying DDoS filtering.
You could also try AWS ARM EC2, but they’re expensive for bandwidth by comparison.
For backups, consider using rsync.net. for a server, have you looked at dedicated servers before? OVH has some cheap servers every once in a while that should be better in theory than most VPS.
Thanks. I actually selfhost my backup server. So I’m not backing up to a VPS. I use the VPS as a hub in a hub and wheel configuration to connect multiple servers (including a dedicated backup server).
That’s really unfortunate. I love Digital Ocean and spend about $800/month with them for work.
Can you tell me more about the traffic they are mistakenly flagging as a DDOS? I ask because I have regular DB and file backups happening and if we had traffic shutdown on production assets for 3-4 hours, it would be a big fucking deal.
So each time I get shut down is during a large extended data transfer. I have my VPS server set up as a VPN hub that connects multiple servers. So typically when my traffic gets diverted to a black hole by DO, there was a consistent roughly 35MB/s inbound/outbound vpn traffic stream for 4-5 hours going through the VPS. My server gets shut down for 3-4 hours and I get a email notice that my server was under a massive DDoS attack and they diverted traffic to a black hole. I always respond informing them that it’s not a DDoS and explain the situation. They typically respond with “Utilize a service like Cloudfare which has DdoS protection”.
I’ve been really happy with them as a provider otherwise but this is a dealbreaker for me.
Hmm, that really doesn’t sound like a traffic pattern that would be confused with a DDoS attack. I would be frustrated as hell too.
What’s concerning is that our traffic would look very similar. We have a VPN dedicated droplet that allows access to our DO private network where the rest of our resources can be accessed. We also have high throughput periods though not as sustained as yours.
Try Hetzner “cloud” offer (shared vCPU). They price it by CPU, RAM and storage, the transfer is 20 TB for all plans. You can choose to be hosted in Germany, Finland or the US (typically, sometimes a specific plan is temporarily sold out in a certain region).
I don’t think you’ll find any guarantees about that since it will vary wildly depending on their bandwidth use at any given time, your ISP’s, and everything in between. I don’t think they cap it on purpose if that’s what you mean, but they do have an upper capacity limit, naturally. Generally speaking you should get some pretty decent speeds, like the other comment mentioned.
I can’t speak to the moral side, but it’s worth noting that from a privacy perspective the major cloud providers basically don’t want to be able to interact with your data.
I work as a cloud engineer and regularly engage with support from Google and Amazon and in general they can only see stuff like metadata and resource configuration, as well as the raw hardware health for your resources. For anything further generally you’re going to have to explicitly provide information, share your screen, etc.
Just wanted to clear up that tidbit. Again, doesn’t help with any moral objections you may have though.
I appreciate your insight. That’s good to know. My journey into self hosting started with searching for alternatives to google products so I’m naturally hesitant to touch anything under their umbrella.
I have used Vultr and I’m quite happy with them, however I had not moved backup level data into the servers so can’t attest that they’ll work great for you.
Maybe not what you’re looking for, but maybe its time to switch to a mesh style VPN, and use your DO VPS’ as coordinator nodes, so they don’t route all that traffic. Tailscale, head scale, netbird, nebula are all options.
I tried to set up a nebula network but it seems like it has trouble when your hosts are behind a VPN service. The VPN must block the port or protocol the lighthouse is connecting with and I can’t figure out a way to bypass the VPN (at least on Mac split tunneling isn’t supported). I’m assuming you’re familiar with mesh networks…do you have any good youtube videos or resources you would recommend? The nice thing about VPN is it’s crazy simple to set up and seems to work with all types of system configurations. Nebula was pretty simple but seems like a pain to troubleshoot so far.
Funny you mention that. I was about to make a post about Nebula earlier. I learned about it through YouTuber apalrd a few months back and it seems perfect. I’m still trying to understand some of the complexities when utilizing a service that requires circumventing the mesh network for public access such as Nextcloud. I’ll probably make a post about this after I’ve done some more research. I think there’s some good discussion to be had about such a setup.
If you’re feeling adventurous, lowendtalk is quite a deep rabbit hole.
Personally, I use rsync.net for backup stuff. Way better for backup than standard vps because everything you put there is automatically snapshotted every day (you can configure how long you want to keep the snapshot). No full shell access, but you can still use rclone there.
You are not logged in. However you can subscribe from another Fediverse account, for example Lemmy or Mastodon. To do this, paste the following into the search field of your instance: !selfhosted@lemmy.world
A place to share alternatives to popular online services that can be self-hosted without giving up privacy or locking you into a service you don’t control.
Rules:
Be civil: we’re here to support and learn from one another. Insults won’t be tolerated. Flame wars are frowned upon.
No spam posting.
Posts have to be centered around self-hosting. There are other communities for discussing hardware or home computing. If it’s not obvious why your post topic revolves around selfhosting, please include details to make it clear.
Don’t duplicate the full text of your blog or github here. Just post the link for folks to click.
Submission headline should match the article title (don’t cherry-pick information from the title to fit your agenda).
No trolling.
Resources:
selfh.st Newsletter and index of selfhosted software and apps
Try using ICE instead of proxying all your traffic through a VPS. If you’re just using the VPS for session establishment you won’t be using a lot of bandwidth and won’t get blocked or go over quota. Try searching for things like “wireguard mesh stun”.
I would try Vultr, if DO is being weird. They are awesome and have very similar pricing to DO. Hetzner is ok as well, but they have annoying DDoS filtering.
You could also try AWS ARM EC2, but they’re expensive for bandwidth by comparison.
For backups, consider using rsync.net. for a server, have you looked at dedicated servers before? OVH has some cheap servers every once in a while that should be better in theory than most VPS.
Thanks. I actually selfhost my backup server. So I’m not backing up to a VPS. I use the VPS as a hub in a hub and wheel configuration to connect multiple servers (including a dedicated backup server).
That’s really unfortunate. I love Digital Ocean and spend about $800/month with them for work.
Can you tell me more about the traffic they are mistakenly flagging as a DDOS? I ask because I have regular DB and file backups happening and if we had traffic shutdown on production assets for 3-4 hours, it would be a big fucking deal.
So each time I get shut down is during a large extended data transfer. I have my VPS server set up as a VPN hub that connects multiple servers. So typically when my traffic gets diverted to a black hole by DO, there was a consistent roughly 35MB/s inbound/outbound vpn traffic stream for 4-5 hours going through the VPS. My server gets shut down for 3-4 hours and I get a email notice that my server was under a massive DDoS attack and they diverted traffic to a black hole. I always respond informing them that it’s not a DDoS and explain the situation. They typically respond with “Utilize a service like Cloudfare which has DdoS protection”.
I’ve been really happy with them as a provider otherwise but this is a dealbreaker for me.
Hmm, that really doesn’t sound like a traffic pattern that would be confused with a DDoS attack. I would be frustrated as hell too.
What’s concerning is that our traffic would look very similar. We have a VPN dedicated droplet that allows access to our DO private network where the rest of our resources can be accessed. We also have high throughput periods though not as sustained as yours.
How many servers are you connecting to on the outside? You might have to stagger them for connections and keep the number under their radar.
Though I have to wonder how a primary mirror would handle this for some large distro like Arch.
Give Oracles always free tier a try. I shuffle over a TB through them every month with no issues.
Just beware of their reclamation policy:
https://docs.oracle.com/en-us/iaas/Content/FreeTier/freetier_topic-Always_Free_Resources.htm#compute__idleinstances
Try linode, i used to use them in a past and it was faultless.
Linode was bought by Akamai in 2022, it might not be the same as it was previously.
I used it before and still use it. No issues with my $5 linode.
Hey, some companies still don’t fix what ain’t broke.
If you want privacy try njalla. A bit more expensive but they do try hide as much data as possible and I’ve never had any downtime with them.
This looks great for privacy but their servers are hosted only in Sweden, which might be an issue since I’ll need good latency and high bandwidth.
Try Hetzner “cloud” offer (shared vCPU). They price it by CPU, RAM and storage, the transfer is 20 TB for all plans. You can choose to be hosted in Germany, Finland or the US (typically, sometimes a specific plan is temporarily sold out in a certain region).
Looks promising. Do you know what their network speeds are? I can’t seem to find that in their FAQs.
you’ll get a shared connection afaik. Speeds ranged from 1-2gbit/s on my server
That’s pretty decent. I tried speed testing some other recommendations and I was seeing 35 MB/s.
I don’t think you’ll find any guarantees about that since it will vary wildly depending on their bandwidth use at any given time, your ISP’s, and everything in between. I don’t think they cap it on purpose if that’s what you mean, but they do have an upper capacity limit, naturally. Generally speaking you should get some pretty decent speeds, like the other comment mentioned.
Are the leading clouds off the table for cost? Azure, AWS, Google?
I can’t remember if cloudflare actually hosts or just proxies.
They have metered bandwidth so it’s not suitable for OP’s bandwindth-heavy usage.
I prefer to shy away from those companies, especially Google, for moral/privacy reasons.
I can’t speak to the moral side, but it’s worth noting that from a privacy perspective the major cloud providers basically don’t want to be able to interact with your data.
I work as a cloud engineer and regularly engage with support from Google and Amazon and in general they can only see stuff like metadata and resource configuration, as well as the raw hardware health for your resources. For anything further generally you’re going to have to explicitly provide information, share your screen, etc.
Just wanted to clear up that tidbit. Again, doesn’t help with any moral objections you may have though.
I appreciate your insight. That’s good to know. My journey into self hosting started with searching for alternatives to google products so I’m naturally hesitant to touch anything under their umbrella.
Acronyms, initialisms, abbreviations, contractions, and other phrases which expand to something larger, that I’ve seen in this thread:
3 acronyms in this thread; the most compressed thread commented on today has 8 acronyms.
[Thread #380 for this sub, first seen 27th Dec 2023, 21:55] [FAQ] [Full list] [Contact] [Source code]
I have used Vultr and I’m quite happy with them, however I had not moved backup level data into the servers so can’t attest that they’ll work great for you.
I might be able to help here. Is it outbound or inbound networking? Is there a ticket number you’d be okay sharing?
I can’t say I have your usecase but I’ve been happy with RackNerd. Support has been top notch.
Maybe not what you’re looking for, but maybe its time to switch to a mesh style VPN, and use your DO VPS’ as coordinator nodes, so they don’t route all that traffic. Tailscale, head scale, netbird, nebula are all options.
This honestly sounds like the best solution to this particular problem.
I tried to set up a nebula network but it seems like it has trouble when your hosts are behind a VPN service. The VPN must block the port or protocol the lighthouse is connecting with and I can’t figure out a way to bypass the VPN (at least on Mac split tunneling isn’t supported). I’m assuming you’re familiar with mesh networks…do you have any good youtube videos or resources you would recommend? The nice thing about VPN is it’s crazy simple to set up and seems to work with all types of system configurations. Nebula was pretty simple but seems like a pain to troubleshoot so far.
You’re trying to route nebula thru another VPN? That sounds like trouble… Can you not do that?
Ideally I would be able to split tunnel around the vpn but I don’t have the option on mac
Funny you mention that. I was about to make a post about Nebula earlier. I learned about it through YouTuber apalrd a few months back and it seems perfect. I’m still trying to understand some of the complexities when utilizing a service that requires circumventing the mesh network for public access such as Nextcloud. I’ll probably make a post about this after I’ve done some more research. I think there’s some good discussion to be had about such a setup.
If you’re feeling adventurous, lowendtalk is quite a deep rabbit hole.
Personally, I use rsync.net for backup stuff. Way better for backup than standard vps because everything you put there is automatically snapshotted every day (you can configure how long you want to keep the snapshot). No full shell access, but you can still use rclone there.