I have a server running Debian with 24 TB of storage. I would ideally like to back up all of it, though much of it is torrents, so only the ones with low seeders really need backed up. I know about the 321 rule but it sounds like it would be expensive. What do you do for backups? Also if anyone uses tape drives for backups I am kinda curious about that potentially for offsite backups in a safe deposit box or something.
TLDR: title.
Edit: You have mentioned borg and rsync, and while borg looks good, I want to go with rsync as it seems to be more actively maintained. I would like to also have my backups encrypted, but rsync doesn’t seem to have that built in. Does anyone know what to do for encrypted backups?
I use Kopia to backup all personal data (nextcloud, immich, configs, etc) daily to another disk in the same server and also to backblaze B2. Its not proper 321 but feels good enough. I dont backup downloadable content because its expensive
I can chain an encryption process to that, so it encrypts then backs up.
I can then mount the encrypted, remote files so that I can easily get to them locally easily (e.g. I could run diff or md5 on select files as naturally as if they were local).
And it supports the rsync --backup options so that it can move locally deleted files elsewhere on the backup instead of deleting them there. I can set up a dir structure such as
Oldfiles/20240301
Oldfiles/20240308
Etc that preserve deletions.
I have a machine at my parents’ house that has a single 20TB drive in it. I’ll log in once in a while and initiate an rsync to bring that up to current with my RAID at home. The specific reason I do it manually is in case there’s a ransomware attack. I won’t copy bad data. That’s also the reason I start it from the backup machine. The main machine doesn’t connect, the backup machine does, so ransomware wouldn’t cross that virtual boundary.
I backup my /home folder on my PC to my NAS using restic (used to use borg, but restic is more flexible). I backup somewhat important data to an external SSD on a weekly basis and very important data to cloud storage on a nightly basis. I don’t backup my *arr media at all (unless you count the automated snapshots on my NAS), as it’s not really important to me and can simply be redownloaded in most cases.
So I don’t and wouldn’t apply the 321 rule to all data as it’s simply too expensive for the amount of data I have and it’d take months to upload with my non-fiber internet connection. But you should definitely apply it to data that’s important to you.
I might be crazy but I have a 20TB WD Red Pro in a padded, water proof, locking, case that I take a full backup on and then drive it over to a family members 30m away once a month or so.
It’s a full encrypted backup of all my important stuff in a relatively different geographic location.
All of my VM data backs up hourly to my NAS as well. Which then gets backed up onto the large drive monthly.
Monthly granularity isn’t that good to be fair but it’s better than nothing. I should probably back up the more important rapidly changing stuff online daily.
30 minutes can vary a lot depending on traffic. If there’s traffic, it can take me 30-40 minutes to get home from work even though it’s only 11 miles away and ~15 mins with no traffic.
I have my BD/DVD/CD collection backed up to S3 Glacier. It’s incredibly cheap, offsite, and they worry about the infrastructure. The amount of Hard drive and infrastructure space you’ll need to back up nearly that amount will cost you the about the same give or take. Yes it’ll cost a bit in the event of a catastrophic restore, but if I have something happen at the house, at least I have an offsite backup.
How much does Glacier cost you? Last time I checked, some hosts had warm storage for around the same price, at least during Black Friday or New Year sales.
I can’t recall storage costs (they’re on the website somewhere but are not straightforward).
I was paying maybe $7 a month for a few hundred Gb, although not all of that was glacier.
But retrieval was a pain. There’s no straightforward way to convert back from glacier for a lot of files and there’s a delay. The process creates a non-glacier copy with a limited lifespan to retrieve.
Then the access costs were maybe $50 to move stuff out.
I moved to rsync.net for the convenience and simplicity. It even supported setting up rclone to access s3 directly. So I could do cloud-to-cloud to copy the files over.
I have a storage VPS with HostHatch - 10TB for $10/month. That pricing was from a Black Friday sale a few years ago. They may not offer it that cheap again, but it’s worth keeping an eye out for their sales. They had something similar last year but double the price, which is still a good deal.
I use Borgbackup to back up the data to the HostHatch VPS. The most important data has a second copy stored with pcloud - I’ve got a lifetime 2TB storage plan with them. I know lifetime accounts are kinda sketchy which is why it’s just a secondary backup and not the primary one.
I don’t have any “disposable” files like torrents though. All the stuff I back up are things like servers that run my websites and email, family photos, CDs I’ve ripped myself, etc. I’ve only got a few TB total.
What I use is Borg. I use Borg to backup the server to a local NAS. Then I have a NAS at my grand parents house which I use to store the backups of the NAS it self.
As of today I’m actually in a lucky position where I am now able to set up a secondary NAS at my brother in laws and use that as a backup server that I can back up to essentially in real time.
All it’ll cost me is the hardware and the electricity.
Anything I can download again doesn’t get backup, but it sits on a RAID-1. I am ok at losing it due to carelessness but not due to a broken disk. I try to be carefully when messing with it and that’s enough, I can always download again.
Anything like photos notes personal files and such gets backedup via restic to a disk mounted to the other side of the house. Offsite backup i am thinking about it, but not really got to it yet. Been lucky all this time.
From 10tb of stuff, the totality of my backupped stuff amount to 700gb. Since 90% of are photos, the backup size is about 700gb too. The actually part of that 700gb that changes (text files, documents…) amount to negligible. The photos never change, at most grow a bit over time.
For offsite I backup to aws Glacier. Cheap to store expensive to retrieve. When the house burns down I’ll still have the photos somewhere and at that point the cost is negligible compared to losing them since it really is worst case scenario.
For offsite I backup to aws Glacier. Cheap to store expensive to retrieve. When the house burns down I’ll still have the photos somewhere and at that point the cost is negligible compared to losing them since it really is worst case scenario.
Short answer: figure out how much of that is actually irreplaceable and then find a friend or friends who’d be willing to set aside some of their storage space for your backups in exchange for you doing the same.
Tailscale makes the networking logistics incredibly simple and then you can do the actual backups however you see fit.
It depends on the value of the data. Can you afford to replace them? Is there anything priceless on there (family photos etc)?
Will the time to replace them be worth it?
If its not super critical, raid might be good enough, as long as you have some redundancy. Otherwise, categorizing your data into critical/non-critical and back it up the critical stuff first?
RAID is not backup. Many failure sources from theft over electrical issues to water or fire can affect multiple RAID drives equally, not to mention silent data corruption or accidental deletions.
Its not, but if the value of the data is low, its good enough. There is no point backing up linux isos, but family photos definitely should be properly backed up according to 3-2-1.
Yeah…I’ve never totally lost my main storage and had to recover from backups. But on a number of occasions, I have been able to recover something that was inadvertently wiped. RAID doesn’t provide that.
Also, depending upon the structure of your backup system, if someone compromises your system, they may not be able to compromise your backups.
If you need continuous uptime in the event of a drive failure, RAID is an entirely reasonable thing to have. It’s just…not a replacement for backups.
Oh, all my drives are RAID too, mostly for the convenience of being able to use them while I order a replacement for a failed drive and not having to restore from backup once I get that.
You are not logged in. However you can subscribe from another Fediverse account, for example Lemmy or Mastodon. To do this, paste the following into the search field of your instance: !selfhosted@lemmy.world
A place to share alternatives to popular online services that can be self-hosted without giving up privacy or locking you into a service you don’t control.
Rules:
Be civil: we’re here to support and learn from one another. Insults won’t be tolerated. Flame wars are frowned upon.
No spam posting.
Posts have to be centered around self-hosting. There are other communities for discussing hardware or home computing. If it’s not obvious why your post topic revolves around selfhosting, please include details to make it clear.
Don’t duplicate the full text of your blog or github here. Just post the link for folks to click.
Submission headline should match the article title (don’t cherry-pick information from the title to fit your agenda).
No trolling.
Resources:
selfh.st Newsletter and index of selfhosted software and apps
I use Kopia to backup all personal data (nextcloud, immich, configs, etc) daily to another disk in the same server and also to backblaze B2. Its not proper 321 but feels good enough. I dont backup downloadable content because its expensive
I use rclone, which is essentially rsync for cloud services. It supports encrypion out of the box.
I like the versatility of rclone.
It can copy to a cloud service directly.
I can chain an encryption process to that, so it encrypts then backs up.
I can then mount the encrypted, remote files so that I can easily get to them locally easily (e.g. I could run diff or md5 on select files as naturally as if they were local).
And it supports the rsync --backup options so that it can move locally deleted files elsewhere on the backup instead of deleting them there. I can set up a dir structure such as Oldfiles/20240301 Oldfiles/20240308 Etc that preserve deletions.
Acronyms, initialisms, abbreviations, contractions, and other phrases which expand to something larger, that I’ve seen in this thread:
6 acronyms in this thread; the most compressed thread commented on today has 10 acronyms.
[Thread #642 for this sub, first seen 30th Mar 2024, 20:45] [FAQ] [Full list] [Contact] [Source code]
I have a machine at my parents’ house that has a single 20TB drive in it. I’ll log in once in a while and initiate an rsync to bring that up to current with my RAID at home. The specific reason I do it manually is in case there’s a ransomware attack. I won’t copy bad data. That’s also the reason I start it from the backup machine. The main machine doesn’t connect, the backup machine does, so ransomware wouldn’t cross that virtual boundary.
I would like to replicate your setup in the future. How do you connect between the two machines, using tailscale or something like that?
It’s just over ssh. They’ve both got their own subdomains.
I backup my /home folder on my PC to my NAS using restic (used to use borg, but restic is more flexible). I backup somewhat important data to an external SSD on a weekly basis and very important data to cloud storage on a nightly basis. I don’t backup my *arr media at all (unless you count the automated snapshots on my NAS), as it’s not really important to me and can simply be redownloaded in most cases.
So I don’t and wouldn’t apply the 321 rule to all data as it’s simply too expensive for the amount of data I have and it’d take months to upload with my non-fiber internet connection. But you should definitely apply it to data that’s important to you.
I might be crazy but I have a 20TB WD Red Pro in a padded, water proof, locking, case that I take a full backup on and then drive it over to a family members 30m away once a month or so.
It’s a full encrypted backup of all my important stuff in a relatively different geographic location.
All of my VM data backs up hourly to my NAS as well. Which then gets backed up onto the large drive monthly.
Monthly granularity isn’t that good to be fair but it’s better than nothing. I should probably back up the more important rapidly changing stuff online daily.
30 minutes, 30 miles, or 30 metres?
Yes.
I’m sure one can reasonably infer that I do not mean 30 meters.
Conveniently at highway speeds 30 minutes and 30 miles away are essentially equal.
I’ll try and use appropriate notation next time
I was just joking :)
30 minutes can vary a lot depending on traffic. If there’s traffic, it can take me 30-40 minutes to get home from work even though it’s only 11 miles away and ~15 mins with no traffic.
I put the prndl in r and just goose it
I have my BD/DVD/CD collection backed up to S3 Glacier. It’s incredibly cheap, offsite, and they worry about the infrastructure. The amount of Hard drive and infrastructure space you’ll need to back up nearly that amount will cost you the about the same give or take. Yes it’ll cost a bit in the event of a catastrophic restore, but if I have something happen at the house, at least I have an offsite backup.
How much does Glacier cost you? Last time I checked, some hosts had warm storage for around the same price, at least during Black Friday or New Year sales.
I can’t recall storage costs (they’re on the website somewhere but are not straightforward).
I was paying maybe $7 a month for a few hundred Gb, although not all of that was glacier.
But retrieval was a pain. There’s no straightforward way to convert back from glacier for a lot of files and there’s a delay. The process creates a non-glacier copy with a limited lifespan to retrieve.
Then the access costs were maybe $50 to move stuff out.
I moved to rsync.net for the convenience and simplicity. It even supported setting up rclone to access s3 directly. So I could do cloud-to-cloud to copy the files over.
I have a storage VPS with HostHatch - 10TB for $10/month. That pricing was from a Black Friday sale a few years ago. They may not offer it that cheap again, but it’s worth keeping an eye out for their sales. They had something similar last year but double the price, which is still a good deal.
I use Borgbackup to back up the data to the HostHatch VPS. The most important data has a second copy stored with pcloud - I’ve got a lifetime 2TB storage plan with them. I know lifetime accounts are kinda sketchy which is why it’s just a secondary backup and not the primary one.
I don’t have any “disposable” files like torrents though. All the stuff I back up are things like servers that run my websites and email, family photos, CDs I’ve ripped myself, etc. I’ve only got a few TB total.
What I use is Borg. I use Borg to backup the server to a local NAS. Then I have a NAS at my grand parents house which I use to store the backups of the NAS it self.
As of today I’m actually in a lucky position where I am now able to set up a secondary NAS at my brother in laws and use that as a backup server that I can back up to essentially in real time.
All it’ll cost me is the hardware and the electricity.
Anything I can download again doesn’t get backup, but it sits on a RAID-1. I am ok at losing it due to carelessness but not due to a broken disk. I try to be carefully when messing with it and that’s enough, I can always download again.
Anything like photos notes personal files and such gets backedup via restic to a disk mounted to the other side of the house. Offsite backup i am thinking about it, but not really got to it yet. Been lucky all this time.
From 10tb of stuff, the totality of my backupped stuff amount to 700gb. Since 90% of are photos, the backup size is about 700gb too. The actually part of that 700gb that changes (text files, documents…) amount to negligible. The photos never change, at most grow a bit over time.
For offsite I backup to aws Glacier. Cheap to store expensive to retrieve. When the house burns down I’ll still have the photos somewhere and at that point the cost is negligible compared to losing them since it really is worst case scenario.
For offsite I backup to aws Glacier. Cheap to store expensive to retrieve. When the house burns down I’ll still have the photos somewhere and at that point the cost is negligible compared to losing them since it really is worst case scenario.
Important stuff (about 150G) is synced to all my machines and a b2 Backblaze bucket.
I have a rented seed box for those low seeder torrents.
The stuff I can download again is only on a mirrored lvm pool with an lvmcache. I don’t have any redundancy for my monerod data which is on an nvme.
I’m moving towards an immutable OS with 30 days of snapshots. While not the main reason, it does push one to practicing better sync habits.
Short answer: figure out how much of that is actually irreplaceable and then find a friend or friends who’d be willing to set aside some of their storage space for your backups in exchange for you doing the same.
Tailscale makes the networking logistics incredibly simple and then you can do the actual backups however you see fit.
It depends on the value of the data. Can you afford to replace them? Is there anything priceless on there (family photos etc)? Will the time to replace them be worth it?
If its not super critical, raid might be good enough, as long as you have some redundancy. Otherwise, categorizing your data into critical/non-critical and back it up the critical stuff first?
RAID is not backup. Many failure sources from theft over electrical issues to water or fire can affect multiple RAID drives equally, not to mention silent data corruption or accidental deletions.
Its not, but if the value of the data is low, its good enough. There is no point backing up linux isos, but family photos definitely should be properly backed up according to 3-2-1.
Yeah…I’ve never totally lost my main storage and had to recover from backups. But on a number of occasions, I have been able to recover something that was inadvertently wiped. RAID doesn’t provide that.
Also, depending upon the structure of your backup system, if someone compromises your system, they may not be able to compromise your backups.
If you need continuous uptime in the event of a drive failure, RAID is an entirely reasonable thing to have. It’s just…not a replacement for backups.
Oh, all my drives are RAID too, mostly for the convenience of being able to use them while I order a replacement for a failed drive and not having to restore from backup once I get that.