So I have a few 5TB external drives with all my media. I mainly just hook them up to my xbox and use kodi to play the files locally. I should probably be investing into a NAS or some sort of JBOD, but that’s a whole other issue.
So no drive is backed up but they are all new (about 6 months old). Obviously, hdd drives can have mechanical failure, be a dud, or just plain suck. I do smart check and make sure the drive health is fine. I have been reading about bit rot and not sure if that is something I have to worry about and an immediate thing (or it’s just some snake oil shit). I want to make sure the data stays readable and in best shape I can have it in for as long as possible. From the reading I have done surfin the web “refreshing” the data is usually done by rewriting the data. I guess what I am worried about is data corruption. With all this being on a HDD, mechanical failure is probably a bigger issue, and is something I should eventually get to with actual backups/parity. Drives are getting cheaper but I don’t have the cash to drop right now on better/larger/enterprise (or NAS) drives to set all that up. I don’t really want to re-download 20+ TB of stuff just to rewrite my data or shuffle data back and forth between my computer and the drive to rewrite. I could be going about this all wrong, so some pointer or input is helpful.
I use CrystalDiskInfo and HD Tune Pro to check drive health, but kinda just wanted to know if there are programs make help against data corruption, if it’s even something that I should be immediately worried about, if I’m going about this in a dumb way, or if I should just start saving and work towards bulding a NAS, JBOD, ZFS or some shit.
(if I’m honest all that shit seems out of my ballpark cause I like to just download and play, but it may be time to learn more about all that shit with regards to raid, parity, and having true backups)
tl;dr : I want to keep my shit for as long as possible on my HDDs. Back ups = good, but short of doing that, how to make sure data I currently have stays healthy on current drives?
Short of backups, you need checksums and the like to detect corruption, and parity data to repair any corruption that may be found. If you received the files using torrents, those have checksums (SHA-1), so you can just tell your torrent client to recheck.
However, that won’t help you recover data. If the torrent dies, you won’t be able to get the lost data again. This is where parity comes in. It can take many forms, such as ZFS or RAID 5/6. If you simply want a file/folder-level approach, you can build recovery volumes in WinRARv when you compress. But you probably want to create PAR2 recovery volumes instead. I haven’t used it in years, but QuickPAR should still work. The big limitation is that it can’t handle subdirectories, so you need a unique set for each folder.
Side note: my setup is 24/7, mostly consumer-grade, high-quality parts. I’ve never had an issue with bitrot or silent corruption from bad RAM. I have, however lost a lot of data from drive failures, human error, and a particularly stupid decision by WD that led to me having the wrong driver. Don’t try to predict drive failures with SMART and the like. Most failures won’t show in SMART until at least some of the data is already gone. But if you get a SMART error, data loss is imminent.
You are not logged in. However you can subscribe from another Fediverse account, for example Lemmy or Mastodon. To do this, paste the following into the search field of your instance: !piracy@lemmy.dbzer0.com
⚓ Dedicated to the discussion of digital piracy, including ethical problems and legal advancements.
Short of backups, you need checksums and the like to detect corruption, and parity data to repair any corruption that may be found. If you received the files using torrents, those have checksums (SHA-1), so you can just tell your torrent client to recheck.
However, that won’t help you recover data. If the torrent dies, you won’t be able to get the lost data again. This is where parity comes in. It can take many forms, such as ZFS or RAID 5/6. If you simply want a file/folder-level approach, you can build recovery volumes in WinRARv when you compress. But you probably want to create PAR2 recovery volumes instead. I haven’t used it in years, but QuickPAR should still work. The big limitation is that it can’t handle subdirectories, so you need a unique set for each folder.
Side note: my setup is 24/7, mostly consumer-grade, high-quality parts. I’ve never had an issue with bitrot or silent corruption from bad RAM. I have, however lost a lot of data from drive failures, human error, and a particularly stupid decision by WD that led to me having the wrong driver. Don’t try to predict drive failures with SMART and the like. Most failures won’t show in SMART until at least some of the data is already gone. But if you get a SMART error, data loss is imminent.