For those of you running Unraid and backing up their data to Backblaze, how are you doing it?
I’ve been playing a bit with KopiaUI but what is the easiest and most straight forward way?
Bonus points if I can have the same “client/software/utility” backup data from non-servers (Windows, macOS and Linux) in my network to said Unraid server.
I don’t want to complicate the setup with a bunch of dependencies and things that would make the recovery process long and tedious or require in-depth knowledge to use it for the recovery process.
To recap:
Workstations/laptop > back up data with what? > Unraid
It’s pretty self run, if you’re a bit nervous, try making a folder locally on your workstation and try syncing with backblaze, just to get a feel for it. We all do a few test runs first anyway. Essentially you’ll need to
Install rclone
run rclone config, this will guide you step by step through adding a remote, adding credentials, etc, there’s a full guide here
Try running a copy job for your test folder. It’ll be something like rclone copy /path/to/local/folder remotenameyoucreated:/
Then go check and see if it showed up in back blaze. Play with some of the flags, like maybe you want -v so it’ll print out everything it does. During testing --dry-run can be a lifesaver.
One big caveat, make sure you read the manual on the difference between rclone copy and rclone sync, make sure you understand both before choosing on one of them.
I’m purposely leaving out my script because I think if you’re getting started in scripting you should start small. There’s no big script that will apply to every system. Just try copying one directory to backblaze first on your workstation, then try another, try adding those to a script and running them as a script. Mine really did start as
It’ll just grow with time as you add more to it, add caveats, add rules, etc. Our labs are evolving for sure, there’s no silver bullet answer. Good luck, good testing, and ping back here if you have any other questions!
My fault! This is exactly what I was going to do as I’m unfamiliar with rclone. My comment about the script was to see how to do checks to make sure the remote is actually available before trying, how to make sure the job ran successfully, how to send a notification upon success/failure, etc.
I’m pretty novice at bash but I know other languages very well. Concepts apply more or less the same across languages, the only thing that changes in most cases is syntax.
Anyways, good write up and I appreciate your feedback.
I’m still working on the amount changed, making sure I’m happy with it before it’s completely automated, but I’m working on parsing the results of summary and --dry-run on a precheck, so if oh, 10% of the files would be changed or more than cancel it, send me a notification, and for that I’d run it manually myself. Still fine tuning and not quite happy with it yet.
My first iteration was to cherry pick a few “key” files that would be randomly around my file system. Things that I will probably never ever change. This has been proven to work for a long time and honestly has saved my ass because I did accidently wipe out a few files once and this verified that my backups wouldn’t run until I fixed it. It’s a bit dumb but it did the trick for me:
#!/bin/bash
# check_file takes in a path to a local file and it's "known good hash"
function check_file {
actual=($(md5sum "$1"))
if [[ "$actual" != "$2" ]]; then
echo "ERROR: $1 did not match it's hash value."
echo "$actual != $2"
echo "Possible attack. Exiting"
exit 1
fi
echo "Validated $1 matches the checksum on file"
}
echo Starting Safety Checks
check_file /mnt/user/myshare/mything b04b917c1f66e52adf2722d35f9b51b6
# about 5 random per share
rclone copy /mnt/user/myshare myremote:/myshare
Like I said, don’t judge too much, but for me it’s a “poor man’s ransomware checker”. If any of those have been modified, do not perform a backup, notify me, and shutdown.
The script I’m building is coming up nicely. I have it checking for error codes, logging to a file and sending discord notifications. Next step is to finish up all the rclone copy’s and syncs I want to do and then trying to encrypt the data client side before uploading to cloud. I’ve seen mention of crypt but can’t find it in the official rclone docs (yet).
crypt is pretty simple honestly, go through prebuild but it’s essentially just sitting on top of another remote. So I usually use the convention {remotename}-crypt. It’ll simply rename and encrypt anything and then pass it into the remove you already set up. A good way to test it is once you’ve copied some stuff over with the crypt set up try to mount it using rclone mount and try to read some of your files, it should be fine.
Don’t forget to back up (securely, and obviously not on your encrypted share) your rclone config! if those keys are lost than so is your data.
I don’t need the whole rclone.conf file right? Just the authentication stuff for the underlying remote, the password for crypt and the salt for crypt. If I have that shouldn’t I be able to recreate the access and subsequently the decryption key to be used on some other machine?
You are not logged in. However you can subscribe from another Fediverse account, for example Lemmy or Mastodon. To do this, paste the following into the search field of your instance: !selfhosted@lemmy.world
A place to share alternatives to popular online services that can be self-hosted without giving up privacy or locking you into a service you don’t control.
Rules:
Be civil: we’re here to support and learn from one another. Insults won’t be tolerated. Flame wars are frowned upon.
No spam posting.
Posts have to be centered around self-hosting. There are other communities for discussing hardware or home computing. If it’s not obvious why your post topic revolves around selfhosting, please include details to make it clear.
Don’t duplicate the full text of your blog or github here. Just post the link for folks to click.
Submission headline should match the article title (don’t cherry-pick information from the title to fit your agenda).
No trolling.
Resources:
selfh.st Newsletter and index of selfhosted software and apps
Mind sharing the script (without the sensitive data)?
I haven’t taken the plunge into rclone because of the scripting part of the equation. Just not great at bash.
It’s pretty self run, if you’re a bit nervous, try making a folder locally on your workstation and try syncing with backblaze, just to get a feel for it. We all do a few test runs first anyway. Essentially you’ll need to
rclone config, this will guide you step by step through adding a remote, adding credentials, etc, there’s a full guide hererclone copy /path/to/local/folder remotenameyoucreated:/Then go check and see if it showed up in back blaze. Play with some of the flags, like maybe you want
-vso it’ll print out everything it does. During testing--dry-runcan be a lifesaver.One big caveat, make sure you read the manual on the difference between
rclone copyandrclone sync, make sure you understand both before choosing on one of them.I’m purposely leaving out my script because I think if you’re getting started in scripting you should start small. There’s no big script that will apply to every system. Just try copying one directory to backblaze first on your workstation, then try another, try adding those to a script and running them as a script. Mine really did start as
It’ll just grow with time as you add more to it, add caveats, add rules, etc. Our labs are evolving for sure, there’s no silver bullet answer. Good luck, good testing, and ping back here if you have any other questions!
My fault! This is exactly what I was going to do as I’m unfamiliar with rclone. My comment about the script was to see how to do checks to make sure the remote is actually available before trying, how to make sure the job ran successfully, how to send a notification upon success/failure, etc.
I’m pretty novice at bash but I know other languages very well. Concepts apply more or less the same across languages, the only thing that changes in most cases is syntax.
Anyways, good write up and I appreciate your feedback.
I’m still working on the amount changed, making sure I’m happy with it before it’s completely automated, but I’m working on parsing the results of
summaryand--dry-runon a precheck, so if oh, 10% of the files would be changed or more than cancel it, send me a notification, and for that I’d run it manually myself. Still fine tuning and not quite happy with it yet.My first iteration was to cherry pick a few “key” files that would be randomly around my file system. Things that I will probably never ever change. This has been proven to work for a long time and honestly has saved my ass because I did accidently wipe out a few files once and this verified that my backups wouldn’t run until I fixed it. It’s a bit dumb but it did the trick for me:
Like I said, don’t judge too much, but for me it’s a “poor man’s ransomware checker”. If any of those have been modified, do not perform a backup, notify me, and shutdown.
The script I’m building is coming up nicely. I have it checking for error codes, logging to a file and sending discord notifications. Next step is to finish up all the rclone copy’s and syncs I want to do and then trying to encrypt the data client side before uploading to cloud. I’ve seen mention of crypt but can’t find it in the official rclone docs (yet).
crypt is pretty simple honestly, go through prebuild but it’s essentially just sitting on top of another remote. So I usually use the convention
{remotename}-crypt. It’ll simply rename and encrypt anything and then pass it into the remove you already set up. A good way to test it is once you’ve copied some stuff over with the crypt set up try to mount it usingrclone mountand try to read some of your files, it should be fine.Don’t forget to back up (securely, and obviously not on your encrypted share) your rclone config! if those keys are lost than so is your data.
I don’t need the whole rclone.conf file right? Just the authentication stuff for the underlying remote, the password for crypt and the salt for crypt. If I have that shouldn’t I be able to recreate the access and subsequently the decryption key to be used on some other machine?
I do, but it’s more of a don’t forget how it was made thing for me. I encrypted my file and put it on another cloud provider in case I lose it