In the latest version of Nephele, you can now create a WebDAV server that deduplicates files that you add to it.
I created this feature because every night at midnight, my Minecraft world that my friends and I play on gets backed up. Our world has grown to about 5 GB, but every night, the same files get backed up over and over. It’s a waste of space to store the same files again and again, but I want the ability to roll back our world to any day in the past.
So with this new feature of Nephele, I can upload the Minecraft backup and only the files that have changed will take up additional space. It’s like having infinite incremental backups that never need a full backup after the first time, and can be accessed instantly.
Nephele will only delete a file from the file storage once all copies that share the same file contents have been deleted, so unlike with most incremental backup solutions, you can delete previous backups easily and regain space.
Edit: So, I think my post is causing some confusion. I should make it clear that my use case is specific for me. This is a general purpose deduplicating file server. It will take any files you give it and deduplicate them in its storage. It’s not a backup system, and it’s not a versioning system. My use case is only one of many you can use a deduplicating file server for.
You might be thinking of incremental backups which also saves space but is not the same thing.
If you for example ran deduplication on a file server and a bunch of users uploaded the same files in multiple different directories, deduplication would remove all duplicate copies and just link them together. This has nothing to do with snapshots. btrfs might support deduplication but now this software does too. Your comment was completely unnecessary since not everything in the world can or should run btrfs
Their response further below clearly shows that they didn’t bother researching btrfs or a similar filesystem that can do this.
Now of course they can do with their time what they want, but I am also free to point out that there are other ways (that are maybe more established) to reach the same goal.
You are not logged in. However you can subscribe from another Fediverse account, for example Lemmy or Mastodon. To do this, paste the following into the search field of your instance: !selfhosted@lemmy.world
A place to share alternatives to popular online services that can be self-hosted without giving up privacy or locking you into a service you don’t control.
Rules:
Be civil: we’re here to support and learn from one another. Insults won’t be tolerated. Flame wars are frowned upon.
No spam posting.
Posts have to be centered around self-hosting. There are other communities for discussing hardware or home computing. If it’s not obvious why your post topic revolves around selfhosting, please include details to make it clear.
Don’t duplicate the full text of your blog or github here. Just post the link for folks to click.
Submission headline should match the article title (don’t cherry-pick information from the title to fit your agenda).
No trolling.
Resources:
selfh.st Newsletter and index of selfhosted software and apps
No, please research what deduplication is before commenting.
https://en.wikipedia.org/wiki/Data_deduplication
You might be thinking of incremental backups which also saves space but is not the same thing.
If you for example ran deduplication on a file server and a bunch of users uploaded the same files in multiple different directories, deduplication would remove all duplicate copies and just link them together. This has nothing to do with snapshots. btrfs might support deduplication but now this software does too. Your comment was completely unnecessary since not everything in the world can or should run btrfs
I am aware of the difference, but if you read the OP they are using it mainly for something that could also be done with btrfs.
Sure but maybe they don’t want to use btrfs. Ever thought about that?
Their response further below clearly shows that they didn’t bother researching btrfs or a similar filesystem that can do this.
Now of course they can do with their time what they want, but I am also free to point out that there are other ways (that are maybe more established) to reach the same goal.