In the latest version of Nephele, you can now create a WebDAV server that deduplicates files that you add to it.
I created this feature because every night at midnight, my Minecraft world that my friends and I play on gets backed up. Our world has grown to about 5 GB, but every night, the same files get backed up over and over. It’s a waste of space to store the same files again and again, but I want the ability to roll back our world to any day in the past.
So with this new feature of Nephele, I can upload the Minecraft backup and only the files that have changed will take up additional space. It’s like having infinite incremental backups that never need a full backup after the first time, and can be accessed instantly.
Nephele will only delete a file from the file storage once all copies that share the same file contents have been deleted, so unlike with most incremental backup solutions, you can delete previous backups easily and regain space.
Edit: So, I think my post is causing some confusion. I should make it clear that my use case is specific for me. This is a general purpose deduplicating file server. It will take any files you give it and deduplicate them in its storage. It’s not a backup system, and it’s not a versioning system. My use case is only one of many you can use a deduplicating file server for.
Snapshots are accessible in read only mode without reverting to it, snapshots can be easily configured to be taken automatically with a simple cron job, btrfs allows full control of snapshots over SSH, and you can easily copy a snapshot to another btrfs filesystem on the same or remote server.
Also btrfs follows the Unix philosophy, so of course you will be using additional tools with it, but btrbk for example makes all of the above really easy with no additional tools needed.
Obviously there are differences, but serving WebDAV on top of a btrfs filesystem is very similar to what you have made.
It very much is not. Again, btrfs will only deduplicate data under certain circumstances, like if you copy a file to a new location. If I take a USB stick with an 8gb movie file on it and copy that to btrfs twice, it will take up 16gb on disk. If I copy it to btrfs once, then copy it from there to a new location, it will take up 8gb on disk. Btrfs does not deduplicate files, it deduplicates copies. I want something that deduplicates files.
If you run WebDAV on top of btrfs and try what I’m using it for, it literally will not deduplicate anything, because you’re always writing new files to it, not copying existing files.
Triggering a snapshot with a cron job doesn’t mean it’s automatic to btrfs. The action still happens only when triggered. Btrfs doesn’t take snapshots for you.
What good is management through SSH? I want a deduplicating file server, not a versioning file system I have to manage over SSH server. If I wanted versioning like that, I would just use git.
And again, adding tools on top of btrfs to recreate something similar to what I’ve made here does not mean I reinvented btrfs. Btrfs is a COW FS. I wrote a deduplicating file server. I honestly can’t believe you don’t see the difference here. Like, are you trolling?
I feel like you misinterpreted my post to mean that my use case is the only thing you could use my server for, and you’re just running with it, even though I’ve told you multiple times, I wrote a deduplicating file server, not an incremental backup system, and not a versioning system. The fact that I’m using it for incremental backups is inconsequential to what it actually does. It deduplicates files and serves them from WebDAV. AFAIK, there’s no other open source server that does that.
You are not logged in. However you can subscribe from another Fediverse account, for example Lemmy or Mastodon. To do this, paste the following into the search field of your instance: !selfhosted@lemmy.world
A place to share alternatives to popular online services that can be self-hosted without giving up privacy or locking you into a service you don’t control.
Rules:
Be civil: we’re here to support and learn from one another. Insults won’t be tolerated. Flame wars are frowned upon.
No spam posting.
Posts have to be centered around self-hosting. There are other communities for discussing hardware or home computing. If it’s not obvious why your post topic revolves around selfhosting, please include details to make it clear.
Don’t duplicate the full text of your blog or github here. Just post the link for folks to click.
Submission headline should match the article title (don’t cherry-pick information from the title to fit your agenda).
No trolling.
Resources:
selfh.st Newsletter and index of selfhosted software and apps
Snapshots are accessible in read only mode without reverting to it, snapshots can be easily configured to be taken automatically with a simple cron job, btrfs allows full control of snapshots over SSH, and you can easily copy a snapshot to another btrfs filesystem on the same or remote server.
Also btrfs follows the Unix philosophy, so of course you will be using additional tools with it, but btrbk for example makes all of the above really easy with no additional tools needed.
Obviously there are differences, but serving WebDAV on top of a btrfs filesystem is very similar to what you have made.
It very much is not. Again, btrfs will only deduplicate data under certain circumstances, like if you copy a file to a new location. If I take a USB stick with an 8gb movie file on it and copy that to btrfs twice, it will take up 16gb on disk. If I copy it to btrfs once, then copy it from there to a new location, it will take up 8gb on disk. Btrfs does not deduplicate files, it deduplicates copies. I want something that deduplicates files.
If you run WebDAV on top of btrfs and try what I’m using it for, it literally will not deduplicate anything, because you’re always writing new files to it, not copying existing files.
Triggering a snapshot with a cron job doesn’t mean it’s automatic to btrfs. The action still happens only when triggered. Btrfs doesn’t take snapshots for you.
What good is management through SSH? I want a deduplicating file server, not a versioning file system I have to manage over SSH server. If I wanted versioning like that, I would just use git.
And again, adding tools on top of btrfs to recreate something similar to what I’ve made here does not mean I reinvented btrfs. Btrfs is a COW FS. I wrote a deduplicating file server. I honestly can’t believe you don’t see the difference here. Like, are you trolling?
I feel like you misinterpreted my post to mean that my use case is the only thing you could use my server for, and you’re just running with it, even though I’ve told you multiple times, I wrote a deduplicating file server, not an incremental backup system, and not a versioning system. The fact that I’m using it for incremental backups is inconsequential to what it actually does. It deduplicates files and serves them from WebDAV. AFAIK, there’s no other open source server that does that.