I’m currently debating on how to manage files on my servers. I have a jellyfin and a minecraft server on which I need to add, remove or download files quite often. I don’t really want to use scp for everything, so I was wondering what everyone uses.
Edit: I’m looking for a gui solution, but a somewhat automated process of backups etc. is also nice
Edit 2: For anyone wondering what my final solution was: I am currently using a wireguard vpn on a raspberry pi to access my servers. I use Xpipe as a gui interface to transfer my files. I also just use tmux and ssh to execute commands and run services.
For automatic stuff I use synching to keep my home an documents in sync.
I don’t normally use samba remotely but I have a wire guard VPN that I connect to if I’m not at home and that gives me access to samba or any other “local” services I might need when away
pterodactyl looks really neat, will definitely look into that. I have a manual system for my media library, so I want to add the directories with artwork and movies manually to the directory which jellyfin reads.
I remember some years ago scp had a big issue, can’t recall what, though. But that made me have a look at rsync, and I’ve been using that ever since. Flags are a bit atteocious, but I’ve aliases rsync -avz status=progress to copy and it’s been happy days. One other benefit - incremental copy. Helps in cases where a copy procedure had been stopped for whatever reason.
I wouldn’t really recommend NFS unless you need to remote mount as a “true filesystem” with full support for things like sockets, locking and other UNIX filesystem features or you need top performance. It is so difficult to do authentication and UID mapping that it typically isn’t worth it for simpler use cases like “add, remove or download files”.
scp can be slow with large numbers of small files. rsync is much better at that and can do differential transfers if you need that. Since rsync can also run over SSH it can be very easy to just use it as a default.
ftp ans sftp are completely different things. ftp is a very old protocol and can be run encrypted as ftps. sftp runs over SSH and is pretty common to use.
Naming is truly one of the hardest problems in computer science.
what! thats so dumb xD. so ftps stands for file transfer protocol secure, and sftp stands for ssh file transfer protocol? we have reached recursive acronyms.
Yup super confusing. The way I remember it is that https is http + SSL so ftps is ftp + SSL. The weird one sftp is an entirely different protocol that uses SSH to transfer files.
Okay this has made it so much worse. i figured that the s in https and ftps stands for secure like a reasonable person. of course it doesnt xD. you would think that this is at least consistent but alas. guess what s in ssh doesnt stand for :)
I don’t know if the s is actually “SSL” or “secure” but the point is that the are the same protocol, running over an encryption layer. So adding an s suffix is running the same protocol over some encrypted transport. You see this s suffix for lots of things like irc/ircs and dav/davs.
This is different to sftp which isn’t related to ftp at all other than they are both protocols that transfer files.
Thanks😅. I wasn’t aware, and until today thought that sftp is just like “the secure version of ftp” (never heard of ftps), and not different things at all. I have still a lot to learn in this field.
Edit:I deleted the comentarios because i am unsure on what i am using right now and i don’t want to confuse people. I’ll check it when i have some free time
Unless you went out of your way to set up FTP and get a TLS certificate I would put my money on you using SFTP which uses SSH for authentication and transport security. It doesn’t require anything to set up other than TOFU server keys and a client key or password for authentication.
Which is probably the right thing to use. Really you shouldn’t be using FTP anymore. Probably you just want HTTP for public data and SFTP for private authenticated data.
Rsync is smart enough to be able to only send files you didnt’ have previously like a regular mv command.
Or you can sync the whole directory and have it also delete removed files.
Edit: If you wan’t something to automatically “sync/backup” the data, why don’t you try and use syncthing? P2P syncing solution that might be exactly what you need in your hse case.
When it comes to media like my music library and comics for example I just use rsync to move new ones to the server every time I collect some new (I have about 60gb of it)
No, I only have new files on my local machine, and as I collect new stuff I move them to my server with rsync and just remove it onthe local machine.
Rsync is more than smart enough to, if you keep same folder structure to easilly move the files to the server without you having to manually copy other each file.
It’s not very sexy, but I just use SFTP and filezilla. Filezilla is available for all the major operating systems, and an ssh server is basically required anyways.
rsync for backups. SMB / NFS for things I need on a daily basis. SSH FTP (use whatever client you want) for things I need rarely but don’t wanna bother with SCP and I especially don’t bother ever setting up vsftpd for ‘real’ FTP.
Nemo/most file explorers have a remote server option. For nemo, “Files > Connect to server”. Once configured, you should be able to view your server files from your file explorer and uploading files should be as easy as drag and drop. I’m pretty sure nautilis has something similar too!
It’s not the best for security because if someone gets your computer, they can access your server through the file explorer. But it’s fast and convenient if you are already using your file explorer.
Generally you can even just type it into the location bar, no need to find a specific dialog. For example if I type CTRL+l then sftp://myserver.local into Nautilus it pops up a username+password dialog (or just logs in using my keys).
You are not logged in. However you can subscribe from another Fediverse account, for example Lemmy or Mastodon. To do this, paste the following into the search field of your instance: !selfhosted@lemmy.world
A place to share alternatives to popular online services that can be self-hosted without giving up privacy or locking you into a service you don’t control.
Rules:
Be civil: we’re here to support and learn from one another. Insults won’t be tolerated. Flame wars are frowned upon.
No spam posting.
Posts have to be centered around self-hosting. There are other communities for discussing hardware or home computing. If it’s not obvious why your post topic revolves around selfhosting, please include details to make it clear.
Don’t duplicate the full text of your blog or github here. Just post the link for folks to click.
Submission headline should match the article title (don’t cherry-pick information from the title to fit your agenda).
No trolling.
Resources:
selfh.st Newsletter and index of selfhosted software and apps
I use samba for normal usage and rsync for backups.
Do you automate your backups in some way? And can you also use samba remotely
For automatic stuff I use synching to keep my home an documents in sync.
I don’t normally use samba remotely but I have a wire guard VPN that I connect to if I’m not at home and that gives me access to samba or any other “local” services I might need when away
Plex/Jellyfin is automatically managed by Sonarr/Radarr so I don’t touch those.
But for game servers I use Pterodactyl which has a nice WebUI to manage the server and its files, and has automated backups.
pterodactyl looks really neat, will definitely look into that. I have a manual system for my media library, so I want to add the directories with artwork and movies manually to the directory which jellyfin reads.
https://github.com/filebrowser/filebrowser might be what you want for that, just a basic web based file management tool.
But you could also just use SMB and access the shares directly from file explorer.
SCP?
sorry, should’ve clarified: secure copy, it’s an ssh kind of way of copying files to a server
Yeah, I had to look it up but the above was the first search result 🙂
docker, persistent shared storage
Being the weirdo I am I have 2 things I use.
WinSCP to get that classic FTP look of file transfers and VS Code when working on smaller compose files.
NFS comes to mind, naturally.
I remember some years ago scp had a big issue, can’t recall what, though. But that made me have a look at rsync, and I’ve been using that ever since. Flags are a bit atteocious, but I’ve aliases
rsync -avz status=progresstocopyand it’s been happy days. One other benefit - incremental copy. Helps in cases where a copy procedure had been stopped for whatever reason.I wouldn’t really recommend NFS unless you need to remote mount as a “true filesystem” with full support for things like sockets, locking and other UNIX filesystem features or you need top performance. It is so difficult to do authentication and UID mapping that it typically isn’t worth it for simpler use cases like “add, remove or download files”.
scpcan be slow with large numbers of small files.rsyncis much better at that and can do differential transfers if you need that. Sincersynccan also run over SSH it can be very easy to just use it as a default.deleted by creator
ftpanssftpare completely different things.ftpis a very old protocol and can be run encrypted asftps.sftpruns over SSH and is pretty common to use.Naming is truly one of the hardest problems in computer science.
what! thats so dumb xD. so
ftpsstands forfile transfer protocol secure, andsftpstands forssh file transfer protocol? we have reached recursive acronyms.Yup super confusing. The way I remember it is that
httpsishttp+ SSL softpsisftp+ SSL. The weird onesftpis an entirely different protocol that uses SSH to transfer files.Okay this has made it so much worse. i figured that the
sinhttpsandftpsstands forsecurelike a reasonable person. of course it doesnt xD. you would think that this is at least consistent but alas. guess whatsinsshdoesnt stand for :)I don’t know if the
sis actually “SSL” or “secure” but the point is that the are the same protocol, running over an encryption layer. So adding anssuffix is running the same protocol over some encrypted transport. You see thisssuffix for lots of things likeirc/ircsanddav/davs.This is different to
sftpwhich isn’t related toftpat all other than they are both protocols that transfer files.Thanks😅. I wasn’t aware, and until today thought that sftp is just like “the secure version of ftp” (never heard of ftps), and not different things at all. I have still a lot to learn in this field.
Edit:I deleted the comentarios because i am unsure on what i am using right now and i don’t want to confuse people. I’ll check it when i have some free time
Unless you went out of your way to set up FTP and get a TLS certificate I would put my money on you using SFTP which uses SSH for authentication and transport security. It doesn’t require anything to set up other than TOFU server keys and a client key or password for authentication.
Which is probably the right thing to use. Really you shouldn’t be using FTP anymore. Probably you just want HTTP for public data and SFTP for private authenticated data.
I tend to use rsync 99% times.
how would you do that with a large media library?
I would just use rsync on the whole folder.
Rsync is smart enough to be able to only send files you didnt’ have previously like a regular mv command.
Or you can sync the whole directory and have it also delete removed files.
Edit: If you wan’t something to automatically “sync/backup” the data, why don’t you try and use syncthing? P2P syncing solution that might be exactly what you need in your hse case.
I’m now considering syncing my minecraft world with syncthing, I already use it for some things but don’t know why I didn’t think of doing that.
On the other hand, if I have a 100+ gb media library, it seems kinda over the top to also have it fully copied on my local machine. Do you do this?
When it comes to media like my music library and comics for example I just use rsync to move new ones to the server every time I collect some new (I have about 60gb of it)
so you basically have a copy of your media library on a local machine?
No, I only have new files on my local machine, and as I collect new stuff I move them to my server with rsync and just remove it onthe local machine.
Rsync is more than smart enough to, if you keep same folder structure to easilly move the files to the server without you having to manually copy other each file.
sounds good, do you have any docs on how to do that?
Sftp using the KDE file browser Dolphin. Keeps it simple for my monkey brain as if I’m accessing the drive locally.
It’s not very sexy, but I just use SFTP and filezilla. Filezilla is available for all the major operating systems, and an ssh server is basically required anyways.
NFS, SMB, and/or sshfs
Sshfs is also a good candidate if for whatever reason you want security / encryption
I mostly want to upload to a media server from my desktop, as that is where I download and manage the files before they go on the server
Use Filezilla and SFTP.
I’ll also take a look at this
rsync for backups. SMB / NFS for things I need on a daily basis. SSH FTP (use whatever client you want) for things I need rarely but don’t wanna bother with SCP and I especially don’t bother ever setting up vsftpd for ‘real’ FTP.
Nemo/most file explorers have a remote server option. For nemo, “Files > Connect to server”. Once configured, you should be able to view your server files from your file explorer and uploading files should be as easy as drag and drop. I’m pretty sure nautilis has something similar too!
It’s not the best for security because if someone gets your computer, they can access your server through the file explorer. But it’s fast and convenient if you are already using your file explorer.
Generally you can even just type it into the location bar, no need to find a specific dialog. For example if I type
CTRL+lthensftp://myserver.localinto Nautilus it pops up a username+password dialog (or just logs in using my keys).Wouldn’t FileBrowser work? Idk if you’re on docker or not though
I use rsync and SFTP to transfer files to a remote server and NFS over the LAN.