So, I was having a phone conversation with my boss yesterday. The topic was a clustered filesystem that can hold huge amounts of data, and how we would best allow local users to access this data without yhe storage cluster becoming overloaded (because of various reasons, the I/O is relatively slow. This was built for quantity, not speed).

Rights now there’s an SMB share, and we’re looking at replacing that so that we can have better control over the data throughput. My suggestion is to simply spin up an FTP server.

Then my boss asks: “I’m just curious, but would rsync or NFS work as a protocol instead?”

Well, it’s a valid question, so the only thing I could do was reply with the honest answer as to why I chose FTP. Paraphrased and translated:

“Because some 20 years ago my then username carried a lot of recognition in certain communities revolving around software and media distribution, whose rights holders would not necessarily approve of said distribution. We used FTP, because when you’re on an ADSL from 2002, you want to have as much fine control as you can to make sure your internet connection doesn’t get flooded with requests. One connection at a time, and only one file at a time, which would be ideal in our particular case.”

The response I got was a chuckle and that he couldn’t think of a better endorsement of FTP as a preferred transfer protocol.

So there you have it - My career revolves a lot of skills that I picked up whole sailing the high seas. And coincidentally, my career now also involves literally sailing the high seas as these storage clusters are used on survey ships.

Lol @ “some 20 years ago … ADSL from 2002”. Thanks for making me feel old!

Horses were domesticated some 6000 years ago. I feel so old!

deadcatbounce
link
fedilink
English
710d

Ceph?

@neidu2@feddit.nl
creator
link
fedilink
English
1010d

No. Beegfs.

adr1an
link
fedilink
English
29d

Do check seaweedfs too! Haven’t tried it (yet) but their ‘erasure coding’ reads as super sophisticated to me ;)

I wonder how it compares to beegfs

mub
link
fedilink
English
4
edit-2
9d

Using FTP (I assume you mean SFTP) will buy you some performance, as would other protocols that are faster and requiring less compute than SMB.

I predict whatever solution you use will only buy you time. Usage is bound to increase so you’ll still hit the performance limits for the hardware platform at some point, unless you can constrain the simultaneous connections. File sizes will impact scalability a lot as well.

You can’t guess this one. You need to test.

tl;dr - I suspect you can’t win.

@onlinepersona@programming.dev
link
fedilink
English
1
edit-2
9d

Was IPFS considered? I’ve tried it myself but it seems like an unstable product and I’m not sure if it’s living up to its promise…

Anti Commercial-AI license

@neidu2@feddit.nl
creator
link
fedilink
English
29d

Unusable in our case

Piracy: ꜱᴀɪʟ ᴛʜᴇ ʜɪɢʜ ꜱᴇᴀꜱ
!piracy@lemmy.dbzer0.com
Create a post
⚓ Dedicated to the discussion of digital piracy, including ethical problems and legal advancements.

Rules • Full Version

1. Posts must be related to the discussion of digital piracy

2. Don’t request invites, trade, sell, or self-promote

3. Don’t request or link to specific pirated titles, including DMs

4. Don’t submit low-quality posts, be entitled, or harass others



Loot, Pillage, & Plunder

📜 c/Piracy Wiki (Community Edition):


💰 Please help cover server costs.

Ko-Fi Liberapay
Ko-fi Liberapay

  • 1 user online
  • 106 users / day
  • 270 users / week
  • 1K users / month
  • 3.5K users / 6 months
  • 1 subscriber
  • 3.4K Posts
  • 82.2K Comments
  • Modlog