I intend to upload an archive of a website I scraped for the last few weeks. The torrent is about 4 TB large and contains 400000 folders which in turn contain 40 .jpg’s on average (plus some metadata).
Should I just create the torrent with the files as they are right now, or should I put the individual folders in archives (or maybe even the entire torrent?)?
Yes, you can, I would say that it would be good to deduplicate files first.
Also put a txt file with locations and file hashes in torrent, it could be usable if some valuable files were lost in the future due to seeders being inactive and torrent becoming dead, to make .torrent file smaller you would choose a bigger piece size, but in this case users would have to waste additional traffic if they are downloading only files they choose.
It would be nice then to have BEP-47 support for file alignment, but this comes with many padding files locations in torrent’s metadata which makes it bigger.
If you have a big nested structure of folders, then every file there would have something like this info in metadata:
folder1/folder2/folder3/file.jpg
All of these problems could be solved, if you’re choosing format to BitTorrent v2 only while creating the torrent, but keep in mind that it won’t work with Transmissions and uTorrents. qBittorrents are working.
You could choose a hybrid, but it makes torrent size a lot bigger.
So overall, bestly you would choose v2 or hybrid (with hybrids sacrificing torrent’s size greatly, but providing your users with file hashes, deduplication and reduced bandwidth), or make at least a v1 torrent, optionally in a client that supports BEP-47 after deduplication and adding .txt
Jpeg is already compressed, so compressing them again won’t do anything but make it impossible for people to selectively download just the image/folder they desire.
Metadata on the other hand sounds like text files, which compress very well. Wether the space savings is worth it is hard to answer without more info. I’d personally lean towards not archiving it.
wow, dedication bud, few weeks to web rip. well done. true pirate here. tx 4 ur service. I’d love to know which site. 😛 feel free to DM me. I’m curious.
Makes sense. What about individual folders? I’m concerned that the massive amount of small files could slow down the download quite a bit. Unfortunately, I’m not too familiar with the BitTorrent protocol, so I don’t know if this would have an impact.
You are not logged in. However you can subscribe from another Fediverse account, for example Lemmy or Mastodon. To do this, paste the following into the search field of your instance: !piracy@lemmy.dbzer0.com
⚓ Dedicated to the discussion of digital piracy, including ethical problems and legal advancements.
Yes, you can, I would say that it would be good to deduplicate files first.
Also put a txt file with locations and file hashes in torrent, it could be usable if some valuable files were lost in the future due to seeders being inactive and torrent becoming dead, to make .torrent file smaller you would choose a bigger piece size, but in this case users would have to waste additional traffic if they are downloading only files they choose.
It would be nice then to have BEP-47 support for file alignment, but this comes with many padding files locations in torrent’s metadata which makes it bigger.
If you have a big nested structure of folders, then every file there would have something like this info in metadata: folder1/folder2/folder3/file.jpg
All of these problems could be solved, if you’re choosing format to BitTorrent v2 only while creating the torrent, but keep in mind that it won’t work with Transmissions and uTorrents. qBittorrents are working.
You could choose a hybrid, but it makes torrent size a lot bigger.
So overall, bestly you would choose v2 or hybrid (with hybrids sacrificing torrent’s size greatly, but providing your users with file hashes, deduplication and reduced bandwidth), or make at least a v1 torrent, optionally in a client that supports BEP-47 after deduplication and adding .txt
Only if you can achieve significant compression. 15% or more.
I’d put it all in a single archive.
Why not both?
Jpeg is already compressed, so compressing them again won’t do anything but make it impossible for people to selectively download just the image/folder they desire.
Metadata on the other hand sounds like text files, which compress very well. Wether the space savings is worth it is hard to answer without more info. I’d personally lean towards not archiving it.
wow, dedication bud, few weeks to web rip. well done. true pirate here. tx 4 ur service. I’d love to know which site. 😛 feel free to DM me. I’m curious.
NGL this style of writing looks like a fed trying hard to look like a l33t h4xer arrrgh.
“a pirate is free!”
CIA is just an information gathering agency with no legal power, the FBI has the authority to make arrests.
Who even cares about CIA and FBI?
Their perception is too exagerated.
Leave them as is. I hate having to deal with archives inside torrents unless there’s a very good reason for them.
People will want to download chicks or parts of the torrent, you should leave them separate.
You always choose to only download the parts of the torrent you want, no need to keep them seperate.
Makes sense. What about individual folders? I’m concerned that the massive amount of small files could slow down the download quite a bit. Unfortunately, I’m not too familiar with the BitTorrent protocol, so I don’t know if this would have an impact.
Depends on final size of the folder zips. 20-30mb? Sure. 200-300? Nah.
Small files don’t impact BitTorrent transfers. They’re sent in pieces of a fixed size, and those pieces can contain multiple files.