Id like to put my lab servers to work archiving US federal data thats likely to get pulled - climate and biomed data seems mostly likely. The most obvious strategy to me seems like setting up mirror torrents on academictorrents. Anyone compiling a list of at-risk data yet?
EDIT: Added script. Note that the script doesn’t include archiving to
archivebox, since its API isn’t available in stable verison yet. You can add a
function depending on your setup. Personally, I am depending on Caddy and
docker, so I am using caddy module [1] to execute commands with this in my Caddyfile:
Thank you for the warning. You are correct. It’s prune to command injection. I will validate the URL before executing it. This shoud suffice until archivebox’s rest API is available in stable.
I am using archivebox, it is pretty straight-forward to self-host and use.
However, it is very difficult to archive most news sites with it and many other sites as well. Most cookie etc pop ups on a site will render the archived page unusable and often archiving won’t work at all because some bot protection (Cloudflare etc.) will kick-in when archivebox tries to access a site.
If anyone else has more success using it, please let me know if I am doing something wrong…
Monolith has the same problem here. I think the best resolution might be some sort of browser-plugin based solution where you could say “archive this” and have it push the result somewhere.
I wonder if I could combine a dumb plugin with Monolith to do that… A weekend project perhaps.
archive.org is hosted in the US and could end up being a valid target. It doesn’t strike me as being a very good place to securely store anything nowadays. I’d consider anything hosted in the US to be out.
In that they’re a single organization, yes, but I’m a single person with significantly fewer resources. Non-availability is a significantly higher risk for things I host personally.
There was the attack on the Internet archive recently, are there any good options out there to help mirror some of the data or otherwise provide redundancy?
I guess they back either other up. Like archive.is is able to take archives from archive.org but the saved page reflects the original URL and the original archiving time from the wayback machine (though it also notes the URL used from wayback itself plus the time they got archived it from wayback).
For myself: Wayback It saves link to multiple different web archives and gives me pdf and warc files.
For others: Archive team have a few active projects to save at risk data and there is IRC channel in which people can suggest adding other websites for saving. They also have wiki with explanations how people can help.
I archive youtube videos that I like with TubeArchivist, I have a playlist for random videos i’d like to keep, and also subscribe to some of my favourite creator so I can keeptheir videos, even when I’m offline
Seems nice, but you need an external Player to watch the content, which can be goof for some people, but I like the webUI of TubeArchivist (even though it can be enhanced for sure)
Monolith can be particularly handy for this. I used it in a recent project to archive the outgoing links from my own site. Coincidentally, if anyone is interested in that, it’s called django-cool-urls.
You are not logged in. However you can subscribe from another Fediverse account, for example Lemmy or Mastodon. To do this, paste the following into the search field of your instance: !selfhosted@lemmy.world
A place to share alternatives to popular online services that can be self-hosted without giving up privacy or locking you into a service you don’t control.
Rules:
Be civil: we’re here to support and learn from one another. Insults won’t be tolerated. Flame wars are frowned upon.
No spam posting.
Posts have to be centered around self-hosting. There are other communities for discussing hardware or home computing. If it’s not obvious why your post topic revolves around selfhosting, please include details to make it clear.
Don’t duplicate the full text of your blog or github here. Just post the link for folks to click.
Submission headline should match the article title (don’t cherry-pick information from the title to fit your agenda).
No trolling.
Resources:
selfh.st Newsletter and index of selfhosted software and apps
NOAA is at risk I think.
Everything is at risk.
I have a script that archives to:
I used to solely depend on archive.org, but after the recent attacks, I expanded my options.
Script: https://gist.github.com/YasserKa/9a02bc50e75e7239f6f0c8f04fe4cfb1
EDIT: Added script. Note that the script doesn’t include archiving to archivebox, since its API isn’t available in stable verison yet. You can add a function depending on your setup. Personally, I am depending on Caddy and docker, so I am using caddy module [1] to execute commands with this in my
Caddyfile:[1] https://github.com/abiosoft/caddy-exec
isn’t this prone to a
or something similar at the end of the URL?
if you can
docker exec, you have a lot of privileges already, so be sure to make sure this is not a dangerThank you for the warning. You are correct. It’s prune to command injection. I will validate the URL before executing it. This shoud suffice until archivebox’s rest API is available in stable.
Would you be willing to share it?
Sure.
I hope you are also donating to the projects for uploading multiple copies to different services.
One option that I’ve heard of in the past
https://archivebox.io/
This seems pretty cool. I might actually host this.
I am using archivebox, it is pretty straight-forward to self-host and use.
However, it is very difficult to archive most news sites with it and many other sites as well. Most cookie etc pop ups on a site will render the archived page unusable and often archiving won’t work at all because some bot protection (Cloudflare etc.) will kick-in when archivebox tries to access a site.
If anyone else has more success using it, please let me know if I am doing something wrong…
Monolith has the same problem here. I think the best resolution might be some sort of browser-plugin based solution where you could say “archive this” and have it push the result somewhere.
I wonder if I could combine a dumb plugin with Monolith to do that… A weekend project perhaps.
That looks useful, I might host that. Does anyone have an RSS feed of at risk data?
Going to check that out because…yeah. Just gotta figure out what and where to archive.
Eyy, I want that!
https://wiki.archiveteam.org/
they have an automatic VM that dowloads stuff in distributed manner and uploads to archive.org
archive.org is hosted in the US and could end up being a valid target. It doesn’t strike me as being a very good place to securely store anything nowadays. I’d consider anything hosted in the US to be out.
Depends on the threat model.
NOAA and others gets underfunded/change of menagement and need to close down open access to stuff.
or
Data becomes illegal to possess and feds start knocking on Web Archive doors.
or
Web archive will do something stupid and will get sued/DDOSed
In only one very unlikely scenario it won’t be availble due to recent events. But still redundancy would be good regardless of recent stuff.
I don’t self-host it, I just use archive.org. That makes it available to others too.
It’s a single point of failure though.
In that they’re a single organization, yes, but I’m a single person with significantly fewer resources. Non-availability is a significantly higher risk for things I host personally.
There was the attack on the Internet archive recently, are there any good options out there to help mirror some of the data or otherwise provide redundancy?
Yes. This isn’t something you want your own machines to be doing if something else is already doing it.
But then who backs up the backups?
Realize how how much they are supporting and storing.
Come back to the comments after.
I guess they back either other up. Like archive.is is able to take archives from archive.org but the saved page reflects the original URL and the original archiving time from the wayback machine (though it also notes the URL used from wayback itself plus the time they got archived it from wayback).
Your argument is that a single backup is sufficient? I disagree, and I think that so would most in the selfhosted and datahoarder communities.
shaarli bookmarks + hecat (
shaarli_apiimporter +download_media/archive_webpagesprocessors +html_tableexporter for the HTML index)I use M-Discs to long term archival.
I heard news recently that some companies recently started shipping non-m disks labelled as m-disks. You may want to have a look
For myself: Wayback It saves link to multiple different web archives and gives me pdf and warc files.
For others: Archive team have a few active projects to save at risk data and there is IRC channel in which people can suggest adding other websites for saving. They also have wiki with explanations how people can help.
Linkding/Linkwarden
I archive youtube videos that I like with TubeArchivist, I have a playlist for random videos i’d like to keep, and also subscribe to some of my favourite creator so I can keeptheir videos, even when I’m offline
I’ll add pinchflat as an alternative with the same aim.
Seems nice, but you need an external Player to watch the content, which can be goof for some people, but I like the webUI of TubeArchivist (even though it can be enhanced for sure)
You can actually play from the UI too, but it’s not particularly nice to use (or intended to be used that way).
Flash drives and periodic transfers.
Monolith can be particularly handy for this. I used it in a recent project to archive the outgoing links from my own site. Coincidentally, if anyone is interested in that, it’s called django-cool-urls.