Say you decide to self-host a Lemmy instance. When you create that instance, do you immediately need to download and store all the data that has ever been posted to all federated Lemmy instances? Or perhaps you only need to download and store everything that is posted to the federated Lemmy instances from that point forward? Or better yet, do you only store what the users on that instance do (i.e. their posts, and posts to the communities hosted on that instance)?
Correct. All also includes communities fetched but not subscribed to, however these are more like stubs. They are in your database but not being updated with activity since no one is subscribed. At least that’s my understanding.
Yes this was unclear to me until I set up my own instance is may be a big deal breaker to the average user. If you can’t view “All” and really see all communities on all federated instances then there is no ability to surf Lemmy. At the very least it should have all communities of instances from either a list or user subscriptions.
edit: if you can share your script that would be amazing, I was thinking of having to do the same thing…
I’m not really sure how this is supposed to work long-term, then. I can’t imagine anyone wants to be on an instance with only a fraction of the content available. It makes perfect sense when subscribing, but surfing All loses its appeal. I understand the challenges, but I hope there’s a creative solution at some point. It seems like folks will gravitate to the instances with the most stability and users.
I think you’re right. People will gravitate to the most stable large instances because their “All” will be as close to 100% as possible without doing anything special. I wrote a script to seed instances and update subscriptions, but it uses a single account that is subscribed to everything so that other users can see everything. That’s not something that would normally happen. Maybe that needs to be part of the base software?
Knowing that instances only pull posts/comments that occur after the first subscription, it will become less and less viable to choose a small instance if Lemmy doesn’t add the option of adjustable pull settings.
I don’t suppose your script is published anywhere? My comment adjacent to yours mentions how something like Mastodon’s Relay system would really help solve this issue, and it sounds like what you’ve made is probably the closest thing we’d have to a relay system for a while (given the core devs being super super busy with the existing issues).
On a side note, I wish there was also a way to set the homepage of an instance to “All” as well (which can be done user-side, but not globally), my instance only has a meta-community for announcements, so I can imagine that it just looks like an absolute ghost town to anyone who stops by.
That‘s so helpful, thank you for sharing it with us! One question: If I want to update my known communities in a month or so, can I just rerun the script or will that cause issues?
You should be able to rerun it anytime. It only gets stuff that doesn’t exist on your instance. That’s how it was designed. It is dependent on browse.feddit.de however. :(
It’d be really cool if we had something similar to Mastodon’s “Relays” where you basically subscribe to a firehose of posts from everyone whose on a server connected to that relay (they show up in the “Federated Timeline”).
I don’t know exactly how this would work for Lemmy, but it seems like if we had a system like this it could really help tackle this issue.
Would you say it makes sense to have accounts on the 2-3 instances that you’re most interested in rather than 2 account and being dependent on federation?
There’s no general answer, it depends on your personal preference.
If you want to have most content available, register on an instance which has an according policy; which federates with anybody and is federated by everybody (both directions can make a difference).
The downside however is, this also opens the door to all sorts of bad actors, including bots and spam.
So I personally tried to strike a balance and am so far quite happy on lemm.ee.
This tool is pretty handy to make informed decisions: https://fba.ryona.agency/ It allows you to check federation status both ways.
You are not logged in. However you can subscribe from another Fediverse account, for example Lemmy or Mastodon. To do this, paste the following into the search field of your instance: !selfhosted@lemmy.world
A place to share alternatives to popular online services that can be self-hosted without giving up privacy or locking you into a service you don’t control.
Rules:
Be civil: we’re here to support and learn from one another. Insults won’t be tolerated. Flame wars are frowned upon.
No spam posting.
Posts have to be centered around self-hosting. There are other communities for discussing hardware or home computing. If it’s not obvious why your post topic revolves around selfhosting, please include details to make it clear.
Don’t duplicate the full text of your blog or github here. Just post the link for folks to click.
Submission headline should match the article title (don’t cherry-pick information from the title to fit your agenda).
No trolling.
Resources:
selfh.st Newsletter and index of selfhosted software and apps
That is exactly what that means and it’s frustrating to say the least, because it’s not clear that’s what’s happening.
Just so I am understanding the feeds…
Subscribed - just the stuff you are subscribed to
Local - just the stuff in your instance
All - the stuff you subscribe to, the stuff in your instance, and stuff that people in your instance follow from other instances
That correct?
Correct. All also includes communities fetched but not subscribed to, however these are more like stubs. They are in your database but not being updated with activity since no one is subscribed. At least that’s my understanding.
Thanks for the info!
Yes this was unclear to me until I set up my own instance is may be a big deal breaker to the average user. If you can’t view “All” and really see all communities on all federated instances then there is no ability to surf Lemmy. At the very least it should have all communities of instances from either a list or user subscriptions.
edit: if you can share your script that would be amazing, I was thinking of having to do the same thing…
I’m not really sure how this is supposed to work long-term, then. I can’t imagine anyone wants to be on an instance with only a fraction of the content available. It makes perfect sense when subscribing, but surfing All loses its appeal. I understand the challenges, but I hope there’s a creative solution at some point. It seems like folks will gravitate to the instances with the most stability and users.
I think you’re right. People will gravitate to the most stable large instances because their “All” will be as close to 100% as possible without doing anything special. I wrote a script to seed instances and update subscriptions, but it uses a single account that is subscribed to everything so that other users can see everything. That’s not something that would normally happen. Maybe that needs to be part of the base software?
Have you got a link to that script? I want to seed my local private instance!
https://github.com/jheidecker/lemmony
Knowing that instances only pull posts/comments that occur after the first subscription, it will become less and less viable to choose a small instance if Lemmy doesn’t add the option of adjustable pull settings.
I don’t suppose your script is published anywhere? My comment adjacent to yours mentions how something like Mastodon’s Relay system would really help solve this issue, and it sounds like what you’ve made is probably the closest thing we’d have to a relay system for a while (given the core devs being super super busy with the existing issues).
On a side note, I wish there was also a way to set the homepage of an instance to “All” as well (which can be done user-side, but not globally), my instance only has a meta-community for announcements, so I can imagine that it just looks like an absolute ghost town to anyone who stops by.
https://github.com/jheidecker/lemmony
That‘s so helpful, thank you for sharing it with us! One question: If I want to update my known communities in a month or so, can I just rerun the script or will that cause issues?
You should be able to rerun it anytime. It only gets stuff that doesn’t exist on your instance. That’s how it was designed. It is dependent on browse.feddit.de however. :(
That’s cool, thanks i’ll check it out. I also found https://github.com/Fmstrat/lcs
It’d be really cool if we had something similar to Mastodon’s “Relays” where you basically subscribe to a firehose of posts from everyone whose on a server connected to that relay (they show up in the “Federated Timeline”).
I don’t know exactly how this would work for Lemmy, but it seems like if we had a system like this it could really help tackle this issue.
I think there is already some software that fetches content but it’s early in development.
Would you say it makes sense to have accounts on the 2-3 instances that you’re most interested in rather than 2 account and being dependent on federation?
There’s no general answer, it depends on your personal preference.
If you want to have most content available, register on an instance which has an according policy; which federates with anybody and is federated by everybody (both directions can make a difference).
The downside however is, this also opens the door to all sorts of bad actors, including bots and spam.
So I personally tried to strike a balance and am so far quite happy on lemm.ee.
This tool is pretty handy to make informed decisions: https://fba.ryona.agency/ It allows you to check federation status both ways.
Thanks for that link. Really interesting.