A better "All" browsing experience for small Lemmy instances - GitHub - jheidecker/lemmony: A better "All" browsing experience for small Lemmy instances
v.0.0.6
v0.0.4 - Per requests and concerns: Defaults changed and options added to prevent overloading servers, hitting rate-limiting, filtering to top x communities, etc!
So if I’m understanding this right, the bot account you create for this is the one subscribing to every community, so it’s known to the local system, right? As long as I’m not mixing up my main account and my bot account, there should be no observable change on my own account?
How is storage affected on this? If the bot account is subscribing to a number of communities across the fediverse, all that remote content is going to take up quite a bit of space, no?
So if I’m understanding this right, the bot account you create for this is the one subscribing to every community, so it’s known to the local system, right?
Yes
As long as I’m not mixing up my main account and my bot account, there should be no observable change on my own account?
Correct, I have it functioning this way and it works great.
How is storage affected on this? If the bot account is subscribing to a number of communities across the fediverse, all that remote content is going to take up quite a bit of space, no?
It does and it will continue to grow. This not not something the tool takes care of, not cleaning up anything old or stale. Space management and “unfollow” is on the roadmap! Currently I can only speak for myself and it is EVERYTHING and it is about 0.25 GB / day of database, and 6-10 GB / day of images.
And will 2FA be supported at any point?
Not on the roadmap. I don’t know how api calls in general work with 2fa since I have not tested or enabled it on my instance. :( Sorry.
EDIT: Changed database/pictures ratio after double checking actual numbers and not looking at used filesystem. :(
Hey, thanks for replying! Good to know about the subscription ownership. I only ran the discovery portion of the script just to dip my toe in the water, because I feared what would happen if I subscribed. That said, if I use the discovery flag, that’s just exposing the different communities to my instance, right? It’s not going to retrieve any remote content?
Good to see that storage management is roadmapped. I probably won’t invoke the Subscribe side of this for my bot account until then.
Understandable about 2FA. I’ll just be smart about my password usage and watch for any updates on that.
It retrieves the last 10 posts and adds the community reference to your local database. It is the same as putting “!community@instance.com” in the search box and clicking search. The retrieval happens whether you look at the results or not.
It’s not meaningful though. It allows one to optionally limit to the top-n communities by active-user count, but the default is still unlimited and the docs still do nothing to explain the abusive load that excessive subscription puts on instances that host communities.
My dude, I appreciate your spirit, but we’re not going to focus on your irrational fear of abuse. I’ll defend myself for being accused of any such thing, or for being irresponsible. This is intended to make things better, and there’s no evidence it’s doing anything other than that.
If you want to contribute, by all means, show us where there is a problem, other than in your imagination, and it will be seriously considered. Until then, your opinion is still valuable, but you are speaking with authority about something you know little of.
My dude… If you want to contribute, by all means, show us where there is a problem, other than in your imagination, and it will be seriously considered.
I had no idea we were doing condescending pet names, this is a fun game. My sweet summer child…
The defaults of this script encourage single-user instances admins to bump their sub count ~70x from something like 100 communities to something more like 7000 communities.
Users of this script actually literally don’t understand how federation works. They think they’re proxying through to the upstream instance while they browse rather than getting firehosed with the entire lemmyverse by they’re asleep.
It doesn’t take a rocket surgeon to figure out that global federation worker queues are not in great shape, or that a default that encourages single-user instance owners who have no idea what they’re doing to bump their sub count 70x isn’t helping the situation. If you think this is in my head I can’t help you. But I can help others understand that running this script with default settings is an awful and unnecessary idea.
Your argument does not gain validity by adding irrelevant verbosity:
Federation ain’t doing great.
The linked issue has nothing to do with this script or lemmony.
Federated replication load scales with the number of instances multiplied by the number of communities they subscribe to.
That’s a hasty generalization that you just made up.
Server counts are growing at ~10x per month.
That’s great! I hope they keep growing!
The defaults of this script encourage single-user instances admins to bump their sub count ~70x from something like 100 communities to something more like 7000 communities.
Nobody is encouraging anyone to do anything.
Users of this script actually literally don’t understand how federation works. They think they’re proxying through to the upstream instance while they browse rather than getting firehosed with the entire lemmyverse by they’re asleep.
That single user asked a question and got berated by a jerk.
It doesn’t take a rocket surgeon to figure out that global federation worker queues are not in great shape, or that a default that encourages single-user instance owners who have no idea what they’re doing to bump their sub count 70x isn’t helping the situation. If you think this is in my head I can’t help you. But I can help others understand that running this script with default settings is an awful and unnecessary idea.
You can help others understand what it is. That’s a great thing to do. It would be nice if you could do that without being a dick.
I think it depends on your instance. This version (same script, just updated) allows for more options IF your instance IS mass Overloaded, or you are scared it will be.
When I discovered, I felt bad for not checking. As for the load stuff. I intended and wanted to see All the things, and I don’t currently have resource problems for my instance. :) We’ll see how that fairs as things continue to grow!
A clean up would be nice, maybe something where if there are no subscribers, or local user actions on a community (votes, comments) after a while it starts to remove them.
The script gets all of the publicly federated communities and “makes them known” to your local instance and then subscribes to them. “All” should be populated with activity from around the Lemmyverse.
Doesn’t that significantly increase the load on your instance and, if many instances use it, all instances? This system isn’t designed with the idea that each instance receives everything from every other instance.
It might be better to run this on a single dedicated site which people can come to to browse. If you could learn where each user had their account, you could send their upvotes, downvotes and comments to that instance.
It increases load during execution. Afterward it’s not significant. My instance is heavily instrumented and monitored. The load this incurs subscribing to 24000 communities is less than adding a single, moderately active user to your instance.
It’s a huge miss if the intended design was to silo information.
What this provides, as far as I’m concerned, is essential to prevent centralization to a few instances.
Is there a better way to do it inherently in Lemmy itself? Probably, and I am excited to help with that!
Well that sounds quite reasonable then. It definitely answers a need for better discoverability of material on Lemmy. And it would be great if something like this could ultimately be integrated into Lemmy itself.
I think your idea is on the right track when thinking longer term and assuming the worst case in both design and admin behavior. :)
The whole network needs to be split into “active” and “archive.” New activity (or at the very least stubs to where new activity is happening) needs to be updated regardless of where it occurs without having to capture anything extra.
It increases load during execution. Afterward it’s not significant. My instance is heavily instrumented and monitored. The load this incurs subscribing to 24000 communities is less than adding a single, moderately active user to your instance.
It’s a huge miss if the intended design was to silo information.
What this provides, as far as I’m concerned, is essential to prevent centralization to a few instances.
Is there a better way to do it inherently in Lemmy itself? Probably, and I am excited to help with that!
Uhh… if your script is subbing to 24k remote communities, those will continue to grow from then on, unless you start purging communities at some point. After one user subscribes to a community, all new content gets indexed and stored on your instance. Pict-rs can cache images short term (and eventually clear them out), but Postgres will start growing very quickly and never slow down until it fills up disks.
Hosting an instance myself, I’m not amused, because if forces my instance to literally sync all content there is on the lemmyverse, drastically increasing traffic, storage use etc.
Please don’t force resource consumption beyond any rational usage!
People can defederate from an instance for any reason they want, but if I get what you’re trying to say: you think people should defederate from any instance that has a user that subscribes to all of their communities.
This is a cool idea. I was curious how subs work and it makes a lot more sense to me now that I’ve spent the time getting my instance setup and read a ton of docs. It feels weird trying to build your list and communities on your own account, while it also populates your instance’s All list. Feels like all of my interests are just out there flapping in the breeze lmao.
I actually wrote it with the flip side of your centralization argument in mind. If a community exists outside of the popular ones a user may never even know of its existence. Having more show up SHOULD be better to prevent centralization no? It requires the users to change their browsing behaviour but at least they don’t have gonsearching offsite.
You are not logged in. However you can subscribe from another Fediverse account, for example Lemmy or Mastodon. To do this, paste the following into the search field of your instance: !selfhosted@lemmy.world
A place to share alternatives to popular online services that can be self-hosted without giving up privacy or locking you into a service you don’t control.
Rules:
Be civil: we’re here to support and learn from one another. Insults won’t be tolerated. Flame wars are frowned upon.
No spam posting.
Posts have to be centered around self-hosting. There are other communities for discussing hardware or home computing. If it’s not obvious why your post topic revolves around selfhosting, please include details to make it clear.
Don’t duplicate the full text of your blog or github here. Just post the link for folks to click.
Submission headline should match the article title (don’t cherry-pick information from the title to fit your agenda).
For anyone who might know:
So if I’m understanding this right, the bot account you create for this is the one subscribing to every community, so it’s known to the local system, right? As long as I’m not mixing up my main account and my bot account, there should be no observable change on my own account?
How is storage affected on this? If the bot account is subscribing to a number of communities across the fediverse, all that remote content is going to take up quite a bit of space, no?
And will 2FA be supported at any point?
Yes
Correct, I have it functioning this way and it works great.
It does and it will continue to grow. This not not something the tool takes care of, not cleaning up anything old or stale. Space management and “unfollow” is on the roadmap! Currently I can only speak for myself and it is EVERYTHING and it is about 0.25 GB / day of database, and 6-10 GB / day of images.
Not on the roadmap. I don’t know how api calls in general work with 2fa since I have not tested or enabled it on my instance. :( Sorry.
EDIT: Changed database/pictures ratio after double checking actual numbers and not looking at used filesystem. :(
Hey, thanks for replying! Good to know about the subscription ownership. I only ran the discovery portion of the script just to dip my toe in the water, because I feared what would happen if I subscribed. That said, if I use the discovery flag, that’s just exposing the different communities to my instance, right? It’s not going to retrieve any remote content?
Good to see that storage management is roadmapped. I probably won’t invoke the Subscribe side of this for my bot account until then.
Understandable about 2FA. I’ll just be smart about my password usage and watch for any updates on that.
Thanks again!
It retrieves the last 10 posts and adds the community reference to your local database. It is the same as putting “!community@instance.com” in the search box and clicking search. The retrieval happens whether you look at the results or not.
Gotcha. Well that was a miscalculation on my part haha. Good to know for the future, though.
Saving this for when I make my own personal instance.
I was working on something like this to add a bot user and follow active communities but got distracted. Looks like I can quit that side project hahah
No. Keep at it. More alternatives is betterer.
deleted by creator
I salute your efforts for working on it anyway
ahah thanks, for posterity this is how far i got. Only needed to add the command to follow https://github.com/Death916/lemmy-federation-bot
I’m happy to help or take PRs for lemmony. There is also https://github.com/Fmstrat/lcs which I didn’t know about until well into lemmony.
If I’ve already run the last version recently is there a reason to run this new version immediately?
Not really. If it worked before, it should work the same, just has more options for control and granularity.
This seems pretty helpful, will spin it up soon and see how it goes.
So how does this operate now?
Sorry, by comparing now to what; or was as that a superfluous “now,” like “come on now?”
Well the last one would have just mass Overloaded instances. Just wondering how this one operates differently to prevent that.
The relevant change is this: https://github.com/jheidecker/lemmony/commit/9b0b7232a1a942615eb67a34f2526328a23fcdd5
It’s not meaningful though. It allows one to optionally limit to the top-n communities by active-user count, but the default is still unlimited and the docs still do nothing to explain the abusive load that excessive subscription puts on instances that host communities.
My dude, I appreciate your spirit, but we’re not going to focus on your irrational fear of abuse. I’ll defend myself for being accused of any such thing, or for being irresponsible. This is intended to make things better, and there’s no evidence it’s doing anything other than that.
If you want to contribute, by all means, show us where there is a problem, other than in your imagination, and it will be seriously considered. Until then, your opinion is still valuable, but you are speaking with authority about something you know little of.
I had no idea we were doing condescending pet names, this is a fun game. My sweet summer child…
It doesn’t take a rocket surgeon to figure out that global federation worker queues are not in great shape, or that a default that encourages single-user instance owners who have no idea what they’re doing to bump their sub count 70x isn’t helping the situation. If you think this is in my head I can’t help you. But I can help others understand that running this script with default settings is an awful and unnecessary idea.
Your argument does not gain validity by adding irrelevant verbosity:
The linked issue has nothing to do with this script or lemmony.
That’s a hasty generalization that you just made up.
That’s great! I hope they keep growing!
Nobody is encouraging anyone to do anything.
That single user asked a question and got berated by a jerk.
You can help others understand what it is. That’s a great thing to do. It would be nice if you could do that without being a dick.
Sigh. No. No, that’s actually how the computational complexity scales and it’s not a difficult analysis to perform. Good luck to you though.
I think it depends on your instance. This version (same script, just updated) allows for more options IF your instance IS mass Overloaded, or you are scared it will be.
I wrote http://github.com/fmstrat/lcs a while back which does similar, but only grabs Top feed communities to reduce load.
I have been using lcs on another account for a small instance and it has been amazing for making the instance feel connected. Thanks!
When I discovered, I felt bad for not checking. As for the load stuff. I intended and wanted to see All the things, and I don’t currently have resource problems for my instance. :) We’ll see how that fairs as things continue to grow!
A clean up would be nice, maybe something where if there are no subscribers, or local user actions on a community (votes, comments) after a while it starts to remove them.
Wouldn’t doing this result in a massive amount of unnecssary load on the larger instances?
I don’t really think so, but i’m open to working with anyone if they see this happening, up to deleting the entire project.
Yes, it would.
Dumb question I know, but do I leave this running on my instance or on my own computer?
It doesn’t matter. Most of the work is happening on the instance, regardless of where the script is running.
Doesn’t that significantly increase the load on your instance and, if many instances use it, all instances? This system isn’t designed with the idea that each instance receives everything from every other instance.
It might be better to run this on a single dedicated site which people can come to to browse. If you could learn where each user had their account, you could send their upvotes, downvotes and comments to that instance.
It increases load during execution. Afterward it’s not significant. My instance is heavily instrumented and monitored. The load this incurs subscribing to 24000 communities is less than adding a single, moderately active user to your instance.
It’s a huge miss if the intended design was to silo information.
What this provides, as far as I’m concerned, is essential to prevent centralization to a few instances.
Is there a better way to do it inherently in Lemmy itself? Probably, and I am excited to help with that!
Well that sounds quite reasonable then. It definitely answers a need for better discoverability of material on Lemmy. And it would be great if something like this could ultimately be integrated into Lemmy itself.
I think your idea is on the right track when thinking longer term and assuming the worst case in both design and admin behavior. :)
The whole network needs to be split into “active” and “archive.” New activity (or at the very least stubs to where new activity is happening) needs to be updated regardless of where it occurs without having to capture anything extra.
It is a significant difference if an instance has to federate literally every post made on the lemmyverse.
It increases load during execution. Afterward it’s not significant. My instance is heavily instrumented and monitored. The load this incurs subscribing to 24000 communities is less than adding a single, moderately active user to your instance.
It’s a huge miss if the intended design was to silo information.
What this provides, as far as I’m concerned, is essential to prevent centralization to a few instances.
Is there a better way to do it inherently in Lemmy itself? Probably, and I am excited to help with that!
Uhh… if your script is subbing to 24k remote communities, those will continue to grow from then on, unless you start purging communities at some point. After one user subscribes to a community, all new content gets indexed and stored on your instance. Pict-rs can cache images short term (and eventually clear them out), but Postgres will start growing very quickly and never slow down until it fills up disks.
Yup. 256 GB should be enough database space for anyone though.
Sorry, but unfortunately, it is, forcing a instance to sync literally every single post made on the lemmyverse.
Hosting an instance myself, I’m not amused, because if forces my instance to literally sync all content there is on the lemmyverse, drastically increasing traffic, storage use etc.
Please don’t force resource consumption beyond any rational usage!
people should just block Federation from the name says enough “Lemmy.management” to start with. one less activitypub abuser.
People can defederate from an instance for any reason they want, but if I get what you’re trying to say: you think people should defederate from any instance that has a user that subscribes to all of their communities.
deleted by creator
This is a cool idea. I was curious how subs work and it makes a lot more sense to me now that I’ve spent the time getting my instance setup and read a ton of docs. It feels weird trying to build your list and communities on your own account, while it also populates your instance’s All list. Feels like all of my interests are just out there flapping in the breeze lmao.
deleted by creator
I actually wrote it with the flip side of your centralization argument in mind. If a community exists outside of the popular ones a user may never even know of its existence. Having more show up SHOULD be better to prevent centralization no? It requires the users to change their browsing behaviour but at least they don’t have gonsearching offsite.
deleted by creator