I run my own instance because the content on it will always be very snappy, since it mirrors the text content that I’m subscribed to. It’s distributed like email or RSS are, in that you choose to subscribe to something, and then it is federated onto your instance when new activity occurs.
Link thumbnails do get mirrored. My understanding is the front end of Lemmy is pretty heavy for the big instances and the burden of federating to another instance is pretty small. One thing I’ve noticed on my instance is that sometimes inbound federation can be pretty annoyingly slow.
Not only annoyingly slow, but I tend to get a massive influx of posts from one community all at once. It fills my entire page with that single community. It’s been my biggest annoyance so far.
Your instance is it’s own thing and will only push data that’s from communities created on your instance to others.
However, since your instance will have limited users, it will consume fewer resources and your browsing experience would be faster.
I’m not sure of where the limits are on the pushing side of things. If people from a lot of instances are subscribed to a particular community in an instance, there is a lot of outgoing data from there. This is something I’m curious about.
Broadly, yes. The way federation works means anything any user on your instance is interested in will be sent to you once (at least posts/comments/votes/etc). Whenever someone on your instance views that thing that is a request that would otherwise be made to another instance. This does, however, increase the load of federation on servers hosting popular communities, as now they have to send each post/comment/vote/whatever to your instance. Unlike bit torrent there is only one place responsible for sending you all of the content that exists in a community, so the fediverse doesn’t get p2p-style network effects where every peer/sever helps even a bit.
A single user instance is a little inefficient, unless you are actually looking at most/all of the content your instance receives, in which case it is probably a wash. The ideal for how federation is implemented in ActivityPub would be many similarly-sized (in terms of user count) instances with the most popular communities being spread out among them.
Sadly right now the most popular instances (lemmy.world, lemmy.ml, lemmynsfw.com, kbin.social) are both where users and communities are, so the real gains to help those instances (several of which continue to struggle under the load) are really only medium and larger sized instances.
Isn’t a semi-significant amount of the per instance load in the database interactions though? I don’t know if that’s still true after some of the optimizations in 0.18, but by having your own instance, even 1 person instance, you no longer load the database of another instance with calculating hot/active/whatever for things you want to see, and you don’t load it when you do pulls.
There is certainly an inflection point. I am not sure where it falls. If you rarely use Lemmy the other servers are expending effort federating to you for no reason, so for a heavy single user it is probably a positive. I imagine it also varies based on how you browse as “new” is probably “lighter” than “hot” or “active”.
Sure, but the server calculates the hot/active every 15 minutes regardless of whether or not you browse it. It’s calculated on a timed job so it’s always fresh and ready for everyone.
I was unaware that was the case, that’s pretty expensive and inefficient to do it for every user (because each user’s “Subscribed” feed is likely different) every 15 minutes regardless of how recently they’ve visited. The inflection point is probably lower for the benefits of even single-user instances in that case.
All the communities you’re subscribed to will now also have to push all their updates (posts, comments, upvotes) to your server, even when you’re not interacting with Lemmy.
As someone else mentioned, it would only be efficient once you have a decent (hard to pinpoint) amount of users on your server.
So say you have your own personal instance, and you use that to follow community news on lemmy.world. If throughout the day that community receives 10 new topics, 50 comments and 100 upvotes, it would have to make 160 calls to your server.
So when you decide to read those 10 topics (if you even read all of them), you would then make roughly 10 api calls.
You would be saving those last mentioned 10 calls by using your own instance, but at the cost of 160 calls made throughout the day.
The thing is that when you interact with the remote server directly it’s not 10 api calls, it’s 10 full-blown HTML webpages that have to be served to you, which are way bigger than REST API calls.
Generally speaking, lemmy is much more cpu bound than it is bound by bandwidth - so the added bytes don’t matter that much. The example above was just for 1 community. Now imagine the user is subscribed to a dozen communities, but doesn’t even browse lemmy that day. That’s probably thousands of api calls made to keep his server on sync, and 0 requests saved.
Like the big instances have literallyhundreds of thousands of workers running in order to get all the updates out. If one of those calls fails, it gets put back into the queue for retry.
OP asked if having his server added to the lemmiverse would alleviate the load “Like with torrent”. That is demonstrably not the case - it only adds more workload on the other servers, with a break even point that’s highly variable. Yes, your server will be nice and snappy, but the origin servers have to pay the price - death by a thousand papercuts synchronisation calls.
So you need just 15 more users on your instance to break even, if you have 17 in total, you’ve saved 10 calls.
In this particular example, yes. But only if those 15 people subscribe to the exact same communities. If they don’t, the calculation gets even more complicated.
I’m not sure why exactly you’re opposed to federation when that’s one of the biggest points of fediverse.
Some people seem to be under the impression that setting up their own personal server is relieving the pressure on the network.
What I trying to get across is that’s not the case, unless it’s being used by a reasonable amount of people.
I can’t tell you what the sweet spot is - but my guess would be that it’s only going to be at least several dozen, more if their interests (subscriptions) don’t overlap very well.
It will take part of the load off, but the server still has a push updates to every instance subscribed to one of its communities. But I would guess that is much less of a demand than having an active user using the instance.
Should be. One instance hosting lot of community vs one instance hosting one community is less load. But your instance have to be popular enough to make a dent in the popular instances users.
Yes but only if you have a lot of users eventually. So your server can serve up hundreds of requests from several different federated servers (eg: maybe some content from beehaw, other content from lemmy.world) instead of 1 server (maybe lemmy.world) having to serve that same content 100 times from all over the place.
You are not logged in. However you can subscribe from another Fediverse account, for example Lemmy or Mastodon. To do this, paste the following into the search field of your instance: !selfhosted@lemmy.world
A place to share alternatives to popular online services that can be self-hosted without giving up privacy or locking you into a service you don’t control.
Rules:
Be civil: we’re here to support and learn from one another. Insults won’t be tolerated. Flame wars are frowned upon.
No spam posting.
Posts have to be centered around self-hosting. There are other communities for discussing hardware or home computing. If it’s not obvious why your post topic revolves around selfhosting, please include details to make it clear.
Don’t duplicate the full text of your blog or github here. Just post the link for folks to click.
Submission headline should match the article title (don’t cherry-pick information from the title to fit your agenda).
No trolling.
Resources:
selfh.st Newsletter and index of selfhosted software and apps
I think I’m going to host my own. I have a bunch of good, short domains I could set it up with.
I run my own instance because the content on it will always be very snappy, since it mirrors the text content that I’m subscribed to. It’s distributed like email or RSS are, in that you choose to subscribe to something, and then it is federated onto your instance when new activity occurs.
Text gets mirrored but not images. In general it helps, but not by much unless you host for several hundred users.
Link thumbnails do get mirrored. My understanding is the front end of Lemmy is pretty heavy for the big instances and the burden of federating to another instance is pretty small. One thing I’ve noticed on my instance is that sometimes inbound federation can be pretty annoyingly slow.
Not only annoyingly slow, but I tend to get a massive influx of posts from one community all at once. It fills my entire page with that single community. It’s been my biggest annoyance so far.
This is fixed in 0.18.x
I’m running 0.18.1-rc.4 and it’s fixed lots of issues
That’s great news! Hopefully it releases soonish.
Your instance is it’s own thing and will only push data that’s from communities created on your instance to others.
However, since your instance will have limited users, it will consume fewer resources and your browsing experience would be faster.
I’m not sure of where the limits are on the pushing side of things. If people from a lot of instances are subscribed to a particular community in an instance, there is a lot of outgoing data from there. This is something I’m curious about.
Broadly, yes. The way federation works means anything any user on your instance is interested in will be sent to you once (at least posts/comments/votes/etc). Whenever someone on your instance views that thing that is a request that would otherwise be made to another instance. This does, however, increase the load of federation on servers hosting popular communities, as now they have to send each post/comment/vote/whatever to your instance. Unlike bit torrent there is only one place responsible for sending you all of the content that exists in a community, so the fediverse doesn’t get p2p-style network effects where every peer/sever helps even a bit.
A single user instance is a little inefficient, unless you are actually looking at most/all of the content your instance receives, in which case it is probably a wash. The ideal for how federation is implemented in ActivityPub would be many similarly-sized (in terms of user count) instances with the most popular communities being spread out among them.
Sadly right now the most popular instances (lemmy.world, lemmy.ml, lemmynsfw.com, kbin.social) are both where users and communities are, so the real gains to help those instances (several of which continue to struggle under the load) are really only medium and larger sized instances.
What is the easiest way to learn about ActivityPub?
These posts are really useful to get a grasp of ActivityPub (if you have a programming background): https://rknight.me/building-an-activitypub-server/ https://blog.joinmastodon.org/2018/06/how-to-implement-a-basic-activitypub-server/
And of course the official spec (although its less useful): https://www.w3.org/TR/activitypub/
A highly opinionated guide to learning about ActivityPub
Isn’t a semi-significant amount of the per instance load in the database interactions though? I don’t know if that’s still true after some of the optimizations in 0.18, but by having your own instance, even 1 person instance, you no longer load the database of another instance with calculating hot/active/whatever for things you want to see, and you don’t load it when you do pulls.
There is certainly an inflection point. I am not sure where it falls. If you rarely use Lemmy the other servers are expending effort federating to you for no reason, so for a heavy single user it is probably a positive. I imagine it also varies based on how you browse as “new” is probably “lighter” than “hot” or “active”.
Sure, but the server calculates the hot/active every 15 minutes regardless of whether or not you browse it. It’s calculated on a timed job so it’s always fresh and ready for everyone.
I was unaware that was the case, that’s pretty expensive and inefficient to do it for every user (because each user’s “Subscribed” feed is likely different) every 15 minutes regardless of how recently they’ve visited. The inflection point is probably lower for the benefits of even single-user instances in that case.
In effect, not really.
All the communities you’re subscribed to will now also have to push all their updates (posts, comments, upvotes) to your server, even when you’re not interacting with Lemmy.
As someone else mentioned, it would only be efficient once you have a decent (hard to pinpoint) amount of users on your server.
They’ll do it once, though. Then every time you view it, you’re helping the bigger servers by serving it from your instance.
Yes. Once for every post, comment and vote.
So say you have your own personal instance, and you use that to follow community
newson lemmy.world. If throughout the day that community receives 10 new topics, 50 comments and 100 upvotes, it would have to make 160 calls to your server.So when you decide to read those 10 topics (if you even read all of them), you would then make roughly 10 api calls.
You would be saving those last mentioned 10 calls by using your own instance, but at the cost of 160 calls made throughout the day.
The thing is that when you interact with the remote server directly it’s not 10 api calls, it’s 10 full-blown HTML webpages that have to be served to you, which are way bigger than REST API calls.
Generally speaking, lemmy is much more cpu bound than it is bound by bandwidth - so the added bytes don’t matter that much. The example above was just for 1 community. Now imagine the user is subscribed to a dozen communities, but doesn’t even browse lemmy that day. That’s probably thousands of api calls made to keep his server on sync, and 0 requests saved.
Like the big instances have literally hundreds of thousands of workers running in order to get all the updates out. If one of those calls fails, it gets put back into the queue for retry.
OP asked if having his server added to the lemmiverse would alleviate the load “Like with torrent”. That is demonstrably not the case - it only adds more workload on the other servers, with a break even point that’s highly variable. Yes, your server will be nice and snappy, but the origin servers have to pay the price - death by a thousand
papercutssynchronisation calls.Yeah you’re right, I just felt the need to point out that API calls are not really comparable to serving a full website.
So you need just 15 more users on your instance to break even, if you have 17 in total, you’ve saved 10 calls.
In this particular example, yes. But only if those 15 people subscribe to the exact same communities. If they don’t, the calculation gets even more complicated.
Some people seem to be under the impression that setting up their own personal server is relieving the pressure on the network. What I trying to get across is that’s not the case, unless it’s being used by a reasonable amount of people.
I can’t tell you what the sweet spot is - but my guess would be that it’s only going to be at least several dozen, more if their interests (subscriptions) don’t overlap very well.
It will take part of the load off, but the server still has a push updates to every instance subscribed to one of its communities. But I would guess that is much less of a demand than having an active user using the instance.
Should be. One instance hosting lot of community vs one instance hosting one community is less load. But your instance have to be popular enough to make a dent in the popular instances users.
Yes but only if you have a lot of users eventually. So your server can serve up hundreds of requests from several different federated servers (eg: maybe some content from beehaw, other content from lemmy.world) instead of 1 server (maybe lemmy.world) having to serve that same content 100 times from all over the place.
If its only you and no other users, then not.