Update : I made a follow-up post containing a Nginx-based solution to cache map tiles from OSM and limit the amount of PII you send
While monitoring the logs in Rethink DNS (awesome app BTW) today, I noticed the Immich app making requests to api-l.cofractal.com.
After reaching out on Immich’s discord, the devs explained to me that it is used as a tile provider for the map feature. I can confirm it is not realistic to self-host a tile provider without heavily tuning down the level of details on the map (which would still require a lot of disk space and CPU time). I understand the need for a third-party service to provide the map tiles, but I’m concerned by this one.
Visiting cofractal.com only tells us that they’re selling APIs. I did not find any details about the company, not even the country they’re registered in. The website is also missing informations about what they are logging or not. Everything else seems gated behind a login page, but they “are not currently accepting new customers”. The whois for the domain says they’re in California. Digging a bit more, I find AS26073 which apparently is the same company.
This bothers me, because Cofractal gets sent every location you viewed (and the zoom level) on Immich’s map, along with your client’s IP address and a “Referrer” header pointing to your Immich instance. This sounds like a lot of PII to me. It’s also behind cloudflare which gets to see the same stuff.
When asked about it, one dev (thanks to them for almost instantly replying to every concern/question I threw at them) explained that they personally know the people behind Cofractal. According to this Immich dev, Cofractal provides free access to its paid service to Immich’s user base as a way to support the project, with the side benefit of load testing their platform.
This explanations seems plausible and reasonable to me. However, I do not personally know the people behind Cofractal, and by default, I do not trust for-profit companies to act in an altruistic way. Here’s a summary of everything that makes me uneasy about this company :
it does not say anything about the kind of data they are logging or not
it requires digging through whois records to find the most basic info about the company
it freely provides access to its normally paid service (for the whole Immich user base), but it does not communicate about it (or it is really hard to find)
it does not communicate about anything : searching for its name only returns its home page and websites with informations on Autonomous Systems
It is not mentioned anywhere in the whole immich.app website (searching for site:immich.app "cofractal" gave me no result). Not even a “Thank You” or “Sponsor” note on the homepage for the free API
(it is behind cloudflare)
The dev I talked to encouraged me to create a feature request, and seemed favorable to adding a switch for disabling maps client side. It is already possible to disable it server-wide, and the “URL to a style.json map theme” option seems to provide a way to customize the tile provider. Which leads to this post : I’m trying to collect feedback on this before creating the feature request.
It should be made prominently clear to server admins that leaving maps enabled causes clients to send requests to a third party-server and give details about what is sent (viewed locations, zoom level, IP address, Immich instance URL). The Post Install Steps in the docs and a paragraph above the switch on the config page seem like good places to me. Are there other/more appropriate place for such a warning ?
The “URL to a style.json map theme” option should probably be renamed to make it clearer that it allows changing tile providers. Or better yet, it could be reworked to make it easier to choose which third-party you decide to trust
What do you think about the idea of providing instance admins with a list of choices for tile providers ? Maybe with a short pros/cons list in the docs for each provider. I’d be fine with using a more reputable provider with the extra step of configuring my own API key (which would probably require proxying requests to the tile provider to not share the API key with all clients)
Should the Immich server proxy requests to the tile provider in any case ? Since the tile provider has access to the Referrer and Origin headers (which is probably required for CORS), they are currently able to link user IP addresses with Immich instances. Proxying requests with the Immich server should prevent that.
I would go as far as making maps disabled by default for new installs. I understand that “disabling by default would be a significant downgrade for a majority of users”, but I feel like there’s a strong overlap between the self-hosting and privacy communities. So we should at least have some debate about it
I’ve also been told that I’m the first one to raise concerns about this, which leads to one more question : Did nobody complain because nobody noticed ? Or are my concerns unjustified ?
I don’t use Immich, but if you open a feature request I’d suggest asking for open street maps, which offers a self hosted tile service https://www.maptiler.com/server/
When I mentionned that “I can confirm it is not realistic to self-host a tile provider”, it’s because I tried to run maptiler : it maxed out my CPU for 2 hours before my disk got filled while trying to generate the tiles from OSM data (and it was just for France)
Edit : Anyway, I don’t think this should be in Immich’s scope. Simply providing an easy option to switch tile providers would allow people motivated enough to host maptiler to use it
I can really recommend pmtiles. They are already pre-generated and easily hostable with something like Minio without the need for a dedicated server. My europe pmtile file (only one file!) is around 40-50 gb.
I don’t self host presently, but if I get my server hardware back (moved out of the country a while) I want to dabble with a self hosted photo solution, so I’m glad to have found your post that keeps this fresh in my mind.
I think disabling by default and having a clear explanation of what enabling it involves is good.
Maybe in the initial account creation/onboarding on a new instance, have it ask if server wide maps should be enabled using the default provider, with clear text about what that involves.
The option to use other providers sounds good too.
Leaflet is just the frontend. You need the actual tiles as well. You can either generate your own tiles or use something like mapbox, who generate the tiles for you and keep them updated.
Seems like a very reasonable objection to me. I’d guess that most of us Immich users are using it in the first place because it improves the privacy of our photos, and a third party seeing our location data certainly undermines that.
I would have complained had I noticed, so you might be the first one to notice. Immich’s userbase isn’t huge right now, it’s definitely possible.
Featurewise, I’d like: a) a clearly documented way to disable map data leaving my server; b) a set of well-integrated choices (maybe even just two, as long as one of them is something like openstreetmap); c) the current configurability to be well documented.
I’d love it if all such outbound data streams are also documented. Many security and privacy-focused products give you a “quiet” mode of some kind, where you can turn off everything that sends your data somewhere else. It’s a requirement in many enterprise installations.
Thanks for the detailed feedback. According to one Immich dev, they used to use OSM’s raster tile provider but switched away from it since they were causing too much load on OSM’s servers.
There does not seem to be any non-commercial vector-tile provider at the moment (though OSM seems to be currently working on it), and it seems really overkill to try and self-host a tile provider (at least with the default level of details). Maybe the way is to find a balanced level of details that makes it reasonable to self host
They could host their own caching proxy between OSM and their users though.
Also, Home Assistant uses OpenStreetMap and they have more users than Immich does.
Edit: Home Assistant does use OSM data, but they use it via another third-party called CARTO, who at least have a proper site: https://carto.com/basemaps. Tiles come from URLs like https://basemaps.cartocdn.com/rastertiles/voyager/12/657/1580@2x.png
Home assistant’s main use case is showing you where your house is on a single map, though. Not sure how immich works, but if it’s one tile per photo with location data, that would be a MUCH bigger ask.
Or you can quite easily configure nginx as your personal caching proxy with an arbitrarily long TTL/retention duration (you can check out my follow-up post for instructions on doing that)
At the end of the day, someone has to pay for it. Either the users pay, or Immich’s developers pay, or a map provider pays (by offering it for free and covering the costs).
You could also start by denying any outside connection to anything except private IP ranges for any docker container, and only allow it on a need to have basis.
It’s not enough to rely on the the good will and savvy of whoever made the software, you have to make the restrictions stick.
It’s the clients (web/android app, probably iOS too) that are making these requests.
To the best of my knowledge, the Immich server inside the container is not making requests to the outside. It is merely sending a style.json to the client displaying a map, which then fetches tiles from the Cofractal URL inside this JSON.
Sounds reasonable. You said they already have a switch to turn it off, which is good. I don’t think it’s unreasonable to have it off by default, with a button to turn it on right where the map data should appear.
Quoting one dev from the conversation I had on Discord :
the one run by OSM is not intended for general purpose use because that results in way too much load on their system. We used to use theirs, but as Immich grew we decided that we should relieve them of that
I guess you (and they) are talking about raster tiles, since OSM does not seem to provide vector tiles
OSM’s core tile servers have dozens of cores, hundreds of GB of RAM each, and the rendering and lookup databases are a few TB. That’s not trivial to self host, especially since one self hosted tile server cannot always keep up with a user flick scrolling.
Edit: car GPS maps and the old TomTom and Garmin devices have significantly less metadata embedded than a modern map.
TBF osm tile servers have the task of providing immediate feedback to the mappers editing osm. The immich map doesn’t need to update live, redrawing each tile like once a month would be fine
A static PNG tile database for world.osm is even larger. Without a solid vector tile solution, this is the most efficient data format for disk space.
Also, there’s a post render CDN cache in front of the rendering layer to offset load, plus there’s I think some internal caching in renderd. It’s a pretty complex machine, but databases of the world are in fact huge.
I don’t use Immich, but want to say that your post here is very detailed and level-headed - and I appreciate it. Giving users more choice over how their data is used is always good.
Good on you for having such a detailed and well-meaning post, and good on the devs for being so receptive to ideas. I run Immich but am definitely a novice when it comes to self hosting things, so I had no idea this was going on. I don’t have any suggestions for the feature request but the proposed changes to the documentation would be good for people to know its thete and can be changed.
You are not logged in. However you can subscribe from another Fediverse account, for example Lemmy or Mastodon. To do this, paste the following into the search field of your instance: !selfhosted@lemmy.world
A place to share alternatives to popular online services that can be self-hosted without giving up privacy or locking you into a service you don’t control.
Rules:
Be civil: we’re here to support and learn from one another. Insults won’t be tolerated. Flame wars are frowned upon.
No spam posting.
Posts have to be centered around self-hosting. There are other communities for discussing hardware or home computing. If it’s not obvious why your post topic revolves around selfhosting, please include details to make it clear.
Don’t duplicate the full text of your blog or github here. Just post the link for folks to click.
Submission headline should match the article title (don’t cherry-pick information from the title to fit your agenda).
No trolling.
Resources:
selfh.st Newsletter and index of selfhosted software and apps
Completely reasonable to have this obvious and easy to toggle. In fact it should be opt-in as part of the initial setup flow.
I had no idea. I assumed the maps come from OSM.
They seem to have have addressed this in the new version 1.110.0 by hosting their own tile server
I don’t use Immich, but if you open a feature request I’d suggest asking for open street maps, which offers a self hosted tile service https://www.maptiler.com/server/
When I mentionned that “I can confirm it is not realistic to self-host a tile provider”, it’s because I tried to run maptiler : it maxed out my CPU for 2 hours before my disk got filled while trying to generate the tiles from OSM data (and it was just for France)
Edit : Anyway, I don’t think this should be in Immich’s scope. Simply providing an easy option to switch tile providers would allow people motivated enough to host maptiler to use it
Edit bis : More details on how hard it is to host your own tile provider are available on the official OSM wiki
Lol, no kidding!
I can really recommend pmtiles. They are already pre-generated and easily hostable with something like Minio without the need for a dedicated server. My europe pmtile file (only one file!) is around 40-50 gb.
Acronyms, initialisms, abbreviations, contractions, and other phrases which expand to something larger, that I’ve seen in this thread:
3 acronyms in this thread; the most compressed thread commented on today has 10 acronyms.
[Thread #886 for this sub, first seen 25th Jul 2024, 08:15] [FAQ] [Full list] [Contact] [Source code]
I would have assumed that maps are disabled by default and all requests proxied by the server to some mapbox api that has been set by the admin
This is a great post! I don’t use immich; I use ente.io and I don’t host it, but I do know they use OSM, as confirmed in #14 of their privacy policy:
Open Street Maps
I don’t self host presently, but if I get my server hardware back (moved out of the country a while) I want to dabble with a self hosted photo solution, so I’m glad to have found your post that keeps this fresh in my mind.
I think disabling by default and having a clear explanation of what enabling it involves is good.
Maybe in the initial account creation/onboarding on a new instance, have it ask if server wide maps should be enabled using the default provider, with clear text about what that involves.
The option to use other providers sounds good too.
Can it just use leaflet
deleted by creator
No need to be rude…
Leaflet is just the frontend. You need the actual tiles as well. You can either generate your own tiles or use something like mapbox, who generate the tiles for you and keep them updated.
Can’t you just pull from OSM? I think that’s what leaflet does
You can pull from any provider you want, with Leaflet. Including OSM. But! OSM does not have the resources to provide tiles for everyone. https://operations.osmfoundation.org/policies/tiles/
Thanks for the detailed feedback. According to one Immich dev, they used to use OSM’s raster tile provider but switched away from it since they were causing too much load on OSM’s servers.
There does not seem to be any non-commercial vector-tile provider at the moment (though OSM seems to be currently working on it), and it seems really overkill to try and self-host a tile provider (at least with the default level of details). Maybe the way is to find a balanced level of details that makes it reasonable to self host
They could host their own caching proxy between OSM and their users though.
Also, Home Assistant uses OpenStreetMap and they have more users than Immich does.
Edit: Home Assistant does use OSM data, but they use it via another third-party called CARTO, who at least have a proper site: https://carto.com/basemaps. Tiles come from URLs like
https://basemaps.cartocdn.com/rastertiles/voyager/12/657/1580@2x.pngHome assistant’s main use case is showing you where your house is on a single map, though. Not sure how immich works, but if it’s one tile per photo with location data, that would be a MUCH bigger ask.
Oh yeah, that’s a good catch. Hosting their own proxy/CDN in front of OSM should be doable though.
Or you can quite easily configure nginx as your personal caching proxy with an arbitrarily long TTL/retention duration (you can check out my follow-up post for instructions on doing that)
I can see that as a paid premium feature. There are costs to host anything.
At the end of the day, someone has to pay for it. Either the users pay, or Immich’s developers pay, or a map provider pays (by offering it for free and covering the costs).
You could also start by denying any outside connection to anything except private IP ranges for any docker container, and only allow it on a need to have basis.
It’s not enough to rely on the the good will and savvy of whoever made the software, you have to make the restrictions stick.
It’s the clients (web/android app, probably iOS too) that are making these requests.
To the best of my knowledge, the Immich server inside the container is not making requests to the outside. It is merely sending a
style.jsonto the client displaying a map, which then fetches tiles from the Cofractal URL inside this JSON.Rethink is awesome, thanks for sharing that app!!
Sounds reasonable. You said they already have a switch to turn it off, which is good. I don’t think it’s unreasonable to have it off by default, with a button to turn it on right where the map data should appear.
Wonder why they wouldn’t use OSM.
Quoting one dev from the conversation I had on Discord :
I guess you (and they) are talking about raster tiles, since OSM does not seem to provide vector tiles
I don’t understand why it even needs to connect to OSM servers. Download the maps locally and plot the points locally.
I am not an IT expert though…
OSM’s core tile servers have dozens of cores, hundreds of GB of RAM each, and the rendering and lookup databases are a few TB. That’s not trivial to self host, especially since one self hosted tile server cannot always keep up with a user flick scrolling.
Edit: car GPS maps and the old TomTom and Garmin devices have significantly less metadata embedded than a modern map.
TBF osm tile servers have the task of providing immediate feedback to the mappers editing osm. The immich map doesn’t need to update live, redrawing each tile like once a month would be fine
A static PNG tile database for world.osm is even larger. Without a solid vector tile solution, this is the most efficient data format for disk space.
Also, there’s a post render CDN cache in front of the rendering layer to offset load, plus there’s I think some internal caching in renderd. It’s a pretty complex machine, but databases of the world are in fact huge.
You could probably save a lot of space by throwing out the higher zoom levels. Level 16 or 17 should be plenty to mark where a photo was taken
When I use Organic Maps, I download the parts of the map I need (my country). Is there a reason something like Immich couldn’t do something similar?
It’s hundreds of MB per map section, but I have hundreds of GB of photos so it would be a drop in the bucket.
Only reason is it’s hard to do
Isn’t vector tiles generated locally? A raster tile needs to be generated every time it changes.
Can immich be configured to work like pixelfed?
I don’t use Immich, but want to say that your post here is very detailed and level-headed - and I appreciate it. Giving users more choice over how their data is used is always good.
Good on you for having such a detailed and well-meaning post, and good on the devs for being so receptive to ideas. I run Immich but am definitely a novice when it comes to self hosting things, so I had no idea this was going on. I don’t have any suggestions for the feature request but the proposed changes to the documentation would be good for people to know its thete and can be changed.