AFAIK every NAS just uses unauthenticated connections to pull containers, I’m not sure how many actually allow you to log in even (raising the limit to a whopping 40 per hour).
So hopefully systems like /r/unRAID handle the throttling gracefully when clicking “update all”.
Anyone have ideas on how to set up a local docker hub proxy to keep the most common containers on-site instead of hitting docker hub every time?
Did they really? Oh my god please tell me your joking, that a company as modern as docker got a freaking oracle CEO. They pulled a Jack Barker. Did he bring his conjoined triangles of success?
you could also self host something like gitlab, which bundles this or sonatype nexus which can serve as a repository for several kinds of artifacts including container images.
Instead of using a sort of Docker Hub proxy, you can also use GitHub’s repository or Quay. If the project allows it, you can easily switch to these alternatives. Alternatively, you can build the Docker image yourself from the source. It’s usually not a difficult process, as most of it is automated.
Or what I personally would probably do is just update the image a day later if I hit the limit.
Would you be able to share more info? I remember reading their issues with docker, but I don’t recall reading about whether or what they switched to.
What is it now?
They run their own registry at lscr.io. You can essentially prefix all your existing linuxserver image names with lscr.io/ to pull them from there instead.
Same here. I’ve been building a bootstrap script, and each time I test it, it tears down the whole cluster and starts from scratch, pulling all of the images again. Every time I hit the Docker pull limit after 10 - 12 hours of work, I treat that as my “that’s enough work for today” signal. I’m going to need to set up a caching system ASAP or the hours I work on this project are about to suddenly get a lot shorter.
You can host your own with harbor, and set up replication per repo (pull upstream tags)
If you need a commercial product/support you can use MSR v4.
Harbor can install on any K8s cluster using helm, with just a couple of dependencies (cert-manager, postgres op, redis-op)
Replication stuff you can easily add.
I have some no-warranty terraform I could share if there is some interest.
Artifactory is mandatory in some industries because it will keep all the versions of the images forever so that you can build your projects reliably without an internet connction.
Well shit, I still rely on Docker Hub even for automated pulls so this is just great. I guess i’m going back to managing VMs with OpenTofu and package managers.
What are our alternatives if we use Podman or K8s?
The issue isn’t Docker vs Podman vs k8s vs LXC vs others. They all use OCI images to create your container/pod/etc. This new limit impacts all containerization solutions, not just Docker. EDIT: removed LXC as it does not support OCI
Instead, the issue is Docker Hub vs Quay vs GHCR vs others. It’s about where the OCI images are stored and pulled from. If the project maintainer hosts the OCI images on Docker Hub, then you will be impacted by this regardless of how you use the OCI images.
Some options include:
For projects that do not store images on Docker Hub, continue using the images as normal
Become a paid Docker member to avoid this limit
When a project uses multiple container registries, use one that is not Docker Hub
For projects that have community or 3rd party maintained images on registries other than Docker Hub, use the community or 3rd party maintained images
For projects that are open source and/or have instructions on building OCI images, build the images locally and bypass the need for a container registry
For projects you control, store your images on other image registries instead of (or in addition to) Docker Hub
Use an image tag that is updated less frequently
Rotate the order of pulled images from Docker Hub so that each image has an opportunity to update
Pull images from Docker Hub less frequently
For images that are used by multiple users/machine under your supervision, create an image cache or image registry of images that will be used by your users/machines to mitigate the number of pulls from Docker Hub
Encourage project maintainers to store images on image registries other than Docker Hub (or at least provide additional options beyond Docker Hub)
Do not use OCI images and either use VM or bare metal installations
Use alternative software solutions that store images on registries other than Docker Hub
Lxc doesn’t use oci images? I always end up using docker in lxc when dockeris the only option (which I have not figured how to makw work on my airgapped side
incus may be an option for you though. It supports both LXC/LXD and OCI (although not nearly as well as Docker/Podman/Kubernetes - I don’t think it supports any compose files).
The problem is that the main container can (and usually does) rely on other layers, and you may need to pull updates for those too. Updating one app can take 5-10 individual pulls.
You are not logged in. However you can subscribe from another Fediverse account, for example Lemmy or Mastodon. To do this, paste the following into the search field of your instance: !selfhosted@lemmy.world
A place to share alternatives to popular online services that can be self-hosted without giving up privacy or locking you into a service you don’t control.
Rules:
Be civil: we’re here to support and learn from one another. Insults won’t be tolerated. Flame wars are frowned upon.
No spam posting.
Posts have to be centered around self-hosting. There are other communities for discussing hardware or home computing. If it’s not obvious why your post topic revolves around selfhosting, please include details to make it clear.
Don’t duplicate the full text of your blog or github here. Just post the link for folks to click.
Submission headline should match the article title (don’t cherry-pick information from the title to fit your agenda).
No trolling.
Resources:
selfh.st Newsletter and index of selfhosted software and apps
Use a service that’s not Docker hub
Forgejo gives you a registry built-in.
Also is it just me or does the docker hub logo look like it’s giving us the middle finger?
How long since getting an oracle CEO did this take?
Did they really? Oh my god please tell me your joking, that a company as modern as docker got a freaking oracle CEO. They pulled a Jack Barker. Did he bring his conjoined triangles of success?
A “jack barker” 🤣
https://distribution.github.io/distribution/
is an opensource implementation of a registry.
you could also self host something like gitlab, which bundles this or sonatype nexus which can serve as a repository for several kinds of artifacts including container images.
Gitea and therefore Forgejo also have container registry functionality, I use that for private builds.
oh, thats good to know, forgejo seems way nicer for self hosting than the limited gitlab open source core.
Codeberg as woodpecker CI
Jumping on the forgejo love train
Instead of using a sort of Docker Hub proxy, you can also use GitHub’s repository or Quay. If the project allows it, you can easily switch to these alternatives. Alternatively, you can build the Docker image yourself from the source. It’s usually not a difficult process, as most of it is automated. Or what I personally would probably do is just update the image a day later if I hit the limit.
You can also host your own with harbor (or MSR v4 if you want a commercial product.) You can set them up to replicate upstream.
Fortunately linuxserver’s main hosting is no longer dockerhub.
Would you be able to share more info? I remember reading their issues with docker, but I don’t recall reading about whether or what they switched to. What is it now?
They run their own registry at
lscr.io. You can essentially prefix all your existing linuxserver image names withlscr.io/to pull them from there instead.It’s actually a redirect service around ghcr to provide them analytics. There’s more info in their FAQ.
https://docs.linuxserver.io/FAQ/
They do it since quite some time now, right?
Couple of years, yeah.
deleted by creator
Huh. I was just considering establishing a caching registry for other reasons. Ferb, I know what we’re going to do today!
Same here. I’ve been building a bootstrap script, and each time I test it, it tears down the whole cluster and starts from scratch, pulling all of the images again. Every time I hit the Docker pull limit after 10 - 12 hours of work, I treat that as my “that’s enough work for today” signal. I’m going to need to set up a caching system ASAP or the hours I work on this project are about to suddenly get a lot shorter.
Do you have a good resource for how one can go about this?
You can host your own with harbor, and set up replication per repo (pull upstream tags) If you need a commercial product/support you can use MSR v4.
Harbor can install on any K8s cluster using helm, with just a couple of dependencies (cert-manager, postgres op, redis-op) Replication stuff you can easily add.
I have some no-warranty terraform I could share if there is some interest.
That’s what we do internally for our openshift deployment. It will reach out if not in harbor and then cache it there for everyone else to use.
I’ve only done my “is it even possible” research so far, but these look promising:
https://medium.com/@amandubey_6607/docker-registry-caching-a2dfefecfff5
https://github.com/obeone/multi-registry-cache
https://www.squid-cache.org/ Should work too I think
Much appreciated <3
Is there a project that acts like a registry? It can proxy the request with TTL, and you can push images to it too?
Almost all of them. Forgejo handles containers already for example
Pull through Cache / proxy is what you’re looking for.
How? I was looking for this (although not very thoroughly)
[Edit] found it https://forgejo.org/docs/v1.21/user/packages/container/
Artifactory is mandatory in some industries because it will keep all the versions of the images forever so that you can build your projects reliably without an internet connction.
I think most self-hosted Git+CI/CD platforms have container registry as a feature, but I’m not aware of a service that is just a standalone registry.
It’s easy to oversee because of the generic name, but this is pretty much that: https://hub.docker.com/_/registry
Edit: forgot there’s jfrog artifactory as well
Well shit, I still rely on Docker Hub even for automated pulls so this is just great. I guess i’m going back to managing VMs with OpenTofu and package managers.
What are our alternatives if we use Podman or K8s?
The issue isn’t Docker vs Podman vs k8s
vs LXCvs others. They all use OCI images to create your container/pod/etc. This new limit impacts all containerization solutions, not just Docker. EDIT: removed LXC as it does not support OCIInstead, the issue is Docker Hub vs Quay vs GHCR vs others. It’s about where the OCI images are stored and pulled from. If the project maintainer hosts the OCI images on Docker Hub, then you will be impacted by this regardless of how you use the OCI images.
Some options include:
Lxc doesn’t use oci images? I always end up using docker in lxc when dockeris the only option (which I have not figured how to makw work on my airgapped side
https://github.com/lxc/lxc/commit/0ef43a5c1d635835950335375f0dd71bf1abe384
It does.
Does that mean I could rip the oci image out of a docker and run ir as an lxc?
I much prefer dealing with lxc systems than docker even if dockers are lighterweigth.
I feel more in control with using lxc
If you’re using just raw LXC it’s easy…
lxc-create <<name>> -t oci -- --url docker://alpine:latestWill spawn an alpine docker container for instance.
If you’re using something like proxmox though, it’s a little more annoying since there’s some work to get proxmox to add it to it’s container list.
Great, I’ll try with docker-mailserver, which I’m running inside docker inside a debian lxc
@interdimensionalmeme @wireless_purposely832
I believe Graber did a talk at FOSDEM this year about using OCI images in Incus.
Ah, you’re right. I’ll edit my comment.
incus may be an option for you though. It supports both LXC/LXD and OCI (although not nearly as well as Docker/Podman/Kubernetes - I don’t think it supports any compose files).
Do you mean a Docker container registry? If so, here are a couple options:
I think they mean pull-through cache. https://shipyard.build/blog/how-to-docker-registry-pull-through-cache/
If only they used a distributed protocol like ipfs, we wouldn’t be in this situation
I’m quite new to docker for NAS stuff - how many pulls would the average person do? like, i don’t think i even have 10 containers 🤨
I’m running ~30 containers, but they don’t typically all get new updates at the same time.
Updates are grabbed nightly, and I think the most I’ve seen update at once is like 6 containers.
Could be a problem for setting up a new system, or experimenting with new toys.
The problem is that the main container can (and usually does) rely on other layers, and you may need to pull updates for those too. Updating one app can take 5-10 individual pulls.