I use nftables to set my firewall rules. I typically manually configure the rules myself. Recently, I just happened to dump the ruleset, and, much to my surprise, my config was gone, and it was replaced with an enourmous amount of extremely cryptic firewall rules. After a quick examination of the rules, I found that it was Docker that had modified them. And after some brief research, I found a number of open issues, just like this one, of people complaining about this behaviour. I think it’s an enourmous security risk to have Docker silently do this by default.
I have heard that Podman doesn’t suffer from this issue, as it is daemonless. If that is true, I will certainly be switching from Docker to Podman.
I’m a podman user, but what’s the point of using podman if you are going to use a daemon and run it as root? I like podman so I can specifically avoid those things.
I am using it as a migration tool tbh. I am trying to get to rootless, but some of the stuff I host just don’t work well in rootless yet, so I use rootful for those containers. Meanwhile, I am using rootless for dev purposes or when testing out new services that I am unsure about.
Podman also has good integration into Cockpit, which is nice for monitoring purposes.
podman-compose is different from docker-compose. It runs your containers in rootless mode. This may break certain containers if configured incorrectly. This is why I suggested podman-docker, which allows podman to emulate docker, and the native docker-compose tool. Then you use sudo docker-compose to run your compose files in rootful mode.
It isn’t that much better. I use it as drop-in docker replacement. It’s better integrated with things like cockpit though and the idea is that it’s easier to eventually migrate to rootless if you’re already in the podman ecosystem.
Ok that sounds intetesting, I’ve found Cockpit easier to use than Proxmox, I’m new to virtualization and I don’t want do nesting… I fear it will complicate things when I’ll need to do GPU passthrough.
How is Podman integrated into Cockpit?
Also, I had so much trouble trying to bridge my Home Assistant VM to my LAN. Are there any tutorials on how to do this from Cockpit?
Your containers show up in Cockpit under the “Podman containers” section and you can view logs, type commands into their consoles, etc. You can even start up containers, manage images, etc.
Are there any tutorials on how to do this from Cockpit?
I have not done this personally, but I would assume you need to create a bridge device in Network Manager or via Cockpit and then tell your VM to use that. Keep in mind, bridge devices only work over Ethernet.
ofcourse docker desktop is not affected because it’s a vm running docker running on your computer. the nftables inside the container are modified by docker
How come I don’t see my previous rules when I dump the ruleset, then? I have my rules written in /etc/nftables.conf, and they were previously applied by running # nft -f /etc/nftables.conf. Now, when I dump the current ruleset with # nft list ruleset, those previous rules aren’t there — all I see are Docker’s rules.
Yeah, it needs those rules for e.g. port-forwarding into the containers.
But it doesnt really ‘nuke’ existing ones.
I have simply placed my rules at higher priority than normal. Very simple in nftables and good to not have rules mixed between nftables and iptables in unexpected ways.
You should filter as early as possible anyways to reduce ressource usage on e.g. connection tracking.
How come I don’t see my previous rules when I dump the ruleset, then? I have my rules written in /etc/nftables.conf, and they were previously applied by running # nft -f /etc/nftables.conf. Now, when I dump the current ruleset with # nft list ruleset, those previous rules aren’t there — all I see are Docker’s rules.
This is standard, but often unwanted, behavior of docker.
Docker creates a bunch of chain rules, but IIRC, doesn’t modify actual incoming rules (at least it doesn’t for me) it just will make a chain rule for every internal docker network item to make sure all of the services can contact each other.
Yes it is a security risk, but if you don’t have all ports forwarded, someone would still have to breach your internal network IIRC, so you would have many many more problems than docker.
I think from the dev’s point of view (not that it is right or wrong), this is intended behavior simply because if docker didn’t do this, they would get 1,000 issues opened per day of people saying containers don’t work when they forgot to add a firewall rules for a new container.
Option to disable this behavior would be 100x better then current, but what do I know lol
IIRC, doesn’t modify actual incoming rules (at least it doesn’t for me)
How come I don’t see my previous rules when I dump the ruleset, then? I have my rules written in /etc/nftables.conf, and they were previously applied by running # nft -f /etc/nftables.conf. Now, when I dump the current ruleset with # nft list ruleset, those previous rules aren’t there — all I see are Docker’s rules.
Don’t know what it’s actually doing, I’m just learning how to work with nftables, but I saved that link in case oneday I want to manage the iptables rules myself :)
Yes it is a security risk, but if you don’t have all ports forwarded, someone would still have to breach your internal network IIRC, so you would have many many more problems than docker.
I think from the dev’s point of view (not that it is right or wrong), this is intended behavior simply because if docker didn’t do this, they would get 1,000 issues opened per day of people saying containers don’t work when they forgot to add a firewall rules for a new container.
My problem with this, is that when running a public facing server, this ends up with people exposing containers that really, really shouldn’t be exposed.
Resulting in exposed services. Feel free to look at shodan or zoomeye, internet connected search engines, for exposed versions of this service. This service is highly dangerous to expose, as it gives people an in to your system via the docker socket.
So uh, I just spun up a vps a couple days ago, few docker containers, usual security best practices… I used ufw to block all and open only ssh and a couple others, as that’s what I’ve been told all I need to do. Should I be panicking about my containers fucking with the firewall?
Docker will have only exposed container ports if you told it to.
If you used -p 8080:80 (cli) or - 8080:80 (docker-compose) then docker will have dutifully NAT’d those ports through your firewall. You can either not do either of those if it’s a port you don’t want exposed or as @moonpiedumplings@programming.dev says below you can ensure it’s only mapped to localhost (or an otherwise non-public) IP.
Probably not an issue, but you should check. If the port opened is something like 127.0.0.1:portnumber, then it’s only bound to localhost, and only that local machine can access it. If no address is specified, then anyone with access to the server can access that service.
An easy way to see containers running is: docker ps, where you can look at forwarded ports.
Alternatively, you can use the nmap tool to scan your own server for exposed ports. nmap -A serverip does the slowest, but most indepth scan.
Just waking up, I’ve been running docker on my nas for a few years now and was never made aware of this - the nas ports appear safe, but the vps is not, so I swapped in 127.0.0.1 in front of the port number (so it’s now 127.0.0.1:8080:80 or what have you), and that appears to resolve it. I have nginx running so of course that’s how I want to have a couple things exposed, not everything via port.
My understanding was that port:port just was local for allowing redirection from container to host, and you’d still need to do network firewall management yourself to allow stuff through, and that appears the case on my home network, so I never had reason to question it. Thanks, I learned something today :)
Might do the same to my nas containers, too, just to be safe. I’m using those containers as a testbed for the vps containers so I don’t want to forget…
My solution to this has been to not forward the ports on individual services at all. I put a reverse proxy in front of them, refer to them by container name in the reverse proxy settings, and make sure they’re on the same docker network.
But… You literally have ports rules in there. Rules that expose ports.
You don’t get to grumble that docker is doing something when you’re telling it to do it
Dockers manipulation of nftables is pretty well defined in their documentation. If you dig deep everything is tagged and natted through to the docker internal networks.
As to the usage of the docker socket that is widely advised against unless you really know what you’re doing.
Dockers manipulation of nftables is pretty well defined in their documentation
Documentation people don’t read. People expect, that, like most other services, docker binds to ports/addresses behind the firewall. Literally no other container runtime/engine does this, including, notably, podman.
As to the usage of the docker socket that is widely advised against unless you really know what you’re doing.
Too bad people don’t read that advice. They just deploy the webtop docker compose, without understanding what any of it is. I like (hate?) linuxserver’s webtop, because it’s an example of the two of the worst footguns in docker in one
To include the rest of my comment that I linked to:
Do any of those poor saps on zoomeye expect that I can pwn them by literally opening a webpage?
No. They expect their firewall to protect them by not allowing remote traffic to those ports. You can argue semantics all you want, but not informing people of this gives them another footgun to shoot themselves with. Hence, docker “bypasses” the firewall.
On the other hand, podman respects your firewall rules. Yes, you have to edit the rules yourself. But that’s better than a footgun. The literal point of a firewall is to ensure that any services you accidentally have running aren’t exposed to the internet, and docker throws that out the window.
You originally stated:
I think from the dev’s point of view (not that it is right or wrong), this is intended behavior simply because if docker didn’t do this, they would get 1,000 issues opened per day of people saying containers don’t work when they forgot to add a firewall rules for a new container.
And I’m trying to say that even if that was true, it would still be better than a footgun where people expose stuff that’s not supposed to be exposed.
But that isn’t the case for podman. A quick look through the github issues for podman, and I don’t see it inundated with newbies asking “how to expose services?” because they assume the firewall port needs to be opened, probably. Instead, there are bug reports in the opposite direction, like this one, where services are being exposed despite the firewall being up.
(I don’t have anything against you, I just really hate the way docker does things.)
Sure, I get it, this stuff should be accessible for all. Easy to use with sane defaults and all that. But at the end of the day anyone wanting to using this stuff is exposing potential/actual vulnerabilites to the internet (via the OS, the software stack, the configuration, … ad nauseum), and the management and ultimate responsibility for that falls on their shoulders.
If they’re not doing the absolute minimum of R’ingTFM for something as complex as Docker then what else has been missed?
People expect, that, like most other services, docker binds to ports/addresses behind the firewall
Unless you tell it otherwise that’s exactly what it does. If you don’t bind ports good luck accessing your NAT’d 172.17.0.x:3001 service from the internet. Podman has the exact same functionality.
That assumes you’re on some VPS with a hardware firewall in front.
Often enough you’re on a dedicated server that’s directly exposed to the internet, with those iptables rules being the only thing standing between your services and the internet.
What difference does it make if you open the ports yourself for the services you expose, or Docker does it for you? That’s all that Docker is meant to do, act as convenience so you don’t have to add/remove rules as the containers go up/down, or remember Docker interfaces.
If by any chance you are making services listen on 0.0.0.0 and covering them up with a firewall that’s very bad practice.
You need to be able to have multiple nodes in one LAN access ports on each others’ containers without exposing those to the world and without using additional firewalls in front of the nodes.
That’s why kubernetes ended up removing docker support and instead recommends podman or using containerd natively.
You are not logged in. However you can subscribe from another Fediverse account, for example Lemmy or Mastodon. To do this, paste the following into the search field of your instance: !selfhosted@lemmy.world
A place to share alternatives to popular online services that can be self-hosted without giving up privacy or locking you into a service you don’t control.
Rules:
Be civil: we’re here to support and learn from one another. Insults won’t be tolerated. Flame wars are frowned upon.
No spam posting.
Posts have to be centered around self-hosting. There are other communities for discussing hardware or home computing. If it’s not obvious why your post topic revolves around selfhosting, please include details to make it clear.
Don’t duplicate the full text of your blog or github here. Just post the link for folks to click.
Submission headline should match the article title (don’t cherry-pick information from the title to fit your agenda).
No trolling.
Resources:
selfh.st Newsletter and index of selfhosted software and apps
There’s also rootless docker. There shouldn’t be any more firewall shenanigans.
CC BY-NC-SA 4.0
That’s another good reason to use podman, rules are on nft and separated from your rules.
Acronyms, initialisms, abbreviations, contractions, and other phrases which expand to something larger, that I’ve seen in this thread:
6 acronyms in this thread; the most compressed thread commented on today has 5 acronyms.
[Thread #589 for this sub, first seen 11th Mar 2024, 10:15] [FAQ] [Full list] [Contact] [Source code]
If you use firewalld, both docker and podman apply rules in a special zone separate from your main one.
That being said, podman is great. Podman in rootful mode, along with podman-docker and docker-compose, is basically a drop-in replacement for Docker.
I’m a podman user, but what’s the point of using podman if you are going to use a daemon and run it as root? I like podman so I can specifically avoid those things.
I am using it as a migration tool tbh. I am trying to get to rootless, but some of the stuff I host just don’t work well in rootless yet, so I use rootful for those containers. Meanwhile, I am using rootless for dev purposes or when testing out new services that I am unsure about.
Podman also has good integration into Cockpit, which is nice for monitoring purposes.
Is it? Last time I tried none of my docker compose files would start correctly in podman compose.
podman-compose is different from docker-compose. It runs your containers in rootless mode. This may break certain containers if configured incorrectly. This is why I suggested podman-docker, which allows podman to emulate docker, and the native docker-compose tool. Then you use
sudo docker-composeto run your compose files in rootful mode.How is Podman rootful better than Docker? I was mostly attracted by the rootless path, but the breakage deterred me. Would you be so kind to tell me ?
It isn’t that much better. I use it as drop-in docker replacement. It’s better integrated with things like cockpit though and the idea is that it’s easier to eventually migrate to rootless if you’re already in the podman ecosystem.
Ok that sounds intetesting, I’ve found Cockpit easier to use than Proxmox, I’m new to virtualization and I don’t want do nesting… I fear it will complicate things when I’ll need to do GPU passthrough.
How is Podman integrated into Cockpit?
Also, I had so much trouble trying to bridge my Home Assistant VM to my LAN. Are there any tutorials on how to do this from Cockpit?
Your containers show up in Cockpit under the “Podman containers” section and you can view logs, type commands into their consoles, etc. You can even start up containers, manage images, etc.
I have not done this personally, but I would assume you need to create a bridge device in Network Manager or via Cockpit and then tell your VM to use that. Keep in mind, bridge devices only work over Ethernet.
Yes, I want to reach my HA VM from my LAN connected devices.
deleted by creator
Wow, thanks for the heads up.
Looks like it affects dockerd, but not docker desktop.
Any idea of the docker implementation in Proxmox or TrueNAS? (TrueNAS does containers if I remember right?)
Correct me if I’m wrong, but I don’t think Proxmox uses Docker. I’m pretty sure its containers are LXC containers.
Correct
Oh, yea, you’re right. Thanks
ofcourse docker desktop is not affected because it’s a vm running docker running on your computer. the nftables inside the container are modified by docker
It doesn’t nuke your rules. Just ads to them.
How come I don’t see my previous rules when I dump the ruleset, then? I have my rules written in
/etc/nftables.conf, and they were previously applied by running# nft -f /etc/nftables.conf. Now, when I dump the current ruleset with# nft list ruleset, those previous rules aren’t there — all I see are Docker’s rules.You can use a bridge network or the host network.
In bridge network it is like a NAT host. With its own firewall settings.
In host network mode, it will just open the port it needs.
I could be misunderstanding your comment, but you don’t seem to have answered my question of why I don’t see my rules anymore.
Yeah, it needs those rules for e.g. port-forwarding into the containers.
But it doesnt really ‘nuke’ existing ones.
I have simply placed my rules at higher priority than normal. Very simple in nftables and good to not have rules mixed between nftables and iptables in unexpected ways.
You should filter as early as possible anyways to reduce ressource usage on e.g. connection tracking.
How come I don’t see my previous rules when I dump the ruleset, then? I have my rules written in
/etc/nftables.conf, and they were previously applied by running# nft -f /etc/nftables.conf. Now, when I dump the current ruleset with# nft list ruleset, those previous rules aren’t there — all I see are Docker’s rules.So better put Docker in a VM so it can’t do any harm to the host?
Lol.
It is enough to put it into its own network namespace.
You can somehow change that behavior: Prevent docker from manipulating iptables
This is standard, but often unwanted, behavior of docker.
Docker creates a bunch of chain rules, but IIRC, doesn’t modify actual incoming rules (at least it doesn’t for me) it just will make a chain rule for every internal docker network item to make sure all of the services can contact each other.
Yes it is a security risk, but if you don’t have all ports forwarded, someone would still have to breach your internal network IIRC, so you would have many many more problems than docker.
I think from the dev’s point of view (not that it is right or wrong), this is intended behavior simply because if docker didn’t do this, they would get 1,000 issues opened per day of people saying containers don’t work when they forgot to add a firewall rules for a new container.
Option to disable this behavior would be 100x better then current, but what do I know lol
How come I don’t see my previous rules when I dump the ruleset, then? I have my rules written in
/etc/nftables.conf, and they were previously applied by running# nft -f /etc/nftables.conf. Now, when I dump the current ruleset with# nft list ruleset, those previous rules aren’t there — all I see are Docker’s rules.Prevent docker from manipulating iptables
Don’t know what it’s actually doing, I’m just learning how to work with nftables, but I saved that link in case oneday I want to manage the iptables rules myself :)
Good luck. Your going to have to change the rules whenever the up address of the container changes.
If you are talking about the IP address then just add a static address, no? I do it anyway in my docker compose:
I’m not an expert so maybe I’m wrong, if so do not hesitate to correct me !
EDIT: If the IP address doesn’t change, you do not need to change to routing and iptables/nftables rules. ??
My problem with this, is that when running a public facing server, this ends up with people exposing containers that really, really shouldn’t be exposed.
Excerpt from another comment of mine:
It’s only docker where you have to deal with something like this:
Originally from here, edited for brevity.
Resulting in exposed services. Feel free to look at shodan or zoomeye, internet connected search engines, for exposed versions of this service. This service is highly dangerous to expose, as it gives people an in to your system via the docker socket.
So uh, I just spun up a vps a couple days ago, few docker containers, usual security best practices… I used ufw to block all and open only ssh and a couple others, as that’s what I’ve been told all I need to do. Should I be panicking about my containers fucking with the firewall?
Docker will have only exposed container ports if you told it to.
If you used
-p 8080:80(cli) or- 8080:80(docker-compose) then docker will have dutifully NAT’d those ports through your firewall. You can either not do either of those if it’s a port you don’t want exposed or as @moonpiedumplings@programming.dev says below you can ensure it’s only mapped to localhost (or an otherwise non-public) IP.Thanks - more detailed reply below :)
Actually, ufw has its own separate issue you may need to deal with. (Or bind ports to localhost/127.0.0.1 as others have stated.)
Probably not an issue, but you should check. If the port opened is something like
127.0.0.1:portnumber, then it’s only bound to localhost, and only that local machine can access it. If no address is specified, then anyone with access to the server can access that service.An easy way to see containers running is:
docker ps, where you can look at forwarded ports.Alternatively, you can use the
nmaptool to scan your own server for exposed ports.nmap -A serveripdoes the slowest, but most indepth scan.Just waking up, I’ve been running docker on my nas for a few years now and was never made aware of this - the nas ports appear safe, but the vps is not, so I swapped in
127.0.0.1in front of the port number (so it’s now127.0.0.1:8080:80or what have you), and that appears to resolve it. I have nginx running so of course that’s how I want to have a couple things exposed, not everything via port.My understanding was that port:port just was local for allowing redirection from container to host, and you’d still need to do network firewall management yourself to allow stuff through, and that appears the case on my home network, so I never had reason to question it. Thanks, I learned something today :)
Might do the same to my nas containers, too, just to be safe. I’m using those containers as a testbed for the vps containers so I don’t want to forget…
My solution to this has been to not forward the ports on individual services at all. I put a reverse proxy in front of them, refer to them by container name in the reverse proxy settings, and make sure they’re on the same docker network.
But… You literally have ports rules in there. Rules that expose ports.
You don’t get to grumble that docker is doing something when you’re telling it to do it
Dockers manipulation of nftables is pretty well defined in their documentation. If you dig deep everything is tagged and natted through to the docker internal networks.
As to the usage of the docker socket that is widely advised against unless you really know what you’re doing.
Documentation people don’t read. People expect, that, like most other services, docker binds to ports/addresses behind the firewall. Literally no other container runtime/engine does this, including, notably, podman.
Too bad people don’t read that advice. They just deploy the webtop docker compose, without understanding what any of it is. I like (hate?) linuxserver’s webtop, because it’s an example of the two of the worst footguns in docker in one
To include the rest of my comment that I linked to:
You originally stated:
And I’m trying to say that even if that was true, it would still be better than a footgun where people expose stuff that’s not supposed to be exposed.
But that isn’t the case for podman. A quick look through the github issues for podman, and I don’t see it inundated with newbies asking “how to expose services?” because they assume the firewall port needs to be opened, probably. Instead, there are bug reports in the opposite direction, like this one, where services are being exposed despite the firewall being up.
(I don’t have anything against you, I just really hate the way docker does things.)
Sure, I get it, this stuff should be accessible for all. Easy to use with sane defaults and all that. But at the end of the day anyone wanting to using this stuff is exposing potential/actual vulnerabilites to the internet (via the OS, the software stack, the configuration, … ad nauseum), and the management and ultimate responsibility for that falls on their shoulders.
If they’re not doing the absolute minimum of R’ingTFM for something as complex as Docker then what else has been missed?
Unless you tell it otherwise that’s exactly what it does. If you don’t bind ports good luck accessing your NAT’d 172.17.0.x:3001 service from the internet. Podman has the exact same functionality.
That assumes you’re on some VPS with a hardware firewall in front.
Often enough you’re on a dedicated server that’s directly exposed to the internet, with those iptables rules being the only thing standing between your services and the internet.
What difference does it make if you open the ports yourself for the services you expose, or Docker does it for you? That’s all that Docker is meant to do, act as convenience so you don’t have to add/remove rules as the containers go up/down, or remember Docker interfaces.
If by any chance you are making services listen on 0.0.0.0 and covering them up with a firewall that’s very bad practice.
There’s no alternative for 0.0.0.0 and a firewall if you’re e.g. using kubernetes.
I’m fairly sure you can find an alternative to whatever problem you’re having.
You need to be able to have multiple nodes in one LAN access ports on each others’ containers without exposing those to the world and without using additional firewalls in front of the nodes.
That’s why kubernetes ended up removing docker support and instead recommends podman or using containerd natively.