Brian Lovin
/
Hacker News
Daily Digest email

Get the top HN stories in your inbox every day.

SamuelAdams

Wait until they learn about Docker ignoring iptable rules.

https://www.baeldung.com/linux/docker-container-published-po...

jwitthuhn

This bit me the first time I deployed using docker.

Everything but 80/443 blocked by the firewall. "Surely it is safe to run my app server on port 8000 because no one can access that port externally."

Docker stepped in to help by making sure people could access that port.

justsomehnguy

My favourite part in Docker is what if you have 127.0.0.1 configured as your resolver (eg you are running Unbound) then Docker helpfully configure containers to use 1.1.1.1 instead.

Thanks, my ass, and give back my time trying to diagnose [a container with no diagnostic tools, because it's a lightweight container] why it can't connect to someservice.on.my.lan.

kccqzy

Nobody should ever mess with a system's resolver settings.

I've recently had to debug a bug with a container (probably not Docker itself) unable to reach the internet. It turns out the machine is behind a NAT64 and has IPv6 addresses only, yet something in the container helpfully reconfigures the resolver to be 8.8.8.8 which of course is unreachable.

judge2020

Wouldn't 127.0.0.1 be the loopback for the container itself? So you'd need to set the DNS to the gateway IP or dynamically by resolving `host.docker.internal` (assuming `--add-host=host.docker.internal:host-gateway`)

teekert

This is such scary stuff. I love the ease of docker (compose) but stuff like this makes me miss the old days. No surprises. Now you can deploy a huge stack of stuff but it may work differently from expectations.

ActorNightly

Its scary stuff because the common theme of software dev/deployment is to follow common patterns without actually understanding what goes on under the hood.

michaelcampbell

That issue bit me with a dockerized mysql db. I think because I was not familiar with `ufw` at the time that after I set it up I tested to ensure you couldn't get to the db.

Surprise enused.

hackyhacky

I agree this is surprising behavior. Other comments have placed blame at the feet of Docker, iptables, or Ufw. However the easiest workaround is to tell docker to publish the ports ONLY on 127.0.0.1, making them inaccessible from elsewhere:

- ports: 127.0.0.1:8000:8000

heywoodlh

I have found that another long-term solution is to switch to rootless Docker[0] or Podman for local dev. That way iptables isn't mangled with because you don't elevate to a privileged user to use it. Not being privileged has some downsides, but I think the tradeoffs are worth it. Docker Desktop and Rancher Desktop also is great on Linux for this reason, although, they both require a desktop environment.

(Not disagreeing with your point -- yours is a great/quick workaround -- it just seems that many people don't know about rootless Docker)

[0] https://docs.docker.com/engine/security/rootless/

jvda

I don‘t get this. What‘s the point of publishing them on some port if you don‘t that to be.. well, public?

berkle4455

Interesting. I guess I've been lucky as I use externally managed stateful and stateless rules (aws security groups + network acls)

yrro

Developers learning how their system actually works? I'll believe it when I see it... ;)

More seriously, I believe nftables has the capacity to help here: now there can be several chains which all attach to the same hook. So docker can put rules into the FORWARD base chain of the filter table (belonging to iptables, mine is empty because I don't use iptables):

    # nft list chain ip filter FORWARD
    table ip filter {
            chain FORWARD {
                    type filter hook forward priority filter; policy accept;
            }
    }
... but then the filter_FORWARD base chain of the firewalld table (belonging to firewalld) _also_ gets to process packets. Note the numerically higher filter priority, which means that this chain runs later:

    # nft list chain inet firewalld filter_FORWARD
    table inet firewalld {
            chain filter_FORWARD {
                    type filter hook forward priority filter + 10; policy accept;
                    ct state { established, related } accept
                    ct status dnat accept
                    iifname "lo" accept
                    ip6 daddr { ::/96, ::ffff:0.0.0.0/96, 2002::/24, 2002:a00::/24, 2002:7f00::/24, 2002:a9fe::/32, 2002:ac10::/28, 2002:c0a8::/32, 2002:e000::/19 } log prefix "RFC3964_IPv4_REJECT: " reject with icmpv6 type addr-unreachable
                    jump filter_FORWARD_POLICIES_pre
                    jump filter_FORWARD_IN_ZONES
                    jump filter_FORWARD_OUT_ZONES
                    jump filter_FORWARD_POLICIES_post
                    ct state { invalid } meta pkttype host log prefix "STATE_INVALID_DROP: "
                    ct state { invalid } drop
                    meta pkttype host log prefix "FINAL_REJECT: "
                    reject with icmpx type admin-prohibited
            }
    }
Now if only tools would learn to stay in their lane, cease assuming they are the only tool managing the nftables state, we'd have ufw managing its own table, docker managing its own table, firewalld managing its own...

Muromec

But docker doesn't ignore iptables, it adds iptables rule to forward packets to docker table in iptables.

yjftsjthsd-h

Okay, docker doesn't ignore firewall rules, it bypasses them. Still annoying.

jeroenhd

I don't think that's fair. Linux users misunderstand nftables/iptables and UFW pretends to have complete control when it really doesn't.

It's annoying, but the fault lies squarely with the lack of reporting of other chains in UFW.

drdaeman

Set `{"iptables": false}` in daemon.json (or pass `--iptables=false` to the daemon) and it won't do it. The defaults are convenient enshittification for your casual security-ignorant developer.

masom

It does not bypasses them, it uses the firewall to add forwarding rules to the container. If you read https://www.baeldung.com/linux/docker-container-published-po... there's a bunch of ways to prevent this from happening.

By enabling port forwarding and exposing, you're essentially asking docker to configure the host for it. It's likely a surprising behaviour, but seems like ufw has a bug if you have multiple chains and groups instead of just the default one.

ilyt

Not really. Just ufw wasn't made to deal with it.

TacticalCoder

Several containers and VMs by default just go a bit "overboard" I'd say, adding network interfaces and firewalling rules left and right. It takes people installing these for the first time by surprise.

yrro

It would be fairer to say that docker rudely stomps all over your iptables configuration. ;)

ilyt

It's probably ufw putting its rules in INPUT and OUTPUT chain and docker needing them in FORWARD.

Don't blame ufw incompetence on iptables.

There is separate issue with bridging aptly described here

https://wiki.libvirt.org/Net.bridge.bridge-nf-call_and_sysct...

but IIRC docker doesn't work in that config by default.

IIRC Centos/RHEL comes with bridging iptables pre-broken because they're bad at making firewall rules there.

gnfargbl

Not just docker, either. If you configure a NodePort on your kubernetes cluster, you'll run into exactly the same issue: ufw-sourced iptables rules are overriden by kubernetes-sourced rules.

Here's the only fix I was able to find, which doesn't seem well documented. Run

    kubectl -n kube-system edit configmap kube-proxy
and edit in this:

    nodePortAddresses:
      - 192.168.0.0/16
...where the CIDR block should correspond to the local network on your machine.

bandyaboot

They can learn about what Docker does with iptables the traditional way, by reading Docker’s networking documentation.

https://docs.docker.com/network/packet-filtering-firewalls/

It says explicitly that docker and ufw should be considered incompatible. Docker also has a configuration key to prevent it from modifying iptables.

Dylan16807

People only have so much time to read documentation. You can't expect them to read an entire textbook before they start using Docker.

pests

And people wonder why we aren't taken seriously like other engineering fields.

Imagine a civil engineer telling someone they don't have time to read the documentation on the bolts their using to hold up a bridge.

zelphirkalt

Isn't that simply because Docker adds its own firewall rules, in order to realize its docker networks?

yomlica8

Can anyone recommend a decent book on linux firewalls, iptables and the like? Every time I wade into this I feel I'm missing to much base knowledge to make good decisions.

dharmab

A classic on *nix networking fundamentals: https://beej.us/guide/bgnet/

It includes a further reading list: https://beej.us/guide/bgnet/html/split/more-references.html#...

These don't cover iptables and other firewalls themselves, but they give you enough knowledge that you can read the iptables manpage and other manuals and understand them.

wjholden

Do you specifically want to learn networking for Linux? If not, the Network+ and/or CCNA certifications are a great place to start for generic network education.

CoastalCoder

I was about to ask a similar question.

I don't know much about Linux networking. But recently I dove into it a little to let my laptop connect to my desktop via Ethernet (when present), but use Wifi for everything else.

I got it working-ish using "nmtui". But I'm left pretty confused about the relationship between all the network-related tools / services / files.

E.g., is "nmtui" just a convenience wrapper around thinks like iptables and resolved? Does it work around them? Which tools are mean to be used together, vs. which ones are redundant / incompatible? And then there's systemd as well.

vaylian

nmtui is a terminal user interface (tui) to control Network Manager (nm). Network Manager is a complete solution for networking that takes care of all the individual parts of networking. But as far as I know it does not change your firewall rules. You would need to use iptables for that directly, because this is not something that can be reasonably automated without actually knowing what the user wants.

systemd is an init system with a lot of bells and whistles. One of the whistles is systemd-networkd, which is also a complete networking solution, but it also delegates some work to systemd-resolved. It also doesn't take care of firewalling.

If you want to get a better understanding on how networks are set up, then you should rather look for more low-level tools like iproute2 (which provides the "ip" tool) and dhclient/dhcpd. For wireless you should look at wpa_supplicant (which is afaik used by Network Manager under the hood). And then there is of course iptables which is used to manipulate the flow of packages through the network. It should be noted that eBPF is intended to replace iptables but knowledge of iptables will still prove useful for many years to come.

seba_dos1

nmtui is just a console UI for NetworkManager. Similar UI is most likely provided by your desktop environment of choice as well. It has nothing to do with iptables.

CoastalCoder

Thanks, that's helpful.

Any suggestion for something explaining how all these tools and service fit together (or don't)?

rfmoz

Linux Firewalls: Enhancing Security with Nftables and Beyond

tambourine_man

It’s these kinds of things that makes me realize I don’t really know what I’m doing regarding networks. I would never have imagined.

Even FreeBSD’s stack, which was always much more straightforward to me, behaves like this, it seems. There’s no hope.

ktm5j

It's one of those things where the more I learn just makes me better realize how little I understand.

tambourine_man

Can you imagine trying to debug this? I pity the poor soul. Hopefully all this debate will raise it to the heights of Google search results.

binkHN

FWIW, this is the same behavior on OpenBSD—DHCP listens directly on bpf, which sees traffic before the packet filter.

le-mark

That is a surprising revalation; that iptables filters traffic depending on Linux implementations details. One could imagine the outcry if firewall vendor X suffered a similar “feature”. Or is this well known for Linux iptable users?

zamadatix

If you used Linux like you would a true firewall and filtered a bridged or routed packet it should filter fine. It's really an interaction of giving low level network access to an application on the same box you're trying to do high level filtering on, then being surprised the high level filter misses the low level data.

I've never liked the way packet sockets are exposed on any operating system though. Exposing them is such an afterthought that the only way to use them is to basically act like the rest of the networking system plain doesn't exist. I shouldn't have to have raw network permissions to send and receive any packet just to be able to mark that I want to send and receive e.g. LLDP (or, on Windows, make a driver that even allows me a way to send such packets from user space in the first place). Operating systems truly offer "TCP/IP" (and UDP and maybe a few other select protocols, depending what you load) stacks not "network stacks" which give you access to each piece equally. Even plain raw IP sockets are increasingly ignored.

/rant of a network guy.

NoZebra120vClip

> If you used Linux like you would a true firewall

Host-based firewalls are, indeed, fake firewalls, and it's unfortunate that they chose to use the same term for functions that superficially look like they do the same thing, but are way less effective, and encourage bad network design.

The classic architecture for security was: edge router -> firewall -> DMZ -> bastion -> DMZ -> office router

You could put more firewalls, and switches and stuff, in there, but it is really important to have a bastion host as a chokepoint in the DMZ. You could have an IPS doing this instead. Or you could have a proxy server such as SOCKS.

But your firewall is pretty far away from any host that could get bright ideas about messing with it. (Because it is possible for client hosts to mess with router "firewall" settings via UPnP.)

This collapsing of topology also happens with other stuff, like NIDS/HIDS intrusion detection. Yeah, IDS can be more effective in certain ways when it has access to host-based logs and resources. But it's different than an isolated IDS listening on a TAP/SPAN where nobody logs into it and doomscrolls Facebook.

chrisshroba

What is the DMZ, physically? I've always heard of putting your bastion in a DMZ but is that a conceptual layer or some sort of physical box sitting between the firewall and the bastion, and the bastion and the office router?

tptacek

Acting like the whole networking stack doesn't exist is the whole purpose of packet sockets, because sometimes you want to override the things the networking stack does.

You need privileges to open a raw socket (or a packet socket, or anything else that would let you program at the IP layer) because otherwise unprivileged processes could hijack traffic for other applications on the system.

zamadatix

Packet sockets as-is are a great option if that's your use case but if I just want to bind to ethertype 0x88CC I shouldn't need to ask for the same permissions I can use to bind to 0x0800 or to capture all traffic. Sure, that's how things like AF_PACKET are implemented but there is nothing about being lower level that requires this all or nothing game of read/write-full-access-to-all-packets-ignoring-the-rest-of-the-os-stack-state vs bind-a-single-TCP/UDP-port. Just look at the way shared binds and privileged ports are handled for TCP, there is no reason the same concepts couldn't be applied to lower layers for more granularity.

All of that aside, none of this would explain why I need to write a custom network driver in Windows to send more than a couple predefined ethertypes from user space at all or why, in Linux, the network filter layer can't apply to any family used in the sendto() syscall instead of just sendto() calls with AF_INET set. These kinds of decisions aren't rooted in a limited scope of what packet sockets can be for or how they must interact with the rest of the network stack, it's just how they are currently built and exposed. This is great if your use case is to ignore the OS stack, but that doesn't mean doing so is the only conceivable way of packet sockets being built.

If BPF were treated slightly differently in terms of how/when certain abilities were exposed it could be a great answer to all of this. As of right now, it's just a better way to ignore the OS network stack even more than normal though.

smashed

Iptables (aka linux's netfilter) processes DHCP packets like any other packets.

The ISC DHCP server though listens for raw packets and thus completely by-pass the netfilter rules.

This is similar to how you can use wireshark to see raw packets received on the physical port, before any filtering.

Any Linux process running with CAP_NET_RAW can by-pass the firewall in such a way, this includes your typical DHCP server running as root.

The question then should be why is ISC DHCP server using raw sockets? That is probably because DHCP sits in-between OSI layers, it bridges the gap between the mac address world and IP address world.

I'm not sure of the exact technical reason though. The linked SO answers talk about some case where NAT rules could be altering packets, not sure how common NAT+DHCP is used...

In the case of a DHCP client, you do need raw sockets because the Linux IP layer will not let a normal socket send packets with a NULL IP source address.

allarm

> The question then should be why is ISC DHCP server using raw sockets?

I don’t know much about Linux network stack, but probably it’s because DHCPDISCOVER messages sent to ff:FF:FF:FF:FF:FF, and Linux network driver only accepts frames sent to the interface address? Not sure if that’s true though.

smashed

This is just a normal udp broadcast packet sent to FF:FF:FF:FF:FF:FF / 255.255.255.255. You do not need raw sockets to receive it.

There is an ISC dhcp article here with more info on why they need it: https://kb.isc.org/docs/aa-00379

But I just thought about it some more.. In a normal unix socket, you don't set the destination mac address when sending a packet. You just set the destination IP address and the kernel figures out the destination MAC address using a static, cached or dynamic ARP lookup.

But since the dhcp client has not been assigned the IP yet, ARP would fail.

The DHCP server would need to set a static ARP entry before sending the response, and hope the kernel properly fills the destination MAC. This causes more problems..

Its much easier and reliable to just craft the exact headers required as-per the RFC, especially in a cross-platform piece of software like the ISC dhcp server.

ilyt

It's the fact that TCP/UDP socket interface doesn't expose macaddress, dnsmasq IIRC does the same

russdill

It's well known that iptables operates on the normal sockets layer. It would similarly be surprising behavior if you could not see packets that are present but dropped when running tcpdump. Note that this distinction only applies to packets handled locally, not forwarded.

Applications such as tcpdump and dhcp require special privileges to open raw sockets. Note that ebtables (and now by extension nftables) can be used to operate at this level.

pravus

> That is a surprising revalation; that iptables filters traffic depending on Linux implementations details. One could imagine the outcry if firewall vendor X suffered a similar “feature”. Or is this well known for Linux iptable users?

All of these major Linux firewall features have been around for over two decades and the use of multi-stage rule routing with filters is day-to-day for anyone in network ops. Cisco switches I messed with in the 90s had DHCP helpers for relaying packets across VLANs among other similar features. FTP helpers were extremely common before SSL/TLS/sftp became standard due to NAT and how the port directions work in that protocol.

zokier

iptables (well, netfilter at least) is part of Linux, so obviously it depends on Linux implementation details. Or rather its all Linux implementation details and nothing else.

allanrbo

Even within iptables / nftables / nft there's a ton of places to hook in. I always need to take a long hard look at this diagram to get it right: https://en.wikipedia.org/wiki/Iptables#/media/File:Netfilter...

Joel_Mckay

if you have systemd/Netplan/docker, than expect chaotic firewall states.

Each can drop some nasty use-case specific assumptions that cause odd issues in other areas.

Happy computing =)

ilyt

that has nothing to do with any single of those tools, just the fact of trying to manage firewall by more than one tool at once.

Netplan technically could solve it but I have zero trust in Ubuntu not fucking it up or abandoning it in few years. And starting with YAML is already begging to fail

Joel_Mckay

All mentioned items have side-channel borked firewall and route rules in the past. Some bugs intermittently silently block local daemon instances from (re)loading like magic (some bugs only happen when the system is brought up).

If your daily tasks include something less borked, than consider yourself very lucky you live without systemd. If I recall, ufw was intended for simple workstation rule sets.

Personally, for home stuff I tend to use a heavily customized rule-set that interoperates with fail2ban. And a very old repeatably stable approach to setting up the interfaces from a known default state...

https://shorewall.org/

Best of luck, =)

ilyt

I remember hating shorewall and similar ones because, well, I know iptables, and I know exactly what I want so using anything that tries to abstract it into it's own approach is torture as I need to take the rules I want and translate it to whatever mediocre paradigm shorewall (or ufw, or near-any other firewall manager in the wild) decided to put on top of iptables.

I ended up using ferm http://ferm.foo-projects.org/ which is basically a convenience layer over iptables, the keywords are named the same and the rules map nearly 1:1 and the changes of mapping are essentially macro and variable expansion. So it's basically iptables but a lot of tedium removed.

Our biggest one is around 1.5k rules and very manageable, using ferm with rule files generated via Puppet. Every entry gets a comment allowing us to track where it came from too.

> If your daily tasks include something less borked, than consider yourself very lucky you live without systemd. If I recall, ufw was intended for simple workstation rule sets.

Systemd has little to do with any of that

josephcsible

For people who wish this weren't the case, would you want tools like tcpdump and Wireshark to also only see packets that the firewall allows? If not, then what change would you propose exactly that changes the former without breaking the latter?

xuhu

Distinct socket types for monitoring vs receiving/sending.

m463

I've just disabled those exceptional rules when dhcp isn't required.

I'm wondering if this is:

- a default configuration situation (which I think I rectified)

- DHCP exception hardcoded in the firewall binaries

anfractuosity

Intriguing, so there's no way to block DHCP from Linux at all as all firewalls such as ufw/nftables/iptables, would use netfilter behind the scenes?

ilyt

You could probably via ebtables as they inspect ethernet frames directly. That can be used for example to not allow VMs on host to spoof mac addresses.

But easiest way is just don't allow app to run with permission to access raw socket. That's it.

The problem is really that there is no way to get UDP interface that also gives you mac address of the packet so raw sockets are only way to do it.

Similarly there is no interface to send ICMP packets other than raw sockets.

jeroenhd

Not through iptables, and probably not through nftables (though I can't find much documentation on nftables).

eBPF should still work, though. You can also configure iptables to filter based on a bpf program, combining the two. Here's an example: https://github.com/Asphaltt/iptables-bpf

anfractuosity

Thanks a lot, will have a look at your link.

ilyt

ebtables can, it inspects ethernet frames.

tenebrisalietum

DHCP relies on Ethernet broadcasts to function - meaning DHCP messages are received by every NIC on the subnet. So ... already not private.

PF_PACKET is needed to look at those broadcasts because the system might not have an IP, so it can't use TCP or UDP sockets.

PF_PACKET on Linux evidently ignores iptables.

Good news: PF_PACKET requires root to use (more precisely, CAP_NET_RAW capability).

So root processes can totally ignore your firewall. This doesn't matter because:

- a firewall is really for managing external communications. If you have stuff running on localhost sending or receiving unwanted stuff, and you don't trust it, why is it running on your machine then?

- root can already simply disable the firewall by removing iptables rules or adding ones.

You can always move to IPv6 which uses multi-cast and self-generated link-local addresses, meaning PF_PACKET isn't necessary.

justsomehnguy

> are received by every NIC on the subnet

On the brodcast segment, usually L2 VLAN.

/nitpick

yrro

The netdev and bridge tables' hooks run early enough I think?

lathiat

Right: https://wiki.nftables.org/wiki-nftables/index.php/Nftables_f...

"The netdev family is different from the others in that it is used to create base chains attached to a single network interface. Such base chains see all network traffic on the specified interface, with no assumptions about L2 or L3 protocols. Therefore you can filter ARP traffic from here. There is no legacy x_tables equivalent to the netdev family."

globular-toast

What does it even mean to block DHCP? A firewall sits between networks and filters what gets passed between them. Linux can absolutely block DHCP packets being passed between networks. But it has to see them itself, obviously. So you don't want other processes being able to see them too? Well don't run those processes. Why on earth would you run a DHCP server that you hope can never receive any requests?!

lathiat

Whether or not it's sensible is not quite the point. It's surprising and noteworthy to understand. And by understanding helps understand the internals of this aspect of linux networking.

I ran into exactly this problem a few months ago. But it was a lab scenario rather than a real production one.

In my lab at work I had only a single VLAN (and couldn't get a new one added fast enough for the tight 2 day timeline of the case I was working on, though I have since done so). So I had "MAAS" (Metal As a Service) which PXE boot installs the servers plus OpenStack Virtual Machines connected to the same VLAN. Both run a DHCP service.

MAAS (using ISC DHCP Server currently) responds to any DHCP requests while OpenStack only responds to IP/MAC combinations it knows from it's configuration database and ignores others. So OpenStack wasn't impacting MAAS but sometimes MAAS was responding faster than OpenStack giving the wrong IP to VMs. As a temporary workaround I (tried) to add a firewall rule to the MAAS server to block DHCP from the specific MAC addresses of the test VMs but it didn't work.

Investigating that is when I learnt about this surprise behaviour, which was particularly surprising as I am far more knowledgable than average of many aspects of linux networking including some (but by no means extensive) kernel/netfilter internals and can write most iptables rules in my sleep from memory... despite all of that, this one was new to me :)

(As a side note, when using ovn/ovs networking, Openstack will filter external DHCP requests to VMs, which also solves this, but in this case I was specifically testing SR-IOV where the VMs are directly attached to a virtual function on the physical network card, where such filtering is not configured and may or may not be possible depending on the card)

timost

This DHCP behaviour also happens on the client side

jas-

ARP? Don’t you need arptables for this? OSI model 101

Daily Digest email

Get the top HN stories in your inbox every day.

DHCP is not blocked by ufw/iptables - Hacker News