Exits with "unable to detect hairpin mode (is the docker daemon running?)"

[Rycieos]

Version: v0.4.3 docker.
Docker version: 20.10.1 and 20.10.2
OS: CentOS Linux release 8.3.2011 (Core)

After a system update, upon launching I get this error:

$ docker logs ipv6nat
2021/01/09 17:26:57 unable to detect hairpin mode (is the docker daemon running?)

After which the container exits and restarts.

Thinking it might be a permissions issue, I removed all --cap-adds, leaving only the --cap-drop ALL to test, but that broke it more:

2021/01/09 18:07:38 running [/sbin/iptables -t nat -C OUTPUT -m addrtype --dst-type LOCAL -j DOCKER --wait]: exit status 3: addrtype: Could not determine whether revision 1 is supported, assuming it is.
addrtype: Could not determine whether revision 1 is supported, assuming it is.
iptables v1.8.4 (legacy): can't initialize iptables table `nat': Permission denied (you must be root)
Perhaps iptables or your kernel needs to be upgraded.

I then tried to give it --cap-add ALL, but that did not fix it.

Since part of the system update was docker-ce, I thought maybe it had changed the backend rules, but:

# /sbin/iptables-save -t nat
# Generated by iptables-save v1.8.4 on Sat Jan  9 13:09:03 2021
*nat
...
-A OUTPUT ! -d 127.0.0.0/8 -m addrtype --dst-type LOCAL -j DOCKER
...

Clearly the right rule still exists. And checking manually:

# /sbin/iptables -t nat -C OUTPUT -m addrtype --dst-type LOCAL -j DOCKER --wait; echo "$?"
iptables: Bad rule (does a matching rule exist in that chain?).
1
# /sbin/iptables -t nat -C OUTPUT ! -d 127.0.0.0/8 -m addrtype --dst-type LOCAL -j DOCKER --wait; echo "$?"
0

The actual checking commands returns correctly as expected. I am using this code section as the reference: https://github.com/robbertkl/docker-ipv6nat/blob/v0.4.3/manager.go#L79-L86

At this point I downgraded dockerd back to 20.10.1, but I got the same error.

What is strange is that when I first did the system upgrade, dockerd restarted itself as usual, and all my containers came back online with IPv6 working. It was after an OS restart that this error started.

I tried to do a system rollback, but the old package versions couldn't be found, so I'm stuck.

Full package list that I upgraded:

Package	New Version	Old Version
NetworkManager	1:1.26.0-12.el8_3.x86_64	1:1.26.0-9.el8_3.x86_64
NetworkManager-libnm	1:1.26.0-12.el8_3.x86_64	1:1.26.0-9.el8_3.x86_64
NetworkManager-team	1:1.26.0-12.el8_3.x86_64	1:1.26.0-9.el8_3.x86_64
NetworkManager-tui	1:1.26.0-12.el8_3.x86_64	1:1.26.0-9.el8_3.x86_64
gnutls	3.6.14-7.el8_3.x86_64	3.6.14-6.el8.x86_64
iptables	1.8.4-15.el8_3.3.x86_64	1.8.4-15.el8.x86_64
iptables-ebtables	1.8.4-15.el8_3.3.x86_64	1.8.4-15.el8.x86_64
iptables-libs	1.8.4-15.el8_3.3.x86_64	1.8.4-15.el8.x86_64
iptables-services	1.8.4-15.el8_3.3.x86_64	1.8.4-15.el8.x86_64
iwl100-firmware	39.31.5.1-101.el8_3.1.noarch	39.31.5.1-99.el8.1.noarch
iwl1000-firmware	1:39.31.5.1-101.el8_3.1.noarch	1:39.31.5.1-99.el8.1.noarch
iwl105-firmware	18.168.6.1-101.el8_3.1.noarch	18.168.6.1-99.el8.1.noarch
iwl135-firmware	18.168.6.1-101.el8_3.1.noarch	18.168.6.1-99.el8.1.noarch
iwl2000-firmware	18.168.6.1-101.el8_3.1.noarch	18.168.6.1-99.el8.1.noarch
iwl2030-firmware	18.168.6.1-101.el8_3.1.noarch	18.168.6.1-99.el8.1.noarch
iwl3160-firmware	1:25.30.13.0-101.el8_3.1.noarch	1:25.30.13.0-99.el8.1.noarch
iwl3945-firmware	15.32.2.9-101.el8_3.1.noarch	15.32.2.9-99.el8.1.noarch
iwl4965-firmware	228.61.2.24-101.el8_3.1.noarch	228.61.2.24-99.el8.1.noarch
iwl5000-firmware	8.83.5.1_1-101.el8_3.1.noarch	8.83.5.1_1-99.el8.1.noarch
iwl5150-firmware	8.24.2.2-101.el8_3.1.noarch	8.24.2.2-99.el8.1.noarch
iwl6000-firmware	9.221.4.1-101.el8_3.1.noarch	9.221.4.1-99.el8.1.noarch
iwl6000g2a-firmware	18.168.6.1-101.el8_3.1.noarch	18.168.6.1-99.el8.1.noarch
iwl6050-firmware	41.28.5.1-101.el8_3.1.noarch	41.28.5.1-99.el8.1.noarch
iwl7260-firmware	1:25.30.13.0-101.el8_3.1.noarch	1:25.30.13.0-99.el8.1.noarch
kexec-tools	2.0.20-34.el8_3.1.x86_64	2.0.20-34.el8.x86_64
linux-firmware	20200619-101.git3890db36.el8_3.noarch	20200619-99.git3890db36.el8.noarch
microcode_ctl	4:20200609-2.20201112.1.el8_3.x86_64	4:20200609-2.20201027.1.el8_3.x86_64
systemd	239-41.el8_3.1.x86_64	239-41.el8_3.x86_64
systemd-libs	239-41.el8_3.1.x86_64	239-41.el8_3.x86_64
systemd-pam	239-41.el8_3.1.x86_64	239-41.el8_3.x86_64
systemd-udev	239-41.el8_3.1.x86_64	239-41.el8_3.x86_64
tuned	2.14.0-3.el8_3.1.noarch	2.14.0-3.el8.noarch
tzdata	2020f-1.el8.noarch	2020d-1.el8.noarch
docker-ce	3:20.10.2-3.el8.x86_64	3:20.10.1-3.el8.x86_64
docker-ce-cli	1:20.10.2-3.el8.x86_64	1:20.10.1-3.el8.x86_64
docker-ce-rootless-extras	20.10.2-3.el8.x86_64	20.10.1-3.el8.x86_64

Seems like coreos/go-iptables/issues/79 could be related.

[robbertkl]

Hi @Rycieos, thanks for the extensive report!

Could you try a few things for me?

• Do you use --network host? And could you try with --privileged instead of the --cap flags?
• Could you try to use robbertkl/ipv6nat:0.4.2 instead of latest? I've recently upgraded go-iptables, so this could indeed be related to the issue you mentioned.

[Rycieos]

I am running with --network host, yes.

Trying --privileged instead has the same result.

I forgot to mention, I was running v0.4.1 I think before debugging, and updated to see if it resolved my issue. I just tested versions v0.4.1, v0.4.2, and v0.4.3 with both --privileged and --cap-adds, same results.

[robbertkl]

Trying --privileged instead has the same result

Just to be sure: in that case you're running the ipv6nat container with --network host AND --privileged?

And before the system upgrade, it was all working fine?

v0.4.1 and v0.4.2 have been around for quite a while already, but haven't seen an issue like this before. And if the issue started only after the system upgrade, that makes it only stranger, and doesn't seem related to go-iptables. I've also upgraded to newer versions of Alpine and Go for the v0.4.3 container, but this would be unrelated as well, since you have the same issue with v0.4.1 and v0.4.2.

The iptables upgrade of your systems goes from 1.8.4-15.el8.x86_64 to 1.8.4-15.el8_3.3.x86_64, but can't find what the changes are between those. Also, you mentioned doing the check yourself works just fine..

A few more things to try:

• Can you try to list the iptables rules (iptables-save -t nat) and run the check commands (-C) from within the docker-ipv6nat container image? Since you can't attach to the running container (it keeps crashing), you could start temporary container with something like docker run --rm -it --privileged --network host --entrypoint sh robbertkl/ipv6nat and run the commands from there.

• Can you try running the docker-ipv6nat binary directly on the host (so not in a docker container)?

[Rycieos]

Can you try to list the iptables rules (iptables-save -t nat) and run the check commands (-C) from within the docker-ipv6nat container image? Since you can't attach to the running container (it keeps crashing), you could start temporary container with something like docker run --rm -it --entrypoint sh robbertkl/ipv6nat and run the commands from there.

Good idea!

Sanity check from host:

$ sudo iptables-save -t nat
# Generated by iptables-save v1.8.4 on Sat Jan  9 16:06:51 2021
*nat
...
-A OUTPUT ! -d 127.0.0.0/8 -m addrtype --dst-type LOCAL -j DOCKER
...

Container:

$ docker run --rm -it --privileged --network host --entrypoint sh robbertkl/ipv6nat
/ # iptables-save -t nat
# Generated by iptables-save v1.8.4 on Sat Jan  9 21:06:41 2021
*nat
:PREROUTING ACCEPT [0:0]
:INPUT ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:POSTROUTING ACCEPT [0:0]
COMMIT
# Completed on Sat Jan  9 21:06:41 2021

Well that isn't good. Same thing if I do docker run --rm -it --cap-add NET_ADMIN --cap-add NET_RAW --network host --entrypoint sh robbertkl/ipv6nat instead.

Any ideas? I'll keep digging.

[robbertkl]

From within the container (docker run, again with --privileged --network host), could you run these 3 commands:

ls -l `which iptables`
xtables-legacy-multi iptables-save -t nat
xtables-nft-multi iptables-save -t nat

[robbertkl]

Sorry, forgot:

Before running the ls -l command, please run ./docker-ipv6nat-compat once.

[Rycieos]

$ docker run --rm -it --cap-add ALL --network host --entrypoint sh robbertkl/ipv6nat
/ # ls -l `which iptables`
lrwxrwxrwx    1 root     root            20 Dec 28 22:49 /sbin/iptables -> xtables-legacy-multi

/ # xtables-legacy-multi iptables-save -t nat
# Generated by iptables-save v1.8.4 on Sat Jan  9 21:17:25 2021
*nat
:PREROUTING ACCEPT [837:70639]
:INPUT ACCEPT [0:0]
:OUTPUT ACCEPT [138:9289]
:POSTROUTING ACCEPT [942:73933]
COMMIT
# Completed on Sat Jan  9 21:17:25 2021

/ # xtables-nft-multi iptables-save -t nat
# Generated by iptables-save v1.8.4 on Sat Jan  9 21:17:29 2021
*nat
:PREROUTING ACCEPT [0:0]
:INPUT ACCEPT [0:0]
:POSTROUTING ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:DOCKER - [0:0]
-A PREROUTING -m addrtype --dst-type LOCAL -j DOCKER
-A POSTROUTING -s 172.17.0.0/32 ! -o docker0 -j MASQUERADE
-A POSTROUTING -s 10.1.0.0/32 ! -o br-dce386402b8e -j MASQUERADE
...
-A POSTROUTING -s 10.1.0.2/32 -d 10.1.0.2/32 -p tcp -m tcp --dport 443 -j MASQUERADE
-A POSTROUTING -s 10.1.0.2/32 -d 10.1.0.2/32 -p tcp -m tcp --dport 80 -j MASQUERADE
...
-A OUTPUT ! -d 127.0.0.0/32 -m addrtype --dst-type LOCAL -j DOCKER
-A DOCKER -i docker0 -j RETURN
-A DOCKER -i br-dce386402b8e -j RETURN
...
-A DOCKER ! -i br-dce386402b8e -p tcp -m tcp --dport 443 -j DNAT --to-destination 10.1.0.2:443
-A DOCKER ! -i br-dce386402b8e -p tcp -m tcp --dport 80 -j DNAT --to-destination 10.1.0.2:80
...
COMMIT
# Completed on Sat Jan  9 21:17:29 2021
# Warning: iptables-legacy tables present, use iptables-legacy-save to see them

That is identical to what I see on the host. Does that mean the iptables update I got switched from the legacy backend to a newer backend?

And before the system upgrade, it was all working fine?

Correct.

[Rycieos]

Oops, I missed this before:
Running on the host:

$ sudo iptables-save -t nat
# Generated by iptables-save v1.8.4 on Sat Jan  9 16:15:14 2021
...
# Completed on Sat Jan  9 16:15:14 2021
# Warning: iptables-legacy tables present, use iptables-legacy-save to see them
$ sudo iptables-legacy-save -t nat
sudo: iptables-legacy-save: command not found

That happens with iptables -L as well. I don't remember seeing that warning before, so something must have changed with my update.

[Rycieos]

Sorry, forgot:

Before running the ls -l command, please run ./docker-ipv6nat-compat once.

$ docker run --rm -it --cap-add NET_ADMIN --cap-add NET_RAW --network host --entrypoint sh robbertkl/ipv6nat
/ # ls -l `which iptables`
lrwxrwxrwx    1 root     root            20 Dec 28 22:49 /sbin/iptables -> xtables-legacy-multi

/ # ./docker-ipv6nat-compat
2021/01/09 21:25:59 unable to detect hairpin mode (is the docker daemon running?)

/ # ls -l `which iptables`
lrwxrwxrwx    1 root     root            17 Jan  9 21:25 /sbin/iptables -> xtables-nft-multi

Looks like that fixes it to point to the right one. I'm assuming the standard entry point must not be doing that, or it should be working for me.

[robbertkl]

It looks like your system has indeed switched backend, but docker-ipv6nat-compat should pick that up and symlink to the correct version.

Could you try the previous ls -l command again, but this time after you run ./docker-ipv6nat-compat once?

Also: do you normally run the ipv6nat container with the entrypoint set? Or does it use the default entrypoint?

[Rycieos]

Also: do you normally run the ipv6nat container with the entrypoint set? Or does it use the default entrypoint?

Default. Here is my full docker-compose file:

version: '2.3'
services:
  ipv6nat:
    image: robbertkl/ipv6nat:0.4.3
    restart: always
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro
    network_mode: host
    cap_drop:
      - ALL
    cap_add:
      - NET_RAW
      - NET_ADMIN
      - SYS_MODULE

[robbertkl]

Hmm, so it does switch over to xtables-nft-multi (docker-ipv6nat-compat is actually the standard entrypoint), but still doesn't work? That's very strange.

[robbertkl]

1 thing that looks off in your output is that:

• xtables-legacy-multi iptables-save -t nat shows no rules, but it does have counters behind each chain (the numbers in brackets)
• xtables-nft-multi iptables-save -t nat shows Docker's NAT rules, but the counters are all 0

[Rycieos]

I think I got it!

On host:

$ sudo iptables-save -t nat
# Generated by iptables-save v1.8.4 on Sat Jan  9 16:34:29 2021
*nat
...
-A OUTPUT ! -d 127.0.0.0/8 -m addrtype --dst-type LOCAL -j DOCKER

In container (I hacked the entrypoint to not exit after it crashes):

± docker exec -it ipv6nat_ipv6nat_1 sh
/ # ls -l `which iptables`
lrwxrwxrwx    1 root     root            17 Jan  9 21:32 /sbin/iptables -> xtables-nft-multi

/ # iptables-save -t nat
# Generated by iptables-save v1.8.4 on Sat Jan  9 21:33:13 2021
*nat
...
-A OUTPUT ! -d 127.0.0.0/32 -m addrtype --dst-type LOCAL -j DOCKER

It shows the address as 127.0.0.0/32, not 127.0.0.0/8!

[robbertkl]

Wow, that should definitely explain it!

Any idea why that would happen and why only after a system upgrade? Very strange how iptables-save (both version 1.8.4) show different things on the host and in container for the same rule!