unable to detect hairpin mode - iptables >1.8.4 update issues

[dragoangel]

New iptables-1.8.4-15.el8_3.3 packages on CentOS8 leading to non working ipv6nat docker.
mailcow users on CentOS8 who have IPv6 enabled need to lock iptables to latest working version till issue will not be resolved.

Issue described here: robbertkl/docker-ipv6nat#67

If you not yet updated to 1.8.4.-15.el8_3.3 please lock packages version:
sudo dnf install -y 'dnf-command(versionlock)'; sudo dnf versionlock add iptables iptables-ebtables iptables-libs iptables-services

If you already updated iptables you need to downgrade packages and reboot the server:

1. downgrade packages:

tmp_dir=$(mktemp -d -t fix-iptables-XXXXXXXXXX)
wget -O $tmp_dir/iptables-1.8.4-15.el8.x86_64.rpm http://vault.centos.org/8.3.2011/BaseOS/x86_64/os/Packages/iptables-1.8.4-15.el8.x86_64.rpm
wget -O $tmp_dir/iptables-ebtables-1.8.4-15.el8.x86_64.rpm http://vault.centos.org/8.3.2011/BaseOS/x86_64/os/Packages/iptables-ebtables-1.8.4-15.el8.x86_64.rpm
wget -O $tmp_dir/iptables-libs-1.8.4-15.el8.x86_64.rpm http://vault.centos.org/8.3.2011/BaseOS/x86_64/os/Packages/iptables-libs-1.8.4-15.el8.x86_64.rpm
wget -O $tmp_dir/iptables-services-1.8.4-15.el8.x86_64.rpm http://vault.centos.org/8.3.2011/BaseOS/x86_64/os/Packages/iptables-services-1.8.4-15.el8.x86_64.rpm
sudo dnf downgrade -y $tmp_dir/iptables-*
rm -rf $tmp_dir

2. do docker-compose down in mailcow home directory
3. restart your server OS
4. lock packages:
   sudo dnf install -y 'dnf-command(versionlock)'; sudo dnf versionlock add iptables iptables-ebtables iptables-libs iptables-services
5. do docker-compose up -d in mailcow home directory

To unlock iptables packages when issue will be resolved:
sudo dnf versionlock del iptables iptables-ebtables iptables-libs iptables-services

Note: some users can not have iptables-services package, this totally fine, downgrade command will not touch it in this case and downgrade all other packages iptables packages successfully.

[andryyy]

Thank you!

[dragoangel]

Issue also reproduce on Debian 11
Update: Debian 11 is beta and not have docker support, so it musn't be used with mailcow

As discussed in telegram group, here's the workaround for those using Debian 11 with iptables version > 1.8.4.

Check your version by typing: $ sudo iptables --version

Download older packages and dependencies of iptables (in this example my machine usues architecture amd64):

tmp_dir=$(mktemp -d -t iptables_workaround)
wget -O $tmp_dir/iptables_1.8.2-4_amd64.deb http://ftp.debian.org/debian/pool/main/i/iptables/iptables_1.8.2-4_amd64.deb
wget -O $tmp_dir/iptables-dev_1.8.2-4_amd64.deb http://ftp.debian.org/debian/pool/main/i/iptables/iptables-dev_1.8.2-4_amd64.deb
wget -O $tmp_dir/libip4tc0_1.8.2-4_amd64.deb http://ftp.debian.org/debian/pool/main/i/iptables/libip4tc0_1.8.2-4_amd64.deb
wget -O $tmp_dir/libip4tc-dev_1.8.2-4_amd64.deb http://ftp.debian.org/debian/pool/main/i/iptables/libip4tc-dev_1.8.2-4_amd64.deb
wget -O $tmp_dir/libip6tc0_1.8.2-4_amd64.deb http://ftp.debian.org/debian/pool/main/i/iptables/libip6tc0_1.8.2-4_amd64.deb
wget -O $tmp_dir/libip6tc-dev_1.8.2-4_amd64.deb http://ftp.debian.org/debian/pool/main/i/iptables/libip6tc-dev_1.8.2-4_amd64.deb
wget -O $tmp_dir/libiptc0_1.8.2-4_amd64.deb http://ftp.debian.org/debian/pool/main/i/iptables/libiptc0_1.8.2-4_amd64.deb
wget -O $tmp_dir/libiptc-dev_1.8.2-4_amd64.deb http://ftp.debian.org/debian/pool/main/i/iptables/libiptc-dev_1.8.2-4_amd64.deb
wget -O $tmp_dir/libxtables12_1.8.2-4_amd64.deb http://ftp.debian.org/debian/pool/main/i/iptables/libxtables12_1.8.2-4_amd64.deb
wget -O $tmp_dir/libxtables-dev_1.8.2-4_amd64.deb http://ftp.debian.org/debian/pool/main/i/iptables/libxtables-dev_1.8.2-4_amd64.deb

sudo apt install $tmp_dir/* && sudo apt --fix-broken install $tmp_dir/*
cd /opt/mailcow-dockerized && sudo docker-compose down
sudo reboot

//**Lock version of iptables and its dependencies. This will hold back upgrades of the necessary iptables packages when doing apt upgrades**
sudo apt-mark hold iptables libip4tc-dev libip6tc-dev libiptc-dev libiptc0 libxtables-dev libxtables12

cd /opt/mailcow-dockerized && sudo docker-compose up -d

After downgrading, the error messages disappeared and IPv6 mail transmission started working again.

Big thanks to @dragoangel for your help!

[mcmufffin]

On Oracle Linux 8 you can just do "yum downgrade iptables" and it will revert back to iptables-1.8.4-15.0.1.el8.x86_64 which is not affected.

Not sure if the same works for Centos

[dragoangel]

Hi @mcmufffin, for CentOS8 I described all flow in details ;)

[mcmufffin]

Hi @mcmufffin, for CentOS8 I described all flow in details ;)

Hehe, yeah i saw that, but downloading rpm by hand? Ill give that a pass if it works via yum downgrade :)

[dragoangel]

Downgrade not guaranty it will be downgraded to needed version ;)

[mcmufffin]

1.8.4-15.0.1 seem also not to be affacted.

[Daniel15]

According to robbertkl/docker-ipv6nat#67, a patch is being backported for CentOS... Does anyone know the state of a patch for Debian Bullseye (11)?

[mcmufffin]

According to robbertkl/docker-ipv6nat#67, a patch is being backported for CentOS... Does anyone know the state of a patch for Debian Bullseye (11)?

You got that mixed up, the problem was not solved by that mentioned patched but rather indstroduced by it, so CentOS is still not fixed.

It seems like Iptables 1.8.7 includes a fix so it should work on debian 11 bullseye.

[Daniel15]

Oh! Oops.

It seems like Iptables 1.8.7 includes a fix so it should work on debian 11 bullseye.

I still got the "unable to detect hairpin mode" message on Debian 11 with the latest packages including iptables 1.8.7 😢 Downgrading to the 1.8.2 packages mentioned earlier in the thread 'fixed' it.

[Daniel15]

Also, for now I've disabled the Docker userland proxy. Having no IPv6 connectivity is better than having broken IPv6 connectivity with an open relay.