Ephemeral source port ranges and docker build

TLDR; If you are having trouble with docker build and ephemeral port ranges, we can use iptables to solve the issue:

$ sudo iptables -t nat -I POSTROUTING -p tcp -m tcp --sport 32768:61000 -j MASQUERADE --to-ports 49152-61000

I have written previously about how things get interesting with ephemeral port ranges in a Windows and Linux environment and AWS network acls. Today’s post is related to the same topic but specifically relevant if you are building docker images in such an environment.

Let’s start with the Dockerfile:

# Build runtime image
FROM microsoft/dotnet:2.2-aspnetcore-runtime
RUN apt-get -y update
..

The instruction RUN apt-get -y update will make network requests to download resources from the Internet over HTTP. This means it will select a certain source port to make these HTTP requests. However, in a controlled environment, we want to explicitly state the range of ephemeral ports that should be use, else these requests will not succeed.

Let’s see how we can do that.

Background

How does a docker build happen? Inside containers. What do we do if we want to configure the ephemeral port range for these builder containers? We can’t seem to be able to run sysctl in this scenario. We could use docker build --host to share the host’s network namespace. And that will ensure that out host’s ephemeral port range will be used. However, we also had user namespacing turned on in our setup since this is a sensible thing to do. However, we cannot use a user namespace while using the host network. So, what do we do?

Solution

Could we have a iptables rule to perform a source port translation so that anything that is going out of our host always uses a source port from the specified port range? Generally speaking, we will need to perform a variation of Source NAT. However, we will only change the source port and leave the IP address alone. The following rule will do it:

$ sudo iptables -t nat -I POSTROUTING -p tcp -m tcp --sport 32768:61000 -j MASQUERADE --to-ports 49152-61000

Here’s what the above rule does:

  1. We are adding this rule to the nat table (-t nat) in the POSTROUTING(-I POSTROUTING) chain
  2. We want this rule to be applied for TCP (-p tcp) packets which has a source port in the range 32768-61000 (--sport 32768-61000)
  3. If a packet matches our rule, forward it to the MASQUERADE target (-j MASQUERADE)
  4. Once in the MASQUERADE target, change the source port to be in the range 49152-61000 (--to-ports 49152-61000)

The resultant iptables-save will look as follows:

# Generated by iptables-save v1.6.1 on Mon Jan 14 06:27:10 2019
*nat
:PREROUTING ACCEPT [80:4975]
:INPUT ACCEPT [2:136]
:OUTPUT ACCEPT [401:30352]
:POSTROUTING ACCEPT [298:24143]
:DOCKER - [0:0]
-A PREROUTING -m addrtype --dst-type LOCAL -j DOCKER
-A OUTPUT ! -d 127.0.0.0/8 -m addrtype --dst-type LOCAL -j DOCKER
-A POSTROUTING -p tcp -m tcp --sport 32768:61000 -j MASQUERADE --to-ports 49152-61000
-A POSTROUTING -s 172.17.0.0/16 ! -o docker0 -j MASQUERADE
-A DOCKER -i docker0 -j RETURN
COMMIT
# Completed on Mon Jan 14 06:27:10 2019

This assumes you don’t have any other iptables rules configured other than what docker engine configures and then our rule above.

Gotchas

Okay, so where/when do you add this iptables rule? We have to add this after docker engine has started. docker creates its own firewall rules which seems to be like “drop everything else and add my own”. So, here’s how I am doing it in AWS EC2 user data:

  1. EC2 instance initialization
  2. docker daemon starts
  3. Add iptables rule

And a real code snippet:


# Add VPC DNS server as an additional DNS server
# https://github.com/amitsaha/aws-vpc-dns-address
vpc_dns=$(aws-vpc-dns-address)
echo '{"dns":["'"$vpc_dns"'", "8.8.8.8"]}' > /etc/docker/daemon.json
cat /etc/docker/daemon.json
systemctl restart docker

..
# This helps us workaround NACLs in place so that all traffic originating traffic source port is mapped to 
# the allowed ephemeral port range
# We carefully do it after we have restarted docker engine, since it inserts inserts
# its own iptables rules which flushes ours
iptables -t nat -I POSTROUTING -p tcp -m tcp --sport 32768:61000 -j MASQUERADE --to-ports 49152-61000

A more full proof appraoch would be to perhaps write a systemd unit file so that it will always run a program/script to insert the above firewall rule when docker engine is started/restarted.

Know about a better idea? Please let me know.

Learn more

There’s much to learn about iptables. The most relevant ones for this post to be familiar with are: