systemd-nspawn: disabling link-local addressing
I recently migrated my various bespoke web services from a bunch of AWS EC2
instances to a single DigitalOcean droplet. The reason is simple: I ran out of
AWS credits. In an effort to expedite any future migrations, I decided to
containerize all my services. Rather than go with a full-blown container
solution like Docker or LXC, I opted to roll my own using
Everything was set up and working well, until I noticed that it would take my blog a full three seconds to load, whereas it should be pretty much instantaneous. Let’s debug this.
Measuring connection times
Accessing the blog works like this: a local machine connects to the nginx
instance running on
jerrington.me which performs TLS termination and
reverse-proxies the request into a container named
blog. This machine has its
own dead-simple nginx instance running to serve up the static blog files.
First, let’s check the speed of a no-frills TCP connection from my local
host to the remote host. The easy, quick-and-dirty was to accomplish this is
netcat -lp 5000 # on the remote host time netcat -z jerrington.me 5000 # on the local host
-z switch sets netcat into the “zero I/O” mode, which makes it
immediately shut down the connection after establishing it. This can useful (as
it is here) for simple network diagnostics.
This took about 20 milliseconds, so the problem isn’t with the connection to the remote host.
Next let’s try the connection from the remote host to the container. Using the same measurement technique, I found that this took about three seconds. There’s the problem. But why?
What’s stranger is that some containers can be connected to instantly, but the blog container has this flat three second penalty on any new connection.
I should mention at this point that I’m using the NSS mymachines plugin which
gethostbyname to resolve container names to the IP addresses
assigned to them internally. So to connect to the blog container, I just needed
blog rather than an IP address that changes on every reboot due to
I was at a loss about where this three second penalty could possibly be coming
from, until I tried
telnet. In the container I set up another simple server
with netcat, and on the host I connected to it with telnet. Telnet tells you
something very very useful when it’s establishing a connection, and that’s the
order in which it tries IP addresses for the host you’re trying to connect to.
Here’s what I saw:
$ telnet blog 5000 Trying 169.254.0.134... Trying 10.0.0.99... Connected to blog. Escape character is '^]'.
And guess what? Trying the link-local address took about three seconds!
For comparison with the containers that don’t have the mysterious three second penalty, if I telnet into those, then I see that the private IP address is tried first, which immediately succeeds. That’s why those containers could be connected to in a reasonable time.
At this point, rather than continue my investigation of why it takes three seconds for the client to decide that the link-local address is no good, I simply decided to eliminate link-local addressing from my containers.
Disabling link-local addressing
The beautiful thing about systemd-nspawn containers is that the networking is
almost like magic. All you need to do is use
machinectl start blog and tadaa!
The container is now behind NAT. If you want to forward ports from the host
directly into a container, it’s easy to do so by specifying
instance in the
blog.nspawn configuration file for the container.
There’s one problem with magic though, and that’s that nobody actually knows how it works. All this stuff is taken care of by systemd-networkd and it’s unclear how exactly it does these things.
Luckily I stumbed onto a GitHub issue that made mention of the files
These files respectively configure the host and guest systemd-networkd
instances to work together to provide DHCP.
And guess what I found in these files?
It suffices to copy these files into
/etc/systemd/network/ on the host and
/etc/systemd/network/ on the guest and set
LinkLocalAddressing=no in both
to completely eliminate link-local addressing from the network setup.
At last, connection times were normal again, and everybody lived happily every after.