blog.thms.uk

Using a Tailscale exit node on a Docker host

I’ve recently migrated my Mastodon instance to run on Docker Compose. This host sits on my home network, and gets exposed to the internet using Caddy on a VPS that’s connected to my server using Tailscale.

This works great, but I did want to route outgoing traffic through the same VPS, so that other server operators don’t see my domestic IP address in their server logs.

This should be easy enough:

  1. On the edge server add the exit node flag:
sudo tailscale set --advertise-exit-node
  1. Configure the Mastodon host to use the exit node:
sudo tailscale set --exit-node=<exit-node-ip> --exit-node-allow-lan-access=true

Easy as pie (yes, you also need to configure IP forwarding, may need to approve it in the admin UI, and set a policy to allow access - still easy as pie).

I did think it worked, as I ran

curl https://ip.me

and got my edge IP address.

But then I tested it from within the container:

docker exec mastodon-sidekiq-1 curl https://ip.me

and the whole thing just hung there.

The problem

Basically, the Docker host sits as a NAT device in front of the Docker containers. And when you enable the Tailscale exit node, that intercepts traffic on all interfaces and routes it via Tailscale - including traffic destined for your container subnet. So nothing can reach your Docker containers any longer, and connections just hang.

The solution

Firstly, let’s see what IP address we need to route to:

docker inspect mastodon-web-1

In the output look for IPAddress and make a note (in my case it was 172.19.0.2).

Now, check the route that your system has for that IP address:

$ ip route get 172.19.0.2
172.19.0.2 dev tailscale0 table 52 src 100.64.0.8 uid 1000
    cache

This gives us two insights:

  1. Traffic for that IP address is indeed routed towards tailscale0, so our initial theory was correct.
  2. The thing routing it is table 52.

So we need to add our own route to that IP address. And we need to do so with a priority that comes before (i.e. is lower than) the rule for table 52. So let’s look at our existing rules:

$ ip rule show
0:	from all lookup local
5210:	from all fwmark 0x80000/0xff0000 lookup main
5230:	from all fwmark 0x80000/0xff0000 lookup default
5250:	from all fwmark 0x80000/0xff0000 unreachable
5270:	from all lookup 52
32766:	from all lookup main
32767:	from all lookup default

Okay, so the rule for table 52 has a priority of 5270.

Let’s add a new rule with a lower priority of 5269 (replace with a different value if your rules have different priorities, and replace your subnet, if your Docker is configured differently):

ip rule add to 172.16.0.0/12 lookup main priority 5269

This means we are routing all traffic to 172.16.0.0/12 (which covers Docker’s default subnets) outside of Tailscale, and we are doing this with priority 5269 - immediately before Tailscale’s rule. Let’s check the route now:

$ ip route get 172.19.0.2
172.19.0.2 dev br-29ab31f9f5aa src 172.19.0.1 uid 1000
    cache

indeed, this now routes to my Docker bridge device, and we can confirm by running docker exec mastodon-web-1 curl https://ip.me which should now return your exit node’s IP address rather than your own.

Making the solution permanent

I think the best solution is a quick service file that inserts the rule when Tailscale and Docker start up:

[Unit]
Description=Route Docker subnets via main table so exit-node return traffic reaches containers
After=network-online.target tailscaled.service docker.service
Wants=network-online.target

[Service]
Type=oneshot
RemainAfterExit=yes
# delete then add keeps it idempotent if the unit is re-run
# replace your subnets and/or priority as needed
ExecStartPre=-/sbin/ip rule del to 172.16.0.0/12 lookup main priority 5269
ExecStart=/sbin/ip rule add to 172.16.0.0/12 lookup main priority 5269
ExecStop=-/sbin/ip rule del to 172.16.0.0/12 lookup main priority 5269

[Install]
WantedBy=multi-user.target

Finally, enable the service:

sudo systemctl daemon-reload
sudo systemctl enable --now docker-tailscale-routing.service

Wrapping up

A bit of policy routing was all it took: one ip rule to keep Docker’s subnets out of Tailscale’s hands, made permanent with a small service file. My containers now reach the internet through the exit node like everything else, and my domestic IP stays out of other people’s logs.