Docker Bridge
DNS Updater
Abstract
When you run a container on a Docker bridge network, it will be operating on a private virtual network on your Docker host server. Your Docker host will be able to ping each container, but your LAN hosts won't.
When you restart your host, unless you have Docker dependencies setup (i.e. some rudimentary start order) you may find that your containers change IP address on the bridge network. This is fine if you're hiding things behind URLS and Traefik routers (Traefik will auto-update) but if you have something obtuse that wants to use IP, this can cause problems when that IP changes.
The specific use case that triggered this was the author's use of OpenRemote running natively on the host, which uses HTTP calls to update EmonCMS. OpenRemote uses Tomcat, which cannot (without severe bother) communicate with HTTPS hosts - so it was setup to communicate with the Docker bridge IP natively - which caused problems for home automation if that IP changed.
This script will enable your server to update it's /etc/hosts file so that it can resolve containers by name locally. This can also be useful if you're using your Docker host as a DNS server. If you're using pihole (as a container) then you can extend this to run on
Note: This may be considered a pointless article by some, as defining your own Docker bridge network, rather than using the natively defined 'bridge' network can get around this - but either way it's a fun solution to a specific problem and may be useful for someone out there to extend
Table Of Contents
Script
Copy and paste the following into a script somewhere on your Docker host (as root), i.e. /usr/sbin/updatedockerdns.sh
Note: You can update MinContainers to be one less than (or equal to) the number of containers you should typically have running. The script will basically loop until your Docker instance has that many containers running before it updates /etc/hosts.
#!/bin/bash
# Script provided by TechnoBobbins (https://site.gothtech.co.uk)
# Backup current /etc/hosts
cp /etc/hosts /etc/hosts.previous
# Set MinContainer amount & get initial container run count
MinContainers=19
ContainerCount=`/usr/bin/docker ps -q | wc -l`
# Loop until MinContainerCount is reached...
if [ $ContainerCount -lt $MinContainers ]; then
while [ $ContainerCount -lt $MinContainers ]
do
sleep 5
ContainerCount=`/usr/bin/docker ps -q | wc -l`
done
fi
# Check for DOCKER section tags in /etc/hosts
if [ `grep DOCKER /etc/hosts | wc -l` -eq 2 ]; then
# Nothing Required
:
else
# Add the DOCKER section tags if missing...
echo "# BEGIN DOCKER" >> /etc/hosts
echo "# END DOCKER" >> /etc/hosts
fi
# Extract IP and build temporary host info
echo "# BEGIN DOCKER" > /tmp/dockerhosts.txt
docker ps -q | xargs -n 1 docker inspect --format '{{ .Name }}{{range .NetworkSettings.Networks}}|{{.IPAddress}}{{end}}' | sed 's#^/##'| grep 172 | sed 's/|/.internal\|/' | awk -F\| '{print $2 " " $1}' >> /tmp/dockerhosts.txt
echo "# END DOCKER" >> /tmp/dockerhosts.txt
# Generate the temporary hosts file
sed -e '/^# END DOCKER/r /tmp/dockerhosts.txt' -e '/^# BEGIN DOCKER/,/^# END DOCKER /d' /etc/hosts > /tmp/tmphosts
# Overwrite /etc/hosts
mv /tmp/tmphosts /etc/hosts
# Tidy Up
rm /tmp/dockerhosts.txt
Now ensure your script is executable, by typing
sudo chmod a+rx /usr/sbin/updatedockerdns.sh
Service Definition
Copy/paste the following into /etc/systemd/system/dockernetcheck.service
[Unit]
Description=Add Docker Container IPs to /etc/hosts at bootup
After=network.target
[Service]
Type=oneshot
ExecStart=/usr/sbin/updatedockerdns.sh
RemainAfterExit=true
StandardOutput=journal
[Install]
WantedBy=multi-user.target
Amend the location of your updatedockerdns.sh script if you put it somewhere else....
Then run:
sudo systemctl daemon-reload
sudo systemctl enable dockernetcheck
sudo systemctl start dockernetcheck
You've now created a service called dockernetcheck that will run at boot, and wait until it seems 19 containers running. It will then parse the output of docker ps to find all the bridge IP addresses, create a temporary file of hosts (with a .local suffix) and their respective IPs, before adding them to /etc/hosts
It puts the new host/IP mappings within a DOCKER block in /etc/hosts, so it can update them again if the script is manually run.
Consider CRON
Depending on how often you reboot, or play around with containers, consider having CRON run the script periodically.
Troubleshooting
Depending on how complicated you've made your DNS setup, you may find your system appears to "hang" at boot - you'll get to a login prompt but docker may not start.
If you run
sudo systemctl status
...and see any reference to "queued jobs" then you should run
sudo systemctl --list-jobs
If you see dockernetcheck.service is running, then docker is probably stuck pending...you can check this by running
sudo docker image ls
If no images are returned, then it's definitely stuck. To remedy this, run:
sudo systemctl stop dockernetcheck.service
sudo systemctl restart docker
sudo systemctl start dockernetcheck.service
We're currently checking if we need to move the dockernetcheck.service dependency to run after multi-user.target.
NB : This MAY also stop some shutdowns, as the service won't cleanly close off with DNS off. Again, not fatal, but we're checking.