OMV, Docker, Portainer, Traefik, LetsEncrypt!

Abstract

This article details how to setup a secure, relatively hassle free home server environment, with secure remote access, using a combination of popular free, open source software (FOSS) - namely OpenMediaVault (OMV), Docker, Portainer, Traefik, LetsEncrypt - along with some useful containers (like pihole and Fail2Ban) - and then top it off with Google oAuth for security (if you like).

This page exists as (at least for the author) the "really useful" bits of info don't seem to exist on a single page, or aren't quite succinct enough, and/or don't cover the versions the author is using - as well as wanting to explain how some things work 

IMPORTANT : Traefik V3 has introduced changes which will break your installations without amending your config.
If you have followed the instructions below, change EVERY instance of HostHeader to just be Host otherwise traefik will not route your traffic.

Table Of Contents

Introduction / Preface

You may (or may not) have already read the author's review of OpenMediaVault (or OMV) - it's a well regarded and well established home server platform, which the author has used since OMV 3.0

Not that long ago. OMV 5.0 was released, and around the same time, it was apparent that the author's venerable little HP N54L Microserver was struggling to keep up with the demands being put upon it (Zoneminder, I'm looking at you...).  So the author acquired an HP Microserver gen8 to migrate things over to.

This author is also (it has to be said) a bit of a geek.  If something can be tinkered with it invariably will wind up being tinkered with.  This has led to various sweary moments as previously stable installations have fallen over, and required more sweary moments to recover.

By way of two examples:

Since then, and due to having a bit of spare time, the author has discovered Docker, and things have got easier.  Then OMV 5 bundled in Portainer, and easy got easier.  Then the author discovered Traefik and fell down a rabbit hole....

If you are currently part way down the same rabbit hole, and/or swearing because you can't "just make it work", read on...

The goal here was/is to build a home server that does accomplishes following:

..and it's safe to say the above has been achieved.   So here we go...

Prerequisites

This article assumes you have the following:

Step 1 : OpenMediaVault 5.0

Installation

First off, get over the OMV download page, and get a copy of the OMV 5 ISO.  You can opt to burn it to CD and boot up to install OMV, or you can use a utility to put the ISO onto a USB stick and boot from that.

Note: All roads here start with OMV 5.  If you're not wanting to install a new system from scratch, just bear in mind that OMV 5 is based on Debian linux, specifically Debian 10 (buster) - and some things later in this article may well rely on Debian 10 being the underlying host.

To steal from the previous OMV review:

"The installation process will be familiar to anyone who has installed Linux before - and maybe a little unfriendly for those that haven't (although there is a "quick" option).  It's a text based menu selection system to let you customise your installation ("root or sudo" etc) - and probably the most important thing to note is that if you choose to install OMV - remove ANY disks you DON'T want the operating system to use BEFORE you install!"

"Yes, in an (admirable) attempt to keep the operating system and data on separate disks, OMV will "claim" anything it sees when it installs.  So, if you install it on a shiny 1TB drive, you might be a bit upset when it only uses 2GB and you have a lot of unusable space. "

The above is quite important now , as most hard drives are "not small" (in fact it's getting harder to find small drives).  When this author installed OMV 3, it was onto an 80GB drive.  Now it's sat on a 1TB drive.  But hey.

For this article, we're going to assume you've:

FYI, the author has a RAID-0 (mirror) configuration on a pair of disks, but (for the hell of it) has also hacked the OS drives to be in a RAID-0 (mirror) as well.

Minimal Configuration

Once you're into the admin GUI, there are a few things you should do to prepare things for later steps

OMV uses port 80 (http) by default.  You'll want this to be used by Traefik later, so in the OMV GUI, go to System > General Settings and change the Port from 80 to 81, and click Save (and then Apply Settings).  You'll manually have to then change your browser URL to access the GUI on it's new URL (i.e. http://youromvhostip:81)
Note: Later on, we'll need to know the IP of your OMV host - so let's assume it's 192.168.1.100

You now want to make sure that OMV is using the correct hostname and is on the domain you want to be using.  Navigate to System > Network and check the Hostname is openmediavault and the Domain name is yourdomain.name - obviously replacing that with your actual domain.

Once you have a working File System, you can create Shared Folders.  OMV uses these for most of it's features/plugins.  Using the OMV GUI, go to Access Rights Management > Shared Folders and click Add

Type the name for the Shared Folder (i.e. Docker) and then select which storage Device you wish to place it on (from your list of File Systems).  The Path should be added automatically, and you shouldn't need to change this.  Leave Permissions alone.  Click Save and (if prompted) Apply Settings.

Now you need to make a note of your full path to your new shared folder.  OMV puts links to the File Systems under /srv - so if you named your File System "OMVDataVolume" (as the author did) then you should have a folder called
/srv/dev-disk-by-label-OMVDataVolume

Within that you will have a folder named whatever Path you chose when creating your Shared Folder for Docker, so in theory (if you didn't change it), something like:

/srv/dev-disk-by-label-OMVDataVolume/Docker

The author advises that you create a "host" folder within your Docker folder (you'll see why later) - so from a command prompt, type

sudo mkdir /srv/dev-disk-by-label-OMVDataVolume/Docker/host

..obviously changing "OMVDataVolume" for whatever you called your folder

So your full path (for Docker data) will be something like:

/srv/dev-disk-by-label-OMVDataVolume/Docker/host

Keep that path to hand, you'll need it shortly...

You are now at the minimum position you need to be in to get going with Docker - but first you need to get Docker.  That comes with the "OMV Extras" package.

OMV Extras

Docker is not provided by OMV "out of the box" - but it's easy to get.  You can visit https://omv-extras.org/ to validate these instructions, but all you need to do is:

wget -O - https://github.com/OpenMediaVault-Plugin-Developers/packages/raw/master/install | bash 

The next time you load your OMV GUI you will see an OMV-Extras item at the bottom of the System menu on the left.

Step 2 : Docker

Open up your OMV GUI, and navigate to System > OMV-Extras > Docker.  In the box next to Docker Storage, enter the full path you copied for earlier, i.e. 

/srv/dev-disk-by-label-OMVDataVolume/Docker/host

Click Save and then Apply Settings when prompted (you can ignore the "Status" above, this screen shot was taken after install).

Now you need to select the Docker drop down and click Install

Docker will now install itself.  Once the "Close" button goes blue, you can click it to close it.

Step 3 : Portainer

This is the easiest piece. Similar to installing Docker, you're going to use the same screen, but this time select the Portainer drop down, and click install.  Again, once the "Close" button goes blue, you can click to close it.

 The keen eyed amongst you may note the Docker Storage path above says "SFDocker" - and that's just because the author likes to prefix things, so it's easier to spot an actual Shared Folder from a normal folder.

Once you've 'Closed' the Portainer installation pop up, you should hopefully see the Open Web button next to the Portainer drop down.  Clicking that will open the Portainer GUI, which is installed on port 9000 on your host - so you will now have two GUI URLS:

When you load Portainer for the first time, it will ask you to provide an admin password, and verify it.  Click Create User and you'll then be asked which Docker environment you want to connect to.  You should see four choices:

You want to click on Local and then click on Connect

You should then see something that looks like the following screen.  "OMV" is the portainer instance picking up the name of the server (OMV).

Click on the host will to open up the larger, host specific menu, as shown below...

The screenshot above shows the authors 17 containers and images, 5 networks and 5 volumes - but initially you should only be showing 3 networks and 1 container with 1 image - which will be Portainer.

That's it - you now have Portainer installed and are nearly ready to start (easily) adding Docker containers.

Step 4 : Docker Networking

There are four types of network that Docker can run a container on, or in, these are:

This guide uses a blend of the above.  Traefik will use host networking.  Most containers will use bridge networking but a couple will use macvlan for ease of separation (as not everything needs to go via Traefik.

Setup macvlan 

A macvlan network is just that, "a network" - it can support multiple hosts, as your Docker/Portainer instance will need to assign IPs from the macvlan range to your containers.

The first challenge is 'where' your macvlan range will live, which will determine it's size.  Some considerations:

Note: The author advises using a decent router which can be flashed with something like DD-WRT or Merlin firmware, to give you maximum control of your network - assuming your router is capable of being flashed.

Note: The author has migrated their network to a 10.X.0.0/16 network, with dedicated ranges for DHCP and macvlan.  

The author is not going to worry about what network you have (that's your problem) - but you may need to do some reading on subnet subvision :)  This article will assume your LAN is setup as 192.168.1.0/24 and we'll be using 192.168.1.240/28 for macvlan - giving 14 usable IPs for Docker - which (given we'll also be using bridge mode for some) should be ample.  If you're reading this far, the author assumes you're probably capable of tweaking your LAN to suit.

Docker handles macvlan setup in two pieces:

So, first of all, lets deal with the configuration

Assuming you logged into Portainer and clicked on your host, you should see the larger menu on the left side.  Click Networks to open up the main network screen, and then click the blue + Add Network button at the top.

You then want to:

The screen below shows what you're looking for...

  6. Scroll down and click "Create the network"

You should now see DockerNetwork in the list of networks.  Now you need to use it.  Click + Add Network again.

You then want to:

Your screen should look something like this...

  5. Scroll down and click "Create the network"

You should now see both DockerNetworkConfig and DockerNetwork listed in your Networks screen.

Did you make a mistake?

Normally, Docker and Portainer are quite forgiving - but unfortunately, network configuration is one place where mistakes cannot be easily fixed.   If you want to change any part of the configuration, you'll need to delete DockerNetworkConfig and DockerNetwork and recreate them again.

Step 5 : Removing noexec

Now we're almost ready for our first actual Docker container, but first, we need to deal with a bit of OMV default configuration...

By default, OMV tries to protect you by ensuring that Linux cannot execute (run) any binary code (programs) that you may store on your Data volume.  For most people, that's not a problem as they're just storing files and other things (music, images, documents etc) - however for Docker, it's a problem, because:

How Linux/Unix can access and use a mounted disk/volume is defined in /etc/fstab - so if open a command prompt and type:

sudo cat /etc/fstab

You should see a lot of information, but you're looking for the lines that have your data volume.  Which looks like this:

# >>> [openmediavault]
/dev/disk/by-label/OMVDataVolume                /srv/dev-disk-by-label-OMVDataVolume    ext4    defaults,noexec,nofail,user_xattr,usrjquota=aquota.user,grpjquota=aquota.group,jqfmt=vfsv0,acl      0 2

The line starting /dev/disk is actually all one line - with a large gap between ext4 and defaults.  As you can see, the word "noexec" is shown in the list of options, which means no code can be executed from that volume.

At this point, the average Unix/Linux person would say "I'll just edit /etc/fstab and remove noexec" - which would work fine on most systems - however OMV controls the generation of /etc/fstab - so any edits made to the actual fstab would be wiped out on the next reboot.

There are no OMV GUI options to amend this, so we need to edit the actual file that controls the generation of the fstab for OMV.  In your command prompt, run

sudo nano /etc/openmediavault/config.xml

Note: If for some reason nano doesn't load (command not found) you can run

sudo apt-get install nano

Once nano loads, you need to find the right bit of the file, so press CTRL-W and a search prompt will appear.  Type fstab and press Return.  Your cursor should move to a line saying

<fstab>

...and your cursor will be on the f.  About 13 lines below your cursor should be a section that looks like this:

<mntent>
<uuid>aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa</uuid>
<fsname>/dev/disk/by-label/OMVDataVolume</fsname>
<dir>/srv/dev-disk-by-label-OMVDataVolume</dir>
<type>ext4</type
<opts>defaults,noexec,nofail,user_xattr,usrjquota=aquota.user,grpjquota=aquota.group,jqfmt=vfsv0,acl</opts>

The author's UUID has been hidden above, but you can see our fsname (file system name).

Move your cursor down to the <opts> line and remove the word "noexec" from the line.  Ensure you also remove one of the two commas "," that were before/after it, so there is only one between the prior and subsequent options.  

It should look like this:

<opts>defaults,nofail,user_xattr,usrjquota=aquota.user,grpjquota=aquota.group,jqfmt=vfsv0,acl</opts>

Then press CTRL-X and press Y to save your file.

Now you need to restart your system - so type

sudo reboot

Check your system comes back up ok, and validate your OMV and Portainer GUIs both load - which will prove Portainer has started (and therefore that Docker is running).

Step 6 : First Docker Container - pihole

If you don't know, pihole is a network wide adblocker, which is really handy for cutting out on pesky ads and lowering your bandwidth consumption - which also speeds up your network a bit (less traffic).

However, besides blocking adverts, it also comes with a DHCP server, meaning you can use it as your central DNS and DHCP point - which the author finds very useful, as barring a configuration change on the main router, everything is in one GUI.

To setup a pihole container, you can either download the "Image" and then create the "Container", or you can just do it from the Container screen itself.

Basic Container Information

Click Containers and then click the blue + Add Container button.

Give your container a name, i.e. pihole and then tell it what Image you want.  If you've already downloaded an image, typing part of it's name in the Image box will provide any matches you have locally, but so far we have none, so you can just type in

pihole/pihole 

Note 1: You need to make a choice about the "Always pull the image" option.  With Portainer V1.2.4, it defaults to "on", however, this means that everytime you amend configuration and (re)deploy your container, Portainer will check if there is a new version.  If there is, it will download and try to run with your existing configuration.  As noted above, the author found this broke things - so it may be safer to switch that off, and you manually control when to "pull" a new image down.  

Note 2: If you use the :latest tag on any image, and have "Always pull the image" switched on, you can get into the situation described in Note 2 quite a lot, especially if you are using a popular, frequently updated image.  Thus, you may want to pick a specific tag (such as pihole:v5.0) - or, once you're happy something is stable, "Exporting" a given image and tagging it as "Production" - we'll cover that later.

So, right now your new container screen should look like this...

...but don't click the "Deploy the container" button just yet...we need to setup a few things.

Reviewing the Container information

In another browser tab, go and look at https://hub.docker.com/r/pihole/pihole
Most DockerHub provided images detail the "volumes" (disks) and "environment" variables that an image might define if setup with docker compose - but we're using Portainer, so we need to replicate that differently in the Portainer GUI shortly.

The pihole info on the above link shows us:

services:

  pihole:

    container_name: pihole

    image: pihole/pihole:latest

    ports:

      - "53:53/tcp"

      - "53:53/udp"

      - "67:67/udp"

      - "80:80/tcp"

      - "443:443/tcp"

    environment:

      TZ: 'America/Chicago'

      # WEBPASSWORD: 'set a secure password here or it will be random'

    # Volumes store your data between container upgrades

    volumes:

       - './etc-pihole/:/etc/pihole/'

       - './etc-dnsmasq.d/:/etc/dnsmasq.d/'

    dns:

      - 127.0.0.1

      - 1.1.1.1

    # Recommended but not required (DHCP needs NET_ADMIN)

    #   https://github.com/pi-hole/docker-pi-hole#note-on-capabilities

    cap_add:

      - NET_ADMIN

restart: unless-stopped

As you can see:

Once we get Traefik up (later on) you could put the pihole onto a bridge network and expose it via Traefik too - so it's always good to know which ports are (potentially) in use.

Creating Volumes

So, let's translate this into Portainer settings....on the create container screen, if you scroll down, you'll see a section that looks like this:

Typically Command & logging will be selected by default, but you can select each tab header and move around, without losing anything you've typed (or will type) in, as long as you don't navigate off the create container page.

We're not going to touch Command & logging, but Volumes is the next tab, so click it, and then  click the grey + map additional volume button twice (as we need two volumes)

The Portainer GUI deals with volumes by asking about the container first, and the host second.  This is the reverse of the docker-compose info we saw above, but we know we need two volumes, so let's key those in:

As you can see, we are telling the container that /etc/pihole will actually use:/srv/dev-disk-by-label-OMVDataVolume/Docker/pihole...and that /etc/dnsmasq.d will actually use:/srv/dev-disk-by-label-OMVDataVolume/Docker/dnsmasq

However, we don't have those folders yet - so open a command prompt on your OMV host and run

mkdir /srv/dev-disk-by-label-OMVDataVolume/Docker/pihole
mkdir /srv/dev-disk-by-label-OMVDataVolume/Docker/dnsmasq

That will create the two folders we need, ready for pihole to use.

Note: By default, Docker containers are not persistent - meaning anything you do inside one when it runs, will be lost when it restarts.  This makes them ideal for testing things, but less ideal if you want a container to survive a reboot and carry on providing services.  Volumes are the way to keep your data persistent. 

Note: By creating the folders inside your Docker folder, but not in your Docker/host - you are keeping your Docker core information and pihole container information separate.  This keeps folders a bit cleaner, as Docker itself keeps "a lot" of information within its host folder.

Configuring the Network

Now change to the Network tab - don't worry, as mentioned, nothing you've typed so far will be lost as long as you stay on the container page.

Note: This is the network setting for DNS, but pihole has it's own DNS settings it will take further down.  Either way, pihole needs to get to at least one upstream DNS server - which typically is your router, although you could point it directly at your ISP DNS, or any chosen DNS provider (the author uses their router, but that points to OpenDNS)

Your screen should look something like this:

Configuring the Environment

Now we need to move to the Env tab, to setup the environment variables pihole will run with.

We know from the Docker Hub info that we need 4 variables (WEBPASSWORD, DNS1, DNS2 and TZ) so click the grey + add environment variable button four times and enter the following

Variable Name Value

WEBPASSWORD yourchosenpasswordhere
DNS1 192.168.1.1
DNS2 192.168.1.1
TZ Europe/London

Change WEBPASSWORD to be whatever password you want to login to the pihole GUI.
DNS1 and DNS2 have both been set to your router, so everything is forced to your router and nowhere else.
TZ (as noted above) should be set to whatever TimeZone you are in.

Shortly, when you start pihole for the first time, you'll find some other additional variables are set - but these are all we need for now.

We're going to leave the Labels tab for now, we'll come back to that later with Traefik, so lets skip to the next tab

Set the Restart Policy

Because pihole is going to be your network DNS service, it's a core service.  You want it to be running all the time, which in Docker terms mean "unless stopped".  If your OMV host is restarted, Docker is now installed as a service and will start automatically, and it in turn will start Portainer.  It will also start pihole as long as we set the Restart policy to be unless stopped - and as long as it's run once.

So, within the Restart Policy tab, click unless stopped 

We don't need to play with Runtime & Resources either, so we'll skip along

Amend Capabilities

Docker has the ability to provide containers with a lot of privileged access and system capabilities - so much so that on the author's 1280x1024 screen, the Capabilities tab cannot be displayed properly - so this next screenshot may look a little different as the browser has been "zoomed out" to 80%

It's advisable to do this also, as the rendering can otherwise place option sliders in "odd" places and you don't want to enable the wrong setting - but we know that NET_ADMIN is required.  Make sure yours looks like the following screenshot:

Now the container is ready to go - you can scroll up a little. just above the Advancer container settings section and you'll see a "Deploy the container" button.  Click it!

The button text should change to "Deployment in progress", then you'll be taken back to the Container page, where you should see a new line at the bottom for pihole, with a yellow Status of Starting

Wait a few seconds and click the refresh icon at the top of the page, and the Status should change to a green Healthy

pihole is running!

You should now be able to access pihole by visiting http://192.168.1.241 - and there will be a link to the admin page there.  Alternately you can go directly to http://192.168.1.241/admin

Check you can login with the WEBPASSWORD you specified

Basic pihole configuration

Login to pihole and click Settings from the left menu - then choose DHCP from the top menu.

Ensure that DHCP server enabled is checked, and setup a DHCP range that LAN machines can use - in the example below machines will be allowed to use 192.168.1.10 thru 192.168.1.50 - amend as you so desire. 
Next ensure your router IP is added, as well as your subdomain.  If you like, you can also check Enable DHCP rapid commit (fast address assignment)

Once you've entered all the info, scroll down and click the blue Save button.

Note: Your pihole is now configured to act as a DHCP server, but configuring your router to use pihole as a DHCP server is not something we will actively cover here, as each router can vary wildly - however, if your router is using dnsmasq itself, you can try something like the following in your specific router (custom) config:

dhcp-relay=192.168.1.1,192.168.1.241,br0

The above line will tell a dnsmaq server to forward all DHCP requests received on 192.168.1.1 over to 192.168.1.241 (your pihole IP), if those requests are received on br0 (bridge zero).  Your router setup and bridge naming is going to be unique to your setup.  Worst case is that your router will still be providing DHCP - so just ensure it's not clashing with any ranges defined in this article.

Test pihole

Now you can test pihole out, by either:

You should then be able to see DNS accesses in the pihole GUI Dashboard and Query Log screens.

Did you make a mistake?

It's not impossible to mistype something, or realise you wanted to change a setting.  Don't worry, unlike the macvlan piece, you can edit container configuration.  

If you need to, go the Container screen, select pihole from your list of containers (in the Name column), and you'll then see a screen which has the following buttons at the top.  If you click the blue Duplicate/Edit button, you will go back to a screen very much like creating a new container, but all your values will be as you keyed them.  You can navigate around, change them, and then click "Deploy the container" again.  Just remember the note re: "Always pull the image"

Step 7 : OMV Host to Docker macvlan access

Docker tries, by default, to isolate hosts.  If you put two containers onto the Docker default bridge network, they will not be able to communicate unless it's via direct IP.

The other quirk you may not yet have spotted, is that your OMV host will not be able to access containers on the macvlan network, by IP or otherwise - but your other LAN hosts will be able to access the containers and vice versa.  You can try this by opening a command prompt on your OMV host, and trying 

ping 192.168.1.241

You will probably not get a response.  But try it from your PC or another machine, and it should work

In any case, you're highly likely to want to enable your OMV host to have IP comms to your containers.

To do this, you (rather bizarrely) need another macvlan interface on your OMV host, which will enable you to communicate with the other macvlan interface.

From a command prompt on your OMV host, type these lines

ip link add dockerlink link eth1 type macvlan mode bridge
ip addr add 192.168.1.240/24 dev dockerlink
ip link set dockerlink up

ip route add 192.168.1.240/28 dev dockerlink

Change eth1 for your OMV Host network adaptor - the same one you used as the Parent Network card when you setup the DockerNetworkConfig

The above instructions:

Now you should be able to ping your pihole IP address from the OMV host. Try it.

You should also be able to see it within the output of

ifconfig

..which should look something like:

dockerlink: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
inet 192.168.1.240  netmask 255.255.255.240  broadcast 0.0.0.0
inet6 <IPv6 address here>  prefixlen 64  scopeid 0x20<link>
ether <MAC address here>  txqueuelen 1000  (Ethernet)
RX packets 126920147  bytes 109338028767 (101.8 GiB)
RX errors 0  dropped 9  overruns 0  frame 0
TX packets 86345592  bytes 16466526025 (15.3 GiB)
TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

Did you make a mistake?

If you need to correct something, you'll need to remove the dockerlink, which you can do by typing

ip link set dockerlink down
ip link del dockerlink

Then you can go back and tweak anything you need to.

Make it persistent

Assuming all is good, you need to make dockerlink persistent so that it works after a reboot.

To do this, type the following at a command prompt on your OMV host:

sudo cat > /etc/network/interfaces.d/dockershim
auto dockerlink
iface dockerlink inet manual
pre-up ip link add dockerlink link eth1 type macvlan mode bridge
up ip addr add 192.168.1.240/24 dev dockerlink
post-up ip route add 192.168.1.240/28 dev dockerlink

Then press CTRL-D to save the file.

The file you just created will be read by the interfaces daemon when your OMV host starts, and effectively run the commands you tested.  pre-up is run before the interface comes up.  up is run to start the interface, and pre-up is run once it's up.

Now you need to restart your system - so type

sudo reboot

Once your OMV host restarts, check your OMV, Portainer and pihole GUIs - to validate your pihole container starts up. You should also try pinging pihole from your OMV host again, to ensure your dockerlink interface is working.

Step 8 : Traefik

Now you have OMV, hosting Docker, hosting Portainer, which lets you administer Docker.  It's a little bit meta, but it works.

Traefik is an "edge router" - designed to route incoming queries to your network (or device handling queries for your network) and send them where you need them to go.  The nice thing is, it can do almost all of it fairly automatically.  It talks to Docker to do this - but it also:

For this article, as mentioned right at the start, we're going to run Traefik as a Docker container, in host mode - so it will be on your OMV host IP. 

Install Traefik

To get Traefik running, it's similar to pihole - check the info on Docker Hub and see what volumes etc you need - however this is where the Author started to have problems, as some of the info is there, but all the other Traefik docs are on the Traefik website - but some volumes etc are use case specific - so this article will provide some shortcuts - and explain them as they come up.

First of all, setup a Docker folder for Traefik

sudo mkdir /srv/dev-disk-by-label-OMVDataVolume/Docker/traefik
sudo mkdir /srv/dev-disk-by-label-OMVDataVolume/Docker/traefik/dynamic

Create a blank holding file for your LetsEncrypt information

sudo touch /srv/dev-disk-by-label-OMVDataVolume/Docker/traefik/acme.json

Create a blank holding file for your Traefik static configuration

sudo touch /srv/dev-disk-by-label-OMVDataVolume/Docker/traefik/traefik.toml

Create a blank holding file for your Traefik access logs

sudo touch /var/log/traefik/access.log

Now we can start to setup the Traefik container.

A bit about DNS, LetsEncrypt and some security

It's important to add that LetsEncrypt only supports wildcard domains (i.e. *.yourdomain.com) - and if you have (or plan on having) many containers, and you want to reference them by machine name, ala container.yourdomain.com - you will need wildcards - and therefore you'll need a DNS provider that supports the LetsEncrypt ACME API.

There is a comprehensive (and growing) list of eligible DNS providers - but it's important to stress that while the Traefik website may list your provider, that provider may be linked to a specific version of LEGO - which Traefik may not have included in a release yet.    It's worth checking the LEGO release page for your provider and then checking if that version of LEGO is included in Traefik via the Traefik release page.  One reason this article took a while to write was that the author needed support for their DNS provider - LEGO supported it, but that LEGO version wasn't in Traefik until 2.2.2.

Next up - consider running your containers within a subdomain - not as a subdomain.  i.e. if you want to run ZoneMinder - consider zoneminder.yoursubdomain.yourdomain.com instead of zoneminder.yourdomain.com.  This is for three reasons:

This article will assume you're going to run your containers within a subdomain - and will build some security in for you as a result.

Configure Traefik (Static Config)

Traefik "on it's own" won't do a lot - it relies on a combination of static, dynamic and container configuration.  The static part comes from the traefik.toml configuration file - which can reference "dynamic" configuration files (which are read/re-read on the fly) and from Docker container labels (also read as they change).

Note 1 : Traefik uses the TOML configuration format - and the quote " marks, square brackets [] and backtick ` characters are important required.  If you find something isn't working. check you've not left out a backtick, quote or bracket.

Note 2 : The backtick ` character is NOT an apostrophe.  On most keyboards you can find it on the button left of the 1 button on the number row.

Open a command prompt on your OMV host and type:

sudo nano /srv/dev-disk-by-label-OMVDataVolume/Docker/traefik/traefik.toml

Within the editor, you want to put the following bits of config, which we'll explain as we go...

[global]
checkNewVersion = true
sendAnonymousUsage = false

This tells Traefik to check for new versions, as well as not to send any usage stats back to the maintainers.  This is of course up to you to opt in to/out of - changing false to true will enable it.

[entryPoints]
[entryPoints.web]
address = ":80"
[entryPoints.websecure]
address = ":443"

        [entryPoints.web.http.redirections.entryPoint]
to = "websecure"
scheme = "https"

The above defines two "entryPoints" - which are what Traefik defines (and opens) as valid ways into the Traefik router.  In this case we are defining two - one called web on port 80 (for http) and one called websecure on port 443 (for https).

There is also a rule setup to redirect ALL traffic arriving via the web entryPoint, and send it over to the websecure (https) entrypoint - meaning all web traffic will wind up being secured.  This is a good thing.

One of the brilliant things about Traefik is the built-in LetsEncrypt ACME functionality, which can handle all certificate registration and renewal for you, hands-free, automatically.  So let's sort out how that encryption will be handled.

[certificatesResolvers.mythicbeasts.acme]
email = "your-email-address@here.com"
storage = "/acme.json"

# Uncomment the next line for using the ACME staging server
# caServer = "https://acme-staging-v02.api.letsencrypt.org/directory"

[certificatesResolvers.mythicbeasts.acme.dnsChallenge]
resolvers = [ "ns1.mythic-beasts.com","ns2.mythic-beasts.com" ]
provider = "mythicbeasts"

This section defines the "certificate Resolvers" for your Traefik instance.  In the above, you can see the author is using Mythic Beasts for their DNS provision, and therefore needs to use them.  Obviously change "mythicbeasts" and the resolver name servers for your relevant DNS ACME tag and nameservers.

Also - while you are setting things up, it is really strongly recommended that you uncomment the # caServer line.  This will mean your Traefik instance initially talks to the LetsEncrypt test server.  This will mean you get "invalid" certificates, but you can examine them and prove the requests are working - and (more importantly) not lock yourself of LetsEncrypt for 5 days...

Then we need to ensure we have some log info that we can see in Portainer

[log]
level = "INFO"

The above provides generally enough info for you to see what's going on if you use the Portainer logging view.  You can change INFO to DEBUG if you want all the info you can get.  However, the above only deals with Traefik logs, not "accesses to" Traefik, so lets deal with that:

[accessLog]
filePath = "/var/log/traefik/access.log"
[accessLog.filters]
statusCodes = [ "301-302","400-499" ]

The above tells Traefik to write all it's web access logs to /var/log/traefik/access.log  - which we will need later for fail2ban to access and provide you some additional security.

The filter section tells Traefik that we're only interested in logging http status/error codes 301 & 302, and anything in the 400-499 range.  This is because "redirects" use 301 and 302, and all "unauthorized" type errors are within the 400 range.

Now lets configure the Traefik dashboard

[api]
insecure = false
dashboard = true

The above tells Traefik to disable the insecure access route (we'll define a secure one in a second) and enable the dashboard.  Now we need to get Traefik to see Docker...

[providers.docker]
endpoint = "unix:///var/run/docker.sock"
defaultRule = "Host(`{{ normalize .Name }}`)"
exposedByDefault = false
network = "bridge"

The above defines Docker as a Traefik information provider.  We're exposing the docker.sock file via a Volume we defined above, and we're setting up a default Rule to normalize hostnames.  We are disabling Traefik's ability to automatically wire up containers - so we retain control over what is exposed,  and we're telling Traefik to use the "bridge" network.  This is separate from the "host" Network config above.

Lastly we need to tell Traefik that we want to monitor a folder called /dynamic for any file based configuration.  This will mean that once Traefik is up, we can amend that file and Traefik will reload it instantly, so that there is no need to restart Traefik.

[providers.file]
directory = "/dynamic"

Now press Ctrl-X and press Y to save your file.

Configure Traefik (Dynamic Config)

Now we need to setup the dynamic pieces of Traefik, so, in the same command prompt window, type

sudo nano /srv/dev-disk-by-label-OMVDataVolume/Docker/traefik/dynamic/dynamic.toml

Start the file with

[file]
[http.routers]

That tells Traefik the file is a valid config file, and starts the http.routers configuration section

Now we need to define some static (yes, this is a dynamic file, but go with it) Traefik routers for a couple of dashboards and GUIs 

# This "traefikapi" router will provide access to the Traefik API
[http.routers.traefikapi]
# Define the hostname to be used
rule = "HostHeader(`traefik.yoursubdomain.yourdomain.com`)"
# Define the entrypoint
entrypoints = [ "websecure" ]
# Target the Traefik API service (internal)
service = "api@internal"
# Define the middlewares to use
middlewares = [ "blockexternal","traefik-basic-auth" ]
# Enable TLS
[http.routers.traefikapi.tls]
# Specify which resolver to use
certResolver = "mythicbeasts"

The above tells Traefik we have an http router called traefikapi - you could change traefikapi to something else, but it's what it is.  Going down the lines:

So, let's setup a router for Portainer:

# This "portainer" router provides access to Portainer
# Portainer does not like adding Traefik labels to itself via
# the Portainer UI so we do it here...
[http.routers.portainer]
# Define the hostname to be used
rule = "HostHeader(`portainer.yoursubdomain.yourdomain.com`)"
# Define the entrypoint
entrypoints = [ "websecure" ]
# Target the "portainer" service
service = "portainer"
# Define the middlewares
middlewares = [ "blockexternal" ]
# Enable TLS
[http.routers.portainer.tls]
# Specify which resolver to use
certResolver = "mythicbeasts"

The above is similar to the previous router, except:

Per the comment above, we define the Portainer router here, as Portainer really doesn't like adding labels to itself when you're using the Traefik access route.  You could try via http://youromvhost:9000  but it's safer in the file.

We also should have a router for OMV, so we can secure the GUI.  As it's not a Docker image, it has to go in here:

# Define the "omv" router
[http.routers.omv]
# Define the hostname to be used
rule = "HostHeader(`omv.yoursubdomain.yourdomain.com`)"
# Define the entrypoints
entrypoints = [ "websecure" ]
# Target the "omv" service
service = "omv"
# Define the middlewares
middlewares = [ "blockexternal" ]
# Enable TLS
[http.routers.omv.tls]
# Specify which resolver to use
certResolver = "mythicbeasts"

Nothing exciting in the above, just a new service.

Now let's define a "blackhole" router for external visits we don't want...

# Define a sink or blackhole
[http.routers.nohost]
# Define the hostname to be used
rule = "HostHeader(`yoursubdomain.yourdomain.com`)"
# Define the entrypoint
entrypoints = [ "websecure" ]
# Target the "nohost" service
service = "nohost"
# Define the middlewares to use
middlewares = [ "blockexternal" ]
# Enable TLS
[http.routers.nohost.tls]
# Specify which resolver to use
certResolver = "mythicbeasts"
# Define the domain to use, as wildcard
[[http.routers.nohost.tls.domains]]
sans = [ "*.yourdomain.com"]

You'll note that here we are not using a name within a subdomain, we are using the subdomain itself.  This is so that we can pick up any access to the subdomain itself (on http or https).  We are pointing this to a service (nohost) we will define below, and using the blockexternal middlewares again (more on that later). Additionally we are adding some info the tls section, to ensure the subdomain is picked up by LetsEncrypt.

Lastly, and importantly, we want to handle any traffic that arrives at Traefik with no hostname.  This could happen if someone targets your external forwarded IP - or tries to forward a CNAME or local hosts entry at your IP to probe it. 

# Define our catchall router
[http.routers.catcher]
# Catch anything with no path (/ is always requested)
rule = "PathPrefix(`/`)"
# Define the entrypoint
entrypoints = [ "websecure" ]
# Set this to execute last (last line of defence)
priority = 1
# Define the middlewares to use
middlewares = [ "bouncer" ]
# Target the "nohost" service
service = "nohost"
# Enable TLS
[http.routers.catcher.tls]
# Specify which resolver to use
certResolver = "mythicbeasts
[[http.routers.catcher.tls.domains]]
# Define the domain to use, as catchall/wildcard
sans = [ "*.yourdomain.com" ]

The above uses priority=1 to run after all other routers, giving them a chance to match things first.  By default. Traefik would just return a 404 Not Found error for any other request, but that looks like a "host to probe" to external naughty types - so we don't want that.  We're using a new middleware (bouncer) which we'll get to, and targeting nohost again.

So, now we can define our middlewares

Dynamic Middlewares

Create the HTTP middleware section of the file with

[http.middlewares]

First up, define an HTTP "basic authentication" (username/password) setup for the Traefik dashboard.

You'll need to create an encoded user/password combination first, and for that you'll need htpasswd

htpasswd -nb <user> <passwd>

Obviously swap <user> and <passwd> for your required user & password combination.  If you don't have htpasswd on your system (you may not), then you can use an online tool such as https://www.web2generators.com/apache-tools/htpasswd-generator 

If you use the username admin and a password of password you should get some output like this

admin:$apr1$uqxc0z9g$ukB361ceL17eKK7gBZSkG1 

So, lets put that within (under) the middlewares section header:

[http.middlewares.traefik-basic-auth.basicAuth]
users = [ "admin:$apr1$uqxc0z9g$ukB361ceL17eKK7gBZSkG1" ]

You'll notice that we named this middleware traefik-basic-auth - which we referred to in our manual router - so Traefik will now know how to link them together.

Now we can define some other middlewares to help police/restrict traffic, thus:

[http.middlewares.middlewares-rate-limit]
[http.middlewares.middlewares-rate-limit.rateLimit]
average = 100
burst = 50

The above will allow Traefik to throttle traffic if a client suddenly makes a lot of requests.  

Next:

[http.middlewares.internal-only-ip]
[http.middlewares.internal-only-ip.ipWhitelist]
sourceRange = [ "192.168.1.0/24" ]

The above creates an "internal-only-ip" middleware which includes a whitelist, limited to 192.168.1.0/24 - which is our example LAN range.  Amend if yours is different. 

Next:

[http.middlewares.bouncer]
[http.middlewares.bouncer.redirectRegex]
regex=".*"
replacement="https://www.somewebsite.com"

The above creates a middleware called "bouncer" which will redirect clients to a given URL. This will be used by Traefik to send clients who don't request a valid subdomain. Change www.somewebsite.com to somewhere you'd like any such traffic to go... 

Now lets define some rules around secure HTTP headers:

[http.middlewares.middlewares-secure-headers]
[http.middlewares.middlewares-secure-headers.headers]
accessControlAllowMethods= ["GET", "OPTIONS", "PUT"]
accessControlMaxAge = 100
hostsProxyHeaders = ["X-Forwarded-Host"]
sslRedirect = true
stsSeconds = 63072000
stsIncludeSubdomains = true
stsPreload = true
forceSTSHeader = true
contentTypeNosniff = true
browserXssFilter = true
# sslForceHost = true

# sslForceHost is now deprecated
referrerPolicy = "same-origin"
# featurePolicy = "camera 'none'; geolocation 'none'; microphone 'none'; payment 'none'; usb 'none'; vr 'none';"

# featurePolicy  is now deprecated - permissionsPolicy is the new key
permissionsPolicy = "accelerometer=(), camera=(), geolocation=(), gyroscope=(), magnetometer=(), microphone=(), payment=(), usb=(), vr=(), interest-cohort=()"

[http.middlewares.middlewares-secure-headers.headers.customResponseHeaders]
X-Robots-Tag = "none,noarchive,nosnippet,notranslate,noimageindex,"
server = ""

The above limits what clients can do (i.e. HTTP GET/PUT/OPTIONS), sets the age of connections, adds in proxy headers (so your servers will see client IPs), ensures redirection of HTTP to HTTPS, sets up strict transport security, and sets up some feature security options.  Finally, it adds in some headers to hopefully prevent any search engines attempt to spider your sites, should any be open or the search engine have/gain access.  Remove that last section if you don't mind being indexed.

Now we need to setup the middleware "connection" for oAuth security:

[http.middlewares.middlewares-oauth]
[http.middlewares.middlewares-oauth.forwardAuth]
address = "http://oauth.yoursubdomain.yourdomain.com:4181"
trustForwardHeader = true
authResponseHeaders = ["X-Forwarded-User"]

Note: The address line above must not include https:// - it should be http:// only

The above declares a "middlewares-oauth" middleware which uses forwardAuth - and points to a URL called oath.yoursubdomain.yourdomain.com on port 4181 (we'll be building that shortly - so don't worry).  It also tells the oAuth to trust forwarded headers and add in the X-Forwarded-User so your sites/logs know who logged in.

Now, it would be great to link some of the above middleware chains together, to combine their strengths.  Traefik lets you do this, so here is our first chain

[http.middlewares.chain-oauth]
[http.middlewares.chain-oauth.chain]
middlewares = [ "middlewares-rate-limit", "middlewares-secure-headers", "middlewares-oauth"]

The above creates a chain called chain-oauth which uses our rate limiter, secure headers and then oAuth policies, in that order.    Let's create a slight variant on that as well....

[http.middlewares.blockexternal]
[http.middlewares.blockexternal.chain]
middlewares = [ "middlewares-rate-limit", "middlewares-secure-headers", "internal-only-ip" ]

The above will apply rate limiting, secure headers, and restrict access to internal (LAN) IPs only.  So basically we have two policies - one for outside (external) clients and one for inside (internal) ones - and we'll apply those to containers a bit further on.

The oAuth middlewares will come into play later on, once the oAuth container is setup - right now, they will just be unused.

Lastly (for Traefik at least) we need to define some services for our manually defined routers - so now we'll create an HTTP services section

[http.services]
[http.services.portainer.loadBalancer]
[[http.services.portainer.loadBalancer.servers]]
url = "http://192.168.1.100:9000"
[http.services.omv.loadBalancer]
[[http.services.omv.loadBalancer.servers]]
url = "http://192.168.1.100:81"
[http.services.nohost.loadBalancer]
[[http.services.nohost.loadBalancer.servers]]
url = "http://192.168.1.100:81"

The first two entries above tell Traefik that Portainer lives at 192.168.1.100, which (if you remember) we used as our example IP for the OMV host- and it's on port 9000.  Amend the IP if yours is different.  Similarly we tell Traefik that the OMV GUI itself is on port 81, on the same IP.

Lastly, we have a service called nohost - and we will can this in a few ways:

Portainer Env(ironment)

Now go back to the Portainer tab you left open, and click on the Env section

Enter the variables and values for your specific provider (clicking on Additional configuration on that page will give you the variables/values you need).

Note:  In the example below, the author is (as noted) using Mythic Beasts.  You will have to check with your provider on how to get a user/password for their specific API.

IMPORTANT : If you are using Cloudflare, on the FREE tier, you will NOT be able to use "sub.sub.domain.com" - only "sub.domain.com" entries - so you could bear this in mind accordingly.

Now we're almost good to go - although at this point not everything will work, as we don't have some internal LAN DNS resolution - so that's our next step...

Start Traefik

Staying on the open Portainer tab, and click "Deploy the container" - if all has gone well, you should see Traefik showing as started - but don't try to access the GUI via your selected DNS name yet....

Step 9 : Local DNS

So far, we've setup Traefik routers for:

However, those won't be resolving anywhere for you to find (yet).  So now we're going to go back to piHole and set some local DNS entries up.

Local DNS Records

Login to pihole and choose Local DNS Records from the left menu.

Create an entry for OMV by entering omv.yoursubdomain.yourdomain.com (amending as necessary) and pointing that entry at 192.168.1.100 - your OMV host.

Testing/Explanation

From a client on your network which is using pihole as it's DNS server,  try to visit

http://omv.yoursubdomain.yourdomain.com

If all has gone as planned you should get your OMV GUI, which should be HTTPS secured!  (if it's not, given it a few minutes and try again)

Your browser has used your machine configured to get it's DNS from pihole, which has answered the query for omv.yoursubdomain.yourdomain.com and returned the IP of 192.168.1.100

Your browser has then made an HTTP request to 192.168.1.100 and asked for the site matching the name omv.yoursubdomain.yourdomain.com.  

HTTP uses port 80 by default, and Traefik is now answering queries on port 80.  Traefik will answer the request via it's web entrypoint - which will then redirect to websecure

Next, Traefik will answer the (redirected) query on it's websecure entrypoint - and use the hostheader to see the request is for omv.yoursubdomain.yourdomain.com.  Traefik will then use the omv router defined, to reverse proxy the request to 192.168.1.100:81 - and serve up the GUI.  

In the background, it will also have acquired the relevant HTTPS/TLS certificates, and provided those transparently - as LetsEncrypt will be providing a wildcard certificate.

Three more hosts...

Assuming the omv DNS entry worked, setup two additional entries in pihole:

Again, test those out.  Accessing the Traefik site should prompt you to login - with the user/password combination we created earlier.  The Portainer entry should send you to the Portainer GUI i.e. http://youromvhost:9000

It's worth noting that the oauth entry should give you an error as you've not built it yet - but don't worry, we'll test oAuth in a little while...

The Traefik dashboard doesn't let you actually do anything, but it does show a count of routers/services, and if there are any errors.  The errors can be useful in finding out if you're missing a bit of configuration or have a typo.

If you have got any errors, odds are they are in your dynamic Traefik file.  Any edits you make to that file are loaded by Traefik immediately once the file is saved, so it's quite fast to play with.  The Portainer "log" viewer for your Traefik container can also show you anything which may be amiss.

Step 10 : A moment of meta reflection...

Now you have OMV, Traefik and Portainer all accessible via custom internal URLs.  If you stop for a moment to think how meta this has got, you have:

Don't try to diagram the above, it's messy :)

Step 11 : pihole GUI access via Traefik

pihole is a funny beast.  If you access it via a web browser & IP normally, you get a logo splash screen with a link to the admin page (which you can bookmark) - but it would be nice to access it via something like http://pihole.yoursubdomain.yourdomain.com wouldn't it?  And get straight to the admin page?  And have it secure?

Let's use Traefik to get that done - and now you've completed all the hard work above, this is now where life gets much easier.

Login to your Portainer GUI (which you should be able to do via your new local DNS entry!) and browse to the Containers tab, find pihole, click it and then click Duplicate/Edit.

Labels

Once the page loads, scroll down to Advanced container settings and click Labels.  We're going to need to add several new labels, so click the grey +add label button nine (9) times,  and add the following label Names and Values

Name Value

If all goes well, you should have something that looks like this:

Labels, explained...

Lets breakdown the above:

So now, Traefik will be able to check any traffic coming in the web (port 80 - http) port and redirect it to websecure (port 443 - https), and start up a TLS connection.  If the requested host header is pihole.yoursubdomain.yourdomain.com then Traefik will match the rule, and apply the middlewares - checking if the request is internal or external.  If it's internal, it will rewrite the url request from https://pihole.yoursubdomain.yourdomain.com to be https://pihole.yoursubdomain.yourdomain.com/admin

Your browser URL will not show the /admin part, that's handled silently - so any visit to pihole will drop you directly at the admin page, saving a bit less typing and clicking.

Note: Technically, there should be an additional label defined, called traefik.http.routers.pihole.service with a value of pihole - which would force the http router to use the pihole service (as declared in the loadbalancer label) - however Traefik is clever enough to figure that out - but if you want you can add it in/remove it to see how it works.

Restart pihole - check Traefik

Now you can restart pihole by clicking Deploy the container - wait a few minutes whilst pihole restarts and then login to your Traefik dashboard.

First of all, click on HTTP Routers - you should see the ones defined in traefik.toml - i.e. omv, portainer and traefik.  They should all have a little shield on showing they are TLS enabled.  Additionally, you should see the new pihole router  showing it's HostHeader rule, using the websecure entrypoint with the name pihole@docker using the service pihole.

Note: The author has 16 routers defined - at this point, you should be showing 6, which should be:

Next click on HTTP Services - again you should see your pihole service defined, as pihole@docker with a type of loadbalancer and 1 server.

Additionally you should also see:

Note: The suffixes @internal, @file and @docker tell you where the configuration was defined. 

Finally click on HTTP Middlewares - you should see a Name of mw_piholeadmin@docker, denoting your defined pihole middlewares, and that it's of type addprefix

Additionally you should also see:
Name Type

These are all the middlewares defined in the dynamic.toml file earlier.

So, at this point you are now able to use Portainer to bring down any image, configure a container to use that image and define Labels such that Traefik will provide SSL access to it, as long as you use pihole to setup a local DNS entry.

It's worth clicking on the Rule names in HTTP Routers,  as you will then see a visual representation of how Traefik has built the router, thus:

Below that will also be additional details about the certificate used and the middlewares.

However, it won't show you the actual service (IP & port) it's going to - so click on HTTP Services and click on the service name (i.e. pihole@docker) and you'll see something like this:

...so you can validate the full end to end route.

Step 12 : External Domain Setup (and a quick test)

Now to sort out the external wiring and carry out a quick test to validate things - although this does assume you have access to your router and can setup IP forwarding.

External DNS Entry

Firstly, you'll need to login to your DNS provider and add a couple of entries:

The above will create a valid external DNS lookup for pihole.yoursubdomain.yourdomain.com

Router Port Forwarding

Now you need to ensure that external traffic can get to your server.  Login to your router GUI and find the port forwarding section.

Your specific router may vary, but need to ensure you forward any incoming TCP traffic on port 443 to the internal LAN address of your OMV host - which in this article has been assumed as 192.168.1.100.

You don't need to do any port forwarding for oAuth (and either way you've not configured it yet) - you just need a resolvable DNS entry for Google to make things work - the actual auth happens internally using the piHole DNS entry created earlier.

Security reminder...

Now, if you're worrying at this point, don't - this is because:

Validate the bounce!

From a machine NOT on your LAN (i.e. a mobile phone with 3G/4G/5G connection, open a browser and try to visit https://bouncetest.yoursubdomain.yourdomain.com

Assuming DNS has propagated, your browser should find your external internet IP and try to visit it.  The request will be forwarded into your OMV box, where Traefik will look at it.

As there is no rule for bouncetest.yoursubdomain.yourdomain.com - the PathPrefix rule will kick in (as the default https request will include a /) - and your browser should be redirected to the website you defined in the dynamic.toml!

This means anyone probing hostnames on your IP will get bounced, unless they get lucky and get a valid hostname, in which case blockexternal will throw them out - so let's check that...

Validate blockexternal

From the browser on your test device, try to reach https://pihole.yoursubdomain.yourdomain.com or https://traefik.yoursubdomain.yourdomain.com

You should see a page simply stating "Forbidden" - as your test device is not using an internal LAN IP address.  Traefik has denied access. All is good.

Prove valid traffic

Now it's a good time to show Traefik's dynamic nature.  Back in your Portainer GUI, open the pihole Container and click Duplicate/Edit, scroll down to the advanced section and open the Labels tab again.

Find the label that reads traefik.http.routers.pihole.middlewares and change the value from:

blockexternal@file,mw_piholeadmin

to:

mw_piholeadmin

...and then click Deploy the container.  Internally you'll lose DNS for a few seconds whilst pihole restarts, but once it does, you will see that in your Traefik dashboard, pihole is now not using blockexternal.

If you now go back to your test device and try accessing the https://pihole.yoursubdomain.yourdomain.com again - you should hopefully be presented with your pihole admin page - ideally requesting a login.  This proves that Traefik has handled the request, got it to the right container and is processing middlewares.

Presumably you won't want pihole to be accessible externally, so you can now:

Prove dynamic config

Now open up a command prompt on your OMV host and edit the dynamic.toml file

Find the [https.routers.traefikapi] section and within that, change the line that reads:

middlewares = [ "blockexternal","traefik-basic-auth" ]

to

middlewares = [ "traefik-basic-auth" ]

Save the file.  You can see the change pretty instantly in Traefik.  Now on your test device, try to access https://traefik.yoursubdomain.yourdomain.com

You should now be prompted for your admin username/password - and if you put them in, should get into Traefik!

If you're happy that's all working, edit dynamic.toml again and put the "blockexternal" piece back, and remove your external CNAME for traefik.yoursubdomain.yourdomain.com

Step 13 : Recap/pause

Now you have a fully working OMV host, with Docker, Portainer, Traefik (with automatic LetsEncrypt) and a pihole DNS server.  You have proven traffic is getting to your host and being bounced/blocked/allowed as necessary.

If you're happy with basic username/password security - you can simply create another piece of middleware similar to the one for traefik-basic-auth, and use it on a router.

Alternately you can use labels within Portainer - so if you wanted to use a username of piholeadmin (with the same password as you used for Traefik) you can simply add the label http.middlewares.mw_pihole-auth.basicAuth with the value 

users = [ "piholeadmin:$apr1$uqxc0z9g$ukB361ceL17eKK7gBZSkG1" ] 

Then you can update the existing label traefik.http.routers.pihole.middlewares from:

blockexternal@file,mw_piholeadmin

to

blockexternal@file,mw_pihole-auth,mw_piholeadmin

If you're happy allowing external access now you have a password, remove the blockexternal@file piece

Alternately, if you want better security, keep reading...

Step 14: Fail2ban

Fail2ban is a handy utility to keeps an eye on application logfiles, checking for specific events.  Depending on the rules that are applied, Fail2ban talks to iptables to block access to any IP which hits a rule enough times - i.e. 10 failed login attempts within 60 seconds results in that IP being blocked.

For webservers, you may to keep an eye on the entries 403 Forbidden (someone or something has failed to login) and 404 File Not Found  (someone or something may be probing for specific resources to see if your server is vulnerable).

Given that attacks can be scripted/automated, and most people don't actively review logs, Fail2ban can be handy in providing a deterrent to some attackers....

A bit more meta...

Given that we're playing heavily with Docker, this article will setup Fail2ban in another container, but will have control over your OMV host's iptables and access to the traefik container logs - which (if you recall) were already exposed when we setup the Traefik container - and can be found in /var/log/traefik on your OMV host.

Install Fail2ban

Login to your OMV host and run

sudo mkdir -p /srv/dev-disk-by-label-OMVDataVolume/Docker/fail2ban/data
sudo mkdir -p /srv/dev-disk-by-label-OMVDataVolume/Docker/fail2ban/run

This creates us two folders we need, within our Docker folder for fail2ban - data & run

Load up your Portainer GUI and click on Containers and then click the blue +Add container button

Container Value Host Value

The /var/log entry should be set to Read-only

We're not going to be using Traefik with Fail2ban directly, so we don't need to worry about Labels here.

Note:  Fail2ban keeps it's ban information in a sqlite3 database file, within /data.  As this is now mounted in a volume, it will ensure that bans are persistent across restarts of Fail2ban.

You can now click Deploy the container - and check if Fail2ban starts (it should). 

Configure Fail2ban for Traefik

 At this point we have no useful rules though, so let's create those...open a command prompt on your OMV host and copy/paste the following into it:

sudo cat <<EOF > /srv/dev-disk-by-label-OMVDataVolume/Docker/fail2ban/data/jail.local
[DEFAULT]
# action = %(action_mwl)s
ignoreip = 192.168.1.0/24
destemail = youremail@address.com
sender = youremail@address.com
EOF

Note: The assumption 

You should then edit the /srv/dev-disk-by-label-OMVDataVolume/Docker/fail2ban/data/jail.local to change your email address.

The above has an email command commented out (we'll come back to that) and tells Fail2ban to globally ignore anything from your LAN - as otherwise you'll get yourself locked out pretty fast - and if you're using a headless machine, that can be "a problem".

Now copy/paste the following into your OMV command prompt window:

sudo cat <<EOF > /srv/dev-disk-by-label-OMVDataVolume/Docker/fail2ban/data/filter.d/traefik-auth.local
[Definition]
# Parameter "method" can be used to specify request method
req-method = \S+
# Usage example (for jail.local):
#   filter = traefik-auth[req-method="GET|POST|HEAD"]
failregex = ^<HOST> \- <usrre-<mode>> \[\] \"(?:<req-method>) [^\"]+\" 401\b
            ^<HOST> \- <usrre-<mode>> \[\] \"(?:<req-method>) [^\"]+\" 403\b
            ^<HOST> \- <usrre-<mode>> \[\] \"(?:<req-method>) [^\"]+\" 404\b
ignoreregex =
# Parameter "mode": normal (default), ddos or aggressive
# Usage example (for jail.local):
#   [traefik-auth]
#   mode = aggressive
#   # or another jail (rewrite filter parameters of jail):
#   [traefik-auth-ddos]
#   filter = traefik-auth[mode=ddos]
mode = normal
# part of failregex matches user name (must be available in normal mode, must be empty in ddos mode, and both for aggressive mode):
usrre-normal = (?!- )<F-USER>\S+</F-USER>
usrre-ddos = -
usrre-aggressive = <F-USER>\S+</F-USER>
EOF

The above will create a filter rule file for Traefik which will look out for HTTP 401, 403 and 404 errors specifically.  It also deals with a small problem where if a user is identified in the logs, it would ignore them - so deals with lines with and without users.

Finally we need to enable the jail within Fail2ban that will use the rules, so copy/paste the following into the OMV command prompt window:

sudo cat <<EOF > /srv/dev-disk-by-label-OMVDataVolume/Docker/fail2ban/data/jail.d/traefik-auth.local
[traefik-auth]
destemail = youremail@address.com
sender = youremail@address.com
# action = %(action_mwl)s
enabled = true
port = http,https
filter = traefik-auth[mode=aggressive]
maxretry = 10
findtime = 900
logpath = /var/log/traefik/access.log
ignoreip = 192.168.1.0/24
EOF

The above sets up some email overrides (but we just make sure they match), again disables email, tells Fail2ban the jail is enabled, to monitor ports 80 and 443, use the filter traefik-auth we created above, and where to find the log file.  It also includes a check to not ban the internal LAN range.

Now restart your Fail2ban container so it takes advantage of the new configuration.

A little bit more secure...

Now Traefik's HTTP access log is being monitored by Fail2ban, checking for any 40x errors.  If any one IP generates more than 10, that IP will be REJECTed from accessing your server.  The author has found that in practice this is generally not a lot of hits, as the Traefik bouncer middleware is deflecting most random drivebys, and anyone trying to get into a specific host with any Traefik authorization method (i.e. basic-auth)  will likely fail and generate a 401 and eventually get banned.

Useful Commands

So far, we're not going to get email notifications (that comes later) - so the following commands may be of use - and they can be run from your OMV host command line:

sudo docker exec fail2ban fail2ban-client status

The above will list your Fail2ban jails (there should only be one - traefik-auth.

sudo docker exec fail2ban fail2ban-client status traefik-auth

The above will get you specific information about the traefik-auth jail - including any banned IP addresses.

sudo docker exec fail2ban fail2ban-client set traefik-auth banip 1.1.1.1

The above will ban a specific IP - in the example shown, it's 1.1.1.1 - change as necessary.

sudo docker exec fail2ban fail2ban-client set traefik-auth unbanip 1.1.1.1

The above will unban a specific IP - in the example shown, it's 1.1.1.1 - change as necessary.

Step 15 : oAuth with Google

Whilst Traefik is now set up to do a good job of deflecting random hits, and has basic-auth capabilities, and Fail2ban is there checking those, you should consider what happens if someone knows/cracks/obtains your basic-auth password OR (if you don't use basic-auth) what will stop attempts to login to the routed apps?

Fail2ban is currently only setup to monitor Traefik - and your individual routed web applications will have their own logs.  You could choose to make those application logs visible to Fail2ban (via some mounts) - and tweak Fail2ban to deal with them, or you could put another good, trusted layer on top of all of it - Google oAuth.

Granted this assumes you have a Google account, but using oAuth provides you with two really useful benefits:

Combining that with application security means you can effectively get 3 layers of security (Google account username and password, Google MFA + application security) - and any failed Google logins will show up in the Traefik HTTP access log and mean Fail2ban can deal with them for you.

Create an oAuth "App"

First of all you will need an oAuth "application" to use.  To do this, head over to the Google Developers console (https://console.developers.google.com/ ), and agree to the T's & C's.

Creating an app is easy, but finding the create button is a bit hard.  You need to click Select a project and then click NEW PROJECT

Give your project a name like oAuth or Your Domain Authorisation - it won't display anywhere outside this console.  Then click CREATE

Now select External for the User Type.  If however, you have a Google Apps account, you can link it to your GApps domain and use Internal.  Then click CREATE again.

Verify your domain (TXT and CNAME entry time)

Now you need to verify your domain with Google.  Click Domain verification and then Add domain

Then enter www.yourdomain.com in the box that pops up, thus:

Enter www.yourdomain.com and then click ADD DOMAIN and you will be told you need to verify the domain - so click the Take Me There button

Once you get to the next page, select your DNS provider (if they are in the list) or click Other.  You will be given a DNS TXT record to add to your domain, which will allow Google to query it and verify you own the domain, thus:

Note: You need to copy the whole string, starting with google-site-verification for your DNS TXT record, otherwise Google won't see it. 

Once you've added the TXT record, click VERIFY and (assuming DNS propagation has worked, Google should confirm you own the domain - which you'll need for the next step....

Note: It may take a few minutes for your DNS record update to be visible to Google - you may need to be patient here.

While you are adding the TXT record, you should also add a CNAME record for oauth.yoursubdomain.yourdomain.com and point it to your external IP address - as this will be used in a few minutes.

oAuth consent

Click oAuth consent screen on the left menu, and then select Public from the Application Type section on the right.  Then choose your Application name - this is what will display when you're requested to login.  Just under that section is the Support email box - select your email address from the dropdown provided.

A little further down you'll find four text entry boxes, which looks like this:

You can leave Authorised domains blank - but you should fill out the lower three to be compliant with Google Policy - even though (in theory) it'll only be you accessing the oAuth app!

Once you've added your links, click Save

Note: If you don't have a website, you could host one in an nginx or apache container behind Traefik 

Credentials

Finally you need the credentials that your application will use to communicate with Google.  Select Credentials from the left menu, then click + CREATE CREDENTIALS at the top and select oAuth client ID

You will then be given a screen to provide a name for these oAuth credentials.  First of all for Application type select Web application and then provide a Name (such as Your Domain SSO) - then click CREATE

You will then be given some credentials (Your Client ID and Your Client Secret) which you should copy/paste into a separate document (i.e. notepad) for later use.  You can get back to them, so don't worry.  Once you've copied them, click OK

Finally, you need to click back on Credentials on the left, and then you should see your oAuth 2.0 credential name that you entered, listed on the right.  Click it to get into the details.  You will see the Client ID and secret listed over at the top right, but if you look toward the bottom, you'll see an Authorised redirect URIs section.

In the box there, enter 

https://oauth.yoursubdomain.yourdomain.com/_oauth

...and then click SAVE

That's it, you've got the Google side all complete - now to get oAuth set up in Docker & Traefik!

oAuth Container

Log back into your Portainer GUI, and click on Containers and then Add container.  For the image name, use

thomseddon/traefik-forward-auth:latest

Scroll down to Advanced container settings and select Network

Now select Env and add 9 (nine) variables, as follows:

Now select Labels and add 8 labels, as follows:

Now select Restart policy and ensure that Always is selected.

You can now Deploy the container

You already created internal and external DNS entries for oauth.yoursubdomain.yourdomain.com in previous steps, so everything should now be in place...

Tweak Traefik

Open up your OMV host command prompt and edit the dynamic.toml file and find your traefikapi router

Change the line below it that reads

middlewares = [ "blockexternal","traefik-basic-auth" ]

to read:

middlewares = [ "blockexternal","chain-oauth@file" ]

Save it, and Traefik will reload itself.  Try logging into https://traefik.yoursubdomain.yourdomain.com now - and you should be prompted for a Google login first.  If you're already logged into Google, try it from another browser or an Incognito/Private Browsing window, where you will have no cookies etc.

Recap

The above has created an oAuth container, which a rather cool person (Thom Seddon) has written, which enables Traefik to call on Google's oAuth facilities.

If you now add the chain-oauth@file middlewares to any container (as you've now done for Traefik), this is the whole sequence of events:

Step 16 : Turning Fail2Ban email on

The Fail2ban image contains SMTP and SSMTP functionality built in - it's just that so far, no configuration info has been supplied.

This article has (so far) assumed you have a Google account, for oAuth - now we're going  to make use of your Google creds again!  

Note: This assumes you have access to an external SMTP server (i.e. Google or your ISP) and have an account you can use to connect to.

Update the Fail2Ban Container

Now log into your Portainer GUI, select the Fail2ban container and select Duplicate/edit 

Scroll down to Advanced container settings and select Env

Add 6 new variables, as follows:
Variable Value

Now before you click Deploy the container open up a command prompt on your OMV host and run

sudo nano /srv/dev-disk-by-label-OMVDataVolume/Docker/fail2ban/data/jail.local

Uncomment (remove the leading #) the line that reads:

# action = %(action_mwl)s

Press CTRL-X and press Y to save the file.  Now run 

sudo nano /srv/dev-disk-by-label-OMVDataVolume/SFDocker/fail2ban/data/jail.d/traefik-auth.local

Uncomment (remove the leading #) the line that reads:

# action = %(action_mwl)s

Press CTRL-X and press Y to save the file.  

Note: You may also want to validate the email addresses used in both files are able to send via your given SMTP server.  For safety, you may just wish to ensure all sender/recipient addresses are just your normal email address.

Now you can click Deploy the container  and you should hopefully shortly receive an email saying Fail2ban has started.  If you restart the container, you should get two emails, one saying it was stopped, and another saying it was started.

You will now receive emails if any bans occur.

Step 17 : Over to you...

So, now you can see how relatively easy it is to create a Container with Portainer, and use Portainer to apply the relevant labels for Traefik.  You can opt to use external DNS and the chain-oauth middlewares to make your applications securely accessible from outside.

You may also want to head over to Shodan (https://www.shodan.io/) and put your external IP in the box at the top, and see what Shodan knows about your IP.  It scans IPs and details what services are active.  Ideally all you'll see if you followed the above is an open TCP port (443) showing HTTPS services, and a valid LetsEncrypt certificate, and likely an HTTP 302 redirect to your chosen "bounce" site.

Shodan may (if you're unlucky, and it scans you during setup) pick up the TRAEFIK DEFAULT CERT, which will highlight you are using Traefik - but as long as you limit external IPs and are using SNI/URLs as detailed above, that should be all it sees!

Or, if you're still itching for more - why not read our guides to:

And, to help keep those Docker images you used up to date, you can check out our CWATCH container, which will alert you if any of them get new versions pushed to Docker Hub!

Have fun!

If this article was helpful, please let us know!

We'll publish some more useful Traefik & Container snippets in other articles.