OMV, Docker, Portainer, Traefik, LetsEncrypt!

Prerequisites

This article assumes you have the following:

1. A server of some sort to install things onto

2. Some hard drives to go into the server (at least two, for OpenMediaVault)

3. An internet connection (you're reading this, aren't you?)

4. A router with some ability to amend IP config/routing (if required) and port forwarding (required)

5. A domain name

6. A domain registrar that provides an ACME V2 API (i.e. one of the ones listed here...)

7. A mobile internet device to test external things quickly via 3G/4G or other connection

8. Time

Step 1 : OpenMediaVault 5.0

Installation

First off, get over the OMV download page, and get a copy of the OMV 5 ISO. You can opt to burn it to CD and boot up to install OMV, or you can use a utility to put the ISO onto a USB stick and boot from that.

Note: All roads here start with OMV 5. If you're not wanting to install a new system from scratch, just bear in mind that OMV 5 is based on Debian linux, specifically Debian 10 (buster) - and some things later in this article may well rely on Debian 10 being the underlying host.

To steal from the previous OMV review:

"The installation process will be familiar to anyone who has installed Linux before - and maybe a little unfriendly for those that haven't (although there is a "quick" option). It's a text based menu selection system to let you customise your installation ("root or sudo" etc) - and probably the most important thing to note is that if you choose to install OMV - remove ANY disks you DON'T want the operating system to use BEFORE you install!"

"Yes, in an (admirable) attempt to keep the operating system and data on separate disks, OMV will "claim" anything it sees when it installs. So, if you install it on a shiny 1TB drive, you might be a bit upset when it only uses 2GB and you have a lot of unusable space. "

The above is quite important now , as most hard drives are "not small" (in fact it's getting harder to find small drives). When this author installed OMV 3, it was onto an 80GB drive. Now it's sat on a 1TB drive. But hey.

For this article, we're going to assume you've:

• Installed OMV

• Given it an IP address (or sorted one via DHCP) during installation

• Powered down, installed the disks you want to use for "main storage", powered back up

• Setup whatever RAID and FileSystem configuration you opt for

• Validated you have console access - directly or via SSH. The author recommends putty as an SSH client.

FYI, the author has a RAID-0 (mirror) configuration on a pair of disks, but (for the hell of it) has also hacked the OS drives to be in a RAID-0 (mirror) as well.

Minimal Configuration

Once you're into the admin GUI, there are a few things you should do to prepare things for later steps

• Change the web port

OMV uses port 80 (http) by default. You'll want this to be used by Traefik later, so in the OMV GUI, go to System > General Settings and change the Port from 80 to 81, and click Save (and then Apply Settings). You'll manually have to then change your browser URL to access the GUI on it's new URL (i.e. http://youromvhostip:81)
Note: Later on, we'll need to know the IP of your OMV host - so let's assume it's 192.168.1.100

• Setup a Shared Folder for Docker

Once you have a working File System, you can create Shared Folders. OMV uses these for most of it's features/plugins. Using the OMV GUI, go to Access Rights Management > Shared Folders and click Add

Type the name for the Shared Folder (i.e. Docker) and then select which storage Device you wish to place it on (from your list of File Systems). The Path should be added automatically, and you shouldn't need to change this. Leave Permissions alone. Click Save and (if prompted) Apply Settings.

Now you need to make a note of your full path to your new shared folder. OMV puts links to the File Systems under /srv - so if you named your File System "OMVDataVolume" (as the author did) then you should have a folder called
/srv/dev-disk-by-label-OMVDataVolume

Within that you will have a folder named whatever Path you chose when creating your Shared Folder for Docker, so in theory (if you didn't change it), something like:

/srv/dev-disk-by-label-OMVDataVolume/Docker

The author advises that you create a "host" folder within your Docker folder (you'll see why later) - so from a command prompt, type

sudo mkdir /srv/dev-disk-by-label-OMVDataVolume/Docker/host

..obviously changing "OMVDataVolume" for whatever you called your folder

So your full path (for Docker data) will be something like:

/srv/dev-disk-by-label-OMVDataVolume/Docker/host

Keep that path to hand, you'll need it shortly...

You are now at the minimum position you need to be in to get going with Docker - but first you need to get Docker. That comes with the "OMV Extras" package.

OMV Extras

Docker is not provided by OMV "out of the box" - but it's easy to get. You can visit https://omv-extras.org/ to validate these instructions, but all you need to do is:

1. Get to a command prompt on your OMV box

2. Type (or cut & paste) the following command

wget -O - https://github.com/OpenMediaVault-Plugin-Developers/packages/raw/master/install | bash

The next time you load your OMV GUI you will see an OMV-Extras item at the bottom of the System menu on the left.

Click Save and then Apply Settings when prompted (you can ignore the "Status" above, this screen shot was taken after install).

Now you need to select the Docker drop down and click Install

Docker will now install itself. Once the "Close" button goes blue, you can click it to close it.

Step 3 : Portainer

This is the easiest piece. Similar to installing Docker, you're going to use the same screen, but this time select the Portainer drop down, and click install. Again, once the "Close" button goes blue, you can click to close it.

The keen eyed amongst you may note the Docker Storage path above says "SFDocker" - and that's just because the author likes to prefix things, so it's easier to spot an actual Shared Folder from a normal folder.

Once you've 'Closed' the Portainer installation pop up, you should hopefully see the Open Web button next to the Portainer drop down. Clicking that will open the Portainer GUI, which is installed on port 9000 on your host - so you will now have two GUI URLS:

1. OMV http://yourhost:81

2. Portainer http://yourhost:9000

When you load Portainer for the first time, it will ask you to provide an admin password, and verify it. Click Create User and you'll then be asked which Docker environment you want to connect to. You should see four choices:

• Local - Remote - Agent - Azure

You want to click on Local and then click on Connect

You should then see something that looks like the following screen. "OMV" is the portainer instance picking up the name of the server (OMV).

The screenshot above shows the authors 17 containers and images, 5 networks and 5 volumes - but initially you should only be showing 3 networks and 1 container with 1 image - which will be Portainer.

That's it - you now have Portainer installed and are nearly ready to start (easily) adding Docker containers.

Step 4 : Docker Networking

There are four types of network that Docker can run a container on, or in, these are:

1. none
   A container will have no network stack, and not be accessible other than via file mounts

2. host
   A container will run with the host IP, and "expose" ports

3. bridge
   A container will run on a network Docker creates, only accessible from the Docker host itself and to other Docker containers via IP

4. macvlan
   A container will have it's own IP address and be fully accessible/routable on your network

This guide uses a blend of the above. Traefik will use host networking. Most containers will use bridge networking but a couple will use macvlan for ease of separation (as not everything needs to go via Traefik.

Setup macvlan

A macvlan network is just that, "a network" - it can support multiple hosts, as your Docker/Portainer instance will need to assign IPs from the macvlan range to your containers.

The first challenge is 'where' your macvlan range will live, which will determine it's size. Some considerations:

1. How big is your existing network?
   Your home network will be using an RFC1918 address, and will be one of the following:
   192.168.0.0/16 Technically allows for 65536 hosts
   172.16.0.0/12 Technically allows for 1048576 hosts
   10.0.0.0/8 Technically allows for 16777216 hosts
   Now, you may think "great", you have 65K spare IPs - but consider this - many home networks typically "only" readily support 255 hosts "out of the box" - this is because your router will typically be something like 192.168.1.1, or 192.168.0.1 and it will only assign/route addresses on it's subnet, which is typically a /24.

2. How many containers are you going to want?
   You need to provide a range of IPs big enough to support however many containers you might want to give a dedicated LAN IP to. Bear in mind not ALL your containers will need a macvlan address

3. Avoid collisions! (DHCP is out there)
   As noted in point 1 above, your router is probably limiting you to 255 addresses (and it's already stolen one). It will also likely be providing IP addresses to devices you connect, both wirelessly and wired. If you're not careful in your design, you may find Docker hands out an address your router may hand out to a device.

Note: The author advises using a decent router which can be flashed with something like DD-WRT or Merlin firmware, to give you maximum control of your network - assuming your router is capable of being flashed.

Note: The author has migrated their network to a 10.X.0.0/16 network, with dedicated ranges for DHCP and macvlan.

The author is not going to worry about what network you have (that's your problem) - but you may need to do some reading on subnet subvision :) This article will assume your LAN is setup as 192.168.1.0/24 and we'll be using 192.168.1.240/28 for macvlan - giving 14 usable IPs for Docker - which (given we'll also be using bridge mode for some) should be ample. If you're reading this far, the author assumes you're probably capable of tweaking your LAN to suit.

Docker handles macvlan setup in two pieces:

1. Configuration The definition of the network

2. Creation The implementation of the network

So, first of all, lets deal with the configuration

Assuming you logged into Portainer and clicked on your host, you should see the larger menu on the left side. Click Networks to open up the main network screen, and then click the blue + Add Network button at the top.

You then want to:

1. Provide a name for your macvlan network configuration (i.e. DockerNetworkConfig)
   (Don't call it "DockerNetwork", or ensure you append "Config" to whatever you do call it)

2. Change the driver to macvlan

3. Make sure Configuration is selected (it is the default selection)

4. Enter the name of your "Parent network card" - this will be the name of the "nic" (Network Interface Card) your OMV host is using for it's network access. If you don't know what that is (or can't remember), open a command prompt on your host, and run
   sudo ifconfig
   That will list all the network adaptors. You are looking for your OMV host IP - the line above it will have the name of the interface - so for the author, we see the following lines in amongst the output:
   eno1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
   inet 10.0.90.1 netmask 255.255.0.0 broadcast 10.0.255.255
   As you can see, the authors OMV interface is "eno1" but yours could be anything like eth0, eth1 etc.

5. If the IPV4 Network configuration section add your:

   1. Subnet This is your whole network subnet - not the DockerNetwork subnet (i.e. 192.168.1.0/24)

   2. Gateway This should be your router IP (i.e. 192.168.1.1)

   3. IP Range This is the DockerNetwork subnet - our /28 - so 192.168.1.240/28

   4. Exclude IPs If you're using a range where something exists, exclude any existing IPs here
      Note: There is a bug in portainer v1.2.4 which means exclude doesn't work. Be aware.

The screen below shows what you're looking for...

5. Scroll down and click "Create the network"

You should now see both DockerNetworkConfig and DockerNetwork listed in your Networks screen.

Did you make a mistake?

Normally, Docker and Portainer are quite forgiving - but unfortunately, network configuration is one place where mistakes cannot be easily fixed. If you want to change any part of the configuration, you'll need to delete DockerNetworkConfig and DockerNetwork and recreate them again.

Step 6 : First Docker Container - pihole

If you don't know, pihole is a network wide adblocker, which is really handy for cutting out on pesky ads and lowering your bandwidth consumption - which also speeds up your network a bit (less traffic).

However, besides blocking adverts, it also comes with a DHCP server, meaning you can use it as your central DNS and DHCP point - which the author finds very useful, as barring a configuration change on the main router, everything is in one GUI.

To setup a pihole container, you can either download the "Image" and then create the "Container", or you can just do it from the Container screen itself.

Basic Container Information

Click Containers and then click the blue + Add Container button.

Give your container a name, i.e. pihole and then tell it what Image you want. If you've already downloaded an image, typing part of it's name in the Image box will provide any matches you have locally, but so far we have none, so you can just type in

pihole/pihole

Note 1: You need to make a choice about the "Always pull the image" option. With Portainer V1.2.4, it defaults to "on", however, this means that everytime you amend configuration and (re)deploy your container, Portainer will check if there is a new version. If there is, it will download and try to run with your existing configuration. As noted above, the author found this broke things - so it may be safer to switch that off, and you manually control when to "pull" a new image down.

Note 2: If you use the :latest tag on any image, and have "Always pull the image" switched on, you can get into the situation described in Note 2 quite a lot, especially if you are using a popular, frequently updated image. Thus, you may want to pick a specific tag (such as pihole:v5.0) - or, once you're happy something is stable, "Exporting" a given image and tagging it as "Production" - we'll cover that later.

So, right now your new container screen should look like this...

...but don't click the "Deploy the container" button just yet...we need to setup a few things.

Reviewing the Container information

In another browser tab, go and look at https://hub.docker.com/r/pihole/pihole
Most DockerHub provided images detail the "volumes" (disks) and "environment" variables that an image might define if setup with docker compose - but we're using Portainer, so we need to replicate that differently in the Portainer GUI shortly.

The pihole info on the above link shows us:

services:

pihole:

container_name: pihole

image: pihole/pihole:latest

ports:

- "53:53/tcp"

- "53:53/udp"

- "67:67/udp"

- "80:80/tcp"

- "443:443/tcp"

environment:

TZ: 'America/Chicago'

# WEBPASSWORD: 'set a secure password here or it will be random'

# Volumes store your data between container upgrades

volumes:

- './etc-pihole/:/etc/pihole/'

- './etc-dnsmasq.d/:/etc/dnsmasq.d/'

dns:

- 127.0.0.1

- 1.1.1.1

# Recommended but not required (DHCP needs NET_ADMIN)

# https://github.com/pi-hole/docker-pi-hole#note-on-capabilities

cap_add:

- NET_ADMIN

restart: unless-stopped

As you can see:

1. The container_name is pihole, but you can set that to whatever you want (but we're using pihole)

2. The ports are the list of ports which docker will expose "through" your host if you are using host mode.
   In this article, we're going to use macvlan - so we don't need to do anything there - but to explain:

   1. 53:53/tcp This means link your host port 53 (tcp) to the pihole container port 53 (tcp)
      Port 53 is for DNS.

   2. 53:53/udp The same as the tcp item above, but using udp instead of tcp

   3. 67:67/udp As per port 53, this is exposing port 67, which is used for DHCP. It uses udp.

   4. 80:80/tcp This is the web interface (GUI) port for pihole

   5. 443:443/tcp This is the secure web interface (GUI) port for pihole

Once we get Traefik up (later on) you could put the pihole onto a bridge network and expose it via Traefik too - so it's always good to know which ports are (potentially) in use.

3. There is a TZ environment variable - which is used for TimeZone. The author uses "Europe/London" - you will need to find/pick yours

4. WEBPASSWORD is the pihole admin password. In the above example it's commented out, and says if you don't set it, it will generate a random one for you. Which means checking the container logs. Which is a pain, so we'll just set one in a minute.

5. volumes shows us there are two "drives" or "folders" the container uses:

   1. /etc/pihole The pihole data

   2. /etc/dnsmasq.d The dnsmasq (DHCP server) data

6. dns is setting the DNS servers for pihole - however if you scroll down the DockerHub page, you'll also see a table of other optional environment variables - including DNS1 and DNS2. We'll be using those.

7. cap_add refers to special capabilities your container can have. pihole needs to have NET_ADMIN

Creating Volumes

So, let's translate this into Portainer settings....on the create container screen, if you scroll down, you'll see a section that looks like this:

Typically Command & logging will be selected by default, but you can select each tab header and move around, without losing anything you've typed (or will type) in, as long as you don't navigate off the create container page.

We're not going to touch Command & logging, but Volumes is the next tab, so click it, and then click the grey + map additional volume button twice (as we need two volumes)

The Portainer GUI deals with volumes by asking about the container first, and the host second. This is the reverse of the docker-compose info we saw above, but we know we need two volumes, so let's key those in:

As you can see, we are telling the container that /etc/pihole will actually use:/srv/dev-disk-by-label-OMVDataVolume/Docker/pihole...and that /etc/dnsmasq.d will actually use:/srv/dev-disk-by-label-OMVDataVolume/Docker/dnsmasq

However, we don't have those folders yet - so open a command prompt on your OMV host and run

mkdir /srv/dev-disk-by-label-OMVDataVolume/Docker/pihole
mkdir /srv/dev-disk-by-label-OMVDataVolume/Docker/dnsmasq

That will create the two folders we need, ready for pihole to use.

Note: By default, Docker containers are not persistent - meaning anything you do inside one when it runs, will be lost when it restarts. This makes them ideal for testing things, but less ideal if you want a container to survive a reboot and carry on providing services. Volumes are the way to keep your data persistent.

Note: By creating the folders inside your Docker folder, but not in your Docker/host - you are keeping your Docker core information and pihole container information separate. This keeps folders a bit cleaner, as Docker itself keeps "a lot" of information within its host folder.

Configuring the Network

Now change to the Network tab - don't worry, as mentioned, nothing you've typed so far will be lost as long as you stay on the container page.

• For Network, select DockerNetwork from the dropdown.

• For Hostname, use piholeengine
  This is because we want to use another name for Traefik later. Your container will still be called pihole

• For Domain Name, enter yourdomain.name

• For IPv4 Address, enter 192.168.1.241
  192.168.1.240 is used by the network interface itself, so we need to specify the next one.

• For Primary DNS Server, enter 192.168.1.1
  ...or your router IP address

Note: This is the network setting for DNS, but pihole has it's own DNS settings it will take further down. Either way, pihole needs to get to at least one upstream DNS server - which typically is your router, although you could point it directly at your ISP DNS, or any chosen DNS provider (the author uses their router, but that points to OpenDNS)

Your screen should look something like this:

Configuring the Environment

Now we need to move to the Env tab, to setup the environment variables pihole will run with.

We know from the Docker Hub info that we need 4 variables (WEBPASSWORD, DNS1, DNS2 and TZ) so click the grey + add environment variable button four times and enter the following

Variable Name Value

WEBPASSWORD yourchosenpasswordhere
DNS1 192.168.1.1
DNS2 192.168.1.1
TZ Europe/London

Change WEBPASSWORD to be whatever password you want to login to the pihole GUI.
DNS1 and DNS2 have both been set to your router, so everything is forced to your router and nowhere else.
TZ (as noted above) should be set to whatever TimeZone you are in.

Shortly, when you start pihole for the first time, you'll find some other additional variables are set - but these are all we need for now.

We're going to leave the Labels tab for now, we'll come back to that later with Traefik, so lets skip to the next tab

Set the Restart Policy

Because pihole is going to be your network DNS service, it's a core service. You want it to be running all the time, which in Docker terms mean "unless stopped". If your OMV host is restarted, Docker is now installed as a service and will start automatically, and it in turn will start Portainer. It will also start pihole as long as we set the Restart policy to be unless stopped - and as long as it's run once.

So, within the Restart Policy tab, click unless stopped

We don't need to play with Runtime & Resources either, so we'll skip along

Amend Capabilities

Docker has the ability to provide containers with a lot of privileged access and system capabilities - so much so that on the author's 1280x1024 screen, the Capabilities tab cannot be displayed properly - so this next screenshot may look a little different as the browser has been "zoomed out" to 80%

It's advisable to do this also, as the rendering can otherwise place option sliders in "odd" places and you don't want to enable the wrong setting - but we know that NET_ADMIN is required. Make sure yours looks like the following screenshot:

Now the container is ready to go - you can scroll up a little. just above the Advancer container settings section and you'll see a "Deploy the container" button. Click it!

Test pihole

Now you can test pihole out, by either:

• Manually changing a device on your LAN to use 192.168.1.241 as it's DNS server
  or

• Changing your master DHCP (or network) settings on your router, to tell all DHCP devices to use 192.168.1.241 for DNS

You should then be able to see DNS accesses in the pihole GUI Dashboard and Query Log screens.

Did you make a mistake?

It's not impossible to mistype something, or realise you wanted to change a setting. Don't worry, unlike the macvlan piece, you can edit container configuration.

If you need to, go the Container screen, select pihole from your list of containers (in the Name column), and you'll then see a screen which has the following buttons at the top. If you click the blue Duplicate/Edit button, you will go back to a screen very much like creating a new container, but all your values will be as you keyed them. You can navigate around, change them, and then click "Deploy the container" again. Just remember the note re: "Always pull the image"

Step 8 : Traefik

Now you have OMV, hosting Docker, hosting Portainer, which lets you administer Docker. It's a little bit meta, but it works.

Traefik is an "edge router" - designed to route incoming queries to your network (or device handling queries for your network) and send them where you need them to go. The nice thing is, it can do almost all of it fairly automatically. It talks to Docker to do this - but it also:

• allows for manual configuration, so you can setup your own routing

• includes LetsEncrypt, so any website in your network can be given an SSL certificate automatically. It even renews them for you!

• provides filters, in the way of "router rules" and "middlewares" so you can secure access to services

For this article, as mentioned right at the start, we're going to run Traefik as a Docker container, in host mode - so it will be on your OMV host IP.

Install Traefik

To get Traefik running, it's similar to pihole - check the info on Docker Hub and see what volumes etc you need - however this is where the Author started to have problems, as some of the info is there, but all the other Traefik docs are on the Traefik website - but some volumes etc are use case specific - so this article will provide some shortcuts - and explain them as they come up.

First of all, setup a Docker folder for Traefik

sudo mkdir /srv/dev-disk-by-label-OMVDataVolume/Docker/traefik
sudo mkdir /srv/dev-disk-by-label-OMVDataVolume/Docker/traefik/dynamic

Create a blank holding file for your LetsEncrypt information

sudo touch /srv/dev-disk-by-label-OMVDataVolume/Docker/traefik/acme.json

Create a blank holding file for your Traefik static configuration

sudo touch /srv/dev-disk-by-label-OMVDataVolume/Docker/traefik/traefik.toml

Create a blank holding file for your Traefik access logs

sudo touch /var/log/traefik/access.log

Now we can start to setup the Traefik container.

1. Access your Portainer GUI and login

2. Go to the Container section and click the blue + Add container button

3. Set the Name to be traefik

4. Set the Image to be traefik:v2.2.7
   Note: Traefik has had major changes in the past, and traefik:latest is certainly a container you don't want breaking further down the line. This article has been written using Traefik V2.2.7. You can swap images once you are setup, but ensure you are stable first. Traefik image tags are available by clicking here

5. On the Volumes tab, let's reference the outline files you created above, and give Traefik access to Docker

   1. Click the grey map additional volume button five times. These will ALL be Bind volumes - so click Bind on each

      1. Volume 1 (LetsEncrypt persistent config)
         Container Path /acme.json
         Host Path /srv/dev-disk-by-label-OMVDataVolume/Docker/traefik/acme.json

      2. Volume 2 (Traefik dynamic config folder)
         Container Path /dynamic
         Host Path /srv/dev-disk-by-label-OMVDataVolume/Docker/traefik/dynamic

      3. Volume 3 (Traefik main config)
         Container Path /traefik.toml
         Host Path /srv/dev-disk-by-label-OMVDataVolume/Docker/traefik/traefik.toml

      4. Volume 4 (Traefik access log folder)
         Container Path /var/log/traefik
         Host Path /var/log/traefik

      5. Volume 5 (Traefik access to Docker) - this should be set "Read-only"
         Container Path /var/run/docker.sock
         Host Path /var/run/docker.sock

6. On the Network tab

   1. Change the Network to host

   2. Set the hostname to traefik

   3. Set the primary DNS IP to your pihole IP (192.168.1.241)

7. On the Environment tab, create one environment variable - it will be TZ, per the pihole TZ

8. On the Restart policy tab, ensure unless stopped is selected

9. Hold off on clicking "Deploy the container" for a moment - still a few things to do...leave the browser tab open :)

A bit about DNS, LetsEncrypt and some security

It's important to add that LetsEncrypt only supports wildcard domains (i.e. *.yourdomain.com) - and if you have (or plan on having) many containers, and you want to reference them by machine name, ala container.yourdomain.com - you will need wildcards - and therefore you'll need a DNS provider that supports the LetsEncrypt ACME API.

There is a comprehensive (and growing) list of eligible DNS providers - but it's important to stress that while the Traefik website may list your provider, that provider may be linked to a specific version of LEGO - which Traefik may not have included in a release yet. It's worth checking the LEGO release page for your provider and then checking if that version of LEGO is included in Traefik via the Traefik release page. One reason this article took a while to write was that the author needed support for their DNS provider - LEGO supported it, but that LEGO version wasn't in Traefik until 2.2.2.

Next up - consider running your containers within a subdomain - not as a subdomain. i.e. if you want to run ZoneMinder - consider zoneminder.yoursubdomain.yourdomain.com instead of zoneminder.yourdomain.com. This is for three reasons:

1. People may guess (or try) for well know container names/applications under your domain

2. If no-one knows about your subdomain, they're unlikely going to guess/stumble on your containers within it

3. You can still use your subdomain for other features

This article will assume you're going to run your containers within a subdomain - and will build some security in for you as a result.

Configure Traefik (Static Config)

Traefik "on it's own" won't do a lot - it relies on a combination of static, dynamic and container configuration. The static part comes from the traefik.toml configuration file - which can reference "dynamic" configuration files (which are read/re-read on the fly) and from Docker container labels (also read as they change).

Note 1 : Traefik uses the TOML configuration format - and the quote " marks, square brackets [] and backtick ` characters are important required. If you find something isn't working. check you've not left out a backtick, quote or bracket.

Note 2 : The backtick ` character is NOT an apostrophe. On most keyboards you can find it on the button left of the 1 button on the number row.

Open a command prompt on your OMV host and type:

sudo nano /srv/dev-disk-by-label-OMVDataVolume/Docker/traefik/traefik.toml

Within the editor, you want to put the following bits of config, which we'll explain as we go...

[global]
checkNewVersion = true
sendAnonymousUsage = false

This tells Traefik to check for new versions, as well as not to send any usage stats back to the maintainers. This is of course up to you to opt in to/out of - changing false to true will enable it.

[entryPoints]
[entryPoints.web]
address = ":80"
[entryPoints.websecure]
address = ":443"

[entryPoints.web.http.redirections.entryPoint]
to = "websecure"
scheme = "https"

The above defines two "entryPoints" - which are what Traefik defines (and opens) as valid ways into the Traefik router. In this case we are defining two - one called web on port 80 (for http) and one called websecure on port 443 (for https).

There is also a rule setup to redirect ALL traffic arriving via the web entryPoint, and send it over to the websecure (https) entrypoint - meaning all web traffic will wind up being secured. This is a good thing.

One of the brilliant things about Traefik is the built-in LetsEncrypt ACME functionality, which can handle all certificate registration and renewal for you, hands-free, automatically. So let's sort out how that encryption will be handled.

[certificatesResolvers.mythicbeasts.acme]
email = "your-email-address@here.com"
storage = "/acme.json"

# Uncomment the next line for using the ACME staging server
# caServer = "https://acme-staging-v02.api.letsencrypt.org/directory"

[certificatesResolvers.mythicbeasts.acme.dnsChallenge]
resolvers = [ "ns1.mythic-beasts.com","ns2.mythic-beasts.com" ]
provider = "mythicbeasts"

This section defines the "certificate Resolvers" for your Traefik instance. In the above, you can see the author is using Mythic Beasts for their DNS provision, and therefore needs to use them. Obviously change "mythicbeasts" and the resolver name servers for your relevant DNS ACME tag and nameservers.

Also - while you are setting things up, it is really strongly recommended that you uncomment the # caServer line. This will mean your Traefik instance initially talks to the LetsEncrypt test server. This will mean you get "invalid" certificates, but you can examine them and prove the requests are working - and (more importantly) not lock yourself of LetsEncrypt for 5 days...

Then we need to ensure we have some log info that we can see in Portainer

[log]
level = "INFO"

The above provides generally enough info for you to see what's going on if you use the Portainer logging view. You can change INFO to DEBUG if you want all the info you can get. However, the above only deals with Traefik logs, not "accesses to" Traefik, so lets deal with that:

[accessLog]
filePath = "/var/log/traefik/access.log"
[accessLog.filters]
statusCodes = [ "301-302","400-499" ]

The above tells Traefik to write all it's web access logs to /var/log/traefik/access.log - which we will need later for fail2ban to access and provide you some additional security.

The filter section tells Traefik that we're only interested in logging http status/error codes 301 & 302, and anything in the 400-499 range. This is because "redirects" use 301 and 302, and all "unauthorized" type errors are within the 400 range.

Now lets configure the Traefik dashboard

[api]
insecure = false
dashboard = true

The above tells Traefik to disable the insecure access route (we'll define a secure one in a second) and enable the dashboard. Now we need to get Traefik to see Docker...

[providers.docker]
endpoint = "unix:///var/run/docker.sock"
defaultRule = "Host(`{{ normalize .Name }}`)"
exposedByDefault = false
network = "bridge"

The above defines Docker as a Traefik information provider. We're exposing the docker.sock file via a Volume we defined above, and we're setting up a default Rule to normalize hostnames. We are disabling Traefik's ability to automatically wire up containers - so we retain control over what is exposed, and we're telling Traefik to use the "bridge" network. This is separate from the "host" Network config above.

Lastly we need to tell Traefik that we want to monitor a folder called /dynamic for any file based configuration. This will mean that once Traefik is up, we can amend that file and Traefik will reload it instantly, so that there is no need to restart Traefik.

[providers.file]
directory = "/dynamic"

Now press Ctrl-X and press Y to save your file.

Configure Traefik (Dynamic Config)

Now we need to setup the dynamic pieces of Traefik, so, in the same command prompt window, type

sudo nano /srv/dev-disk-by-label-OMVDataVolume/Docker/traefik/dynamic/dynamic.toml

Start the file with

[file]
[http.routers]

That tells Traefik the file is a valid config file, and starts the http.routers configuration section

Now we need to define some static (yes, this is a dynamic file, but go with it) Traefik routers for a couple of dashboards and GUIs

# This "traefikapi" router will provide access to the Traefik API
[http.routers.traefikapi]
# Define the hostname to be used
rule = "HostHeader(`traefik.yoursubdomain.yourdomain.com`)"
# Define the entrypoint
entrypoints = [ "websecure" ]
# Target the Traefik API service (internal)
service = "api@internal"
# Define the middlewares to use
middlewares = [ "blockexternal","traefik-basic-auth" ]
# Enable TLS
[http.routers.traefikapi.tls]
# Specify which resolver to use
certResolver = "mythicbeasts"

The above tells Traefik we have an http router called traefikapi - you could change traefikapi to something else, but it's what it is. Going down the lines:

• The rule line tells Traefik to use this router if the requested website host header is traefik.yoursubdomain.yourdomain.com - if it isn't this particular router won't be used.

• The entryPoints line tells Traefik to also only use this router when the request came in via the websecure (https) entryPoint we defined back in the main traefik.toml file

• The service line references an internal Traefik service called api@internal

• The middlewares line tells Traefik to apply some rules (to be defined below) to the router

• The http.routers.traefikapi.tls line enables TLS for this router, and the certResolver refers back to the section defined in the main traefik.toml file

So, let's setup a router for Portainer:

# This "portainer" router provides access to Portainer
# Portainer does not like adding Traefik labels to itself via
# the Portainer UI so we do it here...
[http.routers.portainer]
# Define the hostname to be used
rule = "HostHeader(`portainer.yoursubdomain.yourdomain.com`)"
# Define the entrypoint
entrypoints = [ "websecure" ]
# Target the "portainer" service
service = "portainer"
# Define the middlewares
middlewares = [ "blockexternal" ]
# Enable TLS
[http.routers.portainer.tls]
# Specify which resolver to use
certResolver = "mythicbeasts"

The above is similar to the previous router, except:

• The service is not internal to traefik, and therefore we need to change it. We'll define it further down.

• The middlewares are different (we're not using the same auth, as Portainer has it's own login)

Per the comment above, we define the Portainer router here, as Portainer really doesn't like adding labels to itself when you're using the Traefik access route. You could try via http://youromvhost:9000 but it's safer in the file.

We also should have a router for OMV, so we can secure the GUI. As it's not a Docker image, it has to go in here:

# Define the "omv" router
[http.routers.omv]
# Define the hostname to be used
rule = "HostHeader(`omv.yoursubdomain.yourdomain.com`)"
# Define the entrypoints
entrypoints = [ "websecure" ]
# Target the "omv" service
service = "omv"
# Define the middlewares
middlewares = [ "blockexternal" ]
# Enable TLS
[http.routers.omv.tls]
# Specify which resolver to use
certResolver = "mythicbeasts"

Nothing exciting in the above, just a new service.

Now let's define a "blackhole" router for external visits we don't want...

# Define a sink or blackhole
[http.routers.nohost]
# Define the hostname to be used
rule = "HostHeader(`yoursubdomain.yourdomain.com`)"
# Define the entrypoint
entrypoints = [ "websecure" ]
# Target the "nohost" service
service = "nohost"
# Define the middlewares to use
middlewares = [ "blockexternal" ]
# Enable TLS
[http.routers.nohost.tls]
# Specify which resolver to use
certResolver = "mythicbeasts"
# Define the domain to use, as wildcard
[[http.routers.nohost.tls.domains]]
sans = [ "*.yourdomain.com"]

You'll note that here we are not using a name within a subdomain, we are using the subdomain itself. This is so that we can pick up any access to the subdomain itself (on http or https). We are pointing this to a service (nohost) we will define below, and using the blockexternal middlewares again (more on that later). Additionally we are adding some info the tls section, to ensure the subdomain is picked up by LetsEncrypt.

Lastly, and importantly, we want to handle any traffic that arrives at Traefik with no hostname. This could happen if someone targets your external forwarded IP - or tries to forward a CNAME or local hosts entry at your IP to probe it.

# Define our catchall router
[http.routers.catcher]
# Catch anything with no path (/ is always requested)
rule = "PathPrefix(`/`)"
# Define the entrypoint
entrypoints = [ "websecure" ]
# Set this to execute last (last line of defence)
priority = 1
# Define the middlewares to use
middlewares = [ "bouncer" ]
# Target the "nohost" service
service = "nohost"
# Enable TLS
[http.routers.catcher.tls]
# Specify which resolver to use
certResolver = "mythicbeasts
[[http.routers.catcher.tls.domains]]
# Define the domain to use, as catchall/wildcard
sans = [ "*.yourdomain.com" ]

The above uses priority=1 to run after all other routers, giving them a chance to match things first. By default. Traefik would just return a 404 Not Found error for any other request, but that looks like a "host to probe" to external naughty types - so we don't want that. We're using a new middleware (bouncer) which we'll get to, and targeting nohost again.

So, now we can define our middlewares

Dynamic Middlewares

Create the HTTP middleware section of the file with

[http.middlewares]

First up, define an HTTP "basic authentication" (username/password) setup for the Traefik dashboard.

You'll need to create an encoded user/password combination first, and for that you'll need htpasswd

htpasswd -nb <user> <passwd>

Obviously swap <user> and <passwd> for your required user & password combination. If you don't have htpasswd on your system (you may not), then you can use an online tool such as https://www.web2generators.com/apache-tools/htpasswd-generator

If you use the username admin and a password of password you should get some output like this

admin:$apr1$uqxc0z9g$ukB361ceL17eKK7gBZSkG1

So, lets put that within (under) the middlewares section header:

[http.middlewares.traefik-basic-auth.basicAuth]
users = [ "admin:$apr1$uqxc0z9g$ukB361ceL17eKK7gBZSkG1" ]

You'll notice that we named this middleware traefik-basic-auth - which we referred to in our manual router - so Traefik will now know how to link them together.

Now we can define some other middlewares to help police/restrict traffic, thus:

[http.middlewares.middlewares-rate-limit]
[http.middlewares.middlewares-rate-limit.rateLimit]
average = 100
burst = 50

The above will allow Traefik to throttle traffic if a client suddenly makes a lot of requests.

Next:

[http.middlewares.internal-only-ip]
[http.middlewares.internal-only-ip.ipWhitelist]
sourceRange = [ "192.168.1.0/24" ]

The above creates an "internal-only-ip" middleware which includes a whitelist, limited to 192.168.1.0/24 - which is our example LAN range. Amend if yours is different.

Next:

[http.middlewares.bouncer]
[http.middlewares.bouncer.redirectRegex]
regex=".*"
replacement="https://www.somewebsite.com"

The above creates a middleware called "bouncer" which will redirect clients to a given URL. This will be used by Traefik to send clients who don't request a valid subdomain. Change www.somewebsite.com to somewhere you'd like any such traffic to go...

Now lets define some rules around secure HTTP headers:

[http.middlewares.middlewares-secure-headers]
[http.middlewares.middlewares-secure-headers.headers]
accessControlAllowMethods= ["GET", "OPTIONS", "PUT"]
accessControlMaxAge = 100
hostsProxyHeaders = ["X-Forwarded-Host"]
sslRedirect = true
stsSeconds = 63072000
stsIncludeSubdomains = true
stsPreload = true
forceSTSHeader = true
contentTypeNosniff = true
browserXssFilter = true
# sslForceHost = true

# sslForceHost is now deprecated
referrerPolicy = "same-origin"
# featurePolicy = "camera 'none'; geolocation 'none'; microphone 'none'; payment 'none'; usb 'none'; vr 'none';"

# featurePolicy is now deprecated - permissionsPolicy is the new key
permissionsPolicy = "accelerometer=(), camera=(), geolocation=(), gyroscope=(), magnetometer=(), microphone=(), payment=(), usb=(), vr=(), interest-cohort=()"

[http.middlewares.middlewares-secure-headers.headers.customResponseHeaders]
X-Robots-Tag = "none,noarchive,nosnippet,notranslate,noimageindex,"
server = ""

The above limits what clients can do (i.e. HTTP GET/PUT/OPTIONS), sets the age of connections, adds in proxy headers (so your servers will see client IPs), ensures redirection of HTTP to HTTPS, sets up strict transport security, and sets up some feature security options. Finally, it adds in some headers to hopefully prevent any search engines attempt to spider your sites, should any be open or the search engine have/gain access. Remove that last section if you don't mind being indexed.

Now we need to setup the middleware "connection" for oAuth security:

[http.middlewares.middlewares-oauth]
[http.middlewares.middlewares-oauth.forwardAuth]
address = "http://oauth.yoursubdomain.yourdomain.com:4181"
trustForwardHeader = true
authResponseHeaders = ["X-Forwarded-User"]

Note: The address line above must not include https:// - it should be http:// only

The above declares a "middlewares-oauth" middleware which uses forwardAuth - and points to a URL called oath.yoursubdomain.yourdomain.com on port 4181 (we'll be building that shortly - so don't worry). It also tells the oAuth to trust forwarded headers and add in the X-Forwarded-User so your sites/logs know who logged in.

Now, it would be great to link some of the above middleware chains together, to combine their strengths. Traefik lets you do this, so here is our first chain

[http.middlewares.chain-oauth]
[http.middlewares.chain-oauth.chain]
middlewares = [ "middlewares-rate-limit", "middlewares-secure-headers", "middlewares-oauth"]

The above creates a chain called chain-oauth which uses our rate limiter, secure headers and then oAuth policies, in that order. Let's create a slight variant on that as well....

[http.middlewares.blockexternal]
[http.middlewares.blockexternal.chain]
middlewares = [ "middlewares-rate-limit", "middlewares-secure-headers", "internal-only-ip" ]

The above will apply rate limiting, secure headers, and restrict access to internal (LAN) IPs only. So basically we have two policies - one for outside (external) clients and one for inside (internal) ones - and we'll apply those to containers a bit further on.

The oAuth middlewares will come into play later on, once the oAuth container is setup - right now, they will just be unused.

Lastly (for Traefik at least) we need to define some services for our manually defined routers - so now we'll create an HTTP services section

[http.services]
[http.services.portainer.loadBalancer]
[[http.services.portainer.loadBalancer.servers]]
url = "http://192.168.1.100:9000"
[http.services.omv.loadBalancer]
[[http.services.omv.loadBalancer.servers]]
url = "http://192.168.1.100:81"
[http.services.nohost.loadBalancer]
[[http.services.nohost.loadBalancer.servers]]
url = "http://192.168.1.100:81"

The first two entries above tell Traefik that Portainer lives at 192.168.1.100, which (if you remember) we used as our example IP for the OMV host- and it's on port 9000. Amend the IP if yours is different. Similarly we tell Traefik that the OMV GUI itself is on port 81, on the same IP.

Lastly, we have a service called nohost - and we will can this in a few ways:

1. You can use an invalid internal URL - any visitor to a service using nohost will just get a "Bad Gateway" error

2. You can use a valid internal URL - but you'll want to ensure said URL has it's own blocking/security policy

3. Use a valid internal URL so you can use it internally, but ensure the blockexternal middleware is linked - so anyone outside your LAN is just given a Forbidden error.

Portainer Env(ironment)

Now go back to the Portainer tab you left open, and click on the Env section

Enter the variables and values for your specific provider (clicking on Additional configuration on that page will give you the variables/values you need).

Note: In the example below, the author is (as noted) using Mythic Beasts. You will have to check with your provider on how to get a user/password for their specific API.

IMPORTANT : If you are using Cloudflare, on the FREE tier, you will NOT be able to use "sub.sub.domain.com" - only "sub.domain.com" entries - so you could bear this in mind accordingly.

Step 9 : Local DNS

So far, we've setup Traefik routers for:

• traefik.yoursubdomain.yourdomain.com

• portainer.yoursubdomain.yourdomain.com

• omv.yoursubdomain.yourdomain.com

However, those won't be resolving anywhere for you to find (yet). So now we're going to go back to piHole and set some local DNS entries up.

Local DNS Records

Login to pihole and choose Local DNS Records from the left menu.

Create an entry for OMV by entering omv.yoursubdomain.yourdomain.com (amending as necessary) and pointing that entry at 192.168.1.100 - your OMV host.

Step 11 : pihole GUI access via Traefik

pihole is a funny beast. If you access it via a web browser & IP normally, you get a logo splash screen with a link to the admin page (which you can bookmark) - but it would be nice to access it via something like http://pihole.yoursubdomain.yourdomain.com wouldn't it? And get straight to the admin page? And have it secure?

Let's use Traefik to get that done - and now you've completed all the hard work above, this is now where life gets much easier.

Login to your Portainer GUI (which you should be able to do via your new local DNS entry!) and browse to the Containers tab, find pihole, click it and then click Duplicate/Edit.

Step 13 : Recap/pause

Now you have a fully working OMV host, with Docker, Portainer, Traefik (with automatic LetsEncrypt) and a pihole DNS server. You have proven traffic is getting to your host and being bounced/blocked/allowed as necessary.

If you're happy with basic username/password security - you can simply create another piece of middleware similar to the one for traefik-basic-auth, and use it on a router.

Alternately you can use labels within Portainer - so if you wanted to use a username of piholeadmin (with the same password as you used for Traefik) you can simply add the label http.middlewares.mw_pihole-auth.basicAuth with the value

users = [ "piholeadmin:$apr1$uqxc0z9g$ukB361ceL17eKK7gBZSkG1" ]

Then you can update the existing label traefik.http.routers.pihole.middlewares from:

blockexternal@file,mw_piholeadmin

to

blockexternal@file,mw_pihole-auth,mw_piholeadmin

If you're happy allowing external access now you have a password, remove the blockexternal@file piece

Alternately, if you want better security, keep reading...

Step 14: Fail2ban

Fail2ban is a handy utility to keeps an eye on application logfiles, checking for specific events. Depending on the rules that are applied, Fail2ban talks to iptables to block access to any IP which hits a rule enough times - i.e. 10 failed login attempts within 60 seconds results in that IP being blocked.

For webservers, you may to keep an eye on the entries 403 Forbidden (someone or something has failed to login) and 404 File Not Found (someone or something may be probing for specific resources to see if your server is vulnerable).

Given that attacks can be scripted/automated, and most people don't actively review logs, Fail2ban can be handy in providing a deterrent to some attackers....

A bit more meta...

Given that we're playing heavily with Docker, this article will setup Fail2ban in another container, but will have control over your OMV host's iptables and access to the traefik container logs - which (if you recall) were already exposed when we setup the Traefik container - and can be found in /var/log/traefik on your OMV host.

Install Fail2ban

Login to your OMV host and run

sudo mkdir -p /srv/dev-disk-by-label-OMVDataVolume/Docker/fail2ban/data
sudo mkdir -p /srv/dev-disk-by-label-OMVDataVolume/Docker/fail2ban/run

This creates us two folders we need, within our Docker folder for fail2ban - data & run

Load up your Portainer GUI and click on Containers and then click the blue +Add container button

• Name the container fail2ban

• For the Image, use crazymax/fail2ban:latest

• Under Advanced container settings

  • Volumes

    • You need to add 3, all set as Bind, as follows:

Container Value Host Value

• /var/run/fail2ban /srv/dev-disk-by-label-OMVDataVolume/Docker/fail2ban/run

• /data /srv/dev-disk-by-label-OMVDataVolume/Docker/fail2ban/data

• /var/log /var/log

The /var/log entry should be set to Read-only

• Network

  • Set the Network to host

  • Set the hostname to fail2ban

  • Set the Domain Name to yoursubdomain.yourdomain.com

  • Set the Primary DNS Server to 192.168.1.241

• Environment

  • Add a named variable called F2B_IPTABLES_CHAIN and set it's value to INPUT

  • Add a named variable called F2B_LOG_LEVEL and set it's value to DEBUG

  • Note: We'll come back to SMTP notifications later. We'll also set DEBUG to INFO later.

• Restart policy

  • Set this to Unless stopped

• Capabiilities

  • As Fail2ban needs to control iptables, enable the NET_ADMIN setting

We're not going to be using Traefik with Fail2ban directly, so we don't need to worry about Labels here.

Note: Fail2ban keeps it's ban information in a sqlite3 database file, within /data. As this is now mounted in a volume, it will ensure that bans are persistent across restarts of Fail2ban.

You can now click Deploy the container - and check if Fail2ban starts (it should).

Configure Fail2ban for Traefik

At this point we have no useful rules though, so let's create those...open a command prompt on your OMV host and copy/paste the following into it:

sudo cat <<EOF > /srv/dev-disk-by-label-OMVDataVolume/Docker/fail2ban/data/jail.local
[DEFAULT]
# action = %(action_mwl)s
ignoreip = 192.168.1.0/24
destemail = youremail@address.com
sender = youremail@address.com
EOF

Note: The assumption

You should then edit the /srv/dev-disk-by-label-OMVDataVolume/Docker/fail2ban/data/jail.local to change your email address.

The above has an email command commented out (we'll come back to that) and tells Fail2ban to globally ignore anything from your LAN - as otherwise you'll get yourself locked out pretty fast - and if you're using a headless machine, that can be "a problem".

Now copy/paste the following into your OMV command prompt window:

sudo cat <<EOF > /srv/dev-disk-by-label-OMVDataVolume/Docker/fail2ban/data/filter.d/traefik-auth.local
[Definition]
# Parameter "method" can be used to specify request method
req-method = \S+
# Usage example (for jail.local):
# filter = traefik-auth[req-method="GET|POST|HEAD"]
failregex = ^<HOST> \- <usrre-<mode>> \[\] \"(?:<req-method>) [^\"]+\" 401\b
^<HOST> \- <usrre-<mode>> \[\] \"(?:<req-method>) [^\"]+\" 403\b
^<HOST> \- <usrre-<mode>> \[\] \"(?:<req-method>) [^\"]+\" 404\b
ignoreregex =
# Parameter "mode": normal (default), ddos or aggressive
# Usage example (for jail.local):
# [traefik-auth]
# mode = aggressive
# # or another jail (rewrite filter parameters of jail):
# [traefik-auth-ddos]
# filter = traefik-auth[mode=ddos]
mode = normal
# part of failregex matches user name (must be available in normal mode, must be empty in ddos mode, and both for aggressive mode):
usrre-normal = (?!- )<F-USER>\S+</F-USER>
usrre-ddos = -
usrre-aggressive = <F-USER>\S+</F-USER>
EOF

The above will create a filter rule file for Traefik which will look out for HTTP 401, 403 and 404 errors specifically. It also deals with a small problem where if a user is identified in the logs, it would ignore them - so deals with lines with and without users.

Finally we need to enable the jail within Fail2ban that will use the rules, so copy/paste the following into the OMV command prompt window:

sudo cat <<EOF > /srv/dev-disk-by-label-OMVDataVolume/Docker/fail2ban/data/jail.d/traefik-auth.local
[traefik-auth]
destemail = youremail@address.com
sender = youremail@address.com
# action = %(action_mwl)s
enabled = true
port = http,https
filter = traefik-auth[mode=aggressive]
maxretry = 10
findtime = 900
logpath = /var/log/traefik/access.log
ignoreip = 192.168.1.0/24
EOF

The above sets up some email overrides (but we just make sure they match), again disables email, tells Fail2ban the jail is enabled, to monitor ports 80 and 443, use the filter traefik-auth we created above, and where to find the log file. It also includes a check to not ban the internal LAN range.

Now restart your Fail2ban container so it takes advantage of the new configuration.

A little bit more secure...

Now Traefik's HTTP access log is being monitored by Fail2ban, checking for any 40x errors. If any one IP generates more than 10, that IP will be REJECTed from accessing your server. The author has found that in practice this is generally not a lot of hits, as the Traefik bouncer middleware is deflecting most random drivebys, and anyone trying to get into a specific host with any Traefik authorization method (i.e. basic-auth) will likely fail and generate a 401 and eventually get banned.

Useful Commands

So far, we're not going to get email notifications (that comes later) - so the following commands may be of use - and they can be run from your OMV host command line:

sudo docker exec fail2ban fail2ban-client status

The above will list your Fail2ban jails (there should only be one - traefik-auth.

sudo docker exec fail2ban fail2ban-client status traefik-auth

The above will get you specific information about the traefik-auth jail - including any banned IP addresses.

sudo docker exec fail2ban fail2ban-client set traefik-auth banip 1.1.1.1

The above will ban a specific IP - in the example shown, it's 1.1.1.1 - change as necessary.

sudo docker exec fail2ban fail2ban-client set traefik-auth unbanip 1.1.1.1

The above will unban a specific IP - in the example shown, it's 1.1.1.1 - change as necessary.