Skip to content

Installing Traefik Enterprise On-Premises

This installation guide is for users who want to install a cluster on-premises.

Requirements

In order to start a Traefik Enterprise cluster with 1 controller, 2 proxies and a plugin registry, ensure that:

  • There are at least 4 Hosts/Virtual Machines
  • Network stack allows communication on required ports
  • Controller Host/VM can reach https://v4.license.containous.cloud
  • The teectl binary is installed, for cluster management

Networking

Only the hosts which run proxies have to be reachable from the internet. On Linux, root access is needed to bind to ports below 1024.

Default Traefik Enterprise paths

The default configration paths for traefikee includes: - /var/run/traefikee - /var/lib/traefikee - /var/run/secrets Read and write access are required on them. See the teectl referece for customization options.

Please refer to the documentation for more information

Multiple instances on the same Host/VM

Running multiple controllers or proxies on the same host is not recommended, and should be used only when upgrading a cluster for which there are no spare hosts.

Installation behind a proxy

In order to be able to install Traefik Enterprise behind a proxy, you must make sure that each Traefik Enterprise instance has the proper HTTP_PROXY and HTTPS_PROXY environment variables defined.

Controller advertise address

The controller advertise address must be an external address reachable by the proxies. Do not use a dynamic address allocation feature, such as ":4242" under Linux, as it may propagate the wrong address to the proxies, causing a connection failure upon restart.

State Directory Management

By default, the statedir path is /var/lib/traefikee.

  • The state directory's contents are used to recover a cluster's state in the event of a controller restart. It has to persist between two executions.
  • In order to re-initialize a cluster, you have to delete the state directory before re-installing it. If this is not done, it will recover its previous state.

Install traefikee

First, download traefikee on each virtual machine, by using one of the following links:

Check the integrity of the downloaded file

Use the sha256 checksums of the binaries:

# Compare this value to the one found in traefikee_checksums.txt
sha256sum ./traefikee_v2.10.18_linux_amd64.tar.gz
# Compare this value to the one found in traefikee_checksums.txt
shasum -a256 ./traefikee_v2.10.18_darwin_amd64.tar.gz
# Compare this value to the one found in traefikee_checksums.txt
Get-FileHash traefikee_v2.10.18_windows_amd64.zip -Algorithm SHA256
Extract the downloaded archive

Use the following commands to extract the archive:

# Compare this value to the one found in traefikee_checksums.txt
tar -zxvf traefikee_v2.10.18_linux_amd64.tar.gz
# Compare this value to the one found in traefikee_checksums.txt
tar -zxvf -a256 ./traefikee_v2.10.18_darwin_amd64.tar.gz
# Compare this value to the one found in traefikee_checksums.txt
Expand-Archive traefikee_v2.10.18_windows_amd64.zip

Copy the traefikee binary to your PATH or add its location to your environment ($PATH or %PATH% depending on your OS) and make sure it's executable:

# Example with /usr/local/bin
# These command may need sudo rights
cp traefikee /usr/local/bin/traefikee
chmod a+x /usr/local/bin/traefikee

# Should print "/usr/local/bin/traefikee"
command -v traefikee
# Example with C:\Program Files
Copy-Item "traefikee.exe" -Destination "C:\Program Files\traefikee.exe"

# Should print "C:\Program Files\traefikee.exe"
where traefikee

You can now test your installation by executing traefikee:

traefikee --help
traefikee    Traefik Enterprise (Enterprise Edition) is a Cloud Native Edge Routing Platform based on Traefik Proxy,
a modern HTTP reverse proxy and load balancer made to deploy microservices with ease.
Complete documentation is available at https://doc.traefik.io.

Usage: traefikee [command] [resource] [flags]

Use "traefikee [command] --help" for help on any command.

Commands:
    controller    Start a new controller
    proxy         Start a new proxy
    tokens        Get tokens from the cluster
    version       Print version

Generating the teectl Configuration

Install the teectl management tool on a machine that has access to a controller node by following the instructions on this guide.

Once teectl is installed, it is possible to generate a set of cluster credentials that enables remote access to the API:

teectl setup --onpremise.hosts="<ip1>,<ip2>,<ip3>"

The host IPs that will be used for controllers should also be set in the onpremise.hosts option.

This command creates a new teectl configuration under ~/.config/traefikee/default.yaml and generates a bundle.zip which carries a set of needed assets for the installation. The bundle.zip needs to be copied onto each controller host.

To further configure the teectl setup, please refer to the reference documentation.

Connecting to a Controller Locally

You can connect to a local controller through a socket connection by specifying the socket option in the teectl command (defaults to /var/run/traefikee/teectl.sock).

teectl get nodes --socket=/var/run/traefikee/teectl.sock
teectl get nodes --socket=\\.\pipe\teectl.sock

In this case, no bundle.zip is needed to start the Controller and the above step can be skipped.

Generating a Pre-Shared Token

The Pre-Shared Token is used to secure communications between the controllers and the plugin registry.

Token Strength

As a security best practice, it is recommended that the token be at least 24 characters long made up of random characters.

A Pre-Shared Token can be generated using openssl:

openssl rand -base64 32
l9RLmLbAwrW73GzbDJ6+T+KOQ7u0sQt/we6OOGj0AKU

Alternatively an online generator can be used:

  1. Open the random string generator web page.
  2. Set Number of strings to 1.
  3. Set Length to 32.
  4. Click the Generate button.

Start the Controller

Start a new controller by specifying the advertise address and Traefik Enterprise license:

traefikee controller --advertise="10.0.0.1:4242" --license="<your-license-key>" --api.bundle="<path-to>/bundle.zip" --plugin.url="https://plugins.mydomain" --plugin.token="<pre-shared-token>"
traefikee controller --advertise="10.0.0.1:4242" --license="<your-license-key>" --api.bundle="<path-to>/bundle.zip" --discovery.static.peers="10.0.0.1:4242,10.0.0.2:4242,10.0.0.3:4242" --plugin.url="https://plugins.mydomain" --plugin.token="<pre-shared-token>"
traefikee controller --advertise="10.0.0.1:4242" --license="<your-license-key>" --api.bundle="<path-to>/bundle.zip" --discovery.dns.domain="traefikee.mydomain" --discovery.dns.port="4242" --discovery.dns.bootstrapexpected=2 --plugin.url="https://plugins.mydomain" --plugin.token="<pre-shared-token>"
INFO[2020-01-20T15:00:35-03:00] Using static discovery to discover other controllers  node=controller-0 role=controller
INFO[2020-01-20T15:00:35-03:00] Initializing a new node                       role=controller node=controller-0
INFO[2020-01-20T15:00:35-03:00] Discovering cluster                           node=controller-0 role=controller
INFO[2020-01-20T15:00:35-03:00] Found peers [10.0.0.1:4242]                    node=controller-0 role=controller
INFO[2020-01-20T15:00:35-03:00] Initial leader runs at 10.0.0.1:4242           node=controller-0 role=controller
INFO[2020-01-20T15:00:35-03:00] Initializing a new Traefik Enterprise cluster          node=controller-0 role=controller
INFO[2020-01-20T15:00:35-03:00] API started                                   role=controller node=controller-0
INFO[2020-01-20T15:00:35-03:00] Node started                                  role=controller node=controller-0
INFO[2020-01-20T15:00:36-03:00] Node is ready                                 node=controller-0 role=controller
INFO[2020-01-20T15:00:36-03:00] Running the cluster info controller           service=provider node=controller-0 role=controller
INFO[2020-01-20T15:00:36-03:00] Running the cluster license controller with frequency "24h0m0s"  node=controller-0 role=controller service=cleanup
INFO[2020-01-20T15:00:36-03:00] Running the cluster cleanup controller        role=controller node=controller-0 service=cleanup

Controller socket on Windows systems

When running on Windows systems, it is required that both the API and the cluster socket paths be defined as named pipes, for example:

traefikee controller --api.socket="\\.\pipe\teectl.sock" --socket="\\.\pipe\cluster.sock" ...

Node Name

By default Traefik Enterprise will use the hostname as the node. This behavior can be overridden by specifying the --name flag on the controller command.

Multi Controller with DNS

When using DNS as the discovery method it is a good practice to set the flag discovery.dns.bootstrapexpected, as this will make the controllers wait until the specified number of nodes are reachable before bootstraping the cluster.

Get the Tokens

Get the generated proxy node token:

traefikee tokens | grep 'TRAEFIKEE_PROXY_TOKEN' | cut -d '=' -f2

Get the generated controller node token:

traefikee tokens | grep 'TRAEFIKEE_CONTROLLER_TOKEN' | cut -d '=' -f2

Share the Tokens

All proxies require a token to start. Any additional controller also require a token to start.

Sharing the token between controllers and proxies can be achieved using environment variables or a shared persistent storage.

To use environment variables, set the token value directly with the command argument --jointoken.value. To use a shared persistent storage solution, set the path to the target directory with --jointoken.file.path on both proxies and controllers instead.

The remaining steps on this guide assumes the tokens are shared using environment variables named ${PROXY_NODE_TOKEN} and ${CONTROLLER_NODE_TOKEN}.

Start Additional Controllers (Multi Controller Only)

Start additional controllers by adding the join token flag:

traefikee controller --advertise="10.0.0.1:4242" --license="<your-license-key>" --api.bundle="<path-to>/bundle.zip" --discovery.static.peers="10.0.0.1:4242,10.0.0.2:4242,10.0.0.3:4242" --plugin.url="https://plugins.mydomain" --plugin.token="<pre-shared-token>" --jointoken.value="${CONTROLLER_NODE_TOKEN}"
traefikee controller --advertise="10.0.0.1:4242" --license="<your-license-key>" --api.bundle="<path-to>/bundle.zip" --discovery.dns.domain="traefikee.mydomain" --discovery.dns.port="4242" --discovery.dns.bootstrapexpected=2 --plugin.url="https://plugins.mydomain" --plugin.token="<pre-shared-token>" --jointoken.value="${CONTROLLER_NODE_TOKEN}"

Start Proxies

Start a proxy by specifying the controller discovery method, static in this example, and the join token:

traefikee proxy --jointoken.value="$PROXY_NODE_TOKEN" --discovery.static.peers="10.0.0.1:4242"
traefikee proxy --jointoken.value="$PROXY_NODE_TOKEN" --discovery.dns.domain="traefikee.mydomain"
INFO[2020-01-20T15:09:47-03:00] Applying a new static configuration           node=proxy-0 role=proxy module=server
INFO[2020-01-20T15:09:47-03:00] Using static discovery                        node=proxy-0 role=proxy peers="10.0.0.1:4242"
INFO[2020-01-20T15:09:47-03:00] Discovering controllers                       node=proxy-0 role=proxy
INFO[2020-01-20T15:09:47-03:00] Found controllers [10.0.0.1:4242]             node=proxy-0 role=proxy
INFO[2020-01-20T15:09:47-03:00] Waiting for the join tokens to be available   node=proxy-0 role=proxy
INFO[2020-01-20T15:09:47-03:00] Looking for a reachable peer                  role=proxy node=proxy-0
INFO[2020-01-20T15:09:47-03:00] Joining cluster                               role=proxy joined-peer="10.0.0.1:4242" node=proxy-0
INFO[2020-01-20T15:09:47-03:00] Node started                                  node=proxy-0 role=proxy
INFO[2020-01-20T15:09:47-03:00] Node is ready                                 node=proxy-0 role=proxy

Repeat the steps above for each proxy, then verify that the Traefik Enterprise installation was successful:

teectl get nodes
ID                         NAME           STATUS  ROLE
c5j53krue2avv77ajr8h5bcoz  controller-0   Ready   Controller (Leader)
3l5xt87fkc2ztlqlkwcpavuev  proxy-0        Ready   Proxy / Ingress
yjtz8kvnsgmqmuycup69vx180  proxy-1        Ready   Proxy / Ingress

Start the Plugin Registry

Start the plugin registry by specifying the controller discovery method, static in this example, and the join token:

traefikee plugin-registry --token="<pre-shared-token>" --jointoken.value="$PROXY_NODE_TOKEN" --discovery.static.peers="10.0.0.1:4242"

Apply a Static Configuration

A Traefik Enterprise cluster is created without any default configuration. To allow the controller to listen to a provider and proxies to manage incoming traffic, it is necessary to apply a static configuration.

Below is an example of a static configuration:

File provider

entryPoints:
  web:
    address: ":80"
  websecure:
    address: ":443"

providers:
  file:
    filename: dynamic_conf.yml
[entryPoints]
  [entryPoints.web]
    address = ":80"
  [entryPoints.websecure]
    address = ":443"

[providers]
  [providers.file]
    filename = "/etc/traefikee/dynamic_conf.toml"

Important

When running multiple controllers, make sure to replicate the configuration file on all nodes at the same path.

Apply the configuration using the following command:

teectl apply --file=config.yaml
teectl apply --file=config.toml

Systemd (Linux only)

When installing under Linux, you can leverage systemd to improve the management of a Traefik Enterprise deployment while also allowing it to run as a daemon.

Systemd limits

The suggested value of LimitNoFile=16384 is reasonable to start with but it should be increased for more demanding environments to prevent the error: accept4: too many open files. The system-wide limits that are typically defined at /etc/security/limits.conf are not inherited by Systemd. The value has to be configured explicitly in the Unit files that controlled the running of proxy and controller.

Another benefit is the ability to load a file of environment variables, which may contain your license key, node join token, and other sensitive information. This avoids setting variables on the host terminal or exposing the keys directly in the unit file.

VM/host preparation

It is always recommended to run system processes unprivileged. For that, a specific user and group must be created on each host/VM (shown here with example user and group IDs of 1500):

groupadd -g 1500 traefikee
useradd -g traefikee --no-user-group --home-dir="/opt/traefikee" --shell="/usr/sbin/nologin" --system --uid="1500" traefikee

OS Compatibility

This guide, including any example config files, was created using Debian Linux. There may be slight differences on other distributions.

Unit File Placement

Usually, systemd files can be placed under /etc/systemd/system, but this may not be the case on all distributions.

Unit Install Target

The examples use network-online as the target group dependency, but this may not be the same on distributions other than Debian. To find out what targets are actually available, run: systemctl list-units --type target

Controllers

Env File (optional)

Store the license key and the bind address for the controller and the preshared token into a new environment file.

Sensitive Information

Make sure this file has restricted permissions.

File permissions

We recommend giving this file restricted access as it contains sensitive information.

# Saved as /opt/traefikee/controller.env, update the bind address on each host according to your needs.
# Define your pre shared plugin registry token.
# Use an external address, reachable by the proxies.
CONTROLLER_BIND_ADDRESS="10.0.0.10:4242"
TRAEFIKEE_LICENSE="<my-license>"
REGISTRY_TOKEN="<my-preshared-token>"

Unit File

[Unit]
Description=Traefik Enterprise Controller
After=network-online.target
Wants=network-online.target systemd-networkd-wait-online.service

[Service]
EnvironmentFile=-/opt/traefikee/controller.env
Restart=on-abnormal
User=traefikee
Group=traefikee
ExecStart=/opt/traefikee/bin/traefikee controller --advertise=${CONTROLLER_BIND_ADDRESS} --license=${TRAEFIKEE_LICENSE} --api.socket=/opt/traefikee/run/teectl.sock --socket=/opt/traefikee/run/cluster.sock --statedir=/opt/traefikee/data --jointoken.file.path=/opt/traefikee/tokens --plugin.token=${REGISTRY_TOKEN} --api.autocerts
PrivateTmp=true
PrivateDevices=false
ProtectHome=true
ProtectSystem=full
ReadWritePaths=/opt/traefikee
PermissionsStartOnly=true
ExecStartPre=/usr/bin/mkdir -p /opt/traefikee/run /opt/traefikee/data /opt/traefikee/tokens
ExecStartPre=-chown -R traefikee.traefikee /opt/traefikee
ExecStartPre=-chmod -R 700 /opt/traefikee

NoNewPrivileges=true
LimitNOFILE=16384

[Install]
WantedBy=multi-user.target

For this guide, the systemd unit above was saved as /etc/systemd/system/traefikee-controller.service.

Systemd ExecStartPre paths

In some systemd configurations all ExecStartPre commands must have an absolute path. If this is the case, please update the Unit file to have absolute paths.

Plugin registry

Env File (optional)

Store the list of controller peers to connect to, the node join token and the pre shared token into a new environment file.

File permissions

We recommend giving this file restricted access as it contains sensitive information.

# Saved as /opt/traefikee/registry.env
CONTROLLER_PEERS="10.0.0.10:4242"
PROXY_NODE_TOKEN="<my-token>"
REGISTRY_TOKEN="<my-preshared-token>"

Unit File

[Unit]
Description=Traefik Enterprise registry
After=network-online.target
Wants=network-online.target systemd-networkd-wait-online.service

[Service]
EnvironmentFile=-/opt/traefikee/registry.env
Restart=on-abnormal
User=traefikee
Group=traefikee
ExecStart=/usr/local/bin/traefikee plugin-registry --name=traefikee-registry --plugindir=/opt/traefikee/registry/plugins --token=${REGISTRY_TOKEN} --discovery.static.peers=&{CONTROLLER_PEERS} --jointoken.value=${PROXY_JOIN_TOKEN} --statedir=/opt/traefikee/plugins/data
PrivateTmp=true
PrivateDevices=false
ProtectHome=true
ProtectSystem=full
ReadWritePaths=/opt/traefikee
ExecStartPre=mkdir -p /opt/traefikee/registry/data /opt/traefikee/registry/plugins
LimitNOFILE=16384

; The following additional security directives only work with systemd v229 or later.
CapabilityBoundingSet=CAP_NET_BIND_SERVICE
AmbientCapabilities=CAP_NET_BIND_SERVICE
NoNewPrivileges=true

[Install]
WantedBy=multi-user.target

For this guide, the systemd unit above was saved as /etc/systemd/system/traefikee-registry.service.

Systemd ExecStartPre paths

In some systemd configurations all ExecStartPre commands must have an absolute path. If this is the case, please update the Unit file to have absolute paths.

Proxies

Env File (optional)

Store the list of controller peers to connect to and the node join token into a new environment file.

File permissions

We recommend giving this file restricted access as it contains sensitive information.

# Saved as /opt/traefikee/proxies.env
CONTROLLER_PEERS="10.0.0.10:4242"
PROXY_NODE_TOKEN="<my-token>"

Unit File

[Unit]
Description=Traefik Enterprise proxy
After=network-online.target
Wants=network-online.target systemd-networkd-wait-online.service

[Service]
EnvironmentFile=-/opt/traefikee/traefikee.env
Restart=on-abnormal
User=traefikee
Group=traefikee
ExecStart=/opt/traefikee/bin/traefikee proxy --jointoken.value=${PROXY_NODE_TOKEN} --discovery.static.peers=${CONTROLLER_PEERS} --statedir=/opt/traefikee/data
PrivateTmp=true
PrivateDevices=false
ProtectHome=true
ProtectSystem=full
ReadWritePaths=/opt/traefikee
ExecStartPre=mkdir -p /opt/traefikee/data
LimitNOFILE=16384

; The following additional security directives only work with systemd v229 or later.
CapabilityBoundingSet=CAP_NET_BIND_SERVICE
AmbientCapabilities=CAP_NET_BIND_SERVICE
NoNewPrivileges=true

[Install]
WantedBy=multi-user.target

For this guide, the systemd unit above was saved as /etc/systemd/system/traefikee-proxy.service.

Systemd ExecStartPre paths

In some systemd configurations all ExecStartPre commands must have an absolute path. If this is the case, please update the Unit file to have absolute paths.

Tip

For the first boot, the node join token must be retrieved from a controller and updated on each proxy node before starting them. This can be automated or handled manually as a one-time operation.

With everything in place, proceed to enable the units files on systemd with:

systemctl enable traefikee-controller
systemctl enable traefikee-proxy
Created symlink /etc/systemd/system/multi-user.target.wants/traefikee-controller.service → /etc/systemd/system/traefikee-controller.service.
Created symlink /etc/systemd/system/multi-user.target.wants/traefikee-proxy.service → /etc/systemd/system/traefikee-proxy.service.

Now both the controller and proxies should be manageable by systemd as usual:

systemctl start traefikee-controller
systemctl start traefikee-proxy

Remote Access Through teectl

Once your cluster is ready, if you want to operate it remotely using the teectl tool, you need to generate credentials from your cluster using traefikee generate credentials on one of your controllers and use teectl to import the cluster credentials.

traefikee generate credentials --onpremise.hosts="<ip1>,<ip2>,<ip3>" --cluster=default --socket="/opt/traefikee/run/teectl.sock" > config.yaml
teectl cluster import --file="config.yaml"
traefikee generate credentials --onpremise.hosts="<ip1>,<ip2>,<ip3>" --cluster=default --socket="\\.\pipe\teectl.sock" > config.yaml
teectl cluster import --file="config.yaml"

You can now use teectl to operate your cluster.

teectl get nodes
ID                         NAME          STATUS  ROLE
c5j53krue2avv77ajr8h5bcoz  controller-0  Ready   Controller (Leader)
3l5xt87fkc2ztlqlkwcpavuev  proxy-0       Ready   Proxy / Ingress
yjtz8kvnsgmqmuycup69vx180  proxy-1       Ready   Proxy / Ingress

What's Next?

Now that the cluster is ready, we recommend reading the various operating guides to dive into all features that Traefik Enterprise provides.

License Monitoring

When a Traefik Enterprise controller starts for the first time, it checks the license validity.

If the license is valid, another check is done once every 24 hours.

If the controller can't communicate with the license server, a 72-day grace period starts to recover from this situation.

Once the grace period is over, the controller stops updating the proxies configuration.

Please look at the FAQ to know how to implement the license monitoring.