Installing Traefik Enterprise Edition with Multiple Control Nodes on TraefikEE On-Premise

This installation guide is for experts who want to install their TraefikEE cluster (Traefik Enterprise Edition) with more than one control node in an on-premise environment.

Work In Progress

This documentation is a first version and will be improved in the next releases.

Requirements

  • In order to start a Traefikee cluster with 3 control nodes and 2 data nodes, prepare 6 Virtual Machines:
    • VM1: 10.10.0.1 → bootstrap-node
    • VM2: 10.10.0.2 → control-node-1
    • VM3: 10.10.0.3 → control-node-2
    • VM4: 10.10.0.4 → control-node-3
    • VM5: 10.10.0.5 → data-node-1
    • VM6: 10.10.0.6 → data-node-2
  • Bootstrap and control node VMs can reach https://v3.license.containous.cloud.

Virtual Machine networking

All of the virtual machines must be able to communicate with each other. Only the virtual machines which contain data nodes must be reachable from the internet.

Number of Virtual Machines

Multiple TraefikEE nodes can be installed on the same virtual machine. To allow it, each node must specify its:

  • advertise and listen address with its own port on the machine
  • sock
  • statedir

Please refer to the traefikee command-line tool reference documentation for more details.

Installation behind a proxy

In order to be able to install TraefikEE behind a proxy, you must make sure that each TraefikEE instance has the proper HTTP_PROXY and HTTPS_PROXY environment variables defined.

Install traefikee

First, download traefikee on each virtual machine, by using one of the following links:

Check the integrity of the downloaded file

Use the sha256 checksums of the binaries:

# Compare this value to the one found in traefikee_checksums.txt
sha256sum ./traefikee_v1.1.15_linux_amd64.tar.gz
# Compare this value to the one found in traefikee_checksums.txt
shasum -a256 ./traefikee_v1.1.15_darwin_amd64.tar.gz
# Compare this value to the one found in traefikee_checksums.txt
Get-FileHash traefikee_v1.1.15_windows_amd64.zip -Algorithm SHA256
Extract the downloaded archive

Use the following commands to extract the archive:

# Compare this value to the one found in traefikee_checksums.txt
tar -zxvf traefikee_v1.1.15_linux_amd64.tar.gz
# Compare this value to the one found in traefikee_checksums.txt
tar -zxvf -a256 ./traefikee_v1.1.15_darwin_amd64.tar.gz
# Compare this value to the one found in traefikee_checksums.txt
Expand-Archive traefikee_v1.1.15_windows_amd64.zip

Copy the traefikee binary to your PATH or add its location to your environment ($PATH or %PATH% depending on your OS) and make sure it's executable:

# Example with /usr/local/bin
# These command may need sudo rights
cp traefikee /usr/local/bin/traefikee
chmod a+x /usr/local/bin/traefikee

# Should print "/usr/local/bin/traefikee"
command -v traefikee
# Example with C:\Program Files
Copy-Item "traefikee.exe" -Destination "C:\Program Files\traefikee.exe"

# Should print "C:\Program Files\traefikee.exe"
where traefikee

You can now test your installation by executing traefikee:

traefikee
Usage: traefikee [flags] <command> [<arguments>]

Use "traefikee <command> --help" for help on any command.

Initialize the Bootstrap Node

By default, the traefikee bootstrap command line deploys a cluster with 1 Control Node. In order to initialize a cluster with more than one Control Node, the number of Control Nodes must provided thanks to the option --controlnodes.

TraefikEE uses the Raft protocol to manage its cluster.

To initialize a cluster with more than one Control Node, an ephemeral control node called Bootstrap node is created. It has to initialize the cluster with your license information and waiting for the initialization of the other Control Nodes.

In the example, the TraefikEE is created to run with 3 Control Nodes:

traefikee bootstrap \
    --advertise=10.10.0.1:4242 \
    --listen=10.10.0.1:4242 \
    --licensekey=fakeLicense \
    --api \
    --controlNodes=3 \
    --timeout=300
Node hostname

Nodes names are equal to VM hostname.

Customize the sock

By default, the TraefikEE sock is created in /var/run/traefikee.sock. The option --traefikeesocket allows customizing the path.

Customize the state directory

By default, the TraefikEE state directory is created in /var/lib/traefikee_state. The option --statedir allows customizing the path.

Customize the API credential generation directory

By default, TraefikEE stores its generated credentials (certificates and keys) in ./.traefikee. The option --mtlscertsoutputdir allows customizing the path in which they are be stored.

Restoring a backup

In order to perform a backup restoration, specify the --archivepath=./your-backup.tar option instead of the --licensekey option. To find out more about the backup and restore system, see here.

API credentials

When installing your cluster, TraefikEE will generate credentials (certificates and keys) that are necessary to communicate with its API. Make sure never to loose those credentials as TraefikEE will only generate them once per cluster. If you lost them, you will need to re-install your TraefikEE cluster.

Dashboard security

The option --api allows exposing the TraefikEE dashboard on the port 8080 only on Control Nodes. TraefikEE does not manage the security to access to its dashboard.

Bootstrap stopped after 5 minutes

If all the control nodes are not started within 5 minutes, the control plane is not initialized and then the bootstrap node will stop automatically and the cluster will not be created. To avoid this, you can tune the timeout parameter. For example, to make the bootstrap wait for up to 600 seconds:

  • --timeout=600

Check if the bootstrap node started by listing the cluster nodes from the virtual machine which contains the bootstrap node.

The bootstrap node's role should be Control Node:

traefikee list-nodes
Name           Role
----           --------------
bootstrap      CONTROL NODE (Current Leader)

Get Tokens

Get the control node token generated by the bootstrap node:

traefikee env | grep 'CONTROL_NODE_TOKEN' | cut -d '"' -f2

Repeat the same for the data node token:

traefikee env | grep 'DATA_NODE_TOKEN' | cut -d '"' -f2
How to use the tokens?

All the nodes require a token to start. Tokens can be stored in environment variables on each virtual machine, by setting it over SSH for example. In the following commands, tokens have been stored in environment variables:

  • ${CONTROL_NODE_TOKEN}: control node token
  • ${DATA_NODE_TOKEN}: data node token

Create the Control Nodes

Create the 3 control nodes for your cluster by launching the following commands on each control node virtual machine:

  • Control node 1
traefikee start-control-node \
    --peeraddresses=10.10.0.1:4242,10.10.0.3:4242,10.10.0.4:4242 \
    --advertise=10.10.0.2:4242 \
    --listen=10.10.0.2:4242 \
    --token=${CONTROL_NODE_TOKEN}
  • Control node 2
traefikee start-control-node \
    --peeraddresses=10.10.0.1:4242,10.10.0.2:4242,10.10.0.4:4242\
    --advertise=10.10.0.3:4242 \
    --listen=10.10.0.3:4242 \
    --token=${CONTROL_NODE_TOKEN}
  • Control node 3
traefikee start-control-node \
    --peeraddresses=10.10.0.1:4242,10.10.0.2:4242,10.10.0.3:4242\
    --advertise=10.10.0.4:4242 \
    --listen=10.10.0.4:4242 \
    --token=${CONTROL_NODE_TOKEN}

Verify that you have 4 TraefikEE nodes (3 control nodes and 1 bootstrap) that are control nodes by launching the following from a Control Node virtual machine:

traefikee list-nodes --details
ID                         Name            Membership  Status  Availability  Role
--                         ----            ----------  ------  ------------  ----
ge4ssbnjmsjw3h5bftrkkv5xv  control-node-3  ACCEPTED    READY   ACTIVE        CONTROL NODE
o92ldyepge8379vbo0ktaixi4  control-node-1  ACCEPTED    READY   ACTIVE        CONTROL NODE
v9m7zaq16ipo0ihhstovjo96r  bootstrap-node  ACCEPTED    DOWN    ACTIVE        CONTROL NODE
wnlqy33k9knic4eb44o4an0ve  control-node-2  ACCEPTED    READY   ACTIVE        CONTROL NODE (Current Leader)

Bootstrap down

The list-nodes command shows that the bootstrap node is in state DOWN. The bootstrap node automatically downs when the number of control nodes specified in the option controlNodes is reached or when the timeout is exceeded. If the bootstrap node is not automatically stopped, refer to the section below to stop it manually.

Remove the Bootstrap Node

TraefikEE requires an odd number of control nodes to work flawlessly.

Remove the Bootstrap node (by launching a kill command on the TraefikEE process on the bootstrap virtual machine) to have 3 control nodes (instead of 4):

ps -ef | grep traefikee | awk '{print $2}' | xargs kill -9

Periodically, the cluster cleans the node list and stops tracking the deleted nodes.

Check that there are 3 control nodes left after 1 minute:

traefikee list-nodes
Name            Role
----            --------------
control-node-1  CONTROL NODE (Current Leader)
control-node-3  CONTROL NODE
control-node-2  CONTROL NODE

Leader Election

The Bootstrap node was the previous leader and is now removed from the cluster. The Control nodes elected a new leader. This behavior is what happens when a control node, acting as a leader, is going down in production: TraefikEE stays available.

Get API credentials

When installing your cluster, TraefikEE will generate credentials (certificates and keys) that are necessary to communicate with its API. If you want to use traefikeectl to remotely access to the cluster, please move those credentials to the remote machine as described here before cleaning the bootstrap node file system. For more information please refer below.

Create Data Nodes

Create the data nodes, to handle your application traffic by launching the following commands on each data node virtual machine:

  • Data node 1
traefikee start-data-node \
    --peeraddresses=10.10.0.1:4242,10.10.0.2:4242,10.10.0.3:4242,10.10.0.4:4242 \
    --token=${DATA_NODE_TOKEN}
  • Data node 2
traefikee start-data-node \
    --peeraddresses=10.10.0.1:4242,10.10.0.2:4242,10.10.0.3:4242,10.10.0.4:4242 \
    --token=${DATA_NODE_TOKEN}

Check that the TraefikEE cluster contains the 2 data nodes:

traefikee list-nodes
Name                        Role
----                        --------------
control-node-3              CONTROL NODE
control-node-1              CONTROL NODE (Current Leader)
control-node-2              CONTROL NODE
data-node-1                 DATA NODE
data-node-2                 DATA NODE

Deploy a Configuration

A TraefikEE cluster is created without any default configuration. To allow control nodes to listen to a provider and data nodes to manage incoming traffic, execute the following command from one control node:

traefikee deploy --entryPoints='Name:http Address::80' \
    --entryPoints='Name:https Address::443 TLS' \
    --defaultentrypoints=http,https \
    --consul.endpoint=10.10.0.10:8500 \
    --consul.watch \
    --consul.prefix=traefik

The command described above allows the control nodes to get the backends and frontends stored in a Consul Key Value Store (reachable at the address 10.10.0.10:8500).

Data nodes expose these backends and frontends on the port 80 and 443 (for TLS).

Load balance between data nodes

TraefikEE does not manage load balancing between data nodes. It should be managed externally with an IP table rule for example.

File provider

The Traefik File Provider is not implemented yet in TraefikEE.

Remotely Accessing the TraefikEE Cluster with traefikeectl

The aforementioned operations require to be connected to a control node, because they are executed using the traefikee command. These operations can be executed from a remote machine instead, using traefikeectl.

Why use traefikeectl?

traefikeectl is a command-line tool which makes it easy to manage a TraefikEE cluster. It uses a secure internal API within TraefikEE to execute operations. However, it's also possible to manage a TraefikEE cluster by using the traefikee command from the control node VM.

API credentials deletion

When installing your cluster, TraefikEE will generate credentials (certificates and keys) that are necessary to communicate with its API on the bootstrap node. If you do not want to use traefikeectl, for more security, you can delete these credentials from the directory specified in the option --mtlscertsoutputdir of the bootstrap command (by default ./.traefikee from the install directory on the bootstrap node).

Requirements for traefikeectl

In order to use traefikeectl to remotely access to the cluster:

  • Check if the remote machine can access to the control nodes API port (55055)
  • Check if the files can be moved from the bootstrap node file system to the remote machine.

Install traefikeectl

The traefikeectl tool can be installed on the remote machine by following this documentation.

Get the Credentials

In order to access your cluster remotely with traefikeectl, you need to securely move the credentials from the bootstrap node to the file system of the machine from which you want to use traefikeectl.

On the bootstrap node, the credentials have been generated in the directory specified in the option --mtlscertsoutputdir of the bootstrap command (by default ./.traefikee from the install directory on the bootstrap node).

You can get them by using, for example, a scp command:

scp <BOOTSTRAP_NODE_IP>:<BOOTSTRAP_NODE_MTLS_CERT_DIR> <LOCAL_MTLS_CERT_DIR>

Connect traefikeectl to the New Cluster

Configure traefikeectl to have access to the new cluster.

traefikeectl connect --onpremise.mtlscertsdir="<LOCAL_MTLS_CERT_DIR>" --onpremise.apiaddress="<CONTROL_NODE_IP>:55055"
Retrieving TraefikEE Control credentials...ok
Removing cluster credentials from platform...ok
Credentials saved in "$HOME/.config/traefikee/traefikee", please make sure to keep them safe as they can never be retrieved again.
✔ Successfuly gained access to the cluster. You can now use other traefikeectl commands.
Using a VIP

As the IP address set in the option --onpremise.apiaddress will be the only IP address used to communicate with the cluster API, you can use a Virtual IP (VIP) instead of setting on control node IP. This VIP will allow accessing to all control nodes. Thus, if one control node is down, traefikeectl will be able to contact another one.

You can then provide the path to the certificates using the --onpremise.mtlscertsdir option. Do not forget to specify the --onpremise.apiaddress argument in order for traefikeectl to be able to communicate with the API.

One-time operation

When running traefikeectl connect, your credentials will be retrieved and it will not be possible to do it again in the future without re-installing a TraefikEE cluster. Remember to keep your credentials safe!

Validate the Connection

Check if the traefikeectl command is linked to the cluster by listing the cluster nodes:

traefikeectl list-nodes
Name                        Role              Leader
----                        ----              ------
control-node-3              Control Node
control-node-1              Control Node      YES
control-node-2              Control Node
data-node-1                 Data Node
data-node-2                 Data Node

You can now execute all traefikeectl commands.

What's Next?

Now that you have an up running cluster, you can configure your routing.