In a multi-controller deployment it can take only a single controller failure for a cluster to become unavailable.
A cluster becomes unavailable when a quorum is not longer able to be reached by the Controllers, preventing a new
cluster leader from being elected. The number of controllers needed for a quorum to be reached is
N is the initial number of controllers. For example, if there are 3 Controllers initially, there would need to be
at least 2 controllers to form a quorum.
While a cluster is unavailable, the proxies will continue to operate with their last known configuration.
In a single Controller deployment, if the state of the controller is lost during a controller failure, it must be restored from a backup.
Learn more about Backup and Restore.
Recovering a Cluster¶
In order for the cluster to be recoverable, at least one controller must remain with its state intact. If this is not the case, see the next section.
The cluster can be recovered by using
teectl cannot connect due to a loss of credentials, the
recover command can be run from
This connects to the remaining controller and perform the recovery operation. Watch the controller logs to see the progress of the recovery. Look for the following log message to know the recovery completed successfully:
"Node is recovered and ready"
Once the recovery is complete, get the nodes, and remove the controllers that are down.
teectl get nodes
ID NAME STATUS ROLE 7022qjifgp2srv3tq6rxyjhdm default-proxy-d55569575-n8mmb Ready Proxy muaxom14euley813tv40t9xxs default-controller-1 Down Controller q01yagt5suwv2tooy8amm248k default-controller-0 Ready Controller (Leader) s4eg90svby3aty5r6o9xkcpeh default-proxy-d55569575-kzn82 Ready Proxy w9aggc8l9lmwni7l5y2l1z4ww default-controller-2 Down Controller
teectl delete node --id="muaxom14euley813tv40t9xxs" teectl delete node --id="w9aggc8l9lmwni7l5y2l1z4ww"
The deployment is now acting as a single controller deployment. Additional controllers must now be started to bring the cluster back into highly availability again.
Congratulations! Your Traefik Enterprise cluster is recovered.
Recovering from State Loss¶
If all controllers and their state were lost, it is not possible to use the recovery procedure above to recover the cluster. At this point the cluster must be restored from a backup.
For more information on restoring from a backup, refer to Backup and Restore.