How to Setup Distributed ACME in Traefik Enterprise¶

Traefik Enterprise uses a dedicated agent to issue and share ACME certificates between multiple clusters. This agent operates along with a Vault KV provider, which retrieves generated certificates from a Vault server. The distributed ACME agent only supports the DNS challenge, because the HTTP and TLS challenges are inherently multicluster-friendly.

This guide assumes that you are using Kubernetes.

Configure the Agent¶

The agent stores generated certificates in a Vault server. It must have a KV Secrets Engine version 2 enabled.

Dedicated Secrets Engine

For security reasons, the best practice is to use a dedicated Secrets Engine to store certificates.

Enabling a Secrets Engine

Here is an example to enable a new Secrets Engine under the "customPath" path: vault secrets enable -path=customPath -version=2 kv

In order to enable mTLS between the Traefik Enterprise controllers and the Distributed ACME Agent, you must provide certificates in the configuration of the agent. It is recommended to store those certificates in Kubernetes TLS Secrets and mount them on your ACME Agent and Traefik Enterprise Controller deployments or StatefulSets.

Additionally, you have to configure the certificate resolvers to use for solving DNS challenges. You'll need to configure at least one DNS provider for the agent to work.

Required Environment Variables

All the DNS providers have their own sets of required environment variables you'll need to set in order to authenticate. The Traefik documentation contains a list of providers and their associated variables.

To avoid recreating new ACME accounts every time the agent restarts, it persists account credentials on its filesystem. Mounting a Persistent Volume to /var/lib/acme-agent/state/state.json allows the agent to keep its state across restarts.

Below is a complete example of how to deploy the ACME agent with mTLS. Commented lines potentially need to be edited.

deployment.yml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: acme-agent-deployment
  namespace: traefikee
  labels:
    app: acme-agent
spec:
  selector:
    matchLabels:
      app: acme-agent
  template:
    metadata:
      labels:
        app: acme-agent
    spec:
      containers:
        - name: acme-agent
          image: traefik/acme-agent:v1.0.0
          # Edit these environment variables with what your DNS provider requires.
          env:
            - name: CLOUDFLARE_DNS_API_TOKEN
              value: $CLOUDFLARE_API_TOKEN
          volumeMounts:
            # Here we mount the configuration of the agent.
            - name: config
              mountPath: /etc/acme-agent/
            # This allows preserving the state if the agent restarts.
            - name: state
              mountPath: /var/lib/acme-agent/state/
            - name: mtls-ca
              mountPath: /var/lib/acme-agent/mtls/ca/
            - name: mtls-server
              mountPath: /var/lib/acme-agent/mtls/server/
      volumes:
        - name: config
          configMap:
            name: acme-agent-config
            items:
              - key: config.yml
                path: config.yml
        - name: state
          persistentVolumeClaim:
            claimName: acme-agent-state
        - name: mtls-ca
          secret:
            secretName: acme-agent-mtls-ca
            items:
              - key: tls.crt
                path: tls.crt
              - key: tls.key
                path: tls.key
        - name: mtls-server
          secret:
            secretName: acme-agent-mtls-server
            items:
              - key: tls.crt
                path: tls.crt
              - key: tls.key
                path: tls.key

service.yml

apiVersion: v1
kind: Service
metadata:
  name: acme-agent
  namespace: traefikee
spec:
  selector:
    app: acme-agent
  ports:
    # This needs to match the port on which the agent is listening.
    - port: 5505

configmap.yml

apiVersion: v1
kind: ConfigMap
metadata:
  name: acme-agent-config
  namespace: traefikee
data:
  # Edit the agent configuration here. Note that the agent must be restarted to take changes into account.
  config.yml: |-
    listenAddr: 0.0.0.0:5505
    tls:
      ca: /var/lib/acme-agent/mtls/ca/tls.crt
      cert: /var/lib/acme-agent/mtls/server/tls.crt
      key: /var/lib/acme-agent/mtls/server/tls.key
    vault:
      url: http://vault.traefikee.svc.cluster.local:8200
      token: root
      enginePath: customPath
    certificateResolvers:
      myResolver:
        email: [email protected]
        caServer: https://acme-staging-v02.api.letsencrypt.org/directory
        provider: cloudflare

pvc.yml

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: acme-agent-state
  namespace: traefikee
spec:
  accessModes:
    - ReadWriteOnce
  # Replace the storageClassName with the storage you use in your Kubernetes cluster.
  storageClassName: local-path
  resources:
    requests:
      # The state only consists of ACME account credentials which are a few kilobytes per account used.
      storage: 10Mi

secrets.yml

apiVersion: v1
kind: Secret
metadata:
  name: acme-agent-mtls-ca
  namespace: traefikee
type: kubernetes.io/tls
data:
  tls.crt: # Your base64-encoded certificate goes here
  tls.key: # Your base-64 encoded key goes here
---
apiVersion: v1
kind: Secret
metadata:
  name: acme-agent-mtls-server
  namespace: traefikee
type: kubernetes.io/tls
data:
  tls.crt: # Your base64-encoded certificate goes here
  tls.key: # Your base64-encoded key goes here
---
apiVersion: v1
kind: Secret
metadata:
  name: acme-agent-mtls-client
  namespace: traefikee
type: kubernetes.io/tls
data:
  tls.crt: # Your base64-encoded certificate goes here
  tls.key: # Your base64-encoded key goes here

Configure Traefik Enterprise¶

In the Traefik Enterprise static configuration, you'll need to enable the Distributed ACME certificate resolver as well as the Vault KV provider. The certificate resolver will issue requests to the Distributed ACME agent, and the Vault KV provider is in charge of fetching certificates from the Vault server that runs with the ACME agent.

Just like the ACME agent, Traefik Enterprise controllers can be configured to use mTLS, in which case the certificate resolver configuration needs paths to the mTLS certificates. It is also recommended to store them in Kubernetes TLS Secrets and mount them on your Traefik Enterprise Controller StatefulSets.

Below is an example of a Traefik Enterprise static configuration that uses the distributed ACME agent:

entryPoints:
  https:
    address: 0.0.0.0:443

providers:
  kubernetesCRD: { }
  plugin:
    vault:
      url: http://vault.traefikee.svc.cluster.local:8200
      token: root
      enginePath: customPath

certificatesResolvers:
  # Define your certificate resolver here.
  myResolver:
    distributedACME:
      url: https://acme-agent.traefikee.svc.cluster.local:5505
      tls:
        ca: /var/lib/acme-agent/mtls/ca/tls.crt
        cert: /var/lib/acme-agent/mtls/client/tls.crt
        key: /var/lib/acme-agent/mtls/client/tls.key

If you are using mTLS, which is recommended for a production environment, you also need to update your Traefik Enterprise StatefulSet and add the following volumes and volume mounts so that it has access to mTLS certificates:

traefikee-mtls-patch.yml

spec:
  template:
    spec:
      containers:
        - name: "default-controller"
          volumeMounts:
          - name: mtls-ca
            mountPath: /var/lib/acme-agent/mtls/ca/
          - name: mtls-client
            mountPath: /var/lib/acme-agent/mtls/client/
      volumes:
      - name: mtls-ca
        secret:
          secretName: acme-agent-mtls-ca
      - name: mtls-client
        secret:
          secretName: acme-agent-mtls-client

You can apply the above patch on your existing Traefik Enterprise StatefulSet with the following command:

Bash

kubectl patch statefulset -n traefikee default-controller -p "$(cat traefikee-mtls-patch.yml)"

Fish

kubectl patch statefulset -n traefikee default-controller -p (cat traefikee-mtls-patch.yml | string collect)

PowerShell

kubectl patch statefulset -n traefikee default-controller -p $(Get-Content traefikee-mtls-patch.yml -Raw)

Configure your IngressRoutes¶

To use the distributed ACME agent you must reference the certificate resolver configured in your static configuration:

apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
  name: myservice
  namespace: traefikee
spec:
  entryPoints:
    - https
  routes:
    - match: Host(`myservice.corp.com`)
      kind: Rule
      services:
        - name: myservice
          namespace: traefikee
          port: 80
  tls:
    # Reference your certificate resolver here.
    certResolver: myResolver

Configuration References¶

Agent Configuration¶

`listenAddr`¶

Required, Default="0.0.0.0:5505"

Defines the interface and port on which the distributed ACME agent listens for requests.

File (YAML)

listenAddr: 0.0.0.0:5505

`checkRenewalInterval`¶

Optional, Default="24h"

Defines the interval at which the agent checks if ACME certificates need to be renewed.

File (YAML)

checkRenewalInterval: 24h

`tls.ca`, `tls.cert` and `tls.key`¶

Optional, Default=""

Paths to certificates required for enabling mTLS to secure communication between Traefik Enterprise and the ACME agent. If these are not set, mTLS is disabled and the connection between the ACME agent and Traefik Enterprise controllers is not secured.

File (YAML)

tls:
  ca: /var/lib/acme-agent/mtls/ca/ca.pem
  cert: /var/lib/acme-agent/mtls/server/server.pem
  key: /var/lib/acme-agent/mtls/server/server-key.pem

`vault.url`¶

Required, Default=""

Defines the URL of the Vault server, including the scheme and port.

File (YAML)

vault:
  url: https://vault.devault.svc.cluster.local:9200

`vault.tls`¶

Optional, Default=""

Paths to certificates required for enabling mTLS to secure communication between ACME agent and Vault. If these are not set, mTLS is disabled and the connection between the ACME agent and Vault controllers is not secured.

File (YAML)

vault:
    tls:
      ca: /var/lib/acme-agent/mtls/ca/ca.pem
      cert: /var/lib/acme-agent/mtls/client/client.pem
      key: /var/lib/acme-agent/mtls/client/client-key.pem
      insecureSkipVerify: false

`vault.token`¶

Optional(one of vault.token or vault.appRole must be set), Default=""

Defines the token to authenticate with Vault.

Supported Authentication Mechanisms

The Vault certificate resolver currently supports token authentication only.

File (YAML)

vault:
  token: myVaultToken

`vault.appRole`¶

Optional(one of vault.token or vault.appRole must be set), Default=None

Enables the AppRole authentication method. See vault.appRole.* options to see what needs to be provided.

`vault.appRole.roleID`¶

Required, Default=""

Defines the ID of the role to use when authenticating to Vault with AppRole.

File (YAML)

vault:
  appRole:
    roleID: "4cf1dc0d-f431-f76d-42fd-ed1264f0b893"

`vault.appRole.secretID`¶

Required, Default=""

Defines the ID of the secret to use when authenticating to Vault with AppRole.

File (YAML)

vault:
  appRole:
    secretID: "63cffb45-b404-18c4-cdbb-af723aaef1e7"

`vault.appRole.path`¶

Optional, Default="approle"

Defines the path under which the AppRole authentication method is enabled in Vault.

File (YAML)

vault:
  appRole:
    path: "customAppRolePath"

`vault.enginePath`¶

Optional, Default="tls"

Defines the path under which the PKI secret engine is enabled.

Vault Namespaces

If using Vault Namespaces, simply prefix the engine path with the name of the namespace. If using the engine path secret under the default namespace, set default/secret as the engine path.

File (YAML)

vault:
  enginePath: customPath

`vault.syncInterval`"¶

Optional, Default="1m"

Defines the interval at which the agent checks if an ACME certificates was manually deleted in Vault.

File (YAML)

vault:
  syncInterval: 1m

`certificateResolvers.resolverName.email`¶

Required, Default=""

The email to use for the ACME account of this resolver.

File (YAML)

certificateResolvers:
  resolverName:
    email: [email protected]

`certificateResolvers.resolverName.caServer`¶

Required, Default=""

The URL to the ACME directory from which to obtain certificates.

File (YAML)

certificateResolvers:
  resolverName:
    caServer: https://acme-staging-v02.api.letsencrypt.org/directory

`certificateResolvers.resolverName.provider`¶

Required, Default=""

The DNS provider used to automate DNS verifications for the ACME challenges.

Required Environment Variables

All the DNS providers have their own sets of required environment variables you'll need to set in order to authenticate. The Traefik documentation contains a list of providers and their associated variables.

File (YAML)

certificateResolvers:
  resolverName:
    provider: cloudflare

`certificateResolvers.resolverName.keyType`¶

Required, Default="RSA4096"

The key type to use for the private keys associated with the generated certificates. Allowed values are the following: EC256, EC384, RSA2048, RSA4096, RSA8192.

File (YAML)

certificateResolvers:
  resolverName:
    keyType: RSA4096

Certificate Resolver Configuration¶

`url`¶

Required, Default=""

The URL of the Distributed ACME Agent.

File (YAML)

url: https://dacme.example.org

File (TOML)

url = "https://dacme.example.org"

`tls.ca`, `tls.cert` and `tls.key`¶

Optional, Default=""

Paths to certificates required for enabling mTLS to secure communication between Traefik Enterprise and the ACME agent. If these are not set, mTLS is disabled and the connection between the ACME agent and Traefik Enterprise controllers is not secured.

File (YAML)

tls:
  ca: /var/lib/acme-agent/mtls/ca/ca.pem
  cert: /var/lib/acme-agent/mtls/server/server.pem
  key: /var/lib/acme-agent/mtls/server/server-key.pem

File (TOML)

[tls]
  ca = "/var/lib/acme-agent/mtls/ca/ca.pem"
  cert = "/var/lib/acme-agent/mtls/server/server.pem"
  key = "/var/lib/acme-agent/mtls/server/server-key.pem"

Configuration Examples¶

Agent Configuration¶

Below is a complete distributed ACME agent configuration example:

listenAddr: 0.0.0.0:5505
checkRenewalInterval: 24h
tls:
  ca: /path/to/mtls/ca.pem
  cert: /path/to/mtls/server/cert.pem
  key: /path/to/mtls/server/key.pem
vault:
  url: https://vault.corp.com:8200
  token: myVaultToken
  enginePath: customPath
  syncInterval: 1m
certificateResolvers:
  myResolver:
    email: [email protected]
    caServer: https://acme-staging-v02.api.letsencrypt.org/directory
    provider: cloudflare

Certificate Resolver Configuration¶

Below is a complete distributed ACME certificate resolver configuration example:

File (YAML)

url: https://acme-agent.traefikee.svc.cluster.local:5505
tls:
  ca: /var/lib/acme-agent/mtls/ca/tls.crt
  cert: /var/lib/acme-agent/mtls/client/tls.crt
  key: /var/lib/acme-agent/mtls/client/tls.key

File (TOML)

url = "https://acme-agent.traefikee.svc.cluster.local:5505"
[tls]
  ca = "/var/lib/acme-agent/mtls/ca/tls.crt"
  cert = "/var/lib/acme-agent/mtls/client/tls.crt"
  key = "/var/lib/acme-agent/mtls/client/tls.key"

How to Setup Distributed ACME in Traefik Enterprise¶

Configure the Agent¶

Configure Traefik Enterprise¶

Configure your IngressRoutes¶

Configuration References¶

Agent Configuration¶

listenAddr¶

checkRenewalInterval¶

tls.ca, tls.cert and tls.key¶

vault.url¶

vault.tls¶

vault.token¶

vault.appRole¶

vault.appRole.roleID¶

vault.appRole.secretID¶

vault.appRole.path¶

vault.enginePath¶

vault.syncInterval"¶

certificateResolvers.resolverName.email¶

certificateResolvers.resolverName.caServer¶

certificateResolvers.resolverName.provider¶

certificateResolvers.resolverName.keyType¶

Certificate Resolver Configuration¶

url¶

tls.ca, tls.cert and tls.key¶

Configuration Examples¶

Agent Configuration¶

Certificate Resolver Configuration¶

`listenAddr`¶

`checkRenewalInterval`¶

`tls.ca`, `tls.cert` and `tls.key`¶

`vault.url`¶

`vault.tls`¶

`vault.token`¶

`vault.appRole`¶

`vault.appRole.roleID`¶

`vault.appRole.secretID`¶

`vault.appRole.path`¶

`vault.enginePath`¶

`vault.syncInterval`"¶

`certificateResolvers.resolverName.email`¶

`certificateResolvers.resolverName.caServer`¶

`certificateResolvers.resolverName.provider`¶

`certificateResolvers.resolverName.keyType`¶

`url`¶

`tls.ca`, `tls.cert` and `tls.key`¶