Traefik & Kubernetes¶
The Kubernetes Ingress Controller, The Custom Resource Way.
Traefik used to support Kubernetes only through the Kubernetes Ingress provider, which is a Kubernetes Ingress controller in the strict sense of the term.
However, as the community expressed the need to benefit from Traefik features without resorting to (lots of) annotations, we ended up writing a Custom Resource Definition (alias CRD in the following) for an IngressRoute type, defined below, in order to provide a better way to configure access to a Kubernetes cluster.
Configuration Requirements¶
All Steps for a Successful Deployment
- Add/update all the Traefik resources definitions
- Add/update the RBAC for the Traefik custom resources
- Use Helm Chart or use a custom Traefik Deployment
- Enable the kubernetesCRD provider
- Apply the needed kubernetesCRD provider configuration
- Add all needed traefik custom resources
Initializing Resource Definition and RBAC
# All resources definition must be declared
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
name: ingressroutes.traefik.containo.us
spec:
group: traefik.containo.us
version: v1alpha1
names:
kind: IngressRoute
plural: ingressroutes
singular: ingressroute
scope: Namespaced
---
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
name: middlewares.traefik.containo.us
spec:
group: traefik.containo.us
version: v1alpha1
names:
kind: Middleware
plural: middlewares
singular: middleware
scope: Namespaced
---
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
name: ingressroutetcps.traefik.containo.us
spec:
group: traefik.containo.us
version: v1alpha1
names:
kind: IngressRouteTCP
plural: ingressroutetcps
singular: ingressroutetcp
scope: Namespaced
---
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
name: tlsoptions.traefik.containo.us
spec:
group: traefik.containo.us
version: v1alpha1
names:
kind: TLSOption
plural: tlsoptions
singular: tlsoption
scope: Namespaced
---
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
name: traefikservices.traefik.containo.us
spec:
group: traefik.containo.us
version: v1alpha1
names:
kind: TraefikService
plural: traefikservices
singular: traefikservice
scope: Namespaced
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
name: traefik-ingress-controller
rules:
- apiGroups:
- ""
resources:
- services
- endpoints
- secrets
verbs:
- get
- list
- watch
- apiGroups:
- extensions
resources:
- ingresses
verbs:
- get
- list
- watch
- apiGroups:
- extensions
resources:
- ingresses/status
verbs:
- update
- apiGroups:
- traefik.containo.us
resources:
- middlewares
- ingressroutes
- traefikservices
- ingressroutetcps
- tlsoptions
verbs:
- get
- list
- watch
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
name: traefik-ingress-controller
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: traefik-ingress-controller
subjects:
- kind: ServiceAccount
name: traefik-ingress-controller
namespace: default
Resource Configuration¶
When using KubernetesCRD as a provider, Traefik uses Custom Resource Definition to retrieve its routing configuration. Traefik Custom Resource Definitions are a Kubernetes implementation of the Traefik concepts. The main particularities are:
- The usage of
name
andnamespace
to refer to another Kubernetes resource. - The usage of secret for sensible data like:
- TLS certificate.
- Authentication data.
- The structure of the configuration.
- The obligation to declare all the definitions.
The Traefik CRD are building blocks which you can assemble according to your needs. See the list of CRDs in the dedicated routing section.
LetsEncrypt Support with the Custom Resource Definition Provider¶
By design, Traefik is a stateless application, meaning that it only derives its configuration from the environment it runs in, without additional configuration. For this reason, users can run multiple instances of Traefik at the same time to achieve HA, as is a common pattern in the kubernetes ecosystem.
When using a single instance of Traefik with LetsEncrypt, no issues should be encountered, however this could be a single point of failure. Unfortunately, it is not possible to run multiple instances of Traefik 2.0 with LetsEncrypt enabled, because there is no way to ensure that the correct instance of Traefik will receive the challenge request, and subsequent responses. Previous versions of Traefik used a KV store to attempt to achieve this, but due to sub-optimal performance was dropped as a feature in 2.0.
If you require LetsEncrypt with HA in a kubernetes environment, we recommend using TraefikEE where distributed LetsEncrypt is a supported feature.
If you are wanting to continue to run Traefik Community Edition, LetsEncrypt HA can be achieved by using a Certificate Controller such as Cert-Manager. When using Cert-Manager to manage certificates, it will create secrets in your namespaces that can be referenced as TLS secrets in your ingress objects. When using the Traefik Kubernetes CRD Provider, unfortunately Cert-Manager cannot interface directly with the CRDs yet, but this is being worked on by our team. A workaround is to enable the Kubernetes Ingress provider to allow Cert-Manager to create ingress objects to complete the challenges. Please note that this still requires manual intervention to create the certificates through Cert-Manager, but once created, Cert-Manager will keep the certificate renewed.
Provider Configuration¶
endpoint
¶
Optional, Default=empty
[providers.kubernetesCRD]
endpoint = "http://localhost:8080"
# ...
providers:
kubernetesCRD:
endpoint = "http://localhost:8080"
# ...
--providers.kubernetescrd.endpoint=http://localhost:8080
The Kubernetes server endpoint as URL.
When deployed into Kubernetes, Traefik will read the environment variables KUBERNETES_SERVICE_HOST
and KUBERNETES_SERVICE_PORT
or KUBECONFIG
to construct the endpoint.
The access token will be looked up in /var/run/secrets/kubernetes.io/serviceaccount/token
and the SSL CA certificate in /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
.
Both are provided mounted automatically when deployed inside Kubernetes.
The endpoint may be specified to override the environment variable values inside a cluster.
When the environment variables are not found, Traefik will try to connect to the Kubernetes API server with an external-cluster client.
In this case, the endpoint is required.
Specifically, it may be set to the URL used by kubectl proxy
to connect to a Kubernetes cluster using the granted authentication and authorization of the associated kubeconfig.
token
¶
Optional, Default=empty
[providers.kubernetesCRD]
token = "mytoken"
# ...
providers:
kubernetesCRD:
token = "mytoken"
# ...
--providers.kubernetescrd.token=mytoken
Bearer token used for the Kubernetes client configuration.
certAuthFilePath
¶
Optional, Default=empty
[providers.kubernetesCRD]
certAuthFilePath = "/my/ca.crt"
# ...
providers:
kubernetesCRD:
certAuthFilePath: "/my/ca.crt"
# ...
--providers.kubernetescrd.certauthfilepath=/my/ca.crt
Path to the certificate authority file. Used for the Kubernetes client configuration.
namespaces
¶
Optional, Default: all namespaces (empty array)
[providers.kubernetesCRD]
namespaces = ["default", "production"]
# ...
providers:
kubernetesCRD:
namespaces:
- "default"
- "production"
# ...
--providers.kubernetescrd.namespaces=default,production
Array of namespaces to watch.
labelselector
¶
Optional,Default: empty (process all Ingresses)
[providers.kubernetesCRD]
labelselector = "A and not B"
# ...
providers:
kubernetesCRD:
labelselector: "A and not B"
# ...
--providers.kubernetescrd.labelselector="A and not B"
By default, Traefik processes all Ingress objects in the configured namespaces. A label selector can be defined to filter on specific Ingress objects only.
See label-selectors for details.
ingressClass
¶
Optional, Default: empty
[providers.kubernetesCRD]
ingressClass = "traefik-internal"
# ...
providers:
kubernetesCRD:
ingressClass: "traefik-internal"
# ...
--providers.kubernetescrd.ingressclass=traefik-internal
Value of kubernetes.io/ingress.class
annotation that identifies Ingress objects to be processed.
If the parameter is non-empty, only Ingresses containing an annotation with the same value are processed.
Otherwise, Ingresses missing the annotation, having an empty value, or the value traefik
are processed.
throttleDuration
¶
Optional, Default: 0 (no throttling)
[providers.kubernetesCRD]
throttleDuration = "10s"
# ...
providers:
kubernetesCRD:
throttleDuration: "10s"
# ...
--providers.kubernetescrd.throttleDuration=10s
Further¶
Also see the full example with Let's Encrypt.