Open Policy Agent (OPA) User Guide

Open Policy Agent (OPA) is an open source, general-purpose policy engine that can be used to enforce unified, context-aware access policies across the application stack. Policies are definied using a high-level, declarative rules language called Rego.

Traeifk Enterprise 2.4 and later includes an OPA Middleware that supports Rego policies. This guide demonstrates how to use this Middleware to enforce policies for services running on a Kubernetes cluster.

Prerequisites

To complete this tutorial, you will need a few things:

• A running Kubernetes cluster
• The kubectl binary installed and configured to connect to the cluster
• Traefik Enterprise installed on the cluster, with a static configuration applied
• Optional: The installation manifest (as output by teectl during installation) saved as manifest.yaml

Deploy a Demo Service

For purposes of this demonstration, you will run an instance of whoami, a simple service that does nothing more than return some information about the requests it receives.

The file whoami-kubernetes.yaml, below, creates a Kubernetes Deployment with a single instance of whoami, an accompanying Kubernetes Service, and an IngressRoute that instructs Traefik Enterprise to expose it on the loopback interface at whoami.localhost:

whoami-kubernetes.yaml

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: whoami
  namespace: traefikee
spec:
  replicas: 1
  selector:
    matchLabels:
      app: whoami
  template:
    metadata:
      labels:
        app: whoami
    spec:
      containers:
        - name: whoami
          image: traefik/whoami
---
apiVersion: v1
kind: Service
metadata:
  name: whoami
  namespace: traefikee
  labels:
    app: whoami
spec:
  type: ClusterIP
  ports:
    - port: 80
      name: whoami
  selector:
    app: whoami
---
apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
  name: whoami
  namespace: traefikee
spec:
  entryPoints:
    - web
  routes:
    - match: Host(`whoami.localhost`)
      kind: Rule
      services:
        - name: whoami
          namespace: traefikee
          port: 80

You should deploy this and the other configuration files in this guide using kubectl, following the pattern:

kubectl apply -f whoami-kubernetes.yaml

Once the service is deployed, you can verify that it's running and producing typical output:

Command

curl -H Host:whoami.localhost http://localhost

Expected Output

Hostname: whoami-deployment-676886cb66-52h4n
IP: 127.0.0.1
IP: 10.1.0.123
RemoteAddr: 10.1.0.122:54010
GET / HTTP/1.1
Host: whoami.localhost
User-Agent: curl/7.64.1
Accept: */*
Accept-Encoding: gzip
X-Forwarded-For: 192.168.65.3
X-Forwarded-Host: whoami.localhost
X-Forwarded-Port: 80
X-Forwarded-Proto: http
X-Forwarded-Server: traefik-77fdb5c487-42cd6
X-Real-Ip: 192.168.65.3

Secure the Service with OPA

Enabling OPA-based access policies on your new service is a two-step process.

First, define an instance of the Traefik Enterprise OPA Middleware:

opa-middleware.yaml

---
apiVersion: traefik.containo.us/v1alpha1
kind: Middleware
metadata:
  name: my-opa-plugin
  namespace: traefikee
spec:
  plugin:
    opa:
      allow: data.example.authz.allow
      forwardHeaders:
        Group: data.example.authz.group
      policy: |
        package example.authz
        default allow = false
        default group = ""

        auth := split(input.headers.Authorization, " ")
        #[header, payload, signature]
        jwtDecode := io.jwt.decode(auth[1])

        payload := jwtDecode[1]

        allow {
          payload["email"] == "[email protected]"
        }

        # if allow is true, group will hold payload["grp"] value
        group = g {
          allow
          g = payload["grp"]
        }

Notice that this configuration includes an OPA policy definition (in Rego). It defines two policies, each of which depends on the contents of a JSON Web Token (JWT) that must be passed with each request:

• Access is only granted if the JWT includes a specific email address in the email claim ([email protected]).
• If access is granted, the contents of the grp claim from the JWT will be forwarded as an additional header called Group.

Next, to apply these policies to your whoami service, update its IngressRoute to enable the OPA Middleware:

opa-ingressroute.yaml

---
apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
  name: whoami
  namespace: traefikee
spec:
  entryPoints:
    - web
  routes:
    - match: Host(`whoami.localhost`)
      kind: Rule
      services:
        - name: whoami
          namespace: traefikee
          port: 80
      middlewares:
        - name: my-opa-plugin

If you compare this file to whoami-kubernetes.yaml, above, you will see that it adds just two lines of configuration to the original IngressRoute.

Test the Service

Apply the above configuration and call the whoami service, as before. You will see that without a JWT, access is now forbidden:

Command

curl --verbose -H Host:whoami.localhost http://localhost

Expected Output

< HTTP/1.1 403 Forbidden
< Date: Thu, 12 Nov 2020 13:33:51 GMT
< Content-Length: 0
<
* Connection #0 to host whoami.localhost left intact

Now call the service with a properly formed JWT token and observe the results:

Command

curl --verbose -H "Authorization: Bearer eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJzdWIiOiIxMjM0NTY3ODkwIiwibmFtZSI6IkpvaG4gRG9lIiwiaWF0IjoxNTE2MjM5MDIyLCJlbWFpbCI6ImZvb0BleGFtcGxlLmNvbSIsImp0aSI6IjM1MWY4YTEzLTIyNGMtNGQwNy05ODZhLWYxZTQ2MjEzNTBjYiIsImV4cCI6MTYwODE5ODQ3OX0.X_N-hCn64Yz2JWMJTT5Qi7LB-9ylXL8-bi8lSOzWrnY" -H Host:whoami.localhost http://localhost

Expected Output

< HTTP/1.1 200 OK
< Content-Length: 686
< Content-Type: text/plain; charset=utf-8
< Date: Thu, 17 Dec 2020 12:54:46 GMT
< 
Hostname: whoami-75d5976d8d-jl2p4
IP: 127.0.0.1
IP: 10.1.0.9
RemoteAddr: 10.1.0.7:47714
GET / HTTP/1.1
Host: whoami.localhost
User-Agent: curl/7.64.1
Accept: */*
Accept-Encoding: gzip
Authorization: Bearer eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJzdWIiOiIxMjM0NTY3ODkwIiwibmFtZSI6IkpvaG4gRG9lIiwiaWF0IjoxNTE2MjM5MDIyLCJlbWFpbCI6ImZvb0BleGFtcGxlLmNvbSIsImp0aSI6IjM1MWY4YTEzLTIyNGMtNGQwNy05ODZhLWYxZTQ2MjEzNTBjYiIsImV4cCI6MTYwODE5ODQ3OX0.X_N-hCn64Yz2JWMJTT5Qi7LB-9ylXL8-bi8lSOzWrnY
Group: 
X-Forwarded-For: 192.168.65.3
X-Forwarded-Host: whoami.localhost
X-Forwarded-Port: 80
X-Forwarded-Proto: http
X-Forwarded-Server: default-proxy-f59689c97-kscbd
X-Real-Ip: 192.168.65.3

Because the JWT included the address [email protected] in its payload, access is now granted.

Something is still missing, however. The Group header is still blank. Try sending a new request with a JWT that includes the grp payload, and you will see that the header is populated and forwarded, as specified in the OPA policy:

Command

curl --verbose -H "Authorization: Bearer eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJzdWIiOiIxMjM0NTY3ODkwIiwibmFtZSI6IkpvaG4gRG9lIiwiaWF0IjoxNTE2MjM5MDIyLCJlbWFpbCI6ImZvb0BleGFtcGxlLmNvbSIsImdycCI6Imdyb3VwMSIsImp0aSI6IjM1MWY4YTEzLTIyNGMtNGQwNy05ODZhLWYxZTQ2MjEzNTBjYiIsImV4cCI6MTYwODIxNDI4MH0.k6k9EyCeZQS34j6Ke3Ey7Gc-ZGWuHUusB3Xd_Ua7j7Y" -H Host:whoami.localhost http://localhost

Expected Output

< HTTP/1.1 200 OK
< Content-Length: 712
< Content-Type: text/plain; charset=utf-8
< Date: Thu, 17 Dec 2020 13:12:20 GMT
< 
Hostname: whoami-75d5976d8d-jl2p4
IP: 127.0.0.1
IP: 10.1.0.9
RemoteAddr: 10.1.0.7:56266
GET / HTTP/1.1
Host: whoami.localhost
User-Agent: curl/7.64.1
Accept: */*
Accept-Encoding: gzip
Authorization: Bearer eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJzdWIiOiIxMjM0NTY3ODkwIiwibmFtZSI6IkpvaG4gRG9lIiwiaWF0IjoxNTE2MjM5MDIyLCJlbWFpbCI6ImZvb0BleGFtcGxlLmNvbSIsImdycCI6Imdyb3VwMSIsImp0aSI6IjM1MWY4YTEzLTIyNGMtNGQwNy05ODZhLWYxZTQ2MjEzNTBjYiIsImV4cCI6MTYwODIxNDI4MH0.k6k9EyCeZQS34j6Ke3Ey7Gc-ZGWuHUusB3Xd_Ua7j7Y
Group: group1
X-Forwarded-For: 192.168.65.3
X-Forwarded-Host: whoami.localhost
X-Forwarded-Port: 80
X-Forwarded-Proto: http
X-Forwarded-Server: default-proxy-f59689c97-kscbd
X-Real-Ip: 192.168.65.3

More About JWTs

The tools at JSONWebToken.io are among many online resources that can help you learn more about JWTs and explore the tokens provided with the examples in this guide.

Deploying Policies as Bundles

Traefik Enterprise also supports deploying OPA policies as bundles.

This section assumes you have Traefik Enterprise running on your cluster with a static configuration applied and the whoami service deployed.

You will also need the example bundle file (opa.tar.gz).

You may also wish to have handy your Traefik Enterprise installation manifest (manifest.yaml), as output by teectl during installation.

Configure the Bundle

First, create a ConfigMap that holds the bundle:

kubectl create configmap -n traefikee --from-file=./opa.tar.gz opa-bundle

You must complete the above step before proceeding further.

Next, patch the Traefik Enterprise proxy deployment (default-proxy) on your cluster to add the ConfigMap as a volume and mount it. The file below provides the necessary patches:

opa-bundle-patch.yaml

spec:
  template:
    spec:
      volumes:
      - name: "opa-bundle"
        configMap:
          name: "opa-bundle"
      containers:
      - name: "default-proxy"
        volumeMounts:
          - name: "opa-bundle"
            mountPath: "/var/lib/traefikee/opa"

Apply the patches from the command line:

Bash

kubectl patch deployment default-proxy --patch "$(cat opa-bundle-patch.yaml)" -n traefikee

PowerShell

kubectl patch deployment default-proxy --patch $(Get-Content opa-bundle-patch.yaml -Raw) -n traefikee

Updating Your Manifest

In addition to patching your running default-proxy deployment, you may also wish to add the patches from opa-bundle-patch.yaml to your manifest.yaml, in case you wish to reinstall Traefik Enterprise in the future. For guidance, consult proxy-bundle-example.yaml, below, which shows a partial manifest. The relevant sections are near the end of the file.

proxy-bundle-example.yaml

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: "default-proxy"
  namespace: traefikee
  labels:
    app: traefikee
    release: "default"
spec:
  replicas: 1
  selector:
    matchLabels:
      app: traefikee
      release: "default"
      component: proxies
  template:
    metadata:
      labels:
        app: traefikee
        release: "default"
        component: proxies
    spec:
      affinity:
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
            - weight: 100
              podAffinityTerm:
                labelSelector:
                  matchExpressions:
                    - key: component
                      operator: In
                      values:
                        - proxies
                topologyKey: "kubernetes.io/hostname"
      terminationGracePeriodSeconds: 30
      automountServiceAccountToken: false
      initContainers:
        - name: wait-dns
          image: busybox:1.31.1
          command: ['sh', '-c', 'until nslookup -type=a default-ctrl-svc.traefikee.svc.cluster.local; do echo waiting for published dns records; sleep 1; done;']
          resources:
            requests:
              memory: "10Mi"
              cpu: "100m"
            limits:
              memory: "100Mi"
              cpu: "1000m"
      containers:
        - name: "default-proxy"
          image: traefikee:latest
          imagePullPolicy: IfNotPresent
          env:
            - name: POD_NAME
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name
            - name: POD_NAMESPACE
              valueFrom:
                fieldRef:
                  fieldPath: metadata.namespace
          securityContext:
            readOnlyRootFilesystem: true
            capabilities:
              drop:
                - ALL
              add:
                - NET_BIND_SERVICE
          ports:
            - containerPort: 8484
              name: distributed
            - containerPort: 80
              name: http
            - containerPort: 443
              name: https
          # readinessProbe:
          #   tcpSocket:
          #     port: http
          #   initialDelaySeconds: 2
          #   periodSeconds: 5
          resources:
            requests:
              memory: "100Mi"
              cpu: "100m"
            limits:
              memory: "4Gi"
              cpu: "1000m"
          volumeMounts:
            - name: "default-proxy-data"
              mountPath: "/var/lib/traefikee"
            - name: "join-token"
              mountPath: "/var/run/secrets"
            # Add the "opa-bundle" VolumeMount here
            - name: "opa-bundle"
              mountPath: "/var/lib/traefikee/opa"
          command:
            - "/traefikee"
            - "proxy"
            - "--role=ingress"
            - "--name=$(POD_NAME)"
            - "--discovery.dns.domain=default-ctrl-svc.$(POD_NAMESPACE)"
            - "--jointoken.file.path=/var/run/secrets"
            - "--log.level="
            - "--log.filepath="
            - "--log.format="
      volumes:
        - name: "default-proxy-data"
          emptyDir: {}
        - name: "join-token"
          secret:
            secretName: "default-tokens"
        # Add the "opa-bundle" ConfigMap as a volume here
        - name: "opa-bundle"
          configMap:
            name: "opa-bundle"

Once the default-proxy deployment is patched, you can define the IngressRoute and Middleware that will enable OPA policy enforcement for your whoami service:

opa-bundle.yaml

---
apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
  name: whoami-opa-bundle
  namespace: traefikee
spec:
  entryPoints:
    - web
  routes:
    - match: Host(`whoami-bundle.localhost`)
      kind: Rule
      services:
        - name: whoami
          namespace: traefikee
          port: 80
      middlewares:
        - name: opa-bundle
---
apiVersion: traefik.containo.us/v1alpha1
kind: Middleware
metadata:
  name: opa-bundle
  namespace: traefikee
spec:
  plugin:
    opa:
      allow: data.example.authz.allow
      bundlePath: /var/lib/traefikee/opa/opa.tar.gz

Test the Bundle

As with the previous example, when you call the whoami service without supplying a JWT, access is denied:

Command

curl --verbose -H Host:whoami-bundle.localhost localhost

Expected Output

< HTTP/1.1 403 Forbidden
< Date: Thu, 17 Dec 2020 18:49:53 GMT
< Content-Length: 0
< 
* Connection #0 to host localhost left intact
* Closing connection 0

Supplying a properly formed JWT with the request, on the other hand, produces the expected whoami output:

Command

curl --verbose -H "Authorization: Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiIxMjM0NTY3ODkwIiwibmFtZSI6IkpvaG4gRG9lIiwiaWF0IjoxNTE2MjM5MDIyLCJlbWFpbCI6ImZvb0BleGFtcGxlLmNvbSIsImdycCI6Imdyb3VwMSJ9.fWnRCDbqpBMyS6uRAyQYLPBpiz9YRQgmAGt2mo-NvK8" -H Host:whoami-bundle.localhost http://localhost

Expected Output

< HTTP/1.1 200 OK
< Content-Length: 713
< Content-Type: text/plain; charset=utf-8
< Date: Thu, 17 Dec 2020 18:39:09 GMT
< 
Hostname: whoami-75d5976d8d-sfgsj
IP: 127.0.0.1
IP: 10.1.0.9
RemoteAddr: 10.1.0.10:39064
GET / HTTP/1.1
Host: whoami-bundle.localhost
User-Agent: curl/7.64.1
Accept: */*
Accept-Encoding: gzip
Authorization: Bearer eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJzdWIiOiIxMjM0NTY3ODkwIiwibmFtZSI6IkpvaG4gRG9lIiwiaWF0IjoxNTE2MjM5MDIyLCJlbWFpbCI6ImZvb0BleGFtcGxlLmNvbSIsImdycCI6Imdyb3VwMSIsImp0aSI6IjM1MWY4YTEzLTIyNGMtNGQwNy05ODZhLWYxZTQ2MjEzNTBjYiIsImV4cCI6MTYwODIxNDI4MH0.k6k9EyCeZQS34j6Ke3Ey7Gc-ZGWuHUusB3Xd_Ua7j7Y
X-Forwarded-For: 192.168.65.3
X-Forwarded-Host: whoami-bundle.localhost
X-Forwarded-Port: 80
X-Forwarded-Proto: http
X-Forwarded-Server: default-proxy-6ddc545cbb-w6g2x
X-Real-Ip: 192.168.65.3

For more information on crafting OPA policies, consult the project documentation.