Transparency Logs

Traefik Hub can keep a second, cryptographic record alongside your normal logs and access logs. If someone later asks whether a log line was edited or removed, this record proves that it wasn't. That matters most for compliance, where an auditor needs proof your logs are complete and unaltered.

This is a Hub-wide setting, not something you attach to a route or a middleware. Turn it on once, and it covers log lines and access log entries from every gateway, API, AI, and MCP, though not request or response bodies.

Token exchange delegates authority to the caller, and an authorization step such as AuthZEN decides whether to allow the action. Transparency Logs records that decision. See Audit authorization decisions for the records this produces.

License Requirement

Transparency Logs is a licensed feature. Contact the Traefik Labs sales team to add it to your license.

How transparency logs stay tamper-evident

• Every entry gets a cheap, verifiable proof of inclusion. Each log line and access log entry is hashed and added to an append-only Merkle tree, built with Tessera. Proving one entry belongs to the tree costs about 30 hashes whether the tree holds a thousand entries or a billion, so verification stays cheap as the log grows.
• Periodic checkpoints prove nothing was edited or removed. Traefik Hub signs a checkpoint, a record of the tree's current size and root hash. Because the tree only ever grows, Tessera can prove any two checkpoints are consistent, meaning the newer one adds entries on top of the older one and never rewrites what was already there.
• The log itself isn't enough to verify anything. It only stores IDs and hashes, not content, so verifying an entry later needs both the transparency log and your original log file.

Closing the trust gap with a witness

On its own, a checkpoint only proves something if you trust whoever stores it. A witness closes that gap. Witnesses are outside parties that periodically cosign a checkpoint after checking it's consistent with the last checkpoint they've already cosigned. Adding a witness is optional, but without one you're exposed to a split view. Someone with access to the log's storage could fork history at an old checkpoint, grow a different, conflicting tree from that point on, and produce a second checkpoint that's equally valid. Nothing in the log itself tells you which of the two is genuine.

A witness only cosigns the checkpoint it's actually shown. Rewriting history after that point would need a cosignature for a different, conflicting checkpoint that the witness never received. Any new checkpoint it does see must still be consistent with the one it already holds.

A witness must be reachable. If a configured witness is unreachable, Traefik Hub still starts and keeps serving traffic. It retries building the transparency log appender every 10 seconds in the background, and until that succeeds, incoming log lines aren't recorded to the transparency log at all. If the appender already initialized and a witness later becomes unreachable, Traefik Hub keeps appending entries to the tree, but stops publishing checkpoints until the witness is reachable again. Either way, it logs an error each time cosigning fails, naming the reason, for example Transparency log witness is unreachable; the checkpoint cannot be cosigned. Ensure your witnesses are as available as Traefik Hub itself.

Prerequisites

• You must set log.format and accessLog.format to json. Otherwise, Traefik Hub will not start, with the error transparency logs require the Traefik log format to be json, got "common" (and the equivalent message for the access log format). Both values.yaml examples below set these values.
• If you also send access logs to an OTLP collector (accessLog.otlp), you must set accessLog.dualOutput: true. Otherwise, Traefik Hub will not write the access log file, while still hashing and committing each entry to the transparency log, so a verifier will report those entries as not found.
• Traefik Hub writes logs to stdout by default on Kubernetes. Set log.filePath and accessLog.filePath so they're written somewhere a verifier can read them later. Write them to a persistent volume, because files on the pod's ephemeral storage are lost when the pod is replaced. Traefik Hub then no longer writes logs to stdout, so kubectl logs shows nothing if startup fails. Remove log.filePath to see the startup error.
• Set log.level to INFO or lower. Traefik Hub writes authorization decisions at the info level, so a higher level leaves them out of the transparency log. The default is ERROR. The Helm chart sets INFO.

Enable transparency logs

First, generate a signing key with the keygen command that ships with Traefik Hub:

traefik-hub keygen --name my-hub-signer --outputDir ./keys

This writes private.key (used to sign checkpoints) and public.key (shared with anyone who needs to verify them) to ./keys. --name becomes part of the key's identity: Tessera writes it as the checkpoint's origin, so anyone verifying the log later needs this exact value. Use a schema-less URL for it, such as example.com/my-log. It doesn't need to resolve to anything, but a plain name risks colliding with another log's origin, since nothing enforces uniqueness otherwise. Pick a name you won't need to change. Mount the private key into your Traefik Hub pod, then point transparencyLogs.signerPrivateKey at it:

values.yaml

log:
  format: json
accessLog:
  format: json
hub:
  transparencyLogs:
    driver:
      posix:
        path: /data/transparency-logs
    signerPrivateKey: /etc/traefik-hub/transparency-logs/private.key
    checkpointInterval: 10s

driver is where the tree itself is stored. This can be a local path (as above), a GCS bucket backed by Spanner, or an S3 bucket backed by MySQL. If you use a local path, mount a persistent volume there. Without one, the tree lives on the pod's own ephemeral storage and is lost on every restart or reschedule, along with your entire tamper-evident history. checkpointInterval controls how often a new checkpoint is signed; the example above signs one every 10 seconds.

Choosing a witness

Before adding a witness, keep in mind:

• A witness you run yourself only proves to you that nothing changed. You could still roll back your log and your own witness together before anyone checks. For the guarantee to mean anything to an outside auditor, the witness needs to run independently of you.

• Traefik operates a hosted witness group for Traefik Hub customers, so you don't have to run one yourself. It's three separate witness endpoints:

  • https://witness0.traefik.io
  • https://witness1.traefik.io
  • https://witness2.traefik.io

  Two of the three must cosign a checkpoint (threshold: 2). The group publishes its current endpoints and public keys in its witness group descriptor. Fetch a witness's key from there instead of copying it once, since a key can rotate.

  Contact your Traefik Labs account team to onboard your log with the hosted witness group.

  If you prefer to self-host instead, run your own with the open source transparency-dev/witness implementation, or ask another team to run one independently on your behalf.

• Witnesses aren't dynamic. Whoever runs the witness, Traefik's hosted group or your own, has to add your log to its configuration before it will cosign anything. Send them your log's origin (the --name you gave keygen) and its public key, then confirm they've added it before continuing.

• Don't generate a witness's keypair with traefik-hub keygen. It only produces plain Ed25519 keys, and some verification tooling only recognizes the cosignature format a witness key needs. If you're self-hosting a witness with transparency-dev/witness, generate its keypair with that project's own generate_keys command instead, which produces the correct format.

Add the witness's URL and public key under witnessGroup:

values.yaml

log:
  format: json
accessLog:
  format: json
hub:
  transparencyLogs:
    driver:
      posix:
        path: /data/transparency-logs
    signerPrivateKey: /etc/traefik-hub/transparency-logs/private.key
    checkpointInterval: 10s
    witnessGroup:
      threshold: 1
      witnesses:
        - url: https://witness.example.com
          key: /etc/traefik-hub/transparency-logs/witness-public.key

threshold is how many witnesses must cosign a checkpoint before Traefik Hub considers it committed; with one witness configured, set it to 1. Using Traefik's hosted witness group instead means listing all three witnesses here with threshold: 2. See the reference page's example for the three-entry shape.

Once you have the configuration adjusted to your needs, use Helm to deploy Traefik Hub with the new values:

Install Traefik Hub using Helm

helm repo add --force-update traefik https://traefik.github.io/charts
helm upgrade --install traefik -n traefik --wait \
   --values values.yaml \
   traefik/traefik

Testing before you go to production

Setting transparencyLogs.debug: true also stores the full rendered log line in the transparency log, not only its hash, which makes it easier to check things are working while you set this up. Turn it off again before going to production to avoid storing your log content a second time. See the Metrics section of the reference page to monitor transparency logs once it's running.

Audit authorization decisions

Traefik Hub writes two kinds of record for a request that passes through an authorization middleware. Together they show who made the request and which policy allowed or refused it.

Record	Written by	Contains	Doesn't contain
Decision record	The authzen and authzen-mcp middlewares, once per authorization decision	The result (allow, deny, or bypass) and the rule that applied	The user, the agent, or the resource
Access record	Traefik Hub, once per request	The request path, the status returned to the client, and the user and agent when you configure it	The policy

Traefik Hub commits both records to the transparency log. It writes decision records to the Traefik Hub log (log.filePath) and access records to the access log (accessLog.filePath). Keep both files. The transparency log stores only hashes, so checking an entry later needs the log line from the file that contains it.

Configure identity and trace fields

By default, the access record doesn't identify the user or the agent, and neither record contains a trace_id. Configure these settings to record them:

Setting	Set on	Effect
usernameClaim	The jwt or oAuthTokenExchange middleware	Sets ClientUsername in the access record. The value is - when unset.
forwardHeaders	The jwt or oAuthTokenExchange middleware	Copies token claims, such as the user and the agent, into request headers. You choose the header names. The upstream service also receives these headers.
accessLog.fields.headers	Static configuration	Keeps the named request headers in the access record. Traefik Hub drops all request headers by default.
tracing	Static configuration	Adds trace_id to both records. Without it, no field joins them. Traefik Hub generates the identifier even when no OpenTelemetry collector receives the traces.

Use the claims that name the user and the agent in your tokens. The example assumes sub names the user and azp names the agent.

JWT middleware options

usernameClaim: sub
forwardHeaders:
  X-Evidence-User: sub
  X-Evidence-Agent: azp

values.yaml

accessLog:
  fields:
    headers:
      defaultMode: drop
      names:
        X-Evidence-User: keep
        X-Evidence-Agent: keep
tracing:
  otlp:
    enabled: true

To also send the traces to a collector, set tracing.otlp.http.enabled and tracing.otlp.http.endpoint.

List every header in accessLog.fields.headers. The access record omits any header that forwardHeaders sets but the list doesn't name.

Clients Can Send the User and Agent Headers

Set forwardHeaders on every route that uses these header names. accessLog.fields.headers applies to every route. On a route that doesn't set forwardHeaders for a header you set to keep, the access record contains whatever value the client sent. A client can then forge the user and agent on that route.

Set usernameClaim and forwardHeaders on the middleware whose token names the user and the agent.

• On a route without token exchange, set them on the jwt middleware. They read the token the client sent.
• On a route with a token exchange middleware, set them on that middleware to read the exchanged token. The exchanged token must be a JWT, or the request fails with a 500.
• To record both the original caller and the exchanged identity, set forwardHeaders on both middlewares with different header names. Set usernameClaim on only one, because both write ClientUsername and the later middleware overwrites the earlier one.
• When a token has no claim that forwardHeaders names, Traefik Hub removes that header from the request, so the access record has no field for it. This also removes any value the client sent for that header.

When a token exchange middleware runs before AuthZEN, mappings that read $token read the exchanged token's claims. See Attach the middleware to a route for what this means for your policies.

Forward the traceparent header between requests. When one operation sends two requests through the gateway, for example a client calls an MCP server and the MCP server calls an API, the MCP server must forward the traceparent header. Both requests then share one trace_id. Otherwise each request gets its own trace_id, and the records of the two requests don't join.

Read a decision record

Traefik Hub writes each decision record as one JSON line. The examples are formatted for readability.

An authzen-mcp decision that allows a tool call:

{
  "level": "info",
  "middlewareName": "apps-user-policy@kubernetescrd",
  "middlewareType": "authzen-mcp",
  "decision": "allow",
  "policy": "user-tool-call",
  "trace_id": "5e5e8662ec810c75a0a215e2d8d7e56b",
  "transparency_log_id": "a5499d5c-7cd8-439b-88a8-47924b9db2ab",
  "gateway_id": "6ff586e3-9fe2-4e6a-af4c-437c58f56579",
  "time": "2026-10-06T05:21:10Z",
  "message": "Request allowed"
}

An authzen decision that refuses a request:

{
  "level": "info",
  "middlewareName": "apps-api-refunds@kubernetescrd",
  "middlewareType": "authzen",
  "decision": "deny",
  "reason": "payments API, refund",
  "trace_id": "5e5e8662ec810c75a0a215e2d8d7e56b",
  "transparency_log_id": "38dbf645-b0af-4a15-b037-acc5eefcae29",
  "gateway_id": "6ff586e3-9fe2-4e6a-af4c-437c58f56579",
  "time": "2026-10-06T05:21:10Z",
  "message": "Request denied"
}

A decision record contains these fields:

Field	Content
middlewareName, middlewareType	The middleware that made the decision. Each key appears twice when an authentication middleware such as jwt ran earlier in the chain, once for that middleware and once for this one. Read the last occurrence, and use a JSON parser that keeps the last value, such as jq.
decision	allow, deny, or bypass. A bypass means an authzen-mcp policy with strategy: bypass skipped the PDP. Match on this field, not on message, to tell these results apart.
policy	The name of the policy that applied. Written by authzen-mcp only. Name every policy, because the field is absent when a policy has no name.
reason	The rendered reason template. Present only when you set reason on the middleware or policy.
trace_id	Joins this record to the access record. Present only when tracing is enabled.
transparency_log_id	The identifier of this record in the transparency log.
gateway_id, time, level, message	The gateway identifier, the decision time, the log level, and a summary of the result.

The authzen and authzen-mcp middlewares record the rule that applied in different fields:

• authzen-mcp writes the policy name.
• authzen writes the reason value. The record has no rule when reason isn't set, so set a distinct reason on each authzen middleware.

An authzen-mcp deny record has neither policy nor reason when no policy applied. Read message to find the cause:

• No policy matched the request means no policy's match expression was true. The client gets the defaultOnDenyResponse, and the record has no reason even when you set defaultReason.
• Token audience does not include the configured resource identifier means resourceIdentifier is set and the token's aud claim doesn't contain it. The client gets a 403, and defaultOnDenyResponse doesn't apply.

The reason option accepts a Go template that can reference the context the PDP returns, such as {{ .Decision.reason }}. If your PDP returns no context, as OpenFGA does, this template renders <no value>, and Traefik Hub writes <no value> into the record. Set reason to a fixed string instead, for example reason: "payments API, refund".

Read an access record

Field	Content
ClientUsername	The user, from usernameClaim. The value is - when the route's middleware doesn't set usernameClaim.
request_<header name>	A request header kept by accessLog.fields.headers, such as request_X-Evidence-User.
RequestPath	The request path.
DownstreamStatus	The status returned to the client.
trace_id, TraceId	Joins this record to the decision records. Present only when tracing is enabled.
transparency_log_id	The identifier of this record in the transparency log.

{
  "RequestPath": "/refunds",
  "DownstreamStatus": 403,
  "ClientUsername": "[email protected]",
  "request_X-Evidence-User": "[email protected]",
  "request_X-Evidence-Agent": "agent-1",
  "trace_id": "5e5e8662ec810c75a0a215e2d8d7e56b",
  "transparency_log_id": "d5b33cf1-1386-4254-b597-6f6fd40dc9af"
}

When ClientUsername is -, read the forwarded header field instead, such as request_X-Evidence-User.

Correlate records by trace identifier

Use trace_id to collect every record of one request. This example reads the files at log.filePath and accessLog.filePath, named traefik.log and access.log, and finds the records of the most recent denied request:

TRACE=$(jq -r 'select(.decision=="deny") | .trace_id' traefik.log | tail -1)

jq --arg t "$TRACE" 'select(.trace_id==$t) | {middlewareType, decision, policy, reason}' traefik.log
jq --arg t "$TRACE" 'select(.trace_id==$t) | {RequestPath, DownstreamStatus, ClientUsername}' access.log

A tool call that the MCP route allows and the API route refuses produces these records:

Record	Content
Decision	authzen-mcp, policy agent-tool-call, allow
Decision	authzen-mcp, policy user-tool-call, allow
Decision	authzen, reason payments API, refund, deny
Access	/refunds, status 403, user and agent
Access	/mcp, status 200, user and agent

The MCP route can return 200 even though the API route refused the call, because the MCP server returns its own response. Identify the refusal from the deny decision, and the user and agent from the access records.

Identify refusals without a decision record

Some requests produce no decision record. To find the failures, check DownstreamStatus in the access record.

Situation	Response to the client	Records written
The PDP is unreachable or returns an error	500	An error line with no decision field, and the access record
The mapping reads a missing field without the optional form, such as $request.headers['X-Id']	500	An error line with no decision field, and the access record
The mapping reads a missing field with the optional form, such as $request.headers[?'X-Id']	The onDenyResponse	A deny decision with an error field naming the missing attribute and no reason, and the access record
The authzen mapping reads request.body, and the body is larger than maxRequestBodySize	413	The access record only
The authzen mapping reads request.body, and the body can't be decoded as its Content-Type	500	An error line with no decision field, and the access record
An authzen-mcp POST request has a body larger than hub.mcpgateway.maxRequestBodySize	413	The access record only
An authzen-mcp POST request has an empty body or a body that isn't valid JSON	400	The access record only
An authzen-mcp request uses a method other than POST, such as GET or DELETE	The upstream's response	The access record only
A token exchange middleware fails to exchange the token	401	The access record only
log.level is above info	The normal response	The access record only

Related Content

• See the Transparency Logs reference for the full transparencyLogs configuration, including the storage driver and witness group fields.
• See the AuthZEN middleware reference for the reason option and the decision record that authzen writes.
• See Configure AuthZEN Authorization for API Gateway to produce these records for an API route.
• See Configure AuthZEN Authorization for MCP Gateway to produce these records for an MCP route.