Tracing
The tracing system allows developers to visualize call flows in their infrastructure. Traefik Hub API Gateway uses OpenTelemetry, an open standard designed for distributed tracing.
Helm Chart Configuration Options
Traefik Hub API Gateway can be configured to provide traces in the OpenTelemetry format using the Helm Chart values.
To know more about the Helm Chart options, refer to the Helm Chart reference page (Options tracing.otlp).
Configuration Options
The section below describes how to configure Traefik Hub API Gateway to provide tracing using the install configuration.
| Field | Description | Default | Required |
|---|---|---|---|
tracing.addInternals | Enables tracing for internal resources (e.g.: ping@internals). | false | No |
tracing.serviceName | Service name used in selected backend. | "traefik" | No |
tracing.sampleRate | The proportion of requests to trace, specified between 0.0 and 1.0. | 1.0 | No |
tracing.globalAttributes | Applies a list of shared key:value attributes on all spans. | No | |
tracing.capturedRequestHeaders | Defines the list of request headers to add as attributes. It applies to client and server kind spans. | No | |
tracing.capturedResponseHeaders | Defines the list of response headers to add as attributes. It applies to client and server kind spans. | False | |
tracing.safeQueryParams | By default, all query parameters are redacted. Defines the list of query parameters to not redact. | No | |
hub.tracing.additionalTraceHeaders.traceContext | Configuration object to duplicate Trace Context headers into dedicated headers. | No | |
hub.tracing.additionalTraceHeaders.traceContext.traceParent | Name of the header to replicate the entire traceparent value. | "" | No |
hub.tracing.additionalTraceHeaders.traceContext.traceState | Name of the header to replicate the tracestate value. | "" | No |
hub.tracing.additionalTraceHeaders.traceContext.traceId | Extracts the trace-id from the traceparent header and duplicates it into a dedicated header. | "" | No |
hub.tracing.additionalTraceHeaders.traceContext.parentId | Extracts the parent-id from the traceparent header and duplicates it into a dedicated header. | "" | No |
tracing.otlp.http | This instructs the exporter to send the tracing to the OpenTelemetry Collector using HTTP. Setting the sub-options with their default values. | null/false | No |
tracing.otlp.http.endpoint | URL of the OpenTelemetry Collector to send tracing to. Format=" <scheme>://<host>:<port><path>" | "http://localhost:4318/v1/tracing" | Yes |
tracing.otlp.http.headers | Additional headers sent with tracing by the exporter to the OpenTelemetry Collector. | No | |
tracing.otlp.http.tls.ca | Path to the certificate authority used for the secure connection to the OpenTelemetry Collector, it defaults to the system bundle. | "" | No |
tracing.otlp.http.tls.cert | Path to the public certificate used for the secure connection to the OpenTelemetry Collector. When using this option, setting the key option is required. | "" | No |
tracing.otlp.http.tls.key | This instructs the exporter to send the tracing to the OpenTelemetry Collector using HTTP. Setting the sub-options with their default values. | ""null/false "" | No |
tracing.otlp.http.tls.insecureskipverify | If insecureSkipVerify is true, the TLS connection to the OpenTelemetry Collector accepts any certificate presented by the server regardless of the hostnames it covers. | false | Yes |
tracing.otlp.grpc | This instructs the exporter to send tracing to the OpenTelemetry Collector using gRPC. | false | No |
tracing.otlp.grpc.endpoint | Address of the OpenTelemetry Collector to send tracing to. Format=" <host>:<port>" | "localhost:4317" | Yes |
tracing.otlp.grpc.headers | Additional headers sent with tracing by the exporter to the OpenTelemetry Collector. | No | |
tracing.otlp.grpc.insecure | Allows exporter to send tracing to the OpenTelemetry Collector without using a secured protocol. | false | Yes |
tracing.otlp.grpc.tls.ca | Path to the certificate authority used for the secure connection to the OpenTelemetry Collector, it defaults to the system bundle. | "" | No |
tracing.otlp.grpc.tls.cert | Path to the public certificate used for the secure connection to the OpenTelemetry Collector. When using this option, setting the key option is required. | "" | No |
tracing.otlp.grpc.tls.key | This instructs the exporter to send the tracing to the OpenTelemetry Collector using HTTP. Setting the sub-options with their default values. | ""null/false "" | No |
tracing.otlp.grpc.tls.insecureskipverify | If insecureSkipVerify is true, the TLS connection to the OpenTelemetry Collector accepts any certificate presented by the server regardless of the hostnames it covers. | false | Yes |
Generative AI Span Attributes Early Access
This feature is currently in early access.
When you enable the AI Gateway, Traefik Hub enriches request spans with OpenTelemetry GenAI semantic-convention attributes. The AI middlewares (chat-completion, responses-api, messages-api) and the MCP middleware each add their own set of attributes. A first group of attributes is always present, and a second group is opt-in because it can carry sensitive content.
These attributes are only emitted when the AI Gateway is enabled and the request is handled by an AI or MCP middleware.
Under detailed trace verbosity, each Hub middleware span is named middleware <type> (for example, middleware chat-completion or middleware mcp), making middleware spans easier to identify.
AI middleware spans
The following attributes are always present, they aren't configurable:
| Attribute | Description | Example |
|---|---|---|
gen_ai.operation.name | The Generative AI operation being performed. | chat |
gen_ai.provider.name | The Generative AI provider handling the request, resolved from the provider field or the upstream host. | openai |
gen_ai.request.model | The model requested by the client. | gpt-4o |
gen_ai.response.model | The model that produced the response. | gpt-4o-2024-08-06 |
gen_ai.conversation.id | The conversation identifier for the request, extracted automatically for the Responses API or read from a request header when traces.conversationIdHeader is set. | conv_123 |
error.type | The class of error the operation ended with, on error. | 500 |
traefik.middleware.type | The Hub middleware category, for selecting AI spans. | AI |
The following attributes are opt-in, grouped into levels set with traces.level. Each level
includes everything from the level below it, minus anything listed in traces.excludeList:
| Attribute | Level | Description |
|---|---|---|
server.address, server.port | minimal (default) | The address and port of the upstream provider. |
gen_ai.response.finish_reasons | detailed | The upstream stop reasons, such as end_turn or content_filter. |
gen_ai.usage.input_tokens, gen_ai.usage.output_tokens | detailed | Input and output token counts. |
gen_ai.usage.cache_creation.input_tokens, gen_ai.usage.cache_read.input_tokens | detailed | The cache token breakdown, present only when the provider reports those counts. |
gen_ai.system_instructions | content | The system prompt sent with the request. |
gen_ai.tool.definitions | content | The tools advertised in the request. |
gen_ai.input.messages | content | The request chat history, including tool-call parts. |
gen_ai.output.messages | content | The response messages, including tool-call parts. |
gen_ai.system_instructions captures the system prompt only. System-role messages sent inline in
the request body are part of the conversation and appear under gen_ai.input.messages instead.
gen_ai.usage.input_tokens already includes the cached tokens when the provider reports prompt
caching. gen_ai.usage.cache_creation.input_tokens and gen_ai.usage.cache_read.input_tokens
are the breakdown within it, not additional tokens on top, and appear only when the provider
reports those counts.
Set the level you need on the AI middleware:
apiVersion: traefik.io/v1alpha1
kind: Middleware
metadata:
name: chatcompletion
spec:
plugin:
chat-completion:
observability:
traces:
level: content
conversationIdHeader: X-Conversation-Id
conversationIdHeader is optional. Set it on Chat Completion and Messages API to populate
gen_ai.conversation.id, since neither API exposes a native conversation field the way the
Responses API does.
At content, gen_ai.input.messages, gen_ai.output.messages, and gen_ai.system_instructions
capture the prompt and completion content, which can contain personal or sensitive data. Set
content only when your tracing backend is allowed to store this content.
MCP middleware spans
The following attributes are emitted on every MCP middleware span:
| Attribute | Description | Example |
|---|---|---|
gen_ai.operation.name | The MCP operation being performed. | execute_tool |
gen_ai.tool.name | The name of the tool being called, on tool operations. | get_weather |
gen_ai.prompt.name | The name of the prompt being requested, on prompt operations. | code_review_prompt |
mcp.method.name | The JSON-RPC method invoked. | tools/call |
mcp.protocol.version | The MCP protocol version negotiated with the client. | 2025-06-18 |
mcp.resource.uri | The URI of the resource read, on resource operations. | file:///README.md |
mcp.session.id | The MCP session identifier. | session-123 |
jsonrpc.request.id | The JSON-RPC request identifier. | 1 |
The following attributes are opt-in. Each flag is off by default and is enabled under traces.genAI on the MCP middleware:
| Attribute | Config flag | Description |
|---|---|---|
gen_ai.tool.call.arguments | toolCallArguments | The arguments passed to the tool call. |
gen_ai.tool.call.result | toolCallResult | The result returned by the tool call. |
Enable them on the MCP middleware:
apiVersion: traefik.io/v1alpha1
kind: Middleware
metadata:
name: mcp
spec:
plugin:
mcp:
traces:
genAI:
toolCallArguments: true
toolCallResult: true
toolCallArguments and toolCallResult capture the tool inputs and outputs, which can contain personal or sensitive data. Enable them only when your tracing backend is allowed to store this content.
Additional Information
Default protocol
The OpenTelemetry trace exporter will export traces to the collector to https://localhost:4318/v1/traces using HTTP by default,
use the tracing.otlp.grpc option explicitly to enable GRPC instead.
Trace sampling
By default, the OpenTelemetry trace exporter will sample 100% of traces. See the OpenTelemetry's SDK configuration to customize the sampling strategy.
Propagation
Traefik Hub API Gateway supports the OTEL_PROPAGATORS env variable to set up the propagators. The supported propagators are:
- tracecontext (default)
- baggage (default)
- b3
- b3multi
- jaeger
- xray
- ottrace
OTEL_PROPAGATORS=b3,jaeger
Forward OTel trace-id using a Header
To facilitate the propagation of OpenTelemetry (OTel) trace identifiers through dedicated headers, Traefik Hub API Gateway offers the tracing.additionalTraceHeaders.traceContext configuration.
This feature allows the duplication of Trace Context headers into custom headers, enabling enhanced traceability across services.
Example
When migrating from a legacy tracing solution to OpenTelemetry, you may need to continue supplying a legacy tracing header.
By using the duplicate headers option, you can automatically replicate the traceparent and tracestate values, as well as extract the trace-id and parent-id into custom headers.
For example, to replicate the trace-id in a header called E2ETrackingID, configure:
- Helm Chart Values
- Install Configuration
tracing:
additionalTraceHeaders:
traceContext:
traceId: E2ETrackingID
hub:
tracing:
additionalTraceHeaders:
traceContext:
traceId: E2ETrackingID
# ...
If the request arrives with an E2ETrackingID header, it will be overwritten by the trace-id extracted from the OpenTelemetry traceparent.
This helps maintain backward compatibility with existing applications that rely on a legacy tracing header, while still adopting the standard W3C trace context format for OTel-based tracing.
When configuring tracing using install configuration, the hub prefix is required (e.g., hub.tracing.additionalTraceHeaders). However, when using Helm Chart values, the prefix is not needed (e.g., tracing.additionalTraceHeaders).
This is because the Helm values are mapped directly to Traefik Hub's internal configuration, while the install configuration follows the binary CLI structure.
