Configure AI Gateway Observability

Early Access

This feature is currently in early access.

The AI Gateway middlewares (Chat Completion, Responses API, and Messages API) emit OpenTelemetry GenAI metrics and traces for every request they handle. This page describes the metrics and span attributes you can turn on, and how to configure them.

Overview

Add an observability block to allow metrics and span attributes for a middleware:

• metrics controls which GenAI and Hub metrics the middleware records, and which API Management labels it adds to them.
• traces controls which GenAI span attributes the middleware adds to the request span.

Each block picks a level that turns on a preset group of metrics or attributes, then narrows that group with an optional excludeList.

No metrics by default

Set metrics.level to detailed to get any GenAI metrics at all, including gen_ai.client.operation.duration and gen_ai.client.token.usage. At the default minimal level, the middleware records no GenAI metrics.

Message capture and PII

At traces.level: content, the middleware captures request and response messages and system instructions, which can contain personally identifiable information. Set content only when your tracing backend is allowed to store this content.

Configuration Examples

Add an observability block to any AI middleware (Chat Completion, Responses API, or Messages API). The following example uses Chat Completion:

apiVersion: traefik.io/v1alpha1
kind: Middleware
metadata:
  name: chatcompletion
spec:
  plugin:
    chat-completion:
      model: gpt-4o
      observability:
        metrics:
          level: detailed
          excludeList:
            - gen_ai.server.time_per_output_token
          addLabels:
            - app_id
            - app_name
            - user_id
        traces:
          level: content
          excludeList:
            - gen_ai.tool.definitions
          conversationIdHeader: X-Conversation-Id

Choose a metrics level

Set metrics.level to detailed to record GenAI metrics. detailed allows all six metrics: the two client metrics, the three server metrics, and the Hub token-saved counter. Drop the ones you don't need with excludeList:

spec:
  plugin:
    chat-completion:
      observability:
        metrics:
          level: detailed
          excludeList:
            - gen_ai.server.time_per_output_token

Add API Management labels

metrics.addLabels adds the calling application and user identifiers as labels on the GenAI metrics. These values come from API Management, so the labels appear only for traffic managed through API Management. Setting addLabels doesn't turn metrics on by itself: the value for metrics.level still needs to be detailed.

spec:
  plugin:
    chat-completion:
      observability:
        metrics:
          level: detailed
          addLabels:
            - app_id
            - app_name
            - user_id

Choose a traces level

Set traces.level to detailed or content to add span attributes beyond the always-on ones. detailed adds finish reasons and token counts. content adds everything in detailed, plus input and output messages, system instructions, and tool definitions, the attributes that can carry prompt and completion content. Drop individual attributes with excludeList:

spec:
  plugin:
    chat-completion:
      observability:
        traces:
          level: content
          excludeList:
            - gen_ai.tool.definitions

Group requests from the same conversation

By default, every request gets traced on its own, even the fifth message in the same back-and-forth conversation. That makes a conversation hard to review, since you have to find each request separately and piece it back together.

gen_ai.conversation.id fixes that. Every request from the same conversation carries the same value, so you can filter or group by it in your tracing UI and see the whole exchange in order. Use it to debug why turn four got a bad response, or to measure cost or latency per conversation instead of per request.

How the ID gets populated depends on the middleware:

• Responses API: the middleware reads OpenAI's native conversation field from the request or response body automatically. No configuration needed.
• Chat Completion and Messages API: neither API has a conversation field, so nothing gets extracted automatically. To get correlation here:
  1. Generate one conversation ID per chat session in your client or application.
  2. Send it as a header on every request in that conversation.
  3. Set traces.conversationIdHeader to that header's name:

spec:
  plugin:
    chat-completion:
      observability:
        traces:
          conversationIdHeader: X-Conversation-Id

Setting conversationIdHeader on the Responses API middleware overrides the ID extracted from the body. Use this if you want your own ID scheme instead of OpenAI's.

If neither a header nor a body value is available, the middleware omits the attribute. That request doesn't get grouped.

Configuration Options

metrics

Field	Description	Required	Default
observability.metrics.level	Metrics detail level. A string, minimal or detailed. minimal records no GenAI metrics. detailed records all six (see Emitted metrics)	No	minimal
observability.metrics.excludeList	Metric names to drop from the selected level. An array of strings, for example [gen_ai.server.time_per_output_token]	No	[]
observability.metrics.addLabels	API Management labels to add to the GenAI metrics. An array of app_id, app_name, user_id, for example [app_id, user_id]. Populated only for API Management traffic	No	[]

note

The app_id, app_name, and user_id labels are populated for traffic managed through API Management. They are omitted for traffic that does not carry API Management context.

traces

Field	Description	Required	Default
observability.traces.level	Span attribute detail level. A string, minimal, detailed, or content. Each level includes everything from the level below it (see Emitted span attributes)	No	minimal
observability.traces.excludeList	Span attribute names to drop from the selected level. An array of strings, for example [gen_ai.tool.definitions]	No	[]
observability.traces.conversationIdHeader	Header to read gen_ai.conversation.id from. Needed on Chat Completion and Messages API to populate the attribute. Overrides the Responses API's body-extracted ID	No	""

Emitted metrics

Set metrics.level to detailed to enable all six metrics below. Drop specific ones with metrics.excludeList, using the metric name shown in the table.

Metric	Type	Description
gen_ai.client.operation.duration	Histogram	Duration of GenAI operations, in seconds
gen_ai.client.token.usage	Histogram	Input and output tokens used per operation
gen_ai.server.time_to_first_token	Histogram	Time to the first streamed response token
gen_ai.server.time_per_output_token	Histogram	Average time per streamed output token
gen_ai.server.request.duration	Histogram	Duration of the upstream request, in seconds
traefik.hub.semantic_cache.token_saved	Counter	Tokens saved by the semantic cache on a cache hit

Streaming required

gen_ai.server.time_to_first_token and gen_ai.server.time_per_output_token are recorded only for streamed responses. Set stream: true in the request to get these two metrics.

Emitted span attributes

The following attributes are always present, they aren't configurable:

Attribute	Description	Example
gen_ai.operation.name	The Generative AI operation being performed.	chat
gen_ai.provider.name	The Generative AI provider handling the request, resolved from the provider field or the upstream host.	openai
gen_ai.request.model	The model requested by the client.	gpt-4o
gen_ai.response.model	The model that produced the response.	gpt-4o-2024-08-06
gen_ai.conversation.id	The conversation identifier for the request, extracted automatically for the Responses API or read from a request header when traces.conversationIdHeader is set.	conv_123
error.type	The class of error the operation ended with, on error.	500
traefik.middleware.type	The Hub middleware category, for selecting AI spans.	AI

The following attributes are opt-in, grouped into levels set with traces.level. Each level includes everything from the level below it, minus anything listed in traces.excludeList:

Attribute	Level	Description
server.address, server.port	minimal (default)	The address and port of the upstream provider.
gen_ai.response.finish_reasons	detailed	The upstream stop reasons, such as end_turn or content_filter.
gen_ai.usage.input_tokens, gen_ai.usage.output_tokens	detailed	Input and output token counts.
gen_ai.usage.cache_creation.input_tokens, gen_ai.usage.cache_read.input_tokens	detailed	The cache token breakdown, present only when the provider reports those counts.
gen_ai.system_instructions	content	The system prompt sent with the request.
gen_ai.tool.definitions	content	The tools advertised in the request.
gen_ai.input.messages	content	The request chat history, including tool-call parts.
gen_ai.output.messages	content	The response messages, including tool-call parts.

System instructions

gen_ai.system_instructions captures the system prompt only. System-role messages sent inline in the request body are part of the conversation and appear under gen_ai.input.messages instead.

Provider cache tokens

gen_ai.usage.input_tokens already includes the cached tokens when the provider reports prompt caching. gen_ai.usage.cache_creation.input_tokens and gen_ai.usage.cache_read.input_tokens are the breakdown within it, not additional tokens on top, and appear only when the provider reports those counts.

Related Content

• Chat Completion middleware
• Responses API middleware
• Semantic Cache middleware
• GenAI metrics reference
• GenAI span attributes reference