Monitor Token Usage and Cache Savings

Early Access

This feature is currently in early access.

Traefik Hub AI Gateway emits OpenTelemetry GenAI metrics for every AI request, including input and output token counts. The Chat Completion, Messages API, and Responses API middlewares extract token counts from both buffered and streamed responses, giving you consistent observability regardless of whether your clients use "stream": true.

When you pair an AI middleware with the Semantic Cache, you can also track how many tokens each cache hit avoids sending upstream with the traefik.hub.semantic_cache.token_saved counter, enabled via observability.metrics.

This guide covers enabling streaming token metrics, configuring Responses API semantic caching, and setting up the saved-tokens counter.

Prerequisites

AI Gateway enabled on your cluster
Metrics collection enabled. See the observability reference
At least one of the AI middlewares deployed: Chat Completion, Messages API, or Responses API
For saved-tokens metrics: the Semantic Cache middleware deployed alongside an AI middleware

How token metrics work

The Chat Completion, Messages API, and Responses API middlewares emit gen_ai.client.token.usage histograms for every successful response. Token extraction works for both response types:

Non-streamed responses: token counts are parsed from the complete JSON body.
Streamed responses (SSE): the middleware reads usage data from the SSE stream as chunks arrive and forwards each chunk to the client immediately. The full body is never buffered for metric purposes.

Set observability.metrics.level: detailed on the AI middleware to enable gen_ai.client.token.usage — the default minimal level emits no GenAI metrics, streamed or not.

The emitted counter carries these labels:

Label	Values
`gen_ai.token.type`	`input`, `output`
`gen_ai.response.model`	Model name from response
`gen_ai.system`	Provider identifier

note

For the Chat Completion middleware, the upstream must include stream_options.include_usage: true in the streamed response for token counts to be available. This is standard behavior for OpenAI-compatible providers but may differ for custom backends.

For the Responses API middleware, token counts come from the terminal response.completed event. If the stream ends before this event arrives, no token-usage datapoint is recorded for that response.

Configure Responses API semantic caching

The Semantic Cache middleware now supports the OpenAI Responses API format, in addition to Chat Completions and Messages API.

Set clientRequestFormat: responsesAPI on your Semantic Cache middleware:

semantic-cache-responses.yaml
apiVersion: traefik.io/v1alpha1
kind: Middleware
metadata:
  name: semantic-cache-responses
spec:
  plugin:
    semantic-cache:
      clientRequestFormat: responsesAPI
      vectorizer:
        openai:
          model: text-embedding-3-small
          token: urn:k8s:secret:ai-keys:openai-token
      vectorDB:
        redis:
          endpoints: ["redis.default.svc.cluster.local:6379"]
          collectionName: responses_cache
          maxDistance: 0.4
          ttl: 1800

The cache key is built from the request input field (both plain-string and message-array shapes are supported). By default, the top-level instructions field is included in the cache key.

Set ignoreInstructions: true to drop the instructions field from the cache key. Two requests with the same input but different instructions will then share a cache entry:

semantic-cache-responses-ignore-instructions.yaml
apiVersion: traefik.io/v1alpha1
kind: Middleware
metadata:
  name: semantic-cache-responses
spec:
  plugin:
    semantic-cache:
      clientRequestFormat: responsesAPI
      ignoreInstructions: true
      vectorizer:
        openai:
          model: text-embedding-3-small
          token: urn:k8s:secret:ai-keys:openai-token
      vectorDB:
        redis:
          endpoints: ["redis.default.svc.cluster.local:6379"]
          collectionName: responses_cache
          maxDistance: 0.4
          ttl: 1800

Both streaming and non-streaming Responses API requests are cached. On a cache hit, streamed requests receive a replayed text/event-stream response with the usage fields stripped.

Enable the saved-tokens counter

The traefik.hub.semantic_cache.token_saved counter records how many input and output tokens each cache hit avoids sending to the upstream LLM. It triggers only on cache hits, and — like every GenAI metric — is only emitted when observability.metrics.level is detailed.

Set observability.metrics.level: detailed on your AI middleware. The example below uses the Responses API middleware; the same block is available on Chat Completion and Messages API. Use excludeList if you want the other five detailed-tier metrics off and only this one on:

responses-api-with-token-saved.yaml
apiVersion: traefik.io/v1alpha1
kind: Middleware
metadata:
  name: responses-api
spec:
  plugin:
    responses-api:
      token: urn:k8s:secret:ai-keys:openai-token
      model: gpt-4o
      observability:
        metrics:
          level: detailed

The counter is emitted as:

OpenTelemetry: traefik.hub.semantic_cache.token_saved
Prometheus: traefik_hub_semantic_cache_token_saved_total

It carries the same labels as gen_ai.client.token.usage: gen_ai.token.type (input or output), gen_ai.response.model, and any API management labels (app_id, app_name, user_id).

Middleware ordering is required

The AI middleware must appear before the Semantic Cache middleware in the router's middleware chain. On a cache hit, the Semantic Cache returns immediately without calling the next middleware. If the AI middleware is listed after the cache, the counter is never reached.

Order the middlewares correctly

Place the AI middleware first and the Semantic Cache second in your IngressRoute:

ingressroute.yaml
apiVersion: traefik.io/v1alpha1
kind: IngressRoute
metadata:
  name: ai-api
spec:
  entryPoints:
    - websecure
  routes:
    - match: Host(`api.example.com`) && PathPrefix(`/v1/responses`)
      kind: Rule
      middlewares:
        - name: responses-api              # AI middleware first
        - name: semantic-cache-responses   # cache second
      services:
        - name: openai-proxy
          port: 443

If you reverse this order, the Semantic Cache hits and returns before the AI middleware can emit the counter. Token-usage (gen_ai.client.token.usage) and duration metrics are not affected by ordering.

Troubleshooting

The saved-tokens counter never appears in Prometheus

Confirm observability.metrics.level: detailed is set on the AI middleware, not the Semantic Cache.
Check that the AI middleware is listed before the Semantic Cache in the IngressRoute middlewares list.
Confirm there are cache hits. The counter triggers only on cache hits. Send a semantically similar request twice and check for a non-zero traefik_hub_semantic_cache_token_saved_total.
Confirm AI Gateway metrics are enabled. See the observability reference.

Token counts are zero for streamed responses

For the Responses API middleware, token counts come from the terminal response.completed event in the stream. If the upstream omits this event or the stream ends early, no datapoint is recorded for that response — it's absent from the metric, not a zero. Check that your upstream provider emits a response.completed frame at the end of the stream.

For Chat Completion, check that the upstream includes stream_options.include_usage in its streamed responses.

Prerequisites​

How token metrics work​

Configure Responses API semantic caching​

Enable the saved-tokens counter​

Order the middlewares correctly​

Troubleshooting​

Next steps​