Responses API

The Responses API middleware promotes any route to an OpenAI-compatible Responses API endpoint. It adds GenAI metrics, central governance of model/parameters, and (optionally) lets clients override them. For how this format compares to Chat Completions and the Messages API, see the Endpoint Format Comparison.

Key Features and Benefits

• One-line enablement: attach middleware, your route becomes an AI Responses API endpoint.
• Governance: lock or allow overrides for model, temperature, topP, tools, and more.
• Metrics: emits OpenTelemetry GenAI spans and counters.
• Tool control: configure and limit the number of tools clients can use.
• Works for local or cloud models: All you need is a Kubernetes Service pointing at the upstream host.

Requirements

• You must have AI Gateway enabled:

  helm upgrade traefik traefik/traefik -n traefik --wait \
    --reset-then-reuse-values \
    --set hub.aigateway.enabled=true

• If routing to a cloud LLM provider, define a Kubernetes ExternalName service.

Model Compatibility

The middleware is designed for the OpenAI Responses API format. When routing to other providers:

• Parameter names may differ (e.g., maxOutputTokens vs max_tokens)
• Parameter limits vary by model and provider
• Tool support is provider-specific

For non-OpenAI providers, you may need to use a proxy service that translates between the Responses API format and your target provider's format.

How It Works

1. Intercepts the request and validates it against the OpenAI Responses API schema.
2. Applies governance by rewriting model, param fields, or instructions if overrides are denied.
3. Starts a GenAI span and records the input tokens.
4. Forwards the (possibly rewritten) request to the upstream LLM.
5. Records usage metrics from the response (model, input/output tokens, latency).

Configuration Example

Middleware

apiVersion: traefik.io/v1alpha1
kind: Middleware
metadata:
  name: responsesapi
spec:
  plugin:
    responses-api:
      token: urn:k8s:secret:ai-keys:openai-token
      model: gpt-4o
      allowModelOverride: false
      allowParamsOverride: true
      params:
        temperature: 1
        topP: 0.9
        maxOutputTokens: 1024
        maxToolCall: 20
        store: true
        tools:
          - type: web_search

Secret

apiVersion: v1
kind: Secret
metadata:
  name: ai-keys
type: Opaque
# Option 1: Plain text
stringData:
  openai-token: sk-proj-XXXXX
# Option 2: Pre-base64 encoded data
# data:
#   openai-token: c2stcHJvai1YWFhYWA==

IngressRoute

apiVersion: traefik.io/v1alpha1
kind: IngressRoute
metadata:
  name: openai-responses
spec:
  routes:
    - kind: Rule
      match: Host(`ai.example.com`)
      middlewares:
        - name: responsesapi
      services:
        - name: openai
          port: 443
          passHostHeader: false

Service

apiVersion: v1
kind: Service
metadata:
  name: openai
spec:
  type: ExternalName
  externalName: api.openai.com
  ports:
    - port: 443

Configuration Options

Field	Description	Required	Default
token	URN of a Kubernetes Secret holding the API key (for example, urn:k8s:secret:<secretname>:<key>)	No
model	Default model to use (for example, gpt-4o, gpt-4-turbo). Required when allowModelOverride is false. Starting in v3.21.0, must be unset when it is true	Conditional
allowModelOverride	true = clients may set the model field; false = middleware rewrites to model	No	auto (true if model empty, else false)
allowParamsOverride	true = clients may override params; false = middleware enforces params	No	auto (true if params empty, else false)
instructions	System instructions to include in every request	No
provider	GenAI provider name for gen_ai.provider.name. A string, for example azure.ai.openai. Overrides host-based resolution. Falls back to openai when the host doesn't match a known provider. Also selects the prices a cost limit uses. When unset, cost limit pricing detects the provider from the model name.	No
params	Block containing default generation parameters	No
params.temperature	Sampling temperature between 0 and 2. Higher values make output more random	No
params.topP	Nucleus sampling parameter. An alternative to temperature sampling	No
params.maxOutputTokens	Maximum number of tokens to generate in the response (OpenAI Responses API format)	No
params.maxToolCall	Maximum number of tools that can be configured in a request. Requests exceeding this limit will be rejected	No
params.store	Whether to store the conversation for future reference (OpenAI feature)	No
params.tools	Array of tool configurations. Each tool must have a type field (for example, web_search, file_search, function)	No
params.tools[].type	Type of tool: web_search, file_search, code_interpreter, image_generation, function, mcp, etc.	Yes
params.tools[].name	Name of the tool (required for function type)	No
params.tools[].description	Description of what the tool does (for function type)	No
params.tools[].parameters	JSON Schema object describing the tool's parameters (for function type)	No
params.tools[].serverLabel	Label for an mcp tool server	No
params.tools[].serverURL	URL of the mcp tool server	No
params.tools[].allowedTools	Optional allowlist of mcp tool names	No
params.tools[].requireApproval.value	Approval policy for mcp tools. Use never or always	No
params.tools[].requireApproval.filter.never.toolNames	MCP tool names that do not require approval	No
params.tools[].requireApproval.filter.always.toolNames	MCP tool names that always require approval	No
params.tools[].headers	Additional headers sent to the mcp tool server	No

The table above lists the fields Traefik Hub actively governs. The following field types in the request body pass through to the upstream provider unchanged:

• Newer or provider-specific fields on a tool, for example a tool_search tool's format or a web_search tool's filters and search_context_size.
• Model-specific settings, like a reasoning value tuned for the model the client originally requested. Pinning a model with allowModelOverride: false only replaces the model field, so these settings still pass through to the pinned model unchanged, and the provider rejects the request with a 400 if it doesn't accept them. This happens because reasoning settings are model-specific. A value valid for one model may not exist on another. There's no way yet to adjust these settings for the pinned model, so treat a 400 after pinning as a likely cause before assuming a config error.

Parameter Override Behavior

The middleware supports two modes for handling parameters:

Mode 1: Allow Parameters Override (allowParamsOverride: true)

When enabled (default), the middleware acts as a default value provider:

• If a client provides a value for a parameter, the client's value is used.
• If a client doesn't provide a value, the configured default is applied.
• Tools follow the same pattern: client-provided tools take precedence, configured tools are used as fallback.

spec:
  plugin:
    responses-api:
      model: gpt-4o
      allowParamsOverride: true  # Clients can override
      params:
        temperature: 0.7
        maxOutputTokens: 1000

Mode 2: Force Parameters (allowParamsOverride: false)

When turned off, the middleware enforces the configured values:

• All configured parameters override client values.
• Clients cannot change these settings.
• Useful for strict governance and cost control.

spec:
  plugin:
    responses-api:
      model: gpt-4o
      allowParamsOverride: false  # Enforce configured values
      params:
        temperature: 0.5
        maxOutputTokens: 500
        tools:
          - type: web_search

Model Override Behavior

The allowModelOverride setting controls whether clients can specify their own model:

• allowModelOverride: true: Clients choose the model. Starting in v3.21.0, leave model unset, because the middleware refuses model together with allowModelOverride: true.
• allowModelOverride: false (default when model is set): Clients must use the configured model. If they omit the model field or specify a different model, the configured model is used.

spec:
  plugin:
    responses-api:
      allowModelOverride: true  # Clients choose the model

Model Access Policies

Model access policies restrict which models a caller can reach, and control what a model discovery call (GET /v1/models) returns to them. Use them when a route serves more than one caller identity: reserving an expensive model for one team, hiding an entire model family from everyone, or publishing a short, renamed catalog to a chat client such as Open WebUI instead of the provider's full list.

policies and listPolicies govern which models a caller can reach. Both evaluate the caller's JWT claims, using the same expression language as MCP TBAC. modelAliases renames a model unconditionally, without evaluating any claims:

• policies decide whether a request naming a given model reaches the provider. Both policies and defaultAction require allowModelOverride: true. Without it, the middleware rejects the configuration and every request on the route returns 404. allowModelOverride defaults to true only when model is unset, and the middleware rejects model together with allowModelOverride: true. To use policies, leave model unset. Each request then names its own model in the model field.
• listPolicies decide what a GET /v1/models call returns: which models are hidden from the list, and which extra models are added. They run independent of allowModelOverride. They only run against GET /v1/models. A direct GET /v1/models/{id} request bypasses them and returns that model whether or not it's hidden or denied from the list.
• modelAliases maps an upstream model to a different name shown in the list and accepted from callers instead of the upstream identifier. Unlike policies and listPolicies, an alias has no match condition. It applies to every caller.

See Restrict and Rename AI Models by Identity for a complete walkthrough that combines all three.

spec:
  plugin:
    responses-api:
      allowModelOverride: true
      policies:
        - match: Equals(`req.model`, `gpt-4o`) && Contains(`jwt.groups`, `admin`)
          action: allow
        - match: Equals(`req.model`, `gpt-4o`)
          action: deny
      defaultAction: allow
      listPolicies:
        - match: Prefix(`res.model.id`, `o1-`)
          action: hide
      modelAliases:
        - model: gpt-4o
          exposedAs: acme-flagship

With this configuration, only callers whose JWT carries the admin group can reach gpt-4o, whether they request it by its upstream identifier or its alias acme-flagship. Every other caller's request for either name is denied by the second policies rule. A request for any other model is allowed by defaultAction: allow. This rule set covers gpt-4o and acme-flagship only. A request naming a snapshot such as gpt-4o-2024-08-06 is also allowed by defaultAction: allow. See Match every name the provider accepts.

Every caller's model list hides o1-* models and shows gpt-4o under the name acme-flagship instead of its real identifier.

defaultAction defaults to deny

When defaultAction is left unset, the middleware denies every model no rule explicitly allows, including models no rule addresses at all. A policies list containing only allow rules becomes an allowlist unless you also set defaultAction: allow, as the example in this section does to let every other model through.

Match every name the provider accepts

Write a rule for every name the provider accepts for a model. policies compare the model name as a string, and a provider can accept more than one name for the same model. OpenAI serves gpt-4o and dated snapshots such as gpt-4o-2024-08-06. Under defaultAction: allow, a deny rule on gpt-4o alone leaves the snapshots reachable.

Choose the match function by the names you need to cover:

• Equals compares the exact string, so it covers only the one name you write.
• OneOf compares against a list of exact names, for example OneOf(`req.model`, `gpt-4o`, `gpt-4o-2024-08-06`). It covers those names and nothing else, so a snapshot released later isn't covered.
• Prefix covers every name that starts with the text you give. Use it only when no other model shares the prefix. Prefix(`req.model`, `gpt-4o`) also matches gpt-4o-mini.

To avoid listing names, set defaultAction: deny and add allow rules for only the models you want.

Fields

Field	Description	Required	Default
policies	Request-time rules deciding whether a call may target a given model. Requires allowModelOverride: true. The middleware rejects the configuration otherwise	No
policies[].match	Expression matching the request. The only fields the match expression can reference are the model the request targets as req.model, after modelAliases resolve, and the caller's JWT claims as jwt (for example, Equals(`req.model`,`gpt-4o`) && Contains(`jwt.groups`,`admin`)). Any other field name is accepted without an error and never matches, so a deny rule that misspells one denies nothing. An omitted match matches every request	No	matches everything
policies[].action	allow or deny	Yes
defaultAction	Outcome for a request no policies rule matches. Requires allowModelOverride: true	No	deny
listPolicies	Rules deciding what a GET /v1/models response shows for the calling identity	No
listPolicies[].match	Expression matching a model in the response. The only fields the match expression can reference are the model as res.model (the matching entry in the response body's data[] array) and the caller's JWT claims as jwt (for example, Equals(`res.model.id`,`gpt-4o`) && Contains(`jwt.groups`,`admin`)). Any other field name is accepted without an error and never matches, so a hide rule that misspells one hides nothing. An omitted match matches every model	No	matches everything
listPolicies[].action	show, hide, or append	Yes
listPolicies[].models	Models to add to the response. Required when action is append	Conditional
listPolicies[].models[].id	Identifier of the appended model	Yes
listPolicies[].models[].displayName	Display name, returned as display_name in the Anthropic format only	No
listPolicies[].models[].ownedBy	Owner, returned as owned_by in the OpenAI format only	No
listPolicies[].models[].created	Creation time in Unix seconds, returned as created (OpenAI format) or created_at in RFC 3339 (Anthropic format)	No
listDefaultAction	Outcome for a model no listPolicies rule matches: show or hide	No	show
modelAliases	Upstream-model-to-alias mappings, applied unconditionally to every caller	No
modelAliases[].model	The upstream model identifier being aliased	Yes
modelAliases[].exposedAs	The name shown in the list and accepted from callers instead of model	Yes
statusCodeOnDeny	HTTP status code returned when a policies rule denies a request	No	403

For policies and for listPolicies show and hide rules, the first matching rule decides, so list the more specific rules first. append rules are independent of that order.

When listPolicies hide every model for a caller, for example under listDefaultAction: hide when no show rule matches that caller, GET /v1/models still returns 200 with an empty data array. A model that policies deny for a caller still appears in the list unless a listPolicies rule hides it.

Adding a model not in the upstream list

Add a listPolicies rule with action: append to add an entry the upstream response doesn't list on its own, for example a fine-tuned model reachable through this route but absent from the provider's own catalog:

listPolicies:
  - match: Contains(`jwt.groups`, `admin`)
    action: append
    models:
      - id: gpt-4o-ft-support
        ownedBy: acme

With this configuration, only callers whose JWT carries the admin group see gpt-4o-ft-support in GET /v1/models. An append policy's match can only check jwt, since there's no existing list entry yet for it to evaluate res.model against.

The upstream has to serve GET /v1/models. append adds to that list, and an error response passes through with nothing appended.

Appending a model to the list doesn't grant access to it. If allowModelOverride is true, add a matching policies rule so a request naming gpt-4o-ft-support is allowed. Otherwise defaultAction governs it like any other name.

How an alias applies end to end

Once modelAliases maps a model, the alias replaces the identifier everywhere a caller interacts with it, for every caller:

• The GET /v1/models response shows the alias, not the upstream identifier.
• listPolicies match against the upstream identifier, before aliases rename it. Write res.model.id rules for the upstream name, not the alias.
• A request naming the alias resolves to the upstream identifier before policies evaluates it. See Order of evaluation.
• A request naming the upstream identifier directly resolves to itself. policies see the same upstream name whether the caller sent the alias or the identifier. You can't allow an alias while refusing its upstream identifier. A caller who knows gpt-4o can use it wherever acme-flagship is allowed.
• The response reports the alias back to the caller, but observability reports the real upstream identifier in traefik.hub.gen_ai.upstream.model. See AI Gateway Observability's emitted span attributes.
• A Token Rate Limit with costLimit prices an alias as its upstream model only when it runs after this middleware. Placed before it, the quota sees the alias, which has no price, so every aliased request is refused with 422, or counted as zero when allowMissingPricing is true.
• Each alias must be unique in both directions: one exposed name maps to exactly one upstream model, and one upstream model maps to exactly one exposed name. A modelAliases list that breaks either rule fails validation before the middleware starts.

Order of evaluation

Traefik Hub resolves any modelAliases mapping first, then evaluates policies against the resolved upstream model name.

Write policies rules against the upstream identifier. A caller requesting the model by its alias or by its upstream identifier resolves to the same name before policies sees it, so one rule governs both:

match: Equals(`req.model`, `gpt-4o`) && Contains(`jwt.groups`, `admin`)

This rule matches a request for gpt-4o and a request for acme-flagship alike, since both resolve to gpt-4o before the match runs. See Step 3 of Restrict and Rename AI Models by Identity for a complete walkthrough that applies this pattern.

Tool Control

The middleware provides governance over tool usage:

Configure Default Tools

Provide a list of tools that will be available by default:

params:
  tools:
    - type: web_search
    - type: file_search
    - type: function
      name: get_weather
      description: Get current weather for a location
      parameters:
        type: object
        properties:
          location:
            type: string
            description: City name
        required:
          - location

Tool-Specific Options

Different tool types support additional configuration options:

• function tools: Support name, description, and parameters fields
• mcp tools: Support extensive configuration options including server connections, authentication, and tool-specific parameters
• Built-in tools (web_search, file_search, code_interpreter, image_generation): Have their own specific configuration options

Sub-fields not listed above, such as newer or provider-specific extensions on a built-in tool, are preserved as sent by the client and forwarded to the upstream provider unchanged.

When configuring tools in the middleware, use Traefik Hub field names. For MCP tools, translate upstream Responses API fields such as server_label, server_url, allowed_tools, and require_approval to serverLabel, serverURL, allowedTools, and requireApproval.

The middleware converts these fields to the upstream wire format before forwarding the request. For example, requireApproval.value: never is forwarded as "require_approval": "never". Client request bodies still use the upstream Responses API field names, such as server_label and require_approval.

Refer to the respective tool documentation for tool behavior details and supported values.

Configure an MCP Tool

Use camelCase field names in middleware config even if the upstream provider docs show snake_case:

params:
  tools:
    - type: mcp
      serverLabel: dmcp
      serverURL: https://dmcp-server.deno.dev/sse
      allowedTools:
        - roll_dice
        - lookup_spell
      requireApproval:
        value: never

Limit Tool Count

Use maxToolCall to prevent clients from requesting too many tools:

params:
  maxToolCall: 5  # Maximum 5 tools per request
  tools:
    - type: web_search

When a client request exceeds this limit, it will be rejected with a 400 Bad Request response.

Request Body Size Limits

The middleware enforces a maximum request body size based on the AI Gateway configuration:

helm upgrade traefik traefik/traefik -n traefik --wait \
  --set hub.aigateway.enabled=true \
  --set hub.aigateway.maxRequestBodySize=10485760  # 10MB

Requests exceeding this size will receive a 413 Request Entity Too Large response.

Streaming Support

The middleware fully supports streaming responses. When a client sets "stream": true in the request, the response will be streamed back as server-sent events (SSE).

Metrics and Streaming

Token usage metrics are recorded for streaming responses. The middleware reads the token counts from the terminal response.completed event, so an interrupted stream that never sends that event yields no usage. Duration metrics cover the whole response.

Metrics

The middleware emits OpenTelemetry GenAI metrics when metrics are enabled.

You can also enable additional server metrics and detailed trace attributes through the observability.metrics and observability.traces configuration blocks. See AI Gateway Observability.

Working with Other AI Middlewares

The Responses API middleware can be combined with other AI Gateway middlewares for enhanced functionality. See the Adapting AI Middlewares for Responses API guide for more details.

OpenAI Responses API vs Chat Completions

The Responses API is OpenAI's successor to the Chat Completions API, designed for agent-like applications:

Feature	Chat Completions	Responses API
Request format	messages[] array	input string + optional instructions
Response format	choices[].message.content	output[] array
Built-in tools	Function calling only	Web search, file search, code interpreter, image generation
State management	Client-managed	Server-managed (via previous_response_id).

Important

• Streaming and Generic Middlewares: Content Guard, LLM Guard, and Semantic Cache stream through unchanged when configured with request rules only. Configuring response rules makes them buffer: the middleware waits for the complete response, processes it, and sends it as a single chunk. Semantic Cache streams on a cache miss and replays the stored response on a hit.
• State Management: The middleware does not currently manage conversation state (previous_response_id). Each request is treated as stateless.
• Metrics with Streaming: Token usage metrics are recorded for streaming requests, read from the terminal response.completed event; an interrupted stream that never sends it yields no usage.

Troubleshooting

Request Entity Too Large

Problem: Receiving 413 Request Entity Too Large errors.

Solution: Increase the AI Gateway's max request body size:

helm upgrade traefik traefik/traefik -n traefik --wait \
  --reset-then-reuse-values \
  --set hub.aigateway.maxRequestBodySize=20971520  # 20MB

Tool Count Exceeded

Problem: Receiving 400 Bad Request: Maximum X tools allowed, got Y.

Solution: Either reduce the number of tools in your request or increase maxToolCall:

params:
  maxToolCall: 50  # Increase limit

Model Override Denied

Problem: Client's model selection is being overridden.

Solution: Remove model and enable model override in the middleware:

spec:
  plugin:
    responses-api:
      allowModelOverride: true  # Clients choose the model

Model Denied by Policy

Problem: Receiving a 403 (or the configured statusCodeOnDeny) response with a message like The model "X" is not allowed.

Solution: Check the policies list for a rule that allows the caller's JWT claims to reach that model, and check defaultAction. It denies any model no rule matches. Remember policies require allowModelOverride: true. Without it, the middleware rejects the configuration and every request on the route returns 404.

Metrics Not Being Recorded

Problem: No metrics are being recorded.

Solution: Ensure:

1. AI Gateway is enabled with metrics
2. The request includes the required headers (Hub-App-Name, Hub-App-Id)

"Request has been aborted ... net/http: abort Handler" in Debug Logs

Problem: At debug log level, you see a line like:

Request has been aborted [<client-ip> - /v1/responses]: net/http: abort Handler

Solution: This is expected behavior, not a problem to fix. Streaming clients disconnect the moment they receive the terminal SSE event, before the reverse proxy's copy loop reaches a clean end. Go's net/http/httputil.ReverseProxy raises the http.ErrAbortHandler sentinel in that case, and Traefik's recovery middleware logs it at debug level and continues. It does not retry, drop the response, or affect the client. Check the access log for the actual outcome: a DownstreamStatus: 200 with the full response body confirms the client received the complete answer.

Related Content

• Learn how to use the Responses API with other middlewares in our Responses API Guide
• Configure Content Guard to protect sensitive data
• Set up Semantic Cache to reduce costs
• Use LLM Guard for custom content policies
• Follow the Restrict and Rename AI Models by Identity guide to configure model access policies end to end