Skip to main content

Chat Completion

The Chat Completion middleware "promotes" any route to a chat-completion endpoint. It adds GenAI metrics, central governance of model/parameters, and (optionally) lets clients override them.

Key Features and Benefits​

  • One-line enablement: attach middleware, your route becomes an AI endpoint.
  • Governance: lock or allow overrides for model, temperature, topP, etc.
  • Metrics: emits OpenTelemetry GenAI spans and counters.
  • Works for local or cloud models: All you need is a Kubernetes Service pointing at the upstream host.

Requirements​

  • You must have AI Gateway enabled:

    helm upgrade traefik traefik/traefik -n traefik --wait \
      --reset-then-reuse-values \
      --set hub.aigateway.enabled=true

  • If routing to a cloud LLM provider, define a Kubernetes ExternalName service.

How It Works​

  1. Intercepts the request and validates it against the OpenAI chat-completion schema.
  2. Applies governance by rewriting model or param fields if overrides are denied.
  3. Starts a GenAI span and records the prompt tokens.
  4. Forwards the (possibly rewritten) request to the upstream LLM.
  5. Records usage metrics from the response (model, prompt/completion tokens, latency).

Configuration Example​

apiVersion: traefik.io/v1alpha1
kind: Middleware
metadata:
  name: chatcompletion
spec:
  plugin:
    chat-completion:
      token: urn:k8s:secret:ai-keys:openai-token
      model: gpt-4o
      allowModelOverride: false
      allowParamsOverride: true
      params:
        temperature: 1
        topP: 1
        maxTokens: 2048
        frequencyPenalty: 0
        presencePenalty: 0

Configuration Options​

FieldDescriptionRequiredDefault
tokenURN of a Kubernetes Secret holding the API key (for example, urn:k8s:secret:<secretname>:<key>)No
modelDefault (fallback) model name enforced when overrides are denied. Starting in v3.21.0, must be unset when allowModelOverride is trueNo
allowModelOverridetrue = clients may set the model field; false = middleware rewrites to modelNoauto (true if model empty, else false)
allowParamsOverridetrue = clients may override params; false = middleware enforces paramsNoauto (true if params empty, else false)
paramsBlock containing default generation parameters applied when client omits themNo
params.temperatureDefault temperature valueNo
params.topPDefault top‑p valueNo
params.maxTokensDefault max token countNo
params.maxCompletionTokensDefault max completion token count (used by newer models like GPT-5)No
params.frequencyPenaltyDefault frequency‑penalty valueNo
params.presencePenaltyDefault presence‑penalty valueNo
providerGenAI provider name for gen_ai.provider.name. A string, for example azure.ai.openai. Overrides host-based resolution. Falls back to openai when the host doesn't match a known provider. Also selects the prices a cost limit uses. When unset, cost limit pricing detects the provider from the model name.No
GPT-5 and maxCompletionTokens

OpenAI's GPT-5 models (such as gpt-5-nano) require the maxCompletionTokens parameter instead of the older maxTokens parameter. If you need to support both GPT-4 and GPT-5 models, use Model-Based Routing to create separate routes with dedicated Chat Completion middleware configurations—one using maxTokens for GPT-4 routes, and another using maxCompletionTokens for GPT-5 routes.

tip

Combine Chat Completion with Semantic Cache or Content Guard by listing multiple middlewares in the same IngressRoute.

Compression

Chat middlewares strip Accept-Encoding from client requests to keep the body readable by governance filters, but always request compressed responses from the backend for efficiency.

Standard flow: Client (uncompressed) → Traefik Hub → Backend (compressed) → Traefik Hub (decompresses) → Client (uncompressed)

If you need compressed responses for your clients (for example, to reduce bandwidth on mobile apps or slow networks), add Traefik's standard Compress middleware before the AI middlewares:

With Compress middleware: Client (uncompressed) → Traefik Hub → Backend (compressed) → Traefik Hub (decompresses + re-compresses) → Client (compressed)

However, this creates double-compression overhead because Traefik Hub must decompress the backend response to apply governance filters, then the Compress middleware re-compresses it for the client. For best performance, avoid the Compress middleware on AI routes unless client compression is essential.

Model Access Policies​

Model access policies restrict which models a caller can reach, and control what a model discovery call (GET /v1/models) returns to them. Use them when a route serves more than one caller identity: reserving an expensive model for one team, hiding an entire model family from everyone, or publishing a short, renamed catalog to a chat client such as Open WebUI instead of the provider's full list.

policies and listPolicies govern which models a caller can reach. Both evaluate the caller's JWT claims, using the same expression language as MCP TBAC. modelAliases renames a model unconditionally, without evaluating any claims:

  • policies decide whether a request naming a given model reaches the provider. Both policies and defaultAction require allowModelOverride: true. Without it, the middleware rejects the configuration and every request on the route returns 404. allowModelOverride defaults to true only when model is unset, and the middleware rejects model together with allowModelOverride: true. To use policies, leave model unset. Each request then names its own model in the model field.
  • listPolicies decide what a GET /v1/models call returns: which models are hidden from the list, and which extra models are added. They run independent of allowModelOverride. They only run against GET /v1/models. A direct GET /v1/models/{id} request bypasses them and returns that model whether or not it's hidden or denied from the list.
  • modelAliases maps an upstream model to a different name shown in the list and accepted from callers instead of the upstream identifier. Unlike policies and listPolicies, an alias has no match condition. It applies to every caller.

See Restrict and Rename AI Models by Identity for a complete walkthrough that combines all three.

spec:
  plugin:
    chat-completion:
      allowModelOverride: true
      policies:
        - match: Equals(`req.model`, `gpt-4o`) && Contains(`jwt.groups`, `admin`)
          action: allow
        - match: Equals(`req.model`, `gpt-4o`)
          action: deny
      defaultAction: allow
      listPolicies:
        - match: Prefix(`res.model.id`, `o1-`)
          action: hide
      modelAliases:
        - model: gpt-4o
          exposedAs: acme-flagship

With this configuration, only callers whose JWT carries the admin group can reach gpt-4o, whether they request it by its upstream identifier or its alias acme-flagship. Every other caller's request for either name is denied by the second policies rule. A request for any other model is allowed by defaultAction: allow. This rule set covers gpt-4o and acme-flagship only. A request naming a snapshot such as gpt-4o-2024-08-06 is also allowed by defaultAction: allow. See Match every name the provider accepts.

Every caller's model list hides o1-* models and shows gpt-4o under the name acme-flagship instead of its real identifier.

defaultAction defaults to deny

When defaultAction is left unset, the middleware denies every model no rule explicitly allows, including models no rule addresses at all. A policies list containing only allow rules becomes an allowlist unless you also set defaultAction: allow, as the example in this section does to let every other model through.

Match every name the provider accepts​

Write a rule for every name the provider accepts for a model. policies compare the model name as a string, and a provider can accept more than one name for the same model. OpenAI serves gpt-4o and dated snapshots such as gpt-4o-2024-08-06. Under defaultAction: allow, a deny rule on gpt-4o alone leaves the snapshots reachable.

Choose the match function by the names you need to cover:

  • Equals compares the exact string, so it covers only the one name you write.
  • OneOf compares against a list of exact names, for example OneOf(`req.model`, `gpt-4o`, `gpt-4o-2024-08-06`). It covers those names and nothing else, so a snapshot released later isn't covered.
  • Prefix covers every name that starts with the text you give. Use it only when no other model shares the prefix. Prefix(`req.model`, `gpt-4o`) also matches gpt-4o-mini.

To avoid listing names, set defaultAction: deny and add allow rules for only the models you want.

Fields​

FieldDescriptionRequiredDefault
policiesRequest-time rules deciding whether a call may target a given model. Requires allowModelOverride: true. The middleware rejects the configuration otherwiseNo
policies[].matchExpression matching the request. The only fields the match expression can reference are the model the request targets as req.model, after modelAliases resolve, and the caller's JWT claims as jwt (for example, Equals(`req.model`,`gpt-4o`) && Contains(`jwt.groups`,`admin`)). Any other field name is accepted without an error and never matches, so a deny rule that misspells one denies nothing. An omitted match matches every requestNomatches everything
policies[].actionallow or denyYes
defaultActionOutcome for a request no policies rule matches. Requires allowModelOverride: trueNodeny
listPoliciesRules deciding what a GET /v1/models response shows for the calling identityNo
listPolicies[].matchExpression matching a model in the response. The only fields the match expression can reference are the model as res.model (the matching entry in the response body's data[] array) and the caller's JWT claims as jwt (for example, Equals(`res.model.id`,`gpt-4o`) && Contains(`jwt.groups`,`admin`)). Any other field name is accepted without an error and never matches, so a hide rule that misspells one hides nothing. An omitted match matches every modelNomatches everything
listPolicies[].actionshow, hide, or appendYes
listPolicies[].modelsModels to add to the response. Required when action is appendConditional
listPolicies[].models[].idIdentifier of the appended modelYes
listPolicies[].models[].displayNameDisplay name, returned as display_name in the Anthropic format onlyNo
listPolicies[].models[].ownedByOwner, returned as owned_by in the OpenAI format onlyNo
listPolicies[].models[].createdCreation time in Unix seconds, returned as created (OpenAI format) or created_at in RFC 3339 (Anthropic format)No
listDefaultActionOutcome for a model no listPolicies rule matches: show or hideNoshow
modelAliasesUpstream-model-to-alias mappings, applied unconditionally to every callerNo
modelAliases[].modelThe upstream model identifier being aliasedYes
modelAliases[].exposedAsThe name shown in the list and accepted from callers instead of modelYes
statusCodeOnDenyHTTP status code returned when a policies rule denies a requestNo403

For policies and for listPolicies show and hide rules, the first matching rule decides, so list the more specific rules first. append rules are independent of that order.

When listPolicies hide every model for a caller, for example under listDefaultAction: hide when no show rule matches that caller, GET /v1/models still returns 200 with an empty data array. A model that policies deny for a caller still appears in the list unless a listPolicies rule hides it.

Adding a model not in the upstream list​

Add a listPolicies rule with action: append to add an entry the upstream response doesn't list on its own, for example a fine-tuned model reachable through this route but absent from the provider's own catalog:

listPolicies:
  - match: Contains(`jwt.groups`, `admin`)
    action: append
    models:
      - id: gpt-4o-ft-support
        ownedBy: acme

With this configuration, only callers whose JWT carries the admin group see gpt-4o-ft-support in GET /v1/models. An append policy's match can only check jwt, since there's no existing list entry yet for it to evaluate res.model against.

The upstream has to serve GET /v1/models. append adds to that list, and an error response passes through with nothing appended.

Appending a model to the list doesn't grant access to it. If allowModelOverride is true, add a matching policies rule so a request naming gpt-4o-ft-support is allowed. Otherwise defaultAction governs it like any other name.

How an alias applies end to end​

Once modelAliases maps a model, the alias replaces the identifier everywhere a caller interacts with it, for every caller:

  • The GET /v1/models response shows the alias, not the upstream identifier.
  • listPolicies match against the upstream identifier, before aliases rename it. Write res.model.id rules for the upstream name, not the alias.
  • A request naming the alias resolves to the upstream identifier before policies evaluates it. See Order of evaluation.
  • A request naming the upstream identifier directly resolves to itself. policies see the same upstream name whether the caller sent the alias or the identifier. You can't allow an alias while refusing its upstream identifier. A caller who knows gpt-4o can use it wherever acme-flagship is allowed.
  • The response reports the alias back to the caller, but observability reports the real upstream identifier in traefik.hub.gen_ai.upstream.model. See AI Gateway Observability's emitted span attributes.
  • A Token Rate Limit with costLimit prices an alias as its upstream model only when it runs after this middleware. Placed before it, the quota sees the alias, which has no price, so every aliased request is refused with 422, or counted as zero when allowMissingPricing is true.
  • Each alias must be unique in both directions: one exposed name maps to exactly one upstream model, and one upstream model maps to exactly one exposed name. A modelAliases list that breaks either rule fails validation before the middleware starts.

Order of evaluation​

Traefik Hub resolves any modelAliases mapping first, then evaluates policies against the resolved upstream model name.

Write policies rules against the upstream identifier. A caller requesting the model by its alias or by its upstream identifier resolves to the same name before policies sees it, so one rule governs both:

match: Equals(`req.model`, `gpt-4o`) && Contains(`jwt.groups`, `admin`)

This rule matches a request for gpt-4o and a request for acme-flagship alike, since both resolve to gpt-4o before the match runs. See Step 3 of Restrict and Rename AI Models by Identity for a complete walkthrough that applies this pattern.

Common Deployment Patterns​

Local Inference (In-Cluster Model Server)​

Deploy an LLM—such as Ollama inside your cluster and expose it with a ClusterIP Service. Attach chat-completion directly to the route:

Service that points at the local model runtime
apiVersion: v1
kind: Service
metadata:
  name: ollama-svc
spec:
  selector:
    app: ollama
  ports:
    - port: 11434
      targetPort: 11434

No API token is needed because the model runs locally, but the middleware still records metrics and enforces any parameter rules you set.

Model-Based Routing​

This pattern lets you expose many local models behind one hostname, with routing driven by the model field in the JSON payload.

Model-based routing
apiVersion: traefik.io/v1alpha1
kind: IngressRoute
metadata:
  name: multi-model-ai
spec:
  routes:
    - kind: Rule
      match: Host(`ai.localhost`) && Model(`qwen2.5:0.5b`)
      middlewares:
        - name: chatcompletion
      services:
        - name: ollama-external
          port: 11434
          passHostHeader: false

Example: Supporting Both GPT-4 and GPT-5​

When routing to OpenAI, GPT-5 models require maxCompletionTokens while GPT-4 models use maxTokens. Create separate routes with dedicated middlewares:

Chat-completion middleware for GPT-5 models
apiVersion: traefik.io/v1alpha1
kind: Middleware
metadata:
  name: chatcompletion-gpt5
spec:
  plugin:
    chat-completion:
      token: urn:k8s:secret:ai-keys:openai-token
      model: gpt-5-nano
      allowModelOverride: false
      allowParamsOverride: false
      params:
        temperature: 1
        topP: 1
        maxCompletionTokens: 2048
        frequencyPenalty: 0
        presencePenalty: 0

Cloud LLM With a Friendly Custom Path​

When you proxy to a provider like Gemini or Cohere, you may want a shorter public path (for example, /api/gemini/chat). Use a replace-path-regex middleware before chat-completion:

Chat-completion middleware (governance + metrics)
apiVersion: traefik.io/v1alpha1
kind: Middleware
metadata:
  name: chatcompletion
spec:
  plugin:
    chat-completion:
      token: urn:k8s:secret:ai-keys:gemini-token
      model: gemini-2.0-flash
      allowModelOverride: false
      allowParamsOverride: true
      params:
        temperature: 0.8
        maxTokens: 4096

With this pattern you get a clean public URL while still benefiting from governance, metrics, and model-based routing.

Provider Compatibility Information

You can find a full list of information about compatibility paths for different providers in the AI Gateway Overview Page.

For example, Google exposes two URLs for Gemini:

  • /v1beta/openai/chat/completions: drop-in replacement for OpenAI SDKs. Use this if your client already talks to /v1/chat/completions.
  • /v1beta/models/gemini-2.0-flash:chat (or …:streamGenerateContent): native REST shape. Use this if you control the client request format.

Pick the one that matches your client, then set replacePathRegex.replacement accordingly to avoid a scenario where Gemini rejects your request even though the gateway added all the right headers.

Streaming Support​

The middleware fully supports streaming responses. When a client sets "stream": true in the request, the response streams back as server-sent events (SSE), and each chunk forwards to the client as it arrives.

Metrics and Streaming

Token usage metrics are recorded for streamed responses, but only when the response contains a usage chunk. An OpenAI-compatible provider emits that chunk only when the client sets stream_options: {"include_usage": true} in the request body. Chat Completion does not add this field on the client's behalf, so a streamed request without it produces no token metrics. Note that adding a Token Rate Limit with clientRequestFormat: ccr to the same route does set it on the upstream request, which makes token metrics available for streamed requests as a side effect.

Related Content​