Messages API

Early Access

This feature is currently in early access.

The Messages API middleware promotes any route to an Anthropic Messages API endpoint. It adds GenAI metrics, central governance of model/parameters, and (optionally) lets clients override them.

Key Features and Benefits

• One-line enablement: attach middleware, your route becomes an Anthropic Messages API endpoint.
• Governance: lock or allow overrides for model, temperature, topP, topK, and more.
• Metrics: emits OpenTelemetry GenAI spans and counters, including Anthropic-specific cache_creation_input_tokens and cache_read_input_tokens.
• Prompt caching: attach an Anthropic ephemeral cache_control marker to the last cacheable block for reduced inference costs.
• Works for local or cloud models: All you need is a Kubernetes Service pointing at the upstream host.

Requirements

• You must have AI Gateway enabled:

  helm upgrade traefik traefik/traefik -n traefik --wait \
    --reset-then-reuse-values \
    --set hub.aigateway.enabled=true

• If routing to a cloud LLM provider (for example, Anthropic), define a Kubernetes ExternalName service.

How It Works

1. Intercepts the request and validates it against the Anthropic Messages API schema.
2. Injects authentication by setting X-Api-Key or Authorization: Bearer headers from the configured credentials.
3. Applies governance by rewriting model or param fields if overrides are denied.
4. Marks the request format as messagesAPI in the request context so other AI middlewares in the chain (Content Guard, LLM Guard, Semantic Cache, Token Rate Limit) automatically adapt to the Anthropic payload format.
5. Starts a GenAI span and records the input tokens.
6. Forwards the (possibly rewritten) request to the upstream LLM.
7. Records usage metrics from the response (model, input/output tokens, cache tokens, latency).

Configuration Example

Middleware

apiVersion: traefik.io/v1alpha1
kind: Middleware
metadata:
  name: messagesapi
spec:
  plugin:
    messages-api:
      apiKey: urn:k8s:secret:ai-keys:anthropic-api-key
      model: claude-opus-4-5
      allowModelOverride: false
      allowParamsOverride: true
      systemPrompt: "You are a helpful assistant. Answer only questions related to our product."
      allowInlineSystemMessages: false  # requires Claude Opus 4.8+
      params:
        temperature: 1
        topP: 0.9
        topK: 50
        maxTokens: 1024

Secret

apiVersion: v1
kind: Secret
metadata:
  name: ai-keys
type: Opaque
# Option 1: Plain text
stringData:
  anthropic-api-key: sk-ant-XXXXX
# Option 2: Pre-base64 encoded data
# data:
#   anthropic-api-key: c2stYW50LVHYWFHYW==

IngressRoute

apiVersion: traefik.io/v1alpha1
kind: IngressRoute
metadata:
  name: anthropic-messages
spec:
  routes:
    - kind: Rule
      match: Host(`ai.example.com`)
      middlewares:
        - name: messagesapi
      services:
        - name: anthropic
          port: 443
          passHostHeader: false

Service

apiVersion: v1
kind: Service
metadata:
  name: anthropic
spec:
  type: ExternalName
  externalName: api.anthropic.com
  ports:
    - port: 443

Configuration Options

Field	Description	Required	Default
apiKey	URN of a Kubernetes Secret holding the Anthropic API key. Sets the X-Api-Key request header (for example, urn:k8s:secret:<secretname>:<key>)	No
authToken	URN of a Kubernetes Secret holding a bearer token. Sets the Authorization: Bearer header (for example, urn:k8s:secret:<secretname>:<key>)	No
model	Default model to use (for example, claude-opus-4-5, claude-3-7-sonnet-latest). Required when allowModelOverride is false	Conditional
allowModelOverride	true = clients may set the model field; false = middleware rewrites to model	No	auto (true if model empty, else false)
allowParamsOverride	true = clients may override params; false = middleware enforces params	No	auto (true if params empty, else false)
systemPrompt	Gateway-enforced system prompt. Replaces the client's top-level system field. Set it to an empty string ("") to strip any client-supplied system prompt. Not subject to allowParamsOverride.	No
allowInlineSystemMessages	When false, removes any entry in the messages[] array whose role is system. Requires Claude Opus 4.8 or later. Not subject to allowParamsOverride.	No	true
params	Block containing default generation parameters	No
params.temperature	Sampling temperature. Higher values make output more random	No
params.topP	Nucleus sampling parameter. An alternative to temperature sampling	No
params.topK	Only sample from the top K options for each subsequent token	No
params.maxTokens	Maximum number of tokens to generate in the response	No
params.stopSequences	Custom sequences that will cause the model to stop generating	No
params.serviceTier	Service tier for routing. Must be auto or standard_only	No
params.inferenceGeo	Geographic region for inference routing	No
params.cacheControl	Attach an Anthropic ephemeral cache_control marker to the last cacheable request block	No
params.cacheControl.ttl	Cache lifetime. Must be 5m or 1h	No
params.outputConfig	Output configuration options	No
params.outputConfig.effort	Controls the reasoning effort applied to the generation	No
params.outputConfig.format.schema	JSON Schema for structured output format	No
metrics	Opt-in configuration that adds extra API Management labels to the emitted GenAI metrics. The labels are sourced from the Hub-* request headers set by API Management, so they are only populated when the route is governed by API Management	No

The params.* fields map to the Anthropic Messages API request parameters. For accepted values and detailed semantics (for example, valid params.outputConfig.effort levels, params.inferenceGeo regions, and params.stopSequences), see the Anthropic Messages API reference.

Parameter Override Behavior

The middleware supports two modes for handling parameters:

Mode 1: Allow Parameters Override (allowParamsOverride: true)

When enabled (default when no params are set), the middleware acts as a default value provider:

• If a client provides a value for a parameter, the client's value is used.
• If a client doesn't provide a value, the configured default is applied.

spec:
  plugin:
    messages-api:
      model: claude-opus-4-5
      allowParamsOverride: true  # Clients can override
      params:
        temperature: 0.7
        maxTokens: 1000

Mode 2: Force Parameters (allowParamsOverride: false)

When turned off, the middleware enforces the configured values:

• All configured parameters override client values.
• Clients cannot change these settings.
• Useful for strict governance and cost control.

spec:
  plugin:
    messages-api:
      model: claude-opus-4-5
      allowParamsOverride: false  # Enforce configured values
      params:
        temperature: 0.5
        maxTokens: 500

Model Override Behavior

The allowModelOverride setting controls whether clients can specify their own model:

• allowModelOverride: true: Clients can use any model they specify. The configured model acts as a fallback when clients omit the model field.
• allowModelOverride: false (default when model is set): The configured model is always used, even if the client specifies a different one.

spec:
  plugin:
    messages-api:
      model: claude-opus-4-5
      allowModelOverride: true  # Client can request claude-3-7-sonnet-latest instead

System Prompt Governance

The middleware provides two optional and independent controls over system prompts. When set, they are always enforced and independent from the allowParamsOverride field. These controls protect against prompt injection, where a client attempts to override the AI's behaviour by injecting unauthorized instructions into the request.

systemPrompt

The systemPrompt field controls the top-level system property of every Anthropic request:

systemPrompt value	Effect on the request
not set	Client's system field is forwarded unchanged
"You are a helpful assistant."	Client's system field is replaced with the configured value
"" (empty string)	Client's system field is removed entirely

For example, inject a gateway system prompt as follows:

spec:
  plugin:
    messages-api:
      model: claude-opus-4-5
      systemPrompt: "You are a helpful assistant. Answer only questions related to our product."

Note

Traefik Hub wraps the configured value in an Anthropic text content block before forwarding: [{"type": "text", "text": "…"}]. This is valid for all Anthropic models and behaves identically to a plain string system prompt. For instance, if you use systemPrompt: "You are a helpful assistant." in your middleware configuration, Traefik sends "system": [{ "type": "text", "text": "You are a helpful assistant." }]

Alternatively, to strip any client-supplied system prompt, use the following format for the systemPrompt field:

spec:
  plugin:
    messages-api:
      model: claude-opus-4-5
      systemPrompt: ""

allowInlineSystemMessages

Some clients may attempt to pass system instructions as {"role": "system", ...} entries inside the messages[] array. Setting allowInlineSystemMessages: false removes all such entries before the request reaches the upstream model.

note

This control requires Claude Opus 4.8 or later.

For example:

spec:
  plugin:
    messages-api:
      model: claude-opus-4-5
      allowInlineSystemMessages: false

Combining both controls

The two fields compose independently. A fully locked-down configuration that injects a gateway prompt and blocks all client-side system content:

spec:
  plugin:
    messages-api:
      model: claude-opus-4-5
      systemPrompt: "You are a customer support agent for Acme Corp. Do not discuss competitors."
      allowInlineSystemMessages: false

With this configuration:

• The client's top-level system field is replaced with the gateway prompt.
• Any {"role":"system"} entries in the client's messages[] array are stripped.

Prompt Caching

Anthropic supports prompt caching to reduce costs on repeated calls with the same long context. Use params.cacheControl to instruct the middleware to attach an ephemeral cache_control marker to the last eligible request block:

spec:
  plugin:
    messages-api:
      model: claude-opus-4-5
      params:
        cacheControl:
          ttl: 5m   # Supported values: "5m" or "1h"

Caching Costs

Anthropic charges a small creation fee for cached content but reduces costs significantly on cache hits. See Anthropic's prompt caching guide for details on supported models and minimum cacheable token counts.

Request Body Size Limits

The middleware enforces a maximum request body size based on the AI Gateway configuration:

helm upgrade traefik traefik/traefik -n traefik --wait \
  --set hub.aigateway.enabled=true \
  --set hub.aigateway.maxRequestBodySize=10485760  # 10MB

Requests exceeding this size will receive a 413 Request Entity Too Large response.

Streaming Support

The middleware fully supports streaming responses. When a client sets "stream": true in the request, the response is streamed back as server-sent events (SSE).

Metrics and Streaming

Token usage metrics are recorded for streamed responses too. They are extracted from the usage fields in the message_start and message_delta events.

Metrics

The middleware emits OpenTelemetry GenAI metrics when metrics are enabled. When prompt caching is active, the Anthropic cache_creation_input_tokens and cache_read_input_tokens usage fields are added to the input token count reported by gen_ai.client.token.usage.

Working with Other AI Middlewares

When the Messages API middleware is present in the middleware chain, the AI Gateway automatically propagates the Anthropic Messages API format to the other AI middlewares in the chain. Content Guard, LLM Guard, Semantic Cache, and Token Rate Limit all detect the format and adapt their behavior without any additional configuration.

For detailed examples of combining these middlewares, see the Using AI Middlewares with Messages API guide.

Middleware Order Matters

Place the Messages API middleware first in the chain among the AI kind. It sets the request format in the context, which the other AI middlewares rely on to detect the Anthropic format. Guards placed before it cannot detect the format. See Middleware Order for details.

Troubleshooting

Request Entity Too Large

Problem: Receiving 413 Request Entity Too Large errors.

Solution: Increase the AI Gateway's max request body size:

helm upgrade traefik traefik/traefik -n traefik --wait \
  --reset-then-reuse-values \
  --set hub.aigateway.maxRequestBodySize=20971520  # 20MB

Model Override Denied

Problem: Client's model selection is being overridden.

Solution: Enable model override in the middleware:

spec:
  plugin:
    messages-api:
      model: claude-opus-4-5
      allowModelOverride: true  # Allow client to choose model

Authentication Errors

Problem: Receiving 401 Unauthorized responses from Anthropic.

Solution: Ensure the API key secret exists and is referenced correctly:

# Check secret exists
kubectl get secret ai-keys -n apps

# Verify secret content
kubectl get secret ai-keys -n apps -o yaml

Reference in middleware using the URN format:

apiKey: urn:k8s:secret:ai-keys:anthropic-api-key

For workspace API keys or alternative authentication methods, use authToken instead:

authToken: urn:k8s:secret:ai-keys:anthropic-bearer-token

Prompt Caching Not Working

Problem: cache_read_input_tokens is always 0 in metrics.

Solution: Ensure the cacheControl.ttl is set and the request payload exceeds Anthropic's minimum cacheable token count for your model. Verify your model supports prompt caching by checking the Anthropic prompt caching documentation.

Related Content

• Compare the endpoint middleware formats in the Endpoint Format Comparison
• Learn how to use the Messages API with other middlewares in our Messages API guide
• Configure Content Guard to protect sensitive data
• Set up Semantic Cache to reduce costs
• Use LLM Guard for custom content policies
• Monitor token usage with Token Rate Limit
• Route through AWS Bedrock instead of Anthropic's direct API with Bedrock Mantle