Skip to main content

Using AI Middlewares with Messages API

This guide explains how to use Traefik Hub's AI Gateway middlewares with the Anthropic Messages API format.

When messages-api is in the chain, followed by Content Guard, LLM Guard, Semantic Cache, and Token Rate Limit, these middlewares automatically detect the Messages API format and adapt their behavior accordingly. No additional clientRequestFormat configuration is needed.

Also applies to Bedrock Mantle

The Bedrock Mantle middleware produces the same Anthropic Messages API format as messages-api. Everywhere this guide references messages-api in a middleware chain, you can substitute bedrock-mantle. The auto-detection and ordering behavior described below apply identically.

Overview​

The Anthropic Messages API uses a different request/response format compared to Chat Completions and Responses API. For a side-by-side comparison of the three endpoint formats, see Endpoint Format Comparison in the AI Gateway overview.

Prerequisites​

Before starting, ensure you have:

  1. AI Gateway enabled in your Traefik Hub installation:

    helm upgrade traefik traefik/traefik -n traefik --wait \
      --reset-then-reuse-values \
      --set hub.aigateway.enabled=true

  2. An Anthropic API key stored in a Kubernetes Secret:

    apiVersion: v1
    kind: Secret
    metadata:
      name: ai-keys
      namespace: apps
    type: Opaque
    stringData:
      anthropic-api-key: sk-ant-XXXXX

  3. An ExternalName service pointing to Anthropic:

    apiVersion: v1
    kind: Service
    metadata:
      name: anthropic
      namespace: apps
    spec:
      type: ExternalName
      externalName: api.anthropic.com
      ports:
        - port: 443

Auto-Detection of Messages API Format​

The Messages API middleware marks the request format in the request context. The guard and cache middlewares (Content Guard, LLM Guard, Semantic Cache, and Token Rate Limit) read that context and adapt their field extraction and response shaping accordingly. No clientRequestFormat configuration is needed on them. This is the same mechanism used by the Chat Completion and Responses API governance middlewares.

For this to work, the Messages API middleware must come first among the AI middlewares in the chain so it sets the format before the other middlewares run:

middlewares:
  - name: messagesapi              # sets format in context (must be first among other AI middleware)
  - name: content-guard-messages   # auto-detects Messages API format
  - name: llm-guard-messages       # auto-detects Messages API format
  - name: semantic-cache-messages  # auto-detects Messages API format

Using Content Guard​

The Content Guard middleware detects and masks PII in requests and responses. When the Messages API middleware is in the chain, it automatically extracts text from relevant fields in this format.

Configuration Example​

apiVersion: traefik.io/v1alpha1
kind: Middleware
metadata:
  name: content-guard-messages
  namespace: apps
spec:
  plugin:
    content-guard:
      engine:
        presidio:
          host: http://presidio-analyzer.presidio.svc.cluster.local:5002
          language: en
      request:
        rules:
          - entities:
              - EMAIL_ADDRESS
              - PHONE_NUMBER
              - CREDIT_CARD
            block: false
            mask:
              char: "*"
              unmaskFromLeft: 2
              unmaskFromRight: 2
      response:
        rules:
          - entities:
              - PERSON
              - EMAIL_ADDRESS
            block: false
            mask:
              char: "*"

Testing​

Send a request with PII to verify the middleware masks it before forwarding:

curl -X POST https://ai.localhost/v1/messages \
  -H "Content-Type: application/json" \
  -d '{
    "model": "claude-opus-4-5",
    "messages": [
      {"role": "user", "content": "My email is [email protected] and my phone is 555-123-4567"}
    ]
  }'

The PII is masked in the request before it reaches Anthropic.

Using LLM Guard​

The LLM Guard middleware analyzes content using an external guard service or LLM. When the Messages API middleware is in the chain, it automatically reads from relevant fields and formats deny responses in the Anthropic Messages API shape.

Configuration Example​

apiVersion: traefik.io/v1alpha1
kind: Middleware
metadata:
  name: llm-guard-messages
  namespace: apps
spec:
  plugin:
    llm-guard:
      endpoint: http://sentiment-analyzer.apps.svc.cluster.local:5000/predict
      format:
        custom: {}
      clientConfig:
        timeout: 30s
        headers:
          Content-Type: application/json
      request:
        template: '{"text":"{{ (index .messages 0).content }}"}'
        blockConditions:
          - condition: 'JSONGt(".predictions[0].NEGATIVE", 0.6)'
            reason: 'Negative sentiment detected'

Testing​

Test with negative content:

curl -k -X POST https://ai.localhost/v1/messages \
  -H "Content-Type: application/json" \
  -d '{
    "model": "claude-opus-4-5",
    "messages": [{"role": "user", "content": "I hate everything"}]
  }'

Requests exceeding your threshold are blocked with a properly formatted Anthropic error response.

Test with positive content to confirm it passes through:

curl -k -X POST https://ai.localhost/v1/messages \
  -H "Content-Type: application/json" \
  -d '{
    "model": "claude-opus-4-5",
    "messages": [{"role": "user", "content": "I love learning new things"}]
  }'

Using Semantic Cache​

The Semantic Cache middleware caches responses based on semantic similarity of the request content. When the Messages API middleware is in the chain, it automatically extracts conversation content from the format for cache key generation.

Configuration Example​

apiVersion: traefik.io/v1alpha1
kind: Middleware
metadata:
  name: semantic-cache-messages
  namespace: apps
spec:
  plugin:
    semantic-cache:
      vectorDB:
        redis:
          endpoints:
            - redis-stack.apps.svc.cluster.local:6379
          database: 1
          collectionName: ai_messages_cache
          maxDistance: 0.6
          ttl: 3600  # 1 hour
      vectorizer:
        openai:
          model: text-embedding-3-small
          token: urn:k8s:secret:ai-keys:openai-token
          dimensions: 1536
      readOnly: false
      allowBypass: true

Testing​

Make the same request twice to confirm caching works:

First request (cache miss):

curl -ik -X POST https://ai.localhost/v1/messages \
  -H "Content-Type: application/json" \
  -d '{"model":"claude-opus-4-5","messages":[{"role":"user","content":"What is the capital of France?"}]}'

The response includes X-Cache-Status: Miss.

Second request (cache hit):

curl -ik -X POST https://ai.localhost/v1/messages \
  -H "Content-Type: application/json" \
  -d '{"model":"claude-opus-4-5","messages":[{"role":"user","content":"What is the capital of France?"}]}'

The response includes X-Cache-Status: Hit and is served without consuming Anthropic tokens.

Complete Integration Example​

Here's a complete example combining all middlewares for a production Anthropic Messages API route:

Downstream authentication

This example governs the upstream provider call but does not authenticate the clients calling the gateway. For a production route, add a downstream authentication middleware (for example, API key or JWT) so only authorized clients can reach the endpoint.

---
apiVersion: traefik.io/v1alpha1
kind: Middleware
metadata:
  name: content-guard-messages
  namespace: apps
spec:
  plugin:
    content-guard:
      engine:
        presidio:
          host: http://presidio-analyzer.presidio.svc.cluster.local:5002
      request:
        rules:
          - entities:
              - EMAIL_ADDRESS
              - PHONE_NUMBER
              - SSN
            mask:
              char: "*"
---
apiVersion: traefik.io/v1alpha1
kind: Middleware
metadata:
  name: messagesapi
  namespace: apps
spec:
  plugin:
    messages-api:
      apiKey: urn:k8s:secret:ai-keys:anthropic-api-key
      model: claude-opus-4-5
      allowModelOverride: false
      params:
        temperature: 0.7
        maxTokens: 1024
        cacheControl:
          ttl: 5m
---
apiVersion: traefik.io/v1alpha1
kind: Middleware
metadata:
  name: semantic-cache-messages
  namespace: apps
spec:
  plugin:
    semantic-cache:
      vectorDB:
        redis:
          endpoints:
            - redis-stack.apps.svc.cluster.local:6379
          database: 1
          collectionName: ai_messages_cache
      vectorizer:
        openai:
          model: text-embedding-3-small
          token: urn:k8s:secret:ai-keys:openai-token

Middleware Order​

The order of middlewares matters. The Messages API middleware must come first in the chain among other AI middleware so it can mark the request format in the context before the guard and cache middlewares run:

  1. Messages API: Validates the request, applies governance (model/param enforcement), injects authentication headers, and marks the format context for the downstream middlewares.
  2. Content Guard / LLM Guard: Reads the format from the context, then masks or blocks content before the request is forwarded upstream.
  3. Semantic Cache: Caches the final governed response.

If you can't guarantee the Messages API middleware runs first in the chain, set clientRequestFormat: messagesAPI explicitly on Content Guard, LLM Guard, Semantic Cache, or Token Rate Limit instead of relying on auto-detection.

Streaming Support​

Content Guard, LLM Guard, and Semantic Cache do not support true streaming mode. When "stream": true is set, these middlewares buffer the complete response, process it, then send it to the client as a single chunk.

For real-time streaming, use a separate route with only the messages-api middleware:

routes:
  # Non-streaming with full middleware stack
  - kind: Rule
    match: Host(`ai.localhost`) && Path(`/v1/messages`)
    middlewares:
      - name: messagesapi
      - name: content-guard-messages
      - name: semantic-cache-messages

  # Streaming with governance only
  - kind: Rule
    match: Host(`ai.localhost`) && Path(`/v1/messages/stream`)
    middlewares:
      - name: messagesapi

Related Content​