AI Gateway

AI Gateway is a service-based solution integrated within Traefik Hub, designed to simplify the management and integration of multiple Large Language Model (LLM) providers.

Configuration Example

• Define your AIService

Anthropic

apiVersion: hub.traefik.io/v1alpha1
kind: AIService
metadata:
  name: ai-anthropic
  namespace: traefik
spec:
  anthropic:
    token: "YOUR_ANTHROPIC_TOKEN"
    model: "anthropic-model-name"

AzureOpenAI

apiVersion: hub.traefik.io/v1alpha1
kind: AIService
metadata:
  name: ai-azureopenai
  namespace: traefik
spec:
  azureOpenai:
    apiKey: "YOUR_AZURE_OPENAI_API_KEY"
    baseUrl: "https://your-azure-openai-endpoint.azure.com/"
    deploymentName: your-deployment-name
    model: "azureopenai-model-name"

Bedrock

apiVersion: hub.traefik.io/v1alpha1
kind: AIService
metadata:
  name: ai-bedrock
  namespace: traefik
spec:
  bedrock:
    model: "bedrock-model-name"
    systemMessage: true

info

When using Bedrock, Traefik Hub will automatically detect the necessary AWS credentials needed from your environmental variables.

Cohere

apiVersion: hub.traefik.io/v1alpha1
kind: AIService
metadata:
  name: ai-cohere
  namespace: traefik
spec:
  cohere:
    token: "YOUR_COHERE_TOKEN"
    model: "cohere-model-name"

Gemini

apiVersion: hub.traefik.io/v1alpha1
kind: AIService
metadata:
  name: ai-gemini
  namespace: traefik
spec:
  gemini:
    apiKey: "YOUR_GEMINI_API_KEY"
    model: "gemini-model-name"

Mistral

apiVersion: hub.traefik.io/v1alpha1
kind: AIService
metadata:
  name: ai-mistral
  namespace: traefik
spec:
  mistral:
    apiKey: "YOUR_MISTRAL_API_KEY"
    model: "mistral-model-name"

Ollama

apiVersion: hub.traefik.io/v1alpha1
kind: AIService
metadata:
  name: ai-ollama
  namespace: traefik
spec:
  ollama:
    baseUrl: "http://ollama.ollama.svc.cluster.local:11434"
    model: "llama3.2"

OpenAI

apiVersion: hub.traefik.io/v1alpha1
kind: AIService
metadata:
  name: ai-openai
  namespace: traefik
spec:
  openai:
    token: "YOUR_OPENAI_TOKEN"
    model: "o1-preview"

• Attach your preferred AIService to an ingressRoute as a TraefikService

IngressRoute

apiVersion: traefik.io/v1alpha1
kind: IngressRoute
metadata:
  name: ai-test
  namespace: traefik
spec:
  routes:
    - kind: Rule
      match: Host(`ai.localhost`)
      services:
        - kind: TraefikService
          name: traefik-ai-openai@ai-gateway-service

IngressRoute with API Management Enabled

apiVersion: traefik.io/v1alpha1
kind: IngressRoute
metadata:
  annotations:
    hub.traefik.io/api: "ai@traefik"
  name: ai-test
  namespace: traefik
spec:
  routes:
    - kind: Rule
      match: Host(`ai.localhost`)
      services:
        - kind: TraefikService
          name: traefik-ai-openai@ai-gateway-service

Configuration options

Below is a table outlining the available configuration options/fields for the AIService resource:

AI Providers	Field	Description	Required	Possible Values / Examples
OpenAI, Cohere, Anthropic	token	Authentication token for AI providers	Yes	A string provided by the AI provider (e.g., sk-XXXX)
Gemini, AzureOpenAI, Mistral	apiKey	API Key for AI providers	Yes	A string key provided by the AI provider (e.g., APIKEY123)
Ollama, AzureOpenAI	baseUrl	Base URL for AI providers that require it	Yes	A valid URL (e.g., https://api.ollama.com)
AzureOpenAI	deploymentName	Deployment name for AzureOpenAI	Yes	A string specifying the deployment (e.g., deployment1)
All Providers	model	Specifies the AI model to use	Yes	Model name as a string (e.g., gpt-4)
Bedrock	region	AWS region for Bedrock (auto-detected if AWS is the cloud provider)	Yes	AWS region codes (e.g., us-west-2)
Bedrock	systemMessage	Enables system message for Bedrock	No	A boolean (e.g., true )
All Providers	params.frequencyPenalty	Penalty for frequency in model outputs	No	Float value (e.g., 0.5)
All Providers	params.maxTokens	Maximum number of tokens per request	No	Integer value (e.g., 1500)
All Providers	params.presencePenalty	Penalty for presence in model outputs	No	Float value (e.g., 0.3)
All Providers	params.temperature	Controls randomness in responses	No	Float range typically between 0.0 and 2.0 (e.g., 0.7)
All Providers	params.topP	Controls diversity via nucleus sampling	No	Float range typically between 0.0 and 1.0 (e.g., 0.9)

Related content

• Learn more about the AI Gateway feature in its dedicated section.