Configuration

Semantic Router uses one canonical YAML document across the CLI, Dashboard, Helm, and Operator. The top-level structure is:

version:
listeners:
providers:
routing:
entrypoints:
recipes:
global:

Most deployments begin with version, listeners, providers, and one top-level routing profile. Add entrypoints and recipes when one deployment needs several isolated policies. Add global settings only for shared services or runtime behavior that differs from the built-in defaults.

What belongs where

Section	Owns
version	Canonical schema version. Use v0.3.
listeners	Public Router listeners and timeouts.
providers	Logical provider models, physical backend endpoints, pricing, capabilities, and defaults.
routing	The default recipe: model cards, signals, projections, decisions, strategy, algorithms, and route plugins.
entrypoints	Public virtual model aliases mapped to named recipes.
recipes	Additional isolated routing profiles that share providers and global infrastructure.
global	Router services, stores, integrations, observability, learning, and router-owned model assets.

Keep these boundaries clear:

• signals detect facts;
• projections combine evidence;
• decisions define eligibility and route policy;
• algorithms choose or coordinate candidate models;
• plugins add behavior at route-specific hook points; and
• providers bind logical model names to inference endpoints.

The Routing Pipeline explains the design. Capability pages under Capabilities document each signal, projection, decision, algorithm, plugin, and global block.

Capability catalog

Use this catalog to choose a reusable building block, then open its guide for configuration details. The inventory comes from config/fragments/; each one-line goal comes from the matching guide's Overview. The documentation build regenerates this block and fails if the checked-in catalog has drifted.

Signals

Family and type	Use it to	Reusable fragment	Guide
authz — heuristic signal	authz turns identity and policy bindings into reusable routing inputs under routing.signals.role_bindings.	config/fragments/signal/authz/	Guide
classifier — learned signal	classifier exposes reusable label scores from a local native sequence classifier or a configured external LLM.	config/fragments/signal/classifier/	Guide
complexity — learned signal	complexity estimates whether a request is easy, medium, or hard by comparing it with configured example sets.	config/fragments/signal/complexity/	Guide
context — heuristic signal	context detects requests that need a larger effective context window.	config/fragments/signal/context/	Guide
conversation — heuristic signal	conversation routes on the structure of a chat, such as message count, developer instructions, available tools, or an active tool loop.	config/fragments/signal/conversation/	Guide
domain — learned signal	domain classifies the request topic family.	config/fragments/signal/domain/	Guide
embedding — learned signal	embedding matches requests by semantic similarity to representative examples.	config/fragments/signal/embedding/	Guide
event — heuristic signal	event routes structured event-like requests by event type, severity, urgency, or domain-specific action code.	config/fragments/signal/event/	Guide
fact-check — learned signal	fact-check decides whether a prompt should be treated as evidence-sensitive traffic.	config/fragments/signal/fact-check/	Guide
jailbreak — learned signal	jailbreak detects prompt-injection and jailbreak attempts before the Router commits to a route.	config/fragments/signal/jailbreak/	Guide
kb — learned signal	kb binds routing signals to the output of a named knowledge base instance.	config/fragments/signal/kb/	Guide
keyword — heuristic signal	keyword matches explicit words and phrases in the request.	config/fragments/signal/keyword/	Guide
language — heuristic signal	language detects the request language and exposes it as a routing signal.	config/fragments/signal/language/	Guide
metadata — heuristic signal	metadata matches bounded string values supplied by the caller in request metadata.	config/fragments/signal/metadata/	Guide
modality — learned signal	modality detects whether a request should stay in text generation, switch into image generation, or support both.	config/fragments/signal/modality/	Guide
pii — learned signal	pii detects sensitive personal data in requests.	config/fragments/signal/pii/	Guide
preference — learned signal	preference infers response-style preferences from examples and classifier settings.	config/fragments/signal/preference/	Guide
reask — learned signal	reask detects when the current user turn semantically repeats recent user turns in the same conversation.	config/fragments/signal/reask/	Guide
structure — heuristic signal	structure detects request-shape facts such as many explicit questions, ordered workflow markers, or dense constraint phrasing.	config/fragments/signal/structure/	Guide
user-feedback — learned signal	user-feedback detects correction, dissatisfaction, or escalation feedback from the conversation.	config/fragments/signal/user-feedback/	Guide

Selection algorithms

Family and type	Use it to	Reusable fragment	Guide
automix — selection algorithm	automix is an experimental selector that ranks candidate models by configured quality and cost plus internal verification and escalation estimates.	config/fragments/algorithm/selection/automix.yaml	Guide
hybrid — selection algorithm	hybrid combines Elo ratings, Router-DC description similarity, AutoMix's one-model value estimate, and cost into one weighted candidate score.	config/fragments/algorithm/selection/hybrid.yaml	Guide
kmeans — selection algorithm	kmeans sends a request to the model assigned to its nearest learned cluster.	config/fragments/algorithm/selection/kmeans.yaml	Guide
knn — selection algorithm	knn chooses a candidate from the models that performed well on the most similar recorded requests.	config/fragments/algorithm/selection/knn.yaml	Guide
latency-aware — selection algorithm	latency_aware ranks eligible candidates using observed TTFT and TPOT percentiles and selects the lowest relative-latency score.	config/fragments/algorithm/selection/latency-aware.yaml	Guide
mlp — selection algorithm	mlp runs a trained neural classifier on CPU to map a request to a candidate model.	config/fragments/algorithm/selection/mlp.yaml	Guide
multi-factor — selection algorithm	multi_factor ranks candidates by a configurable combination of quality, latency, cost, and load, then rejects any candidate that violates a hard limit.	config/fragments/algorithm/selection/multi-factor.yaml	Guide
prompt — selection algorithm	prompt uses a concrete helper model to select exactly one model from the matched decision's modelRefs.	config/fragments/algorithm/selection/prompt.yaml	Guide
router-dc — selection algorithm	router_dc embeds the request and each model description, then selects the candidate with the strongest semantic similarity.	config/fragments/algorithm/selection/router-dc.yaml	Guide
static — selection algorithm	static provides deterministic model choice without metrics or learned state.	config/fragments/algorithm/selection/static.yaml	Guide
svm — selection algorithm	svm uses a trained linear or RBF support-vector classifier to map request features to a candidate model.	config/fragments/algorithm/selection/svm.yaml	Guide

Looper algorithms

Family and type	Use it to	Reusable fragment	Guide
confidence — looper algorithm	confidence tries candidate models in order and stops when response confidence reaches a configured threshold.	config/fragments/algorithm/looper/confidence.yaml	Guide
fusion — looper algorithm	fusion asks several models to analyze a request and a judge model to synthesize one final answer.	config/fragments/algorithm/looper/fusion.yaml	Guide
ratings — looper algorithm	ratings calls every candidate model and returns one OpenAI-compatible choice per successful model. max_concurrent limits parallel work; it does not limit the total number of candidates executed.	config/fragments/algorithm/looper/ratings.yaml	Guide
remom — looper algorithm	remom runs several candidate models across bounded rounds and synthesizes their responses into one answer.	config/fragments/algorithm/looper/remom.yaml	Guide
workflows — looper algorithm	workflows runs a bounded, multi-step Router Flow behind one OpenAI-compatible model name.	config/fragments/algorithm/looper/workflows.yaml	Guide

Plugins and bundles

Family and type	Use it to	Reusable fragment	Guide
content-safety — plugin bundle	Content Safety combines supported route-local safety plugins into one reusable policy.	config/fragments/plugin/content-safety/	Guide
context-compression — route plugin	context_compression is a route-local request plugin that reduces large tool/function outputs before the selected provider receives the request.	config/fragments/plugin/context-compression/	Guide
fast-response — route plugin	fast_response is a route-local plugin that returns a deterministic fallback message immediately.	config/fragments/plugin/fast-response/	Guide
hallucination — route plugin	hallucination is a route-local plugin for fact-checking and response-quality screening after the decision already matched.	config/fragments/plugin/hallucination/	Guide
header-mutation — route plugin	header_mutation is a route-local plugin for adding, updating, or deleting downstream headers.	config/fragments/plugin/header-mutation/	Guide
image-gen — route plugin	image_gen is a route-local plugin for handing a matched route off to an image-generation backend.	config/fragments/plugin/image-gen/	Guide
memory — route plugin	memory is a route-local plugin for retrieving and storing conversation memory.	config/fragments/plugin/memory/	Guide
rag — route plugin	rag retrieves external context for a matched route before generation.	config/fragments/plugin/rag/	Guide
request-params — route plugin	request_params is a route-local plugin that validates and trims OpenAI Chat Completions request bodies before they are forwarded to backends.	config/fragments/plugin/request-params/	Guide
response-cache — route plugin	response_cache is the route-local plugin for reusing exact or semantically compatible prior responses.	config/fragments/plugin/response-cache/	Guide
response-jailbreak — route plugin	response_jailbreak is a route-local plugin for screening the model response before it is returned.	config/fragments/plugin/response-jailbreak/	Guide
router-replay — route plugin	router_replay is a route-local plugin for overriding replay/debug capture on one route.	config/fragments/plugin/router-replay/	Guide
system-prompt — route plugin	system_prompt is a route-local plugin for inserting or modifying the system prompt on matched traffic.	config/fragments/plugin/system-prompt/	Guide
tool-selection — route plugin	tool_selection is a decision plugin that controls how tools are chosen for a matched route.	config/fragments/plugin/tool-selection/	Guide
tools — route plugin	tools is a route-local plugin for tool filtering and semantic tool selection.	config/fragments/plugin/tools/	Guide

For route-local RAG with backend: external_api, custom request templates are parsed as a typed, non-null JSON object or array before placeholder substitution. Supported request formats are pinecone, weaviate, elasticsearch, and custom; custom templates and all hybrid children are validated when the configuration loads. The lowercase ${user_content}, ${top_k}, and ${threshold} tokens are reserved for runtime substitution. Other braced lowercase tokens are rejected before environment expansion; intentional environment references in templates use uppercase names such as ${RAG_TENANT}. Successful response bodies use an exact 16 MiB default limit; set the positive backend_config.max_response_body_bytes field to override that limit up to 64 MiB. The router accepts a response at the configured byte count and rejects one byte more before JSON decoding, so it never accepts a valid truncated prefix.

Minimal example

version: v0.3

listeners:
  - name: http-8899
    address: 0.0.0.0
    port: 8899
    timeout: 300s

providers:
  defaults:
    default_model: local/general
  models:
    - name: local/general
      provider_model_id: my-served-model
      backend_refs:
        - name: primary
          endpoint: host.docker.internal:8000
          protocol: http

routing:
  strategy: priority
  modelCards:
    - name: local/general
      modality: text
      capabilities: [chat]
  signals:
    keywords:
      - name: needs_explanation
        operator: OR
        keywords: ["explain", "walk me through"]
  decisions:
    - name: explanatory_answer
      description: Prefer an explanatory answer when the request asks for one.
      priority: 100
      rules:
        operator: AND
        conditions:
          - type: keyword
            name: needs_explanation
      modelRefs:
        - model: local/general

global:
  services:
    observability:
      metrics:
        enabled: true

Requests using an automatic model alias enter the default routing profile. A concrete provider model name is a direct pass-through request and bypasses recipe signals, decisions, route plugins, cache, learning, and session routing.

Validate and serve

vllm-sr validate --config config.yaml
vllm-sr serve --config config.yaml

Validation catches schema errors, unresolved references, incompatible recipe boundaries, invalid provider bindings, and unsupported plugin or algorithm settings before the Router starts.

Environment references and secrets

Keep credentials outside the YAML file:

api_key: ${MODEL_API_KEY}

Supported string substitutions are:

• ${VAR} and $VAR;
• ${VAR:-default} when VAR is unset or empty;
• ${VAR-default} when VAR is unset; and
• $$ for a literal $.

For a custom Recipe, authorize required host variables explicitly with --recipe-env NAME. Kubernetes deployments place sensitive environment values in Secrets rather than ConfigMaps or Helm values. See Security Hardening.

Entrypoints and recipes

An entrypoint maps one or more public model aliases to a recipe. A recipe owns its signal, projection, decision, algorithm, plugin, cache, replay, learning, and routing state. Providers, stores, and router-owned classifier assets may be shared without allowing policy state to cross recipe boundaries.

In the schema, entrypoints[].model_names lists the public aliases, entrypoints[].recipe selects a named recipe, and recipes[].routing contains that recipe's policy.

If no decision matches, the recipe uses providers.defaults.default_model. The virtual entrypoint name never reaches a backend.

See Models, Entrypoints, and Serving for built-in virtual models, CLI serving, backend binding, forking, packaging, and migration. See Virtual Models for the complete schema.

Configuration workflows

The canonical document can be authored or applied through several interfaces:

• local CLI and YAML;
• Dashboard setup and visual routing tools;
• Helm or vllm-sr serve --target k8s;
• the Kubernetes Operator; and
• the routing DSL.

Configuration Workflows explains which interface owns which part of the document and how to avoid competing sources of truth.

Reference sources

• config/config.yaml is the exhaustive canonical example.
• config/fragments/ contains reusable signal, decision, algorithm, and plugin fragments.
• Providers and routing tutorials describe shared runtime configuration.
• Unified Config Contract v0.3 records the design behind the current contract.

Avoid copying the exhaustive example as an application config. Start with the smallest document that describes the deployment, then add only the capabilities and services it uses.