Skip to content

How Routing Works

When you send a request, you choose a model (e.g. Claude Opus 4.8, GPT 5.4, Gemini 3.1 Pro) and one of the supported API formats. The router takes care of everything between your request and a healthy backend that can serve that model.

The request lifecycle

graph LR
    A[Your request<br/>model + format] --> B[Normalise<br/>to internal form]
    B --> C[Select best<br/>available backend]
    C --> D{Healthy?}
    D -- yes --> E[Translate &<br/>dispatch]
    D -- no --> C
    E --> F[Response in your<br/>original format]
  1. Normalise — your request is parsed into a single internal representation, regardless of which format you sent.
  2. Select — the router picks the best available backend for the requested model.
  3. Translate & dispatch — parameters are translated into the form the chosen backend needs, then the request is sent.
  4. Return — the response is returned in the format you called with.

Built-in reliability

The router improves availability automatically — you don't configure anything:

Mechanism What happens
Retry A transient error is retried on the same backend.
Failover If a backend is unavailable, the router automatically tries the next available backend for the same model.
Fallback If every backend for the requested model is exhausted, the router can fall back to a configured alternative model.

Because all three formats flow through the same pipeline, failover and fallback apply equally no matter how you call the API.

What you don't have to think about

  • Where the model runs. The router resolves the best backend for you; your code only ever names the model.
  • Per-backend quirks. Parameter translation is handled by the router — see Parameter Support by Model for what each model family accepts.
  • Format translation. Send OpenAI- or Anthropic-style requests; the router maps them to whatever the model needs and maps the response back.