Routing overview

Outgate selects an upstream from a single gateway endpoint.

Routing is how Outgate decides where each request should go once it enters the gateway.

Instead of your application choosing a specific provider, you send every request to a single endpoint. Outgate evaluates the request and forwards it to the most appropriate upstream based on the routing logic you define.

This keeps your clients simple while giving you control over reliability, cost, and performance in one place.

How routing works

A router is a provider that points to other providers.

A router behaves like a provider, but instead of pointing to a single upstream, it points to a set of providers.

When a request reaches the gateway, the router evaluates it and selects one upstream. That decision happens inline as part of the request. There is no separate service or delayed process. Every request is handled immediately and consistently.

From your application perspective, nothing changes. You still call one endpoint. From the system perspective, each request becomes a decision about where it should go.

Router types

Outgate provides three routing strategies.

Outgate provides three routing strategies. Each one represents a different way of making the upstream selection decision.

These strategies share the same underlying model. The only difference is how the upstream is selected.

Failover routing

Keep requests successful when providers fail.

Failover routing focuses on keeping requests successful.

You define an ordered list of providers. The router sends the request to the first provider. If that request fails, the router automatically moves to the next one, and continues until a provider succeeds or all options are exhausted.

This creates a natural primary and fallback structure. You can tune how quickly failover happens by adjusting timeouts, and you can change behavior across providers by overriding models for fallback paths.

Failover routing is usually the first step beyond a single provider. It allows you to absorb outages and rate limits without changing application logic.

Weighted routing

Distribute traffic by relative weights.

Weighted routing focuses on distribution.

Each provider is assigned a weight, and requests are distributed across providers based on those proportions. Over time, the traffic you observe will reflect the weights you configured.

This is particularly useful when introducing a new model. You can start by sending a small portion of traffic to it, observe behavior, and gradually increase its share.

Weighted routing does not retry across providers. Each request is handled by a single upstream, which keeps latency predictable and makes experimentation easier to reason about.

Smart routing

Select a provider dynamically for each request.

Smart routing focuses on selecting the best option for each request.

Instead of using fixed rules, the router evaluates available models and chooses the one that best fits the request. This decision considers factors such as quality, speed, and cost, along with preferences you configure.

Each upstream contributes one or more candidate models. These candidates are evaluated at runtime, and the selection can vary from one request to another.

Smart routing also integrates with guardrails. Requests can be evaluated for safety before routing, and additional adjustments such as tool filtering can be applied as part of the same decision step.

Designing your routing approach

Start simple and layer in more advanced behavior over time.

Most teams adopt routing incrementally.

• Start with a single provider to validate integration.
• Introduce failover to improve reliability.
• Use weighted routing to test and shift traffic.
• Move to smart routing when optimization becomes important.

The important part is that routing evolves without requiring changes to your application. You continue to call the same endpoint while the behavior behind it becomes more sophisticated.

Working with upstreams

Configure each connected provider independently.

A router is defined by the providers it connects to. Each upstream can have its own configuration, which influences how requests behave.

You might use a high quality model as your primary provider and fall back to a faster or cheaper model when needed. You might assign different weights based on cost constraints or expected latency.

In smart routing, these upstream settings influence how candidates are evaluated. Preferences help guide selection, but the final decision remains dynamic.

Limits and control

Enforce usage limits before requests are sent upstream.

Routing happens at the gateway, which makes it a natural place to enforce limits.

You can apply request or token limits to a router, ensuring that usage stays within defined boundaries before requests reach upstream providers. This simplifies cost control and centralizes enforcement across all applications using the router.

Operational considerations

Routing decisions are in the request path, so tune them deliberately.

Because routing decisions are part of the request path, they directly affect performance.

Short failover chains tend to produce better latency, since each additional attempt adds delay when failures occur. Timeout configuration plays an important role in how quickly the system reacts to unhealthy providers.

Weighted routing should be used intentionally, especially when introducing new models into production. Observability is essential as routing becomes more complex, since it allows you to understand which providers are being used and how they perform over time.

Current limitations

Routers are intentionally simple and predictable.

Routers are designed to be predictable and easy to reason about, which means some advanced compositions are not yet supported.

Routers cannot currently be used as upstreams for other routers, so nested routing is not available. Weighted routing does not include automatic fallback behavior, and smart routing depends on available model metadata to make effective decisions.

Bringing it together

Routing makes provider choice a platform concern instead of application logic.

Once a router is in place, your system becomes simpler to operate.

Your application sends every request to the same endpoint. The gateway evaluates the request, selects an upstream, and returns the response. Routing becomes a property of the platform rather than something each application needs to implement.

This allows teams to move faster while maintaining control over how requests are handled.