How Do I Fail Over When a Provider Is Slow, Not Just Down?

A provider outage usually returns an error that triggers failover. A slow provider may return no error while the customer waits, so ordinary error-based failover may never run.

AI Gateway provides separate controls for choosing historically faster providers and limiting how long a BYOK provider can take to emit its first token.

These tools address different needs. Sorting prefers providers with better recent time-to-first-token performance. A timeout enforces a maximum wait for the first token on BYOK traffic.

Outcome

Configure TTFT sorting on the order flow, add a first-token deadline for BYOK traffic, and read both decisions back out of the response metadata.

Fast Track

1. Add sort: "ttft" under providerOptions.gateway
2. Add a providerTimeouts.byok entry for your BYOK provider
3. Run pnpm latency-guard and read the sort metadata

Hands-on exercise

We cannot schedule a slow provider for practice. Configure both tools, verify the live sorting decision, then inspect a previously captured timeout trace.

Requirements:

• A scripts/latency-guard.ts that sends the order-confirmation prompt to anthropic/claude-sonnet-4.6
• sort: "ttft": rank providers by measured time-to-first-token
• A providerTimeouts.byok deadline for your Anthropic BYOK key, sized to your latency budget
• Print who served the request, how fast, and the full routing.sort metadata

The timeout takes effect only when the request uses an Anthropic BYOK credential. Without one, the script still demonstrates TTFT sorting, but the provider deadline does not apply. Bring Your Own Keys covers the credential setup.

Wrapping the AI call in Promise.race with a timer stops the client request but does not retry another provider, so the customer receives an error. A Gateway timeout can abort the provider attempt and continue to the next eligible provider.

Try It

pnpm latency-guard

The order confirms, and then the metadata shows its work:

One Camarón Cañón and one agua fresca — that'll be $8.50. Napkin situation: manageable.
---
Served by:     anthropic
Response time: 612ms
{
  "option": "ttft",
  "executionOrder": ["anthropic", "vertex", "bedrock"],
  "metrics": {
    "anthropic": { "ttft": 389 },
    "vertex": { "ttft": 501 },
    "bedrock": { "ttft": 548 }
  }
}

executionOrder shows the live sorting decision based on recent measurements. Your providers, values, and order will differ from this example.

A healthy request will not show the timeout firing. The following block is a previously captured and sanitized example trace:

"providerAttempts": [
  { "provider": "anthropic", "success": false, "error": "PROVIDER_TIMEOUT",
    "providerTimeout": true, "configuredTimeoutMs": 10000 },
  { "provider": "vertex", "success": true }
]

The provider exceeded the deadline, so the Gateway aborted that attempt and continued to the next provider. The attempt metadata records the timeout.

Two issues you may encounter:

Your timeout never seems to apply. Check credentialType in the attempt metadata. If it says system, the request did not use your BYOK key. Provider timeouts apply only to BYOK traffic.

A reasoning model keeps crossing the deadline. The clock stops at the first token, including a thinking token. Choose deadlines per model and feature rather than copying one value everywhere.

Commit

git commit -m "feat(latency): sort providers by ttft and add a byok first-token deadline"

Done-When

• sort: "ttft" configured, and the routing.sort metadata shows a measured executionOrder
• A providerTimeouts.byok deadline set for your BYOK provider
• You can say what happens at the deadline: abort, fall to the next provider, PROVIDER_TIMEOUT in the trace
• You can explain why the timeout didn't apply to a credentialType: "system" request

Solution

import { generateText } from "ai";
import { MENU } from "../lib/menu";
 
const result = await generateText({
  model: "anthropic/claude-sonnet-4.6",
  prompt:
    "Confirm this order back with a total: one Camarón Cañón and one " +
    `agua fresca.\n\n${MENU}`,
  providerOptions: {
    gateway: {
      sort: "ttft", // works with system credentials and BYOK
      // Hard first-token deadlines apply only to BYOK credentials.
      // Replace the provider and value to match your own key and latency budget.
      providerTimeouts: {
        byok: { anthropic: 10_000 },
      },
    },
  },
});
 
const routing = result.finalStep.providerMetadata?.gateway?.routing as any;
const attempt = routing?.modelAttempts?.[0]?.providerAttempts?.find(
  (a: any) => a.success
);
 
console.log(result.text.trim());
console.log("---");
console.log(`Served by:     ${routing?.finalProvider}`);
console.log(`Response time: ${Math.round(attempt?.responseTimeMs ?? 0)}ms`);
console.log(JSON.stringify(routing?.sort, null, 2));

TTFT sorting chooses a starting provider. The BYOK deadline defines when the Gateway should stop waiting and continue.

Related Questions

• How do I survive a provider outage?
• How do I pin routing to a specific provider?
• How do I route to the cheapest provider automatically?