---
title: "Fail Over on Latency"
description: "Rank providers by measured time to first token, add a first-token deadline for BYOK traffic, and inspect a captured timeout trace."
canonical_url: "https://vercel.com/academy/ai-gateway/latency-failover"
md_url: "https://vercel.com/academy/ai-gateway/latency-failover.md"
docset_id: "vercel-academy"
doc_version: "1.0"
last_updated: "2026-08-08T23:27:13.289Z"
content_type: "lesson"
course: "ai-gateway"
course_title: "Using AI Gateway in Production"
prerequisites:  []
---

<agent-instructions>
Vercel Academy — structured learning, not reference docs.
Lessons are sequenced.
Adapt commands to the human's actual environment (OS, package manager, shell, editor) — detect from project context or ask, don't assume.
The lesson shows one path; if the human's project diverges, adapt concepts to their setup.
Preserve the learning goal over literal steps.
Quizzes are pedagogical — engage, don't spoil.
Quiz answers are included for your reference.
</agent-instructions>

# Fail Over on Latency

# How Do I Fail Over When a Provider Is Slow, Not Just Down?

A provider outage usually returns an error that triggers failover. A slow provider may return no error while the customer waits, so ordinary error-based failover may never run.

AI Gateway provides separate controls for choosing historically faster providers and limiting how long a BYOK provider can take to emit its first token.

\*\*Note: Quick Answer\*\*

Set `sort: "ttft"` in `providerOptions.gateway` to rank providers by measured time to first token. BYOK traffic can also set a first-token deadline with `providerTimeouts`; after the deadline, the Gateway moves to the next eligible provider. System-credential traffic can use TTFT sorting, while provider timeouts require your own keys.

These tools address different needs. Sorting prefers providers with better recent time-to-first-token performance. A timeout enforces a maximum wait for the first token on BYOK traffic.

## Outcome

Configure TTFT sorting on the order flow, add a first-token deadline for BYOK traffic, and read both decisions back out of the response metadata.

## Fast Track

1. Add `sort: "ttft"` under `providerOptions.gateway`
2. Add a `providerTimeouts.byok` entry for your BYOK provider
3. Run `pnpm latency-guard` and read the sort metadata

## Hands-on exercise

We cannot schedule a slow provider for practice. Configure both tools, verify the live sorting decision, then inspect a previously captured timeout trace.

Requirements:

- A `scripts/latency-guard.ts` that sends the order-confirmation prompt to `anthropic/claude-sonnet-4.6`
- `sort: "ttft"`: rank providers by measured time-to-first-token
- A `providerTimeouts.byok` deadline for your Anthropic BYOK key, sized to your latency budget
- Print who served the request, how fast, and the full `routing.sort` metadata

The timeout takes effect only when the request uses an Anthropic BYOK credential. Without one, the script still demonstrates TTFT sorting, but the provider deadline does not apply. [Bring Your Own Keys](/ai-gateway/save-money/bring-your-own-keys) covers the credential setup.

Wrapping the AI call in `Promise.race` with a timer stops the client request but does not retry another provider, so the customer receives an error. A Gateway timeout can abort the provider attempt and continue to the next eligible provider.

## Try It

```bash
pnpm latency-guard
```

The order confirms, and then the metadata shows its work:

```
One Camarón Cañón and one agua fresca — that'll be $8.50. Napkin situation: manageable.
---
Served by:     anthropic
Response time: 612ms
{
  "option": "ttft",
  "executionOrder": ["anthropic", "vertex", "bedrock"],
  "metrics": {
    "anthropic": { "ttft": 389 },
    "vertex": { "ttft": 501 },
    "bedrock": { "ttft": 548 }
  }
}
```

`executionOrder` shows the live sorting decision based on recent measurements. Your providers, values, and order will differ from this example.

A healthy request will not show the timeout firing. The following block is a previously captured and sanitized example trace:

```
"providerAttempts": [
  { "provider": "anthropic", "success": false, "error": "PROVIDER_TIMEOUT",
    "providerTimeout": true, "configuredTimeoutMs": 10000 },
  { "provider": "vertex", "success": true }
]
```

The provider exceeded the deadline, so the Gateway aborted that attempt and continued to the next provider. The attempt metadata records the timeout.

Two issues you may encounter:

**Your timeout never seems to apply.** Check `credentialType` in the attempt metadata. If it says `system`, the request did not use your BYOK key. Provider timeouts apply only to BYOK traffic.

**A reasoning model keeps crossing the deadline.** The clock stops at the first token, including a thinking token. Choose deadlines per model and feature rather than copying one value everywhere.

\*\*Warning: A timed-out request may still cost you\*\*

Some providers do not support cancelling an in-flight stream. The Gateway can continue to a fallback, while the abandoned request may still create a provider charge. Include that possibility in the latency policy.

## Commit

```bash
git commit -m "feat(latency): sort providers by ttft and add a byok first-token deadline"
```

## Done-When

- [ ] `sort: "ttft"` configured, and the `routing.sort` metadata shows a measured `executionOrder`
- [ ] A `providerTimeouts.byok` deadline set for your BYOK provider
- [ ] You can say what happens at the deadline: abort, fall to the next provider, `PROVIDER_TIMEOUT` in the trace
- [ ] You can explain why the timeout didn't apply to a `credentialType: "system"` request

## Solution

```ts filename="scripts/latency-guard.ts"
import { generateText } from "ai";
import { MENU } from "../lib/menu";

const result = await generateText({
  model: "anthropic/claude-sonnet-4.6",
  prompt:
    "Confirm this order back with a total: one Camarón Cañón and one " +
    `agua fresca.\n\n${MENU}`,
  providerOptions: {
    gateway: {
      sort: "ttft", // works with system credentials and BYOK
      // Hard first-token deadlines apply only to BYOK credentials.
      // Replace the provider and value to match your own key and latency budget.
      providerTimeouts: {
        byok: { anthropic: 10_000 },
      },
    },
  },
});

const routing = result.finalStep.providerMetadata?.gateway?.routing as any;
const attempt = routing?.modelAttempts?.[0]?.providerAttempts?.find(
  (a: any) => a.success
);

console.log(result.text.trim());
console.log("---");
console.log(`Served by:     ${routing?.finalProvider}`);
console.log(`Response time: ${Math.round(attempt?.responseTimeMs ?? 0)}ms`);
console.log(JSON.stringify(routing?.sort, null, 2));
```

TTFT sorting chooses a starting provider. The BYOK deadline defines when the Gateway should stop waiting and continue.

## Related Questions

- [How do I survive a provider outage?](/ai-gateway/stay-reliable/provider-outage)
- [How do I pin routing to a specific provider?](/ai-gateway/stay-reliable/pin-a-provider)
- [How do I route to the cheapest provider automatically?](/ai-gateway/save-money/cheapest-provider-routing)


---

[Full course index](/academy/llms.txt) · [Sitemap](/academy/sitemap.md)