Sort providers by cost, latency, or throughput on AI Gateway

You can now sort the providers behind a model by cost, time to first token (TTFT), or throughput (TPS) in AI Gateway.

The default provider order blends provider reliability, quality of model output, cost, and speed of response. You can now use sort for explicit control over ranking criteria.

For models with many providers and noticeable cost or speed variation, you can use sort to optimize on your dimension of choice. Ranking is computed at request time, so newly added providers, price changes, and shifts in observed latency or throughput flow through automatically without any code changes.

Set sort on providerOptions.gateway to one of the three values:

Link to headingBasic usage

Use sort to ensure optimizing for your metric of choice.

In this example, AI Gateway has over five providers for GPT OSS 120B with different prices, so sorting by cost is a useful option for requests that want to route through the lowest price provider.

Providers are tried in sort order. Fallback to the next provider only happens when the higher-ranked one is unavailable.

import { streamText } from 'ai';
const result = streamText({
  model: 'openai/gpt-oss-120b',
  prompt: 'Summarize this internal document.',
  providerOptions: {
    gateway: {
      sort: 'cost', // Use the lowest cost provider first
    },
  },
});

Link to headingCombine with other routing controls

sort is compatible with other gateway routing options like Zero Data Retention (ZDR).

The example below uses deepseek/deepseek-v4-pro for an interactive request where latency and data retention matter: AI Gateway filters to only providers for Deepseek V4 Pro that have zero data retention, and then sorts the remaining providers by time to first token (TTFT).

import { streamText } from 'ai';
const result = streamText({
  model: 'deepseek/deepseek-v4-pro',
  prompt,
  providerOptions: {
    gateway: {
      zeroDataRetention: true,
      sort: 'ttft', // Among ZDR-compliant providers in this set, try the lowest latency first
    },
  },
});

sort also composes with order: providers listed in order are promoted to the front, and the remaining providers follow the requested sort criterion.

Link to headingInspecting routing decisions

See exactly why each request landed where it did. Every response includes a sort block in the routing metadata showing which providers were considered, the metric values used to rank them, the order they were attempted, and any that were deprioritized due to degraded health.

1
{
2
  "gateway": {
3
    "routing": {
4
      "sort": {
5
        "option": "cost",
6
        "executionOrder": ["novita", "groq", "fireworks", "baseten", "cerebras"],
7
        "metrics": {
8
          "novita": 0.10,
9
          "groq": 0.15,
10
          "cerebras": 0.20,
11
          "fireworks": 0.22,
12
          "baseten": 0.25
13
        },
14
        "deprioritizedProviders": ["cerebras"]
15
      }
16
    }
17
  }
18
}

For more information on sorting via AI Gateway, read the documentation.