How Do I Cut My Token Bill with Caching?

The Taco Tuesday assistant sends the full menu and operating instructions with every order. Those repeated input tokens can become a significant part of the request cost even when the content has not changed.

Prompt caching lets supported providers reuse that stable prefix instead of charging the full input rate each time.

Quick Answer

Set caching: "auto" in providerOptions.gateway. The Gateway adds cache markers for providers that require them, while other providers detect repeated prefixes themselves. Verify the cache hit in usage.inputTokenDetails.cacheReadTokens and compare the two request costs.

Outcome

Enable automatic caching on the ordering assistant's model and prove, with token counts from two consecutive requests, that the second one read the menu from cache instead of paying full price.

Fast Track

Add providerOptions: { gateway: { caching: 'auto' } } to a call with a big stable prefix
Run the same request twice
Compare usage.inputTokenDetails.cacheReadTokens: near zero the first time, nearly the whole menu the second

Hands-on exercise

Build scripts/cache-check.ts with two identical requests and print the token details for each one.

Requirements:

Use an Anthropic model (anthropic/claude-sonnet-4.6), because Anthropic needs the explicit markers, which makes the Gateway's work visible
Send the full TRUCK_INSTRUCTIONS (menu included) as instructions, plus a short customer question
Set caching: 'auto' under providerOptions.gateway
After each call, print inputTokens, cacheReadTokens from usage.inputTokenDetails, and the actual cost from gateway.cost

Caching requires at least two requests to demonstrate a hit. The first request writes the cache entry and may carry a write premium; the second request can read it at the cached-input rate.

Try It

pnpm cache-check

--- Request 1 ---
Input tokens:      612
Cache read tokens: 0
Cost:              $0.002214

--- Request 2 ---
Input tokens:      612
Cache read tokens: 578
Cost:              $0.000371

The second request reports most of the stable prefix as cache-read tokens and shows the lower request cost.

When caching costs you money

Some providers charge more to write a cache entry and less to read it. One-shot prompts can pay the write cost without receiving a later discount. Check the current provider policy and use caching for stable prefixes that repeat.

Two issues you may encounter:

cacheReadTokens is 0 on the second request. Check the provider's current cache lifetime and confirm that the stable prefix is byte-identical. A timestamp or request-specific value in the instructions prevents a prefix match.

The second request used another provider. Cache entries live with a provider, so changing providers can produce a miss. Pin cache-sensitive traffic when consistency matters, as shown in lesson 1.7.

Commit

git commit -m "feat(caching): enable automatic prompt caching for the menu prefix"

Done-When

scripts/cache-check.ts runs two requests and prints both receipts
Request 2 shows cacheReadTokens covering most of the menu
Request 2's actual cost is visibly smaller than request 1's
You can say when caching: 'auto' would lose money (one-shot traffic)

Solution

scripts/cache-check.ts

import { generateText } from "ai";
import { TRUCK_INSTRUCTIONS } from "../lib/menu";
 
async function order(label: string) {
  const result = await generateText({
    model: "anthropic/claude-sonnet-4.6",
    instructions: TRUCK_INSTRUCTIONS,
    prompt: "Which taco should I get if I can't handle spice? Be honest.",
    providerOptions: {
      gateway: { caching: "auto" },
    },
  });
 
  const { inputTokens = 0, inputTokenDetails } = result.usage;
  const cost = result.finalStep.providerMetadata?.gateway?.cost;
 
  console.log(`--- ${label} ---`);
  console.log(`Input tokens:      ${inputTokens}`);
  console.log(`Cache read tokens: ${inputTokenDetails?.cacheReadTokens ?? 0}`);
  console.log(`Cost:              $${cost}`);
}
 
await order("Request 1");
await order("Request 2");

The second request's token metadata tells us whether the provider reused the prefix and how that changed the cost.