Caching

Prompt-prefix caching and the optional response cache — what they do and how they're billed.

Two independent caching layers can sit behind your endpoint. They do different things and bill differently.

1. Prompt-prefix caching (always on)

Most upstream providers cache the prefix of a prompt so a repeated system prompt + tool schema + conversation history isn’t recomputed from scratch on every turn. When the upstream reports cached tokens, they show up on your response:

"usage": {
  "prompt_tokens": 8421,
  "prompt_tokens_details": { "cached_tokens": 7680 },
  "completion_tokens": 142,
  "cost": 0.0142
}

Why it matters for you: long agent loops with a stable preamble get cheaper as the cached prefix grows. To maximize hits, keep the front of your prompt stable (system message, tools, early context) and let only the tail vary. The gateway also normalizes request bodies (canonical key order, stable tool ordering) before forwarding, which lifts upstream cache-hit rates without any change on your side.

This layer is transparent — no flag, no body change.

2. Response cache (opt-in, per-org)

An optional full-response cache: if an org enables it and an identical request arrives within the TTL, we return the stored completion and skip the upstream call entirely.

• Opt-in — off by default; enabled per-org.
• Scope — keyed per-org; never shared across orgs.
• TTL — short (default ~30 minutes), configurable.
• Non-streaming only — streamed requests are never response-cached.
• Privacy — this is the one case where a completion body is held transiently; see privacy & security.

A response-cache hit is flagged with an X-Dynoyard-Cache: HIT response header.

Billing

• Prompt-prefix cache: cached input tokens are reported separately on usage and billed at the model’s reduced cached-input rate. We mirror the upstream provider’s cache discount straight through to you — a cached token costs you what it costs us. You win twice: fewer recomputed tokens and a lower per-token rate on the cached portion of your prompt.
• Response cache: a cache hit is billed at the same cost the original request would have cost (recorded at store time), so spend is predictable whether or not a given request hits cache.

usage.cost on every response always reflects what you were actually charged — caching never makes the returned cost figure wrong.

Computing an effective / blended $/M

To show a blended (cache-adjusted) price across many calls, sum usage.cost and divide by total tokens:

blended $/M = Σ(usage.cost) / Σ(prompt_tokens + completion_tokens) × 1e6

usage.cost already includes the cache discount, so the blended drops as your cache-hit rate rises — and it stays correct automatically if our rates change. This is the recommended way to track effective price.

OpenAI-compatible cache fields. Dynoyard is OpenAI-shaped, not Anthropic-shaped. We do not emit Anthropic’s cache_read_input_tokens / cache_creation_input_tokens. The cached portion is reported the OpenAI-standard way: usage.prompt_tokens_details.cached_tokens (a subset of prompt_tokens, not additive). If a multi-provider cost tool shows Dynoyard as artificially expensive, it’s almost certainly reading the Anthropic field (which is absent) and seeing zero cache — point it at usage.cost or prompt_tokens_details.cached_tokens instead.