Prompt Caching for Production LLM Apps in September 2026: A Practical Developer Guide

Prompt caching is one of the simplest ways to reduce the cost and latency of a production LLM application that repeatedly sends the same instructions, documentation, examples, or tool definitions. Instead of processing an identical prompt prefix from scratch on every request, a provider can reuse previously processed input tokens and charge a lower cache-read rate. The result is not a new model or a replacement for application architecture. It is an optimization for repeated context.

Image: mikemacmarketing / photo on flickr via Wikimedia Commons (CC BY 2.0)

Image: Midjourney; prompt suggested by Grok via Wikimedia Commons (Public domain)

 

 

What prompt caching actually does

An LLM request usually contains two broad sections: a relatively stable prefix and a changing suffix. The prefix may include the system prompt, safety rules, product documentation, tool schemas, response-format instructions, and few-shot examples. The suffix may contain the current user message, retrieved records, conversation turns, or request-specific parameters.

Prompt caching allows the provider to recognize a repeated prefix and reuse internal processing associated with that prefix. The model still generates a new answer for each request, and the dynamic part of the request is still processed normally. Caching therefore does not make generation free, and it does not cache the model's answer unless you implement a separate response cache.

A cache hit normally reduces input-processing cost and often improves time to first token. Output-token charges remain unchanged. If a request contains 20,000 stable tokens and 500 dynamic tokens, caching can make a significant difference because the expensive repeated portion is read from cache while only the new material is processed as ordinary input.

How the major APIs expose caching

OpenAI

OpenAI supports automatic prompt caching for eligible repeated prefixes on supported models. In the typical integration, you do not need to create a cache object manually. Send requests with a stable token sequence at the beginning, and inspect the usage information returned by the API for cached input tokens.

The important implementation detail is prefix identity. The provider compares the beginning of the prompt, so changing an early system instruction, tool definition, or serialized configuration can prevent a hit for everything after that point. Put stable content first and request-specific content later. Avoid inserting a timestamp, request ID, random example, or user-specific instruction near the beginning of the prompt.

OpenAI's pricing pages and model documentation should be treated as the source of truth for current cache-read rates, eligibility, retention windows, and minimum prompt sizes. These values can differ by model and can change independently of the API surface. In production, record both total input tokens and cached input tokens rather than assuming every request is a hit.

Anthropic

Anthropic exposes explicit prompt caching through cache-control markers. A request can mark a content block with a cache policy such as {"cache_control":{"type":"ephemeral"}}. This gives the application more control over where the reusable boundary occurs.

Anthropic's ephemeral caching is designed for temporary reuse rather than permanent storage. The exact lifetime and available duration options depend on the API and model. Anthropic also applies minimum cacheable token thresholds, so a tiny system prompt may not qualify. As of the current documentation, supported Claude models commonly use a minimum of 1,024 tokens, although developers should check the model-specific documentation before relying on that figure.

Anthropic separates cache creation or write pricing from cache-read pricing. A cache write can cost more than an ordinary input because the provider is storing and preparing the prefix. Repeated reads are substantially cheaper. This means caching a prefix for one isolated request is usually a loss; the prefix needs enough reuse to recover the initial write cost.

Google Gemini

Google's Gemini API uses context caching, generally through an explicit cached-content resource. You create cached content with a selected model, provide the reusable contents, and set a time-to-live. Later requests reference that cached content and add the changing portion.

Gemini has model-specific minimum token requirements. Gemini 2.5 models have commonly required at least 2,048 tokens for context caching, while Gemini 3 models have commonly required at least 4,096 tokens. These thresholds and supported features should be verified against the current Gemini API documentation before deployment.

Google also charges for cached storage over time. A long time-to-live can be useful for a frequently used knowledge base or tool catalog, but it can waste money when the cache is rarely accessed. Treat TTL as an operational setting: short for bursty workloads, longer for consistently busy workloads, and carefully monitored for large prefixes.

What belongs in the cached prefix

The best cached material is large, stable, and reused across many requests. Good candidates include:

• System instructions that change only when the application is deployed.
• Tool names, descriptions, JSON schemas, validation rules, and usage policies.
• Long product manuals, internal operating procedures, or domain reference material.
• Few-shot examples that are part of the application's fixed behavior.
• Stable output schemas and formatting rules.
• Shared policy text used by every tenant or a large group of tenants.

Keep the cached prefix deterministic. Serialize JSON with stable key ordering, use consistent whitespace, and avoid embedding values that change on every request. A semantically identical prompt is not necessarily byte-for-byte or token-for-token identical to the cache system.

What belongs in the dynamic suffix

Put user and request-specific material after the reusable prefix. This includes the latest user message, recent conversation turns, retrieved documents, account data, current date, authorization context, tool results, and per-request metadata.

Do not place sensitive tenant data in a shared cache unless the provider's cache isolation guarantees and your own keying strategy are fully understood. A safe design often creates separate cache prefixes per tenant, workspace, or permission boundary. That can reduce cache reuse, but privacy is more important than a theoretical discount.

Conversation history requires special care. If every new turn is appended to the end, the earlier prefix may remain reusable. If the application prepends a new summary or changes the system prompt on every turn, the cache can be invalidated repeatedly. Consider a stable instruction prefix followed by a conversation section, and update summaries at controlled checkpoints rather than on every message.

Pricing and latency: measure the complete request

Prompt caching savings depend on three variables: the number of repeated input tokens, the cache-read price, and the number of times the prefix is reused before it expires. A useful rough calculation is:

net savings = ordinary input cost - cache write cost - cache read cost - cache storage cost

For a small prompt sent only twice, caching may not help. For a 30,000-token tool schema and policy bundle sent thousands of times per hour, it can be one of the highest-impact optimizations available.

Measure more than average latency. Track cache-hit ratio, cached input tokens, cache writes, cache reads, time to first token, total request latency, and cost per successful request. A cache hit may improve time to first token while total latency remains dominated by output generation or a slow tool call.

Common production pitfalls

Cache invalidation

Changing one early token can invalidate the reusable suffix. Version stable prompts explicitly, such as policy-version: 2026-09-01, and deploy prompt changes deliberately. Never assume that updating a prompt automatically removes every old cached copy immediately. Use provider-supported expiration and versioning rather than trying to manage hidden provider caches yourself.

Minimum token thresholds

A prompt shorter than the provider's minimum may never be cached. Do not pad a short prompt with meaningless text merely to cross the threshold. Padding increases cost and can make instructions harder to maintain. Caching is most appropriate when the repeated context is naturally large.

Privacy and authorization

Cached content can contain confidential instructions, customer records, or proprietary documents. Review retention, isolation, encryption, deletion behavior, and provider terms. Never assume that a cache is equivalent to a private in-memory variable controlled only by your application.

Changing tool schemas

Tool definitions are often large and repeated, making them excellent cache candidates. They are also easy to invalidate accidentally. Keep tool ordering stable, avoid dynamically adding unused tools to the beginning of the list, and version schemas when they change. If tool availability differs by user, partition the prefix by permission set.

Ignoring observability

Without usage telemetry, a team can believe caching is active while receiving few or no hits. Log provider usage fields, but do not log the full prompt or sensitive cached content. Aggregated metrics are usually sufficient to identify broken prefix stability.

When caching beats RAG

Prompt caching is often better than RAG when the same relatively static context is needed for nearly every request. Examples include a fixed tool catalog, a company-wide support policy, a coding assistant's framework instructions, or a stable workflow definition. RAG adds embedding, search, filtering, ranking, and document assembly overhead. If the relevant context is already known and reused, caching is simpler and usually faster.

RAG is better when the knowledge base is large, changes frequently, or only a small subset is relevant to each request. Caching an entire million-token corpus may be wasteful when each question needs only a few passages. A hybrid design is common: cache the stable instructions and tool schemas, then retrieve current documents into the dynamic suffix.

When caching beats fine-tuning

Fine-tuning changes model behavior through training examples. Prompt caching does not change behavior; it reduces the cost of repeatedly supplying instructions. If the problem is “the model needs the same policy and schema on every request,” caching is usually the first optimization to test. It preserves editable instructions and avoids a training pipeline.

Fine-tuning becomes more appropriate when the application needs a durable change in style, classification behavior, structured-output reliability, or task-specific performance that cannot be achieved efficiently with instructions. The two techniques are not mutually exclusive: a fine-tuned model can still benefit from caching repeated tools and reference material.

A practical rollout checklist

1. Measure the current input-token cost and time to first token.
2. Separate stable instructions from dynamic user and retrieval content.
3. Make serialization deterministic and keep the stable prefix first.
4. Check the provider's model-specific minimum token and TTL rules.
5. Start with a high-reuse prefix such as tools, policies, or documentation.
6. Inspect cached-token usage in real responses.
7. Track cache-hit ratio, cost, latency, and privacy boundaries.
8. Version prompt changes and test invalidation behavior before deployment.

Prompt caching is most valuable when treated as a measured systems optimization, not a toggle. Build a stable prefix, keep volatile data out of it, understand each provider's pricing and retention rules, and verify the result through usage telemetry. For repeated system prompts and tool schemas, it can deliver lower cost and faster responses with far less operational complexity than adding RAG or retraining a model.