Every time a large language model generates a token, it draws on the content that came before it. If it had to compute everything from scratch at every step, responses would be much slower. When building an agent, the same set of system prompts, tool definitions, and conversation history is used over and over again. If these were reprocessed each time, latency and computational costs would continually increase.
These two types of redundant computations correspond to two concepts that are often confused: KV Cache and Prompt Cache. A model’s processing of a single request is usually split into two stages: prefill and decode. The KV Cache stops the system from re-doing the work on historical token K/V pairs during decoding, and the Prompt Cache lets later requests reuse the same prefix.
In short, the KV Cache is the underlying state and inference mechanism. The Prompt Cache is the strategy or product capability that reuses these preprocessing results across requests. A lot of Prompt Cache implementations rely on reusing pre-computed K/V states.
Tip for reading: This text is going to talk about Q, K, V, prefill, decode, prefix matching, and cache breakpoints (also called cache boundaries). You don’t need to know anything about math or APIs to understand this article. When you’re reading, first think of Q, K, and V as “intermediate vectors” in attention calculations. Then, follow along with the two examples: “Beijing weather” and “product manual.”
KV Cache: “Intermediate Results” During Model Generation
Large models generate content token by token. Whenever a new token is generated, the model has to consider the tokens that have already appeared.
For example, in a standard Transformer, each token makes three sets of vectors—Q, K, and V—at every layer. You can think of Q as “what I’m looking for,” K as “what I have here,” and V as “what information I should extract if I’m selected.”
In autoregressive decoding, the Q values of historical tokens aren’t reused in subsequent steps. However, their K and V values are repeatedly queried by tokens generated later. So, the model stores these K and V values—this is the KV Cache.
For example, if you were to ask, “What’s the weather like in Beijing?” During the prefill phase, the model processes the whole question and stores the K and V values for each token at every layer. Once the decoding phase starts, new Q, K, and V values are calculated only for the token just added to the sequence at each step. The model puts the current K and V together with the cached history, then does attention calculations using the current Q on both the historical and current K and V to predict the next token. This way, you won’t have to keep recalculating the K and V of historical tokens.
But that doesn’t mean long context is free of cost. For standard full-attention models, the longer the context, the more video memory the KV Cache uses, and the more historical K/V pairs usually need to be read at each step. So, long conversations might still feel slow. Attention structures like sliding windows limit the history that can be seen.
The KV Cache is usually managed by the inference engine, and application developers rarely interact with it directly. It’s mostly used for incremental decoding within a single generation, but the cached K/V state can also be used by the inference framework for cross-request prefix reuse. The latter is often called a Prompt Cache or Prefix Cache.
Prompt Cache: Eliminating Redundant Processing of Identical Prefixes
When people hear the term “cache,” many immediately think of an “output cache,” where a previously answered question is simply returned. But the Prompt Cache isn’t the same kind of output cache. Even if there’s a cache hit, the model will still regenerate the response.
The Prompt Cache reuses intermediate results from the prefill phase for prompt prefixes, such as K/V states or other similar preprocessing results. A cache hit reduces redundant prefill computations and shortens the delay for the first token. If the API provider charges for cached inputs, it can also lower the cost of repeated inputs.
For example, let’s say you give the model a 50-page product manual and ask:
Product manual → What's the warranty period?A bit later, you ask another question based on the same manual:
Product manual → What are the requirements for returning an item?The product manual used in both requests is exactly the same, except for the last question. If this common prefix is cached, the second request can reuse the preprocessed results associated with the manual and process only the new question that follows. On the other hand, if you only use this manual once, Prompt Cache might not be that helpful.
For Prompt Caches that use automatic matching or caching based on breakpoints, the reusable portion should typically consist of a continuous, identical prefix starting from the beginning of the prompt. So, content that changes slowly and can be reused in many ways should go at the beginning, while content that changes frequently should go at the end. For example:
Long-term stable content: system prompt, tool definitions
→ Periodically stable content: user configuration, reference documentation, task background
→ Session content: conversation history, task status
→ Current request: current time, temporary information, user questionThis isn’t a fixed classification. The key is to arrange content by stability, but this shouldn’t alter message roles, command priorities, or business semantics. Different service providers may use automatic matching, explicit cache breakpoints, or independent cache objects. Also, keep in mind that minimum length, expiration periods, and billing rules can differ depending on the model. When integrating, it’s a good idea to check the latest model documentation and cache statistics in the response.
Two Common Bad Cases: These Approaches Can Quietly Break Cache Reuse
Here are two common examples. The code uses Anthropic’s cache_control as an example, but other service providers may use automatic matching, different cache markers, or independent cache objects. So, you can’t simply copy these fields across providers.
1. Dynamic Content Can Mess With Prefix Stability
When you’re counting on prefix matching, if the content changes at a certain point, the old prefix following that point usually can’t be reused. So, including a timestamp—which changes with every request—in the cache prefix will affect the fixed rules that follow it.
// ❌ Timestamp is included in the cached prefix and changes every request
const system = [{
type: "text",
text: `Current time: ${new Date().toISOString()}
You are a code assistant. Here are the fixed behavior rules...`,
cache_control: { type: "ephemeral" }
}]A better approach is to put long-term, stable content at the beginning and set a cache breakpoint at the end of the stable prefix. Dynamic information, like timestamps, should be placed after the cache breakpoint.
// ✅ Stable content first; dynamic content after the cache breakpoint
const system = [
{
type: "text",
text: "You are a code assistant. Here are the fixed behavior rules..."
},
{
type: "text",
text: "Here are the fixed tool usage instructions...",
cache_control: { type: "ephemeral" }
},
{
type: "text",
text: `Current time: ${new Date().toISOString()}`
}
]Content like project configurations and reference materials might be somewhere between “long-term stable” and “subject to frequent changes.” You can sort them by stability. If there are multiple cache breakpoints, set breakpoints for stable prefixes of different lengths. You also need consistency beyond just the text: the order of tool definitions, image parameters, and other elements may also participate in prefix matching.
2. Only Caching the System Prompt in Multi-Round Conversations
Multi-round conversations usually include the conversation history in every request. If you set the cache breakpoint only at the end of the system prompt and don’t enable automatic caching, the growing conversation history will still need to be processed repeatedly.
// ❌ Only caches the system prompt; conversation history is outside the cache boundary
const request = {
system: [{
type: "text",
text: "You are a code assistant...",
cache_control: { type: "ephemeral" }
}],
messages: history
}You can use Anthropic’s current auto-caching as an example. Enable cache_control at the top level of the request to automatically move the cache boundary forward as the conversation grows:
// ✅ Automatically cache the prefix of an ever-growing conversation
const request = {
cache_control: { type: "ephemeral" },
system: [
{
type: "text",
text: "You are a code assistant..."
}
],
messages: history
}Once enabled, the next round of requests can use the prefix that was cached from the previous round. It processes only the responses, tool calls, tool results, and current question that were added, and writes a new cache prefix for later requests. So, this reduces unnecessary processing of the conversation history. It does not mean that only the last message results in a cache miss.
If the service provider doesn’t support automatic caching, you need to follow its rules and place an explicit cache breakpoint at a stable position near the end of the conversation.
Setting a cache doesn’t guarantee a hit. When a prefix is encountered for the first time, the system typically has to finish the computation and write it to the cache first. A cache miss may occur if the cache has expired, doesn’t meet the minimum length, the historical prefix has changed, or the cache entry isn’t yet available. As of August 2026, each cache breakpoint in Anthropic will only search for previously written cache entries within the most recent 20 content blocks. A miss may also occur if too many blocks are added during a single Agent cycle.
You can’t just look at whether cache configuration is present in the request to determine whether caching really provides benefits. You should check the cache read, write, and hit metrics that the API returns. If latency is a concern, you should also log first-token latency on the application side and evaluate cache effectiveness together with actual costs.
Finally, How to Tell the Two Apart
| Concept | Core Function |
|---|---|
| KV Cache (Inference Mechanism) | Reuses the K/V pairs of historical tokens during generation to avoid redundant calculations at each step. |
| Prompt Cache / Prefix Cache (Cross-Request Reuse) | Reuses prefill results with the same prompt prefix across different requests. |
So, the two are not equivalent, nor are they entirely unrelated. The KV Cache is the underlying state and inference mechanism. The Prompt Cache reuses the pre-computed prefix state for other requests. For application developers, the best approach is to keep the common prefix stable and put timestamps, temporary information, and the current question as far toward the end as possible.
