Context Engineering & Prompt Caching

When they say “write a good prompt”, they often mean only the wording of the instructions. In practice, it's too narrow. The actual input model consists of system rules, tool schemas, examples, retrieved context, conversation state, user message, and sometimes service runtime items. Context engineering is the decision of what exactly falls into this budget, in what order and with what life expectancy.

WORKING MEMORY OF ONE RUN

instructions
 tool schemas
 examples
 retrieved docs
 recent turns
 current user turn
 runtime artifacts
 token budget / truncation rules
 caching strategy

The word “prompt” is useful to understand broadly. If runtime has added system message, tool definitions, and 12 retrieved chunks to your text, they together form the distribution of the following tokens. An error in RAG, a failed tool schema, or a trashy dialogue history change the answer as real as a poorly written instruction.

prompt = not just a question
prompt = all input,
What model sees before decode

This is a fundamental boundary. Prompt and context conditioning affect the current inference pass: which tokens will be more likely, which patterns are activated, how attention will be used. But after the weight response, the models remain the same. If behavior is to change consistently for all future queries, it is a matter of training/fine-tuning, not context engineering.

Each extra piece of text is paid twice: in tokens and cognitive noise for the model. In a crowded context, useful instructions and real-world evidence begin to compete with trash. Therefore, context engineering is almost always about selection and packaging, not about “stuffing everything that is”.

Better than 3 relevant blocks
20 vaguely related blocks

Better yet, clean transcript
than the full log of the user's life

When the system gets confused, don’t change the model first. Often the reason is simpler: the history of dialogue has grown, tool outputs are not cleaned, retrieval began to bring garbage, the order of blocks has changed, the cache hit has disappeared, the summary has lost an important fact. A good LLM engineer has a reflex to check the packed context first, rather than blaming weights right away.

The most convenient format for engineering is a large static prefix and a compact dynamic tail. In the prefix live instructions, schema, policy, few-shot and description tools. In the tail - a specific user turn, retrieved snippets and minimally needed history. This improves handling, eases debugging and increases the chance of a cache hit.

PROMPT SHAPE

stable prefix
  system / policy
  tool schemas
  output format
  examples

dynamic tail
  retrieved docs
  recent turns
  latest user message

If a critical policy is buried within a long user context, the model may treat it as a fact rather than a rule of conduct. Therefore, system/instruction layers are derived from and separated from sources of knowledge. The point is that the model understands what is the norm of the answer, and what is only material for reasoning.

LLM is easier to work with a context where the parts: ‘INSTRUCTIONS’, ‘TOOLS’, ‘CONTEXT’, ‘USER QUESTION’, ‘OUTPUT FORMAT’ are clearly highlighted. Such a prompt does not make the model “smarter,” but reduces confusion about what it is. This is especially important when retrieved text, chat history and machine-generated fragments are mixed in one window.

[SYSTEM RULES]
[TOOL SCHEMAS]
[RETRIEVED FACTS]
[RECENT DIALOG]
[CURRENT TASK]
[RESPONSE FORMAT]

A few good examples teach the model the desired response style, JSON structure, failures, and edge case handling. But if examples grow into a mini dataset, they begin to eat up the budget and blur the current task. In a mature system, examples are short, curated and reflect only really necessary patterns.

The best RAG systems don’t just insert top-k passages. They sign the source, delete duplicates, sometimes group by document, and explicitly separate the retrieved evidence from the instruction layer. Then the model sees not random text garbage, but a set of facts to use.

retrieve
→ dedup
→ rerank
→ group / trim
→ annotate source
→ pack into context

After the answer, the model does not store the personal memory of the user in its scales. In order for the next move to know the previous context, the system must either send the story back or use a server-side state API that will restore the item chain itself. Therefore, the memory in the product is not the magic of the model, but the infrastructure of state storage and transfer.

Stateful chaining is convenient because the client does not need to collect the entire transcript every time. But that doesn’t mean the previous context has disappeared from the model’s value and focus. The official guides explicitly state that when a chain continues, past input tokens are still counted as inputs of that chain, including runtime artifacts like reasoning items if they are involved in the continuation.

response_1 = create(input=...)
response_2 = create(
  previous_response_id=response_1.id,
  input="one more question"
)

The client doesn't send transcript with his hands.
But the state still participates in the run.

Modern APIs begin to give a separate conversation object, which stores items on the provider side and allows you to continue the chain without constantly retransmitting the entire array of messages. This is convenient for long threads, handoffs between devices and cases, where in addition to message history there is an agentic state. But the engineer still needs to understand the budget and life cycle of these items.

When an app just stores an array of all the messages and sends it on every move, it seems simple at first. Then come problems: runaway cost, giant prompts, random leakage of old tool outputs, conflicting instructions from the distant past, and unpredictable degradation of quality. The longer a product lives, the more trimming, compaction, and separation of persistent state from transcript are needed.

In one application can live simultaneously: a conversational history, user profile, results of past tools, structured task state and retrieval corpus. If you put it all in one sheet of text, the system becomes fragile. Mature architecture stores these entities separately and prompts only what is really needed in the current step.

With a long prompt, a significant part of the time and cost is spent on prefill: the model must run all input tokens and build internal states for further decode. If the next request starts with the same long prefix, the provider can reuse the work already done. Important: This doesn’t mean that the answer just comes from cache like HTTP. The generation of a new response still occurs anew for a new tail and a new sampling path.

CACHE HIT LOGIC

request A: [same prefix][tail A] → prefill full → decode
request B: [same prefix][tail B] → reuse prefix work → prefill tail B → decode

speeding up
The answer is not static memoized object.

The official guide describes automatic caching of prompt prefixes from the threshold of 1024 tokens with a growth step multiple of 128 tokens. In usage, you can see 'cached tokens', and with 'prompt cache key' you can help route similar requests to a single cache path. For an engineer, this means that if an important prefix is stable, it can be made cheap and fast without manual application-level memoization.

If the timestamp, random request ID, tool definitions order or header format changes at the beginning of prompt, the cache will easily fall apart. Therefore, a stable prefix should be determined: the same order of fields, the same schema names, without meaninglessly changing pieces. The tail can change, the prefix must live as an almost immutable artifact.

cache-friendly prompt =
stable front
+ variable end

cache-hostile prompt =
dynamic metadata at the top
+ reordered blocks

Anthropic describes prompt caching more explicitly as checkpoints inside prompt prefix. The hierarchy of blocks is important: usually tools go before the system, system before messages; cache counts prefix according to this structure. The documentation also describes different TTLs, including cache up to an hour, which is useful for long agentic sessions and heavy tool-heavy prompts.

Engineers often lose cache not because feature is bad, but because they make prefix unstable. At the beginning of the query is time, trace id, current date in different formats, dynamically generated list of tools or crude transcript with infinitely expanding blocks. In this mode, prompt caching is formally enabled, but almost never works.

If a dialogue or workflow lasts a long time, keeping a full text history becomes too expensive and harmful. The old turns are then converted into a shorter surrogate: summary, compacted state item, or other equivalent representation. The goal is not a beautiful retelling, but to preserve the working utility of the story with a noticeably smaller token budget.

The official OpenAI guide compaction is described as a way to replace a long sequence of old items with a shorter view while retaining the ability to continue the conversation. This is stronger than just locally making a summary string and pasting it into prompt: the provider can retain the desired internal connectivity of the chain, and the client gets a more compact state for the next moves.

long chain
→ compaction
→ shorter state representation
→ next turn can continue

Victory:
less
less latency
less of a noisy past

The guide also shows a separate compaction mode for the whole conversation. This is useful when you need to “pack” thread in advance: before a pause for the night, transfer between services, synchronization between devices or saving a long customer history in a more compact form. Such a step can be made an explicit part of lifecycle, rather than just waiting for emergency overflow.

Anthropic describes “context editing” as a mechanism that can clear old use/result blocks from active context. This is important for agent systems: search results, SQL, browser steps, and intermediate computing quickly clutter the window. If these blocks are not edited or removed, the context swells up, the cache breaks down, and the model begins to rely on outdated intermediate artifacts.

Any compression of history loses detail. Therefore, you cannot make a summary the only carrier of important business facts or workflow state. The correct approach: the summary helps the model remember the general course of the conversation, and the true data about the user, order, task and steps of the pipeline are stored structured in the database and submitted separately if necessary.

In agentic systems, context is not just spent on conversation. Tool descriptions, JSON schemas, tool use instructions, and error-based policy often make up a large part of the input. This makes the tool registry the main object of optimization: names, descriptions, parameters and the number of simultaneously available tools affect the quality of choice, price, and cacheability.

If the search returned 50 HTML snippets, SQL gave a wide spreadsheet, and the browser tool kept a huge DOM snapshot, the model should not see it raw. Usually, an intermediate step is needed: extract, normalize, prune, maybe summarize. That is, between tool execution and re-submission to the model should be a layer of context curation.

tool output
→ filter relevant fields
→ remove noise / markup
→ maybe summarize
→ inject compact result

not:
tool output → paste whole blob

Short-term memory is responsible for the continuity of the dialogue, retrieval for the presentation of external knowledge on request, workflow state for the structured state of the process. When a developer tries to solve all three problems with one line, “Put in more history,” the system begins to break down. On the contrary, it is necessary to clearly decide what is stored in transcript, what is stored in the vector store, and what is stored in the fields of the state machine.

Reasoning items, hidden scratch work, intermediate tool traces and service planner steps are needed by the system, but do not have to live in user transcript. The clearer the boundary between “internal workflow” and “useful context for the next move,” the more stable the agent is. Otherwise, the model begins to argue with itself based on the old intermediate steps.

The most reliable way is to collect context not spontaneously, but through an explicit packer. It knows the limits, knows how to trim history, throw out outdated tool results, mix retrieval only for business and keep a stable prefix for cache. In fact, it is a separate layer of the application, no less important than retriever or model client.

RUNTIME CONTEXT PACKER

policy / system
  ↓
tool registry
  ↓
retrieved evidence
  ↓
recent transcript / compact summary
  ↓
current user turn
  ↓
token budgeting + trimming
  ↓
final packed prompt

The total number of tokens is useful, but not enough. For real debugging, it’s more important to know how much went into instructions, how much went into tools, how much went into retrieval, how much went into transcript, how much cached, how many tool outputs were cleaned out, and whether the compaction occurred. This is the only way to understand why latency and cost suddenly rose or why quality suddenly fell.

If the application is based on a long stable prefix, the lack of cache hit immediately hits the latency and budget. Therefore, it is worth logging “cached tokens”, the share of requests from hit, the average size of the cached prefix and cases when the hit rate collapsed after the release. This is often the earliest signal that someone has inadvertently made prompt prefix unstable.

If there is no final packed prompt or at least layered representation in traces, you are practically blind. Good observability allows you to answer: what documents were mixed, what summary was used, what tools were available, what prefix version worked, whether there was a cache hit and whether there was a compaction. Without this, LLM debugging turns into divination.

prompt/template version
tool registry version
retrieved doc ids
summary / compact state id
input token breakdown
cached_tokens
final output + user signal

The same budget policy is rarely suitable for a quick FAQ response, for research-agent, for voice session, and for code-assistant. Somewhere the minimum latency is important and there is almost no history, some where long continuity is needed, some where retrieval dominates transcript. Therefore, a production system typically has multiple packer policies and multiple compaction/caching profiles tied to specific product surfaces.

At an early stage, it seems that everything is solved with one successful phrase in the system prompt. At the mature stage, the focus shifts: stable prefix, understandable layers of context packing, token budgets, cache-friendly design, compaction strategy, cleanup tool traces, observability over packed context, route-specific policies. This is what makes a “model demo” into a live LLM product.

MATURITY LADDER

prompt wording only
  ↓
structured prompt templates
  ↓
retrieval + memory separation
  ↓
state management
  ↓
prompt caching / compaction
  ↓
observability of packed context
  ↓
context engineering as first-class system design