LLM Pipeline Patterns

Before sending a request to LLM or a search engine, reformulate it in a different way. This compensates for inaccurate user formulations and aligns the semantics to the index vector space.

Breaking a long document into chunks for indexing or transferring into context. Strategies: fixed-size, recursive semantic, sentence-based, paragraph-based, sliding window with overlap.

Maximum filling of the context window with relevant information from different sources: retrieved chunks + metadata + examples + history. The goal is to convey everything the model needs in one call.

Instead of searching by raw query, ask LLM to generate a hypothetical answer, and then search by its embedding. This works because the answer embedding is semantically closer to the actual documents than the question embedding.

Explicitly instructing the model to “think out loud” before answering. The model generates intermediate steps of reasoning, which reduces errors on complex tasks - especially mathematics, logic, and multi-step inferences.

Passing input→output examples directly to the prompt. Dynamic version - examples are selected from the repository through semantic search, relevant to the current request. This is in-context learning without fine-tuning.

LLM generates or optimizes a prompt for another LLM call. It is used in automatic optimization of prompts (APE, DSPy-style), where the model itself offers improvements for the task.

Assign a role to system prompt to control style, tone, level of expertise, and output format. Activates the desired distribution of tokens in the model’s weights - it “switches the register”.

Interleaved reasoning + action. The model alternates between "Thought: [reasoning]" and "Action: [calling a tool]" and "Observation: [result]". This allows the agent to adapt to the results of the tools on the fly.

One incoming request launches N parallel LLM calls with different prompts, configurations, or subtasks. All calls are independent - there are no dependencies between them. The main goal is either to cover different aspects of the problem, or to collect a variety of answers for subsequent selection of the best one.

Assembling the results from N parallel calls into one final response. Strategies: concatenation, map-reduce summarization, voting, LLM synthesis.

Two-stage search/generation: in the first stage, many candidates are quickly received (recall); in the second, an expensive model re-evaluates and rearranges them in descending order of relevance (precision). Key insight: fast retriever (BM25, vector search) can find, but ranks poorly. A slow reranker (cross-encoder, LLM) can accurately compare query-document pairs because it sees both at once - this is a fundamentally different type of attention.

Run the same prompt N times with temperature > 0, get different reasoning paths, take the answer by majority vote. Statistically more reliable than a single greedy call.

For a document longer than the context window: split into chunks, summarize each in parallel (map), then combine the summaries into the final one (reduce). Can be done recursively.

Router determines the type of request and routes it to a specialized prompt/agent. Each “expert” is an LLM with a narrow system imperative. This is a prompt-level analogue of an architectural MoE.

The output of one LLM call becomes the input of the next. Each step transforms the data: extract → analyze → generate → format. Allows you to break a complex task into manageable subtasks.

LLM or deterministic code decides on the next step. If condition A is branch 1, if B is branch 2. Allows you to build non-linear pipelines with if-else logic.

Split into two independent LLM calls: Planner creates a list of steps, Executor executes each step in turn. Planner sees the entire task strategically, Executor is focused on one step - this reduces errors.

LLM first breaks down a problem into subtasks from simple to complex. Then solves them sequentially, using simple answers to solve complex ones. An analogue of dynamic programming for LLM.

The model generates a draft, criticizes it, improves it - and so on for N iterations. This simulates the editing process. Stopping based on quality condition or number of iterations.

Force JSON/structured output via Pydantic, Instructor, function calling or JSON mode. Scheme error → immediate retry with a description of the error in the prompt.

Separate LLM call to evaluate the quality of the main output. Judge receives the original question + answer and gives a score based on the criteria. Works as an automated reviewer.

If there is an error, do not just repeat the call, but pass on to the next attempt a description of what went wrong. LLM sees his mistake and takes it into account. Exponential backoff for rate limits.

Checking that every factual statement in the answer is supported by sources. Either through the NLI model (entailment), or through a separate LLM call, or through a citation search.

If the main model fails, automatic transition to the backup model occurs. Hierarchy: first the expensive model, in case of timeout/error - cheaper, then deterministic fallback.

Security layer before and after the LLM call. Input guardrails: toxicity, PII detection, jailbreak detection. Output guardrails: checking for admissibility, filtering personal data, subject restrictions.

A combination of keyword search (BM25/TF-IDF) and semantic (vector) search. BM25 is precise for exact terms and abbreviations, vector for meaning. RRF (Reciprocal Rank Fusion) combines ranks.

LLM generates N rephrases of the original query, each searched in the index, the results deduplicated and merged. Covers different semantic aspects of one question.

After the initial response, the model determines what knowledge is missing, formulates a follow-up query and performs an additional search. The cycle continues until the context is complete. This is a RAG with an internal feedback loop.

Index small chunks for precise search, but pass their parent (large) chunks into the context. Search accuracy + context completeness. Different granularities for different tasks.

When the dialogue history exceeds the context window, compress old messages into a compact summary, leaving new ones complete. Rolling summary is updated with each new message.

Extracting and accumulating information about entities (people, companies, products) from the dialogue into a separate store. The next time an entity is mentioned, its profile is pulled into the context.

Past interactions are saved as embeddings. With a new request, search for semantically similar episodes from the past and add them to the context. Long-term memory without storing a full log.

Allocation of a special zone in the prompt for intermediate calculations, notes and agent state. The model writes to the scratchpad during reasoning and reads from it - this is external “working memory”.

LLM decides when and which tool to call, generates call parameters, receives the result and continues reasoning. Tools: search, calculator, database, API, browser, interpreter code.

After performing an action, the agent reflects: what happened, what went wrong, what needs to be corrected. This is a supervised self-improvement within one inference loop without updating the weights.

Several LLM agents with different roles interact through a common message bus or hierarchically. Orchestrator delegates tasks to specialized subagents and collects the results. Allows you to solve problems that require parallel expertise in different domains.

A dedicated critical agent looks at the output of the main agent and issues specific comments based on the specified criteria (checklist). The main agent fixes it. The critic can be stronger than the main one - asymmetry is useful.

CoT extension: instead of one chain, there is a tree of reasoning options. At each step, N continuations are generated, they are evaluated, and beam search selects the best. Expensive, but powerful for complex planning.

Forced limitation of token space during generation at the logits level. Outlines/LMQL/Guidance masks invalid tokens at every moment of time - 100% guarantee of valid JSON/regex without retry.

After receiving the LLM output, a chain of deterministic transformations: strip markdown, extract JSON, normalize, deduplicate, sort, format. The less logic you put on LLM, the more predictable the system.

Processing tokens as they are generated - without waiting for a complete response. Allows you to terminate generation early (early stopping), show progress to the user, and process the beginning of the response in parallel.

Compress long context before sending to expensive model. The small model evaluates the “importance” of each token and removes the unimportant ones. 4-10x compression with minimal quality loss.

Caching responses not by exact string match, but by semantic proximity of the request. If cosine distance < threshold - return cache. Reduces costs on repeated requests in different formulations.

The query complexity classifier directs simple queries to the cheap, fast model, and complex queries to the expensive one. 70-80% of queries in an average product are simple. RouteLLM, LLM Router.

The small draft model generates K tokens quickly, the large model verifies them in one forward pass. Accepted tokens are free, rejected tokens are regenerated. 2-3x speedup with the same accuracy.

Anthropic and OpenAI support caching of KV cache prefixes. If system prompt + context are the same between requests, they are not recalculated. Up to 90% savings on prefill tokens for long system prompts.