DSPy & GEPA: Implementation Cookbook

In conventional prompt engineering, task signature, formatting, reasoning strategy, few-shot examples, and model-specific hacks are mixed into one fragile text. DSPy lays this out in layers and allows you to change the layer without rewriting the entire system. That is why it is useful not only for optimization, but also for normal architecture.

LM specific model and generation kwargs
Signature What Comes In and What Should Come Out
Module: Predict / CoT / ReAct / PoT
Adapter as a signature turns into messages and back
Metric is considered a good answer.
Optimizer How to Find the Best Instructions / Demos / Weights

Bottom line: code controls flow and prompts become compiled artifact

DSPy’s official programming overview doesn’t start with optimizers, but with task design and pipeline. The right start: first decide what inputs and outputs the system needs, where retrieval or tools are needed, and then think about how to compile prompt internals.

1. determine the external contract of the task
2. make the simplest working pipeline
3. collect characteristic dips in dev examples
4. describe metric.
5. Then turn on the optimizer.

The DSPy documentation clearly suggests a cyclical process: programme, assemble devset, define metrics, optimize, go back and change the pipeline itself if the metric or architecture is weak.

Explore manual runs, 10-20 cases
   ↓
Evaluate  devset + baseline metric
   ↓
Optimize  few-shot / instruction / weights
   ↓
Re-design if metric is bad or pipeline is poorly decomposed
   ↓
Harden    observability, cache, save/load, deployment

GEPA does not replace a DSPy program, but sits on top of it as a reflective optimizer. Its task is not to invent the entire architecture for you, but to improve the text components of an existing program using score, feedback and execution traces.

dspy.LM specifies model and generation defaults. Documentation shows work with OpenAI, Anthropic, Gemini, Vertex, Databricks, Ollama, local OpenAI-compatible endpoints and other LiteLLM providers. LM is cached by default, stores call history, usage and cost metadata, and can switch via dspy.configure and dspy.context.

lm = dspy.LM("openai/gpt-4o-mini")
dspy.configure(lm=lm)

with dspy.context(lm=other_lm):
    ...

generation kwargs: temperature / max_tokens / stop / cache
diverse rollouts: rollout_id + temperature>0
history: prompt / messages / response / usage / cost / model
reasoning models: model_type="responses"

DSPy: Signature is a declarative specification for input/output behavior. It's about describing what to do, not manually dictating models how to ask.

Inline: "question -> answer"
Typed inline: "context: list[str], question -> answer"
Class-based:
class Emotion(dspy.Signature):
    """Classify emotion."""
    sentence: str = dspy.InputField()
    sentiment: Literal[...] = dspy.OutputField()

Class-based signatures are needed when you want to clearly define the meaning of the task: docstring, field descriptions, output limitations and complex types. DSPy supports not only 'str' but also 'int', 'bool', 'list[str]', 'dict[...]', 'optional[...]', custom types and special types like 'dspy'. Image' and dspy. History.

The basic data container in DSPy is ‘Example’. It is similar to dict but knows which fields are inputs. 'Prediction' is a special subclass 'Example' that returns modules. Metric, eval and optimization work with the same semantics of objects.

x = dspy.Example(question="Q", answer="A").with_inputs("question")
x.inputs() # only inputs
x.labels() # The rest as labels/metadata

pred = program(**x.inputs())
metric(x, pred)

dspy.Module is an analogue of nn.Module for LM programs. Inside, you can store sub-modules, predictors, and your control flow. All serious DSPy systems, from multi-hop RAG to tool agents, end up with their own forward() classes.

class MyProgram(dspy.Module):
    def __init__(self):
        self.retrieve = dspy.Predict("question -> query")
        self.answer = dspy.ChainOfThought("context, question -> answer")

    def forward(self, question):
        query = self.retrieve(question=question).query
        context = search(query)
        return self.answer(context=context, question=question)

Officially, adapters are responsible for translating signature, demos, input data, and DSPy types into LM messages and parsing the response back to Prediction. They also manage history and function calls. By docs, `ChatAdapter' is used by default; `JSONAdapter' is good for native structured output and lower latency.

The documentation says plainly: dspy. Predict is the basic module on top of which the rest are built. It simply displays input signature in outputs through the language model and does not extend the signature with additional reasoning fields.

The API defines ‘ChainOfThought’ as a module that reasoning and step by step before predicting output. Learn docs further emphasize that in many cases, simply replacing 'Predict' with 'ChainOfThought' immediately improves quality.

signature: "document -> summary"
↓
CoT extends it to an additional reasoning field
↓
response.reasoning + response.summary

dspy.ReAct implements the classic pattern "Reasoning and Acting": the model sees the list of tools, reason, decides to call the tool or complete the task. In DSPy, this pattern is generalized to arbitrary signatures, not just “question-> answer” chat agents.

agent = dspy.ReAct(
    "question -> answer",
    tools=[search_web, get_weather],
    max_iters=5,
)

The official learn page summarises `ProgramOfThought` is as follows: LM learns to output code, and the result of execution of this code determines the answer. The API specifies that the module runs Python programs and requires runtime to be interpreted.

This is a family for cases when text reasoning is not enough. CodeAct is inherited from both ReAct and ProgramOfThought: it is essentially a tool agent with code interpreter. RLM is a standalone experimental abstraction for very large contexts, where LM explores data through sandboxed Python REPL and recursive sub-LLM calls instead of directly loading giant context into prompt.

Not all built-in modules are about reasoning style. Some of them manage the compute budget and execution policy. 'BestOfN' repeats the module N times and takes the best one by 'reward fn'. Refine does the same thing, but adds a feedback loop between attempts. MultiChainComparison brings several reasoning attempts into one final answer. Parallel is a utility for multithreaded pair launch (module, example).

BestOfN different rollout id choose the best answer
Refine try to get feedback again
MultiChainComparison several reasoning paths
Parallel batch evaluation / batch execution on streams

Learn docs emphasize three types of values on example: inputs, intermediate labels, and final label. Important practical conclusion: DSPy can often be used without intermediate labels and even without final labels, if the metric is able to measure quality by inputs and output.

The DSPy documentation literally says: framework machine-learning-like, so automatic metric is mandatory for both evaluation and optimization. For simple tasks, metric can be an ‘accuracy’/‘exact match’; for most production tasks, metric must be a small DSPy program or a function that checks several properties of long-form output.

def metric(example, pred, trace=None):
    score_1 = ...
    score_2 = ...

    if trace is None:
        return (score_1 + score_2) / 2.0  # evaluation / optimization
    else:
        return score_1 and score_2         # bootstrap demos

Before running a compile, you need to understand the baseline. Docs offer either a regular Python loop or a built-in dspy.evaluate.Evaluate that helps with multi-thread eval, progress bar and sample predictions table.

evaluator = Evaluate(
    devset=devset,
    num_threads=1,
    display_progress=True,
    display_table=5,
)
evaluator(program, metric=metric)

The official DSPy heuristics are good to start with: already 20 dev examples are useful, 200 go far; for optimizers, you can get value from 30 examples, but it is better to aim for 300+. For many, prompt optimizers docs recommend an atypical split `20% train / 80% val` because prompt optimization is retrained on a small trainset. For GEPA, the more classic ML approach is recommended: maximize the trainset and keep the val large enough to reflect downstream distribution.

Docs describe this family as an automatic signature extension of optimized examples. LabeledFewShot just picks up labeled demos. BootstrapFewShot uses the teacher program to generate demonstrations and filters them with metrics. 'BootstrapFewShotWithRandomSearch' amplifies search, and 'KNNFewShot' picks up close examples before bootstrap.

Copro makes coordinate-ascent over instructions. MIPROv2 combines instruction generation and few-shot search through Bayesian Optimization. SIMBA seeks complex examples through stochastic minibatches and generates self-reflective rules. GEPA analyzes trajectories and textual feedback. This family is worth taking when the main problem is not in the raw control flow, but in how the subtasks are formulated.

‘BootstrapFinetune’ distill-it prompt-based DSPy program in weight updates: the program remains the same in structure, but the steps begin to be performed by finetuned models. BetterTogether combines prompt optimization and finetuning in a sequence like ‘prompt → weight → prompt’. Ensemble uses DSPy programs as one program transformation layer.

DSPy docs themselves give the starting heuristics. They are not a substitute for experimentation, but work well as the first decision tree.

about 10 examples
  → BootstrapFewShot

50+ examples
  → BootstrapFewShotWithRandomSearch

0-shot instruction optimization
  MIPROv2 in 0-shot configuration

200+ examples and more budget optimization
  → MIPROv2

There is already a strong large model and you need an efficient small runtime.
  → BootstrapFinetune

Track-level feedback and rich failure analysis
  → GEPA

According to official GEPA docs and paper: the optimizer takes the seed candidate, runs it on the training signal, receives traces and numeric score, reflects what worked and what did not, generates a new instruction and selects candidates through the Pareto-style frontier. This is no longer a blind search for scalar reward, but a search for textual diagnoses.

seed DSPy program
      ↓
run on minibatch / trainset
      ↓
metric(example, pred, trace, pred_name, pred_trace)
      ↓
score + feedback
      ↓
reflection LM writes lesson / new instruction
      ↓
candidate added to frontier
      ↓
new candidate selected by pareto or current_best

GEPA docs formalize the metric interface more strictly than a conventional eval loop. It can ask for feedback for a specific predictor. To do this, not only "gold" and "pred" arrive in metric, but also "trace", "pred name", "pred trace". If you have a multi-module system, this allows you to give local feedback to the step that is currently being optimized.

def metric(gold, pred, trace=None, pred_name=None, pred_trace=None):
    if pred_name is None:
        return total_score

    if pred_name == "categories_module.predict":
        return dspy.Prediction(
            score=total_score,
            feedback="Missed one facility category and over-predicted another."
        )

The official GEPA API is not about the number of parameters, but how they change the search regime.

auto = "light" | "medium" | "heavy"
reflection minibatch size = 3 # by doc default
candidate_selection_strategy = "pareto" | "current_best"
reflection lm = strong LM # docs directly suggest a strong reflection model
max_full_evals / max_metric_calls = budget caps

GEPA advanced docs describe custom `instruction proposer` and custom `component selector` separately. That is, you can separately decide which components to update and who will write the new instructions. For most tasks, docs are advised to start with a default proposer, and customization to include only when default is already understood.

GEPA can be used as a batch inference-time search strategy. To do this, transmit “valset=trainset”, include “track stats=True” and “track best outputs=True”, and then use “detailed results” and frontier best outputs.

Not just compile prompts forever,
but also look for the best outputs on batch tasks,
When the price of additional search is justified by quality

The initial stage of DSPy should be super-pragmatic. The goal is not to “turn on compile right away,” but to build a system skeleton and understand its failure surface.

Day 1
  select an external contract
  Write a 1-module baseline on Predict or CoT
  Hands to run 15-20 representative cases

Day 2
  devset
  3-5 basic failure modes
  metric

Day 3
  baseline score
  Decide whether to decompose 2-3 submodules
  Then try the Optimizer.

Proper DSPy decomposition follows locally measured steps, not “how beautiful it sounds.” Each module must have a clear contract and a clear reason to exist. If a step cannot be separately explained or evaluated, it is usually a poor module boundary.

GEPA should be added not “when you heard about the new paper”, but when the team has already stabilized program boundaries and is able to give useful textual feedback. Prior to this, MIPROv2 or simple few-shot methods usually give more value per engineering hour.

You got a baseline?
You got devset?
Does metric correlate with human judgment?
Do you see local failure modes?
Can feedback be expressed in text?

If you say yes everywhere, GEPA is right.

DSPy should be implemented at maturity, not one big leap. If the team is building an LM system for the first time, it’s better to gradually add layers, otherwise complexity will explode before real quality comes along.

Stage 1 Predict / CoT + manual eval
Stage 2  Example / metric / Evaluate / devset
Stage 3  MIPROv2 / bootstrap optimizers
Stage 4  GEPA + predictor-level feedback
Stage 5  finetune / ensemble / prod hardening

DSPy is able to show LM history through 'inspect history', but the docs themselves honestly state the limitations: this log covers only LM calls and poorly reflects the role of retrievers, tools and custom modules. For serious work, you need tracing. Official tutorials systemically recommend MLflow autologging for compiles, evals and traces.

inspect_history
  + Quickly view text LM calls
  You can't see the whole pipeline.

MLflow tracing / autologging
  + compiles
  + evals
  + traces from program execution
  + explainability under optimization

Compiled DSPy programs can be saved as state-only JSON or as a whole program directory via `save program=True' and then downloaded via `dspy.load'. The cache documentation separately emphasizes: LM calls are cached by default, prompt caching can be enabled from supported providers, and if necessary, you can write your own dspy.clients. Cache'.

program code
compiled program state
dataset snapshot
metric version
LM / adapter config
optimizer run metadata

The official deployment tutorial describes two main paths: FastAPI for lightweight REST-serving and MLflow for more production-grade deployment with version and artifact management. Practically, this means: while the product is young, FastAPI is enough; as soon as you want to manage lifecycle compiled programs, tracing and versioning, move to MLflow-centered workflow.

DSPy covers runtime concerns directly in tutorials. Streamify includes token streaming and intermediate status streaming. Asyncify and async patterns are needed when an I/O-bound system serves many concurrent requests. Usage tracking is available at the prediction level by enabling "track usage=True". This is the final production contour.

Streaming
  streamify + StreamListener
  Only string fields are streamed

Async
  sync: exploration / research / simpler debugging
  async: concurrency / throughput / many requests

Usage
  dspy.configure(track_usage=True)
  prediction.get_lm_usage()

Before you sell, check:
  baseline frozen
  metric versioned
  traces working
  saved compiled artifacts
  cache policy decided
  deployment path chosen
  rollback story exists