{ "type": "new-runtime-knowledge-guide", "version": 1, "generated_at": "2026-07-23", "volume": 34, "slug": "dspy-gepa-reference", "title": "DSPy & GEPA: Implementation Cookbook", "description": "A big volume about how to really build systems on DSPy: all the key abstractions from official documentation, a built-in module map, data design and metrics, the choice of an optimizer, deep dive by GEPA and step-by-step rollout plans from the first baseline to production.", "track": { "slug": "prompting-and-adaptation", "title": "Prompting & Adaptation", "route": "https://newruntime.com/learn/tracks/prompting-and-adaptation/", "position": 3 }, "counts": { "sections": 8, "cards": 41 }, "routes": { "html": "https://newruntime.com/learn/dspy-gepa-reference/", "json": "https://newruntime.com/learn/dspy-gepa-reference.json" }, "sections": [ { "number": "01", "title": "Foundations – What is DSPy?", "intro": "The official idea of DSPy is simple, but the implications are big: stop encoding the logic of the system in rows of prompts and raise the level of abstraction to software components. The key to quality here is not “magical prompts”, but decomposition, metrics and compile loop.", "cards": [ { "number": "01", "title": "DSPy separates task from prompt strings", "tag": "foundation", "description": "In conventional prompt engineering, task signature, formatting, reasoning strategy, few-shot examples, and model-specific hacks are mixed into one fragile text. DSPy lays this out in layers and allows you to change the layer without rewriting the entire system. That is why it is useful not only for optimization, but also for normal architecture.", "blocks": [ { "kind": "code", "label": "Diagram", "language": "text", "value": "LM specific model and generation kwargs\nSignature What Comes In and What Should Come Out\nModule: Predict / CoT / ReAct / PoT\nAdapter as a signature turns into messages and back\nMetric is considered a good answer.\nOptimizer How to Find the Best Instructions / Demos / Weights\n\nBottom line: code controls flow and prompts become compiled artifact" }, { "kind": "list", "label": "When it has the maximum effect", "items": [ "There are several LM steps that need to be supported.", "The model or provider will change.", "Improved quality through eval loop", "For one throwaway prompt it can be overkill" ] } ] }, { "number": "02", "title": "First, control flow, then optimization.", "tag": "design rule", "description": "DSPy’s official programming overview doesn’t start with optimizers, but with task design and pipeline. The right start: first decide what inputs and outputs the system needs, where retrieval or tools are needed, and then think about how to compile prompt internals.", "blocks": [ { "kind": "code", "label": "Right order.", "language": "text", "value": "1. determine the external contract of the task\n2. make the simplest working pipeline\n3. collect characteristic dips in dev examples\n4. describe metric.\n5. Then turn on the optimizer." } ] }, { "number": "03", "title": "Lifecycle project on DSP", "tag": "lifecycle", "description": "The DSPy documentation clearly suggests a cyclical process: programme, assemble devset, define metrics, optimize, go back and change the pipeline itself if the metric or architecture is weak.", "blocks": [ { "kind": "code", "label": "Diagram", "language": "text", "value": "Explore manual runs, 10-20 cases\n ↓\nEvaluate devset + baseline metric\n ↓\nOptimize few-shot / instruction / weights\n ↓\nRe-design if metric is bad or pipeline is poorly decomposed\n ↓\nHarden observability, cache, save/load, deployment" } ] }, { "number": "04", "title": "Where in this picture sits GEPA", "tag": "gepa slot", "description": "GEPA does not replace a DSPy program, but sits on top of it as a reflective optimizer. Its task is not to invent the entire architecture for you, but to improve the text components of an existing program using score, feedback and execution traces.", "blocks": [ { "kind": "list", "label": "What should exist before GEPA", "items": [ "working DSPy program", "train/dev split", "A metric that can explain failure", "GEPA won't save chaotic system without eval loop" ] } ] } ] }, { "number": "02", "title": "Core Abstractions - Basic Bricks", "intro": "If you look at the official Learn + API, the central abstractions of DSPy are `LM`, `Signature`, `Module`, `Adapter`, `Example` and `Prediction`. Understanding these layers makes all optimizers and tutorials predictable.", "cards": [ { "number": "05", "title": "LM – One client over model, cache and metadata", "tag": "lm", "description": "dspy.LM specifies model and generation defaults. Documentation shows work with OpenAI, Anthropic, Gemini, Vertex, Databricks, Ollama, local OpenAI-compatible endpoints and other LiteLLM providers. LM is cached by default, stores call history, usage and cost metadata, and can switch via dspy.configure and dspy.context.", "blocks": [ { "kind": "code", "label": "Key Opportunities of LM", "language": "text", "value": "lm = dspy.LM(\"openai/gpt-4o-mini\")\ndspy.configure(lm=lm)\n\nwith dspy.context(lm=other_lm):\n ...\n\ngeneration kwargs: temperature / max_tokens / stop / cache\ndiverse rollouts: rollout_id + temperature>0\nhistory: prompt / messages / response / usage / cost / model\nreasoning models: model_type=\"responses\"" }, { "kind": "list", "label": "Practical conclusions", "items": [ "Maintain baseline LM globally through 'configure'", "Use 'context' for experiments and A/B", "Rollout id is required for controlled diversity", "Changing 'rollout id' to 'temperature=0' is useless" ] } ] }, { "number": "06", "title": "Signature – a declarative contract of a task", "tag": "signature", "description": "DSPy: Signature is a declarative specification for input/output behavior. It's about describing what to do, not manually dictating models how to ask.", "blocks": [ { "kind": "code", "label": "Signature shapes", "language": "python", "value": "Inline: \"question -> answer\"\nTyped inline: \"context: list[str], question -> answer\"\nClass-based:\nclass Emotion(dspy.Signature):\n \"\"\"Classify emotion.\"\"\"\n sentence: str = dspy.InputField()\n sentiment: Literal[...] = dspy.OutputField()" } ] }, { "number": "07", "title": "InputField / OutputField / type system", "tag": "fields", "description": "Class-based signatures are needed when you want to clearly define the meaning of the task: docstring, field descriptions, output limitations and complex types. DSPy supports not only 'str' but also 'int', 'bool', 'list[str]', 'dict[...]', 'optional[...]', custom types and special types like 'dspy'. Image' and dspy. History.", "blocks": [ { "kind": "list", "label": "When to switch from inline to class-based", "items": [ "You need \"Literal[...]\" or structured output.", "You want to clearly describe the inputs/limitations", "Do metric as a DSPy program", "Don't ditch the field name by hand too soon" ] } ] }, { "number": "08", "title": "Example & Prediction: Data and results in one semantics", "tag": "data", "description": "The basic data container in DSPy is ‘Example’. It is similar to dict but knows which fields are inputs. 'Prediction' is a special subclass 'Example' that returns modules. Metric, eval and optimization work with the same semantics of objects.", "blocks": [ { "kind": "code", "label": "Minimum data contract", "language": "text", "value": "x = dspy.Example(question=\"Q\", answer=\"A\").with_inputs(\"question\")\nx.inputs() # only inputs\nx.labels() # The rest as labels/metadata\n\npred = program(**x.inputs())\nmetric(x, pred)" } ] }, { "number": "09", "title": "Module - the basic class of programs in DSPy", "tag": "module", "description": "dspy.Module is an analogue of nn.Module for LM programs. Inside, you can store sub-modules, predictors, and your control flow. All serious DSPy systems, from multi-hop RAG to tool agents, end up with their own forward() classes.", "blocks": [ { "kind": "code", "label": "The skeleton of the custom program", "language": "python", "value": "class MyProgram(dspy.Module):\n def __init__(self):\n self.retrieve = dspy.Predict(\"question -> query\")\n self.answer = dspy.ChainOfThought(\"context, question -> answer\")\n\n def forward(self, question):\n query = self.retrieve(question=question).query\n context = search(query)\n return self.answer(context=context, question=question)" } ] }, { "number": "10", "title": "Adapter – the layer between the signature and the real LM API", "tag": "adapters", "description": "Officially, adapters are responsible for translating signature, demos, input data, and DSPy types into LM messages and parsing the response back to Prediction. They also manage history and function calls. By docs, `ChatAdapter' is used by default; `JSONAdapter' is good for native structured output and lower latency.", "blocks": [ { "kind": "table", "headers": [ "Adapter", "When to take", "What to remember" ], "rows": [ [ "ChatAdapter", "A universal start for almost everything", "More boilerplate tokens; docs say it can fallback in JSONAdapter" ], [ "JSONAdapter", "structured output + latency sensitive path", "Need a model with native structured-output support" ], [ "Selection process", "`dspy.configure(adapter=...)' or `with dspy.context(adapter=...) *", "First stabilize the signature, then change the adapter." ] ] } ] } ] }, { "number": "03", "title": "Modules – How DSP implements behavior", "intro": "Modules in DSPy abstract prompting technique, but remain compatible with any signature. This is what makes it portable: the same contract can be run through Predict, ChainOfThought, ReAct or ProgramOfThought without changing the external interfaces.", "cards": [ { "number": "11", "title": "Predict is the most fundamental module", "tag": "predict", "description": "The documentation says plainly: dspy. Predict is the basic module on top of which the rest are built. It simply displays input signature in outputs through the language model and does not extend the signature with additional reasoning fields.", "blocks": [ { "kind": "list", "label": "Best start for", "items": [ "classification", "field extraction", "short, normalized responses", "Hard reasoning is best to start with CoT" ] } ] }, { "number": "12", "title": "ChainOfThought - reasoning field over base signature", "tag": "cot", "description": "The API defines ‘ChainOfThought’ as a module that reasoning and step by step before predicting output. Learn docs further emphasize that in many cases, simply replacing 'Predict' with 'ChainOfThought' immediately improves quality.", "blocks": [ { "kind": "code", "label": "What's going on under the hood", "language": "text", "value": "signature: \"document -> summary\"\n↓\nCoT extends it to an additional reasoning field\n↓\nresponse.reasoning + response.summary" } ] }, { "number": "13", "title": "ReAct — managed tool-using agent", "tag": "react", "description": "dspy.ReAct implements the classic pattern \"Reasoning and Acting\": the model sees the list of tools, reason, decides to call the tool or complete the task. In DSPy, this pattern is generalized to arbitrary signatures, not just “question-> answer” chat agents.", "blocks": [ { "kind": "code", "label": "ReAct skeleton", "language": "text", "value": "agent = dspy.ReAct(\n \"question -> answer\",\n tools=[search_web, get_weather],\n max_iters=5,\n)" } ] }, { "number": "14", "title": "ProgramOfThought: When LM Should Write Code", "tag": "pot", "description": "The official learn page summarises `ProgramOfThought` is as follows: LM learns to output code, and the result of execution of this code determines the answer. The API specifies that the module runs Python programs and requires runtime to be interpreted.", "blocks": [ { "kind": "list", "label": "Best suited for", "items": [ "formalization", "symbolism", "tasks where the answer is more convenient to get through the code", "For conventional extraction, this is too complicated." ] } ] }, { "number": "15", "title": "CodeAct, RLM and execution-heavy family", "tag": "execution", "description": "This is a family for cases when text reasoning is not enough. CodeAct is inherited from both ReAct and ProgramOfThought: it is essentially a tool agent with code interpreter. RLM is a standalone experimental abstraction for very large contexts, where LM explores data through sandboxed Python REPL and recursive sub-LLM calls instead of directly loading giant context into prompt.", "blocks": [ { "kind": "table", "headers": [ "Module", "Idea", "When you need it." ], "rows": [ [ "CodeAct", "tools + code interpreter", "tool agent, where it is necessary not only to name the API, but also to process the result" ], [ "RLM", "recursive code exploration over large context", "Long documents, context rot, data exploration through code" ], [ "RLM caveat", "Experimental+: Sandbox Runtime", "Don’t take this first step if conventional RAG is already solving the problem." ] ] } ] }, { "number": "16", "title": "BestOfN, Refine, MultiChainComparison, Parallel — inference-time control", "tag": "control", "description": "Not all built-in modules are about reasoning style. Some of them manage the compute budget and execution policy. 'BestOfN' repeats the module N times and takes the best one by 'reward fn'. Refine does the same thing, but adds a feedback loop between attempts. MultiChainComparison brings several reasoning attempts into one final answer. Parallel is a utility for multithreaded pair launch (module, example).", "blocks": [ { "kind": "code", "label": "Diagram", "language": "text", "value": "BestOfN different rollout id choose the best answer\nRefine try to get feedback again\nMultiChainComparison several reasoning paths\nParallel batch evaluation / batch execution on streams" } ] } ] }, { "number": "04", "title": "Data & Evals – Optimization doesn’t work without it", "intro": "The official DSPy assessment philosophy is very correct: first devset, then metric, then baseline. Intermediate labels are almost never required. If you don’t know how to reliably say that there is a “good output”, no optimizer will save the situation.", "cards": [ { "number": "17", "title": "Data model — inputs, intermediates, final labels", "tag": "data", "description": "Learn docs emphasize three types of values on example: inputs, intermediate labels, and final label. Important practical conclusion: DSPy can often be used without intermediate labels and even without final labels, if the metric is able to measure quality by inputs and output.", "blocks": [ { "kind": "list", "label": "Implications for Datasets Design", "items": [ "Do not mark all the internal steps in advance.", "Start with at least a few representative inputs.", "Labels are needed where metrics depend on them.", "Do not confuse metadata with inputs in 'with inputs()'" ] } ] }, { "number": "18", "title": "Metric design – the heart of DSPy and especially GEPA", "tag": "metric", "description": "The DSPy documentation literally says: framework machine-learning-like, so automatic metric is mandatory for both evaluation and optimization. For simple tasks, metric can be an ‘accuracy’/‘exact match’; for most production tasks, metric must be a small DSPy program or a function that checks several properties of long-form output.", "blocks": [ { "kind": "code", "label": "Normal metrics in DSPy", "language": "python", "value": "def metric(example, pred, trace=None):\n score_1 = ...\n score_2 = ...\n\n if trace is None:\n return (score_1 + score_2) / 2.0 # evaluation / optimization\n else:\n return score_1 and score_2 # bootstrap demos" }, { "kind": "list", "label": "Practice", "items": [ "Make metric simple first", "Iteratively improve it with the system", "Long-form output almost always requires composite metrics.", "Don’t optimise on a judge you don’t trust." ] } ] }, { "number": "19", "title": "Evaluate utility and baseline before optimizer", "tag": "evaluate", "description": "Before running a compile, you need to understand the baseline. Docs offer either a regular Python loop or a built-in dspy.evaluate.Evaluate that helps with multi-thread eval, progress bar and sample predictions table.", "blocks": [ { "kind": "code", "label": "Minimum baseline loop", "language": "text", "value": "evaluator = Evaluate(\n devset=devset,\n num_threads=1,\n display_progress=True,\n display_table=5,\n)\nevaluator(program, metric=metric)" } ] }, { "number": "20", "title": "How much data is needed and how to divide train / val / test", "tag": "splits", "description": "The official DSPy heuristics are good to start with: already 20 dev examples are useful, 200 go far; for optimizers, you can get value from 30 examples, but it is better to aim for 300+. For many, prompt optimizers docs recommend an atypical split `20% train / 80% val` because prompt optimization is retrained on a small trainset. For GEPA, the more classic ML approach is recommended: maximize the trainset and keep the val large enough to reflect downstream distribution.", "blocks": [ { "kind": "table", "headers": [ "Phase", "Practice", "Why?" ], "rows": [ [ "Exploration", "20+ dev examples", "catching obvious failure modes" ], [ "General prompt optimizers", "20% train / 80% val", "More stable validation, less overfit" ], [ "GEPA", "More train, compact but representative val", "reflection learns on richer training signal" ], [ "Production", "mandatory held-out test", "There is no real uplift." ] ] } ] } ] }, { "number": "05", "title": "Optimizer Map: What DSPy can compile", "intro": "According to official optimization docs, DSPy optimizers are divided not by brands, but by the type of object being changed: few-shot demos, instructions, model weight, program transformations and meta-sequences. This is a handy map to choose from, not just a list of API names.", "cards": [ { "number": "21", "title": "Families of optimizers in DSPy", "tag": "optimizer map", "description": "", "blocks": [ { "kind": "table", "headers": [ "Family", "What changes", "Top names in the API" ], "rows": [ [ "Automatic few-shot learning", "demonstration", "LabeledFewShot, BootstrapFewShot, BootstrapFewShotWithRandomSearch, BootstrapRS, KNNFewShot" ], [ "Automatic instruction optimization", "instructions and sometimes demos", "COPRO, MIPROv2, SIMBA, GEPA" ], [ "Automatic finetuning", "weights of underlying LMs", "BootstrapFinetune" ], [ "Program transformations", "execution policy", "Ensemble" ], [ "Meta-optimizers", "sequence of strategies", "BetterTogether" ], [ "Other specialized names in current API", "special retrieval / rule / similarity helpers", "InferRules, KNN, KNNFewShot" ] ] } ] }, { "number": "22", "title": "Few-shot family - when the main problem is good demos", "tag": "few-shot", "description": "Docs describe this family as an automatic signature extension of optimized examples. LabeledFewShot just picks up labeled demos. BootstrapFewShot uses the teacher program to generate demonstrations and filters them with metrics. 'BootstrapFewShotWithRandomSearch' amplifies search, and 'KNNFewShot' picks up close examples before bootstrap.", "blocks": [ { "kind": "list", "label": "When to take", "items": [ "Dataset is small but representative", "It depends on examples-in-context.", "If you want strict 0-shot, see MIPROv2/GEPA." ] } ] }, { "number": "23", "title": "Instruction family — COPRO, MIPROv2, SIMBA, GEPA", "tag": "instructions", "description": "Copro makes coordinate-ascent over instructions. MIPROv2 combines instruction generation and few-shot search through Bayesian Optimization. SIMBA seeks complex examples through stochastic minibatches and generates self-reflective rules. GEPA analyzes trajectories and textual feedback. This family is worth taking when the main problem is not in the raw control flow, but in how the subtasks are formulated.", "blocks": [] }, { "number": "24", "title": "BootstrapFinetune, BetterTogether, Ensemble", "tag": "weights", "description": "‘BootstrapFinetune’ distill-it prompt-based DSPy program in weight updates: the program remains the same in structure, but the steps begin to be performed by finetuned models. BetterTogether combines prompt optimization and finetuning in a sequence like ‘prompt → weight → prompt’. Ensemble uses DSPy programs as one program transformation layer.", "blocks": [ { "kind": "list", "label": "Where appropriate.", "items": [ "There is a strong prompt baseline, but you need a cheaper runtime.", "latency/cost at scale", "Do not jump in finetune until metric and prompts are stabilized." ] } ] }, { "number": "25", "title": "Official heuristics – which optimizer to take first", "tag": "selection", "description": "DSPy docs themselves give the starting heuristics. They are not a substitute for experimentation, but work well as the first decision tree.", "blocks": [ { "kind": "code", "label": "Diagram", "language": "text", "value": "about 10 examples\n → BootstrapFewShot\n\n50+ examples\n → BootstrapFewShotWithRandomSearch\n\n0-shot instruction optimization\n MIPROv2 in 0-shot configuration\n\n200+ examples and more budget optimization\n → MIPROv2\n\nThere is already a strong large model and you need an efficient small runtime.\n → BootstrapFinetune\n\nTrack-level feedback and rich failure analysis\n → GEPA" } ] } ] }, { "number": "06", "title": "GEPA Deep Dive – Reflective Optimization as It Is", "intro": "GEPA is not just another prompt optimizer, but a reflective optimizer. It works on execution traces, can use predictor-level feedback, and has its own data split philosophy. For complex modular programs, this is often a qualitatively different mode of improvement compared to scalar-only search.", "cards": [ { "number": "26", "title": "GEPA compile loop — score, traces, reflection, frontier", "tag": "core loop", "description": "According to official GEPA docs and paper: the optimizer takes the seed candidate, runs it on the training signal, receives traces and numeric score, reflects what worked and what did not, generates a new instruction and selects candidates through the Pareto-style frontier. This is no longer a blind search for scalar reward, but a search for textual diagnoses.", "blocks": [ { "kind": "code", "label": "Diagram", "language": "text", "value": "seed DSPy program\n ↓\nrun on minibatch / trainset\n ↓\nmetric(example, pred, trace, pred_name, pred_trace)\n ↓\nscore + feedback\n ↓\nreflection LM writes lesson / new instruction\n ↓\ncandidate added to frontier\n ↓\nnew candidate selected by pareto or current_best" } ] }, { "number": "27", "title": "Metric contract in GEPA — `trace`, `pred_name`, `pred_trace`", "tag": "feedback api", "description": "GEPA docs formalize the metric interface more strictly than a conventional eval loop. It can ask for feedback for a specific predictor. To do this, not only \"gold\" and \"pred\" arrive in metric, but also \"trace\", \"pred name\", \"pred trace\". If you have a multi-module system, this allows you to give local feedback to the step that is currently being optimized.", "blocks": [ { "kind": "code", "label": "Metric shape for GEPA", "language": "python", "value": "def metric(gold, pred, trace=None, pred_name=None, pred_trace=None):\n if pred_name is None:\n return total_score\n\n if pred_name == \"categories_module.predict\":\n return dspy.Prediction(\n score=total_score,\n feedback=\"Missed one facility category and over-predicted another.\"\n )" }, { "kind": "list", "label": "Why is it important?", "items": [ "multi-head extraction tasks", "RAG with separate retrieval and answer stages", "Repeated local failure modes agents", "If feedback is only global, GEPA will learn rougher" ] } ] }, { "number": "28", "title": "The main hyperparameters of GEPA", "tag": "knobs", "description": "The official GEPA API is not about the number of parameters, but how they change the search regime.", "blocks": [ { "kind": "code", "label": "What's the real first spin?", "language": "text", "value": "auto = \"light\" | \"medium\" | \"heavy\"\nreflection minibatch size = 3 # by doc default\ncandidate_selection_strategy = \"pareto\" | \"current_best\"\nreflection lm = strong LM # docs directly suggest a strong reflection model\nmax_full_evals / max_metric_calls = budget caps" } ] }, { "number": "29", "title": "GEPA Advanced Extensions", "tag": "advanced", "description": "GEPA advanced docs describe custom `instruction proposer` and custom `component selector` separately. That is, you can separately decide which components to update and who will write the new instructions. For most tasks, docs are advised to start with a default proposer, and customization to include only when default is already understood.", "blocks": [ { "kind": "list", "label": "When to Really Climb Advanced", "items": [ "Multiple components are needed and selective update is required.", "non-standard instruction search format", "Default proposer is almost always sufficient" ] } ] }, { "number": "30", "title": "GEPA as a batch inference-time search", "tag": "search", "description": "GEPA can be used as a batch inference-time search strategy. To do this, transmit “valset=trainset”, include “track stats=True” and “track best outputs=True”, and then use “detailed results” and frontier best outputs.", "blocks": [ { "kind": "code", "label": "When it's useful.", "language": "text", "value": "Not just compile prompts forever,\nbut also look for the best outputs on batch tasks,\nWhen the price of additional search is justified by quality" } ] }, { "number": "31", "title": "When GEPA wins and when it breaks", "tag": "failure modes", "description": "", "blocks": [ { "kind": "table", "headers": [ "Script", "Verdict", "Why" ], "rows": [ [ "Multi-module system with rich traces", "GEPA is very appropriate.", "You can locally explain which predictor is wrong and how." ], [ "enterprise extraction with predictor-level feedback", "fit", "The official tutorial shows such a regime." ], [ "scalar metric without explanation", "Yes, but the value is lower.", "reflection impoverished; search gets closer to blind optimization" ], [ "failure", "GEPA won't.", "It polishes text components rather than redesigning pipelines." ], [ "No representative data", "result will be fragile", "Reflection learns from misallocation of tasks" ] ] } ] } ] }, { "number": "07", "title": "Cookbook – How to Start a Project Right", "intro": "The most common mistake DSPy teams make is arguing too early about optimizers and designing the data/metric layer too late. Below is a plan for how to decompose the system in the beginning so that you can actually get sustainable quality through DSPy and GEPA.", "cards": [ { "number": "32", "title": "Implementation plan 0: 1: First 2-3 days of the project", "tag": "start plan", "description": "The initial stage of DSPy should be super-pragmatic. The goal is not to “turn on compile right away,” but to build a system skeleton and understand its failure surface.", "blocks": [ { "kind": "code", "label": "Diagram", "language": "sql", "value": "Day 1\n select an external contract\n Write a 1-module baseline on Predict or CoT\n Hands to run 15-20 representative cases\n\nDay 2\n devset\n 3-5 basic failure modes\n metric\n\nDay 3\n baseline score\n Decide whether to decompose 2-3 submodules\n Then try the Optimizer." }, { "kind": "list", "label": "Anti-start pattern", "items": [ "7-step agent to baseline", "Build datasets without a list of failure modes", "Run GEPA before the first fair eval" ] } ] }, { "number": "33", "title": "How to Decompose a System in DSPy From the Beginning", "tag": "decomposition", "description": "Proper DSPy decomposition follows locally measured steps, not “how beautiful it sounds.” Each module must have a clear contract and a clear reason to exist. If a step cannot be separately explained or evaluated, it is usually a poor module boundary.", "blocks": [ { "kind": "table", "headers": [ "A sign.", "Good module.", "Bad module." ], "rows": [ [ "Contract", "clear input/output", "do-it-yourself" ], [ "Observability", "You can look at the track and see what happened.", "Error only appears at the end of the program." ], [ "Optimization", "Predictor-level feedback", "You cannot localize what to improve." ] ] } ] }, { "number": "34", "title": "Pattern cookbook – where to start for typical tasks", "tag": "patterns", "description": "", "blocks": [ { "kind": "table", "headers": [ "Task type", "Starter module", "First metric.", "First Optimizer" ], "rows": [ [ "Classification / routing", "Predict", "accuracy / macro-F1", "BootstrapFewShot or MIPROv2" ], [ "Structured extraction", "CoT + class-based signature", "field accuracy / exact match", "GEPA if you have text feedback by field" ], [ "RAG answerer", "Module(retrieve → answer)", "answer correctness + grounding", "MIPROv2, then GEPA for multi-stage feedback" ], [ "Tool agent", "ReAct", "task success + tool correctness", "MIPROv2 or GEPA if trace-level errors are visible" ], [ "Formal reasoning / math", "ChainOfThought or ProgramOfThought", "exact answer", "GEPA / MIPROv2" ], [ "Huge context exploration", "RLM", "answer accuracy + exploration efficiency", "only after baseline; not the first step of the project" ], [ "Cheap production runtime", "First, a strong prompted program", "task success + latency/cost", "BootstrapFinetune / BetterTogether" ] ] } ] }, { "number": "35", "title": "When to add GEPA to the roadmap", "tag": "adoption", "description": "GEPA should be added not “when you heard about the new paper”, but when the team has already stabilized program boundaries and is able to give useful textual feedback. Prior to this, MIPROv2 or simple few-shot methods usually give more value per engineering hour.", "blocks": [ { "kind": "code", "label": "Normal readiness check", "language": "text", "value": "You got a baseline?\nYou got devset?\nDoes metric correlate with human judgment?\nDo you see local failure modes?\nCan feedback be expressed in text?\n\nIf you say yes everywhere, GEPA is right." } ] }, { "number": "36", "title": "Rollout on team maturity", "tag": "maturity", "description": "DSPy should be implemented at maturity, not one big leap. If the team is building an LM system for the first time, it’s better to gradually add layers, otherwise complexity will explode before real quality comes along.", "blocks": [ { "kind": "code", "label": "Diagram", "language": "text", "value": "Stage 1 Predict / CoT + manual eval\nStage 2 Example / metric / Evaluate / devset\nStage 3 MIPROv2 / bootstrap optimizers\nStage 4 GEPA + predictor-level feedback\nStage 5 finetune / ensemble / prod hardening" } ] }, { "number": "37", "title": "The most common architectural errors when implementing DSPy", "tag": "pitfalls", "description": "", "blocks": [ { "kind": "code", "label": "Don't do that.", "language": "sql", "value": "Write a huge agent immediately.\n First, find out if 1-2 modules and retrieval are enough.\n\nMeasure only the aggregate score\n not visible, retrieval, reasoning or formatting breaks the task\n\nIt's too early to sign words with your hands.\n First let metric and optimizer show where the problem is.\n\nRun GEPA without text feedback\n The meaning of reflective optimization is lost.\n\nConfuse devset and testset\n Optimization becomes self-deception" } ] } ] }, { "number": "08", "title": "Ops & Production to bring the system to work", "intro": "DSPy docs cover not only the modeling layer, but also the real operational contour: tracing, optimizer tracking, save/load, cache, deployment, streaming, async. This is important because without observability, the compile loop quickly turns into a black box.", "cards": [ { "number": "38", "title": "Debugging & Observability – ‘Inspect history’ and Tracing", "tag": "observability", "description": "DSPy is able to show LM history through 'inspect history', but the docs themselves honestly state the limitations: this log covers only LM calls and poorly reflects the role of retrievers, tools and custom modules. For serious work, you need tracing. Official tutorials systemically recommend MLflow autologging for compiles, evals and traces.", "blocks": [ { "kind": "code", "label": "Diagram", "language": "sql", "value": "inspect_history\n + Quickly view text LM calls\n You can't see the whole pipeline.\n\nMLflow tracing / autologging\n + compiles\n + evals\n + traces from program execution\n + explainability under optimization" }, { "kind": "list", "label": "When to include", "items": [ "From the first day of work optimization", "Required for GEPA and agents", "without traces team argues about quality blindly" ] } ] }, { "number": "39", "title": "Save / load / cache - reproducibility of experiments", "tag": "artifacts", "description": "Compiled DSPy programs can be saved as state-only JSON or as a whole program directory via `save program=True' and then downloaded via `dspy.load'. The cache documentation separately emphasizes: LM calls are cached by default, prompt caching can be enabled from supported providers, and if necessary, you can write your own dspy.clients. Cache'.", "blocks": [ { "kind": "code", "label": "What to keep as artifacts", "language": "text", "value": "program code\ncompiled program state\ndataset snapshot\nmetric version\nLM / adapter config\noptimizer run metadata" } ] }, { "number": "40", "title": "Deployment – FastAPI for simple, MLflow for more adult circuit", "tag": "deploy", "description": "The official deployment tutorial describes two main paths: FastAPI for lightweight REST-serving and MLflow for more production-grade deployment with version and artifact management. Practically, this means: while the product is young, FastAPI is enough; as soon as you want to manage lifecycle compiled programs, tracing and versioning, move to MLflow-centered workflow.", "blocks": [ { "kind": "list", "label": "Quick heuristics", "items": [ "prototype / internal tool → FastAPI", "managed experiments / versioning → MLflow", "Recruitment without saved artifacts = pain in a week" ] } ] }, { "number": "41", "title": "Streaming, async, usage tracking and production checklist", "tag": "runtime", "description": "DSPy covers runtime concerns directly in tutorials. Streamify includes token streaming and intermediate status streaming. Asyncify and async patterns are needed when an I/O-bound system serves many concurrent requests. Usage tracking is available at the prediction level by enabling \"track usage=True\". This is the final production contour.", "blocks": [ { "kind": "code", "label": "Diagram", "language": "text", "value": "Streaming\n streamify + StreamListener\n Only string fields are streamed\n\nAsync\n sync: exploration / research / simpler debugging\n async: concurrency / throughput / many requests\n\nUsage\n dspy.configure(track_usage=True)\n prediction.get_lm_usage()\n\nBefore you sell, check:\n baseline frozen\n metric versioned\n traces working\n saved compiled artifacts\n cache policy decided\n deployment path chosen\n rollback story exists" } ] } ] } ] }