Dropbox Dash uses an LLM judge to score query-document relevance across ranking, training-data generation, and offline evaluation. Moving that judge to a cheaper model exposed a familiar problem: a prompt tuned for one model did not transfer cleanly to another.
Dropbox first made the objective explicit. Human annotators rate relevance from one to five and explain their decisions. Normalized mean squared error measures disagreement between human and model scores. Malformed JSON is treated as fully incorrect because an answer that cannot be parsed is a production failure, even if its prose looks plausible.
The team then used DSPy’s GEPA optimizer to adapt the program from o3 to gpt-oss-120b. GEPA turns score gaps, human rationales, and model reasoning into structured feedback, revises the prompt, and repeats against the fixed dataset and metric.
The optimizer needed boundaries. Early candidates copied usernames or document phrases from training examples and sometimes changed the required rating scale. Dropbox added constraints against example-specific content and task-definition drift.
The reported result was a 45% NMSE reduction, from 8.83 to 4.86. Model adaptation fell from one or two weeks to one or two days, and the cheaper judge enabled 10 to 100 times more labels at the same cost. For a sensitive existing prompt, Dropbox used an instruction library so DSPy could select small approved bullets instead of rewriting everything.
That last pattern is broadly reusable. Treat prompt optimization like a series of small changes with tests. Define the metric, freeze the task, penalize structural failures, constrain the edit surface, and compare every candidate against a human-calibrated baseline.