---
schema_version: "newruntime-agent-readable-v0.2"
type: "post"
stable_id: "post:dropbox-dspy-relevance-judge"
slug: "dropbox-dspy-relevance-judge"
title: "Dropbox Uses DSPy To Move A Relevance Judge To Cheaper Models"
description: "Dropbox Dash optimized a human-calibrated relevance judge with DSPy, reducing disagreement, shortening model migration, and adding structural-output reliability to the objective."
retrieval_nugget: "Dropbox Dash optimized a human-calibrated relevance judge with DSPy, reducing disagreement, shortening model migration, and adding structural-output reliability to the objective. Dropbox Dash uses an LLM judge to score query-document relevance across ranking, training-data generation, and offline evaluation. Moving that judge to a cheaper model exposed a familiar problem: a prompt tuned for one model did not transfer cleanly to."
status: "published"
published_at: "2026-08-01"
updated_at: "2026-08-01"
record_date: "2026-08-01"
date_kind: "published_at"
topics: ["dspy","evals","llm-judges","model-economics"]
source_urls: ["https://dropbox.tech/machine-learning/optimizing-dropbox-dash-relevance-judge-with-dspy","https://github.com/stanfordnlp/dspy"]
visuals: [{"id":"dropbox-dspy-relevance-judge","kind":"editorial-diagram","role":"hero","src":"https://newruntime.com/images/posts/dropbox-dspy-relevance-judge.webp","alt":"Hand-drawn optimization loop where human relevance labels and JSON validity feed DSPy, which adapts a judge prompt for cheaper target models under regression gates.","caption":"Dropbox optimizes judge behavior against human agreement and parseable output, then constrains changes when production stability matters.","credit":"New Runtime synthesis from Dropbox.Tech","source_url":"https://dropbox.tech/machine-learning/optimizing-dropbox-dash-relevance-judge-with-dspy","generated_with":"gemini-3.1-flash-image","width":1600,"height":900,"legend":[{"label":"Objective","description":"Human 1-to-5 ratings, explanations, and malformed-output penalties define success."},{"label":"Optimize","description":"DSPy analyzes disagreement and searches for prompt changes tailored to the target model."},{"label":"Constrain","description":"Guardrails prevent example copying, rating-scale drift, and unsafe full-prompt rewrites."}]}]
telegram_message_id: 2910
telegram_url: "https://t.me/qwgai/2910"
telegram_message_ids: [2910,2911]
telegram_delivery_mode: "text_then_media"
telegram_media_url: "https://t.me/qwgai/2911"
routes: {"html":"https://newruntime.com/posts/dropbox-dspy-relevance-judge/","markdown":"https://newruntime.com/posts/dropbox-dspy-relevance-judge.md","json":"https://newruntime.com/posts/dropbox-dspy-relevance-judge.json"}
source_format: "markdown"
---

# Dropbox Uses DSPy To Move A Relevance Judge To Cheaper Models

## Retrieval answer

Dropbox Dash optimized a human-calibrated relevance judge with DSPy, reducing disagreement, shortening model migration, and adding structural-output reliability to the objective. Dropbox Dash uses an LLM judge to score query-document relevance across ranking, training-data generation, and offline evaluation. Moving that judge to a cheaper model exposed a familiar problem: a prompt tuned for one model did not transfer cleanly to.

Dropbox Dash uses an LLM judge to score query-document relevance across ranking, training-data generation, and offline evaluation. Moving that judge to a cheaper model exposed a familiar problem: a prompt tuned for one model did not transfer cleanly to another.

Dropbox first made the objective explicit. Human annotators rate relevance from one to five and explain their decisions. Normalized mean squared error measures disagreement between human and model scores. Malformed JSON is treated as fully incorrect because an answer that cannot be parsed is a production failure, even if its prose looks plausible.

The team then used DSPy's GEPA optimizer to adapt the program from o3 to `gpt-oss-120b`. GEPA turns score gaps, human rationales, and model reasoning into structured feedback, revises the prompt, and repeats against the fixed dataset and metric.

The optimizer needed boundaries. Early candidates copied usernames or document phrases from training examples and sometimes changed the required rating scale. Dropbox added constraints against example-specific content and task-definition drift.

The reported result was a 45% NMSE reduction, from 8.83 to 4.86. Model adaptation fell from one or two weeks to one or two days, and the cheaper judge enabled 10 to 100 times more labels at the same cost. For a sensitive existing prompt, Dropbox used an instruction library so DSPy could select small approved bullets instead of rewriting everything.

That last pattern is broadly reusable. Treat prompt optimization like a series of small changes with tests. Define the metric, freeze the task, penalize structural failures, constrain the edit surface, and compare every candidate against a human-calibrated baseline.