Dropbox Dash is a workplace assistant that answers questions over company documents, messages, meetings, and connected apps through a chat agent. On 25 June 2026, Simran Jumani and Dmitriy Meyerzon published How we used DSPy to turn AI evaluations into better responses in Dash chat on dropbox.tech. The article argues that LLM-as-judge evaluations of the full agent trajectory (intent, retrieval, tool use, grounding, and task completion) can drive prompt updates with DSPy rather than remaining dashboard metrics.
That claim is useful to builders because most agent teams still edit system prompts by hand after reading traces. Dropbox says it closed a loop: a small set of human labels improved the judges, the judges scored candidate agent prompts at scale, and GEPA used those scores plus structured judge reasoning to propose the next prompt. Relative to the 11 August 2026 newsroom scan the post was about seven weeks old, and it is distinct from Dropbox's earlier 2026 DSPy work on search-relevance judges. The quality numbers are first-party on internal chats, so they are an engineering case study, not audited customer telemetry.
What we checked
- What concrete DSPy/evaluation method did Dropbox use and what measured response-quality gains were reported? confirmed. Dropbox used human-labeled internal chat traces on five 1-5 dimensions, DSPy GEPA and MIPROv2 to calibrate LLM judges, then GEPA with offline counterfactual replay of historical internal chats to optimize the agent system prompt. It reported a 26 percent reduction in incomplete answers, a 13 percent reduction in missed key aspects within 24 hours of launch, 5.4 percent lower total token usage, and 9.8 percent shorter average completions. [1]
- Is the article newly published or an older item newly discovered by the scan? confirmed. The post is dated 25 June 2026 on the article page and on the dropbox.tech Machine Learning index. The newsroom assessed it on 11 August 2026, about seven weeks later, so it was a newly published 2026 engineering item rather than an old post rediscovered years later. As of 23 October 2026 it is about four months old. [1][2]
- Did Dropbox apply DSPy to the Dash chat agent itself, beyond earlier relevance-judge work? confirmed. Yes. A January 2026 VP talk said DSPy was already used for LLM-as-judge work across more than 30 Dash prompts, and a March 2026 post optimized the search-relevance judge. The June post uses GEPA to optimize the chat agent's system prompt and attributes launch results to that change. [1][4][5]
- Have independent outlets or the official DSPy use-cases list corroborated the June 2026 chat-agent metrics? unresolved. InfoQ independently covered Dropbox's January 2026 context-engine and DSPy prompt-management claims on 18 February 2026, not the June chat-agent percentages. The official DSPy production use-cases page still lists Dropbox only for the March 2026 relevance-judge blog and talk. No independent measurement of the 26, 13, 5.4, or 9.8 percent figures was found. [10][9]
What we found
- The dropbox.tech Machine Learning index lists the article as a featured post dated 25 June 2026, matching the byline of Simran Jumani and Dmitriy Meyerzon. The newsroom flagged the story on 11 August 2026, so this was a newly published 2026 engineering post, not an old item rediscovered years later. As of 23 October 2026 the article is about four months old and remains the primary public source for the Dash chat DSPy results. [1][2]
- Dropbox applied DSPy in two stages. First it optimized LLM judges against human-labeled examples so scores better matched human judgment. Then it used those judges to optimize the chat agent's system prompt. For judges it names GEPA and MIPROv2. For the agent it describes a GEPA round in which a candidate prompt is replayed on representative historical Dropbox internal chats and scored by the evaluation pipeline, with scores and structured judge reasoning as the next-round feedback. [1]
- Human evaluation used sampled internal chats plus trace logs, scored on five dimensions: user intent following, semantic relevance, tool calling, instruction following, and context selection. Several metrics used a 1-5 scale, plus reasoning notes and failure codes such as stale evidence, missing context, unsupported claims, incomplete coverage, or failure to personalize. The post does not report how many chats were labeled or inter-annotator agreement. A February 2026 Dash labeling post, also by Dmitriy Meyerzon, states that human review uses limited non-sensitive internal datasets and that no customer data is reviewed by humans. [1][3]
- The team says prompt optimization used to be manual and was replaced by labeled examples, production-aligned scorers, and offline counterfactual replay. In the first two weeks the loop generated six prompt candidates automatically, compared with five manual prompt changes in the prior month. Launch comparison against the production prompt on the same examples reported a 26 percent reduction in incomplete answers and a 13 percent reduction in missed key aspects, with improvements appearing within the first 24 hours, plus 5.4 percent lower total token usage and 9.8 percent shorter average completions. Candidate edits were constrained to small, targeted instruction updates, with automated checks for prompt structure, completeness, caching behavior, and size limits. The post says statistical tests were used but does not name the test, sample size, or confidence interval. [1]
- The June chat write-up continues a 2026 Dash eval program rather than appearing from nowhere. In January, VP Josh Clemm said Dash had over 30 prompts and that DSPy helped LLM-as-judge work, with plans to try it across ingest, offline evals, and the online agentic platform. In March, Eric Wang and Dmitriy Meyerzon reported that GEPA cut relevance-judge NMSE 45 percent (8.83 to 4.86) when adapting from OpenAI o3 to gpt-oss-120b, and that MIPROv2 on gemma-3-12b improved NMSE from 46.88 to 17.26 while cutting invalid JSON from more than 40 percent of responses to 9 invalid outputs against 847 valid ones. Those earlier metrics are about judges and ranking labels, not the June incomplete-answer figures. Official DSPy docs match the named algorithms: MIPROv2 jointly searches instructions and few-shot demos with Bayesian optimization; GEPA (Genetic-Pareto), from Agrawal et al. 2025 (arXiv:2507.1 [4][5][6][7][8][9]
What it means
Builders should copy the loop, not the headline percentages. The reusable pattern is: label a small internal set with a multi-dimension rubric and failure codes; optimize the judge until it tracks those labels; then optimize the agent prompt against that judge on replayed production traces, with hard limits on how much the prompt may change. DSPy GEPA is a fit when the metric can return text explaining why a trajectory failed, which Dropbox did with structured judge notes. MIPROv2 remains relevant for judge format reliability, as in the March gemma JSON result.
Do not treat 26 percent fewer incomplete answers as a transferable benchmark. The post does not disclose label volume, customer versus employee traffic, or the statistical test. Earlier Dropbox posts state that humans review only non-sensitive internal data. Ship this pattern behind eval gates, hold out a test set the optimizer never sees, and keep human spot-checks on the judge. Dropbox's October 2025 conversational-eval playbook already described that discipline: about 150 canonical queries judged on every pull request, plus live-traffic scoring.
Still open
- How large was the human-labeled chat set, and what was inter-annotator agreement on the five scoring dimensions?
- Which models served as judge versus chat agent in the June work, and was the agent optimized on the same o3 or gpt-oss-120b stack described in the March relevance-judge post?
- Were the 26 percent and 13 percent launch figures measured on employee dogfood traffic, customer production traffic, or the same internal replay set used for optimization?
- Which statistical tests were used, and what were sample size and confidence intervals?