What happened
Ai2 announced XYBench on November 13, 2026. The benchmark covers programming and everyday procedural questions, including cooking and home repair. It was developed with researchers at Carnegie Mellon, NVIDIA, and Johns Hopkins, and won the Best Resource Paper Award at EMNLP 2026. Ai2 describes an “XY problem” as a mismatch between the goal a person wants to achieve (X) and the approach they assume will work (Y), which is what they ask about. Its example is someone asking how to make frosting stick to cupcakes when the cupcakes are still too warm.
The dataset draws on 1,557 Stack Overflow questions, 184 from Stack Exchange, 102 manually curated examples, and 6,272 questions generated from WikiHow guides. For forum questions, Ai2 used high-scoring human answers and accepted answers where available to infer the intended goal and possible solutions, filtering out cases where answering the literal question was sufficient. For WikiHow, Ai2 says it used Sonnet 4.5 to generate questions around plausible misconceptions, then refined the questions so an expert could infer the underlying goal from context.
XYBench scores responses along three dimensions: whether they include a solution to the stated request and to the underlying goal; whether their emphasis falls on the stated request, the goal, or both; and whether they steer users toward the goal while discouraging or avoiding the mistaken approach. Ai2 used Claude Sonnet 4.5 as an automated judge. Two human reviewers independently rated 500 responses; their agreement with the judge ranged from 97.5–100% for whether a solution appeared, 84.6–93.3% for misconception identification, and 74.2–79.8% for pragmatic emphasis.
Ai2 evaluated Claude Sonnet 4.6, GPT-5, Gemini 3.1 Pro, Qwen3.5-9B, and DeepSeek-V4 Pro in a single-turn setting, with one response per question. On Stack Overflow questions, the models included solutions to the literal request in 75–86% of answers, versus 59–71% for the underlying goal. They focused primarily on the goal in only 35–41% of answers, while human reference answers did so 55% of the time. In a separate choice task, five evaluated models plus GPT-4.1 picked the goal-focused answer 71–84% of the time on Stack Overflow questions. In an oracle experiment that explicitly supplied the goal, Sonnet 4.6’s rate of including a goal solution rose from 59% to 67%, but its misconception-identification rate fell from 47% to 34%.
What the archive adds
New Runtime’s earlier coverage of Ai2’s StateBench, dated 2026-11-04 (StateBench), concerned a different evaluation task: tracking state changes across sequences. XYBench is a new benchmark, not a new fact about that release; together, the two releases show Ai2 publishing evaluations for distinct capabilities.
Trend signal
This does not yet fit a pattern New Runtime tracks.
Why it matters
For teams building assistants, a correct answer to the literal request is not necessarily useful. A user may be committed to a remedy that cannot solve the underlying problem; a system that follows that framing can appear compliant while wasting time or recommending unnecessary work. XYBench offers a way to test whether an assistant surfaces a plausible goal, explains why the proposed route may fail, and makes a better alternative central rather than incidental.
The reported gap between selecting a goal-focused answer and writing one is especially relevant to evaluation design. A model’s ability to judge two candidate responses should not be treated as evidence that it will produce the better response in open-ended use. Teams can use separate checks for goal-solution coverage, emphasis, and redirection, while treating benchmark results as a diagnostic rather than a guarantee of field performance.
Limits and uncertainty
Ai2’s evaluation uses one response per question in a single-turn setting, so it does not measure whether clarification, follow-up questions, or multi-turn context improve performance. The questions are English-language and mainly procedural; the authors identify other languages, cultures, and conversational settings as areas for extension. Human review covered 500 responses, and agreement was lower for pragmatic emphasis than for solution presence. The reported benchmark results do not establish how assistants behave across deployed products or user populations.
Skill to build now
Add a small set of “wrong route, real goal” cases to your assistant’s evaluation suite this week. For each, record the user’s literal request and the inferred objective separately, then score whether the response explains the mismatch and makes an effective alternative its main recommendation. Keep the literal-answer and goal-focused-answer metrics distinct.
Reader takeaway
XYBench’s central finding is not simply that models miss user intent: they can often recognize a goal-focused answer when shown one, yet still fail to make that answer the focus when generating a response. Assistant teams should test whether systems act on inferred goals, not only whether they can identify them.