The thesis is Audience-level prediction needs evaluation against measured distributions, not a fluent account of what people might do. The benchmark reports 1,199 tasks drawn from surveys of 1,244 U.S. adults, with 239 public tasks and 960 held out. Its metric is 100 × (1 − total variation distance), comparing predicted and observed answer distributions.
The mechanism is to compare each model's predicted answer distribution with survey ground truth for a defined cohort, rather than grade a free-form explanation. Systems receive a closed-form question tied to an audience and predict percentages across fixed options. The published table lists six LLM scores between 76.2 and 80.6; Rehearsals reports 94.1 for Engram 1.0. The source says base models ran through Kaggle's harness while Engram used Rehearsals' in-house evaluation system with the same case and metric setup. Why now: The benchmark makes a gap between conversational plausibility and audience-level prediction legible as a scored evaluation, including in product and marketing questions. The organizers also report LLM top-answer accuracy of 41% on product questions and 74% on brand-perception questions.
The first implication is that teams should not treat a model's confident audience persona or preference summary as equivalent to measured user research. The evidence boundary is that These are organizer-reported results; base models used Kaggle's harness and Engram used Rehearsals' in-house harness, albeit with the same case and metric setup. The page does not establish independent replication, representativeness beyond the U.S. panel, or predictive validity for every business decision. The first watch condition is whether independent evaluators reproduce the rankings and whether performance generalizes across new populations, survey modes, and decision domains. The next useful evidence would therefore be an independent replication on additional populations and decision settings.