They tested Jev 1.13, Perplexity Decider 1.1 27B, Liquid d1 and d1-ultra, Cloudflare Clef Flash, and OpenAI Decisions using GPT-6 Luna.
The test used 100 human-reviewed traces from an apartment-leasing agent. Each model assigned a failure probability to traces, including failed bookings and missed handoffs to a human. The authors recommend tuning the binary pass/fail threshold on labeled data rather than accepting the default 0.50, using stratified five-fold cross-validation and explicit precision and recall minimums.
With an 80% minimum for both precision and recall, Perplexity, Liquid d1-ultra, and OpenAI Decisions met the stated requirement in this experiment. After tuning, Perplexity reached F1 0.79 at 0.82 cents per 100 evaluation runs; OpenAI Decisions reached F1 0.87 at 3.72 cents per 100 runs.
The source does not establish that one model is best across other datasets or evaluation tasks. It also excludes latency from the comparison and bases its results on a single set of 100 traces.