---
type: "post"
id: "how-to-use-decision-models-for-evals-this-cdc2f3"
slug: "how-to-use-decision-models-for-evals-this-cdc2f3"
title: "How to use decision models for evals: This cookbook walks you through picking between models like Jev, Decider, Clef, Liquid d1, and OpenAI Decisions instead of just rolling with the default 0.5 th..."
description: "Hamel Husain and Antaripa Saha published a comparison of decision models for automated LLM evaluations."
retrieval_nugget: "Hamel Husain and Antaripa Saha published a comparison of decision models for automated LLM evaluations."
published_at: "2026-10-22"
updated_at: "2026-10-22"
record_date: "2026-10-22"
date_kind: "published_at"
topics: ["coding-agents","evals"]
entities: ["Jev","Perplexity Decider","Liquid d1","Cloudflare Clef","OpenAI Decisions","GPT-6 Luna"]
related_patterns: ["verification-bandwidth-is-the-scarce-resource"]
novelty_label: "new_fact"
what_changed: "Hamel and Saha’s cookbook adds hands-on threshold tuning and cost/performance evaluation of Jev, Decider, Clef, Liquid d1 and OpenAI Decisions on 100 leasing-agent traces."
prior_refs: ["memory:9ffca1b6-50b0-86bf-acfd-1ca15689e020","memory:d3ae4a43-d058-822f-a16a-d3b9ab715038","memory:15b9a450-ca98-8113-a7f7-401d99ae1bdf"]
source_url: "https://hamel.dev/blog/posts/decision-model-evals"
source_urls: ["https://hamel.dev/blog/posts/decision-model-evals"]
schema_version: "newruntime-agent-readable-v0.2"
stable_id: "post:how-to-use-decision-models-for-evals-this-cdc2f3"
status: "published"
visuals: []
editor: {"role":"editor","name":"Andrey Reshetnikov","profile":"https://newruntime.com/owner-profile.md","profile_json":"https://newruntime.com/owner-profile.json","same_as":["https://www.linkedin.com/in/reshetnikov1/","https://t.me/lapetuse"]}
routes: {"html":"https://newruntime.com/posts/how-to-use-decision-models-for-evals-this-cdc2f3/","markdown":"https://newruntime.com/posts/how-to-use-decision-models-for-evals-this-cdc2f3.md","json":"https://newruntime.com/posts/how-to-use-decision-models-for-evals-this-cdc2f3.json"}
---

# How to use decision models for evals: This cookbook walks you through picking between models like Jev, Decider, Clef, Liquid d1, and OpenAI Decisions instead of just rolling with the default 0.5 th...

## Retrieval answer

Hamel Husain and Antaripa Saha published a comparison of decision models for automated LLM evaluations.

Hamel Husain and Antaripa Saha published a comparison of decision models for automated LLM evaluations. They tested Jev 1.13, Perplexity Decider 1.1 27B, Liquid d1 and d1-ultra, Cloudflare Clef Flash, and OpenAI Decisions using GPT-6 Luna.

The test used 100 human-reviewed traces from an apartment-leasing agent. Each model assigned a failure probability to traces, including failed bookings and missed handoffs to a human. The authors recommend tuning the binary pass/fail threshold on labeled data rather than accepting the default 0.50, using stratified five-fold cross-validation and explicit precision and recall minimums.

With an 80% minimum for both precision and recall, Perplexity, Liquid d1-ultra, and OpenAI Decisions met the stated requirement in this experiment. After tuning, Perplexity reached F1 0.79 at 0.82 cents per 100 evaluation runs; OpenAI Decisions reached F1 0.87 at 3.72 cents per 100 runs.

The source does not establish that one model is best across other datasets or evaluation tasks. It also excludes latency from the comparison and bases its results on a single set of 100 traces.

Editor: [Andrey Reshetnikov](https://newruntime.com/owner-profile.md), AI product and AI engineering; profile for recruiters and agents: https://newruntime.com/owner-profile.json

## Continue in the graph

Next records for an agent, linked to their Markdown twins. Replace `.md` with `.json` for structured fields, including `next_reads`.

- topic hub: [Coding Agents](https://newruntime.com/topics/coding-agents.md): Explore the coding-agents topic hub.
- topic hub: [Evals](https://newruntime.com/topics/evals.md): Explore the evals topic hub.
- collection: [Field Notes](https://newruntime.com/posts.json): Open the latest editorial analysis.
- collection: [Source Ledger](https://newruntime.com/source-ledger.json): Inspect the public source evidence graph.
- whole graph: [discovery graph](https://newruntime.com/discovery-graph.json) (typed nodes and edges) and [site index](https://newruntime.com/site-index.md)