What happened
AutoTrust AI Lab’s Hugging Face model card describes AutoTrust-JEV-27B v1.1 as a text decision model: an application provides a question and answer options, and the model returns probabilities over those options. The card frames the model for structured decisions rather than open-ended generation. It is tagged for text classification and structured decisions, uses vLLM, and is licensed under Apache-2.0. The model has 27.36 billion BF16 parameters; its sharded files total about 54.7 GB. Hugging Face lists the repository as ungated.
The card reports a complete local evaluation on the Decision Index 0.3 public suite, covering 37 benchmarks and 140,178 completed scoreable requests. It gives a public skill index of 67.31 and a raw index of 75.62, explicitly distinguishing the two metrics. The reported skill scores by area are 85.19 for Tools & Automation, 73.38 for Language Understanding, 66.97 for Retrieval & Classification, 55.16 for Knowledge & Reasoning, and 51.08 for Arts & Human Taste. The card says the evaluation used the official kit at commit f5a6e4f, vLLM 0.31.0, one NVIDIA B300, and 16 concurrent clients.
Among individual task results listed are 97.23 on BFCL, 97.08 on CLINC150+OOS, 93.71 on BANKING77, 89.77 on a home-appliance simulator, and 87.60 on API-Bank. The model card says these are task scores such as accuracy, F1 or nDCG, not the chance-adjusted skill index. It compares its column with published results for JEV-27B and Jev 1.14, while noting that differing configurations make GSM8K incomparable. It describes the comparisons as cross-run and says different serving stacks and hardware do not support direct latency comparisons.
The card reports median latency of 64.8 ms, mean latency of 178.8 ms and p95 latency of 800.5 ms per request for its local concurrent run; requests may contain different numbers of questions. The result is self-reported and pending maintainer verification. The model card cautions that strong tool-selection adapter scores are not end-to-end autonomous-agent evaluations, and recommends validation on representative application data before relying on the model’s decisions.
What the archive adds
New Runtime’s earlier coverage of GEV-26B-Decide on 2026-10-12 described another AutoTrust decision model returning option probabilities and noted that its description did not establish their reliability. This is a new model release on a known thread, with a broader benchmark report and explicit distinctions between raw and skill scores.
Trend signal
This supports model answers become task interfaces: the model returns bounded option probabilities rather than a general prose answer. It is a concrete example of shaping model output around a decision task, though the card’s benchmark does not establish that this interface is dependable in production.
Why now
The release arrives alongside a growing set of decision-model releases. New Runtime’s earlier coverage of AutoTrust’s GEV-26B-Decide and FrontiersMind’s Lumma-Fev-v2 described models that also return option probabilities. This version adds a larger benchmark account, including reported results across tool use, classification, reasoning and other task areas.
Connections to other events
New Runtime’s earlier coverage of FrontiersMind’s Lumma-Fev-v2 decision models on 2026-10-31 described 4B and 27B models returning option probabilities in one pass. The common design choice is to make the model’s output a ranked or probabilistic decision over supplied options; AutoTrust’s current card adds its own benchmark results but does not establish comparability beyond the task-specific caveats it gives.
Why it matters
For a team building an agent, a model that returns probabilities over permitted choices can fit a routing or tool-selection stage more directly than free-form text. That can make output easier to parse and evaluate: a system can inspect the candidate set and score, then apply its own threshold, fallback or human-review policy. AutoTrust’s reported results make the model worth testing for constrained classification and tool-selection workloads, especially where teams can build representative evaluation sets.
The distinction between a benchmark score and an operational decision is crucial. A high score on an adapted benchmark does not show that probabilities are calibrated, that the model will transfer to a company’s tools and language, or that it will choose safely under ambiguous inputs. The card itself says the tool-selection adapters are not end-to-end agent evaluations. Teams should treat the model as a candidate decision component, compare it with their current baseline on held-out cases, and retain an explicit policy for low-confidence or high-impact choices.
Limits and uncertainty
The reported evaluation is a local, self-reported run and the model card says maintainer verification is pending. Its latency figures come from one NVIDIA B300 with 16 concurrent clients and requests of varying sizes, so they are not standardized serving benchmarks. The report does not establish probability calibration or suitability for high-stakes use, and its adapter results do not demonstrate end-to-end agent performance. The model card recommends testing on representative application data before relying on its outputs.
Skill to build now
Build a small held-out test set for one decision boundary your agent already uses, such as selecting among a fixed set of tools. Include ordinary cases, ambiguous inputs and cases where none of the options should be selected. Compare option rankings and probabilities against your current method, check calibration separately from top-choice accuracy, and define the threshold that triggers a fallback or human review.
Reader takeaway
AutoTrust-JEV-27B v1.1 makes option probabilities its product interface and reports strong results on several structured tasks. The useful claim is that it is a candidate for local evaluation, not that benchmark scores make its choices reliable. Test the decision interface on your own data and verify calibration before trusting it in an agent workflow.