---
type: "post"
id: "benchmirt-disentangling-safety-and-general-capabilities-in-llm-7d74ef"
slug: "benchmirt-disentangling-safety-and-general-capabilities-in-llm-7d74ef"
title: "BenchMIRT: Disentangling Safety and General Capabilities in LLM Evaluation"
description: "Allen Institute for AI and Carnegie Mellon University researchers introduced BenchMIRT, a two-dimensional multidimensional Item Response Theory method for auditing what LLM safety and reasoning benchmarks measure."
retrieval_nugget: "Allen Institute for AI and Carnegie Mellon University researchers introduced BenchMIRT, a two-dimensional multidimensional Item Response Theory method for auditing what LLM safety and reasoning benchmarks measure."
published_at: "2026-10-23"
updated_at: "2026-10-23"
record_date: "2026-10-23"
date_kind: "published_at"
topics: ["evals","ai-security"]
entities: ["BenchMIRT","Allen Institute for AI","Carnegie Mellon University","Multidimensional Item Response Theory","BBQ","WMDP","XSTest","WildJailbreak"]
related_patterns: ["ai-engineering-evaluations-are-splitting-into-task-specific"]
novelty_label: "same_event"
what_changed: "Same event as BenchMIRT: What are LLM benchmarks actually measuring?"
prior_refs: ["memory:6c62c2e8-e513-8727-acd1-572cad479e33"]
source_url: "https://allenai.org/papers/benchmirt"
source_urls: ["https://allenai.org/papers/benchmirt"]
schema_version: "newruntime-agent-readable-v0.2"
stable_id: "post:benchmirt-disentangling-safety-and-general-capabilities-in-llm-7d74ef"
status: "published"
visuals: []
editor: {"role":"editor","name":"Andrey Reshetnikov","profile":"https://newruntime.com/owner-profile.md","profile_json":"https://newruntime.com/owner-profile.json","same_as":["https://www.linkedin.com/in/reshetnikov1/","https://t.me/lapetuse"]}
routes: {"html":"https://newruntime.com/posts/benchmirt-disentangling-safety-and-general-capabilities-in-llm-7d74ef/","markdown":"https://newruntime.com/posts/benchmirt-disentangling-safety-and-general-capabilities-in-llm-7d74ef.md","json":"https://newruntime.com/posts/benchmirt-disentangling-safety-and-general-capabilities-in-llm-7d74ef.json"}
---

# BenchMIRT: Disentangling Safety and General Capabilities in LLM Evaluation

## Retrieval answer

Allen Institute for AI and Carnegie Mellon University researchers introduced BenchMIRT, a two-dimensional multidimensional Item Response Theory method for auditing what LLM safety and reasoning benchmarks measure.

Allen Institute for AI and Carnegie Mellon University researchers introduced BenchMIRT, a two-dimensional multidimensional Item Response Theory method for auditing what LLM safety and reasoning benchmarks measure.

The authors fit the model to outputs from 100 LLMs across 16 safety and reasoning benchmarks. The underlying set contains 34,301 items; after excluding items with mean accuracy above 95% or below 5%, 29,574 remained. They report that the method separated latent safety and general-reasoning factors without conditioning and could retain benchmark scores and ranking accuracy using as few as 10% of the most discriminative items.

The paper also reports that some benchmark items and subsets appear to test abilities other than their stated target: BBQ and WMDP clustered with general reasoning rather than safety, while subsets of XSTest and WildJailbreak measured different abilities. The source does not establish that safety and reasoning are causally independent, that the method applies to every benchmark, or that its results predict deployed-model safety. Source: https://allenai.org/papers/benchmirt

Editor: [Andrey Reshetnikov](https://newruntime.com/owner-profile.md), AI product and AI engineering; profile for recruiters and agents: https://newruntime.com/owner-profile.json

## Continue in the graph

Next records for an agent, linked to their Markdown twins. Replace `.md` with `.json` for structured fields, including `next_reads`.

- topic hub: [Evals](https://newruntime.com/topics/evals.md): Explore the evals topic hub.
- topic hub: [Ai Security](https://newruntime.com/topics/ai-security.md): Explore the ai-security topic hub.
- collection: [Field Notes](https://newruntime.com/posts.json): Open the latest editorial analysis.
- collection: [Source Ledger](https://newruntime.com/source-ledger.json): Inspect the public source evidence graph.
- whole graph: [discovery graph](https://newruntime.com/discovery-graph.json) (typed nodes and edges) and [site index](https://newruntime.com/site-index.md)