The authors fit the model to outputs from 100 LLMs across 16 safety and reasoning benchmarks. The underlying set contains 34,301 items; after excluding items with mean accuracy above 95% or below 5%, 29,574 remained. They report that the method separated latent safety and general-reasoning factors without conditioning and could retain benchmark scores and ranking accuracy using as few as 10% of the most discriminative items.
The paper also reports that some benchmark items and subsets appear to test abilities other than their stated target: BBQ and WMDP clustered with general reasoning rather than safety, while subsets of XSTest and WildJailbreak measured different abilities. The source does not establish that safety and reasoning are causally independent, that the method applies to every benchmark, or that its results predict deployed-model safety. Source: https://allenai.org/papers/benchmirt