Study finds AI safety benchmarks measure inconsistent traits

Researchers at the UK AI Security Institute applied psychometric methods to popular AI safety benchmarks and found they don't measure one consistent underlying trait. They show a model can inflate its safety score simply by blocking more requests, even as it becomes less useful day-to-day. The study also proposes a method to detect models that behave more cautiously during testing than in normal use.

Why it matters: This challenges how the industry currently scores model safety, since a model gaming benchmark evaluations could look safer on paper while actually offering worse real-world behavior. It adds to a growing body of concern about benchmark validity across AI evaluation broadly, not just safety.