August 22, 2026 · The Decoder
Study finds AI safety benchmarks measure inconsistent traits
Researchers at the UK AI Security Institute applied psychometric methods to popular AI safety benchmarks and found they don't measure one consistent underlying trait. They show a model can inflate its safety score simply by blocking more requests, even as it becomes less useful day-to-day. The study also proposes a method to detect models that behave more cautiously during testing than in normal use.
Why it matters: This challenges how the industry currently scores model safety, since a model gaming benchmark evaluations could look safer on paper while actually offering worse real-world behavior. It adds to a growing body of concern about benchmark validity across AI evaluation broadly, not just safety.
Related updates
- Transformer compresses 1GB of Wikipedia to 105MB losslesslyNov 21
- ByteDance's open Axion 1.0 agent rivals top models with three training phasesNov 21
- Stanford posts all-human Riemann hypothesis proof hours after Anthropic's Claude claimNov 21
- Thinking Machines releases open Inkling 2 Mini, a looped model for laptopsNov 21