Publication Date
10-19-2026
Abstract
Governments, insurers, and defenders increasingly rely on published capability claims about AI cybersecurity agents to inform procurement, regulation, and risk assessment. Yet a striking pattern runs through the organizations with the most at stake: enterprises and technology companies deploying frontier AI agents against real production infrastructure consistently decline to publish the accuracy metrics that academic and vendor benchmarks compete to report. This paper investigates that pattern as an economic, organizational, and policy problem rather than a purely technical one. We audit eleven independently published AI cybersecurity benchmarks and deployments using a five-dimension integrity rubric and find that benchmarks relying on outcome-only grading, such as binary flag capture or multiple-choice accuracy, verify the intended method behind a reported success in only a small minority of cases, while benchmarks using process-aware grading verify method in nearly all cases, a separation unlikely to be due to chance within this corpus. We identify a second, largely invisible source of inflation: several benchmarks reuse practice applications or competition archives with extensive public solution write-ups, and we show, drawing on an independent study that quantified this effect directly, that this alone can roughly double a reported score regardless of grading design. We then show that five organizations in our wider corpus, spanning a major payment network, two technology-company security research programmes, and two further vulnerability-discovery efforts, are the only sources that decline to publish any headline accuracy number at all, instead reporting discrete, externally confirmed findings or a custom process metric. We interpret this divergence through the lens of information asymmetry in markets for quality-uncertain goods, and through the organizational-trust literature on human reliance on automation, arguing that unaudited capability claims function similarly to unverifiable quality signals in other markets, while noting that existing certification institutions from adjacent security markets have not yet reached this one. We close with the societal stakes of this dynamic, since a government-backed synthesis report already cites unaudited capability figures from our corpus in a policy-facing document, and offer concrete recommendations for procurement officers, insurers, regulators, and benchmark designers.
Included in
Artificial Intelligence and Robotics Commons, Cybersecurity Commons, Information Security Commons, Management Information Systems Commons, Technology and Innovation Commons, Theory and Algorithms Commons