Title
The Streetlight Effect in AI Safety Assessments
Abstract
As AI systems advance in capabilities, measuring their safety and alignment to human values is becoming paramount. A fast-growing field of AI research is devoted to developing such assessments. However, most current advances therein may be ill-suited for assessing AI systems across real-world deployments.
Standard methods prompt large language models (LLMs) in a questionnaire-style to describe their values or behavior in hypothetical scenarios. By focusing on unaugmented LLMs, they fall short of evaluating AI agents, which could actually perform relevant behaviors, hence posing much greater risks. LLMs’ engagement with scenarios described by questionnaire-style prompts differs starkly from that of agents based on the same LLMs, as reflected in divergences in the inputs, possible actions, environmental interactions, and likely internal processing between both. As such, LLMs’ responses to scenario descriptions are unlikely to be representative of the corresponding LLM agents’ behavior.
We further contend that such assessments make strong assumptions concerning the ability and tendency of LLMs to report accurately about their counterfactual behavior. This makes them inadequate to assess risks from AI systems in real-world contexts as they lack the necessary construct validity. We then argue that a structurally identical issue holds for current AI alignment approaches. Lastly, we discuss improving safety assessments and alignment training by taking these shortcomings to heart.
About Max
Max is a PhD candidate at the Centre for Philosophy and AI Research (PAIR) at FAU. His research focuses on risks from AI systems, specifically on concepts that are often invoked but rarely scrutinized in this context (such as goals, agency, manipulation, and planning). Recently, he has also explored topics at the intersection of philosophy of AI and philosophy of science. Before joining PAIR, he studied philosophy, political science, and neuroscience at Goethe University Frankfurt and Humboldt University of Berlin.