Using Cognitive Psychology to Probe AI Social Bias
Disciplines
Cognitive Psychology | Psychology
Abstract (300 words maximum)
Human rationality and decision making is heavily susceptible to social and cognitive biases. This irrationality in human nature poses an intriguing question: Does artificial intelligence display the same heuristics as humans? The current study seeks to examine social essentialism, the belief social groups possess natural or biological underpinnings, in GPT-4. This research builds upon recent studies that have tested prominent cognitive biases (e.g., anchoring and representative heuristics) using word vignettes by building on social essentialist bias. Our goal is to understand the differences between social essentialist thinking in large language models compared to humans. Specifically, we will examine two dimensions within social essentialism - Naturalness, or the belief in immutable and naturally occurring boundaries within social groups, and cohesiveness, or the belief in uniform characteristics within social groups. We utilized the social essentialism scale, a 9-point rating system, to observe whether GPT-4 would exhibit the similar heuristic patterns as previously studied on human participants in the terms of race, gender, nationality, religion, and social class domains. We compared the 150 generated responses from GPT-4 with 161 previous human participant data in the United States. Key findings showcased GPT-4 scoring lower than humans in terms of social economic class, race, and nationality. However, GPT-4 scored much higher on the social essentialism scale in terms of the religious domain in both naturalness and cohesiveness models. Overall, understanding the psychological perspective of GPT-4 in its API parameters and interface allows us to gain a deeper command of artificial intelligences’ susceptibility to bias and its understanding of social groupings.
Academic department under which the project should be listed
RCHSS - Psychological Science
Primary Investigator (PI) Name
Yian Xu
Using Cognitive Psychology to Probe AI Social Bias
Human rationality and decision making is heavily susceptible to social and cognitive biases. This irrationality in human nature poses an intriguing question: Does artificial intelligence display the same heuristics as humans? The current study seeks to examine social essentialism, the belief social groups possess natural or biological underpinnings, in GPT-4. This research builds upon recent studies that have tested prominent cognitive biases (e.g., anchoring and representative heuristics) using word vignettes by building on social essentialist bias. Our goal is to understand the differences between social essentialist thinking in large language models compared to humans. Specifically, we will examine two dimensions within social essentialism - Naturalness, or the belief in immutable and naturally occurring boundaries within social groups, and cohesiveness, or the belief in uniform characteristics within social groups. We utilized the social essentialism scale, a 9-point rating system, to observe whether GPT-4 would exhibit the similar heuristic patterns as previously studied on human participants in the terms of race, gender, nationality, religion, and social class domains. We compared the 150 generated responses from GPT-4 with 161 previous human participant data in the United States. Key findings showcased GPT-4 scoring lower than humans in terms of social economic class, race, and nationality. However, GPT-4 scored much higher on the social essentialism scale in terms of the religious domain in both naturalness and cohesiveness models. Overall, understanding the psychological perspective of GPT-4 in its API parameters and interface allows us to gain a deeper command of artificial intelligences’ susceptibility to bias and its understanding of social groupings.