Abstract
We illustrate how standard psychometric inventories originally designed for assessing noncognitive human traits can be repurposed as diagnostic tools to evaluate analogous traits in large language models (LLMs). Psychometric profiling enables researchers to study and compare LLMs in terms of noncognitive characteristics, thereby providing a window into the personalities, values, beliefs, and biases these models exhibit (or mimic). We demonstrate one promising approach, zero-shot classification, for several LLMs and psychometric inventories. We conclude by highlighting open challenges and future avenues of research for AI Psychometrics.
Short BioMax Pellert has a background in computer science, the social sciences, cognitive science and economics (University of Vienna, Austria and University of Ljubljana, Slovenia). Currently, he is interim Professor for Social and Behavioural Data Science at the University of Konstanz. He is broadly interested in the social sciences and uses traditional and novel computational methods from domains such as Natural Language Processing to study belief updates, emotional decay on social media, polarization, psychometric aspects of large language models, emotional well-being measured from textual data, semantic embeddings as complements to human ratings and many other interesting phenomena.
Speakers
Speaker: Max Pellert. Professor for Social and Behavioural Data Science at the University of Konstanz.
Host: Mercè Crosas. Head of Computational Social Sciences Programme, BSC.