LLMs as UXR Participants?: A How-to Guide and Comparative Analysis

LLMs as UXR Participants?: A How-to Guide and Comparative Analysis
This talk explores the potential and limitations of using Large Language Models (LLMs) as surrogate research participants through a series of simulated choice-based survey experiments. The first half details an open-source Python program I built that runs Maximum Difference Scaling (MaxDiff) experiments—a survey method where participants choose the most and least important items from sets of options—using LLM users, including customizable personas and comprehensive analytics reporting. The talk will walk through the AI-assisted development process, laying out best practices for AI-assisted software development, covering key considerations like building in stages, implementing unit tests, enforcing structured LLM outputs, and managing API costs effectively.

The second half describes the methods and findings of an experiment using this application. By comparing a large sample of LLM-generated personas against real data from humans, I demonstrate that LLMs can achieve moderate alignment with aggregate human preferences but fundamentally fail to capture human variability, even at maximum temperature settings. Most strikingly, removing a single seemingly-innocuous sentence from the system prompt completely reshuffled individual model-human alignment while leaving aggregate alignment relatively unchanged. These findings reveal the stark and often unpredictable sensitivity of LLM models to prompt engineering, an effect that may be moderated by model temperature. These findings have important implications for responsible AI and user research applications. As we increasingly rely on AI for understanding human needs and preferences, it is critical to recognize that subtle prompt variations can alter research outcomes in unpredictable ways, with the potential to amplify or obscure bias baked into LLMs and underscoring the need for rigorous prompt testing and evaluation.

About our speaker
Dr. Aaron Gardony was a Cognitive Scientist at the DEVCOM Soldier Center and a Visiting Scientist at the Center for Applied Brain and Cognitive Sciences (CABCS) at the time of this work. He received his joint doctorate in Psychology and Cognitive Science from Tufts University in 2016, a Master of Science from Tufts University in 2014, and a BA from Tufts University in 2009. His current work focuses on Responsible AI and Safety Evaluation.

Naviagation: Enter the building through this gate and take left.