Summary: A new study presents an automated cognitive-mapping framework that combines the raw text-processing power of large language models (LLMs) with rigorous mathematical choice models from behavioral decision science. Researchers used an LLM to read, interpret, and categorize thousands of free-text justifications written by participants during simulated gambling tasks. By validating the model’s text classifications against formal mathematical models of actual choices, the team demonstrated that first-person verbal reports are a reliable, scientifically valid data source and that people’s core decision logic shifts systematically with the structure of each choice problem.
Key Facts
- The gambling text experiment: In repeated simulated gambling rounds with changing risk parameters, participants could not simply click an option; after every round they were required to write a short explanation of their thought process and subjective reason for choosing as they did.
- The algorithmic codebook: Drawing on decades of behavioral finance and decision theory, researchers developed a comprehensive taxonomy of possible decision rationales—from risk-seeking “maximax” heuristics that emphasize the best possible outcome to conservative “minimax” strategies that prioritize avoiding large losses.
- LLMs as scalable qualitative auditors: Instead of manually reading and coding thousands of free-text responses, the team fine-tuned large language models to tag each entry with the psychological reasons present in the text, enabling rapid, consistent classification at scale.
- Mathematical choice validation: To ensure the AI’s labels were not spurious, the researchers cross-checked the LLM’s textual tags against independent mathematical models of participants’ observable choices. The two sources aligned closely: what people wrote matched how they actually behaved.
- Dynamic strategic shifting: Results show decision strategies are not fixed personality traits. Individuals systematically adjust their reasoning profiles from round to round in response to the specific structure and stakes of the decision problem.
- Policy and research toolkit: The automated framework makes it feasible to analyze large volumes of open-ended, real-world feedback—helping policymakers and analysts understand how people simplify complex trade-offs in health, finance, and technology adoption.
Source: TUD
Lead author: Dr. Kamil Fuławka, SynoSys. “Our understanding of human decision making improves when people explain their thought processes,” says Dr. Fuławka. “Analyzing free-text reports at scale requires rigorous, scalable tools — and LLMs now make that possible.”
In the study, participants repeatedly chose between monetary lotteries and provided short, free-text explanations after each decision. The research team translated established decision theories into a detailed codebook of potential reasons and used an LLM to identify which reasons appeared in each participant’s verbal report. Independently, the team fit formal mathematical choice models to participants’ selections to test whether the verbalized reasons predicted actual behavior.
The combined approach—self-reports, LLM classification, and mathematical validation—showed that people’s own explanations are a valuable empirical resource. Crucially, the analysis found that reason profiles change predictably with the framing and structure of each choice, rather than reflecting immutable individual traits. Verbal reports produced parsimonious, interpretable representations of decision strategies and, in many cases, improved out-of-sample prediction compared with models inferred from choices alone.
Key Questions Answered
A: Observing final choices tells you what people did but not why. Free-text explanations reveal thought processes, heuristics, and priorities that are invisible in choice data alone. Manual coding of open-text responses is slow and inconsistent. LLMs provide a fast, scalable, and consistent way to read and quantify thousands of personal explanations, giving researchers a richer view of the mechanisms behind decisions.
A: The critical validation step was quantitative: the team translated the AI-identified textual reasons into formal choice assumptions, then tested whether those assumptions predicted participants’ actual gambling behavior. The match between text-derived reasons and observed choices was strong, supporting the accuracy and scientific validity of the LLM-based classifications.
A: High-stakes public choices—retirement planning, medical treatment selection, technology adoption—often involve messy trade-offs that standard surveys and choice data fail to capture. By automatically coding large volumes of open-ended feedback, policymakers can discover how the public simplifies complex problems, which details people attend to, and which misperceptions influence decisions. That insight can guide clearer communications, better-designed interventions, and more human-centered policy instruments.
Editorial Notes
- This article was edited by a Neuroscience News editor.
- Journal paper reviewed in full.
- Additional context added by staff.
About this AI and Decision-Making Research
Author: Benjamin Griebe
Source: TUD
Contact: Benjamin Griebe – TUD
Image credit: Neuroscience News
Original Research: Open access. “Large language models accurately identify decision reasons in verbal reports” by Dirk U. Wulff, Kamil Fuławka, Ralph Hertwig. PNAS. DOI: 10.1073/pnas.2526798123
Abstract
Large language models accurately identify decision reasons in verbal reports
Understanding reasons behind human choices under risk is central to decision science, but traditional methods based solely on behavioral data rely on strict invariance assumptions. This study introduces a scalable framework using large language models to analyze verbal reports and identify articulated reasons for choosing between monetary lotteries. A validated LLM recovered predefined decision reasons from free-text reports and aligned those reasons with participants’ actual choices on the vast majority of trials. The analysis shows that reasons vary systematically with problem structure more than with stable individual differences. Verbal reports provide parsimonious, interpretable representations of decision processes and achieve competitive out-of-sample prediction compared with standard computational models. The framework unlocks the potential of open-ended reports for building context-sensitive, transparent models of human decision making.