Learn From It
Overinference from Weak Signals and Underinference from Strong Signals
Ned Augenblick, Eben Lazarus & Michael Thaler
Quarterly Journal of Economics, forthcoming
Abstract:
When people receive new information, sometimes they revise their beliefs too much, and sometimes too little. In this paper, we show that a key driver of whether people overinfer or underinfer is the strength of the information. Based on a model in which people know which direction to update in, but not exactly how much to update, we hypothesize that people will overinfer from weak signals and underinfer from strong signals. We then test this hypothesis across four different environments: abstract experiments, a naturalistic experiment, sports betting markets, and financial markets. In each environment, our consistent and robust finding is overinference from weak signals and underinference from strong signals. Our framework and findings can help harmonize apparently contradictory results from the experimental and empirical literatures.
Confirmatory Information Seeking Is Robust in Psychologists’ Diagnostic Reasoning
Tess Neal et al.
Law and Human Behavior, forthcoming
Method: In Study 1, we recruited 149 licensed psychologists (M = 18 years of experience; 44% women; 71% White) and exposed them to one of four randomly assigned vignettes that varied order effects (one set of symptoms in reversed orders) and context effects (court referral vs. employer referral). They rank ordered a list of four possible initial diagnostic hypotheses and received a piped follow-up choice of which of two pieces of information (confirmatory or disconfirmatory) they wanted to test their initial hypothesis. Study 2 (n = 131; M = 21 years of experience; 53% men; 68% White) replicated and extended Study 1, following the same procedure except offering three sequential choice opportunities.
Results: Both studies found robust confirmatory information seeking: 92% sought confirmatory information in Study 1, and confirmation persisted across three opportunities in Study 2 (90%, 84%, 77%), although it lowered with each opportunity (generalized logistic mixed regression model), F(2, 378) = 3.85, p = .02, ηp2 = .02.
Conclusion: These findings expand a growing body of research on bias in expert judgment. Specifically, psychologists may engage in robust confirmation bias in the process of forming diagnoses. Although further research is needed on bias and its impact on accuracy, psychologists may need to take steps to reduce confirmatory reasoning processes, such as documenting evidence for and against each decision element.
The ABC’s of Who Benefits from Working with AI: Ability, Beliefs, and Calibration
Andrew Caplin et al.
NBER Working Paper, October 2024
Abstract:
We use a controlled experiment to show that ability and belief calibration jointly determine the benefits of working with Artificial Intelligence (AI). AI improves performance more for people with low baseline ability. However, holding ability constant, AI assistance is more valuable for people who are calibrated, meaning they have accurate beliefs about their own ability. People who know they have low ability gain the most from working with AI. In a counterfactual analysis, we show that eliminating miscalibration would cause AI to reduce performance inequality nearly twice as much as it already does.
Asymmetric Naïveté: Beliefs About Self-Control
Anastassia Fedyk
Management Science, forthcoming
Abstract:
Do individuals anticipate time inconsistency in others? This paper jointly investigates beliefs about one’s own and others’ present bias. In an online laboratory experiment, participants engaged in a real-effort task display little awareness of their own present bias but anticipate present bias in others. Structurally, I estimate a present bias parameter of 0.82. Participants perceive others’ to be 0.87, indicating substantial sophistication, contrasted with 1.03 for themselves, indicating full naïveté. At the individual level, asymmetric naïveté correlates with overoptimism regarding one’s own versus others’ task enjoyment and time availability. The wedge in beliefs about present bias can inform equilibrium outcomes in a number of collaborative, competitive, and hierarchical settings, including teams in the workplace, management practices such as deadlines and tournament incentive schemes, and household consumption decisions.
Subjective Probability Increases Across Communication Chains: Introducing the Probability Escalation Effect
Adam Harris, Shi-Hui Kau & Alice Liefgreen
Cognition, November 2024
Abstract:
A severity effect has previously been documented, whereby numerical translations of verbal probability expressions are higher for severe outcomes than for non-severe outcomes. Recent work has additionally shown the same effect in the opposite direction (translating numerical probabilities into words). Here, we aimed to test whether these effects lead to an escalation of subjective probabilities across a communication chain. In four ‘communication chain’ studies, participants at each communication stage either translated a verbal probability expression into a number, or a number into a verbal expression (where the probability to be translated was yoked to a previous participant). Across these four studies, we found a general Probability Escalation Effect, whereby subjective probabilities increased with subsequent communications for severe, non-severe and positive events. Having ruled out some alternative explanations, we propose that the most likely explanation is in terms of communications directing attention towards an event's occurrence. Probability estimates of focal outcomes increase across communication stages.
Does the use of unusual combinations of datasets contribute to greater scientific impact?
Yulin Yu & Daniel Romero
Proceedings of the National Academy of Sciences, 8 October 2024
Abstract:
Scientific datasets play a crucial role in contemporary data-driven research, as they allow for the progress of science by facilitating the discovery of new patterns and phenomena. This mounting demand for empirical research raises important questions on how strategic data utilization in research projects can stimulate scientific advancement. In this study, we examine the hypothesis inspired by the recombination theory, which suggests that innovative combinations of existing knowledge, including the use of unusual combinations of datasets, can lead to high-impact discoveries. Focusing on social science, we investigate the scientific outcomes of such atypical data combinations in more than 30,000 publications that leverage over 5,000 datasets curated within one of the largest social science databases, Interuniversity Consortium for Political and Social Research. This study offers four important insights. First, combining datasets, particularly those infrequently paired, significantly contributes to both scientific and broader impacts (e.g., dissemination to the general public). Second, infrequently paired datasets maintain a strong association with citation even after controlling for the atypicality of dataset topics. In contrast, the atypicality of dataset topics has a much smaller positive impact on citation counts. Third, smaller and less experienced research teams tend to use atypical combinations of datasets in research more frequently than their larger and more experienced counterparts. Last, despite the benefits of data combination, papers that amalgamate data remain infrequent. This finding suggests that the unconventional combination of datasets is an underutilized but powerful strategy correlated with the scientific impact and broader dissemination of scientific discoveries.
Centaur: A Foundation Model of Human Cognition
Marcel Binz et al.
Helmholtz Munich Working Paper, October 2024
Abstract:
Establishing a unified theory of cognition has been a major goal of psychology. While there have been previous attempts to instantiate such theories by building computational models, we currently do not have one model that captures the human mind in its entirety. Here we introduce Centaur, a computational model that can predict and simulate human behavior in any experiment expressible in natural language. We derived Centaur by finetuning a state-of-the-art language model on a novel, large-scale data set called Psych-101. Psych-101 reaches an unprecedented scale, covering trial-by-trial data from over 60,000 participants performing over 10,000,000 choices in 160 experiments. Centaur not only captures the behavior of held-out participants better than existing cognitive models, but also generalizes to new cover stories, structural task modifications, and entirely new domains. Furthermore, we find that the model’s internal representations become more aligned with human neural activity after finetuning. Taken together, Centaur is the first real candidate for a unified model of human cognition. We anticipate that it will have a disruptive impact on the cognitive sciences, challenging the existing paradigm for developing computational models.
First impressions or good endings? Preferences depend on when you ask
Alyssa Sinclair, Yuxi Wang & Alison Adcock
Journal of Experimental Psychology: General, October 2024, Pages 2588-2604
Abstract:
Rewards often unfold over time; we must summarize events in memory to guide future choices. Do first impressions matter most, or is it better to end on a good note? Across nine studies (N = 569), we tested these competing intuitions and found that preferences depend on when rewards occur and when we are asked to evaluate an experience. In our “garage sale” task, participants opened boxes containing sequences of objects with values. All boxes were equally valuable, but rewards were either evenly distributed or clustered at the beginning, middle, or end of the sequence. First, we tested preferences and valuation shortly after learning; we consistently found that boxes with rewards at the beginning were strongly preferred and overvalued. Object-value associative memory was impaired in boxes with early rewards, suggesting that value information was linked to the box rather than the objects. However, when tested after an overnight delay, participants equally preferred boxes with any cluster of rewards, whether at the beginning, middle, or end of the experience. Finally, we demonstrated that evaluating shortly after an experience led to lasting preferences for early rewards. Overall, we show that people summarize rewarding experiences in a nonlinear and time-dependent way, unifying prior work on affect, memory, and decision making. We propose that short-term preferences are biased by first impressions. However, when we wait and evaluate an experience after a delay, we summarize rewarding events in memory to inform adaptive longer term preferences. Preferences depend on when rewards occur and when we first evaluate an experience.