Know it all
Predicting Experimental Results: Who Knows What?
Stefano DellaVigna & Devin Pope
NBER Working Paper, August 2016
Abstract:
Academic experts frequently recommend policies and treatments. But how well do they anticipate the impact of different treatments? And how do their predictions compare to the predictions of non-experts? We analyze how 208 experts forecast the results of 15 treatments involving monetary and non-monetary motivators in a real-effort task. We compare these forecasts to those made by PhD students and non-experts: undergraduates, MBAs, and an online sample. We document seven main results. First, the average forecast of experts predicts quite well the experimental results. Second, there is a strong wisdom-of-crowds effect: the average forecast outperforms 96 percent of individual forecasts. Third, correlates of expertise -- citations, academic rank, field, and contextual experience -- do not improve forecasting accuracy. Fourth, experts as a group do better than non-experts, but not if accuracy is defined as rank ordering treatments. Fifth, measures of effort, confidence, and revealed ability are predictive of forecast accuracy to some extent, especially for non-experts. Sixth, using these measures we identify 'superforecasters' among the non-experts who outperform the experts out of sample. Seventh, we document that these results on forecasting accuracy surprise the forecasters themselves. We present a simple model that organizes several of these results and we stress the implications for the collection of forecasts of future experimental results.
---------------------
Karla Evans et al.
Proceedings of the National Academy of Sciences, forthcoming
Abstract:
Humans are very adept at extracting the “gist” of a scene in a fraction of a second. We have found that radiologists can discriminate normal from abnormal mammograms at above-chance levels after a half-second viewing (d′ ∼ 1) but are at chance in localizing the abnormality. This pattern of results suggests that they are detecting a global signal of abnormality. What are the stimulus properties that might support this ability? We investigated the nature of the gist signal in four experiments by asking radiologists to make detection and localization responses about briefly presented mammograms in which the spatial frequency, symmetry, and/or size of the images was manipulated. We show that the signal is stronger in the higher spatial frequencies. Performance does not depend on detection of breaks in the normal symmetry of left and right breasts. Moreover, above-chance classification is possible using images from the normal breast of a patient with overt signs of cancer only in the other breast. Some signal is present in the portions of the parenchyma (breast tissue) that do not contain a lesion or that are in the contralateral breast. This signal does not appear to be a simple assessment of breast density but rather the detection of the abnormal gist may be based on a widely distributed image statistic, learned by experts. The finding that a global signal, related to disease, can be detected in parenchyma that does not contain a lesion has implications for improving breast cancer detection.
---------------------
Irrelevant Negative Information Enhances Positive Impressions
Meyrav Shoham, Sarit Moldovan & Yael Steinhart
Journal of Consumer Psychology, forthcoming
Abstract:
This research examines the impact of irrelevant information and its valence (positive or negative) on consumers' evaluations, choices, and post-choice satisfaction, within the context of online reviews. We demonstrate that seemingly irrelevant online reviews can enhance positive impressions, but only if they are labeled with a negative valence (e.g., with a one-star rather than a five-star rating). A series of studies provides support for this positive effect of negatively valenced irrelevant information; namely, the inclusion of a negatively valenced irrelevant review alongside positive reviews leads to greater product preferences, as consumers feel confident that the information they have about the product is more complete. We also demonstrate the moderating role of review source.
---------------------
Mitchel Murdock & Priyali Rajagopal
Journal of Marketing, forthcoming
Abstract:
This research examines the effects of warning messages that emphasize the social consequences of negative health outcomes and demonstrates that when social (versus health) consequences are highlighted it leads to greater perceived temporal proximity of and increased perceived vulnerability to the outcome, thereby affecting risk perceptions, behavioral intentions, and customer perceptions of actual experience. This effect is documented across five studies in different health domains including flossing (study 1), soda consumption (study 2), smoking (study 3), and unprotected UV exposure (study 4, study 5). These findings point to the important role of the consequence type highlighted in warning messages, which can have a significant impact on risk perceptions and consumer experiences.
---------------------
Take It or Leave It: How Choosing versus Rejecting Alternatives Affects Information Processing
Tatiana Sokolova & Aradhna Krishna
Journal of Consumer Research, forthcoming
Abstract:
People can make decisions by choosing or by rejecting alternatives. This research shows that changing a task from choice to rejection makes people more likely to rely on deliberative processing, what we label the task-type effect. To demonstrate this effect, we use a set of established decision biases that can be attenuated under deliberative processing. We show that changing a task from choice to rejection makes people express more consistent preferences between safe and risky options in the Asian disease problem (Study 1A) and in financial decision-making (Study 1B), even with real monetary consequences (Study 1C). Further, switching a task from choice to rejection increases the quality of consideration sets in the context of hotel reviews (Study 2), and leads to more rational decisions in the context of cell phone plan selection (Study 3). Studies 4 and 5 tap into the process underlying the effect of task type. We demonstrate that a rejection task produces decisions similar to those observed in a choice task when decision-makers are cognitively depleted (Study 4), or encouraged to rely on their feelings (Study 5). The findings provide insight into the effect of task type on deliberation and decision outcomes.
---------------------
Embracing Chance Tactically: A Different Perspective on Risk Taking
Orit Tykocinski, Inbal Amir & Shahar Ayal
Journal of Behavioral Decision Making, forthcoming
Abstract:
Although they value certainty, people are willing to take risks to avoid losses. Consequently, they are risk-seeking in the domain of losses but risk-avoidant in the domain of gains. This behavior, frequently demonstrated in framing experiments, is traditionally explained in terms of prospect theory. We suggest a different account whereby involving chance in one's decisions may serve a strategic impression-formation function. In the domain of losses actors may embrace chance to distance themselves from the outcomes and deflect possible blame. Given potential gains, however, actors may avoid uncertainty to enhance their association with valued outcomes. We test this idea by manipulating the level of actors' personal responsibility for the decision outcomes. The results of four studies consistently showed that when personal responsibility is high, the original framing effect is replicated (i.e., greater risk-taking when choices are framed in terms of losses rather than gains). However, when because of assigned role or decision circumstances, actors experience low personal responsibility for the outcomes, and the classic framing effect is eliminated.
---------------------
The sky is falling: Evidence of a negativity bias in the social transmission of information
Keely Bebbington et al.
Evolution and Human Behavior, forthcoming
Abstract:
The method of serial reproduction has revealed that the social transmission of information is characterized by the gradual transformation of the original message. This transformation results from the preferential survival of certain types of information and the resolution of ambiguity. Here we present evidence of a bias favoring the social transmission of negatively-valenced information across multiple transmission episodes. Ninety-two, four-person chains transmitted a story containing unambiguously positive and unambiguously negative story events, along with ambiguous story events that could be interpreted positively or negatively. Analysis using mixed-effects modelling revealed the preferential survival of unambiguously negative events over positive events, and the increasingly negative resolution of ambiguous events across successive transmission episodes. Contrary to predictions, elevated state anxiety did not enhance the social transmission of negatively-valenced information. We also found that the survival of unambiguously negative story events was positively correlated with the negative resolution of ambiguous story events, reflecting a general negativity-bias in the social transmission of information.
---------------------
Classified or Coverup? The Effect of Redactions on Conspiracy Theory Beliefs
Brendan Nyhan et al.
Journal of Experimental Political Science, forthcoming
Abstract:
Conspiracy theories are prevalent among the public. Governments frequently release official documents attempting to explain events that inspire these beliefs. However, these documents are often heavily redacted, a practice that lay epistemic theory suggests might be interpreted as evidence for a conspiracy. To investigate this possibility, we tested the effect of redactions on beliefs in a well-known conspiracy theory. Results from two preregistered experiments indicate that conspiracy beliefs were higher when people were exposed to seemingly redacted documents compared to when they were exposed to unredacted documents that were otherwise identical. In addition, unredacted documents consistently lowered conspiracy beliefs relative to controls while redacted documents had reduced or null effects, suggesting that lay epistemic interpretations of the redactions undermined the effect of information in the documents. Our findings, which do not vary by conspiracy predispositions, suggest policymakers should be more transparent when releasing documents to refute misinformation.
---------------------
Haotian Zhou & Ayelet Fishbach
Journal of Personality and Social Psychology, forthcoming
Abstract:
The authors find that experimental studies using online samples (e.g., MTurk) often violate the assumption of random assignment, because participant attrition — quitting a study before completing it and getting paid — is not only prevalent, but also varies systemically across experimental conditions. Using standard social psychology paradigms (e.g., ego-depletion, construal level), they observed attrition rates ranging from 30% to 50% (Study 1). The authors show that failing to attend to attrition rates in online panels has grave consequences. By introducing experimental confounds, unattended attrition misled them to draw mind-boggling yet false conclusions: that recalling a few happy events is considerably more effortful than recalling many happy events, and that imagining applying eyeliner leads to weight loss (Study 2). In addition, attrition rate misled them to draw a logical yet false conclusion: that explaining one’s view on gun rights decreases progun sentiment (Study 3). The authors offer a partial remedy (Study 4) and call for minimizing and reporting experimental attrition in studies conducted on the Web.
---------------------
Benjamin Storm, Sean Stone & Aaron Benjamin
Memory, forthcoming
Abstract:
The ways in which people learn, remember, and solve problems have all been impacted by the Internet. The present research explored how people become primed to use the Internet as a form of cognitive offloading. In three experiments, we show that using the Internet to retrieve information alters a person’s propensity to use the Internet to retrieve other information. Specifically, participants who used Google to answer an initial set of difficult trivia questions were more likely to decide to use Google when answering a new set of relatively easy trivia questions than were participants who answered the initial questions from memory. These results suggest that relying on the Internet to access information makes one more likely to rely on the Internet to access other information.
---------------------
Trading experience modulates anterior insula to reduce the endowment effect
Lester Tong et al.
Proceedings of the National Academy of Sciences, 16 August 2016, Pages 9238–9243
Abstract:
People often demand a greater price when selling goods that they own than they would pay to purchase the same goods—a well-known economic bias called the endowment effect. The endowment effect has been found to be muted among experienced traders, but little is known about how trading experience reduces the endowment effect. We show that when selling, experienced traders exhibit lower right anterior insula activity, but no differences in nucleus accumbens or orbitofrontal activation, compared with inexperienced traders. Furthermore, insula activation mediates the effect of experience on the endowment effect. Similar results are obtained for inexperienced traders who are incentivized to gain trading experience. This finding indicates that frequent trading likely mitigates the endowment effect indirectly by modifying negative affective responses in the context of selling.
---------------------
Sleep Increases Susceptibility to the Misinformation Effect
Dustin Calvillo et al.
Applied Cognitive Psychology, forthcoming
Abstract:
When individuals witness an event and are exposed to misleading postevent information, they often incorporate the misleading information into their memory for the original event, a phenomenon known as the misinformation effect. The present study examined the role of sleep in the misinformation effect. Participants (N = 177) witnessed two events; were exposed to misleading postevent information immediately, 12 hours later the same day, 12 hours later the next day, or 24 hours later; and then took a recognition test. All groups demonstrated the misinformation effect, and this effect was larger in groups with an overnight retention interval. Signal detection analyses revealed that sleep decreased sensitivity. These results suggest that sleep increases susceptibility to the misinformation effect, which may occur because sleep results in gist-based representations of original events or because sleep improves learning of postevent information. Implications for interviewing eyewitnesses are discussed.
---------------------
Brianna Middlewood, Jonathan Gallegos & Karen Gasper
Creativity Research Journal, Summer 2016, Pages 310-317
Abstract:
Three studies examined the hypothesis that feeling tired along with feeling happy might be linked to the acceptance of atypical ideas. Consistent with this hypothesis, across 3 studies and using 2 different measures of accepting atypical ideas, feelings of happiness and tiredness interacted. When people were high in tiredness, as happiness increased, so too did acceptance of atypical ideas (choosing more unusual exemplars and suggesting more unusual solutions to the gestalt completion test). When people were low in tiredness, happiness had no effect on acceptance of atypical ideas. The studies also examined whether differences in sensation-seeking (Studies 1 & 2) and acquiescence (Study 3) mediated the effect, but neither consistently did so. This work suggests that the combination of feeling tired and happy may enhance acceptance of atypical or unusual ideas, which could potentially help creative thought.
---------------------
The strength to face the facts: Self-regulation defends against defensive information processing
Rachel Ruttan & Loran Nordgren
Organizational Behavior and Human Decision Processes, November 2016, Pages 86–98
Abstract:
Five studies examined the impact of self-regulatory capacities on defensive information processing — the tendency to deny, distort, or avoid diagnostic self-threatening information. Across domains, we found that people low in trait (Studies 1 and 5) and state (Studies 2, 3, and 4) self-regulatory capacities were more likely to deny the validity and importance of negative feedback, and were less willing to seek improvement based on this information. Alternative explanations based on self-esteem and competence-based deficits were ruled out. Moreover, we demonstrated a boundary condition of this effect: For participants high in self-improvement motivation, reduced self-regulatory capacities did not affect defensive processing (Studies 4 and 5). Taken together, these results suggest that trait and state levels of self-regulatory capacities are a key factor in determining whether people engage in defensive information processing.
---------------------
Jennifer Siegel & Julie Bugg
Psychology of Aesthetics, Creativity, and the Arts, forthcoming
Abstract:
Creativity is predominantly measured in scientific research with divergent thinking tasks that assess the potential for creative ideation. The current study aimed to further foster a distinction between divergent thinking and a second measure of creativity, creative achievement (the production of tangible or visible pieces), by examining whether these 2 measures are differentially related to attentional flexibility and hypomania. Evidence was found linking divergent thinking to better attentional flexibility and creative achievement to poorer attentional flexibility in a novel variant of the Stroop task. Additionally, creative achievement, especially nonscience-related (e.g., artistic) achievement, was positively associated with risk for hypomania whereas divergent thinking was not related to hypomania. The findings support a distinction between measures of creativity (divergent thinking ability vs. creative achievement), which may have clinical implications (e.g., for bipolar disorder) and theoretical implications for the study of attentional flexibility and rigidity.