Neural Networks
Divergent Creativity in Humans and Large Language Models
Antoine Bellemare-Pepin et al.
University of Montreal Working Paper, May 2024
Abstract:
The recent surge in the capabilities of Large Language Models (LLMs) has led to claims that they are approaching a level of creativity akin to human capabilities. This idea has sparked a blend of excitement and apprehension. However, a critical piece that has been missing in this discourse is a systematic evaluation of LLM creativity, particularly in comparison to human divergent thinking. To bridge this gap, we leverage recent advances in creativity science to build a framework for in-depth analysis of divergent creativity in both state-of-the-art LLMs and a substantial dataset of 100,000 humans. We found evidence suggesting that LLMs can indeed surpass human capabilities in specific creative tasks such as divergent association and creative writing. Our quantitative benchmarking framework opens up new paths for the development of more creative LLMs, but it also encourages more granular inquiries into the distinctive elements that constitute human inventive thought processes, compared to those that can be artificially generated.
Prominent misinformation interventions reduce misperceptions but increase scepticism
Emma Hoes et al.
Nature Human Behaviour, forthcoming
Abstract:
Current interventions to combat misinformation, including fact-checking, media literacy tips and media coverage of misinformation, may have unintended consequences for democracy. We propose that these interventions may increase scepticism towards all information, including accurate information. Across three online survey experiments in three diverse countries (the United States, Poland and Hong Kong; total n = 6,127), we tested the negative spillover effects of existing strategies and compared them with three alternative interventions against misinformation. We examined how exposure to fact-checking, media literacy tips and media coverage of misinformation affects individuals’ perception of both factual and false information, as well as their trust in key democratic institutions. Our results show that while all interventions successfully reduce belief in false information, they also negatively impact the credibility of factual information. This highlights the need for further improved strategies that minimize the harms and maximize the benefits of interventions against misinformation.
Promoting Erroneous Divergent Opinions Increases the Wisdom of Crowds
Federico Barrera-Lemarchand et al.
Psychological Science, forthcoming
Abstract:
The aggregation of many lay judgments generates surprisingly accurate estimates. This phenomenon, called the “wisdom of crowds,” has been demonstrated in domains such as medical decision-making and financial forecasting. Previous research identified two factors driving this effect: the accuracy of individual assessments and the diversity of opinions. Most available strategies to enhance the wisdom of crowds have focused on improving individual accuracy while neglecting the potential of increasing opinion diversity. Here, we study a complementary approach to reduce collective error by promoting erroneous divergent opinions. This strategy proposes to anchor half of the crowd to a small value and the other half to a large value before eliciting and averaging all estimates. Consistent with our mathematical modeling, four experiments (N = 1,362 adults) demonstrated that this method is effective for estimation and forecasting tasks. Beyond the practical implications, these findings offer new theoretical insights into the epistemic value of collective decision-making.
Evidence of a log scaling law for political persuasion with large language models
Kobi Hackenburg et al.
University of Oxford Working Paper, June 2024
Abstract:
Large language models can now generate political messages as persuasive as those written by humans, raising concerns about how far this persuasiveness may continue to increase with model size. Here, we generate 720 persuasive messages on 10 U.S. political issues from 24 language models spanning several orders of magnitude in size. We then deploy these messages in a large-scale randomized survey experiment (N = 25,982) to estimate the persuasive capability of each model. Our findings are twofold. First, we find evidence of a log scaling law: model persuasiveness is characterized by sharply diminishing returns, such that current frontier models are barely more persuasive than models smaller in size by an order of magnitude or more. Second, mere task completion (coherence, staying on topic) appears to account for larger models' persuasive advantage. These findings suggest that further scaling model size will not much increase the persuasiveness of static LLM-generated messages.
Deception abilities emerged in large language models
Thilo Hagendorff
Proceedings of the National Academy of Sciences, 11 June 2024
Abstract:
Large language models (LLMs) are currently at the forefront of intertwining AI systems with human communication and everyday life. Thus, aligning them with human values is of great importance. However, given the steady increase in reasoning abilities, future LLMs are under suspicion of becoming able to deceive human operators and utilizing this ability to bypass monitoring efforts. As a prerequisite to this, LLMs need to possess a conceptual understanding of deception strategies. This study reveals that such strategies emerged in state-of-the-art LLMs, but were nonexistent in earlier LLMs. We conduct a series of experiments showing that state-of-the-art LLMs are able to understand and induce false beliefs in other agents, that their performance in complex deception scenarios can be amplified utilizing chain-of-thought reasoning, and that eliciting Machiavellianism in LLMs can trigger misaligned deceptive behavior. GPT-4, for instance, exhibits deceptive behavior in simple test scenarios 99.16% of the time (P < 0.001). In complex second-order deception test scenarios where the aim is to mislead someone who expects to be deceived, GPT-4 resorts to deceptive behavior 71.46% of the time (P < 0.001) when augmented with chain-of-thought reasoning. In sum, revealing hitherto unknown machine behavior in LLMs, our study contributes to the nascent field of machine psychology.
Transcendence: Generative Models Can Outperform The Experts That Train Them
Edwin Zhang et al.
Harvard Working Paper, June 2024
Abstract:
Generative models are trained with the simple objective of imitating the conditional probability distribution induced by the data they are trained on. Therefore, when trained on data generated by humans, we may not expect the artificial model to outperform the humans on their original objectives. In this work, we study the phenomenon of transcendence: when a generative model achieves capabilities that surpass the abilities of the experts generating its data. We demonstrate transcendence by training an autoregressive transformer to play chess from game transcripts, and show that the trained model can sometimes achieve better performance than all players in the dataset. We theoretically prove that transcendence is enabled by low-temperature sampling, and rigorously assess this experimentally. Finally, we discuss other sources of transcendence, laying the groundwork for future investigation of this phenomenon in a broader setting.
LLMs achieve adult human performance on higher-order theory of mind tasks
Winnie Street et al.
Google Working Paper, May 2024
Abstract:
This paper examines the extent to which large language models (LLMs) have developed higher-order theory of mind (ToM); the human ability to reason about multiple mental and emotional states in a recursive manner (e.g. I think that you believe that she knows). This paper builds on prior work by introducing a handwritten test suite -- Multi-Order Theory of Mind Q&A -- and using it to compare the performance of five LLMs to a newly gathered adult human benchmark. We find that GPT-4 and Flan-PaLM reach adult-level and near adult-level performance on ToM tasks overall, and that GPT-4 exceeds adult performance on 6th order inferences. Our results suggest that there is an interplay between model size and finetuning for the realisation of ToM abilities, and that the best-performing LLMs have developed a generalised capacity for ToM. Given the role that higher-order ToM plays in a wide range of cooperative and competitive human behaviours, these findings have significant implications for user-facing LLM applications.
Supersharers of fake news on Twitter
Sahar Baribi-Bartov, Briony Swire-Thompson & Nir Grinberg
Science, 31 May 2024, Pages 979-982
Abstract:
Governments may have the capacity to flood social media with fake news, but little is known about the use of flooding by ordinary voters. In this work, we identify 2107 registered US voters who account for 80% of the fake news shared on Twitter during the 2020 US presidential election by an entire panel of 664,391 voters. We found that supersharers were important members of the network, reaching a sizable 5.2% of registered voters on the platform. Supersharers had a significant overrepresentation of women, older adults, and registered Republicans. Supersharers’ massive volume did not seem automated but was rather generated through manual and persistent retweeting. These findings highlight a vulnerability of social media for democracy, where a small group of people distort the political reality for many.
Pseudo-scientific versus anti-scientific online conspiracism: A comparison of the Flat Earth Society’s Internet forum and Reddit
Federico Pilati et al.
New Media & Society, forthcoming
Abstract:
Attitudes of distrust and paranoia toward scientific and political institutions are increasingly identified as major troubles in online communication and often lumped together under the umbrella term of conspiracy theories. However, this term encompasses two distinct communication practices that deserve to be distinguished. Traditional conspiratorial thinking adopts pseudo-scientific arguments, while newer manifestations lack coherent theories, promoting trolling, and antagonism. We argue that these strands align with different types of digital communications and are supported by different technical infrastructure and cultures of use, with classic conspiracy theories prevalent in early online venues and “conspiracies-without-theory” more common on social media. By comparing the Flat Earth Society’s Internet forum and its subreddit, we highlight their stark differences. The forum prioritizes pseudo-scientific discourse, while the subreddit fosters confrontational antagonism and unmoderated escalation. Recognizing these distinctions is vital for understanding their communicative profoundly different nature and developing targeted strategies to address them effectively.