Behind the paper: the role of science in the climate change discussions on Reddit

We talk to Yelena Mejova and colleagues, authors of the recent PLOS Climate publication “The role of science in the climate change discussions on Reddit“

What led you to decide on this research question?
There are several issues that are politically controversial which have to do with scientific consensus; these include the merits of vaccination, the fluoridation of water, and the anthropogenic origins of climate change. Although much has been written in the scientific literature about such issues, and about climate change specifically, we wondered just how much such literature penetrated the everyday discussions about this topic. We chose to study Reddit because it has been a popular forum for discussing both personal and public topics in the US for over a decade (being the “front page of the Internet”). Climate change debate touches not only the political sphere, but personal considerations about what to buy, what to eat, and how to plan one’s future. As these topics are brought up in the thousands of communities (subreddits), we decided to examine just how much scientific literature was cited directly, and what other sources of information people relied on.

How did you go about designing your study?
One of the trickiest steps in the designing of the data was the selection of the relevant discussions. To do this, we compiled a list of 64 phrases that were relevant to the topic, but were not too general. We then manually checked a sample of the data to make sure the posts having these phases were indeed about the topic of climate change. It is not easy, since much of the conversation is contextual (based on other people’s posts), and often non-verbal (using emojis, links, or images). Secondly, we scoured the literature for available sources on scientific publishing, and other kinds of information sources. We especially did not want to introduce our own biases into the definition of political leaning or source quality, both of which we took from journalism databases and fact-checking websites that are acknowledged to be experts. Once the data was prepared and domains selected, the rest of the analysis was fairly straightforward. However, because the data was so large – millions of posts and comments – we also decided to perform a case study of specific Reddit communities as examples of citation behaviors of users having different interests.

Did you encounter any challenges in collecting or interpreting your data?
It took us a while to be certain the data we are selecting is indeed mentioning climate change, since the informal way in which social media users write can be abrupt and depend on the discussion context. It took many rounds of cleaning and labeling to settle on a reasonable approach. Further, we found that the large amount of data available to us was both a blessing and a curse. Whereas we were able to track citations to various information sources over 14 years, the sheer possibilities of the hypothesis space often distracted us from the core insight we were seeking. For instance, we checked the network properties of the subcommunities in the way they were citing different sources, but it was difficult to provide a clear interpretation of the results. Even seasoned big data researchers can get over-excited about the possibilities it holds. However, we are open to collaboration, if somebody in the research community has an interesting research question concerning this dataset.

What struck you most about your results? What are the key messages and who do you hope might benefit from these new insights?
First, we were shocked by the headline result – that only 4% of the links in the posts were to a scientific source. Considering the topic, and its contentiousness, we were expecting at least some of the discussants to go to the primary sources to make their points. The fact that the rate of the scientific linking is a bit higher in the comments – at 6.5% – suggests that it is used more as a response to somebody else’s statement than as an initiator of conversation. It is possible that mass media, newspapers, and social media act as intermediaries and pass along scientific knowledge to their audience, but it is not clear to us just how much nuance these sources retain after the summarization and editorial adjustment. Given how little the scientific literature itself is cited, the role of these intermediaries should be scrutinized even more. Further, it is also possible that the presentation of the scientific sources is not appealing to the general public, and more should be done to summarize and highlight the key points of the scientific articles, such that they are easier to search, quickly scan, and use in a debate.

What further research would you like to see in this area?
Social media continues to be an important source of news, commentary, and educational material for people around the world. Video is becoming increasingly dominant, and new tools are necessary to understand the quality of information in multimedia posted on platforms like TikTok, YouTube, and Instagram. With the rise of generative AI, it is even more important to track whether scientific information is communicated accurately, how original research is cited, and whether it is accessible to the general public.

What made you choose PLOS Climate as a venue for your article?
PLOS Climate is a leading venue for climate-related research and covers a wide scope in the disciplines. We felt our study may reach not only other researchers interested in the quality of information on social media, but also those who engage in scientific communication on a daily basis, and for whom the success of this communication is vitally important. 

Ready to submit your own work to PLOS Climate? Follow our step-by-step guide to the submission process!