a year of content reviews

a year of content reviews

Analyzing a Year of Content Reviews: Trends, Insights, and Implications

1. Introduction

In this paper, we leverage our unique partnership with Facebook’s Community Operations team to quantify a year of content controversies. Specifically, we have focused on understanding the volume and nature of rule-breaking content that could harm life on the site if not handled correctly. We then strive to interpret this information within the ethical framework of the company in order to draw lessons from the statistics. We write as academic researchers but with the practical goals of helping Facebook better understand and aid the organization in its ongoing content management efforts. Since questions and uncertainties exist at all stages of effectively conducting review, we further provide a set of recommendations, raise open questions, and suggest areas for future work. Our hope is that our experience and learnings can help guide future research and industry collaboration.

As an academic research and practical operations organization, we have great interest in obtaining an understanding of the kinds of violations that content reviewers handle on a daily basis so that we can help foster a healthy community, cooperate with law enforcement, and drive future product and policy efforts. Unfortunately, relatively little information about sensitive content flows from companies such as Facebook to academic researchers due to the reputation risks that a company may face over such sensitive data.

Every day, hundreds of millions of people use Facebook to share their thoughts, images, and life events with friends and family in a quick and easy manner. We continually strive to keep the Facebook community a safe and secure place for people to express themselves. Given that some of this sharing may be controversial content, Facebook has an online team of thousands of content reviewers that work 24 hours a day, 7 days a week, 365 days a year to handle such potentially sensitive and harmful content.

2. Methodology

Maintaining and enforcing community guidelines often involves detailed qualitative content reviews. In this paper, we present the results of an analysis that was conducted during the 12 months before September 2019. For one year, we used our custom content review system to examine posts classified as violating at least one of the community standards from dozens of commonly replicated groups. This large volume of community guideline reviews allowed us to capture trends and understand better where improved guidelines or enforcement might be helpful. This paper serves as a traffic report, providing an overview of the patterns we found and the changes designed to reduce hate speech, misinformation, adult nudity and sexual activity, terrorist propaganda, and drug and firearm sales, while limiting mistakes. As always, we remain concerned about the potential implications of this work, including that our decision to share may underestimate real-world prevalence.

3. Key Findings

As shown in Table 1, five of the ten categories are about violence or hate speech, and four are related to broader mental health issues such as self-harm or substance use disorders. Not every example in the dataset neatly fits into only one category, and some categories – like harassment or bullying – have gray areas around questions of how to treat satire or statements made by members of historically marginalized groups.

To answer these questions, we began with a much broader set of individual content review cases collected through Google’s web search product. Based on internal subject matter expertise and a review of the literature, we identified ten categories of content that are often challenging for content reviewers to assess.

– What are the key content moderation challenges on the major online platforms? – How are companies moderating different types of content at scale? – What data is needed to properly assess the state of content moderation and what company practices might be contributing to the problems outlined in Surveying the Digital Future reports?

This analysis was built around three key questions:

4. Implications for Content Creators and Marketers

Given the recent settlement with Crowdtangle’s new owner, the original Facebook partner agreement which provided all the data for our analysis is expected to be revived. Still, the need for consistently representative, open platforms is evident. The proprietary hurdles we encountered as we built this report are significant, and serious questions about the long-term viability of Crowdtangle data, and thus its use for academic research, should be asked. Offering a window into private platform processes is valuable but not the same as opening APIs. Furthermore, easy-to-access and use proprietary datasets can lull researchers into a type of data laziness, and we urge social media platforms to focus on making all data both easier to use and more accessible. In addition to making data more generally available for media research, interoperability of data across platforms is a key concern of researchers.

Content creators and marketers who have yet to dive into the Facebook content contained in the Crowdtangle repository will soon have access to a wealth of understanding. The content insights presented here are limited to text data – significant metadata is not included in our text dataset, so many types of insights are not possible. Frustrations around platform data accessibility across many platforms are widespread. 72% of respondents in a recent Tow Center survey of research tool needs identified these issues as barriers to producing reliable media scholarship.

5. Future Research Directions

Many questions remain around what it means for platforms to “move fast and break things” in content moderation. Without a more complete understanding of how these changes are experienced by users in different populations, it remains difficult to weigh current techniques aimed at modeling and mitigating these effects. Getting a deeper understanding of why effectiveness levels can be unpredictable across subpopulations in the first place may help to guide exploratory, potentially creative techniques that are more effective for differing communities and beliefs. Increased understanding of the unique challenges faced by different content review populations will help to better support these crucial individuals, in part by introducing more helpful interventions and policies that are better aligned with their unique experiences and needs.

Previous work has considered a number of factors that may drive different norms surrounding social interactions online. As we noted earlier in this paper, teasing apart the effects of site design and site content remains an open question in this space. Despite some promising exploratory research in this area, it is still not clear how potential targeted interventions at the content-level (e.g., norm-setting language, etc.) might perform falling on the totem pole of effective content moderation. Research identifying the roles that gatekeepers play in establishing platforms’ interactive norms and monitoring their adherence to these norms has also been shown to help identify ways to encourage or enforce better norms.