The Role of Content Moderators: What they do and why are they needed?

Speaker 1

Welcome everyone to today’s webinar where I will be discussing the role of content moderators exploring what they do and why they are needed.

Speaker 2

I am Dr. Michelle Thio, Clinical Director at Zevo Health and a Chartered Counseling Psychologist by trade. And having worked in the TNS industry for the past five years, I’m really excited to share the knowledge that I have gained to date with you all.

Speaker 1

In today’s webinar, I’m going to be covering three different areas.

Speaker 2

exploring content moderation as an operational function of organizations, sharing how moderation functions from the human perspective, and highlighting the potential consequences of life without content moderation. Starting with an exploration of content moderation as an operational function. So content moderation is a function within trust and safety inclusive of human moderators and AI moderation.

For those who may be unfamiliar, human moderators are often considered the guardians or superheroes of the internet who are hidden behind the veil of online platforms and search engines.

These moderators and AI use platform policies sometimes known as community guidelines or user agreements to make decisions about whether user generated content or user to user interactions remain online or whether accounts need to be suspended or removed. These UGC and UTU include things like videos, comments, likes, etc.

And the largest platforms accessible globally like TikTok, Google, Discord, even gaming platforms and consoles like PlayStation all require content moderation to keep users from viewing potentially harmful content. And similarly, smaller platforms will also require content moderation teams.

So moderation teams are dual purpose, safeguarding users from illegal content like harassment, terrorism, intimate imagery, and child exploitation, as well as upholding the desired cultural ethos of a platform. And one of the key reasons that online platforms require content moderation teams is due to regulation like the EU Digital Services Act.

So the EU DSA is designed to provide greater online safety and organizations for which the DSA applies are required to provide greater transparency on their services, adopt procedures for handling takedown notices, informing users in certain circumstances and addressing complaints, as well as refraining from certain practices such as profiling and or improving user control of their services.

According to the DSA, this regulation will support various personas like the ways shown on this slide. And the DSA is not the only regulation which exists in the space of trust and safety. There are several across various countries and states such as the UK online safety bill and increasing regulation is on the horizon.

Speaker 1

So this commentary here…

Speaker 2

sort of sets the stage for understanding why content moderators exist. So Ofcom is the UK’s communications regulator, and they highlight in their recent documentation that the impact of illegal harms online are not limited to the online world. They have a real impact on people’s lives offline, such as instances of grooming.

And additionally, Ofcom highlights how illegal harms have a wider impact on society as a whole. For example, state sponsored disinformation campaigns. They believe that the current trust and safety investment to tackle online harms is just not sufficient. As mentioned earlier, content moderation also encompasses the use of AI systems. So the use of content moderation AI is pretty standard nowadays.

However, humans are kept in the loop for various reasons. Firstly, since machine learning and AI systems still rely on human labeling and annotation, there is a possibility that unconscious bias may slip through. Having a second reviewer in the form of a human moderator who follows a strict set of policies that are laid out for their platforms minimizes that further bias.

Secondly, much of the online content falls into what are called gray areas in the trust and safety industry. So there are types of content which are on a sort of tightrope of acceptable or unacceptable according to a platform’s policies.

And this could be due to contextual factors, language nuances, which makes it difficult for AI alone to determine whether the content actually violates a platform policy.

Speaker 1

If we take a look at this bell curve,

Speaker 2

It crudely demonstrates where AI is really great at identifying what is very obviously unacceptable, like child abuse, and what is very obviously acceptable, like cute panda videos. And human moderators are required to review the bulk of the content which falls into the middle of the spell curve, which are the gray areas.

So you might take an example like communities like my own, Asian women, who typically look younger than their white counterparts.

AI systems may have a little bit more difficulty discerning whether an image of a young female Asian is a minor or an adult, and it may flag an image as potential child sexual exploitation imagery, but a human reviewer may be able to discern more accurately based on their training that the image is indeed of an adult.

So this is an example of a gray area where human moderation is required to make an accurate decision about removing the image from a platform. So how exactly does human moderation function? It’s important to understand that human moderators are not making subjective decisions.

That is their personal values and beliefs do not result in a user account being suspended or a piece of content being removed from a platform. Each illegal harm online or type of abuse will have its own dedicated policy or number of policies that support the moderator to make the accurate decision based on the platform rules, and these will differ between platforms.

So for example, you will see a screen capture here of META’s Facebook community standards, which outline content that is allowed or not allowed based on various categories such as violence and criminal behavior, safety and objectionable content. Other platforms will have similar guidelines for their users, and it’s these guidelines that human moderators use to make their decisions.

As you can see in this image, these policies are subject to updates. So updates may be made based on how users online behavior changes, regulatory compliance like the DSA, what modern society deems acceptable versus unacceptable as we progress, and any other learnings that the platform takes into account in relation to safeguarding their users from illegal harms.

Moderators may work through thousands of pieces of content daily, spanning from innocuous spam to mid-level egregious content such as bullying to highly egregious content such as suicide or self-injury imagery. Each of these decisions needs to be made swiftly and accurately according to the policies of the platform.

And this requires moderators to use higher cognitive abilities such as multitasking, excellent comprehension and critical thinking, and even cultural context and nuance to safeguard users like you and me who are using online platforms.

Speaker 1

If we look at.

Speaker 2

this in even more detail, this is where we can start to see why human moderators are so necessary. So if you imagine that this young woman is a content creator, she is posting various cooking videos online, so she records her video and uploads it to a video streaming platform.

On her platform, her followers can like the video, they can comment on the video, and they can reshare the video to other platforms. Now imagine that these are some of the comments that show up under the video.

There are two comments here which are very obviously offensive, and based on language and context, the platform’s AI system may pick up the first and the third comment as bullying, harassment, or even possibly hate speech.

If the AI system isn’t 100% certain based on its database that these comments are violating the policy, or where there is a consideration that context is needed in order to make an accurate decision, then the comments will be sent along to a human moderator to review.

It is then the role of a human moderator to look at these comments, review them against the policies platforms for bullying, harassment, hate speech, or any other policies that it might violate, and then take an action according to the level of harm. So while this is a very simplified example of human moderation practices, it can help us understand exactly how this process operates.

And what we do need to bear in mind is that there are varying factors that a moderator is contending with when they are making these decisions. So as previously mentioned, policies are numerous and updates are frequent. So not only are moderators required to comprehend and apply these policies, they must also keep up with the changes and change their behaviors accordingly.

Speed and accuracy are paramount in content moderation. If moderators don’t work fast enough, then there’s a risk that illegal harms are exposed to users online at higher rates.

And if they’re not accurate in their decision making, such as suspending user accounts where there was no policy violation, then there’s a risk to the platform that users lose trust and then stop engaging with the platform altogether. The variance of workflows is also a factor which impacts decision making.

So if moderators are working on content that is highly egregious, like child sexual exploitation throughout their entire shift, then this can be harmful to their psychological well-being, which may have impacts on speed and accuracy.

Conversely, if they are switching between child sexual abuse materials, hate speech, spam, and fraud all within one shift, this can be very mentally taxing as the moderator is working through various different policies throughout their shift. And then there are moderators who may only support non egregious workflows like spam or account authenticity where boredom may be the reality.

Again, this can impact on their productivity levels from a psychological perspective.

Speaker 2

Individuals who feel that their work have meaning and purpose are more likely to perform well. And finally, psychological well-being is another factor which absolutely cannot be forgotten.

So as highlighted already, repeated and ongoing exposure to egregious content has potential impacts on psychological well-being and boredom has potential impacts, cognitive load has potential impacts, and both personally or professionally for moderators, feeling a sense of meaning and purpose in their work has an impact on their psychological well-being.

Psychological well-being is another factor which cannot be forgotten. As highlighted already, repeated and ongoing exposure to egregious content has potential impacts on psychological well-being, so does boredom and so does cognitive load. So personally or professionally for moderators, these are all factors that we need to consider when they’re doing the work that they are doing.

As active fence has shared in a blog post entitled content moderators, the cost of burnout, human moderation is necessary due to the nuance inherent to moderation work. So not only does harmful content come in all forms, content shared by bad actors might be disguised as incapable, hiding them from AI detection.

And unfortunately, platforms simply cannot risk false negatives with the most harmful of content such as child sexual abuse, terrorism, or suicide and self-injury. Moderators are juggling all of these factors when doing the work with the end goal of safeguarding their users.

Speaker 1

despite the challenges in human-

Speaker 2

moderation work, one of the incredible outcomes of this work is the real-world impact that they can make. This is just one recent example of a case in the U.S. where a sex offender was charged with felony counts based on the work of moderators.

This offender was using Amazon Photos to upload child sexual abuse materials, and Amazon Photos Trust and Safety Team, which includes human moderators, reported these images to the National Center for Missing and Exploited Children, along with the user’s email and IP address. Investigators were then able to trace that IP address to the offender, execute a search warrant, and arrest him.

And additionally, NACMEC can add these images to their electronic service provider database to disrupt further sharing of this imagery, and may also be able to identify the children who were exploited and provide them supports to foster healing.

So, moderators and their support teams work tirelessly to ensure images like those uploaded by the sex offender are reported to the relevant authorities so that action can be taken to safeguard the individuals at harm. And particularly in cases like CSAM, suicide and self-injury and terrorism, this work saves lives.

And in fact, there are thousands of human moderators working globally to protect users and children from online harms daily. So, what happens if human moderators no longer exist? In the last section of today’s webinar, I’m going to share some examples of what happens when moderation teams are cut down and highlight some data to demonstrate just how needed they really are.

Trust and safety has gained significant spotlighting in the news due to the potential impacts of life without content moderation practices. A few examples of newsworthy highlights. So, first up, xCorp slashing 30% of trust and safety staff.

This article really highlights comments from the Australian E-Safety Commissioner, Julie Inman Grant, who stated that this reduction in trust and safety staff will almost inevitably result in the platform becoming more toxic and less safe for users.

Furthermore, it highlights how this action by xCorp has impacted the brand’s reputation and advertising revenue as advertisers want to be on platforms that they feel are safe, positive, and non-toxic. So, this example provides insight into how a lack of moderation impacts both users and the business itself.

Speaker 1

when considering generative A.

Speaker 2

ActiveFence has shared this useful blog that really highlights several key considerations including GenAI audio impersonation, a wide reach of election disinformation, which is particularly relevant this year with over 40 national elections due to take place, IP and copyright infringement, malicious large-language models and chatbot proliferation, and enterprise GenAI application deployment risks.

So just to take one of those dangers as an example, ActiveFence has shared in their article that dark web chatter on creating generated AI non-consensual intimate imagery of women increased by 360% in 2023 and generative AI sexual abuse materials producers emerged and increased by 335% across dark web predator communities.

So those shocking statistics demonstrate just how prolific online harms are with the advent of GenAI and just how important it is for content moderators to remove this content and to this article from Tech Policy Press provides great insight into the very high cost of cutting TNS corners.

They outline from a legal perspective when platforms do not sufficiently protect users from online harm or refuse to disclose how they’re protecting users ranging from fines of approximately four hundred thousand to five billion US dollars.

Additionally they highlight the impact on advertising revenue as well as negative user sentiment which leads to disengagement from the platform and overall if users are unhappy with a platform and its ability to moderate harmful content online, in all likelihood we are going to stop using those platforms and move to ones that feel safer.

Without moderation users are risking significantly increased exposure to harmful content of various abuse types so transparency reports from various platforms give us further insight into how the proliferation of abuses online can harm users. Consider this graphic this is from Discord’s Q4 2023 transparency report.

They disabled over 23,000 accounts sharing violent and graphic content, over 10,000 accounts engaging in bullying and harassment, over 52,000 accounts sharing exploitative and unsolicited content, and over 115,000 accounts compromising child safety.

In those abuse areas alone that is over 200,000 accounts sharing content that has a real-world impact on users’ psychological health and well-being and children’s safety online if it was made available. Similarly you might consider this table from Facebook’s Q3 2023 transparency report where they demonstrate that they proactively actioned 99.2 percent suicide and self-injury content that violated their policies. So if all platforms are disabling hundreds of thousands of accounts and removing hundreds of thousands of pieces of content we can only imagine what our online lives would be like if they were to stop doing this altogether. The potential consequences are enormous. Now hopefully I’ve been able to make it clear what life would be like for users if content moderators didn’t do the challenging work that they do. And to return to the crux of this webinar, what do content moderators do and why are they needed?

Speaker 2

They protect users like you and me from viewing or being subjected to abuses online. They ensure that children and vulnerable groups can remain online safely without any fears of being exploited, harassed, or worse. They support platforms to comply with regulation and finally they protect brand reputation.

So not only is it imperative for platforms to have robust and adequately resourced moderation teams in place for the safety of their users from an ethical and moral standpoint, but platforms must also ensure that these teams are in place to protect themselves from half-defines and negative user sentiment, which can have longer-term consequences on their overall viability.

So as a final note, I would like to just give huge praise and thanks to the human moderators who are currently working to keep platforms that we use daily safe from online harms.

Without you, the internet would not be the place that it is today and I hope that this webinar can highlight just how important your roles are to those who are new to the trust and industry or simply enjoy spending time online. Here at Siegel Health, we are protecting content moderators and their support team psychological health through best practice human intervention and our technology.

Our aim is to ensure human moderators can work productively to the best of their abilities in service of safeguarding users online. If you are in need of a well-being service provider for your content moderation teams, visit our website to learn more about our solutions and use the submission form to speak to one of our dedicated trust and safety solutions directors.

We are also committed to engaging with the wider trust and safety community to further ours and your knowledge base to find practical solutions to big problems and ensure online platforms are as safe as they can be. We seek collaboration from those in industry, academics, and civil society alike.

If you’re interested in collaborating with us on a future webinar, joining us as an expert guest on our podcast Zevo Talks, partnering with research around human moderators, psychological health, and well-being, or simply just to share your insights and learnings, you can find us on LinkedIn or use the submission form on our website to get in touch.