Minding the Mental Health of Gaming Content Moderators

Speaker 1

So welcome everybody to today’s webinar discussing content moderation in the gaming industry, specifically focusing on defining content mods in gaming, how to set up a moderation team and some advice about armoring your moderators against psychological harms. So I’m very thrilled to be joined by two experts in the field from Keyword Studios. I have Sharon Fisher and Sabine Ernst with me.

Sharon is the Head of Trust and Safety at Keyword Studios and she is 15 years of experience in the gaming industry, starting at Club Penguin as a moderator and then moving to a startup, which was creating software to keep the internet safe. So she sort of quit her role there mid pandemic and started doing some consultancy and trust and safety. And she’s been at Keywords now for almost two years.

Sabine has been with Keywords since August of 2022 as Trust and Safety Wellbeing Manager.

So she’s been in the gaming industry for close to 10 years and she brings her experience from that moderator perspective, but also having worked as a social worker in the EU previously, she has great experience creating wellbeing programs, understanding the risks to moderators and having previously worked at Xbox, she’s now at Keyword Studios. So welcome both of you. Thank you. Thanks, Michelle.

Great, so very first question for both of you, how does Keywords define a content moderator or using your terminology, a superhero?

Speaker 2

Spoiler alert there, Michelle. Yes, so being a moderator is quite a challenge. And this comes literally from firsthand experience when I joined Coping with many years ago. And through the years, it hasn’t really changed the way that we have done moderation.

Obviously, we now have tooling, but the job itself continues to be the same, which the way that we define it is moderators are the people that wakes up every day, sometimes looks at the worst of the internet. And with that, sometimes they save lives in real life threat cases. So that’s the definition for me of a superhero.

That’s why internally and now externally, we’re calling moderators superheroes. That comes also not only with the title, it comes also, to your point, with an armor. And the term superhero, we take it very responsible in this way. It’s not just go out there and find the internet and good luck because now we’re just titled your superhero.

It comes with a responsibility from the side of keywords to make sure that this team has all of the tools necessary, not only to protect their well-being. We don’t only go and share with them many different techniques and practices that will talk in a little bit about it, but also arming them with a different kind of knowledge. So understanding that we all have bias.

A moderator nowadays, it’s gonna be like the way that I best can describe it is that one person surrounded probably by thousands of people that are staring at him or her day and just trying to wait for a decision.

And no matter what decision you make as a moderator superhero, there’s gonna be pushback now, so we want to make sure that we give them all of those tools to make sure that when they’re making a decision, they’re taking these different pieces in consideration. So let’s have been talk a little bit more about what that armor and what those pieces are.

Speaker 1

Perfect. And just maybe before we jump into that, there has been some concern, I think, in the industry just calling in content moderators sort of the guardians of the internet or superheroes as it might place that kind of additional societal pressure on them. So maybe just from your perspective, what are your thoughts on that since you are calling your moderators superheroes?

Speaker 2

I think that’s the difference between just giving a title to someone and then putting the responsibility on them, and, hey, you’re a superhero, yeah, go and find the internet in just very broad ways, broad words. But it is not just like that.

There has to be, as I say, responsibility from people that is employing them to make sure that you just don’t send them to war with CAPE, but rather to give them all of the different tooling. And that’s what we are doing. And that’s what Sabine will talk about, on what do we mean with the armor itself.

Speaker 3

It actually just reminds me of a toxicity report that I just read recently. And I actually said that players, they see moderators, and the game studios as the main drivers of game safety. And assigning them, it was close to 50%, 40, 50% of the responsibility, while attributing something like 40, 50% also to themselves.

So players, while moderation or moderators were incognito still years ago, nobody really knew about what are they doing about their existence. It is now players also that are asking for them. But what Sharon mentioned is like the angle or the perspective that I would like to take is the meaningful work, the impact that they do.

And as Sharon said, it’s not just, or now you’re a superhero, go and protect everyone from harmful content. It is really understanding the impact that they bring. And during our onboarding, I’ll actually share a little bit what superheroes share with us.

So one of the questions is, give me like a reason, not a reason, sorry, but give me something that excites, or what excites you within that role, within that role being a superhero and something that maybe concerns you.

And most of the answers are actually, when it comes to the excitement, I am part, I’m allowed to make a change, to make the internet a safe place, to ensure safety of those who publish, who consume content, who just want to have a nice time and enjoyable time on the internet. And so with that, I also remember, specifically two superheroes.

One said, my cousin committed suicide because of bullying on the internet. So I want to be a change. One said, girls, women in gaming, right? They have been bullied. So I am here as their ambassador. And so with that impact that they bring, it’s a meaningful work. We actually discussed three main reasons and Sharon mentioned them, or some of them.

So one is really, you are willing, you get up to look at the content that you want to protect another pair of eyes from, right? Knowing that what risk that exposure to harmful content can bring to you. The second one is they identify threats. They know how to look for threats. They get trained on that. And with that, they are actually changing lives. Sometimes they are saving lives.

And the third one is, if you think of the high level of cognitive functions that also a superhero brings to the table, many of them are at least bilingual. They have the ability to engage with communication technology, engage with information, awareness of bias, what Sharon also mentioned, right?

The cultural knowledge to understand also the relevant legal guidelines, even just from a cultural perspective, Europe, America, even just a gun culture to give you an example, right? Linguistic competencies and so on. High level of cognitive functions. And that is or contributes to the armor or to our responsibility to really equip them with what they need to be successful in their day-to-day work.

Speaker 3

And yeah, some of that can be daily self-care. Some of it can be further training on a framework on how to deal with bias. Once you are aware of that, right? Team building, creating awareness. That also means within the company, does HR actually know what a moderator does, who is employed and so on? Like you probably have to stop me, Michelle, because I’ll keep going on and on.

But it is that responsibility also that comes with that terminology of a superhero understanding the impact, but then also saying, we will equip you. You’re not on your own. And here is your armor that you can put on on a day-to-day basis.

Speaker 1

No, that’s really great. And I think it leads nicely sort of into my next question, where, you know, safety by design is a phrase that’s utilized quite commonly within the industry in reference to developing platforms with safety in mind for the players or the users. But is there a way that we can apply safety by design to content moderators, psychological health and safety?

Speaker 2

I believe so, but let me go a little bit onto how did we build it internally and then something could go a little bit more into how do we do it. But to your earlier point, superhero term is not something that we take lightly.

One of the pieces that was very clear to us when we’re even building trust and safety at Keyword Studios is that the title seemed to feed the responsibility but we also needed to figure out who was gonna to support this superhero.

So it’s as much as I wanted to like on day two, go out there and literally like hug and protect the superheroes of Keywords, it was not something that will be scalable for me to try to reach to 300 people.

So a little bit of how we design literally the model of trust and safety is exactly making sure that we have all of the pieces to support the superheroes but not also support them ourselves also for them to identify and be able to help themselves and even their coworkers whenever this kind of situations arises, prior the situations arise, not to put their responsibility on them but the knowledge rather on it.

Like we know and we will always acknowledge that we are not psychologists, we cannot provide that.

That’s something that just the experts can provide but our role as trust and safety and something that we identified thankfully very early on the date is to make sure that these kinds of cases are reduced to protect our superheroes, to give them the tools, to give them the space to say when and if they are not okay and then from there pass them to the experts when needed or give them the avenues to get to the experts.

So safety by design is actually applied on the process that we have designed to bring superheroes literally in front of their desk. How are we to Sabine’s point when I came to keywords, everybody was player support agent, right? So we had to come back from zero and be like, okay, HR, we need your expertise.

And again, this is where we leverage everybody’s expertise and the collaboration has been amazing. We need your expertise to help us define this role but also the skills, but also looking to bias and then within the two teams, we collaborated to make sure that the job description was proper for the superheroes. And then after that, the way that you onboard them, right?

That’s part of our secret recipe. The superhero onboarding is something that is totally different from what you can see in any other organization because first off we have made sure that people just don’t speak the language that there’s other skills that we are considering prior even onboarding them. By the time that they are onboarded, they are the right fit for the role.

They come with a certain mindset and then during the onboarding, we just continue to equip them with not just what the role looks like, the product will look like, the operation of things but also the wellbeing side of things and just make them aware that at Keyword Studios we care about this as much as we care for operations.

So now with that intro, I will let Sabine talk more about like how design works in Keyword.

Speaker 3

So as Sharon mentioned, basic safety by design also means that there isn’t the one size fits all support or package, right? And in the beginning, also when I started, the focus within our department was also internal, how do you call it, knowledge upgrade even. So we were very much focused internally, interdepartmentally, do we all understand that line of moderation, right?

What’s the difference between player support and moderation? So it also means understanding really the role, the challenges, and so we had meetings, workshops with our HR, with all the different regions, we came together, we talked about the role of a moderator.

Sharon mentioned first we focused actually on the hiring and recruiting process that everyone has a job title, a job description, a job role, so that they are also aware of what to expect when they come to work without of course traumatizing them, but also they don’t come to work and they think, oh, I signed up for something totally different, right? So that transparency about it.

And then with HR, what we also did, we collaborated, we also created surveys to reach out to the superheroes and really understand their needs. And sometimes they looked very different than what we thought it is, right? It’s not always the exposure to harmful content. For some projects, it was something different that they were dealing with.

And that helps us then also to tailor that going back to the no one side fits all. And then our focus is actually on proactive intervention. So building that resiliency that when crisis hits, crisis intervention, that they go through it and come out on the other side stronger because our focus is really preemptive intervention.

Like we want to make sure that they are equipped in their day-to-day work. And as I mentioned before, one of it is creating that awareness. We have a two day onboarding where we discuss what is the role of a superhero? How do we identify threats? We have one day that we talk about mental health and the risks of it.

And as Sharon mentioned, not that we are the professionals, but that we understand it and we create that safe space and equipping them with resources, but also included in their day-to-day. So there is one thing providing it, but then also taking them by the hand and implementing daily routines. So we talk about daily self-care, weekly self-care, monthly self-care.

For example, also when we work with EAP programs, making sure that they understand what keywords does, what the work entails. And one of the things is Superheroes Day Work shifts, right? So making sure that we have counseling available, clinical counseling 24 seven and not a Monday to Friday.

So another one is we work, when we look at our superhero well-being program and resilience program, we have three pillars. One is the mental health that I just mentioned. Another one is actually the educational side. We have skill building exercises that they take, that we have a curriculum for that. And we discuss things there like bias, cultural differences, inclusive language.

Speaker 3

So whatever they are dealing with, community guidelines, sometimes the gray areas, the taxonomy, right? What are the challenges there? And where can we also help and provide resources? One is for example, the ever-changing language. So we created a virtual space where all the superheroes can meet and discuss terminology.

And again, as being a non-English speaker or first English isn’t my first language, right? As you can tell, looking for words. Yeah, sometimes I just wanna share those terms or say, hey, did you come across, what does it mean? Making sure, so giving them that space that they also can exchange, right? Their expertise and their ideas. And then the third pillar is leadership.

Often you lead a team that is exposed, that does that work as a superhero. So how do we equip them when their team’s members, when the superheroes come to them, approach them? So how to create that safe space? Also how to set the boundaries and say, this is not my expertise, but I can refer you to someone, right?

Also to saying, this goes beyond my expertise and I can actually cause more harm if I give you some advice here. So that leadership support, team building activities that we do. professional help, training, technology is a huge one also that we are looking into. And then last but not least, since we have teams all over the world, culture, understanding their culture is key.

We have some teams, you have wellbeing activities with them, they’re active, they’re engaging, they will share their feedback. With some, it works more or better in of writing. So send out a survey, ask them what stresses them, what they need. So they’re more confident, more comfortable sharing in writing, for example.

And so also making sure that we really understand the different projects, the different cultures, that all in itself, like that means safety by design. So really looking, listening, understanding, and not just coming and saying, we know what you need, we understand it, this is the program, and now you’re basically on your own and you never hear from us again.

Speaker 1

And I think that when I hear you talk about everything that you have in place to kind of equip all of your superheroes with that armor, it’s really about ensuring that there’s an embedded culture of well-being within the organization.

So at every level, whether you’re collaborating with your HR department on job specifications, or whether you’re getting feedback directly from the moderators, or whether you’re working with the tooling department and trying to figure out, is there a more efficient way that we can have tools work for the superheroes in a way that makes their job a little bit easier?

It’s taking that very holistic approach and ensuring that everyone has a part in designing that safety by design piece for them. Great. So in terms of Keyword Studios being within the gaming industry, is there any difference between moderation in gaming versus maybe on social media platforms or other UGC platforms?

And do you feel maybe it changes the way that providers need to think about well-being service provision for moderators on gaming platforms?

Speaker 2

I think there is a difference between the two, but the wellbeing practices should remain the same. So what I mean with this is we are moderating in-game chat, for example. We know that we will be exposed maybe to some toxicity from the game, some lingo from the game. But somehow, in-game, it’s a little bit more of a control environment.

When we’re talking about social media, it’s literally wild, wild west, and it’s wild card. What’s going to happen? What are we going to be talking about today? Like the scenario and the forum is so open to anything and everything. And it could be that we’re commenting on somebody’s comment. It could be that we’re just expressing our own views.

Like it’s just anything and anything, obviously, because there’s not a specific playground. I like to call it somehow as if it was a game that you’re just focused on. Now, however, within these different platforms and scenarios, you will find different kinds of content. So you see. I think that’s where we need to put our focus on, regardless of the platform or where this is happening.

So we know for sure that sound will be something that will stay longer with our superheroes or with any human being screaming, anything like that. So to Sabine’s point earlier, when we’re talking about safety by design, we need to consider this kind of UGC.

The same breaks that somebody’s taking in reviewing, for example, or acting on in-game chat might be different from somebody that is doing VR moderation. Obviously, there’s other senses that are involved that you have to be more focused. Never mind mobility and all of it.

So we really have to look into each project and continue to challenge ourselves when we are doing all of these processes and the implementation of them on the well-being side of things. You said it really, really nicely, I think, Michelle, on the holistic kind of way that we’re doing things.

And I think that’s one of the pieces that really brings a success where we are so open also as the trust and safety team to hear that feedback. And we sometimes sound like a broken record. Please talk to us. Please talk to us.

But really, that feedback that we have, that conversation that we have, not only with leadership, but also with the superheroes, which at the end, guide how we are producing the content, how we are producing the knowledge of great work we’re putting out there.

Sometimes Sabine’s team just literally sits with one of our projects and deals with their own challenges, which it could go from, I don’t know, this specific title is a lot sexual. So let’s make sure that we give a little bit more into that. How can we support people on that sense?

Obviously, for us, first of all, there should be some technology in place so we can make sure that our superheroes can get to the cases faster, but also they are protected from the worst of the worst. That’s number one for us.

And that should be a practice that no matter where your superheroes, moderators are working, either social media, or in-game chat, or in-game in general, this should be a practice that we will continue to push.

Speaker 2

And I think the industry is now taking and running with, if not for the right reasons, which is the good of humans, now because we know that with all the different laws that are coming and different, it just has to be done.

Also, to consider the sheer volume that we’re seeing currently on social and in gaming, you need some kind of automated tool to help you with, through, and equip and empower the superheroes to get to those cases in a faster manner. So yeah, I will say they are different, different size of challenges for each of them.

However, well-being practices and the safety design principles should be applied equally to both of them.

Speaker 1

Sabine, I don’t know if you have any last thoughts on that or if you’re happy to leave it where Sharon left it there with us.

Speaker 3

Actually there, yeah, to leave it with Sharon. One example that I can also, or just to add to it, now I’m contradicting myself, leave it with Sharon, but still anyone. But what I remembered was exactly that tailoring, right? Also from a content perspective.

So when we do training for community management, that means they are engaging also with users, with players online or with anyone really who creates user-generated content, right? There the focus is more how to have those conversations, how to create that safe space where content moderators usually don’t engage with the users, with the community on the other side.

So that is really, again, understanding the role, then tailoring it, but from a wellbeing perspective, yeah, it’s applied equally with that knowledge of that every role has a bit of a difference there that we are tailoring the content to and the training to.

Speaker 1

Yeah and I think that’s a really good sort of last point to leave it on anyway is that well-being in any space whether it’s trust and safety or outside of trust and safety is deeply personal and the way that we approach well-being when we’re when we are working in this kind of human environment is that we need to make sure that the interventions that we’re providing are at that individual level and like you said Sabine earlier it’s not one size fits all but ensuring that all of the right armor is in place to start with and then tailoring as we need to so I think that’s a great place to leave it.

Great so really appreciate everyone taking the time to join us for this webinar keep an eye out we’ll have a few more webinars coming up with Keyword Studios that we’re very excited to share with you and we look forward to seeing you next time thank you very much.