[Radiolab] Post No Evil

Okay, so we'll try to say something sure. Hey, this is Chad before we launch into this week's podcast. I want to make you wear a boyfriend on the hall. I'll come from my friends, our friends on media, for glasses and see what we have on the show coming out.

I think I'm excited to be in. I'm extremely excited about it. What is it? You are going to be in it.

It's an episode. I don't want to say show because on radio labs, that generally means you're launching a whole new- Yeah, this is an episode on Twitch, which some people know as if it were part of their family. Not people are like, what? Yeah, is that something in any medication for?

So it depends on where you're situated. But what we are going to do is examine how it came to be and how it went to the future of where our culture is going. And for people who don't know what is Twitch. Most people, they've ever heard of it.

They know it's about watching and commenting in real time on people playing video games. No, Drap, hold my gosh, right there. And you guys want to hear profile when these Twitch Superstars? The main character in that story is Ninja.

I'm going to make something like that. What? What? What?

On Twitch? Yes, three contributors who say, save us out loud or point to me. Imagine my name. He is online.

He knows he knows. Another same time he's playing the game, he's even giving advice to young boys with girlfriend problems. And they're like, oh, make one of my kids. They're like, oh, you need to be like, they're like, oh, I'm just like, I'm just like, I'm like, I'm like, putting him wherever it is.

And then like, then I feel like you are. And I'm just like, I just want to make sure that you're just in a sitting position. I think it's something new. Oh, I want you that.

Well, thank you. Thank you for letting me on the top of your show. Absolutely. So Twitch on the media, on the media.

I'll check it out. I'm in there. It's amazing. But not just because of me.

Because you work. Bye. Bye. Bye.

Bye. You're listening. You're listening. Where do you travel?

WNYC. Hey, I'm John. I'm Robert Krollich, and today we have a story about what we can say. And what we can't.

By the way, there's going to be some area of cursor here that we're not going to leave, which makes sense. Given the content of the story. And also, there's some graphics scenes that you got kids with email. My son.

Anyway, this story comes with some producer Simon Adler. Let's start counting 2008. How about what a song? Yes, we can.

I'll see. There are a bit of silly fries up. I'm not going to put that on pictures right now. So December 27th, a sunny Saturday morning, this group of youngutely's women gathered in downtown They're wearing these colorful hats and are singing and swinging directly in front of the glass door headquarters of Facebook.

Yes, it was humble gathering of a few dozen women and babies. That right there is one of the organizers of the gathering. What do you call me that? It's a Facebook nursing.

The intent was really just to be visible and be peaceful and make a quiet point. What do they call me? What do they call me? So Stephanie and the group of mothers, they were on Facebook and many people were in.

They'd have photos taken of themselves occasionally breastfeeding their babies. They wanted to share with their friends what was going on so they would upload those photos of Facebook. These pictures would get taken down and they would receive a warning for Facebook for uploading a pornographic content. And people were really getting their backs up over this.

They wanted Facebook to stop taking their photos down to say that when nudity is not allowed. Breastfeeding is exempt period. Now what Stephanie couldn't have known at the time was that this small, peaceful protest would turn out to be this warning of face off on face black. One of the opening shots.

Facebook trigger and horns now. And what would become a loud, Fuck you Facebook, fuck you Facebook, fuck you, fuck you. And global battle. And that was Facebook's E.

And today playing defense. And now I'm not talking about all the things you've recently heard about, Russian or Fieran's, and Legimetling or Aigha breaches. But rather something that I think is deeper than both of those. Free speech.

Facebook is confused. We can't say, we can't say, but Facebook is a conforte of us. We can't say, and we can't say, We can't say, but we can't say, we can't say, on the end. Thank you, Mr.

Chairman. Mr. Zuckerberg, I'm going to ask you, do you subjectively prioritize or censor speech? Congressman, we don't think what we're doing is censor speech.

But what really grabbed me was discovering that underneath all of this is an actual rule book. A text document that dictates what I can say on Facebook, what you can say on Facebook, and what all 2.2 billion of us can say on Facebook. For everyone in the entire world. For everyone in the entire world.

One set of rules that all 2.2 billion of us are expected to follow. It's an actual document. But yes, it's about 50 pages if you put it off. And in bullet points and if then statements, it sells out sort of a first amendment for the globe.

Which made me wonder, like, why are these rules? How are they written? Can you even have one rule book? Right, exactly.

And so I dove into this rule book and dug up some stories that really put it to the test. Okay. Okay. I'm actually going to hear your English.

Three-ish. Okay. Let's start on that morning. The one that you could argue started it all.

Because in the building right behind those protesting others, there was a group of Facebook employees sitting in a conference room trying to figure out what to do. Cool. So if I... So I was able to get in touch with a couple of former Facebook employees.

One of them was actually in that room at the moment. And now neither of these two were comfortable being identified, but they did give us permission to quote them extensively. How's that? Well, that will take a few minutes.

That's great. So what you're going to hear here is an actually brought into read quotes taken directly from interviews that we did with these two different form of Facebook employees. All right. So at the time when I joined them, there was a small group 12 of us, and that's the lead reason called dreads.

We're sort of called the site integrity team. Again, keep in mind this was nearly 2000. So it's been changed since we can the internet hierarchy. This was like the deep dark past.

MySpace.com is now the most visible website in the US. Facebook had somewhere in the neighborhood of 10 million users. We were smaller than mySpace. The vast majority of them college kids.

And so in those early days, those 12 people, they would sit around in a sort of conference like room with a big glen table. Each of them in front of their own computer. And things would come up onto their screens to Facebook. And like me, like I used their size, something that I thought was wrong.

Exactly. Like a reporting piece of content that you think violates the community standards. This is Kate Klonick. She's a professor of law at St.

John's. And she's a lot of times studying this very thing. And she says in those early days, what would happen is a user would flag a piece of content. And then that content along with an alert would get sent to one of those people sitting in that room.

It would just pop up on the screen. Most of what you were saying was either naked people, blown off heads, or things that there was no clear reason why someone had a board of it. Because it was like a photo of a golden retriever. And people are just in the way.

And every time something popped up onto the screen, the person sitting at that computer would have to make a decision. Whether to leave that thing up or take it down. And at the time, if you didn't know what to do, you would turn to your pod leader. Who was somebody who would have been around nine months longer than you and ask, what do I do with this?

They would either have seen it before and explain it to you. Or you both would know when you'd Google some things. It really was just kind of an ad hoc approach. Was there anything that we didn't standard or any common standard?

What kind of? They had a set of community standards. At the end of the day, there were just kind of one page long. It was not very specific.

Sorry, the guideline for really one page long. There were one page long. And basically, all the stage said was nudity is bad. So it's Hitler.

And if it makes you feel bad, take it down. And so when one of the people sitting in that room would have a breastfeeding picture pop up on the screen in front of them, they'd be like, I can see if he'll be in the rest. So I guess that's nudity. And they would take it down.

Rise up. But that was right. So that's it. Anyway.

Now, it doesn't hurt so people in front of their offices on Saturday. It probably wasn't calling Facebook too much heartache. But you know, hey, we have an opportunity here with, you know, over 10,000 members in our group. According to Stephanie Mira, those protesters were just a tiny fraction of a much larger online group who had organized.

I probably not. So to coincide with the live protest, they just, you know, take up a little bit of encouraging our members that were in the group to do a virtual nursing. A virtual person. Right.

What we did, they posted a message asking their members to for one day change their profile avatar to an image of breastfeeding. And then change their status to the title of our group. A Facebook breastfeeding is not seen. And they caught on.

Social networking by size and a fire for its policy on phones of women breastfeeding. 12,000 members participated and media requests started pouring in. They said the call, hey, they said the rest of the evening. I did hundreds of interviews for print, Chicago Tribune Miami Herald, High Magazine, New York Times, Washington Post.

Tell me about it. You said the interesting thing about it. Dr. Phil, it was a media storm.

And eventually, perhaps as a result of our group and our efforts, Facebook was forced to get much more specific about their rules. So for example, by the news, it was already not allowed on the site. But they had no definition for a newbie. They just had no newbie.

And so the site integrity team, though, 12 people at the time, they realized they had to start spelling out exactly what they meant. Precisely. And so we're in charge of trying to define newbie. So I mean, yeah, the first cut out it was visible male and female genitalia.

And then visible female breasts. And then the question is, well, okay, how much of a breast needs to be showing before it's nude? And the thing that we landed on was if you could see essentially the nipple and Ariola, then that's nude. And it would have to be taken down, which directly would appease these protests.

Because now when a picture would pop up, or another breastfeeding, as long as the child was blocking the view of the nipple and Ariola, they could say, cool, no problem. Then you start getting pictures that are women with just their babies on their chest with their breasts there. Like for example, maybe sleeping on the chest of a bare breast woman and not actively breastfeeding. Okay, now what?

Is this actually breastfeeding? No, it's actually not breastfeeding. Women are just holding the baby in the case. So they included what you can call an attachment clause.

But as soon as they got that rule of place, like you would see a 25 year old woman and a teenage looking boy, and like what the hell is going on there? Oh, yeah, it's really weird if you start entering into a child age. It wasn't even going to bring that up because it's kind of gross. It's like breastfeeding born.

Is that something? So this teen they realized they needed to have a nude and a rule that allowed for breastfeeding, but also had some kind of age gap. So then we were saying, okay, once you've progressed has infancy, then we believe that it's appropriate. But then pictures would start popping up on their screen and they'd be like, wait, is that an infant?

Like, where's the line between infants in public? And so the thing that we landed on was if it looked like a child could walk on his own, then two old. You can have to walk? Too big of a breast.

Yeah, that's a year old in some cases. Yeah, in the World Health Organization, we're getting into 18 months or two years, which meant there were a lot of photos still being taken down. Within days, we've been continuing to hear reports from people that their photographs were still being targeted. But?

I just want to get off for a statement saying, you know, that's where we're going to go on. And keep in mind through this whole episode. Is this perhaps an expect thing? The Facebook.com.

The company was growing really, really fast. It seems like almost everyone on it, and it just got to be a lot more content. When we first launched, we were holding for a number of 400, 500 people, and now we're 100,000. So when I was growing, thousands more people are joining Facebook every day.

The email user so far, the projection of 200 million by the end of the year, now more people in Facebook than the entire US population. Not just within the United States, but also it was growing rapidly more international. You know, you were getting spoken to the media stuff in the internet and Turkey Facebook. Facebook is around.

It's getting bigger out of the view. Korea joins the Facebook. So they have more and more content coming in from all these different places in all different languages. How are we going to keep everybody on the same page?

And so once I saw that this was the operational method for dealing with this, creating this like nesting, set of exceptions and rules and these clear things that had to be there, it had to not be there in order to keep content. We came there for a seizure. And so this small thing of Facebook got a little bigger and bigger and jumped up 60 people in the 100. And they set out to create rules and definitions for every can we go through some sort of ridiculous examples.

Let's go here. So Gore, Gore, you mean violence going to go? Yes. The course entered was, we don't allow graphic violence in Gore.

And then the shorthand definition they used was no insides on the outside. No guts, no blood pouring out of something. Life is a separate issue. There was an excessive blood rule that comes with rules about body fluids.

Demon, for example, would be allowed in like a clinical setting. But like what does a clinical setting mean? And you notice that mean if someone is in a lab code, I want to make favorite examples is like how do you define art? These people are moderating.

They would see images of naked people that were painting for sculptures. Come up. And so what they decided to do is say art with nakedness can stay up. Like it stays up if it is made out of wood made out of metal made out of stone?

Really? Yeah. Because how else do you define art? You have to just be like, is this what you can see with your eyeballs?

And so from now on is they running the problems. Those rules just constantly get updated. Your constant amendments. Yeah, constant amendments.

Your problem? New rule. Not a new problem. Up the rule.

In fact, at this point they're amending these rules up to 20 times a month. Wow. Really? Take for example those rules about breastfeeding.

In 2013, they removed the attachment clause. So they no longer needed to have it to be out physically touching the nipple of a woman. And in fact one nipple and or area could be visible info. But not two.

Only one. Then 2014 they make it so that both nipples or both area of maybe present in the photo. This is what happens in the American law all the time. Yes.

Yeah. It sounds a lot like common law. So common law is the system being back to England where individual judges would make a ruling, which would sort of be a law, but then that law would be amended or a wall by other judges. So the body of law was sort of constantly fleshed out in face any facts.

Literally every time this team is face, but we come up with a rule that they thought was airtight. A plop. Something would be something which the other day that they were prepared for that the rule had to come. as soon as you think, yeah, this is good.

Like the next day something shows up to show you. Yeah, you didn't think about this. For example, sometime around 2011, this content moderator is going through a queue of things. It's apt, reject, accept, escalate, accept.

And she comes upon this image. Oh my god. The photo itself was a teenage girl African by dress and skin breastfeeding a goat, a baby goat. And the moderator first hand says, what the fuck is this?

And we googled breastfeeding goats and found that this was a thing. It turns out it's a survival practice. According to what they found, this is a tradition in Kenya that goes back centuries that in a drought, a known way to help your herd get through the drought is to, if you have a one who's like dating, to have her nurse the kid that maybe go along with her, you know, kid. And so there's nothing sexual about it.

It's just good for the business. And theoretically, if we go point by point through this list, it's an infant. It's sort of good to watch. Maybe there's an issue there.

But there's physical contact between the mouth and the nipple. But obviously breastfeeding as we intended anyway meant human infants. And so in that moment, what they decide to do is remove the photo. And there was a men and asterisk under the rules dating animals are not babies.

We either that's on any future cases people would know what to do. But they removed it. They discovered it was culturally appropriate. Nothing that people do and decide to remove the photo.

Yeah. That our agent visuals are either sort of like, why do we make an exception? Because when a problem grows larger enough, you have to change the rules. If not, we don't.

So it's not one of those cases. The juice wasn't worth the sweets. And like if they were to allow this picture, then they'd have to make some rule about when it was okay to breastfeed an animal. And when it wasn't okay, this is a utilitarian document.

It's not about being right 100% of the time. It's about being able to execute effectively. In other words, we're not trying to be perfect here. And we're not even necessarily trying to be 100% just or fair.

We're just trying to make something that works. One, two, three, four, five, six, seven, eight. And when you step back and look at what Facebook has become, like from 2008 to now in just 10 years. Simon, I've just arrived at the Accenture Tower here in Minilite.

I don't know if he's worth it as one. The idea of a single set of rules that works that can be applied fairly is just a crazy, crazy concept. Because they've gone from something like 70 million users to 2.2 billion. It's hard to take account.

I would say it's about 30 floors. And they've gone from 12 folks sitting around deciding what to take down or leave up to somewhere around 16,000 people. So there's a floor in this building where Facebook supposedly outsourced its content moderators. It's over on 2010.

They decided to start outsourcing some of these places like Nila, where you just are reported or were on the draw. As well as, I mean, we guess that there's those people that are building. That wasn't where we sent a quarter-gareled stack. Oh, it's the same on the characteristics of Facebook and the streets of Tariq's book.

Everybody's beering away. It's not like some of these people who are living in a city of computer and collectively click through around a million flag bits of content that pop onto their screen every day. I'm curious, what's that like? Can I ask you some questions?

We found out pretty quickly. None of these folks were willing to talk to us about what they do. So there's a lot of running away from me happening. Sorry about the effort to be doing that.

No, I don't. Sorry to talk to the difference. No, sorry. Do you work in Facebook?

No, no. I mean, like, you just stay right there. In fact, most people wouldn't even admit they worked for the company. Is there something wrong about being in the idea of this?

Oh, yeah. So when I finally did find someone willing to talk to you. I want to be named or you're not. Or do you not want to be named?

I'd rather not. That's only finding out the industry. I want to be the company. To explain the key and all the other moderators like and were forced to sign these non-disclosure agreements.

Stating they weren't allowed to admit that they worked for Facebook. They're not allowed to talk about what they do. My contract created for him and he's been talking about what kind of moderation was. One.

The several reasons. One is that, until recently Facebook wanted to keep secret what these rules were. So that couldn't be gained. At the same time it creates a sort of separation between these workers and company, which if your Facebook you might want.

I knew I kind of took a monitor of that commitment. Just to give them the nature of the job. You know, you really know the impact that that can have on you until you go through. So this guy talked to you guys for contracting this work several years back and for the duration of the year he jumped to his desk every morning put on his headphones.

You know exactly exactly exactly exactly exactly exactly exactly exactly what he said. I thought it was a image that was a image that was a picture. Five thousand a day you just said? Yeah, he said basically he had to go through an image or somebody's content every three or four seconds.

Wow, all day. All day, that was a day. Well if I can ask what kind of things did you see? I don't know if this is even a whole nother just like video where he has to.

I think he's too much of clicking through. he can cross on people things. I'm heading to, you know, the building spot I tagged to. You know, the content is in drought.

You're like a 13 year old girl heading to a tornado point. And it's not just what? It's over and over and over and over. When did you do this?

Like keep you up and I did this. Absolutely 100%. You catch me talking about these videos and photos when you're trying to relax. He had to start avoiding things.

There were no specific activities I could watch. There was one I think it was a potential one. The microphone has the head to the window. I had the window.

No, I have the walk away. And I just had the head to the window. I thought that it was a different monitor I spoke to described as being the worst side of humanity. You see all of the stuff that you and I don't have to see because they're going around playing clean.

And if we're noting that more and more of this we're being done in an automated fashion, particularly with content like Gore or Terrace propaganda. They're getting better. They get through supervision, they're able to detect hallmarks of the Terrace video or the Gore image. And with Terrace propaganda they now take down 99% of it before anyone's likes it on Facebook.

But moving on to our second story here, there is a kind of content that they are having an incredibly hard time, not just automating but even getting their rule straight on. And that's surrounding heat speech. Oh, good. More laughs.

Well, they will be laughter. Oh, really? There will be comedians. There will be jokes.

All right. Okay. All right. Okay.

We'll take a break in the comeback. No, I think we're going. Okay. So a couple months back.

I think we sent our pair of interns. Oh, we go at the standing room. And why's he eager? I think we're going to get this.

This cramped narrow little comedy club. The kind of place with like the first time I've done. $15. Smash, Rosemary Cocktails.

I got tables. But stills kind of nice. We sent them there to check out someone else who found a fault line in Facebook's rule. I'm really excited to go right on.

The next thing you can come to stage. Can you get over to Mark Jopel's game? Thank you. I guess so.

Matt, I feel like my first one to the city, I was such a carefree brat. You know, I was going, I had these older friends, which I thought was like very cool. And then you just realized that they're alcoholic. So you know, she's got dark curly hair.

It was raised in Oklahoma. I was raised Jewish. So, whenever she was she would read about in Frank a lot. A lot of life.

And when you read that, in Frank, like, this will get funny. She. How did you decide to become a comedian? You know, it was kind of the only thing that ever clicked with me.

And especially political comedy. You know, I used to watch the Daily Show every day. And back in 2016, she started this political running bit that I think can be called sort of a absurdist feminist to come. Now a lot of people think that I'm like an angry feminist.

Which is weird. This guy called me a militant feminist. And I'm like, okay, just because I am training a little gentleman. And the woods.

I just had this running bit online on Facebook and Twitter. She was tweeting, posting jokes. You know, like we have all the buffalo wild wings surrounded. You know, things like that.

Eventually, just a bit on stage. Even wrote some songs. All the white men should die, but not my dad. No, no, no, no.

Anyhow, so about a year into this running bit, Marcia was born at work one day and logs on Facebook. But instead of seeing her in normal newsfeed, there was this message that pops up. It says you posted something that discriminated along the lines of race, gender, or ethnicity group. And so we removed that post.

And so I'm like, what can I possibly oppose it? I really I thought was like a glitch. But then she clicked continue and their highlighted was the violating post. It was a photo of her.

What is the picture? And you described it. The photo is me as what can only be described as a chair of a two little seven year old with big curly hair. And she's wearing this blue floral dress, her teeth are all messed up.

And into the photo Marcia had edited in a speech bubble that says kill all men. And so it's funny. It's funny. It's funny.

So I thought was ridiculous. So she searched through her library of photos and found that kill all men. And I posted it again immediately after like, yeah, and it got removed again. And this time there were consequences.

I got banned for three days after that. Then after several other bands. Shoe for where this is months later. A friend of hers had posted an article underneath it in the comments section.

There were guys posting just really nasty stuff. So I commented underneath those comments. Men are scum, which was very quickly removed. Well, how long did you get banned for this time?

Thirty days. Wow. And so I was dumbfounded. So there's a rule somewhere that if I type men are scum, you take it down.

Yes. And like what could it be? And so Marcia called on her quote, militia. Exactly.

Is this just me female comedians who were sort of like mad on my behalf started experimenting, posting men are scum to see how quickly we got removed and if it would be removed every time. And it was. So they started trying other words. Well, if yeah, they find out where the line is.

My friend put men are just gone. That got removed. Men are the worst. They're moved.

They are. This one girl put men are septic fluid. Band. But we're only at the middle of the saga.

It doesn't end there because they're not really like what the hell is going on. Is it sexism? So they start doing the most bare minimum amount of investigating. She's googled around trying to figure out what these policies are.

And pretty quick she comes across this leaked Facebook document. So this is when I lose my mind. This is when Marc's Zuckerberg becomes my sworn nemesis for the rest of my life. Because what she found today documents Facebook used to train their moderators.

And inside of it in a section detailing who Facebook protected from hate speech, there was a multiple choice question that said, who do we protect? White men or black children. And the correct answer was white men, not black children. Not even kidding.

White men were protected black children or not. That's not a good look. It's racist. Something's going on here.

There's absolutely some sort of un-attressed bias or systematic issue at Facebook. I'm not going to have to say it's not like you're in the middle here. Not long after sitting down with Marcia, Facebook invited me to come up to their offices in California and sit down with them. I mean, one could be in the car.

I'm going to put it in the car. Can I just do your name and your title? I'm Monica Bickerd and I lead the policies for Facebook. My favorite is in charge of all of Facebook's rules, including their policies on hate speech.

And so I ask that they're like, why would there be a rule that protects white men, but not black children? We have made our hate speech policies. Let me erase that. Our hate speech policies have become more detailed over time.

But our main policy is you can't attack a person or group of people based on a protected characteristic like race, religion, or gender. So this takes a couple of weeks to explain. But the gist of it is that the Facebook bar, this idea of protected classes, straight from US anti-discrimination law. These are the laws that make it so that you can't not hire someone, say, based on the religion, their ethnicity, their race.

And so on Facebook, you can't attack someone based on one of these characteristics, meaning you can't say men are trash. Nor could you say women are trash. Because essentially you're attacking all men for being men. Oh, is it the all?

Can I say bodice trash? Yeah, you can say bodice trash because as my source explained to me. The distinction is that in the first instance, you're attacking a category. In the second instance, you're attacking a person, but it's not clear that you're attacking that person because they are a member of a protected category.

So that's why you take it on men or scumba. Why would you leave up black children or scumba? Why would that not be taken out? So traditionally, we allowed speech once there was some other word in it that made it something other than a protected characteristic.

In Facebook, you're are going to be referred to as a non-protected modifier. That means literally not that you can't say that. So traditionally, if you said I don't like this religion, cab driver would be the non-protected modifier because employment is not a protected category. And so what the rule stated was when you add this non-protected modifier to a protected category, in this case, the cab driver's religion, we would allow it because we would have to be a non-protected modifier.

So we would allow it because we can't assume that you're hating this person because this religion, you actually just may not like a cab driver's. So in the case of black children, children is modifying the protected category of black and so children trumps black. Age is a non-protected category. So children becomes a non-protected modifier and their childness trumps their blackness.

You can say what everyone about black children. Whereas in case of white men, you've got gender and race both protected, you can't attack them. That's just a bizarre rule. And we think you go in the direction that you protected class without way of the modifier.

Well, they made this decision as they claimed because their default was to allow speech. They were really trying to incorporate or not to the American free speech decision. And so there's a whole lot of stuff other than none of us would defend as a valuable speech, but didn't rise to the levels that we'd say this is so bad we're going to take it down. And in this case, their concern was we're all members of like, you know, these half a dozen protected categories.

Like we all have gender. We all have sexual orientation. And if you may, the rule is that any time a protected class is mentioned, it could be hate speech. What you are doing at that point is opening up just about every comment that's ever made about anyone on Facebook to potentially be hate speech.

Then you're not left with anything, right? No matter where we draw this line, there are going to be some outcomes that we don't like. There are always going to be casualties. That's why we continue to change policies.

And in fact, since March is the bottle, they've actually updated this rule. So now black children are protected from what they consider the worst forms of hate speech. Now our reviewers take House of Viewer the attack is into consideration. But despite this, there are still plenty of people.

That is why because you are a social network including March, who think this still just isn't good enough. There are not systematic efforts to eliminate white men in the way that there are other groups. That's why you have protected groups. She thinks white men and heterosexuals should not be protected.

Protect the groups who are actually victims of hate speech. Makes sense. Yeah, because in sort of hate speech, or thinking about hate speech, there's this idea of privilege, or historically disadvantaged groups, and that those historically disadvantaged groups should have more protection because of being historically disadvantaged. And the challenge with that that was presented to me was, okay, in the 1940s, you had Japanese soldiers killing millions of Chinese during World War II.

At that same time, you had Japanese-American citizens. The So we asked ourselves a question like are the Japanese and historically advantaged or disadvantaged group? Japanese-Americans pretty easy to make a case and they were disadvantaged, but in China, it's totally different story. In this happened at the exact same moment.

There are two different places, two different cultural stories. When you have a website like Facebook, this transnational community, they realize that they decided that ideas of privilege are so geographically bound. That there is no way to effectively lay and consider who is privileged above, who decided therefore that we are not going to allow historical advantage or historical privilege into the equation at all. And I think it's very important to keep in mind here.

These moderators only have like four or five seconds of the way. We probably can just come to make a decision. In those four seconds, is there enough time to figure out where the world someone is? Particularly give an IP address and can easily be masked.

Go back when you came from. Is it enough time to figure out a person's ethnicity? I bet it's enough time to do that. On top of that, we often don't know an individual's race.

Straight people suck. Other categories are even less clearly. Sexual orientation. And they just realized it would be next to impossible to get anybody to be able to run these calculations effectively.

When we were building that framework, we did a lot of tests. And we saw sometimes that it was just too hard for our viewers to implement a more detailed policy consistently. They just couldn't do it accurately. So we want the policies that he's efficiently detailed to take into account all different types of scenarios, but simple enough that we can apply them consistently and accurately around the world.

And the reality is, anytime that the policies become more complicated, we see dips in our consistency. What Facebook's trying to do is take the first amendment to this high-minded, lofty legal concept and convert it into an engineering manual that can be executed every four seconds or any piece of content from anywhere on the globe. And when you've gotten that fast, sometimes justice loses. That's the tension here.

And I just want to make sure I emphasize that these policies, they're not going to please everybody. Please everybody that's working on the policy team and Facebook, if we want to have one line that we enforce consistently. That means we have to have some pretty objective black and white rules. Like when we come back, those rules, they get toppled.

This is Danny from Denver Colorado. Radio lab supported in part by the Alfred P Slown Foundation, enhancing public understanding of science and technology in the modern world. More information about Sloan at www.slown.org. The federal emergency management agency is fighting for its life.

It was a disaster. It was a good way to be brought. We're waiting up in terms of the FEMA camps. How did the agency task with saving us become so distrusted and despised?

Listen to American emergency, the movement to kill FEMA on this week's on the media from WNYC. Find on the media where every day your podcast. Jet Robert, Media Lab, back to Simon. Facebook, Facebook, Facebook.

So as we just heard before, Facebook is trying to do two competing things at once. They're trying to make rules that are just, but at the same time can be reliably executed by thousands of people's right across the globe in ways that are fair and consistent. And I would argue that this balancing act was put to the test, April 15, 2013. Hey, Carlos, Carlos, let's go.

But if we ask him, we have some breaking news. Otherwise, I wouldn't cut you off so abruptly. On the 15th, 2013, just before three in the afternoon. Two pressure cooker bombs ripped through the crowd near the finish line of the boss America.

And as for the dust being settled, people are bringing into action. This one man, cowboy hat, Caesar Spectator, who's been injured, picks him up, throws him in a wheelchair. And as they're pushing him through the sort of ashy cloud, there's a talker for there. And he snaps this fuck.

And the photo show is that the runner in the cowboy hat and these two other people pushing this man, who is faces action from all of the debris, is sort of standing on end. And you can tell that actually the force of the blast and then the particles that got in there actually holding it in this sort of wedge shape. And one of his legs is completely blown off. And the second one is blown off below the knee, other than the femur bone sticking out, and then sort of skin and muscle attendance.

It's horrific. Meanwhile, on the other side of the country, I remember snippets of the day. Facebook employees were clustering around several desks, staring at the computer screens, watching the news break. And this has occurred just in the last half hour or so.

I have memories of watching some of the coverage. Killing new images just released of the Boston Bony. I remember seeing the photo published online, and it wasn't long after that. Someone had posted on Facebook.

From the folks I spoke to, the order of events here are a little fuzzy, but pretty quickly, the photo is going viral. And we realized we're going to have to deal with it. This image is spreading like wildfire across the platform. It appears to be way outside the rules they've written, but it's in this totally new context.

So they got the other thing together, sat down in the conference room. I don't know, there's probably eight or 10 people thinking about, like, should be allowed. Or should they take it down? According to the rules.

Yeah, so if you recall the no inside on the outside definition that we have in place, meaning you can't see like people's organs or that sort of thing. And if you can't, then we would allow it. And in this photo, you could see you had a phone. And so by the rules, the photo should obviously come down.

Yep. However, half the room says, no, the other people are saying this is newsworthy. Essentially, the photo is being posted everywhere else. It's important.

We need to suspend the rules. We need to make an exception, which immediately receives pushback. Well, I was saying that what we've tried ourselves on was not making those calls. And there are no exceptions.

There's either mistakes or improvements. We made the guidelines for moments like this, which the other side shoots back. Oh my God, are you kidding me? The box of gloves publishing this all over the place?

And we're taking it down. Are you fucking kidding me? They have the guidelines. Let's have kind of sense here.

And yeah, they're kind of the right. But the reality is like, if you say, well, we allowed it because it's newsworthy, how you answer any other questions about any other rest of the stuff. In other words, this is a Pandora's box. And in fact, for reasons that are totally clear, teen consistency, teen follow the rules, eventually wins the day.

They decide to take the photo down. But before they can pull the lever, words are making its way up the chain. And internally within Facebook, according to my sources, and executive language is like a word sent down in order. We were essentially told, make the exception.

I don't care what your guidelines say. I don't care what your reason is. The photo stands. You're not taking this down.

Yes. Yes, that's allowed. This decision means that Facebook has just become a publisher, whether they don't think maybe they have, but they've made a news judgment. And just really, really, they've become CBS ABC New York Times, and they're ultimately always, they're the only ones that just become a news organization.

Yeah. And this brings up a legal question that's at the center of this conversation about receipts. Like Facebook is sort of full-hearted scrapbook for us all, or is it a public square where you should be able to say what you want? Or yeah, is it now a news organization?

That's transparency. Let me get the one. I'm sorry, I'm sorry, I'm going to get the one final question. The kind of relates to what you're talking about in terms of what exactly Facebook is.

And this question has been talking about a lot recently. In fact, it came up with the federal wins. Zuckerberg was testifying in front of Congress. I think about 140 million Americans get their news from Facebook.

So which are you? Are you a tech company? Are you the world's largest publisher? Senator, this is a, I view us as a tech company, because the primary thing that we do is build technology and products.

Senator, your sponsor for your content, which makes that kind of a publisher, right? Well, I agree that we're responsible for the content. But I don't think that that's incompatible with fundamentally at our core, being a technology company with the main thing that we do is how engineers and build products. Basically, Zuckerberg and others have to come here or are you in no, they're not a news organization.

Why? What would be the bounce on that? Well, Facebook currently sits on this little, a legal island where they can't be held liable for much of anything. They're subjective to regulations.

However, where they'd be seeing the eyes of the court as a media organization, that could change. But setting that aside, what really strikes me about all this is here you have a company that really up until this point has been crafting a set of rules that are both as objective as possible and can be executed as consistently as possible. And they've been willing to sacrifice rather large ideas in the name of this. For example, privilege, which we talked about, they decided was too geographically bound to allow for one consistent rule.

But if you ask me, there's nothing more subjective or geographically bound than what people find interesting or important, what people find newsworthy. Now, I'll give you a great example of this. They have just six months after the Boston Air of the BOMM. When this video starts being circulated out of Northern Mexico, and it's a video of a woman being grabbed and forced down to her knees in front of the camera and then a man with his face covered, grabs her head, pulled her head back and slices her head off right in front of the camera.

And this video starts being spread. I can't count how many times. I just reading my Twitter feed. I'm like, oh, you know, like- When person can cross this video, or at least dozens of others like it, which Shannon Young.

My name is Shannon Young. I am a free-man's radio reporter. I've been living here in Mexico for many years now. Her beat is covering the drug war and doing so.

She's actually noticed this strange phenomenon. It first caught my attention in early 2010. She'd be checking social media. You know, you're scrolling through your feed and you see all this news.

People say, oh, there's just three hour ago battle and intense fighting all began long. Folks were posting about clashes between drug cartels and government forces. But then when Shannon would watch the news that night, she'd see reports on the economy and soccer results. But the media wasn't covering it.

There'd be no mention of these attacks. Nothing to do with the violence. And so she and other journalists tried to get the bond with this. Reports on Mexico City with contact the state authorities and public information officer and they'd be like, shooting spongings.

What are you talking about? Nothing's going on. We have no reports of anything. These are just internet rumors.

They didn't want the situation to see not control. But then the video was posted. It opens, looking up when she'll have a car on a sunny day, philanthropy is dry, dusty, and three hours off the shaky, clearly shot on a phone. So you see that?

You can see that. And then the woman taping starts talking. And this woman, she just narrates as they dread along this highway. What am I going to use?

She pans the phone from the passenger window to the windshield, focusing in on these two silver and dried pick up drugs. And she's saying, with these cars over here, they're shot up. Oh, look, you're looking at these 18-wheelers. You know, totally abandoned and got shot up.

I want one sheet. Six of another window to show all the bullet casing, clearing the ground. And she just turned the official denial on its head. The government was saying there's no violence.

Here were cars riddled with bullets. It was impossible to dismiss. And from then on, you had more and more citizens. Citizen journalist uploading anonymously video of the violence.