What we do is basically we combine this very, very deep subject matter expertise with our technology. So we're a technology company. And yeah, we also have sort of experts in the field, in the domain, like experts in the field of researching human trafficking and really understanding that space or in misinformation and different types of misinformation and in hate speech and in terror. And so they speak the languages, you know, they research the space, they understand it, they know the key players, they know the different organizations, the keywords. And, you know, this is an adversarial space. It's constantly changing. And so they make sure that they stay up to date. And what that means is that.
us on the data side, we can basically take their ideas, take their knowledge, and then engineer features out of those to really translate the human knowledge into our models. So then we can go out and automate that and do it at scale. Welcome to Practical AI, a weekly podcast making artificial intelligence practical, productive, and accessible to everyone. Subscribe now if you haven't already. Head to practicalai.fm for all the ways. Special thanks to our partners at Fastly for delivering our shows super fast to wherever you listen. Check them out at fastly.com. And to our friends at fly.io. We deploy our app servers close to our users, and you can too. Learn more at fly.io.
Welcome to another episode of Practical AI. This is Daniel Whitenack. I'm a data scientist with SIL International, and I'm joined as always by my co-host, Chris Benson, who's a tech strategist with Lockheed Martin. How you doing, Chris? Doing very well today, Daniel. How's it going? It's going great. So yesterday was voting day here in the US, and I did go to the voting place. And it was interesting because in line, I could hear people talking about cyber threats to the voting machines and other things like that. And so my mind was already actually thinking about these things because...
We have a really interesting topic to talk about today that's in that same vein. We're privileged today to have with us Matar Haller, who is VP of Data at ActiveFence. Welcome, Matar. Hi, thanks for having me. Yeah, and ActiveFence, I've read a bit about it, and the sort of website talks about this barrage of threats that online platforms are susceptible to now, which ActiveFence is addressing in various interesting ways, which we'll get into. But I'm wondering if you could kind of give us a picture, like if I'm going to run an online platform of some type, maybe it's not even like...
you know, I'm likely not going to start and run the next Facebook, but I might very well, you know, start and run some type of software company that provides a online platform to do something. What should be on my mind? And what's the reality of kind of online threats that I might need to be aware of if I'm getting into that space? Yeah. So first of all, I think there's one thing to think about is that anytime that you have a platform that has any type of user generated content, whether users are uploading photos or they're chatting or they have comments or anything like that, you're going to have tons of data very, very fast. And just it's prime for people to, you know.
post-wonderful things, but also some really, really dark things, which we've all sort of seen and been exposed to. And so one thing to keep in mind is that trust and safety, just basically safety online, it's not really a nice to have anymore. At this point, it's a competitive advantage. It's kind of a basic expectation, right? So users are expecting it. Advertisers are expecting it. Parents are expecting it. The public expects it. So if you're going to spin up a platform, so first of all, best of luck. And second of all, you need to keep this in mind from the get-go before you find yourself down this rabbit hole. One thing that I think is really important to keep in mind is that
Although trust and safety isn't a new industry, really, it's really now finally becoming something that people are aware of. Like I said, it's this basic expectation. Now it's not only users, but also like regulators and legislators. There's new legislation coming in that's making it even more at the forefront. And the basic sort of content moderation that is out today doesn't really make the cut. To follow up like the second part of your question about sort of like what kind of harms are out there.
So online harm is really multidimensional. We can see it in different media types. So we've seen it in games and like merchandise sites, chats, text, you know, video, audio, things like that across many, many different languages and also different types of violations. So you have, you know, white supremacists and terrorists and human trafficking and sort of these like really painful sorts of things. Also goes into like misinformation, disinformation, fraud, spam, cyberbullying, and so forth. And so it's this really, really complex space that you need to have a deep understanding of to understand how to address.
I guess up until this point, you were talking about like content moderation and how it has evolved over time, but it's still kind of lacking in the sort of traditional sense. What does it look like? I mean, is content moderation like people might have in their mind, oh, I have a blog, right? And I'm going to choose whether I allow people to like post a comment or I have to approve that comment before it's posted or something like that. So this sort of content moderation, is it?
In your opinion, kind of as where we sit today, is most content moderation sort of reactive at this point? Or how do you view like, you know, how are most people approaching the problem right now, maybe? And why is that lacking? So there's kind of different levels, I would say, of content moderation. At this point, to just sit and moderate every single comment gets out of hand really, really fast. And so there is some level of automation that, you know, started being introduced, right? And so the first sort of basic level is let's go out and look for keywords, right? Like, I don't want any slurs on my platform. I don't want anyone calling anyone any of the like.
words, you know, I don't want that there. So I'm going to ban all those. I know that people might have, and then people kind of get a little trickier, right? And so I'll say, well, you know, what about if we use an emoji? And so there's different kinds of emojis or combinations of them that also can be used in a hateful way. And so you can say, okay, well, I'm going to, I'm going to ban those. And I'm going to ban like these specific keywords. And I'm going to ban these engrams because, you know, these phrases are bad. Like I hate Jews or I hate what, you know, so I'll ban that. And that, that works. Okay. But very, very quickly, you get to these cases where keywords are either insufficient. So like I said, with emojis or with lead speak. So
You can write for people that aren't familiar. It's basically taking a word and replacing letters with numbers. So Adolf Hitler can be written like pretty much all the numbers. And so it has to evade detections and only on a need to know basis sort of thing. There's also in the keyword space, you have numbers. So like 1488 is a white supremacy number. 88 for Heil Hitler and 14 is like a phrase that they use, like the number of words in the phrase. And so you can say, OK, well, I'll add all these to my dictionary. But then you get to the place where you say, well, what about someone that's telling someone else? Don't call me.
you know, some slur, don't call me that. And so now they're using the slur, right? Is that something that you want to necessarily ban? Maybe in some cases, but in other cases, you're going to need sort of a deeper understanding of how language is being used before you just go out and outright ban it. And so you see this sort of like evolution in terms of the ways or the approaches that platforms are taking to moderate the space and sort of looking at the context in which language is used is sort of the first step in that.
I think I just realized and that was a great explanation. And I just realized how sheltered I am because several of the things you referred to, I just didn't know at all. So, yeah, I guess I'm very sheltered. But I'm curious when we were first starting the conversation a few minutes ago, you also mentioned misinformation. And we were just kind of diving into kind of some of the specific use cases on hate speech. How does misinformation because we've seen, you know, we've been dealing with hate speech for a long time now. But, you know, misinformation in the last few election cycles has really become a huge issue. And obviously in national security issues and things like that, it's big.
How does that fit in? I think in my mind, I've kind of thought about there's hate speech and there's misinformation and all. Is there a connection between them all? Are they bound together in some way the way you see it? Or are these distinct, separate kinds of things? How do you think about it? How do the folks at your company think about it? That's a really interesting question to take sort of these two examples of these two specific violations. And there's, I think violations are sort of on this spectrum, right? And so you can think about these sort of more like evasive violations. So things that are, you know, that kind of are more difficult to find or require subject matter.
knowledge so hate speech right you need to know these keywords you know these things and then you also have these just sort of more like common violations where some of them are like well maybe it's not even a violation like nudity or profanity or things like that where it's just more more out there and everything kind of lies almost sort of on the spectrum you can go like spam and fraud and so forth until you get to like the really dark you know like um like child safety and abuse and things like that and with misinformation it's actually interesting because it's not really trying to evade right like that's the whole point um on the other hand it's really tricky to find and to understand and there's lots of organizations that do really wonderful work of fact checking and of you know like keeping up on the trends and really identifying misinformation and there's also sort of these techniques that we can use to, once we've identified sort of like a specific type of misinformation, then we can use it to find that it's going viral and so forth. It's a real struggle to understand like, you know, kind of how to put that in context because you could say things in a misinformation context that are, you know, there's no hate speech in it, you know, explicitly, there are no banned, there's none of that is there. And yet, as we've seen in recent years, it can do great harm. So it seems like a very hard target to go after and be able to mitigate it in a sane and reasonable way.
Absolutely. I think that's one thing that's really unique about Actifence and what we do is that we're, what we do is we basically, we combine this like very, very deep subject matter expertise with our technology. So we're a technology company and yeah, we also have sort of experts in the field, in the domain, like, you know, experts in the field of researching human trafficking and really understanding that space or in misinformation and different types of misinformation in hate speech and in terror. And so they speak the languages, they, you know, they research the space, they understand it, they know the key players, they know the different organizations, the keywords, and this is an adversarial space. It's constantly changing. And so they make sure that they stay up to date. And then what that means is that.
us on the data side, we can basically take their ideas, take their knowledge, and then engineer features out of those, right? So really translate the human knowledge into our models. So then we can go out and automate that and do it at scale. The reason that it's so interesting is because as you all know, models drift, they decay. And so you can go out and you can retrain your model and get new weights and you're great. Except if you're in an adversarial space, then not only are you drifting, but your reality is like, it's so not stationary. It's just changing from underneath you. So as it's changing from underneath you, you need to like hurry up and re-engineer your features. And so we're constantly...
engineering your features, retrain your models, and also thinking about just like what else can we possibly extract from this data that's coming in, right? We're analyzing text, video, audio, everything, basically anything that we can get our hands on and just really milking whatever we can out of it. That's super interesting. I have so many questions, really interesting, really interesting technology, but also like the infrastructure and management part of that is I'm sure a great challenge, but I'm glad you brought up the modality thing. And I also saw, you know, on your website, you talk about different languages as well, which it seems like this is definitely now in terms of the way people communicate online.
Like you mentioned emojis. I was also thinking of like GIFs or GIFs, depending on who you are, or, you know, posting memes with text, you know, in the image. There's also, you know, of course, like you're talking about videos and audio messages, all of that, I guess, as a more general question. Is language, but sort of like multimodal language, is that like your primary area of research? Or are there other things like outside of communication in terms of like the threats posed to sort of online platforms where someone's not trying to communicate a certain messaging or something, but it's still a threat to the platform in one way or another? I guess maybe spam would be an example of that, but I don't know if you have other examples or is it really kind of from your view, a lot of what you focus on is the communication and language piece.
So the goal, it's not necessarily language. And in a second, I'll talk a lot about like contextual AI and what that means. But really our goal is to enable users to be safe online, to have a safe experience, right? I'm a mom, I have three kids. I started working in ActiveFence and I said, oh gosh, my daughter is not getting a cell phone until she's 35. Like forget about it, right? Because you're suddenly exposed to all this. And then you say, but you know, that's why what we do is so important because that's our whole goal. It's not only about language and that's just one form of communication. There's lots of things out there.
like we're a bunch of concerned parents like let's really make this like the fact that chris is still sort of in this sheltered bubble is amazing like i want everyone to be in this sheltered bubble right um and so that's that's kind of the idea i think i just need to correct it it depends there's some bubbles i'm sheltered from i think and there's some i probably am not probably so yeah yeah like when you're talking about the hate speech the specific numbers that meant stuff i was like i didn't know that so anyway i didn't mean to cut in but why would you um yeah so i mean so so so language is only one part of it and i think one thing that we really get into is um is context and so let me let me take you on a journey through context and we'll end up with at means which to me is like sounds great So we were talking about like the language context, right? And so how keywords and engrams, they just don't cut it, right? You need like language models. So, you know, transformers and so forth to really get an understanding of what is being said and the context in which it's being said. And so those are the kinds of models that we end up training and that we have data for. We have, like I said, we have like our subject matter experts and policy experts that are able to ensure that we're capturing things that are sort of on the edge and border because that's where things get interesting, right? And that's where I want to be able, that's how I'm able to get the difference between, you know, I'm proud of being a whatever because I'm like reclaiming that word versus.
you know, you are a, you're not allowed here. And even within the hate speech of language, there's, you know, insulting hate speech, not insulting hate speech, you know, like, hey, wasn't that a great KKK rally yesterday? I really like your Proud Boys tattoo, right? It's hard to catch those things. That's a surreal statement right there, that example. That's never been said on your podcast, right? No, it's never been said. Just like you said that, I was like, wow, I'm not talking to the people that are saying things like that. So anyway, sorry, go ahead. It's novel to me to hear some of this perspective.
But then we can sort of go to the level of, okay, so let's say we're now in the image space, right? And so one thing that we're able to do, and again, because we have this deep subject matter expertise is we're able to search for logos, right? So logos of terror, you know, like rare terror groups that are small that are hard to find. So we know this space. And so we're able to go out and find, you know, do logo detection, find those particular logos, identify things. And then you say, okay, great. So here's the video, found the ISIS logo. Great, check, terror. And they say, well, wait a minute, but there's also the CNN logo here. So suddenly, even though it's a snippet from ISIS, it's, you know, the context in which it's used.
is not, doesn't make it violative. It's something that's interesting. It's important. It's historical. It's whatever. You can see the same things with like videos of Nazis marching, right? Like sometimes that's glorified and sometimes it's just like historical, you know, it is what it is. So that's another level of context. We have to sort of look at the context of the, you know, one signal out of the image isn't enough or out of the video. Another thing that we like looking at that, you know, is that you can look at the context in which the image is being used, right? What is the title? What is the description? What are the comments? We have an example that I like using where you see sort of non-violative text, you know, I love him.
And you're like, oh, that's fine. Who cares? And then when you zoom out, you see it's like, I love him with a picture of Osama bin Laden. And suddenly that's suddenly more interesting, right? Suddenly it becomes violent. So you can't just take any one piece in isolation. Or there's an example of like some chef that's like, it's hard to do this on a podcast, but like showing knives. And he's like demonstrating these knives and showing these knives and his hands are all cut up because he uses knives. And if you do just object detection, it screams at you like weapon, weapon, weapon. Like, oh gosh, this is like this terrible video. And then you analyze, you know, the title and description and the comments and the channel and everything along with it.
And you're like, no, he's teaching about knives. It's a chef video, really not interesting. And so looking at things, just as keywords in a sentence aren't enough, also just looking at an image by itself isn't going to tell you whether or not something is problematic. And so that's where this idea of contextual AI that we think about a lot is like, what is the context in which something is used? And context can mean lots of things. It can also mean the policy, right? So different platforms have different policies. Some platforms will say baby's first bath is like child abuse, like you cannot have it.
you know, child nudity and others will say not a big deal. So it's like another level of context that our models need to deal with. I mean, just knowing about where NLP models or other models fail, like this area of like sarcasm and humor is so difficult. And there's like this further distinction that you're drawing out, which is, well, there's some memes that are jokes and sarcasm, right? There's some memes that are jokes and sarcasm, like to the point of being very, very harmful. And also that's kind of tied into the context of where they're.
you know, where they're put or the timing of when they're put somewhere or something like that. So I'm, I'm wondering if you could kind of break down, like as, as you're kind of, as you're stepping into addressing some of this, you already mentioned like frequently you're updating like new features, like there's new behaviors that you're seeing that didn't exist before. So like, let's say that ActiveFence like starts to understand that there's some type of new, new behavior that's, that's harmful or something like that. What is your process and how do you think about going from like, knowing this is happening to detecting that this is happening in a repeatable sort of way. So there's sort of a couple of different ways that we're basically staying up to date. So the first is really, really close contact with subject matter experts that are out there gathering information, intelligence, researching, collecting data, building like keyword databases, looking for particular, you know, bad actors that frequently post things. They're really there. And so we're frequently talking to them and understanding like, what was it that made this pilot? Or they'll send us a report and be like, hey.
this is a new hate group or, Hey, this is a new meme. And so that's, that's one thing. The other thing is that even within our models, we're constantly getting feedback. We have something that we call the database of evil, which is like, I mean, might as well call it what it is. That has to be the best name I've ever heard, you know, for it's the database of evil. And it's true to its name. I believe it. Yeah. And so we keep that, that's it, right? So we have data coming in.
We score it, you know, we give it a risk score, which is essentially like the probability that it's violative for some violation. And then we have trained analysts that review it, review the score and sort of can say, yes, no, no. Anything that's verified as being violative, it goes into the database of evil. The database of evil is sort of used for a few things, one of which is new content that comes in. We can say like, well, have we seen this before? Do we know it? Things that we've seen a lot, they're more like, you know, versus things that are brand new.
And it's also used because as we take this feedback, we're constantly, like I said, we're retraining, right? We're learning. And that's like, those are the small adjustments that we can make to our models. And so it's this idea of like constantly getting feedback, both just, you know, from researchers that go out and find things from. The data that's coming in is being scored. And then, you know, we're sort of retraining on top of that. And then, of course, we have our database of evil. So let me ask a question. Obviously, as you pointed out, your database of evil has some has a lot of really explicitly evil stuff. But I'm also imagining that there that there are gray areas that like you kind of mentioned the baby's first bath kind of thing.
You know, that would depend kind of on audience on whether that was if a family member showed me, well, we have a new baby in our family. My niece has a new baby. If she showed me a photo of the new baby having his first bath, that would not be offensive to me. But there are contacts where posting it online, it could become offensive and such. So with these types of gray areas and the fact that you can have one set of content that has a bunch of different, I don't know what you would go, you know, acceptable rankings, if you will, you know, depending on on who is viewing it and what context and all that. How do you approach?
making sense of all the gray area? Like, you know, when there is everything from perfectly fine to absolutely not fine, and it's all valid for the same thing. Right. That's a really pertinent question. And it's something that we're dealing with a lot. Part of it is sort of, and we haven't, I don't think that we've completely cracked it. But one thing that we do do is that we can also have database of evil where evil is relative for the client, right? And so, which also leads to sort of, you know, this idea of customized models per client based on.
the feedback that is coming from them, where basically you can say, okay, so when we have this now, right, we have two clients, one is babies first bath is violative and one it isn't. And so we already are juggling this with different levels of like, of, you know, of human intervention to get it to sort of really to perform, because that's what's training it to get it to that point. And there's even other examples where, you know, if we take it, you know, babies, babies first bath and things like that, people are like, you know, oh, but it's so clear what, you know, child abuse and pedophilia are. And clearly it's not. And even things like asking someone, are your parents home? So if it's a conversation between two children.
on a chat room that's totally fine but it can take a much darker turn when you suddenly see that it's you know a user that is also in you know adult chat rooms or it's being posted like you know at around 8 39 10 p.m when kids are supposed to be starting to go to bed or to be in bed i don't know my kids are going to bed early and so and you know and so you can start so even then like you can say well i have language understanding and you know this is a kid's chat room but suddenly there's like all these other levels that you need to take into account to understand if a phrase really is just Nothing. Just as kind of setting some, I am living what you just described. I've grown kids, but I also have a daughter who is just getting to the point where we're letting her get online and do some of the stuff. And some of it's in supposedly safe environments. But then as the nosy dad who's just worrying about keeping his child safe, there are all sorts of gray areas and stuff. So it's fast. And there are all sorts of moments where I'm having a lot of trouble telling whether it is a safe context or not. It's not very clear. And so I can imagine that that is extremely challenging.
you know, to solve as a technical problem that can be, you know, recreated across a lot of different audiences. Totally. And I think that there's always going to be to some, like, at least when we're like training or whatever, right? There's always, there has to be a human in the loop. And for these gray areas, so we do as much as we can with technology and moving it there. But even if as a parent, you're looking at it and you're saying, you know, I don't know. And so sometimes we can leverage things that you don't have access to, right? Like we can look at the history of the user or the other chat rooms or other things that are going on in the space or how this, you know, who has been in this chat room before. But, you know, sometimes it comes down to, you just don't know. So I,
I have a lot of, Chris always knows I like to ask a lot of practical questions before I get to those in terms of like some of the things you're doing and how you're doing them. I am wondering for a company that's using some of ActiveFence's technology, what does that connection look like? Like one of the examples I'm thinking in my mind is I'm working on a website for some of our partners where people can contribute like and list cards for tools that they're working on, like software tools. And, you know, technically like they could submit anything in that description of that tool, you know? Now I think like...
we have hopefully vetted people that will be submitting content and actually not everyone has accounts. And so it's fairly restricted. But yeah, I'm wondering like in that situation or a much more scaled up situation, what does it, is working with the right view to have like your software platform and then you send off content to some API and get a threat score or something and then you figure out what to do with that threat score in terms of. How does this actually practically work out for a company in terms of, because I imagine it's complicated. Every company has like their different platform, right? And also like, oh, you know, the format of a Facebook message, you know, going to a webhook is going to be different than like a blog post being posted to a content management platform. So in terms of like data moving around, how does that work out practically? Yeah, so there's, I think there's kind of two parts to it, maybe more parts, but.
two main parts of the question. The first is like, how would you as a user interact with us? And so we have a UI, a platform where you can really, you know, see the content that's coming in and you can define sort of codeless workflows where if, you know, something is above a certain risk score threshold, then, you know, it's automatically filtered out. If it's below a particular risk score threshold and you don't even look at it and then like, what is your threshold for human moderation? That sort of gets between, that sort of gets around this sort of like precision recall conundrum where you're like, well, I set a threshold and I always have to choose, you know, what am I maximizing? And you can say, well, let's set one threshold where you're maximizing your precision and, you know, another one where you're comfortable with the recall and then you look in this sort of band.
And then you can use that to moderate. We also have an API. You can send, you know, you can do like synchronous calls for text. So like near real time, really, really fast. So for chat, if you want to try, you know, pre-publish and so forth. And we also have async for text and for images, for video, sort of, and, you know, more like we can send the full context, right? So you can send your content and you have like the body of the media and the title, description, whatever you have. That's for the first part of your question. And for the second part, I think it's maybe like a little bit more interesting kind of, because everyone, you know, you can build an API or whatever, but what we've spent a lot of time on is both like...
optimizing our API, so making sure that it's very robust and responsive and so forth, and then also modeling our data. And so we have a very rich understanding of the world of platforms, like how we can model the world of online media or online platforms or user-generated content. Pick your favorite term. We model it. We have a very robust and flexible schema where we're able to sort of model a user and it's related to the post that they put and how many likes they have. And it's not always relevant. You don't always need to use it all. But we have users and we have contents and we have collections and each of those are modeled a bit differently. And so once the data comes in and we adjust the data and it's modeled like this, then we can go ahead and take it apart and score the different parts of it. And then through our API, which is able to handle really high throughput and fast SLA, basically start giving you responses. And we've done a lot of work.
on our backend optimizing. And so we're batching models on GPUs and doing all sorts of, you know, thinking that, you know, we have all kinds of like code that we've run that basically optimizes like what machine type you want to run and basically make sure that everything runs as smoothly and as robust and reliably as possible to get those responses out. So, Matar, one of my questions is just in the back of my mind is like just the practicalities of running the type of platform that you're building and the service that you're running. Like I could imagine, oh, I have, you know, this model that is able like model one is able to detect like this harmful type of meme and model two is able to detect like this harmful type of video. And then like all of a sudden you're proliferating like hundreds and thousands of models for like little pieces of what you're trying to detect. And then another sort of scenario is like, oh, I'm going to like
try to standardize everything into more generalized models that, you know, handle multiple modes of data or like try to synthesize things together. How just practically as like a development and research team, have you started thinking about like, when is this something maybe we want to combine together in maybe a larger model that's trying to, you know, address like multi-task sort of thing or multiple types of data? And then the other side of that is like, maybe sometimes it is useful to just like spin up hundreds of small models and, you know, ensemble them together in some way. Any thoughts on that? Yeah, so we actually do do that. Sometimes we have models that are like just really lean and we serve them as is. And that's sort of for, like I said, for our like near real time.
responses um for when we do contextual stuff so like i said we really need to extract information as many different ways as we can so we're looking for logos and work when we're listening you know we're listening to the audio and and you know looking for it for known phrases and keywords and language understanding and what have you um and all these like smaller models then we do combine into ensembles we have um we have a feature store that we can basically you know take from um combine train train relevant models um and then and productionize them and then add add like we call them like indicators but you know essentially indicators from which then we can get features and then go to a model um which is which is an ensemble of these um and so we kind of use both approaches based on on like SLA requirements, based on also the explainability that we need. We want to be able to explain why something, right? Like, you know, this particular logo was found because sometimes the moderator may not have the full knowledge that we have. And so a big thing that we deal with is how can we take our intelligence and leverage it to the fullest extent, right? So one way is really to put it in the models. And the other way is really to educate the moderators through explainability of the models. They can really understand, you know, why things, you know, sometimes things aren't obvious. Yeah. And I guess that you started getting to my other question, which is like, how and when do you...
bring in the subject matter experts into the loops? Because I imagine there's like, you know, certain cases where you're highly probable that this is some type of harmful situation. And maybe given a restricted set of subject matter experts in an area, maybe they're restricted to only reviewing, you know, X amount of content per day or something. So is that a situation that you run into where, excuse me, where you have to prioritize, you know, what you're reviewing with subject matter experts based on some predictive measure that you have and kind of do that in some sort of ranked way? Or do you...
handle that in some other way. Do you mean in terms of like what the analysts are reviewing for us? Yeah. For labeling and or like for reviewing the. Right. So I'm assuming that there's a limited number of those people. There's not infinite of those people. So there's not an infinite number of those people. And also we want to be very aware of their well-being. We care a lot about the well-being of people that we work with. Like Act Defense invests a lot in that. And so specifically for example, I want to make sure that I prioritize what it is that they need to review.
I don't always need, you know, I don't need them to review everything, right? I usually, like, what I would go for is I want to review the gray zone, right? So we do, we have implemented active learning, which, you know, is basically to prioritize what it is that we want to train on. And so that also prioritizes what it is that we want to review into label. Because I'm always going for the gray zone, right? Like what, like the things that we're not quite sure of, that we don't really know, that's where it goes to the expert, right? It goes back to Chris's question, like, how do you know? Sometimes you do know, but it's tough. And those are the things that I want to label because those things that are tough are what is going to feed in and to give, like, you know, my distributor, you know, the maximum power that it needs. And how much,
Sorry to steal all the questions, Chris. I'm just so fascinated by all this. No worries. One thing that is always on my mind and maybe I wrestle with sometimes is how much do your sort of data science people or the people that are working sort of with the models directly interact with the subject matter experts and kind of... share knowledge across that boundary. How do you balance that? Because that's always something I think I struggle with in projects is ultimately, it would be great to kind of bring the subject matter experts in all along the way, like in every step of everything, because you learn so much. But the fact of the matter is like, you've got a limited number of those people, but also...
You have to ship things, right? So you can't necessarily have the luxury of always having a discussion before you make a development decision. So how do you balance that, especially because this is such a complicated environment in terms of the subject matter? How have you found ways to balance that and any thoughts or takeaways that you have from that experience? Yeah, so one thing that we did is we actually embedded subject matter experts like researchers into our dev teams to sort of be part of the process. We also have our analysts that are labelers.
you know, they work really closely. They're just, it's part of the same group. And so they're not out there. However, again, it's a limited number. It's, you know, there's, there's an, it's a limited number of violations. There's, you know, we're constantly exposed to new stuff that we have to handle. And there's just a matter of, you know, relationship building and of doing, you know, check bases and constant feedback. So, Hey, this is, you know, like they're kicking off new projects. We come, we learn, like, what are you guys doing? Because a lot of times they're learning on the fly too, right? They have this new, new trend, new thing that they're learning about. And so as they're learning, we're trying to gather as much as we can from them. And then just a constant like feedback, like, you know.
How does this look? How does this look? Is this there? Is this not? But I think the key was embedding them with us. We did have situations where basically we wanted to develop models for things that we didn't want to expose our data scientists to and that only a very, very few number of people in the company can be exposed to because of the nature of the violation. And there, that was much trickier because there, there's like complete dependence of the data scientists. Like, how do you, how do you build and train a model without looking at the data?
So I'm kind of curious because it changed the question I was about to ask you just a little bit with what you just said. So I'm going to kind of combine two things. The sense that I got, because you keep talking about going to the gray area and stuff, is that almost the core of your research effort is to kind of replace the human intuition that's necessary early on to identify the nuance that's there with more and better models as you're moving forward into that. And so I would almost like a bell curve of difficulty where that gray area is the hardest. But I am curious and I'm wondering if that's the case. But I'm also curious when you mention those things like it seems like.
I'm guessing that the things that you really don't want to expose someone to are almost they're not in the gray area. They're way over into the deeply evil side, you know, where it's like you just you're never going to forget having been exposed to that. If you are, how do those balance out? You know, you have that gray area that you're focusing on that you've mentioned several times. And then you have that those kind of things. Are they.
When you have something that's so explicitly evil and it will imprint a human's mind in a very negative way, are those different problems that you're solving as a data scientist a little bit? Whereas the gray area, there's so much nuance there. Do you see what I'm getting at? How do you balance the approach to building models to handle one thing that's really obviously bad and you don't want to get anyone to it versus developing intuition or an alternative to intuition in the gray area? Yeah, so I think, and let me know if this doesn't quite answer your question, but...
A lot of times it can be the same model, right? So you have a model and it knows how to identify it because, you know, it gets the distribution of, you know, the data is distributed in some way along the space, right? And so the things that are very obvious are going to be on like one side of the discriminator boundary, right? Like the really, like pick your favorite violation, right? And I'll give you examples of like things that are like very, very, very clearly violative and they're on that line. And then when you're training your model, you don't only want to give it just like the really horrible examples and then things that are just like, you know, puppies and snowflakes, right? They're very obviously not because those are so, the distribution between them, like they're so far away that your discriminator is, like your discriminator is just never going to.
convert it's going to flip flop back and forth and you know you'll never know and so as we're doing it we're also trying to find things that are like you know on the borderline because that's what's going to help us really make sure that we're able to like find the good decision boundary because at the end of the day it's really important like the basic like the basics is that we have to be able to catch like the isis and we have to be able to catch beheadings you know all these terrible terrible things but we also want to be able to catch things that are less obvious within that still within that space and so that's kind of where i'm talking about the gray area just like you know for the grooming model you can like i don't even want to say but you can think of phrases that are like very obviously grooming right where like you're sexually harassing a minor and it's there and like plain text right but that same model if you want it to be any good it's sure it's helpful that it can find the obvious stuff but you also want to train it on things that are kind of more on like closer to the boundary because that's what that's what'll help you in the long run i have a quick follow-up that as you were talking it was kind of coming into my mind it's a very human question i'm for a moment i want to move you out of the data science bit a little bit and just your organization is in a little bit of a of a unique position on that You mentioned that there are certain things you want to keep as many of your of your folks on your team not exposed to. But that does leave some people exposed to some pretty awful stuff. And that does impact people. We know I'm guessing that you're one of the people that has had to see some of those pretty tough things to see. How do you cope with that a little bit and keep you seem to have be really super grounded in that? And I know like, you know, but as part of the job, you're going to have to cope with some really tough stuff.
and i know there has been things that i have seen myself online that i wish i just had not seen i remember early on in the al-qaeda period some time back and i watched something that was in the news that happened to be out there and i was like i wish i'd never seen that and i will never forget that as long as i live i'm just curious in a human sense how do you cope with terrible things and and keep it in a healthy you know kind of for yourself in a healthy healthy place if that makes sense yeah that does make sense so i think i'm i'm like a very um personally like i'm a very like mission driven person like it's like it's very clear to me why it is that we do what we do. And I really, really believe in it. Like I said, a bunch of us are parents or have nieces and nephews or just care about kids or care about communities and environments. And I just very, very deeply believe in what we at ActiveFence do. And you can really feel that when you're in the office and when you're working with people, everyone is very mission driven. That being said, we also do our absolute best to support and protect everyone that works with us. So whether it's like different wellness support programs, we have a psychologist on staff who specializes in resilience.
you know, she's available to everyone and she has like group and one-on-one and really helps, you know, helps people build this sort of resilience in the face of what it is that we do. And for me, what personally works for me is like just understanding why what we do is so important. And yeah, I've definitely, definitely seen things that will never leave me, never. And I accept that, you know, because I'm doing my part to make everything just- You're helping the world in that way. I get that. Hopefully. I try. We all try. I think even though we've talked about hard things this episode, I'm super encouraged and want to thank you and the team at ActiveFence for what you're doing.
Yeah, it's something that's desperately needed. And thank you so much for digging into these problems and doing it with such technical excellence as well and deep insight. As we close out here and we look to the future, what on the positive side sort of excites you about where this technology is headed as you look to the future? Yeah, so to me, first of all, super exciting that this is no, like I said, like we start off, it's no longer nice to have. It's a basic expectation. So I think, first of all, to me, that's really exciting because people are not...
taking safety for granted. Like they understand how critical it is and it's coming from the users. It's kind of, you know, it's not just sort of like, oh, well, it is what it is. Like this is the price I pay for being online. No, that shouldn't be the price that you pay. So to me, that's exciting. And on the tech side, I think what's cool is that we're seeing a lot of open sourcing of different models, whether it's data generation or, you know, audio transcription or Z shots or like all these things that are just, it's like a candy store, right? Like you can start thinking about all these things that are used for.
You know, technology is used for completely different things. And you can say, well, like, how can I take these ideas and use them to just extract more signal and to look at these things from different angles? And, you know, it's an adversarial space. And so it keeps it interesting, at least. Well, Mithar, it's very inspirational, the work you're doing. I know that it's tough work, but thank you very much and to your teammates for doing the kind of work that you're doing. It was great having you on the show. Looking forward to having you back sometime as you guys surge forward and have some more stuff that you want to share with us. So thank you very much for your time today. Thank you so much for having me. Thank you for caring and for asking really, really interesting questions. I appreciate it.
All right, that is our show for this week. If you dig it, don't forget to subscribe. Head to practicalai.fm for all the ways. And if Practical AI has benefited your life, pay it forward by sharing the show with a friend or colleague. Word of mouth is the number one way people find shows like ours. Thanks again to Fastly for fronting our static assets, to Fly.io for backing our dynamic requests, to Breakmaster Cylinder for the beats, and to you for listening. We appreciate you. That's all for now. We'll talk to you again on the next one.