Welcome to Cybersecurity Insights, the podcast for the CyberEd.io Learning Community. Our goal is to bring cybersecurity practitioners the latest and most relevant education and training to upskill and dive deeper into topics that matter in today's modern cybersecurity world. Good day, everyone. This is Steve King.
I'm the Managing Director for CyberEd.io. With me today is San Bershon, who's the CEO of Skyhawk Security and manages the company's day-to-day operations. He's led product and strategy teams for companies in the security space for over 10 years. He was the VP of Strategy, General Manager, and Site Manager at Dome9, which was acquired by Checkpoint.
He helped develop the company into a leader in the cloud security cluster management space and then was brought to lead Skyhawk Security, which was a spinoff of Redware and backed by Tiger Global Management. He's also been a product leader and manager in technology companies for 20 years, roles going back to IBM and Cisco. Joining us is Amir Shachar, head of Skyhawk's data science group. He has extensive experience in mathematics, computer science, and statistics, a man of my heart.
And I say that only because that was my degree, but that was 114 years ago. He is the author of the mathematical theory of semi-discrete calculus and has co-invented numerous patents in various domains. So I am delighted and honored that he is joining us today as well. So thank you guys for coming.
And we're going to talk about generative AI and your product and the world according to ChatGPT here. So thanks for having us, Steve, today. Very grateful and excited to be here. Yeah, well, thank you.
So Skyhawk recently integrated generative AI into its threat detection process, which the objective was to, you know, significantly increase the speed of detection based on anomalous activity and, of course, lowering operational costs at the same time. The data says that in the vast majority, 78% of the cases that your platform produced alerts earlier after adding that capability to your threat scoring process. Can you guys explain kind of how that works, what you did to get that done, how long that process took, and what that does to your product in terms of its ability to compete in the markets? Sure.
And maybe just a second before we deep dive into this latest addition, let's try to give a bit of a context on what Skyhawk is doing and why it's so important. So you mentioned my background coming from the kind of creating with others the CSPM market. And where most of the cloud security market operates today is on risk management, right? It's finding the vulnerabilities, finding the misconfigurations, the correlation between them, but it's all static misconfiguration data where people are trying to reduce their risk and attack surface, basically improve their posture.
What Skyhawk is doing is actually looking beyond that point into the behavior in runtime. Going into what we call CDR, cloud threat detection and response, and trying to basically detect anomalous behavior, malicious behavior, which is beyond risk. It's beyond the configuration. It's what actually is going on in the account.
And we do it by three layers of machine learning. Basically, we take a lot of data, whether it's log data, telemetry, events from different systems, and we identify anomalies based on them, which are malicious behavior indicators. The second layer that we have is basically correlating all these malicious behavior indicators into an attack storyline. So most of the anomaly detection products out there have a lot of false positives, right?
Every anomaly is basically an alert, but not every anomaly is an incident. The first approach we had before generative AI to reduce the alert fatigue in runtime and make sure that we alert only on real alerts on real events was to try and correlate all these malicious behaviors and make sure that we find the pattern based on the meter framework, attack kill chains, but basically making sure that we have all these indicators co-related into an attack sequence where we can give you an alert that has all the evidence and all the steps that makes us think that the collection of events, of indicators are a real attack. That's before generative AI. Okay so far?
Yeah. Okay. Now we basically integrated a couple of months ago ChatGPT. And the way we integrated ChatGPT was basically in order to address a big problem in the market.
When we alert the customer on event, this event needs to go to an incident responder. There aren't too many cloud expertise out there. Getting incident, experienced incident responders is very, very tough. And for us to make sure that we don't create an alert fatigue and send too many alerts to our customers, we had to really make sure that we send out alerts that we are 100% sure are high probability for an attack.
And that creates a certain capability of what we call the confusion metrics. By integrating ChatGPT, what we could do is we could basically lower the bar and send more alerts to ChatGPT and ask ChatGPT to act like a virtual incident responder. Take the event in abstract terms, the whole chain of events, and analyze them and tell us if as a virtual incident responder ChatGPT sees it as a malicious event, as an incident. And we do it multiple times.
Let's say 1,000 times. And we get basically these 1,000 results back. And based on whether the results have high scores in a small standard deviation or high standard deviation, we basically create a confidence level in the alert. And that allows us to basically improve the confusion metrics, send more alerts for virtual incident responders, and being able to provide eventually a more accurate alert to our customers.
Yeah. When you say you create 1,000 versions or instances, does that mean that each alert that you sent 1,000 different times? Yes. Okay.
So ChatGPT then kind of regenerates each one of those and looks at it from a different point of view. Correct. And that allows us to basically, again, use the fact that ChatGPT is trained on millions of data points and get these different 1,000 opinions that are the composition of the data that was labeled and basically see if there's a consensus on this event being high risk. If there's a consensus, we can say, okay, not only we think it's a high risk, there's also consensus.
And by the way, the other functionality is explainability, is here is why ChatGPT thinks it's malicious and we provide this explainability to the human incident responder. So we're basically addressing a lot of the shortage in manpower by harnessing ChatGPT in the way we used it in order to provide better confidence and better explainability. Yeah. I used to run SOC centers for a living, so I'm particularly excited about that.
Does this mean that the incidents are being regenerated until you get a certain level of confidence before you send it as an alert? How does that work? So it's a combination. Basically, if the version before ChatGPT basically was issuing an alert only when the correlation passed a certain threshold, now it's not only the risk or a threshold, it's also the confidence level from the incident responders.
And part of the reason we were able to advance the detection was because we basically can send lower threshold alerts to ChatGPT. And sometimes the confidence level that we get from ChatGPT before the alert passes the system threshold is so high that it tells us to advance the incident into alert much earlier. That's why in 78% of the cases, we found out that we were able to alert earlier, basically fewer steps into the APT. And so, you know, it's been said that when ChatGPT or different versions of it can't figure something out, it makes stuff up.
Does it ever do that with you? I mean, does it ever say, well, I would never say I've never seen this before, but do you have instances where it actually makes something up to replace the fact that it hasn't seen that before? So first of all, that's why we do it 1,000 times. That's helping us a lot.
And second, we know how to make sure that we augment it and pass on things that even to ChatGPT, we don't pass on everything. We pass on things that pass at a lower threshold, but still a threshold on our side. I see that Amir wanted to add something here. So I'll let the real professional here to say something about it.
Yeah, I mean, just to mention that we do see an improvement across the ChatGPT versions. We did see that in GPT-3, the results were sometimes awkward, but it gets better. And now it makes sense. It makes much more sense.
Yeah. What process did you use to refine all of this so that you got to the point where you were 100% confident that the high-confidence incidents you were identifying were, in fact, carrying that level of confidence? So we have our own human security researchers, and they looked into the results and verified them. We can now say with very high confidence that the ChatGPT results, the ones with the high confidence, are very similar to what a human researcher would say about them.
Yeah. And then the intelligence that you're providing the analyst, is it more extensive than they otherwise might have from a purely mechanical system? It's more explainable. You need to remember that for privacy reasons, we don't send ChatGPT raw customer data.
So part of the things that Amir and Tim did was basically to abstract the events into the general steps or the techniques, methods, and techniques that were taken, and that is the sequence that is sent to ChatGPT. So in terms of raw data, we are still the ones. Before ChatGPT, we still have our own extensive IP generating both the malicious behavior indicators at the low level and the sequence. Did I answer your question?
Yes, yeah. What do you think the impact is going to be, let's say that you set the standard for adoption here for this approach to applied technology, or that Skyhawk suddenly becomes the standard and you guys own 500 customers tomorrow And that's how I view the kind of the crowdsourcing here. We do not retrain again for privacy reasons. We might go to our own private copy and then retrain based on that.
And we have many other ideas, by the way. Amir has a full list of ideas of other things we can do with agents, private copies that we may host. But for the hosted version, we are very, very careful on how we use it to make sure we do not violate any privacy obligations. Yeah.
And there are plenty of those. It seems to me, based upon, you know, the whole process of training through large language models, it would seem to me that the more specific data you fed into that model, the better and faster you'd be able to detect. And my question around that, I guess, is, do you pay any attention to the source of the threat vector? In other words, the actor versus just the vector?
Because it would seem to me that you'd know, based on just technique alone, perhaps, or the origin of a particular threat vector that it probably comes from a certain threat actor. Do you pay any attention to the actor in the part of the calculus? We do. And even before ChatGPT, our detection system had things like C3, right?
What's the user agent? Where is it coming from? Geolocations, right? That type of things, whether it's reputation aspects.
So we can be, and even when we send to ChatGPT, we can be very specific, still in abstract terms, right? You know, we can talk about SCP being the user agent or an abnormal user agent. These are things that are still very abstract in general and are still, on the other hand, for ChatGPT, extremely detailed in order to get to a conclusion. So it's this balance.
And that was exactly Amir's team's job, to find this balance on how to send data that doesn't violate privacy and still extremely detailed for ChatGPT to be able to provide us a good analysis. That was part of the trial and error and work that was done. And the results were sent to our research team, right? So that our researchers could actually validate that what we're getting is valuable.
And that's how we basically do it. The whole trick is in how we abstract it and still we're specific enough to get accurate results. Right. To address the first part of your question, Steve, we did see that fine-tuning the large language model over a large corpus of security data did improve the model performance.
Yeah, I imagine. You guys aren't alone. I'm sure there are tons of other companies doing something similar with that technology. What do you see for the future of cloud security and more broadly, I guess, the whole cybersecurity landscape as a result of this technology that just jumped out at us a couple of months ago.
So first of all, you know, I think that when I'm looking on generative AI currently, ChatGPT and other technologies, I think that it's like, it's probably going to create another cycle of innovation, probably as big as we saw with, you know, cloud, right? The guys among us that were many years in the industry, remember the innovation cycles, right? I think the generative AI generally is a huge opportunity. It will be used by a lot of security companies in creative ways.
Unfortunately, it will be used by the bad actors as well in creative ways, which will create another opportunity, right? I'm not sure that the investment in what's called trust and safety, the ChatGPT safety measures is going to stop bad actors completely. And therefore, the opportunity here to create products that are aimed to deal with the risks that ChatGPT is going to create is also going to be immense. I just want to add to what Amir mentioned.
I mentioned that the attacks, the future attacks will be much more sophisticated because first of all, they will have to overcome generative AI in the defensive part. And also they will use a very evolving generative AI in the offense. Yeah, sure. And that's kind of what I was going to get to here.
I mean, it's kind of, it's like a comic book version of the generative AI with the black hat fighting the generative AI with the white hat who wins and why. How's it going to sort itself out in your minds? I think there will be many, many more kind of sophisticated, you know, sophisticated attacks. And we will see many more creative people like Amir trying to protect against them, right?
And it's probably going to create a complete new vector and maybe even category. I mean, for us as a CD operator, it was very natural to, and where we were with our product, it was very natural to integrate ChatGPT into our detection flow. What we saw so far in the industry, it wasn't so much integrated into detection. I mean, I think that I can probably say Skylock is standing in the same line as Microsoft with its security co-pilot, right?
There aren't too many companies out there that were able to adopt ChatGPT so deeply into security detection. I think that this trend will probably continue at a much higher pace. And we will see many more attacks coming from ChatGPT as well. Yeah, let me, I think I should have rephrased that question because I agree with what you just said.
And that's the future. But in between, in between now and then, yeah, is your vision that we're going to get so good at detection that in such a hurry that we'll actually be able to reduce the frequency of breaches dramatically before the bad guys figure out how to circumvent or get around the detection defenses that we've created? It's definitely going to improve on many factors. And I think that this is the opportunity for companies like us at the front of the innovation in CDR, those that have, you know, track record, been there for a few years to be able to improve that.
But we will have to, you know, stand on our toes and keep innovating because the innovation on the bad actor side will not stop either. No. So just to mention two very promising technologies that are like, you know, the evolution of ChatGPT that will probably be used both for protection and malicious attempts. So the first one is AutoGPT, which is AI agents that can act synchronously and correlate between each other and, you know, plan ahead and stuff like that, which can be used both for defense and offense.
And the other promising technology is, you know, one of the main problems with ChatGPT and large language models in general is their context, which is limited. For the most part, it's limited by, let's say, a couple of thousands of tokens. And we do see first signs of models that are almost not limited at all with the number of tokens that they use. This brings many opportunities, both for defenders and offenders, because you may apply these kind of models to whatever, however length the logs are.
Yeah. And then from an employment, a future employment point of view, let me ask you, Amir, would you, as you scale the business, are you going to be hiring more data scientists or are you going to be hiring more security analysts, assuming that those are two different people? That's a very good question. I think we will.
How about we see both positions as the heart of our business. I mean, in many cases, when people ask me what we are as a company, are we a security company? I look on us as a data company in the security space. Right.
And that's why I think both are so much in the heart of what we're doing. There is software engineering work for us, but most of what we do is centered around data science and security research. If you were going to pursue a post undergraduate program in data science and you had your eye on Skyhawk, what kinds of competencies would you want to develop? You would probably want to develop a very good understanding of large language models, their limitations, what they're capable of, just get a good intuition of cyber threats in general.
But for the most part, I would say just come up with a very creative and open minded approach to account for the ever evolving and the breakneck speed evolving industry and threats and technology. It's basically about adaptation because we learn each week new technologies and new approaches that need to be applied. Yeah. Adaptability.
That's what we look for. Right. Kind of goes to the, in my view, of the typical profile of a successful cybersecurity technology professional that, you know, the ability to adapt quickly and to, and to be curious enough to continue to investigate and discover those are kind of two of the traits that feel to me like they have to be there. Otherwise you probably wouldn't get there.
A lot of creativity. I think that a lot of creativity. I think that we are just the kind of seeing the tip of the iceberg in terms of what will be possible. When you say creativity, do you that sometimes people think, you know, go immediately to art instead of to science.
When I'm saying creativity, I'm thinking of how the technology can be best used. Trying to find how to utilize that technology, even in creative ways in terms of how bad guys will try to use it, right. How the bad actors will try to use it in order to defend against it. It's ever looking exploration, asking questions all the time, trying to really be creative in both utilizing the tools as attack tools, as well as as both defense.
In order to be able to create great defense, we have to be able to think like attackers. So it's really this open mind creativity, trying to stretch the boundaries because we haven't yet, I think. Yeah, that's cool. I'm excited about your company and it's great to see what I consider to be the sort of applied science in day to day action here affecting outcomes.
So that's incredible. I'm conscious of our time here and we're kind of over our 30 minutes. So I wanted to thank you both. Could you It would be a big category, just like the CINA category up to this point, almost independent out of CINA, in my view.
Yeah, yeah, that makes sense. And that's great. All good news. And thank you for sharing that.
I'm sure our audience appreciates it. And I hope that those folks had as good a time as I did learning about how you guys apply ChatGPT and what the future looks like here. It's been a real pleasure getting to chat with you guys. And I appreciate you taking the time out of your day to share with us a little bit.
So thanks for being here today. Thank you for having us. I really appreciate it. It's been a pleasure.
Great. Thank you again to our audience for spending your time with us. And hopefully you found it valuable. And until next time, I'm Steve King, your host, signing off.
Thank you for joining us for another episode of Cybersecurity Insights. You can connect with us on LinkedIn or Facebook, or send us an email at social at cybered.io. For more information about the podcast, visit cybered.io forward slash podcast. Until next week, stay safe and secure, and we'll see you on the next episode of Cybersecurity Insights.