PACKET HACKING VILLAGE - How to Tune Automation to Avoid False Positives
Ziabari, Gita
Every SOC is deluged by massive amounts of logs, suspect files, alerts and data that make it impossible to respond to everything. It is essential to deploy automation to accelerate response time, consistency, scalability and efficiency. This talk will cover techniques to design a reliable automated tool in security. We will discuss about techniques of tunning the automation to avoid false positives and the many struggles we have had in creating appropriate whitelists. We will walk through steps of creating an automated tool and the essential factors to be considered to avoid any false positive.
 All right, good morning. We're out to our second talk and next speaker is, you've been a few times at the packet hacking village talks. Gita Ziyabari. Hi, everyone. Thanks for coming to my presentation. My name is Gita Ziyabari. I'm coming from Verizon, working as a senior consultant engineer software developer and I'm going to talk about how to tune automation to avoid false positive. So I'm going to talk about techniques to design a reliable automated tool, introducing threat intelligence feeds and how to avoid false positive when we generate automated tools and feeds. Techniques to design a reliable automated tool. So there are many different reasons that automation is needed in cybersecurity. There are massive amount of data that needs to be analyzed on an early basis and action is needed. So accelerated response time, consistency, scalability, efficiency, risk reduction, simplified IR process, empowering users, they are all good reasons that we do need to have automation in cybersecurity. However, since we have large amount of data and all of us are very much busy, bad ideas could lead to false positive and false positive is always bad especially in cybersecurity. So automation needs to be done with intelligence. By intelligence, I mean let's just consider creating an automated tool that is pretty much simple to use because no one is going to be interested to learn a new tool that is very much complicated and it's designed to have a lot of let's say GUI pages with lots of maintenance and everything that it just like takes a long time. And think about five years, you are creating an automated tool and you move on. If you want to maintain it all the time and something breaks all the time, then it's not going to be usable. Make it user friendly so that the threat researchers who are basically the target to use the automated tool, they buy it and they start using it right away. Make it like have some documented design framework and think about whoever is going to inherit the framework. It might be an open source you may decide to publish it. So think about those that are going to use and inherit the code. Most of researchers in cybersecurity and analysts, they already have a framework that they are based on it and they are actually using it. So if you want to say that, yeah, I have this automated tool, it's pretty cool and it's just like adding these many features to your analysis and it's so fast and everything, they are going to hesitate using it because they already have something that they are using and switching from one source that is actually working to a new one is going to be a bit hard. So make it... I'm sorry. I apologize. So make it somehow that you guys can hear me? That's good. That's much better. So make it somehow between more dependencies. First of all, to other servers. Again, if you want to publish it in GitHub, people who are going to download your tool, they are not going to have the same servers that you have. So make it easily integrated to any platform that is being used by different analysts, by different groups, by different companies. It just doesn't matter. Just make it as much as possible independent with possibility of integration. Then it comes to planning the automation. Again, simplicity is the most important thing because imagine that you could create an automated tool, a script that could be run through Chrome job, in background, through command line, or you have a pretty nice UI. View UI, believe me, it just requires a lot of maintenance and there are so many patches that you have to just like upgrade. There are so many dependencies and, on the list, they are not going to like it, like trade researchers. They have to just like go to this website, they have to access it, and then they have to log into it. There are so many layers involved. So just try to make it simple. Chrome job. If you can do it through Chrome job, then do it through Chrome job. Some of us like Python, some of us like pairs, some of us like PHP. If you have a framework that is already designed, let's say doing Python, then stick to Python. And do not just switch because you just get bored from one language and you feel that the other language actually has more functionality for doing this tool that you're, for making this tool that you're making. Make it simple. Avoid dependent processes because we all know that when we have dependent processes, if just one of them breaks, then everything is going to be broken. If you're busy working on other stuff, they are going to ask you to fix it and you don't know where to start. What is actually broken to fix it? And if you have to really have chain of processes, then you have some monitoring systems like Nagios, so that it just like a start reporting you if something goes down. And if it goes down, make some really, really like easy script, like a line of command. If it goes down, like if your MongoDB is down, send me the logs, restart it so that your automated tool is not going to be broken forever till you or someone actually finds out what's going on. In cybersecurity, we are actually lucky developers because we do not have QA in most of cases to QAR code. So that's a good part and a bad part. But we have to be our own QA especially because we are actually having a massive amount of data that needs to be analyzed and performance testing is very important. So make sure to have the platform for yourself for being able to test framework actually to be able to test your own code. That is very, very important. I know that we all do unit testing and feature testing. Performance testing is needed and we do need to consider it. I just quickly jumped to threat intelligence feeds. That's the area that I was working in the past few years. And by site, by threat intelligence, I mean indicator-based threat intelligence feeds such as domains, URLs, IP addresses, hashes, email addresses and whatever you want to consider as an indicator. So threat intelligence feeds, you can get it from third party feeds. Like there are some open sources, there are some community-based based on the circle of trust, commercial ones you have to pay, good money. And the government-based internal feeds are also valuable. You get feedbacks from analysts that these are basically the feeds that I could pass you and indicators that I could pass you. Or you create some automated data mining tools or you use the ones that are available and start mining data from the data and from the logs that you have. You just generated how many of you guys are using threat intelligence feeds? How many of you are happy with the result? None. I was expecting it actually. So the reason is that it's just like poor quality control, overlapping indicators, false positives, noise. It's just like massively creating noise. And most of the companies, they would rather not to use it rather than using it because it's just like generating noise and you don't know where is the source coming or if you can't trust it or not. Now, generating high-quality feeds is possible, believe me. The first step that you need to take action is actually having a database. Don't take everything from fly, from different sources, make it to like a JSON or CSV format and then export it to whoever you want to. Have everything in your database and in that case, you would be able to apply quality and you have control on the indicators that you're ingesting in the database. What needs to be done is do duplicating, white listing, filtering, scoring, and aging. If you are inserting an indicator, check your database and see if it exists in the database. If it does, then update it based on the source, like aging, scoring, just like make some changes, but do not reinsert it. When you want to export the feed, it's very important to also do duplicating so that you do not have to duplicate indicators. White listing, it could be done through third-party sources, it could be done through internally. Third-party sources, there are some sources that are available, like external sources, some of them are free, some of them you have to pay some money. Most of them are based on the popularity of the indicators. For example, the websites that are being most visited by different countries, they are being introduced as white lists. Try to scoop up just like the top 1000 and even if you are considering the top 1000, let's say, domains, still apply some filters. Filtering is very important, actually, in feed, because imagine that the adult content-based domains are also pretty much popular and some of them are even in the top 50 indicators. They are not necessarily good or trustable, they are just popular. So, as quantity increases, consider that the possibility of having false positive also increases. You get feedback from your peers, your analysts, your customers about white indicators that actually are false positive immediately, quarantine, then have another media tool to check the results through different sources and then white list then immediately. Everything has to be done in automatic way because you cannot just go through every single indicator one by one. And believe me, one bad indicator could result in many hits like 100k plus. So, when I said that we don't have QA and we have to be our own QA, have your framework, the test framework, and start testing your feeds that you are generating, write a script, see the number of matches and have a statistic about the indicators that are matching a lot of hits. And then quarantine them, analyze them through another source and see if they are actually false positive or they are true positive, immediately white list them. So, you get a data, a big amount of data in indicators, you apply white listing from any source, internal, external, you have a better data, it's not enough though. You do need to do some scoring. So, imagine that you are getting, you are using this many sources, like open sources, different sources, sources coming from your internal or external. And the indicators that are being reported repeatedly through different sources, it means that they are actually happening and they have a better chance to be malicious. So, their score is actually higher than the ones that are coming from a single source. So, they should have a higher score. A score of sources is also important. Check out the results that you are getting, the alerts that is being generated and check out the number of false positive that you are getting from different sources. Have a statistic. Again, here, a test framework is needed if you want to analyze the sources and the score of the sources that are coming. The ones that are noisier and they are generating more false positive, they need to be less scored and the ones that are more reliable, they need to be higher scored. Now, the result that you are getting today may be different one week from now. So, again, you need to have an automated way for getting these scored of especially the sources and also the common number of sources that are reporting an indicator and tune the algorithm somehow that gives you the right results. This is just like an algorithm. You get the indicators, see if it exists in database. If it doesn't exist, then score it based on the source that you have obtained. If it does exist, then check the common sources, the quality, the quantity of the common sources and also the quality of the sources. In this phase, right here, you can apply many different scoring methods. Let's say if you have access to the malware type, then you can start scoring the malware type as well. So, you can just apply intelligence based on your needs if you have the malware type, malware type could be added here. So, filtering, we just talked about like white listing, scoring. There are so many different methods that you could apply in this phase for applying more filtering to have a better filtered data. There are some tools available that you could just like check the indicator through trusted parties that they have access to many AVN genes and you can get a report from them and see what is the number of positives for this particular indicator. Then you want to build up the queries, be as specific as possible. More attributes, it means that you can have a better database of indicators and you can easily search on whatever you are interested. So, make sure that the critical attributes that you are having are indicators, the indicator itself, the type of it, if it is a hash, URL, domain, IP address, email, anything. Unique index based on the sources so that if an alert gets generated, you would know that, okay, this is coming from this source or this source. So, you have a control for the nubbing thing for the algorithm that I just mentioned to avoid false positive. A list of sources, the score that is given currently, data being insertion, malware type and age. So, when it comes to feeds, you cannot say that, okay, I have my feed that I built like a week ago and now it's just like this is what I have and I'm going to submit it. It needs to be updated. Ingestions should happen in real time and you need to apply the intelligence, the algorithms that we just covered and export the data at least on an early basis to have a real time, almost real time, I would say feeds. You could be selective. Now you have the pool of indicators based on different criteria that you defined and attributes. Now you can select what you are interested and actually interest you. Like if you want to select the type of indicator to be only hash, you can apply it. If you want based on the malware type, then you can just like extract it. If you want combination of the stuff that is happening, let's say the most recent indicators that have been seen through different networks, you can just like extract the information in the database. Aging is also another thing that is very much important and needs to be considered. You don't want to save lots of indicators in your database and then get an indicator that is not happening in any database anymore. So, as you see the indicators and as you are ingesting the indicators in your database, you need to age them properly and set it based on the malware type, based on the scoring, set it somehow from like 60 days or like a little bit more and then age them out if they are not being seen. So you don't have like something that is not happening in the network anymore in your feeds. So as I mentioned, like you get your feeds and indicators from different sources, from third party sources, from internal sources, you do apply some intelligence, white listing, scoring, filtering, aging and then you can select your feeds based on the criteria that you are interested. Now imagine that you want to have high-periority feeds that are actually very critical and needs to be taken care of immediately. You can just like have your alerts based on these feeds like very high, you can define medium-periority feeds or like low confidence. And with all these criteria that I just told you, you are going to have a better result, less false positive, selecting true positive and you do need to have aging for having a current indicators and the sort of like a real-time results. Do you have any questions?
 All right, good morning. We're out to our second talk and next speaker is, you've been  a few times at the packet hacking village talks. Gita Ziyabari.  Hi, everyone. Thanks for coming to my presentation. My name is Gita Ziyabari. I'm coming from Verizon,  working as a senior consultant engineer software developer and I'm going to talk about how to  tune automation to avoid false positive. So I'm going to talk about techniques to design a  reliable automated tool, introducing threat intelligence feeds and how to avoid false  positive when we generate automated tools and feeds. Techniques to design a reliable automated  tool. So there are many different reasons that automation is needed in cybersecurity. There  are massive amount of data that needs to be analyzed on an early basis and action is needed. So  accelerated response time, consistency, scalability, efficiency, risk reduction, simplified IR  process, empowering users, they are all good reasons that we do need to have automation in  cybersecurity. However, since we have large amount of data and all of us are very much busy, bad  ideas could lead to false positive and false positive is always bad especially in cybersecurity. So  automation needs to be done with intelligence. By intelligence, I mean let's just consider  creating an automated tool that is pretty much simple to use because no one is going to be  interested to learn a new tool that is very much complicated and it's designed to have a lot of  let's say GUI pages with lots of maintenance and everything that it just like takes a long time. And  think about five years, you are creating an automated tool and you move on. If you want to  maintain it all the time and something breaks all the time, then it's not going to be usable. Make  it user friendly so that the threat researchers who are basically the target to use the automated  tool, they buy it and they start using it right away. Make it like have some documented design  framework and think about whoever is going to inherit the framework. It might be an open  source you may decide to publish it. So think about those that are going to use and inherit the  code. Most of researchers in cybersecurity and analysts, they already have a framework that  they are based on it and they are actually using it. So if you want to say that, yeah, I have  this automated tool, it's pretty cool and it's just like adding these many features to your  analysis and it's so fast and everything, they are going to hesitate using it because they  already have something that they are using and switching from one source that is actually  working to a new one is going to be a bit hard. So make it...  I'm sorry. I apologize. So make it somehow that you guys can hear me? That's good. That's  much better. So make it somehow between more dependencies. First of all, to other servers.  Again, if you want to publish it in GitHub, people who are going to download your tool,  they are not going to have the same servers that you have. So make it easily integrated  to any platform that is being used by different analysts, by different groups, by different  companies. It just doesn't matter. Just make it as much as possible independent with possibility  of integration. Then it comes to planning the automation. Again, simplicity is the most  important thing because imagine that you could create an automated tool, a script that could  be run through Chrome job, in background, through command line, or you have a pretty  nice UI. View UI, believe me, it just requires a lot of maintenance and there are so many  patches that you have to just like upgrade. There are so many dependencies and, on the  list, they are not going to like it, like trade researchers. They have to just like go to  this website, they have to access it, and then they have to log into it. There are so many  layers involved. So just try to make it simple. Chrome job. If you can do it through Chrome  job, then do it through Chrome job. Some of us like Python, some of us like pairs, some  of us like PHP. If you have a framework that is already designed, let's say doing Python,  then stick to Python. And do not just switch because you just get bored from one language  and you feel that the other language actually has more functionality for doing this tool  that you're, for making this tool that you're making. Make it simple. Avoid dependent processes  because we all know that when we have dependent processes, if just one of them breaks, then  everything is going to be broken. If you're busy working on other stuff, they are going  to ask you to fix it and you don't know where to start. What is actually broken to fix it?  And if you have to really have chain of processes, then you have some monitoring systems like  Nagios, so that it just like a start reporting you if something goes down. And if it goes  down, make some really, really like easy script, like a line of command. If it goes down, like  if your MongoDB is down, send me the logs, restart it so that your automated tool is  not going to be broken forever till you or someone actually finds out what's going on.  In cybersecurity, we are actually lucky developers because we do not have QA in most of cases  to QAR code. So that's a good part and a bad part. But we have to be our own QA especially  because we are actually having a massive amount of data that needs to be analyzed and performance  testing is very important. So make sure to have the platform for yourself for being able  to test framework actually to be able to test your own code. That is very, very important.  I know that we all do unit testing and feature testing. Performance testing is needed and  we do need to consider it. I just quickly jumped to threat intelligence  feeds. That's the area that I was working in the past few years. And by site, by threat  intelligence, I mean indicator-based threat intelligence feeds such as domains, URLs, IP  addresses, hashes, email addresses and whatever you want to consider as an indicator. So threat  intelligence feeds, you can get it from third party feeds. Like there are some open sources,  there are some community-based based on the circle of trust, commercial ones you have to  pay, good money. And the government-based internal feeds are also valuable. You get feedbacks  from analysts that these are basically the feeds that I could pass you and indicators  that I could pass you. Or you create some automated data mining tools or you use the  ones that are available and start mining data from the data and from the logs that you  have. You just generated how many of you guys are using threat intelligence feeds? How many  of you are happy with the result? None. I was expecting it actually. So the reason is that  it's just like poor quality control, overlapping indicators, false positives, noise. It's  just like massively creating noise. And most of the companies, they would rather not to  use it rather than using it because it's just like generating noise and you don't know where  is the source coming or if you can't trust it or not. Now, generating high-quality feeds  is possible, believe me. The first step that you need to take action is actually having  a database. Don't take everything from fly, from different sources, make it to like a  JSON or CSV format and then export it to whoever you want to. Have everything in your database  and in that case, you would be able to apply quality and you have control on the indicators  that you're ingesting in the database. What needs to be done is do duplicating, white  listing, filtering, scoring, and aging. If you are inserting an indicator, check your  database and see if it exists in the database. If it does, then update it based on the source,  like aging, scoring, just like make some changes, but do not reinsert it. When you want to export  the feed, it's very important to also do duplicating so that you do not have to duplicate indicators.  White listing, it could be done through third-party sources, it could be done through internally.  Third-party sources, there are some sources that are available, like external sources,  some of them are free, some of them you have to pay some money. Most of them are based on  the popularity of the indicators. For example, the websites that are being most visited by  different countries, they are being introduced as white lists. Try to scoop up just like  the top 1000 and even if you are considering the top 1000, let's say, domains, still apply  some filters. Filtering is very important, actually, in feed, because imagine that the  adult content-based domains are also pretty much popular and some of them are even in  the top 50 indicators. They are not necessarily good or trustable, they are just popular.  So, as quantity increases, consider that the possibility of having false positive also  increases. You get feedback from your peers, your analysts, your customers about white  indicators that actually are false positive immediately, quarantine, then have another  media tool to check the results through different sources and then white list then immediately.  Everything has to be done in automatic way because you cannot just go through every single  indicator one by one. And believe me, one bad indicator could result in many hits like  100k plus. So, when I said that we don't have QA and we have to be our own QA, have  your framework, the test framework, and start testing your feeds that you are generating,  write a script, see the number of matches and have a statistic about the indicators  that are matching a lot of hits. And then quarantine them, analyze them through another  source and see if they are actually false positive or they are true positive, immediately  white list them. So, you get a data, a big amount of data in indicators, you apply white  listing from any source, internal, external, you have a better data, it's not enough though.  You do need to do some scoring. So, imagine that you are getting, you are using this many  sources, like open sources, different sources, sources coming from your internal or external.  And the indicators that are being reported repeatedly through different sources, it means  that they are actually happening and they have a better chance to be malicious. So, their  score is actually higher than the ones that are coming from a single source. So, they  should have a higher score. A score of sources is also important. Check out the results that  you are getting, the alerts that is being generated and check out the number of false  positive that you are getting from different sources. Have a statistic. Again, here, a  test framework is needed if you want to analyze the sources and the score of the sources that  are coming. The ones that are noisier and they are generating more false positive, they  need to be less scored and the ones that are more reliable, they need to be higher scored.  Now, the result that you are getting today may be different one week from now. So, again,  you need to have an automated way for getting these scored of especially the sources and  also the common number of sources that are reporting an indicator and tune the algorithm  somehow that gives you the right results. This is just like an algorithm. You get the  indicators, see if it exists in database. If it doesn't exist, then score it based on  the source that you have obtained. If it does exist, then check the common sources, the  quality, the quantity of the common sources and also the quality of the sources. In this  phase, right here, you can apply many different scoring methods. Let's say if you have access  to the malware type, then you can start scoring the malware type as well. So, you can just  apply intelligence based on your needs if you have the malware type, malware type could  be added here. So, filtering, we just talked about like white listing, scoring. There are  so many different methods that you could apply in this phase for applying more filtering  to have a better filtered data. There are some tools available that you could just like  check the indicator through trusted parties that they have access to many AVN genes and  you can get a report from them and see what is the number of positives for this particular  indicator. Then you want to build up the queries, be as specific as possible. More attributes,  it means that you can have a better database of indicators and you can easily search on  whatever you are interested. So, make sure that the critical attributes that you are  having are indicators, the indicator itself, the type of it, if it is a hash, URL, domain,  IP address, email, anything. Unique index based on the sources so that if an alert  gets generated, you would know that, okay, this is coming from this source or this source.  So, you have a control for the nubbing thing for the algorithm that I just mentioned to  avoid false positive. A list of sources, the score that is given currently, data being  insertion, malware type and age. So, when it comes to feeds, you cannot say that, okay,  I have my feed that I built like a week ago and now it's just like this is what I have  and I'm going to submit it. It needs to be updated. Ingestions should happen in real  time and you need to apply the intelligence, the algorithms that we just covered and export  the data at least on an early basis to have a real time, almost real time, I would say  feeds. You could be selective. Now you have the pool of indicators based on different  criteria that you defined and attributes. Now you can select what you are interested  and actually interest you. Like if you want to select the type of indicator to be only  hash, you can apply it. If you want based on the malware type, then you can just like  extract it. If you want combination of the stuff that is happening, let's say the most  recent indicators that have been seen through different networks, you can just like extract  the information in the database. Aging is also another thing that is very much important  and needs to be considered. You don't want to save lots of indicators in your database  and then get an indicator that is not happening in any database anymore. So, as you see the  indicators and as you are ingesting the indicators in your database, you need to age them properly  and set it based on the malware type, based on the scoring, set it somehow from like 60  days or like a little bit more and then age them out if they are not being seen. So you  don't have like something that is not happening in the network anymore in your feeds. So as  I mentioned, like you get your feeds and indicators from different sources, from third party sources,  from internal sources, you do apply some intelligence, white listing, scoring, filtering, aging  and then you can select your feeds based on the criteria that you are interested. Now  imagine that you want to have high-periority feeds that are actually very critical and  needs to be taken care of immediately. You can just like have your alerts based on these  feeds like very high, you can define medium-periority feeds or like low confidence. And with all  these criteria that I just told you, you are going to have a better result, less false  positive, selecting true positive and you do need to have aging for having a current  indicators and the sort of like a real-time results. Do you have any questions?
DEF_CON_26_PACKET_HACKING_VILLAGE_-_Gita_Ziabari___How_to_Tune_Automation_to_Avoid_Fa_0_263.png DEF_CON_26_PACKET_HACKING_VILLAGE_-_Gita_Ziabari___How_to_Tune_Automation_to_Avoid_Fa_264_887.png DEF_CON_26_PACKET_HACKING_VILLAGE_-_Gita_Ziabari___How_to_Tune_Automation_to_Avoid_Fa_888_1259.png DEF_CON_26_PACKET_HACKING_VILLAGE_-_Gita_Ziabari___How_to_Tune_Automation_to_Avoid_Fa_1260_1379.png DEF_CON_26_PACKET_HACKING_VILLAGE_-_Gita_Ziabari___How_to_Tune_Automation_to_Avoid_Fa_1380_2123.png DEF_CON_26_PACKET_HACKING_VILLAGE_-_Gita_Ziabari___How_to_Tune_Automation_to_Avoid_Fa_2124_2543.png DEF_CON_26_PACKET_HACKING_VILLAGE_-_Gita_Ziabari___How_to_Tune_Automation_to_Avoid_Fa_2544_5039.png DEF_CON_26_PACKET_HACKING_VILLAGE_-_Gita_Ziabari___How_to_Tune_Automation_to_Avoid_Fa_5040_5243.png DEF_CON_26_PACKET_HACKING_VILLAGE_-_Gita_Ziabari___How_to_Tune_Automation_to_Avoid_Fa_5244_5627.png DEF_CON_26_PACKET_HACKING_VILLAGE_-_Gita_Ziabari___How_to_Tune_Automation_to_Avoid_Fa_5628_6035.png DEF_CON_26_PACKET_HACKING_VILLAGE_-_Gita_Ziabari___How_to_Tune_Automation_to_Avoid_Fa_6036_7967.png DEF_CON_26_PACKET_HACKING_VILLAGE_-_Gita_Ziabari___How_to_Tune_Automation_to_Avoid_Fa_7968_12635.png DEF_CON_26_PACKET_HACKING_VILLAGE_-_Gita_Ziabari___How_to_Tune_Automation_to_Avoid_Fa_12636_16979.png DEF_CON_26_PACKET_HACKING_VILLAGE_-_Gita_Ziabari___How_to_Tune_Automation_to_Avoid_Fa_16980_17267.png DEF_CON_26_PACKET_HACKING_VILLAGE_-_Gita_Ziabari___How_to_Tune_Automation_to_Avoid_Fa_17268_18875.png DEF_CON_26_PACKET_HACKING_VILLAGE_-_Gita_Ziabari___How_to_Tune_Automation_to_Avoid_Fa_18876_19427.png DEF_CON_26_PACKET_HACKING_VILLAGE_-_Gita_Ziabari___How_to_Tune_Automation_to_Avoid_Fa_19428_21095.png DEF_CON_26_PACKET_HACKING_VILLAGE_-_Gita_Ziabari___How_to_Tune_Automation_to_Avoid_Fa_21096_22499.png DEF_CON_26_PACKET_HACKING_VILLAGE_-_Gita_Ziabari___How_to_Tune_Automation_to_Avoid_Fa_22500_23843.png DEF_CON_26_PACKET_HACKING_VILLAGE_-_Gita_Ziabari___How_to_Tune_Automation_to_Avoid_Fa_23844_24851.png DEF_CON_26_PACKET_HACKING_VILLAGE_-_Gita_Ziabari___How_to_Tune_Automation_to_Avoid_Fa_24852_25859.png DEF_CON_26_PACKET_HACKING_VILLAGE_-_Gita_Ziabari___How_to_Tune_Automation_to_Avoid_Fa_25860_26315.png DEF_CON_26_PACKET_HACKING_VILLAGE_-_Gita_Ziabari___How_to_Tune_Automation_to_Avoid_Fa_26316_26471.png DEF_CON_26_PACKET_HACKING_VILLAGE_-_Gita_Ziabari___How_to_Tune_Automation_to_Avoid_Fa_26472_26665.png
All right, good morning. We're out to our second talk and next speaker is, you've been a few times at the packet hacking village talks. Gita Ziyabari. Hi, everyone. Thanks for coming to my presentation. My name is Gita Ziyabari. I'm coming from Verizon, working as a senior consultant engineer software developer and I'm going to talk about how to tune automation to avoid false positive. So I'm going to talk about techniques to design a reliable automated tool, introducing threat intelligence feeds and how to avoid false positive when we generate automated tools and feeds. Techniques to design a reliable automated tool. So there are many different reasons that automation is needed in cybersecurity. There are massive amount of data that needs to be analyzed on an early basis and action is needed. So accelerated response time, consistency, scalability, efficiency, risk reduction, simplified IR process, empowering users, they are all good reasons that we do need to have automation in cybersecurity. However, since we have large amount of data and all of us are very much busy, bad ideas could lead to false positive and false positive is always bad especially in cybersecurity. So automation needs to be done with intelligence. By intelligence, I mean let's just consider creating an automated tool that is pretty much simple to use because no one is going to be interested to learn a new tool that is very much complicated and it's designed to have a lot of let's say GUI pages with lots of maintenance and everything that it just like takes a long time. And think about five years, you are creating an automated tool and you move on. If you want to maintain it all the time and something breaks all the time, then it's not going to be usable. Make it user friendly so that the threat researchers who are basically the target to use the automated tool, they buy it and they start using it right away. Make it like have some documented design framework and think about whoever is going to inherit the framework. It might be an open source you may decide to publish it. So think about those that are going to use and inherit the code. Most of researchers in cybersecurity and analysts, they already have a framework that they are based on it and they are actually using it. So if you want to say that, yeah, I have this automated tool, it's pretty cool and it's just like adding these many features to your analysis and it's so fast and everything, they are going to hesitate using it because they already have something that they are using and switching from one source that is actually working to a new one is going to be a bit hard. So make it... I'm sorry. I apologize. So make it somehow that you guys can hear me? That's good. That's much better. So make it somehow between more dependencies. First of all, to other servers. Again, if you want to publish it in GitHub, people who are going to download your tool, they are not going to have the same servers that you have. So make it easily integrated to any platform that is being used by different analysts, by different groups, by different companies. It just doesn't matter. Just make it as much as possible independent with possibility of integration. Then it comes to planning the automation. Again, simplicity is the most important thing because imagine that you could create an automated tool, a script that could be run through Chrome job, in background, through command line, or you have a pretty nice UI. View UI, believe me, it just requires a lot of maintenance and there are so many patches that you have to just like upgrade. There are so many dependencies and, on the list, they are not going to like it, like trade researchers. They have to just like go to this website, they have to access it, and then they have to log into it. There are so many layers involved. So just try to make it simple. Chrome job. If you can do it through Chrome job, then do it through Chrome job. Some of us like Python, some of us like pairs, some of us like PHP. If you have a framework that is already designed, let's say doing Python, then stick to Python. And do not just switch because you just get bored from one language and you feel that the other language actually has more functionality for doing this tool that you're, for making this tool that you're making. Make it simple. Avoid dependent processes because we all know that when we have dependent processes, if just one of them breaks, then everything is going to be broken. If you're busy working on other stuff, they are going to ask you to fix it and you don't know where to start. What is actually broken to fix it? And if you have to really have chain of processes, then you have some monitoring systems like Nagios, so that it just like a start reporting you if something goes down. And if it goes down, make some really, really like easy script, like a line of command. If it goes down, like if your MongoDB is down, send me the logs, restart it so that your automated tool is not going to be broken forever till you or someone actually finds out what's going on. In cybersecurity, we are actually lucky developers because we do not have QA in most of cases to QAR code. So that's a good part and a bad part. But we have to be our own QA especially because we are actually having a massive amount of data that needs to be analyzed and performance testing is very important. So make sure to have the platform for yourself for being able to test framework actually to be able to test your own code. That is very, very important. I know that we all do unit testing and feature testing. Performance testing is needed and we do need to consider it. I just quickly jumped to threat intelligence feeds. That's the area that I was working in the past few years. And by site, by threat intelligence, I mean indicator-based threat intelligence feeds such as domains, URLs, IP addresses, hashes, email addresses and whatever you want to consider as an indicator. So threat intelligence feeds, you can get it from third party feeds. Like there are some open sources, there are some community-based based on the circle of trust, commercial ones you have to pay, good money. And the government-based internal feeds are also valuable. You get feedbacks from analysts that these are basically the feeds that I could pass you and indicators that I could pass you. Or you create some automated data mining tools or you use the ones that are available and start mining data from the data and from the logs that you have. You just generated how many of you guys are using threat intelligence feeds? How many of you are happy with the result? None. I was expecting it actually. So the reason is that it's just like poor quality control, overlapping indicators, false positives, noise. It's just like massively creating noise. And most of the companies, they would rather not to use it rather than using it because it's just like generating noise and you don't know where is the source coming or if you can't trust it or not. Now, generating high-quality feeds is possible, believe me. The first step that you need to take action is actually having a database. Don't take everything from fly, from different sources, make it to like a JSON or CSV format and then export it to whoever you want to. Have everything in your database and in that case, you would be able to apply quality and you have control on the indicators that you're ingesting in the database. What needs to be done is do duplicating, white listing, filtering, scoring, and aging. If you are inserting an indicator, check your database and see if it exists in the database. If it does, then update it based on the source, like aging, scoring, just like make some changes, but do not reinsert it. When you want to export the feed, it's very important to also do duplicating so that you do not have to duplicate indicators. White listing, it could be done through third-party sources, it could be done through internally. Third-party sources, there are some sources that are available, like external sources, some of them are free, some of them you have to pay some money. Most of them are based on the popularity of the indicators. For example, the websites that are being most visited by different countries, they are being introduced as white lists. Try to scoop up just like the top 1000 and even if you are considering the top 1000, let's say, domains, still apply some filters. Filtering is very important, actually, in feed, because imagine that the adult content-based domains are also pretty much popular and some of them are even in the top 50 indicators. They are not necessarily good or trustable, they are just popular. So, as quantity increases, consider that the possibility of having false positive also increases. You get feedback from your peers, your analysts, your customers about white indicators that actually are false positive immediately, quarantine, then have another media tool to check the results through different sources and then white list then immediately. Everything has to be done in automatic way because you cannot just go through every single indicator one by one. And believe me, one bad indicator could result in many hits like 100k plus. So, when I said that we don't have QA and we have to be our own QA, have your framework, the test framework, and start testing your feeds that you are generating, write a script, see the number of matches and have a statistic about the indicators that are matching a lot of hits. And then quarantine them, analyze them through another source and see if they are actually false positive or they are true positive, immediately white list them. So, you get a data, a big amount of data in indicators, you apply white listing from any source, internal, external, you have a better data, it's not enough though. You do need to do some scoring. So, imagine that you are getting, you are using this many sources, like open sources, different sources, sources coming from your internal or external. And the indicators that are being reported repeatedly through different sources, it means that they are actually happening and they have a better chance to be malicious. So, their score is actually higher than the ones that are coming from a single source. So, they should have a higher score. A score of sources is also important. Check out the results that you are getting, the alerts that is being generated and check out the number of false positive that you are getting from different sources. Have a statistic. Again, here, a test framework is needed if you want to analyze the sources and the score of the sources that are coming. The ones that are noisier and they are generating more false positive, they need to be less scored and the ones that are more reliable, they need to be higher scored. Now, the result that you are getting today may be different one week from now. So, again, you need to have an automated way for getting these scored of especially the sources and also the common number of sources that are reporting an indicator and tune the algorithm somehow that gives you the right results. This is just like an algorithm. You get the indicators, see if it exists in database. If it doesn't exist, then score it based on the source that you have obtained. If it does exist, then check the common sources, the quality, the quantity of the common sources and also the quality of the sources. In this phase, right here, you can apply many different scoring methods. Let's say if you have access to the malware type, then you can start scoring the malware type as well. So, you can just apply intelligence based on your needs if you have the malware type, malware type could be added here. So, filtering, we just talked about like white listing, scoring. There are so many different methods that you could apply in this phase for applying more filtering to have a better filtered data. There are some tools available that you could just like check the indicator through trusted parties that they have access to many AVN genes and you can get a report from them and see what is the number of positives for this particular indicator. Then you want to build up the queries, be as specific as possible. More attributes, it means that you can have a better database of indicators and you can easily search on whatever you are interested. So, make sure that the critical attributes that you are having are indicators, the indicator itself, the type of it, if it is a hash, URL, domain, IP address, email, anything. Unique index based on the sources so that if an alert gets generated, you would know that, okay, this is coming from this source or this source. So, you have a control for the nubbing thing for the algorithm that I just mentioned to avoid false positive. A list of sources, the score that is given currently, data being insertion, malware type and age. So, when it comes to feeds, you cannot say that, okay, I have my feed that I built like a week ago and now it's just like this is what I have and I'm going to submit it. It needs to be updated. Ingestions should happen in real time and you need to apply the intelligence, the algorithms that we just covered and export the data at least on an early basis to have a real time, almost real time, I would say feeds. You could be selective. Now you have the pool of indicators based on different criteria that you defined and attributes. Now you can select what you are interested and actually interest you. Like if you want to select the type of indicator to be only hash, you can apply it. If you want based on the malware type, then you can just like extract it. If you want combination of the stuff that is happening, let's say the most recent indicators that have been seen through different networks, you can just like extract the information in the database. Aging is also another thing that is very much important and needs to be considered. You don't want to save lots of indicators in your database and then get an indicator that is not happening in any database anymore. So, as you see the indicators and as you are ingesting the indicators in your database, you need to age them properly and set it based on the malware type, based on the scoring, set it somehow from like 60 days or like a little bit more and then age them out if they are not being seen. So you don't have like something that is not happening in the network anymore in your feeds. So as I mentioned, like you get your feeds and indicators from different sources, from third party sources, from internal sources, you do apply some intelligence, white listing, scoring, filtering, aging and then you can select your feeds based on the criteria that you are interested. Now imagine that you want to have high-periority feeds that are actually very critical and needs to be taken care of immediately. You can just like have your alerts based on these feeds like very high, you can define medium-periority feeds or like low confidence. And with all these criteria that I just told you, you are going to have a better result, less false positive, selecting true positive and you do need to have aging for having a current indicators and the sort of like a real-time results. Do you have any questions?
going to talk about how to tune automation to techniques to design a reliable automated tool, action is needed. So accelerated response time, consistency, scalability, efficiency, risk you are creating an automated tool and you move servers that you have. So make it easily are actually having a massive amount of data that it to whoever you want to. Have everything in results that you are getting, the alerts that is mentioned to avoid false positive. A list of and needs to be considered. You don't want to