AI Village - Machine Learning Security Evasion Competition 2020
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
| Title |
| |
| Title of Series | ||
| Number of Parts | 374 | |
| Author | ||
| License | CC Attribution 3.0 Unported: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor. | |
| Identifiers | 10.5446/49687 (DOI) | |
| Publisher | ||
| Release Date | ||
| Language |
Content Metadata
| Subject Area | ||
| Genre | ||
| Abstract |
|
00:00
Proxy serverAsynchronous Transfer ModeStructural loadPoint (geometry)Different (Kate Ryan album)Slide ruleAdditionWordForm (programming)Position operatorProduct (business)Pulse (signal processing)Internet forumMultiplication signSheaf (mathematics)Price indexArmComputer fontVisualization (computer graphics)Presentation of a groupTotal S.A.CuboidMultiplicationProxy serverSoftwareBeer steinData miningBlogSampling (statistics)Strategy gameRule of inferenceNumberRepresentation (politics)Game theoryMedical imagingProjective planeSinc functionVirtual machineSocial classField (computer science)Polygon meshOrder (biology)DemosceneDecision theoryOverlay-NetzCASE <Informatik>WeightArithmetic meanRing (mathematics)Cellular automatonEndliche ModelltheorieMalwareBinary codeComputer fileInformation securityLink (knot theory)Machine learningString (computer science)Antivirus softwarePlastikkarteKey (cryptography)
06:22
Asynchronous Transfer ModeHash functionoutputMaxima and minimaExecution unitArrow of timeFuzzy logicMathematicsContent (media)QuantumConvex hullTrailFlowchartHill differential equationSample (statistics)Computer wormInformationSicHigh-level programming languageInflection pointPoint cloudScalabilityGraphical user interfaceScripting languageCurve fittingComputer fileMultiplication signPoint (geometry)Virtual machineOrder (biology)Term (mathematics)Image registrationFamilyRule of inferencePlanningPrice indexMultiplicationRevision controlMalwareSampling (statistics)Key (cryptography)Functional (mathematics)Local ringPattern languageQuery languageScalabilityGraphical user interfaceServer (computing)System administratorPatch (Unix)Arithmetic progressionBlack boxGame controllerDebuggerArithmetic meanPartial derivativeCuboidMathematicsScripting languageAdditionEndliche ModelltheorieDifferent (Kate Ryan album)CodeTrailResultantWindowFile formatSheaf (mathematics)WebsiteTouchscreenExtreme programmingCASE <Informatik>Workstation <Musikinstrument>Product (business)2 (number)State of matterProxy serverInformationArmForcing (mathematics)HierarchyElectronic signatureAngleMaxima and minimaNatural languageComputer animation
12:44
Asynchronous Transfer ModeEndliche ModelltheorieData modelTimestampSample (statistics)CodeStrategy gameSmoothingMalwareFunction (mathematics)HeuristicHypercubeEndliche ModelltheorieVirtual machineSampling (statistics)Maxima and minimaInterior (topology)Computer fileINTEGRALMultiplication signAlgorithmAdditionWebsiteCodeMathematical optimizationFunktionenalgebraRoundness (object)Point (geometry)Process (computing)MalwareMereologyMenu (computing)PredictabilityNumberSource codeSmoothingRing (mathematics)RankingRule of inferenceQuery languageStrategy gameData structureSeries (mathematics)Execution unitTimestampFunction (mathematics)InformationZustandsgrößeBlack boxElectric generatorDemosceneArithmetic meanNP-hardWeightFunctional (mathematics)Electronic program guideOrder (biology)Total S.A.Ganzzahlige lineare OptimierungEvent horizonComputer animation
17:13
Asynchronous Transfer ModeChi-squared distributionLink (knot theory)Endliche ModelltheorieEmbedded systemValidity (statistics)1 (number)Function (mathematics)Sampling (statistics)CASE <Informatik>Message passingNumberMultiplication signWebsiteComputer fileCountingEndliche ModelltheorieRight angleFlagMetropolitan area networkMalwareGreatest elementState of matterSeries (mathematics)SoftwareIterationBitElement (mathematics)CodeWindowQuery languageMathematical optimizationWritingDirectory serviceRevision controlVirtual machineContext awareness
20:57
BlogAsynchronous Transfer ModeLine (geometry)Ring (mathematics)WebsiteEmailComputer animation
Transcript: English(auto-generated)
00:02
Good morning, everyone, and welcome to the Machine Learning Security Invasion Competition 2020. My name is Zoltan Balazs, and this is a project together with Hiram Anderson. We are going to talk about machine learning detection
00:21
bypasses. And in the past, you probably have seen some researches where people modified existing images. And the goal was to bypass machine learning classifiers. So for example, for a human viewer,
00:41
the new image looks like the original one. But for a machine learning classifier, this looks like something different, like it will believe that this looks like an ostrich. In this talk, we are going to present machine learning
01:00
bypasses when it comes to malicious software. And there have been some interesting researches in the past regarding this topic as well. For example, last year, there was a new research published where people extracted strings from a known game executable,
01:23
appended this to a known malware, and it was able to bypass a production machine learning model. I also did some research in the past, and like four years ago, you were able to bypass some machine learning models just
01:40
by packing a sample with UPX. In order to advance the field of offensive and defensive machine learning-based malware detection, last year we created a challenge where you had to download 50 working malware samples.
02:01
You had to download three machine learning models with its weights, modify the malware samples to evade detection by all models. And if you were lucky and you had the most points, you were able to win this nice GPU card. In total, 70 people registered for this competition,
02:24
and at least 11 people were able to bypass at least one machine learning model. Congratulations to the winner, William Fleschmann, and I highly recommend you to check out his blog post on how he won this competition.
02:43
There were some other write-ups and papers as well, I do recommend you to check out those as well. You can see one from Jakub and one from Fabrizio on the following links. When it comes to win this competition,
03:02
multiple approaches were used. Some people started with a simple packer, like the one I mentioned. But unfortunately, some of the samples are already packed in a way. And this means that if you start with UPX
03:23
or something similar, it will not work anymore. Another great approach was to add new sections to the executable. For example, you can extract the end user license agreement
03:44
resource from Microsoft files and add it to the malware samples multiple times. This approach was really good at bypassing the detection for the ML models. But unfortunately, again, this broke some of the malware binaries.
04:04
Fun fact, if you just simply add sections to a malware sample, you might be able to bypass some antivirus detection because for performance reasons, some AV engines check the number of sections
04:23
before they evaluate the rest of the rules. At the end, the winning strategy was to just append random data to the end of the executable. This is called as an overlay.
04:41
Even though this is a very simple strategy, it worked during last year competition. And this is also an easy way to bypass if the sample has any kind of self-protection,
05:00
for example. Just by increasing the size of the sample, again, you might be able to bypass some antivirus engines. Again, they can have file size in their rules. And one important thing,
05:23
as you can see on the top right image, that's a visual representation of a malware. And I just appended some random strings to the end of this sample. And if you look at the green visual,
05:43
you can clearly see how this changed the visual representation of the sample. For us, there were some key takeaways from last year competition.
06:00
For example, some of the machine learning models are way too academic, but not very effective in practice. Turnout, it's not just us, but everybody thinks that the EF tool is awesome. This is a Python package you can use to modify binaries.
06:21
And as it is the case with malware, it is always tricky to deal with them. For example, some of the samples do not reproduce the same indicators of compromise over time. This can be because, for example, the command and control server is down, and dealing with packed and protected samples
06:43
can be hard sometimes. I also checked the SSD patches of some of the samples, and it was interesting to see that whenever people added repeating patterns,
07:00
to the sample, for example, the same section, or they added the same overlay over and over again to the sample, then it created a repeating pattern in the SSD patch as well, which can be used for detecting a sample
07:21
which uses a machine learning deviation, for example. This year, we created the defender and the attacker challenge. In the defender challenge, you had to create your own machine learning model and submit this to the competition in a Docker format. And in the attacker challenge,
07:41
now the machine learning models are not available for you. So this is now a black box challenge. And if you win this competition, either the defender or the attacker challenge, you can win some Azure credits for your machine learning research plans.
08:02
The defensive track is already over. We received two submissions that passed the minimum requirements, and the offensive track already started. So I highly recommend you to go to our website, anasag.io, and check out what the competition is.
08:25
In this year, we have used the following malware families. And if you go to our website, review the terms of service, and then you can download the 50 provided malware samples. And after that, it is your time to modify the samples
08:44
in order to evade the detection. And new to this year, you can use an API to check your samples or submit your samples. I also recommend you that you verified the malware functionality remains the same
09:03
in your local Windows box. Then when you uploaded the zip file, so you can just upload the partial zip files, meaning that you only submit some of the samples and not all of them. You can receive one point
09:23
for each bypassed machine learning model, which means that for every sample, you can get up to three points. And as usual, highest score wins. The details about this will be provided by Hiram.
09:41
And in order to claim your price, you have to publish your solution. Please note that you have to keep the file names as it were in the original zip file. This helps us to track which file you modified originally.
10:03
We also provide some additional tips and tricks you might use in this competition. Some of them may not make sense, but you can modify and execute it in a lot of different ways. For example, you can add or remove signatures,
10:21
change section names, properties, modify the import or export tables, create TLS callbacks, change the PA header, fix or change the checksums, add, modify or remove the version information, create new entry points, or just change some code or data in it.
10:42
Still, it's not allowed to create droppers or self-extracting air-hives because this will kind of defeat the purpose of the whole competition. And this year, keep in mind that multiple registration is against the rules and it will result in immediate disqualification.
11:03
Please do join our Slack channel where you can discuss everything with us and you can also discuss your progress with the other participants of this competition.
11:21
Just a side note, the whole front end was created in Python Flask admin and we are using Cloudflare, Nginx, and GUnicorn for scalability and performance reasons. There are some backend scripts running with Python
11:40
as scheduled by Chrome. And as it was the case last year, we still use the VM-REA sandbox to evaluate the samples. As mentioned, we already have an API. So if you want to check your sample against all the machine learning models,
12:03
or just against one machine learning model, you can use the API just to do that. And also you can use the API to get the results. And if you are satisfied with bypassing the machine learning models, you can upload your zip files
12:22
and query the zip status and the sample statuses as well with the API keys. This is all I wanted to share with you guys, but please welcome Hirom, who will present you some other tips and tricks you can use to win this competition.
12:42
Thank you. I'm going to describe to you the example solution in the Machine Learning Security Evasion Competition's Attacker Challenge that has just begun. The models that you'll be attacking this year have been submitted by participants
13:02
of the previous round in the Defender Challenge. Two of the models from the previous round have qualified to be included in this round. In addition, we have hosted our own model for you to attack. That model is trained on the Ember dataset and includes some basic capability
13:22
to detect adversarial examples. The source code and model weights for this defendant Ember model are provided on the competition's GitHub site. However, the remaining models are to you complete black boxes where you only get to observe the hard label predictions,
13:43
that is a zero or a one, for an output that you provide to the machine learning models. The final leaderboard ranking will be set by the following rank ordered criteria. First, the total number of evasions with one point for each of the three ML models
14:03
times 50 malware samples, meaning that the maximum score is 150. Remember though that each evasive sample must reproduce its original functionality in a sandbox in order to be awarded a point. Functionality is verified only
14:22
when you upload a zip file containing your candidate malware samples. It will not be verified when you merely query the machine learning models through the API. In the event of a tie for point number one,
14:41
contestants will be ranked by the number of model queries used through the API. And lastly, the timestamp of your final zip upload would break any subsequent tie. More than likely though, we won't get to point number three, so you should feel incentivized to continue competing
15:01
right up until the competition deadline. Even if you see a perfect score on the leaderboard because you might achieve that same perfect score but do it more efficiently. So as a contestant, you can choose any strategy you'd like to compete. But to demonstrate one possible strategy, we have released some example code
15:22
on the competition's GitHub site. You can find more information about the nitty-gritty details of this approach on the website essentially as using a discrete optimization technique over a space of functionality preserving file modifications. However, the general strategy
15:40
might be more useful for you to adopt. The strategy is really simple, consists of doing a bunch of bulk work using an algorithm in part A and then kind of batting cleanup for manual manipulation of malware samples in part B. And I'm gonna be describing and demoing the code for part A today.
16:02
In it, because we'd like to be efficient in the number of queries against the hosted machine learning models, we'll actually break this attack into two parts. An offline attack where we use the defended Ember model for which we have code to kind of work out our strategy
16:20
and generate initial malware samples. We hope that those seeds might evade some of the online models that are hosted. And then in the online attack, we'll take those initial seeds and the algorithm will further optimize and discover additional file modifications required to evade the online hosted models.
16:44
Some tricks that we're using here include label smoothing where we're converting the hard label outputs into a soft score by averaging four things, the three hard label outputs from the hosted machine learning models,
17:00
as well as a local score from a local machine learning Ember model that will be used as a heuristic to kind of guide the optimization process. So as I demo this code, I want you to please be aware that this code writes malware to disk.
17:20
So please do run this code only using a Linux VM. To begin, we initialized the attack by analyzing a collection of benign files. This init subcommand extracts elements of these benign files that will be later injected into the malware.
17:45
To launch our offline attack, we'll run a local copy of the Ember model in the top window. Then in the bottom window, we'll use the run command passing in malware samples that we downloaded after registering on the website.
18:01
The tool will then write successful evasion attempts to pass one slash success that we've specified in the command line and failed attempts to pass to slash failure folder. And also included in each output directory will be the history of file modifications
18:23
that will be useful if we'd like to pick up to resume a failed attempt. So to demonstrate that, in a second pass of the offline attack, we'll start with the pass one failures
18:41
and iterate on the optimization approach. Again, storing successes and failures to a pass two directory. So after doing that a number of times and having collected a bunch of candidate samples offline that evade the local defended Ember model,
19:02
we'll then use those candidates as seeds for an online attack, which now counts against our API query usage. To do an online attack, simply use this tool with a dash dash online flag and the optimization will then continue
19:21
trying to find file modifications that will bypass all three of the hosted models. Of course, you want to do perhaps as many iterations as necessary in the online version of this attack. But after you've done so, in a final pass of the online attack,
19:41
you can now collect the successful samples into a zip file that you would then validate in a Windows 10 virtual machine and then upload to the website for validation and leaderboard scoring. I want to point out that since there is a chance
20:00
that by running this code, file modifications might break some of the samples, you should always run these samples in a Windows 10 sandbox before uploading to the competition website. Also note that zip file uploads contribute against your API query count. So it is to your benefit to double check your work
20:22
and make sure that any files you upload are functional so you don't have to redo that work and upload again. As a final note, kind of tricky that since the hosted models might be actually changing state and learning from the queries that you and others are giving them,
20:40
there's a possibility that a evasive variant sample that you discovered along the way may no longer evade a model by the time that you upload your zip file. So I don't know if that will be the case, but please be aware that that is a possibility. So with that, good luck on the competition.
21:01
Visit the website and email sec.io. The competition will run for over six weeks and those who are ranked first and second on the leaderboard will win our grand and first prizes respectively, so long as they publish their solution.
21:23
And with that, I'd like to thank our sponsors, Microsoft and Cujo AI with partners MRG Efites and VMRay.