---
author:
- 'Matilda Rhode,  Pete Burnap,  and Kevin Jones [^1]'
bibliography:
- 'refs.bib'
title: 'Distillation for run-time malware process detection and automated process killing'
---

Malicious software (malware) evolves rapidly both to evade detection and take advantage of new vulnerabilities. In the first half of 2019, 3.6 new malware samples appeared each second according to AV-TEST, a security research institute [@avtest]. Automatic detection techniques are necessary to cope with the rate at which new malware samples are being created as the expert human resources required to analyse this volume of malware are limited. Automatic detection uses data to make decisions and this data can be static or dynamic. Dynamic data is less vulnerable to obfuscation techniques [@damodaran2017comparison] and analysis of this data can be conducted offline or at run-time (online). Offline dynamic data has been used to train machine learning models, achieving $>$95% detection accuracy (e.g. [@HuangStokes], [@shijo2015integrated], [@rosenberg2017generic]) using the full post-execution traces of software. Dynamic models are not, however, widely used for endpoint protection as they take much longer to record the execution trace and reach a decision.

**Detection: Online vs. Offline.** Online (run-time) malware detection provides real-time analysis *whilst* a sample executes, whereas offline detection first collects data, then analyses this data. Real-time analysis can prevent problems rather than simply detecting that they have occurred after the fact. In the same way that self-driving cars need to detect a “STOP" sign as early as possible (rather than 10 minutes after it has passed) in order to mitigate the chances of a road accident, online malware detection can detect malware before the malicious payload is delivered. Online detection of malicious software is more challenging than offline detection because typically online detection must use a partial execution trace rather than the full behavioural footprint. In order to protect the machine we can assume that the earlier a malicious process is detected and killed, the more likely we are to have prevented damage to the victim machine/network. This means making decisions based on short behavioural traces. Our previous work [@rhode2018early] found that short offline behavioural traces can still give high detection accuracy for offline detection, but this used machine-level data and was not able to identify the specific malicious processes that needed to be killed to mitigate damage to the endpoint. It is not feasible that a user will wait for the entire execution trace, which can take several minutes to deliver a verdict, therefore, we propose an online run-time behavioural detection tool to kill malware as early as possible with a goal to mitigate harm. Previous work on process killing [@fastSlow2017] has used behavioural traces of 5 to 10 minutes, but this is sufficient time for malware to cause damage. One security vendor has found that Chimera ransomware can encrypt 70 MB worth of files (stored in 1,000 documents) in 18 seconds [@barkly_chimera]. This is the first paper, to the best of our knowledge, to try and detect and kill malicious processes with a goal to mitigate damage to the target endpoint.

**Malicious application vs. malicious process detection**. Killing a malicious sample amounts to killing the processes underlying that application. Previous malware detection research collects behavioural data for the entire (virtual) machine whilst a single application executes. However, on an endpoint in use it is less trivial to distinguish the malicious and benign applications by observing the behaviours of the machine alone - given aggregation of many applications into behavioural traces. Rather, it is necessary to look at the behaviours of the individual processes. This is more challenging than application-level detection because a model is only able to observe partial components of the behavioural trace in each constituent process. Sun et al.[@fastSlow2017] present the only previous work to attempt process-level detection. The authors use API call sequences but find that the online API hooking slows detection and is therefore unsuited to real-time early detection and process killing. In this paper we use machine activity metrics (e.g. CPU use, memory use, network traffic metrics) to capture process-level behaviour as these can quickly be extracted at detection time to enable early process killing.

Two fundamental challenges for run-time malicious process detection and automated process killing are:

*(i)* The false positive rate is prone to increase as a single false positive detection during several hours of true negative detections will terminate that process (and its child processes) this can mean work being lost, interruption to business processes and other significant consequences for the user. Previous malware detection models cannot emphasise the impact of a false positive. We develop a modified loss function inspired by reinforcement learning to teach a recurrent neural network (RNN) the impact of automatic process killing following a positive detection.

*(ii)* The model needs to be quick enough at killing malicious processes to prevent damage and thus justify the use of run-time detection. We propose distilling the learning of the RNN into an algorithm with a shorter inference speed, a Random Forest. The distilled model benefits from the RNN’s learning and requires less data collection, storage and inference time during execution than the RNN thus allowing for a faster response.

This paper contributes to the field of online malware detection in the following ways:

-   This is the first paper, as far as we are aware, to address run-time process killing looking at the tangible impact on the victim endpoint machine.

-   A novel loss function and distilled model for malicious process detection is presented to improve inference speed and response time. Previous work ( [@fastSlow2017; @yuan2017phd]) has used a two-stage model with a lightweight model sending suspicious processes to a more computationally intensive model. This approach is vulnerable to getting stuck in an infinite loop of back-and-forth between the two models, thus consuming as much processing power as if the intensive model were running constantly.

-   The model is judged using a realistic environment based on a typical laptop with between 2 and 35 applications running simultaneously (up to 95 simultaneous processes), this limited processing capability and scale of execution has not previously been investigated for run-time malware detection.

The following paper will first present related work in Section \[related\_work\]. The proposed model is presented in Section \[proposed\_approach\]. In Section \[experiments\] three models are compared: first an offline malware process detection model using the entire execution trace (\[offline\]), second an online run-time model trained with a modified loss function using automated process killing (\[online\]), and third, a distilled Random Forest model for improved inference speed (\[distillation\]). Finally a live test is conducted with ransomware to demonstrate the tangible benefits of the distilled model (\[ransomware\_case\_study\]) followed by a discussion of future work (Section \[future\_work\]) and conclusions (Section \[conclusion\])

Related work {#related_work}
============

Malware detection using dynamic behaviours
------------------------------------------

The data used to distinguish malicious and benign software can be either static or dynamic. Static detection uses data extracted directly from the code of a malware sample, whereas dynamic data records the behavioural traces of malware as it executes. Traditional antivirus techniques create static signatures from known malware and use these to distinguish malicious and benign files. These signatures will not capture new malware variants (zero-days) and can be bypassed by simple obfuscation techniques to disguise even known variants [@you2010malware]. More recently machine learning models trained on static data have shown good detection accuracy, e.g. Chen et. al.[@chen2019gene] achieved 96% detection accuracy using statically-extracted sequences of API calls to train a Random Forest model. However, in direct comparison with dynamic data, static data performed less well and did not even improve accuracy when used to augment dynamic datasets [@damodaran2017comparison]. Adversarial samples present an additional emerging concern to static machine learning models. Adversarial samples are specifically crafted to cause machine learning models to misclassify. Grosse et al.[@grosse2016adversarial] and Kolsnaji et al.[@kolosnjaji2018adversarial] demonstrated that static malware detection models achieving over 90% detection accuracy could be thwarted by injecting code or simply altering the padded code at the end of a compiled binary respectively.

Dynamic behavioural data is more difficult to obfuscate as it is generated by the malware carrying out its functionality. Removing this data is equivalent to removing the functionality, which may be essential to the malicious payload. Again machine learning models have been used to draw out patterns between malicious and benign software using dynamic data. Various dynamic data can be collected to describe malware behaviour. The most commonly used data are API calls made to the operating system, typically recorded in short sequences or by frequency of occurrence. Huang and Stokes’s research [@HuangStokes] reports the highest accuracy in recent malware detection literature with a very large dataset of more than 6 million samples to achieve an accurate detection rate of 99.64% using a neural network trained on the input parameters passed to API calls and the co-occurrence of API calls. Other dynamic data sources include dynamic opcode sequences (e.g. Carlin et al.[@carlin2019cost] achieve 99.04% using a Random Forest), hardware performance counters (e.g. Sayadi [@sayadi2018ensemble] achieve 94% on Linux/Ubuntu malware using a decision tree), and machine activity metrics (e.g. Burnap et al. [@somburnap] achieve 94% using a self-organising map). Recently, Rhode et al. [@rhode2019lab] demonstrated the robustness of machine activity metrics over API calls in detecting malware collected from different sources.

Dynamic detection is more difficult to obfuscate but typically the time taken to collect data is several minutes, making it less attractive for endpoint detection systems. Some progress has been made on early detection of malware. For instance, Rhode et al. [@rhode2018early] were able to detect malware with 94% accuracy within 5 seconds of executions. However, as a sandbox-based method, malware which is inactive for the first 5 seconds is unlikely to be detected with this approach.

Moreover, the majority of dynamic malware detection papers use virtualised environments to collect data. Sophisticated malware can detect when it is running in a virtual machine and behave differently under these conditions [@splitPersonality]. The only way to ensure that the behaviour being observed during execution is the same behaviour as would occur on a victim endpoint is to either perfectly mimic the victim environment, which could be financially or practically impossible if there are a variety of endpoints being detected, or to collect data from the victim endpoint itself. This paper examines the possibility of a run-time malware detection with an automated response to malware being detected.

Run-time malicious process detection
------------------------------------

Run-time malware detection uses dynamic data collected as the malware executes on the target endpoint. In transitioning from a virtual machine to a machine in use, three additional challenges appear for malware detection. The first is that a user is likely to be interacting with the machine so the malware detection model needs to be able to distinguish malicious and benign behaviours. The second is that the malware is now running on a networked machine with potentially sensitive data that could be compromised. Third, detection models need to be lightweight to minimise the chances of disruption to benign use.

Run-time detection on an endpoint in use requires us to distinguish between malware and benignware that are running simultaneously. Offline malware detection research will typically execute malware and benignware one-by-one in a virtual environment and monitor the behaviour on the virtual machine. This is not possible for run-time detection. The individual processes need to be examined instead because looking at behaviour on the overall machine will include data generated by both malicious and benign processes. Moreover we want to isolate the malicious processes so that they can be automatically blocked or investigated further.

The first challenge to detect malicious processes has only previously been addressed by Sun et al. [@fastSlow2017] using sequential API call data. The authors execute up to 5 benign and malicious programs simultaneously achieving 87% detection accuracy after 5 minutes of execution and 91% accuracy after 10 minutes of execution.

The second challenge posed by the risk that malware executes on the endpoint has not been directly addressed by previous work. Some work has looked at early-stage tun-time detection. Das et al. [@das2016semantics] used an FPGA as part of a hybrid hardware-software approach to detect malicious Linux applications using system API calls which are then classified using a multilayer perceptron. Their model was able to detect 46% of malware within the first 30% of its execution with a false-positive rate of 2% in offline testing. These findings however were not tested with multiple benign and malicious programs running simultaneously and do not explain the impact of detecting 46% of malware within 30% of its execution trace in terms of benefits to a user or the endpoint being protected.

The third challenge (to minimise user disruption) can be linked to the second challenge (to bring about early detection). Some previous work has proposed hardware based detection for lightweight monitoring. Syadi et al. [@sayadi2018ensemble] use high performance counters (HPC) as features to train ensemble learning algorithms and scored 0.94 AUC using a dataset of 100 malicious and 100 benign Linux software samples. Ozsoy et al. [@hardwareWindows] use low-level architectural events to train a multilayer perceptron on the more widely used [@win7] (and attacked) Windows operating system. The model was able to detect 94% of malware with a false positive rate of 7% using partial execution traces of 10,000 committed instructions. The hardware based detection models however, are less portable than software-based systems due to the ability for the same operating system to run on a variety of hardware configurations.

Both Sun et al. [@fastSlow2017] and Yuan [@yuan2017phd] propose two-stage models to address the need for lightweight computation. The first stage comprises a lightweight ML model such as a Random Forest to alert suspicious processes, the second being a deep learning model which is more accurate but more computationally intensive to run. Two-stage models, as Sun et al. [@fastSlow2017] note, can get stuck in an infinite loop of analysis in which the first model flags a process as suspicious but the second model deems it benign and this labelling continues repeatedly. This paper proposes using a distilled lightweight model trained on the outputs of an accurate but computationally intensive model to reduce computational complexity and potentially improve detection accuracy.

Distillation for malware detection
----------------------------------

Distillation describes the process of training a new model on the outputs of another trained (set of) model(s). It was first proposed by Hinton et al. [@hinton2015distilling] to capture the benefits of an ensemble of models in a more computationally lightweight model, however, it has also been used as a defence against the development of adversarial samples for fooling machine learning models [@papernot2016distillation]. Celik and McDaniel [@celik2018extending] applied distillation to a number of security datasets including user face authentication and malware family classification to capture the knowledge of a model that had been exposed to “privileged" data. “Privileged" data is data that is expensive or slow to generate and so will not be available during run-time. We use the finding of Celik and McDaniel’s work [@celik2018extending] to train a Random Forest on a short snapshot of behaviour but benefiting from a recurrent neural network which has been trained by observing a longer window of behavioural data.

Proposed Approach {#proposed_approach}
=================

!ht\]

In this paper we attempt to address the problems posed by virtual-machine detecting, using endpoint monitoring of processes with automatic killing for those deemed malicious. Examining malicious behaviour on a live endpoint requires process-level monitoring in order to distinguish between behaviours arising from benign and malicious applications. There are a number of significant challenges in trying to detect and kill malicious processes in real-time by comparison with behavioural malware detection in a sandbox using the entire execution trace. The two most significant challenges are:

1.  **False positives have worse consequences and are more likely to occur**

    A false positive in real-time detection with process killing, potentially results in a benign process being terminated whilst in use. This results in a worse user experience than not permitting a benign program to run in the first place as work-in-progress could be lost, business processes interrupted, etc. False positives are also more likely because the model must make its decisions based on partial execution traces - it cannot wait for the program to finish before classification otherwise all malware would succeed in executing. If the model believes a program is benign once in one thousand predictions, it is achieving 99.9% accuracy but a single malicious prediction will result in the process being killed and therefore a false positive.

2.  **Malicious processes need to be killed as early as possible** The model should kill malicious processes as soon as possible to reduce the chance of damage taking place. This places restrictions on the time that can be taken for data collection, data preprocessing and model inference. Typically a model will perform better with more data, such as a longer window of execution trace prior to classification, but to improve the response speed the model should use as little data processing possible.

The proposed approach to tackle these challenges uses a modified loss-function and decision-making process to reflect the penalties of false-positives (see Figure \[fig:model\_training\]) together with the distillation technique to improve inference speed (see Figure \[fig:model\_inference\]).

### Modified loss function and decision-making {#modified_loss}

Machine learning algorithms continue to update their parameters during the training phase towards the minimisation of some loss function. In supervised learning problems, for which we have the ground truth labels associated with the data, this means minimising the number of wrong predictions over the entire training set.

Online detection with automatic process killing will stop a process (and all child processes) if that process is deemed malicious at *any single point* during execution, the model predictions are not averaged across the entire execution trace. If a benign process is running for 3 minutes and being monitored every second it may be deemed malicious at 2 minutes 45 seconds but labelled benign the rest of the time, an online process killing model will react to that single alert whereas the offline model described in the previous section would average out all predictions and label the process malicious. This results in a dramatic increase in the false-positive rate.

A neural network’s weights are updated during training to minimise the prediction error across all samples in the training set. Using the previous example of the 3-minute benign process, 180 predictions (60 seconds \* 3 minutes) are made and only 1/180 is wrong. This feeds back a very small loss to the machine learning model such that it may not adjust its weights to eliminate the single wrong prediction for that process at the risk of compromising other correct predictions. For this reason we have proposed a modified loss function to capture the problem posed by online detection.

The modified loss function takes inspiration from reinforcement learning. An offline model will use supervised learning, taking a labelled dataset to guide the training of the model to match as many of those labels correctly as possible. Reinforcement learning uses rewards and penalties from the model’s environment. In reinforcement learning, loss functions are replaced by reward functions which update the neural network weights to reinforce model outputs that lead to higher rewards and discourage model outputs that lead to lower rewards.

In the case of online malware detection, we make two assumptions that underpin the new loss function:

1.  True positives are more useful the earlier in the process that they occur

2.  False positives are undesirable

Assumption A is true if we assume that the longer malware executes, the more chance of greater damage. Assumption B is defined in contrast to Assumption A; if a benign word-processing program is being used, a false positive one second into the program opening is just as unhelpful as a false positive 3 hours into the program running.

Formally the loss function for $N$ samples where $y_i$ is the label for sample $i$ of $N$, $p_i$ is the model prediction for sample $i$, and $t_i$ is the execution time left for sample $i$, loss $L$ is:

$$\frac{1}{N}\sum_{i=1}^{N}((p_i - t_i)^2 + \nint{p_i}\cdot(1 - t_i) + y_i\cdot\frac{1}{t_i + 1}) 
    \label{eq:loss_fn}$$

The first term of the modified loss function (\[eq:loss\_fn\]) is the mean squared error of the model prediction, the second term penalises any false positive at a flat rate and the last term scales the loss inversely proportional to the execution time left of a process for a positive prediction.

Distinguishing malicious and benign processes is a binary detection problem. Outputs closer to 0 indicate that the model believes the process to be benign, and outputs closer to 1 indicate the the model believes the process malicious. The default threshold to distinguish malicious and benign samples is 0.5 but this can be adjusted to reduce the chances of throwing a false positive. Section \[online\] details the results of implementing the modified loss function and altering the decision boundary.

### Killing malicious processes as early as possible

Recurrent neural networks (RNNs) were developed to enable neural networks to classify time-series data. Their usefulness was limited until 1997 when Schmidhuber and Hochreiter [@hoc1997] developed the Long-Short Term Memory (LSTM) cell which made use of gating mechanisms, enabling RNNs to learn over longer time series without the gradients exploding or vanishing. Dynamic behavioural data is time-series data and RNNs (using LSTM or Gated Recurrent Unit Cells) have shown good detection accuracy for various malware detection and classification tasks (e.g. [@rhode2018early; @malwareProcessBehaviour; @PascanuMalwareRNN; @Kolosnjaji2016]). Chung et al. [@chung2014gru] have subsequently developed a variant of the LSTM cell, Gated Recurrent Units (GRUs), which are computationally more efficient due to using one fewer gating mechanism whilst generally maintaining the accuracy of LSTMs on most datasets [@GRUcomp], which are the RNN cells used in this paper.

RNNs have previously been used for accurate malware detection, implying that they could work well for malicious process detection, however, these neural networks are computationally intensive both to train and when used for inference. The inference time for a recurrent neural network scales linearly with the window size of the data it is analysing, and larger windows tend to lead to more accurate predictions [@rhode2018early]. Ideally it would be possible to achieve the accuracy of detection of an RNN with a much smaller model. This paper proposes and evaluates the use of distillation to achieve this.

Distillation was first proposed by Hinton, Vinyals and Dean [@hinton2015distilling] as a means to reducing the computational complexity of ensemble machine learning models. One common way to improve the accuracy of a machine learning-based system is to combine the predictions of multiple models together. These models often work better together than individually but running multiple models increases computational cost. Distillation trains an additional model to predict the result of the ensemble (rather than the ground truth labels) in order to mimic their strengths. Models will typically output a set of logits for a multi-class classification problem or a value between 0 and 1 for a binary classification problem which gives nuance to the predictions and can enables the model used for the distillation to learn more than it would from the ground truth labels. Sometimes the distilled model outperforms the original ensemble [@hinton2015distilling].

Here we use a Random Forest to distil the findings of the RNN. Carlin et al. [@carlin2019cost] found that Random Forests gave the best balance between accurate prediction and inference speed in a comparison of 23 machine learning algorithms. Random Forests cannot process time-series data nor can they output non-integer values meaning they cannot ingest the same data as the RNNs nor predict the values between 0 and 1 output by the model. Instead of the full window of data the Random Forest will just see the most recent captured data (see Figures \[fig:model\_training\] and \[fig:model\_inference\]). This may seem counter-intuitive but Celik and McDaniel [@celik2018extending] found that in a number of security applications (including classifying malware into families) distillation could be used to train models with partial information when compared with the full models from which they are distilled. Instead of values between 0 and 1, the Random Forest will be trained to mimic the behaviour of the malware with the adjusted decision threshold to mitigate the chances of false positives. The results of the distillation experiments are in Section \[distillation\].

Features
--------

A variety of features have been used for dynamic malware detection which can be used to inform the choice for malicious process detection. The fundamental difference being that the features collected for dynamic malware detection are usually observing the activity for an entire virtual machine, whether that is API calls, network traffic or hardware performance counters. Data collection for a machine may be faster than ascribing the data to the individual processes that created it, which is one reason we have chosen to use machine activity metrics rather than API hooking. API hooking collects the data point first and then ascribes it to a process which costs processing time, and potentially therefore time during which malware is causing damage. Sun et al.’s work [@fastSlow2017] on process classification has indicated that real-time data collection using API hooks can be slow and this paper seeks to minimise data collection and inference time to limit the potential damage inflicted by malicious processes.

The data collected are numeric machine activity metrics. These continuous data can represent a wider range of states in fewer features than discrete events such as API calls are able to and are collected every second as the process executes. 26 features are used, these were dictated by the attributes available using the Psutil [@psutil] python library, which enables fast data collection inspecting each running process. The machine activity metrics collected are listed in Table \[tab:features\]

[ &gt;p[0.1]{}| &gt;p[0.1]{}| &gt;p[0.1]{}| &gt;p[0.1]{}]{}

**Category**\
**CPU use** (%) & system level & user level\
**Memory use** (bytes) & total & physical (non-swapped) & swap\
**Child processes** & count & maximum process ID & number of threads\
**I/O operation bytes on disk** (bytes) & read & write & non-read-write I/O operations\
**I/O operation count on disk** & read & write & non-read-write I/O operations\
**Priority** & process priority & I/O process priority\
**Misc.** & number of command line arguments passed to process & number of handles being used by process\
**Time** (seconds) & Time since process start\
**Network Packets** & TCP packet count & UDP packet count\
**Network Other** & number of connections currently open & statuses of the ports opened by the process (4 statuses)

Feature normalisation is necessary for RNNs to avoid over-weighting features with higher absolute values. The test, train and validation sets ($x$) are all normalised by subtracting the mean ($\mu$) and dividing by the standard deviation ($\sigma$) of each feature in the training set: $\frac{x - \mu}{\sigma}$. This sets the range of input values largely between -1 and 1 for all input features, avoiding the potential for some features to be weighted more important than others during training purely due to the scalar values of those features.

Environment
-----------

The environment describes the machine used to test the proposed model. The following experiments were conducted using a virtual machine (VM) running with Cuckoo Sandbox [@cuckooSandbox] for ease of collecting data and restarting between experiments. To emulate the capabilities of a typical machine, we took the modal hardware attributes of the top 10 “best seller" laptops according to a popular internet vendor [@amazon], and used these attributes to set up the machine. This resulted in a VM with 4GB RAM, 128GB Storage and dual-core processing running Windows 7 64-bit. Though new PC laptops are typically packaged with Windows 10, Windows 7 is still the most widely used operating system globally [@win7].

In typical machine use, multiple applications run simultaneously. This is not reflected by behavioural malware analysis research in which which samples are injected individually to a virtual machine for observation. The environment used for the following experiments launches multiple applications on the same machine at slightly staggered intervals as if a user were opening them. Each malware is launched with a manageable number (1-3) and a larger number (3-35) of applications. Most of these applications will be benign but some malware are randomly chosen to run simultaneously with the benignware. Each application may launch multiple processes, causing more than 35 processes to run at once. From the existing run-time analysis literature only Sun et al. [@fastSlow2017] run multiple applications at the same time, with a maximum of 5 running simultaneously. We could not find up to date user data on the number of simultaneous applications running on a typical desktop, so have chosen to launch up to 36 applications (35 benign + 1 malicious) at once, which is the largest number of simultaneous apps for run-time data collection to date.

Evaluation metrics {#metrics}
------------------

The evaluation metrics used for machine learning models that do not interact with their environment are not appropriate for a model which kills processes and thus alters its environment. Rather than averaging predictions over the entire execution trace, it is preferable that a benign process is never labelled malicious (thus avoiding it being killed). Malicious processes should be killed and killed as early as possible to reduce the opportunities for damage to the endpoint; for example scenario B (Fig. \[fig:early\_killing\]) is preferable to scenario A (Fig. \[fig:late\_killing\]).

The malware evaluation metric cascades positive detections to child processes at the time they are applied and the success of the model is determined by the shortest malware execution time per process (parent and child processes) prevented from executing. In Figures \[fig:late\_killing\] and  \[fig:early\_killing\] if malicious process A would execute for 120 seconds undisturbed, B for 30 and C for 20 seconds, scenario A gives a FNR over time of $\frac{5 + 4 + 4}{120 + 30 + 20} = 7.65\%$ whilst Scenario B gives a FNR over time of $\frac{1 + 0 + 0}{120 + 30 + 20} = 0.58\%$.

The benignware evaluation metric is measured by the lowest false positive rate as it is not necessarily preferable to kill a benign program after 10 minutes than after 1 second depending on what the user is doing.

The metric used to capture both error in malicious and benign detection is the mean of the false positive rate (FPR) and the false negative rate over time (FNR over time). The mean FPR, FNR over time will be denoted by (FPR + FNR over time) / 2.

Experimental Results {#experiments}
====================

Dataset
-------

The software samples used in this paper are all Windows portable executables. These are the most commonly seen file types submitted to VirusTotal [@vt_stats], a free malware analysis service. We use 3,603 benign and 2,876 malicious samples, with 3,313 for training and validation and 3,946 for testing, which is consistent with previous run-time detection dataset sizes ([@das2016semantics] uses 168 malicious, 370 benign; [@sayadi2018ensemble] uses over 100 each benign and malicious; [@hardwareWindows] uses 1,087 malicious and 467 benign; [@fastSlow2017] uses 9,115 malicious, 877 benign).

Overall there are 57,981 behavioural traces of processes in the training, testing and validation datasets. The benign samples comprise files from VirusTotal [@virusTotal], from free software websites (later verified as benign with VirusTotal), and from a fresh Windows 7 installation. The malicious samples were collected from two different VirusShare [@VirusShare] repositories. The dataset is split in half with the malicious samples in the test set coming from the more recent VirusShare repository, and those in the training set from the earlier repository. This is to increase the chances of simulating a real deployment scenario in which the malware tested contain new functionality by comparison with those in the training set.

The training and validation sets are created from the same set of samples, but the sets of behavioural traces in the benign and validation sets are mutually exclusive. The test set is generated from a distinct set of benign and malicious applications.

In practice a user will launch the same benign application again and again. The proportion of benign applications which are completely new to the system will only represent a small proportion of all benign applications executed due to the repeated use of installed applications (e.g. browsers, email clients, word processors). Though the same applications are used again and again, their behavioural traces may be slightly different, such that a detection model may have seen a behavioural trace for application ‘X’ before but has not seen it behave in the specific way it is acting now. This is analogous to a model which has learned how to recognise objects from images, but it may not have seen a learned object from a particular perspective or angle before.

The validation set represents this group of use cases. To generate the validation set, samples from the training set are run again to generate new unseen behavioural traces. There is increased likelihood that benignware executing will have been seen before but that the malware will be new as known malware samples are filtered out by signature-based anti-virus once they are known to the malware community. This makes the test set malware detection rate and the validation set benignware detection rate important metrics.

Data collection used the Psutil [@psutil] python library to collect machine activity data for running processes and to kill those processes deemed malicious. The RNN and Random Forests were implemented using the Pytorch [@paszke2017automatic] and Scikit-Learn [@scikit-learn] python libraries respectively. The model runs with high priority to make sure the polling is maintained when compute resources are scarce.

Offline detection of malicious processes {#offline}
----------------------------------------

Though this paper is concerned with online detection, here we provide the results for offline detection of malicious processes using the same train, validation and test datasets as a benchmark.

  Hyperparamter    Possible values
  ---------------- --------------------------------
  hidden neurons   \[50-5000\]
  depth            \[1, 2, 3\]
  batch size       [64, 128, 256]{}
  epochs           \[1 - 200\]
  dropout rate     \[0, 0.1, 0.2, 0.3, 0.4, 0.5\]
  window size      \[1 - 30\]

  : Hyperparameter search space[]{data-label="tab:hyperparameters"}

Recurrent neural networks, like any neural network, can be highly sensitive to the hyperparameters selected prior to training. These include the number of neurons (or GRU cells in this case) in the hidden layers and the number of hidden layers. The selection of hyperparamters may be conducted manually but it can also be automated, which is preferred if human resources are not available and if the model will be retrained to incorporate information about new malware in the future. Based on Bergstra and Bengio’s [@bergstra2012random] findings that a random search of the possible set of parameters tends to find higher-accuracy models more quickly by comparison to grid-based searches, a random search was used to find the optimal set of hyperparameters from the search space listed in Table \[tab:hyperparameters\].

The hyperparameters in table \[tab:hyperparameters\] are as follows: hidden neurons refers to the number of GRU cells in each hidden layer; the depth refers to the number of hidden layers in the neural network; the batch size indicates the number of samples predicted by the model between weight updates and the epochs designate the number of times the model will be exposed to the entire dataset; the dropout rate is a regularisation technique first proposed by Srivastava et al. [@srivastava2014dropout] to reduce the chance of model overfitting, the rate is the proportion of neurons randomly turned “off" (set to an activation of 0) during training; the window size gives the length of historical data for each process visible to the model. For example a window size of 5 will allow the model to see the current set of features and the features at the 4 previous time-steps. The data used here is captured every second such that a window size of 5 corresponds to the model being able to see the machine activity metrics from 4, 3, 2, and 1 seconds ago as well as the current metrics. For reproducibility, the loss function used to train the model was mean squared error and the weight update algorithm was Adam [@adam].

  Hyperparamter        Value
  -------------------- --------------------
  hidden neurons       3076
  depth                1
  batch size           256
  epochs               127
  dropout rate         0
  window size          12
  loss function        mean squared error
  weight update rule   Adam [@adam]
  recurrent unit       GRU cell

  : Hyperparameters of the model giving the lowest mean false-positive and false-negative rates[]{data-label="tab:best_hyperparams"}

The model is trained using a random selection of 50,000 malicious and 50,000 benign windows of process activity. The validation set is highly imbalanced due to the intentional setup of the experiment with 1 - 35 benign processes running with just 1 - 2 malicious processes running to mimic the likely proportions of benign and malicious software running on a machine where the majority of applications will be benign. Due to the unbalanced validation set, here the best model is the one that has the lowest mean false positive and false negative rate. The hyperparameters of the best-performing model are detailed in Table \[tab:best\_hyperparams\]. The model prediction for a given process is determined by taking the mean prediction across all windows. The results on the training, validation and test sets are in Table \[tab:offline\_results\].

  Dataset        Acc. (%)   FPR (%)   FNR (%) 
  ------------ ---------- --------- --------- --
  Training          98.01      2.19      1.36 
  Validation        88.55      8.72     31.31 
  Test              77.58     24.18     16.81 

  : Accuracy (Acc.), false positive (FPR) and false negative rates (FNR) for the training, validation and test sets using the offline model[]{data-label="tab:offline_results"}

These results indicate the challenge of detecting individual malicious processes by comparison with malicious applications overall. Our previous work used the same set of samples and was able to achieve 96.01% detection accuracy on the test set with 3.17% and 4.72% false positive and false negative rates respectively using a recurrent neural network by observing the overall behaviour of the virtual machine when each sample ran independently. Run-time malware process detection cannot observe the overall impact of an application on a machine as it is only able to see the partial execution trace up to that point in time and possibly only a part of the overall application behaviour if multiple processes comprise the application.

Online detection with automatic process killing {#online}
-----------------------------------------------

Online detection of malicious processes is only useful if it can prevent the damage caused by malware. Some malware will begin causing damage to an endpoint almost immediately and in that case automatic responses present the quickest and most reliable way to prevent damage. Prevention is preferable to an alert after the event because saves the resources necessary to recover from an attack. A human could respond to run-time alerts but this would require a human to be present and to be able to react quickly to mitigate damage. Human-in-the loop systems are vulnerable to flooding attacks which can slow down response times significantly and thus potentially allow more damage to take place in the delay.

  Dataset                                                
  ------------------------------ ----------- ----------- ----------
                                    Acc. (%)     FPR (%)    FNR (%)
  Offline                          **88.55**    **8.72**      31.31
  Offline model applied online         48.87       53.16   **3.95**
                                                         
                                    Acc. (%)     FPR (%)    FNR (%)
  Offline                          **77.58**   **24.18**      16.81
  Offline model applied online         38.72       72.00   **2.56**

  : Change in accuracy (Acc.), false postive rate (FPR) and false negative rate (FNR) on validation and test datasets used for offline detection for online detection with automatic process killing[]{data-label="tab:online_vs_offline"}

Table \[tab:online\_vs\_offline\] shows the changes in false positive and false negative rates in moving from an offline to an online model. There is a dramatic increase in false positives and a fall in false negatives, and this result is not surprising. As detailed previously in Section \[modified\_loss\], the mean squared error loss function is trying to minimise the number of wrong predictions over the entire training set and does not discriminate between false positives and false negatives even though a false positive results in a process being terminated permanently, whilst a false negative gives the model the chance to reclassify the process at the next time-step.

The modified loss function introduced in Section \[modified\_loss\] is used to train a new model with the goal to reduce the false positive rate. Using the modified loss function, another random search of hyperparameters gives the best model for the new loss function. The best model is the one that has the lowest mean score for false positive rate and the quickest true positive decisions made, as detailed above in Section \[metrics\]. This is necessary in case the new model finds a longer or shorter window of data helpful for example. The hyperparamters of the new RNN using the modified loss function are given in table \[tab:modified\_hyperparameters\], presented together with the offline model values for ease of comparison. Other than the loss function, 3 of 6 hyperparameters remain unchanged. The number of hidden neurons has reduced by more than one third and the window size has quartered. The online model also took far fewer epochs to train.

[ &gt;p[0.1]{}| &gt;p[0.1]{}| &gt;p[0.2]{} ]{} Hyperparamter & Offline model value & Online model value\
hidden neurons & 3076 & 2021\
depth & 1 & 1\
batch size & 256 & 256\
epochs & 127 & 24\
dropout rate & 0 & 0\
window size & 12 & 3\
loss function & Mean sq. error & Modified loss function\
weight update rule & Adam [@adam] & Adam [@adam]\
recurrent unit & GRU cell & GRU cell\

The decision threshold for deeming a process malicious is a value between 0 (benign) and 1 (malicious), with a default value of 0.5. This threshold can be adjusted to further reduce the chances of a false positive classification. Figure \[fig:threshold\] shows how the false positive rate and the false negative rate over time are impacted by altering the threshold value between 0.5 and 1.

[ &gt;p[0.08]{}| &gt;p[0.06]{}| &gt;p[0.06]{}| &gt;p[0.06]{}| &gt;p[0.08]{}]{} Dataset & Acc. (%) & FPR (%) & FNR over time (%) & (FPR + FNR over time) / 2\
Validation & 93.62 & 4.23 & 21.59 & 13.21\
Test & 69.93 & 28.12 & 33.82 & 32.15\

The lowest mean false positive rate and false negative rate across time used a threshold of 0.98. As Figure \[fig:threshold\] illustrates, using the adjusted loss function gives a lower false positive rate at all time-steps. Table \[tab:online\_results\] reports the accuracy metrics for the validation and test sets. The modified loss function RNN with a threshold adjustment has seen a reduction of false positives from 72.00% of all unseen benign samples to 28.12% percent. This still represents a high false positive rate for unseen benign applications. Distilling the RNN into a secondary (Random Forest) model is intended to both reduce the computational costs of running the RNN as well as emphasising the gravity of false positives.

Distilled real-time detection and automatic process killing {#distillation}
-----------------------------------------------------------

Distilling a machine learning model into another model is typically used to reduce the computational complexity during inference, but can sometimes lead to improvements in accuracy beyond the original model [@hinton2015distilling].

Recurrent neural networks are computationally intensive as the number of neurons in a hidden layer are scaled by the number of time-steps used by the model. The online model proposed in the previous Section (\[online\]) makes 18,190 calculations per prediction during inference. Beyond the computational requirements of the RNN there are other steps that delay inference or occupy more memory than the raw machine activity metrics being collected from each process. Carlin et al. [@carlin2019cost] found that a Random Forest gave the best trade off between accuracy and cost of inference. The following table (Table \[tab:rnn\_drawbacks\]) outlines the additional processing and memory requirements required for an RNN by comparison with a Random Forest. Each of the processing requirements in Table \[tab:rnn\_drawbacks\] takes some time. In this paper we are concerned with fast-acting malware and quick responses such that time delays should be minimised.

[ &gt;p[0.2]{}| &gt;p[0.2]{} ]{} **RNN requires for inference but Random Forest does not** & **Memory or processing cost**\

The RNN uses a sequence equivalent to the window size must be stored for each process this requires tracking each process in order to align data at each time-step & memory requirement scales linearly with window size; processing cost to align new data with existing data\

RNN data must be normalised using the mean and standard deviations of the training data & memory requirement to store training data means and standard deviations (2 \* number of features \* bytes per datum); processing requirement O(number of features) to calculate normalised values\

RNN outputs a value between 0 and 1 which must be converted into a decision, but the Random Forest outputs 0 or 1 as default & very small processing cost of O(number of samples in batch)

[ &gt;p[0.06]{}| &gt;p[0.05]{}| &gt;p[0.14]{}| &gt;p[0.09]{}| &gt;p[0.09]{}| &gt;p[0.09]{}| &gt;p[0.09]{}| &gt;p[0.09]{}| &gt;p[0.09]{} ]{} Dataset & Model \# & Model & Acc. (%) & FPR (%) & FNR (%) & FPR across time (%) & FNR across time (%) & (FPR + FNR across time) / 2\
Val. & 1 & Offline model ($\theta$=0.5) & 48.87 & 53.16 & **3.95** & 56.86 & **4.67** & 30.4\
Val. & 2 & Offline model (best $\theta$=0.99) & 78.19 & 22.42 & 14.47 & 22.84 & 13.5 & 18.66\
Val. & 3 & Online model ($\theta$=0.5) & 55.13 & 44.4 & 10.13 & 49.41 & 7.16 & 29.77\
Val. & 4 & Online model (best $\theta$=0.98) & **93.62** & 4.23 & 21.87 & 4.55 & 21.59 & **13.21**\
Val. & 5 & Distilled RF model & 87.81 & **2.59** & 92.62 & **2.93** & 89.63 & 47.78\
Val. & 6 & RF trained directly from data & 87.15 & 12.32 & 26.23 & 10.91 & 29.02 & 18.57\

Test & 1 & Offline model ($\theta$=0.5) & 38.72 & 72.0 & 2.56 & 74.21 & 3.18 & 38.38\
Test & 2 & Offline model (best $\theta$=0.99) & 61.2 & 43.28 & 10.69 & 44.4 & 13.66 & **27.54**\
Test & 3 & Online model ($\theta$=0.5) & 39.39 & 70.27 & **3.28** & 73.31 & **3.46**s & 38.3\
Test & 4 & Online model (best $\theta$=0.98) & 69.93 & 28.12 & 35.06 & 29.24 & 33.82 & 32.15\
Test & 5 & Distilled RF model & **84.35** & **4.76** & 71.97 & **5.11** & 62.9 & 38.54\
Test & 6 & RF trained directly from data & 70.08 & 30.89 & 24.15 & 31.09 & 24.66 & 27.62\

Table \[tab:offline\_online\_RF\_accuracies\] details the accuracy scores for the three models presented in this paper:

-   the offline model trained with a normal (mean-squared error) loss function using a decision threshold of 0.5 and applied to online detection with automatic process killing

-   the online model trained with the modified loss function using a threshold selected for optimal performance on the validation set

-   the distilled random forest trained from the online model (Model 4)

For comparison we have added three additional models:

1.  the offline model trained with an adjusted decision threshold selected for optimal performance on the validation set in online detection with process killing

2.  the online model trained with a modified loss function using a threshold of 0.5

3.  a Random Forest model trained directly from the training data

Model 4 (online model) departs from model 1 (offline model) in two respects: the loss function and the decision threshold. Models 2 and 4 are included to illustrate the separate impacts of these two changes. Both the online and offline models see a large fall in the false positive rate when we allow the “optimal" decision thresholds to be used (0.99 and 0.98 respectively), these thresholds being determined be the mean FPR and FNR over time. On both the test and validation sets, however, Model 4 (online model with adjusted threshold + modified loss function) achieves the lowest FPR indicating that the threshold alone does not account for the fall in FPR.

Model 6 is included to show that we cannot achieve the same results as by simply training a Random Forest in the first place from the training data. The distilled random forest (Model 5) maintains a low false positive rate on completely unseen benign applications of 4.76% compared with 30.89% for Model 6. Even though Model 6 has a lower mean FPR and FNR over time, but 30% of new benign application’s processes and 12% of known benign application’s processes are terminated automatically.

The distilled model does not achieve the lowest (FPR + FNR over time) / 2 but it is the only model to kill less than 25% of the unseen benign processes, its FPR being less than 5%. The distilled model kills 30% of unknown malicious applications. Whilst this figure is low, it is still a significant improvement on a random guess as the model simultaneously maintains a low false positive rate of 4.76%. Due to the lack of labels for individual processes we cannot know whether those processes killed are the ones causing the most damage. Malware may invoke multiple processes in order to achieve its goals but some of those may not be harmful to the user such as processes which conduct reconnaissance to collect non-sensitive information. 30% may be sufficient to limit the damage to the endpoint.

To evaluate the tangible impact for a user, the next Section (\[ransomware\_case\_study\]) conducts a live real-time experiment to see the damage caused with and without the random forest model running.

Case Study: Ransomware damage {#ransomware_case_study}
-----------------------------

The labels provided for the malicious and benign samples in this dataset are denoted at the application level. A single application may invoke tens of processes and without considerable manual reverse engineering effort, it is not possible to determine which processes are causing damage and which are not. Although we can say with confidence that benign processes should not be killed and the processes created by malicious applications can be killed without causing damage, we do not know which malicious processes are causing the (most) damage. It may be that malware runs a number of processes for data exfiltration, but some of these could be gathering data whilst one is responsible for sending it back to the adversary. As long as the latter process is killed, the malware’s capacity to cause damage is limited.

Ransomware is a the broad term given to malware that prevents access to user data (often by encrypting files) and holds the means for retrieving the data (usually a decryption key) from the user until a ransom is paid. It is possible to quantify the damage caused by ransomware using the proportion of modified files as Scaife et al. [@scaife2016cryptolock] have done in developing a ransomware detection system.

From 2000 VirusShare ransomware portable executables it was possible to identify 158 which begin modifying files within the first 30 seconds of execution and do not require internet connection to do so. Some may argue that malware will commonly wait for instructions from a C2 server before carrying out the malicious payload but these 158 samples indicate that malware exists which will begin causing problems for the used within seconds of being launched without any such instruction. These are the types of malware that the proposed model could have a significant impact on.

The 158 samples were executed for 60 seconds each without the process killing model in place 3 times and the average number of files modified recorded. The process killing Random Forest model was then used to detect and kill malicious processes. Across the 158 samples, the number of modified files was reduced from 11,680 to 1,595; equivalent to a reduction of 86.34% fewer files being compromised.

Discussion and Future Work {#future_work}
==========================

Behavioural malware analysis research using machine learning regularly reports $>$95% classification accuracy. Though useful for analysts, behavioural detection should be deployed as part of endpoint defensive systems to leverage the full benefits of a detection model. Dynamic analysis is not typically used for endpoint protection because it takes too long in data collection to deliver the quick verdicts required for good user experience. Run-time detection on the endpoint allows for observation of the full trace without the user having to wait. However, run-time detection also introduces the risk that malware will cause damage to the endpoint. This risk requires that processes detected as malicious are automatically killed as early as possible to avoid harm.

The distilled Random Forest model presented in this paper correctly identified 95.24% of unseen benign applications and was able to have a tangible impact on the damage caused to a user by fast-acting ransomware, preventing 86.34% of the file corruption that took place without the model running.

These results are encouraging but the false negative rate remains high with only unseen 30% of malicious processes being detected. Killing 30% of processes may be sufficient to prevent the malicious payload being executed in many cases but we cannot know this without process-level labels. Future work will seek to improve the accuracy of the model using more granular labels and further test the robustness of the model under capacity strain:

-   The precise accuracy of the model is difficult to calculate without **process-wise labelling of subprocesses**. Future work could train the model with individually labelled processes. This is likely to improve accuracy results and give a better indication of how much damage is being prevented.

-   In these experiments, up to 36 applications were running at once with a maximum of 95 processes being measured and classified simultaneously. Future experiments should test the **robustness and dependability** of this model to discover if there is a capacity limit after which accuracy severely diminishes. Running an even higher number of processes may, for example, slow down the data collection so much that the malware is able to inflict more damage during the processing time.

Conclusions {#conclusion}
===========

Behavioural malware detection is a well-established research field with little translation into models for run-time behavioural detection, though run-time detection enables us to move from analysis post-attack to prevention of damage caused by malware. In this paper we have demonstrated that using a modified loss function and decision threshold to train a recurrent neural network, the increased likelihood of false positives for automatic process killing can be mitigated.

The distilled model presented in this paper captured adversity to false positives whilst demonstrating a tangible benefit to users in the case of fast-acting ransomware, reducing the number of files encrypted by 86.34%.

The distilled model does not yet achieve the detection accuracies of state of the art offline behavioural analysis models, but these models typically use the full post-execution trace of malicious behaviour. Delaying classification until post-execution negates the principal advantages of run-time detection. However, the proposed model presents an initial step towards a fully automated endpoint protection model which becomes increasingly necessary as adversaries become more and more motivated to evade offline automated detection tools.

[^1]: This research was funded by EPSRC and Airbus through iCASE studentship EP/P510452/1 project \#1852525
