model extraction
204907203
Thieves on Sesame Street! Model Extraction of BERT based APIs
We study the problem of model extraction in natural language processing, in which an adversary with only query access to a victim model attempts to reconstruct a local copy of that model. Assuming that both the adversary and victim model fine tune a large pretrained language model such as BERT (Devlin et al. 2019) we show that the adversary does not need any real training data to successfully mount the attack. In fact, the attacker need not even use grammatical or semantically meaningful queries: we show that random sequences of words coupled with task specific heuristics form effective queries for model extraction on a diverse set of NLP tasks, including natural language inference and question answering. Our work thus highlights an exploit only made feasible by the shift towards transfer learning methods within the NLP community: for a query budget of a few hundred dollars, an attacker can extract a model that performs only slightly worse than the victim model. Finally, we study two defense strategies against model extraction membership classification and API watermarking which while successful against naive adversaries, are ineffective against more sophisticated ones.
ICLR
Kalpesh Krishna Gaurav Singh Tomar Ankur P Parikh Nicolas Papernot Mohit Iyyer
67876983
Exploring Connections Between Active Learning and Model Extraction
Machine learning is being increasingly used by individuals, research institutions, and corporations. This has resulted in the surge of Machine Learning as a Service (MLaaS) cloud services that provide (a) tools and resources to learn the model, and (b) a user friendly query interface to access the model. However, such MLaaS systems raise privacy concerns such as model extraction. In model extraction attacks, adversaries maliciously exploit the query interface to steal the model. More precisely, in a model extraction attack, a good approximation of a sensitive or proprietary model held by the server is extracted (i.e. learned) by a dishonest user who interacts with the server only via the query interface. This attack was introduced by Tramer et al. at the 2016 USENIX Security Symposium, where practical attacks for various models were shown. We believe that better understanding the efficacy of model extraction attacks is paramount to designing secure MLaaS systems. To that end, we take the first step by (a) formalizing model extraction and discussing possible defense strategies, and (b) drawing parallels between model extraction and established area of active learning. In particular, we show that recent advancements in the active learning domain can be used to implement powerful model extraction attacks, and investigate possible defense strategies.
USENIX Security Symposium
Varun Chandrasekaran Kamalika Chaudhuri Irene Giacomelli Somesh Jha Songbai Yan
212687783
DeepSniffer: A DNN Model Extraction Framework Based on Learning Architectural Hints
As deep neural networks (DNNs) continue their reach into a wide range of application domains, the neural network architecture of DNN models becomes an increasingly sensitive subject, due to either intellectual property protection or risks of adversarial attacks. Previous studies explore to leverage architecture level events disposed in hardware platforms to extract the model architecture information. They pose the following limitations: requiring a priori knowledge of victim models, lacking in robustness and generality, or obtaining incomplete information of the victim model architecture. Our paper proposes DeepSniffer, a learning based model extraction framework to obtain the complete model architecture information without any prior knowledge of the victim model. It is robust to architectural and system noises introduced by the complex memory hierarchy and diverse run time system optimizations. The basic idea of DeepSniffer is to learn the relation between extracted architectural hints (e.g. volumes of memory reads/writes obtained by side channel or bus snooping attacks) and model internal architectures. Taking GPU platforms as a show case, DeepSniffer conducts model extraction by learning both the architecture level execution features of kernels and the inter layer temporal association information introduced by the common practice of DNN design. We demonstrate that DeepSniffer works experimentally in the context of an off the shelf Nvidia GPU platform running a variety of DNN models. The extracted models are directly helpful to the attempting of crafting adversarial inputs. Our experimental results show that DeepSniffer achieves a high accuracy of model extraction and thus improves the adversarial attack success rate from 14.6%\sim$25.5% (without network architecture knowledge) to 75.9% (with extracted network architecture) The DeepSniffer project has been released in Github.
ASPLOS
Xing Hu Ling Liang Shuangchen Li Lei Deng Pengfei Zuo Yu Ji Xinfeng Xie Yufei Ding Chang Liu Timothy Sherwood Yuan Xie
73729063
Neural Network Model Extraction Attacks in Edge Devices by Hearing Architectural Hints
As neural networks continue their reach into nearly every aspect of software operations, the details of those networks become an increasingly sensitive subject. Even those that deploy neural networks embedded in physical devices may wish to keep the inner working of their designs hidden either to protect their intellectual property or as a form of protection from adversarial inputs. The specific problem we address is how, through heavy system stack, given noisy and imperfect memory traces, one might reconstruct the neural network architecture including the set of layers employed, their connectivity, and their respective dimension sizes. Considering both the intra layer architecture features and the inter layer temporal association information introduced by the DNN design empirical experience, we draw upon ideas from speech recognition to solve this problem. We show that off chip memory address traces and PCIe events provide ample information to reconstruct such neural network architectures accurately. We are the first to propose such accurate model extraction techniques and demonstrate an end to end attack experimentally in the context of an off the shelf Nvidia GPU platform with full system stack. Results show that the proposed techniques achieve a high reverse engineering accuracy and improve the one's ability to conduct targeted adversarial attack with success rate from 14.6\\sim$25.5\ (without network architecture knowledge) to 75.9\ (with extracted network architecture)
ASPLOS 2020
Xing Hu Ling Liang Lei Deng Shuangchen Li Xinfeng Xie Yu Ji Yufei Ding Chang Liu Timothy Sherwood Yuan Xie
213157375
ActiveThief: Model Extraction Using Active Learning and Unannotated Public Data
Machine learning models are increasingly being deployed in practice. Machine Learning as a Service (MLaaS) providers expose such models to queries by third party developers through application programming interfaces (APIs) Prior work has developed model extraction attacks, in which an attacker extracts an approximation of an MLaaS model by making black box queries to it. We design ActiveThief a model extraction framework for deep neural networks that makes use of active learning techniques and unannotated public datasets to perform model extraction. It does not expect strong domain knowledge or access to annotated data on the part of the attacker. We demonstrate that (1) it is possible to use ActiveThief to extract deep classifiers trained on a variety of datasets from image and text domains, while querying the model with as few as 10 30% of samples from public datasets, (2) the resulting model exhibits a higher transferability success rate of adversarial examples than prior work, and (3) the attack evades detection by the state of the art model extraction detection method, PRADA.
AAAI
Soham Pal Yash Gupta Aditya Shukla Aditya Kanade Shirish K Shevade Vinod Ganapathy
224817690
Now You See Me (CME) Concept based Model Extraction
Deep Neural Networks (DNNs) have achieved remarkable performance on a range of tasks. A key step to further empowering DNN based approaches is improving their explainability. In this work we present CME: a concept based model extraction framework, used for analysing DNN models via concept based extracted models. Using two case studies (dSprites, and Caltech UCSD Birds) we demonstrate how CME can be used to (i) analyse the concept information learned by a DNN model (ii) analyse how a DNN uses this concept information when predicting output labels (iii) identify key concept information that can further improve DNN predictive performance (for one of the case studies, we showed how model accuracy can be improved by over 14% using only 30% of the available concepts)
CIKM
Dmitry Kazhdan B Dimanov Mateja Jamnik Pietro Lio' Adrian Weller
221507717
Model extraction from counterfactual explanations
Post hoc explanation techniques refer to a posteriori methods that can be used to explain how black box machine learning models produce their outcomes. Among post hoc explanation techniques, counterfactual explanations are becoming one of the most popular methods to achieve this objective. In particular, in addition to highlighting the most important features used by the black box model, they provide users with actionable explanations in the form of data instances that would have received a different outcome. Nonetheless, by doing so, they also leak non trivial information about the model itself, which raises privacy issues. In this work, we demonstrate how an adversary can leverage the information provided by counterfactual explanations to build high fidelity and high accuracy model extraction attacks. More precisely, our attack enables the adversary to build a faithful copy of a target model by accessing its counterfactual explanations. The empirical evaluation of the proposed attack on black box models trained on real world datasets demonstrates that they can achieve high fidelity and high accuracy extraction even under low query budgets.
ArXiv
U Aivodji Alexandre Bolot Sebastien Gambs
59337129
An Active Learning Approach for Improving the Accuracy of Automated Domain Model Extraction
Domain models are a useful vehicle for making the interpretation and elaboration of natural language requirements more precise. Advances in natural language processing (NLP) have made it possible to automatically extract from requirements most of the information that is relevant to domain model construction. However, alongside the relevant information, NLP extracts from requirements a significant amount of information that is superfluous (not relevant to the domain model) Our objective in this article is to develop automated assistance for filtering the superfluous information extracted by NLP during domain model extraction. To this end, we devise an active learning based approach that iteratively learns from analysts' feedback over the relevance and superfluousness of the extracted domain model elements and uses this feedback to provide recommendations for filtering superfluous elements. We empirically evaluate our approach over three industrial case studies. Our results indicate that, once trained, our approach automatically detects an average of 45% of the superfluous elements with a precision of 96% Since precision is very high, the automatic recommendations made by our approach are trustworthy. Consequently, analysts can dispose of a considerable fraction nearly half of the superfluous elements with minimal manual work. The results are particularly promising, as they should be considered in light of the non negligible subjectivity that is inherently tied to the notion of relevance.
TSEM
Chetan Arora Mehrdad Sabetzadeh Shiva Nejati Lionel C Briand
202579639
BDPL: A Boundary Differentially Private Layer Against Machine Learning Model Extraction Attacks
Machine learning models trained by large volume of proprietary data and intensive computational resources are valuable assets of their owners, who merchandise these models to third party users through prediction service API. However, existing literature shows that model parameters are vulnerable to extraction attacks which accumulate a large number of prediction queries and their responses to train a replica model. As countermeasures, researchers have proposed to reduce the rich API output, such as hiding the precise confidence level of the prediction response. Nonetheless, even with response being only one bit, an adversary can still exploit fine tuned queries with differential property to infer the decision boundary of the underlying model. In this paper, we propose boundary differential privacy \epsilon BDP) as a solution to protect against such attacks by obfuscating the prediction responses near the decision boundary. \epsilon BDP guarantees an adversary cannot learn the decision boundary by a predefined precision no matter how many queries are issued to the prediction API. We design and prove a perturbation algorithm called boundary randomized response that can achieve \epsilon BDP. The effectiveness and high utility of our solution against model extraction attacks are verified by extensive experiments on both linear and non linear models.
ESORICS
Huadi Zheng Qingqing Ye Haibo Hu Chengfang Fang Jie Shi
21327267
Model Extraction Warning in MLaaS Paradigm
Machine learning models deployed on the cloud are susceptible to several security threats including extraction attacks. Adversaries may abuse a model's prediction API to steal the model thus compromising model confidentiality, privacy of training data, and revenue from future query payments. This work introduces a model extraction monitor that quantifies the extraction status of models by continually observing the API query and response streams of users. We present two novel strategies that measure either the information gain or the coverage of the feature space spanned by user queries to estimate the learning rate of individual and colluding adversaries. Both approaches have low computational overhead and can easily be offered as services to model owners to warn them against state of the art extraction attacks. We demonstrate empirical performance results of these approaches for decision tree and neural network models using open source datasets and BigML MLaaS platform.
ACSAC
Manish Kesarwani Bhaskar Mukhoty V Arya Sameep Mehta