commit message classification
230995090
Multi label Classification of Commit Messages using Transfer Learning
Commit messages are used in the industry by developers to annotate changes made to the code. Accurate classification of these messages can help monitor the software evolution process and enable better tracking for various industrial stakeholders. In this paper, we present a state of the art method for commit message classification into categories as per Swanson's maintenance activities i.e. "Corrective" "Perfective" and "Adaptive" This is a challenging task because not all commit messages are well written and informative. Existing approaches rely on keyword based techniques to solve this problem. However, these approaches are oblivious to the full language model and do not recognize the contextual relationship between words. State of the art methodology in Natural Language Processing (NLP) is to train a context aware neural network (Transformer) on a very large data set that encompasses the entire language and then fine tunes it for a specific task. In this way, the model can learn the language, pay attention to the context, and then transfer that knowledge for better performance at the specific task. We use an off the shelf neural network called DistilBERT and fine tune it for commit message classification task. This step is non trivial because programming languages and commit messages have unique keywords, jargon, and idioms. This paper presents our effort in training this model and constructing the data set for this task. We describe the rules used to construct the data set. We validate our approach on industrial projects from GitHub, such as Kubernetes, Linux, TensorFlow, Spark, TypeScript, and PyTorch. We were able to achieve 87% F1 score for the commit message classification task, which is an order of magnitude accurate than previous studies.
2020 IEEE International Symposium on Software Reliability Engineering Workshops (ISSREW)
Muhammad Usman Sarwar Sarim Zafar Mohamed Wiem Mkaouer Gursimran Singh Walia Muhammad Zubair Malik
229545423
Commit Classification using Natural Language Processing: Experiments over Labeled Datasets
Software commits play an important role in collaborative software development. A commit allows developers to collaboratively contribute to open source software projects. Previous studies show that developers spend a lot of time in identifying and prioritizing critical issues and respective commits. For this reason, understanding what a software commit is trying to solve in the project is very important. In this context, commit classification could help managers to plan and allocate resources in advance for the software project. Despite its relevance, we still lack accurate models to support automated classification of commits into relevant categories. In this work, we conduct an empirical study to evaluate commit classification techniques using the commit message given by developers as input. After an ad hoc literature review, we created a dataset based on labeled data from different works found in the literature. Second, we compared five Machine Learning (ML) state or art algorithms seeking a reliable baseline model. Finally, we applied Natural Language Processing (NLP) to a novel ML model aiming at improving results compared to the baseline models. Our findings show that the new NLP based model outperforms the random baseline models by a relevant rate, achieving 91% of F measure for the commit classification task.
CIbSE
Geanderson Esteves dos Santos Eduardo Figueiredo
233690496
Augmenting commit classification by using fine grained source code changes and a pre trained deep neural language model
Abstract Context: Analyzing software maintenance activities is very helpful in ensuring cost effective evolution and development activities. The categorization of commits into maintenance tasks supports practitioners in making decisions about resource allocation and managing technical debt. Objective: In this paper, we propose to use a pre trained language neural model, namely BERT (Bidirectional Encoder Representations from Transformers) for the classification of commits into three categories of maintenance tasks corrective, perfective and adaptive. The proposed commit classification approach will help the classifier better understand the context of each word in the commit message. Methods: We built a balanced dataset of 1793 labeled commits that we collected from publicly available datasets. We used several popular code change distillers to extract fine grained code changes that we have incorporated into our dataset as additional features to BERT's word representation features. In our study, a deep neural network (DNN) classifier has been used as an additional layer to fine tune the BERT model on the task of commit classification. Several models have been evaluated to come up with a deep analysis of the impact of code changes on the classification performance of each commit category. Results and conclusions: Experimental results have shown that the DNN model trained on BERT's word representations and Fixminer code changes (DNN@BERT+Fix_cc) provided the best performance and achieved 79.66% accuracy and a macro average f1 score of 0.8. Comparison with the state of the art model that combines keywords and code changes (RF@KW+CD_cc) has shown that our model achieved approximately 8% improvement in accuracy. Results have also shown that a DNN model using only BERT's word representation features achieved an improvement of 5% in accuracy compared to the RF@KW+CD_cc model.
Inf. Softw. Technol.
Lobna Ghadhab Ilyes Jenhani Mohamed Wiem Mkaouer Montassar Ben Messaoud
20749331
Tracing Your Maintenance Work A Cross Project Validation of an Automated Classification Dictionary for Commit Messages
A commit message is a description of a change in a Version Control System (VCS) Besides the actual description of the change, it can also serve as an indicator for the purpose of the change, e.g. a change to refactor code might be accompanied by a commit message in the form of "Refactored class XY to improve readability" We would label the change in our example a perfective change, according to maintenance literature. This simplified example shows how it is possible to classify a change by its commit message. However, commit messages are unstructured, textual data and efforts to automatically label changes into categories like perfective have only been applied to a small set of projects within the same company or the same community. In this work, we present a cross project evaluated and valid mapping of changes to the code base and their purpose that is usable without any customization on any open source project. We provide further the Eclipse Plug In Subcat which allows for a comfortable analysis of projects from within Eclipse. By using Subcat, we are able to automatically assess if a commit to the code was e.g. a bug fix or a refactoring. This information is very useful for e.g. developer profiling or locating bad smells in modules.
FASE
Andreas Mauczka Markus Huber Christian Schanes Wolfgang Schramm Mario Bernhart Thomas Grechenig
231684268
Phishing Attack Detection using Machine Learning Classification Techniques
Phishing attacks are the most common form of attacks that can happen over the internet. This method involves attackers attempting to collect data of a user without his/her consent through emails, URLs, and any other link that leads to a deceptive page where a user is persuaded to commit specific actions that can lead to the successful completion of an attack. These attacks can allow an attacker to collect vital information of the user that can often allow the attacker to impersonate the victim and get things done that only the victim should have been able to do, such as carry out transactions, or message someone else, or simply accessing the victim's data. Many studies have been carried out to discuss possible approaches to prevent such attacks. This research work includes three machine learning algorithms to predict any websites' phishing status. In the experimentation these models are trained using URL based features and attempted to prevent Zero Day attacks by using proposed software proposal that differentiates the legitimate websites and phishing websites by analyzing the website's URL. From observations, the random forest classifier performed with a precision of 97% a recall 99% and F1 Score is 97% Proposed model is fast and efficient as it only works based on the URL and it does not use other resources for analysis, as was the case for past studies.
2020 3rd International Conference on Intelligent Sustainable Systems (ICISS)
Noor Faisal Abedin Rose Mary Bawm Tawsif Sarwar Mohammed Saifuddin Mohammd Azizur Rahman Sohrab Hossain
204817788
Towards Standardizing and Improving Classification of Bug Fix Commits
Background: Open source software repositories like GitHub are mined to gain useful empirical software engineering insights and answer critical research questions. However, the present state of the art mining approaches suffers from high error rate in the labeling of data that is used for such analysis. This is particularly true when labels are automatically generated from the commit message, and seriously undermines the results of these studies. Aim: Our goal is to label commit comments with high accuracy automatically. In this work, we focus on classifying a commit as a "Bug Fix commit" or not. Method: Traditionally, researchers have utilized keyword based approaches to identify bug fix commits that leads to a significant increase in the error rate. We present an alternative methodology leveraging a deep neural network model called Bidirectional Encoder Representations from Transformers (BERT) that can understand the context of the commit message. We provide the rules for semantic interpretation of commit comments. We construct a hand labeled dataset from real GitHub commits according to these rules and fine tune BERT for classification. Results: Our initial evaluation shows that our approach significantly reduces the error rate, with up to 10% relative improvement in classification over keyword based approaches. Future Direction: We plan on extending our dataset to cover more corner cases and reduce programming language specific biases. We also plan on refining the semantic rules. In this work, we have only considered a simple binary classification problem (Bug Fix or not) which we plan to extend to other classes and extend the approach to consider multiclass problems. Conclusion: The rules, data, and the model proposed in this paper have the potential to be used by people analyzing open source repositories to improve the labeling of data used in their analysis.
2019 ACM/IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM)
Sarim Zafar Muhammad Zubair Malik Gursimran Singh Walia
142503740
On the classification of software change messages using multi label active learning
In this paper, we present a multi label active learning based approach to handle the problem of classification of commit messages. The approach will help developers track software changes, e.g. adding or updating existing features, fixing user reported errors, improving software performance, etc. We first constructed an unlabeled dataset of commit messages where each commit message is represented as a vector of feature values. The set of adopted features were automatically generated from the original commit messages using Term Frequency Inverse Document Frequency (TF IDF) technique. Because many commit messages can be assigned more than one commit class at the same time and in order to reduce the effort needed to assign labels to each instance in a large set of commit messages, we adopted an Active Learning multi label approach. Experimentations have shown that we could train an accurate multi label classifier model, in our case, a binary relevance with logistic regression as a base classifier, by actively querying an oracle for labels during the training process and with a reasonable number of labeled instances.
SAC
Sirine Gharbi Mohamed Wiem Mkaouer Ilyes Jenhani Montassar Ben Messaoud
53283934
Linking Source Code to Untangled Change Intents
Previous work [13] suggests that tangled changes (i.e. different change intents aggregated in one single commit message) could complicate tracing to different change tasks when developers manage software changes. Identifying links from changed source code to untangled change intents could help developers solve this problem. Manually identifying such links requires lots of experience and review efforts, however. Unfortunately, there is no automatic method that provides this capability. In this paper, we propose AutoCILink, which automatically identifies code to untangled change intent links with a pattern based link identification system (AutoCILink P) and a supervised learning based link classification system (AutoCILink ML) Evaluation results demonstrate the effectiveness of both systems: the pattern based AutoCILink P and the supervised learning based AutoCILink ML achieve average accuracy of 74.6% and 81.2% respectively.
2018 IEEE International Conference on Software Maintenance and Evolution (ICSME)
Xiaoyu Liu LiGuo Huang Chuanyin Liu Vincent Ng
6992114
Automatic classication of large changes into maintenance categories
Large software systems undergo significant evolution during their lifespan, yet often individual changes are not well documented. In this work, we seek to automatically classify large changes into various categories of maintenance tasks corrective, adaptive, perfective, feature addition, and non functional improvement using machine learning techniques. In a previous paper, we found that many commits could be classified easily and reliably based solely on the manual analysis of the commit metadata and commit messages (i.e. without reference to the source code) Our extension is the automation of classification by training Machine Learners on features extracted from the commit metadata, such as the word distribution of a commit message, commit author, and modules modified. We validated the results of the learners via 10 fold cross validation, which achieved accuracies consistently above 50% indicating good to fair results. We found that the identity of the author of a commit provided much information about the maintenance class of a commit, almost as much as the words of the commit message. This implies that for most large commits, the Source Control System (SCS) commit messages plus the commit author identity is enough information to accurately and automatically categorize the nature of the maintenance task.
2009 IEEE 17th International Conference on Program Comprehension
Abram Hindle Daniel M German Michael W Godfrey Richard C Holt
56347614
IDENTIFY A PLURALITY OF COMPUTING NODESEACH COMPRISING A SET OF DATA ASSOCATED WITH A TRANSACTION OBTAINA SET OF LOCAL TRANSACTIONSEQUENCE INFORMATION FROMEACH OF THE PLURALITY OF NODES GENERATE ATRANSACTION EXECUTIONSEQUENCE FOR THE TRANSACTION BASED ONTHE OBTAINED SET OF LOCAL TRANSACTIONSEQUENCE
(72) Inventors: Xavier R. GUERIN, White Plains, NY In various embodiments a distributed computing node in a (US) Shicong MENG, Elmsford, NY plurality of distributed computing nodes executes transac (US) tions in a distributed processing system. In one embodiment, a transaction commit message is received from a client com (73) Assignee: International Business Machines puting node for a transaction. The transaction commit mes Corporation, Armonk, NY (US) sage includes at least an identifier of the transaction and a transaction sequence for the transaction. The transaction (21) Appl. No. 14/215,428 sequence indicates a sequence of execution for the transaction (22) Filed: Mar 17, 2014 on the plurality of distributed computing nodes. An entry within the transaction sequence associated with the distrib Publication Classification uted computing node is identified. The entry includes a sequence number for executing the transaction on the distrib (51) Int. Cl. uted computing node with respect to other transactions. The G06F 9/46 (2006.01) transaction is executed based on the sequence number in the G06F 7/30 (2006.01) entry.