
# pull_request_comments_model

## Model description

This model is a fine-tuned version of [distilbert-base-uncased](https://huggingface.co/distilbert-base-uncased) on a pull request comments dataset.
It achieves the following results on the evaluation set:
- Train Loss: 0.0791
- Train Accuracy: 0.9955
- Validation Loss: 0.5019
- Validation Accuracy: 0.8291
- Epoch: 12

## Training and evaluation data

Training and evaluation data used for this model are the pull request comments of the tensorflow repository on GitHub.
In particular, of all the pull request data (commit comments, review comments, events, exc.) only the rows with Type equal to PC (Pull request Comment) or RC (Review Comment) have been entered into the dataset.
These comments has been classified into 4 categories:
1) ML (Machine Learning), if the comment is about specific machine learning aspects, algorithms exc.
2) Code, if the comment concerns either style and documentation in the code or maintainability issues or possible bugs exc.
3) Management, if the comment is about management activities like checking an activity status, assign a review to someone, trigger Jenkins CI
4) Other, if the comment doesn't belong to any of the above categories

## Intended uses & limitations

One possible use of this model could be to label the pull request comments, clearly only on GitHub repositories that are about Machine Learning.
In this way a developer, before reading a comment entirely, can have a preview of what that comment is about.

### Training hyperparameters

The following hyperparameters were used during training:
- batch_size: 32
- num_epochs: 20
- optimizer: {'name': 'Adam', 'learning_rate': {'class_name': 'PolynomialDecay', 'config': {'initial_learning_rate': 2e-05, 'decay_steps': 280, 'end_learning_rate': 0.0, 'power': 1.0, 'cycle': False, 'name': None}}, 'decay': 0.0, 'beta_1': 0.9, 'beta_2': 0.999, 'epsilon': 1e-08, 'amsgrad': False}
- training_precision: float32

### Training results

| Train Loss | Train Accuracy | Validation Loss | Validation Accuracy | Epoch |
|:----------:|:--------------:|:---------------:|:-------------------:|:-----:|
| 1.3102     | 0.4308         | 1.1851          | 0.4701              | 0     |
| 1.1436     | 0.4978         | 0.9891          | 0.6068              | 1     |
| 0.9590     | 0.6183         | 0.8287          | 0.6838              | 2     |
| 0.7801     | 0.6942         | 0.6916          | 0.7692              | 3     |
| 0.6074     | 0.7946         | 0.6212          | 0.8120              | 4     |
| 0.4755     | 0.8817         | 0.5471          | 0.8205              | 5     |
| 0.3503     | 0.9241         | 0.5244          | 0.8376              | 6     |
| 0.2594     | 0.9665         | 0.5171          | 0.8120              | 7     |
| 0.1711     | 0.9844         | 0.4832          | 0.8291              | 8     |
| 0.1474     | 0.9911         | 0.5000          | 0.8205              | 9     |
| 0.1082     | 0.9955         | 0.4875          | 0.8291              | 10    |
| 0.0981     | 0.9933         | 0.4928          | 0.8291              | 11    |
| 0.0791     | 0.9955         | 0.5019          | 0.8291              | 12    |


### Framework versions

- Transformers 4.26.1
- TensorFlow 2.9.0
- Datasets 2.10.1
- Tokenizers 0.13.2
{"language": "en", "license": "apache-2.0", "tags": ["generated_from_keras_callback"], "widget": [{"text": "Please add a detailed comment explaining what this config parameter means.", "example_title": "Code example"}, {"text": "Because if non-chief workers don\u2019t restore the model from the checkpoint, training can\u2018t continue.  For example, I am using tf.estimator.Estimator API and tf.distribute.CollectiveAllReduceStrategy for distributed training. Firstly, I train for 1000 rounds(train_stpes=1000) and save the checkpoint, it works normally. Then I set the train_steps to 2000, only the is_chief role can restore the model from the checkpoint without any error.", "example_title": "ML example"}, {"text": "Hi, Could you please review this PR when you have some time? Thank you very much!", "example_title": "Management example"}, {"text": "Sorry, i was on holiday all the month and i didn\u2019t have so much time to review this PR. I also had a lot of problems with my computer, so i couldn\u2019t work...", "example_title": "Other example"}], "pipeline_tag": "text-classification", "model-index": [{"name": "pull_request_comments_model", "results": []}]}
transformers tf distilbert text-classification generated_from_keras_callback en license:apache-2.0 autotrain_compatible endpoints_compatible region:us
TAGS
#transformers #tf #distilbert #text-classification #generated_from_keras_callback #en #license-apache-2.0 #autotrain_compatible #endpoints_compatible #region-us 

pull\_request\_comments\_model
==============================


Model description
-----------------


This model is a fine-tuned version of distilbert-base-uncased on a pull request comments dataset.
It achieves the following results on the evaluation set:


* Train Loss: 0.0791
* Train Accuracy: 0.9955
* Validation Loss: 0.5019
* Validation Accuracy: 0.8291
* Epoch: 12


Training and evaluation data
----------------------------


Training and evaluation data used for this model are the pull request comments of the tensorflow repository on GitHub.
In particular, of all the pull request data (commit comments, review comments, events, exc.) only the rows with Type equal to PC (Pull request Comment) or RC (Review Comment) have been entered into the dataset.
These comments has been classified into 4 categories:


1. ML (Machine Learning), if the comment is about specific machine learning aspects, algorithms exc.
2. Code, if the comment concerns either style and documentation in the code or maintainability issues or possible bugs exc.
3. Management, if the comment is about management activities like checking an activity status, assign a review to someone, trigger Jenkins CI
4. Other, if the comment doesn't belong to any of the above categories


Intended uses & limitations
---------------------------


One possible use of this model could be to label the pull request comments, clearly only on GitHub repositories that are about Machine Learning.
In this way a developer, before reading a comment entirely, can have a preview of what that comment is about.


### Training hyperparameters


The following hyperparameters were used during training:


* batch\_size: 32
* num\_epochs: 20
* optimizer: {'name': 'Adam', 'learning\_rate': {'class\_name': 'PolynomialDecay', 'config': {'initial\_learning\_rate': 2e-05, 'decay\_steps': 280, 'end\_learning\_rate': 0.0, 'power': 1.0, 'cycle': False, 'name': None}}, 'decay': 0.0, 'beta\_1': 0.9, 'beta\_2': 0.999, 'epsilon': 1e-08, 'amsgrad': False}
* training\_precision: float32


### Training results



### Framework versions


* Transformers 4.26.1
* TensorFlow 2.9.0
* Datasets 2.10.1
* Tokenizers 0.13.2



### Training hyperparameters


The following hyperparameters were used during training:


* batch\_size: 32
* num\_epochs: 20
* optimizer: {'name': 'Adam', 'learning\_rate': {'class\_name': 'PolynomialDecay', 'config': {'initial\_learning\_rate': 2e-05, 'decay\_steps': 280, 'end\_learning\_rate': 0.0, 'power': 1.0, 'cycle': False, 'name': None}}, 'decay': 0.0, 'beta\_1': 0.9, 'beta\_2': 0.999, 'epsilon': 1e-08, 'amsgrad': False}
* training\_precision: float32 ### Training results ### Framework versions


* Transformers 4.26.1
* TensorFlow 2.9.0
* Datasets 2.10.1
* Tokenizers 0.13.2
TAGS
#transformers #tf #distilbert #text-classification #generated_from_keras_callback #en #license-apache-2.0 #autotrain_compatible #endpoints_compatible #region-us 
 ### Training hyperparameters


The following hyperparameters were used during training:


* batch\_size: 32
* num\_epochs: 20
* optimizer: {'name': 'Adam', 'learning\_rate': {'class\_name': 'PolynomialDecay', 'config': {'initial\_learning\_rate': 2e-05, 'decay\_steps': 280, 'end\_learning\_rate': 0.0, 'power': 1.0, 'cycle': False, 'name': None}}, 'decay': 0.0, 'beta\_1': 0.9, 'beta\_2': 0.999, 'epsilon': 1e-08, 'amsgrad': False}
* training\_precision: float32 ### Training results ### Framework versions


* Transformers 4.26.1
* TensorFlow 2.9.0
* Datasets 2.10.1
* Tokenizers 0.13.2
TAGS
#transformers #tf #distilbert #text-classification #generated_from_keras_callback #en #license-apache-2.0 #autotrain_compatible #endpoints_compatible #region-us 
### Training hyperparameters


The following hyperparameters were used during training:


* batch\_size: 32
* num\_epochs: 20
* optimizer: {'name': 'Adam', 'learning\_rate': {'class\_name': 'PolynomialDecay', 'config': {'initial\_learning\_rate': 2e-05, 'decay\_steps': 280, 'end\_learning\_rate': 0.0, 'power': 1.0, 'cycle': False, 'name': None}}, 'decay': 0.0, 'beta\_1': 0.9, 'beta\_2': 0.999, 'epsilon': 1e-08, 'amsgrad': False}
* training\_precision: float32### Training results### Framework versions


* Transformers 4.26.1
* TensorFlow 2.9.0
* Datasets 2.10.1
* Tokenizers 0.13.2