
<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->

# bert-finetuned-sem_eval-english

This model is a fine-tuned version of [bert-base-uncased](https://huggingface.co/bert-base-uncased) on[Multi-Label Classification Dataset
](https://www.kaggle.com/datasets/shivanandmn/multilabel-classification-dataset).
It achieves the following results on the evaluation set:
- Loss: 0.1673
- F1: 0.8389
- Roc Auc: 0.8999
- Accuracy: 0.7046

## Model description

This model is a BERT base uncased model fine-tuned for multi-label classification of research papers into 6 categories: Computer Science, Physics, Mathematics, Statistics, Quantitative Biology, and Quantitative Finance. It classifies papers based on their title and abstract text.

## Intended uses & limitations

This model can be used to automatically tag research papers with relevant categories based on the paper's title and abstract. It works best on academic papers in quantitative research fields. Performance may be lower on papers from other domains or with very short abstracts.
## Training and evaluation data

The model was trained on a dataset of ~15,000 research paper abstracts labeled with one or more of 6 category tags:

* Computer Science
* Physics
* Mathematics
* Statistics
* Quantitative Biology
* Quantitative Finance
* 
The training data includes papers from arXiv and peer-reviewed journals.

The model was evaluated on a held-out test set of ~3,000 labeled research paper abstracts drawn from the same distribution as the training data.


## Training procedure

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 2e-05
- train_batch_size: 8
- eval_batch_size: 8
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- num_epochs: 5

### Training results

| Training Loss | Epoch | Step  | Validation Loss | F1     | Roc Auc | Accuracy |
|:-------------:|:-----:|:-----:|:---------------:|:------:|:-------:|:--------:|
| 0.1857        | 1.0   | 2098  | 0.1924          | 0.8143 | 0.8825  | 0.6760   |
| 0.1586        | 2.0   | 4196  | 0.1673          | 0.8389 | 0.8999  | 0.7046   |
| 0.1194        | 3.0   | 6294  | 0.1777          | 0.8361 | 0.8982  | 0.6989   |
| 0.0975        | 4.0   | 8392  | 0.1958          | 0.8312 | 0.8932  | 0.6946   |
| 0.0695        | 5.0   | 10490 | 0.2113          | 0.8315 | 0.8957  | 0.6918   |


### Framework versions

- Transformers 4.35.2
- Pytorch 2.1.0+cu118
- Tokenizers 0.15.0

{"license": "apache-2.0", "tags": ["generated_from_trainer"], "metrics": ["f1", "accuracy"], "base_model": "bert-base-uncased", "model-index": [{"name": "Multiclass-Classifcation[Bert]", "results": []}]}
transformers tensorboard safetensors bert text-classification generated_from_trainer base_model:bert-base-uncased license:apache-2.0 autotrain_compatible endpoints_compatible region:us