transformers pytorch xlm-roberta generated_from_trainer dataset:massive base_model:FacebookAI/xlm-roberta-base license:mit endpoints_compatible region:us
---
license: mit
base_model: FacebookAI/xlm-roberta-base
tags:
- generated_from_trainer
datasets:
- massive
metrics:
- accuracy
- f1
model-index:
- name: scenario-KD-POST-DIV6-data-AmazonScience-massive-all_1.1
  results: []
---

<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->

# scenario-KD-POST-DIV6-data-AmazonScience-massive-all_1.1

This model is a fine-tuned version of [FacebookAI/xlm-roberta-base](https://huggingface.co/FacebookAI/xlm-roberta-base) on the massive dataset.
It achieves the following results on the evaluation set:
- Loss: 2.0040
- Accuracy: 0.8096
- F1: 0.7738

## Model description

More information needed

## Intended uses & limitations

More information needed

## Training and evaluation data

More information needed

## Training procedure

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 5e-05
- train_batch_size: 32
- eval_batch_size: 32
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- num_epochs: 5

### Training results

| Training Loss | Epoch | Step  | Validation Loss | Accuracy | F1     |
|:-------------:|:-----:|:-----:|:---------------:|:--------:|:------:|
| 4.7457        | 0.27  | 5000  | 4.6586          | 0.5547   | 0.4016 |
| 3.4196        | 0.53  | 10000 | 3.4566          | 0.6755   | 0.5700 |
| 2.9283        | 0.8   | 15000 | 2.9338          | 0.7286   | 0.6523 |
| 2.2088        | 1.07  | 20000 | 2.6995          | 0.7484   | 0.6778 |
| 2.1295        | 1.34  | 25000 | 2.5451          | 0.7582   | 0.6927 |
| 1.9602        | 1.6   | 30000 | 2.3551          | 0.7764   | 0.7218 |
| 1.8919        | 1.87  | 35000 | 2.2104          | 0.7893   | 0.7361 |
| 1.4886        | 2.14  | 40000 | 2.1850          | 0.7908   | 0.7449 |
| 1.4385        | 2.41  | 45000 | 2.1490          | 0.7935   | 0.7470 |
| 1.4356        | 2.67  | 50000 | 2.1004          | 0.7987   | 0.7569 |
| 1.3987        | 2.94  | 55000 | 2.0390          | 0.8032   | 0.7599 |
| 1.1264        | 3.21  | 60000 | 2.0869          | 0.8021   | 0.7606 |
| 1.1303        | 3.47  | 65000 | 2.0537          | 0.8046   | 0.7655 |
| 1.098         | 3.74  | 70000 | 2.0119          | 0.8067   | 0.7677 |
| 1.0142        | 4.01  | 75000 | 2.0310          | 0.8070   | 0.7669 |
| 0.9545        | 4.28  | 80000 | 2.0322          | 0.8073   | 0.7678 |
| 0.9423        | 4.54  | 85000 | 2.0184          | 0.8094   | 0.7720 |
| 0.9343        | 4.81  | 90000 | 2.0040          | 0.8096   | 0.7738 |


### Framework versions

- Transformers 4.33.3
- Pytorch 2.1.1+cu121
- Datasets 2.14.5
- Tokenizers 0.13.3
