
<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->

# wikitext-103-raw-v1-shuffled_default_gpt2_from_scratch

This model is a fine-tuned version of [gpt2](https://huggingface.co/gpt2) on an unknown dataset.
It achieves the following results on the evaluation set:
- Loss: 3.6101
- Accuracy: 0.3736

## Model description

More information needed

## Intended uses & limitations

More information needed

## Training and evaluation data

More information needed

## Training procedure

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 5e-05
- train_batch_size: 64
- eval_batch_size: 64
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- num_epochs: 10.0

### Training results

| Training Loss | Epoch | Step  | Validation Loss | Accuracy |
|:-------------:|:-----:|:-----:|:---------------:|:--------:|
| 5.6446        | 0.5   | 880   | 5.4390          | 0.2190   |
| 4.9746        | 1.0   | 1760  | 4.8573          | 0.2603   |
| 4.5819        | 1.5   | 2640  | 4.4930          | 0.2907   |
| 4.3038        | 2.0   | 3520  | 4.2540          | 0.3138   |
| 4.1286        | 2.5   | 4400  | 4.1119          | 0.3267   |
| 4.0129        | 3.0   | 5280  | 3.9992          | 0.3377   |
| 3.9101        | 3.5   | 6160  | 3.9169          | 0.3451   |
| 3.8433        | 4.0   | 7040  | 3.8462          | 0.3511   |
| 3.7703        | 4.5   | 7920  | 3.8059          | 0.3553   |
| 3.74          | 5.0   | 8800  | 3.7622          | 0.3583   |
| 3.6849        | 5.5   | 9680  | 3.7342          | 0.3610   |
| 3.6577        | 6.0   | 10560 | 3.7092          | 0.3628   |
| 3.6202        | 6.5   | 11440 | 3.6917          | 0.3649   |
| 3.6039        | 7.0   | 12320 | 3.6634          | 0.3676   |
| 3.5688        | 7.5   | 13200 | 3.6442          | 0.3700   |
| 3.5534        | 8.0   | 14080 | 3.6380          | 0.3705   |
| 3.535         | 8.5   | 14960 | 3.6293          | 0.3714   |
| 3.5305        | 9.0   | 15840 | 3.6157          | 0.3726   |
| 3.5134        | 9.5   | 16720 | 3.6139          | 0.3736   |
| 3.503         | 10.0  | 17600 | 3.6101          | 0.3736   |


### Framework versions

- Transformers 4.34.0
- Pytorch 2.1.0+cu121
- Datasets 2.14.5
- Tokenizers 0.14.1

{"license": "mit", "tags": ["generated_from_trainer"], "metrics": ["accuracy"], "base_model": "gpt2", "model-index": [{"name": "wikitext-103-raw-v1-shuffled_default_gpt2_from_scratch", "results": []}]}
transformers pytorch gpt2 text-generation generated_from_trainer base_model:gpt2 license:mit autotrain_compatible endpoints_compatible text-generation-inference region:us